August
27th 2009
Are you part of the solution, or part of the problem?

Posted under: American history, jobs, students, unhappy endings

assessmentClio Bluestocking is cranky, again.  Why?  She has to “get dressed, drive up to the self-proclaimed ‘main campus’ (they aren’t, they just like to think they are) and sit in on one of those hideous Outcomes Assessment meetings run by the OA Borg, a group of True Believers who get paid a lot of money NOT to teach.”  Yeah, that’s a loser of a proposition twice-over:  1) a meeting, run by 2) “Outcomes Assessment” fraudsters. 

Oh, Historiann!  You’re just an old crank too, you might be thinking.  (You might be right.)  For those of you who remain blissfully ignorant of “Outcomes Assessment,” allow me to explain:  academic departments are asked to invent new tests and measures by which to measure their students’ progress, outside of all of those papers and exams we’re assigning to them in our classes to prove that our students are learning something.  That’s right, friends!  It’s redundant work for everyone, except for the “Outcomes Assessment” administrators who are paid to make $hitwork up for faculty and students who would prefer to be left alone to get on with the business of studying physical anthropology, or engineering, or zoology, or Romantic literature, or something else that has actual interest and value to people other than “Outcomes Assessment” administrators.

Why do I call “Outcomes Assessment” a fraud?  Let Clio B. tell the tale:

Meanwhile, at our college, in our department, we all settled on a truce. Do what they ask, generate the data and hand it over with as little disruption to our own teaching as possible. After all, the OA Borg kept telling us, “You are the professionals. You know your subject. We trust you to come up with the most effective assessment instrument. We will accept what you come up with.” If we didn’t comply, then, “THEY will come in and create one for you.”

Someone actually told that to me yesterday. I wanted to tell her, “c’mon! You are far too old to believe that, if we are good little professors, and do exactly what is expected of us, then THEY are going to leave us alone.” I did tell her, “THEY are going to take it over if THEY want to no matter what we do.” She has become assimilated. She honestly believes that she can limit the impact of the system by becoming part of it. Our pity for her prevents us from holding her in contempt.

THEY are actually already taking it over. All of that “we trust you” and “you are the professionals” and “we will accept what you come up with” is just smoke. You see, we came up with ours, and they kept sending it back to us. At first, it was just tweaking the language. “Students will understand the causes of the American Revolution,” had to be “Students will demonstrate an understanding of the causes of the American Revolution.” That sort of thing. Then, their revisions became more detailed. “How does this question show that students are demonstrating the causes of the American Revolution?” they wanted to know.

Ultimately, what they wanted from us was an essay-based exam. Ultimately, we refuse to give it to them.

But, as we know from long experience with stray cats, ex-boyfriends, and telemarketers:  if you feed it, it will just keep coming back!  If you engage with them, they’ll never let you go! 

The OA Borg becomes more and more intrusive with more and more forms and more and more rejection of our own “assessment tools.” They say, “we let you create your own tool because we trust that you know what you are doing.” Then, when we do, they send it back saying “this isn’t good enough.” The process repeats until they are satisfied, which means that they do have requirements for these “instruments,” (please! They are “tests”!) but to keep up the mendacity of “you create the instrument yourselves,” they have to coerce us into figuring out what it is and giving it to them. To keep up the lie that “we aren’t asking for a standardized or common exam” they have to get us to decide that a standardized and common exam is the best option.

Clearly, they do have to coerce our department because we don’t buy it and we have no respect for their process. They want us to give them honest-to-god exams that demonstrate education. We believe that we already do, they just aren’t the same exams approaching the questions of the course in the exact same way. They don’t accept that method because, if their numbers are going to mean anything, they need sameness. To achieve that sameness, they want us to give the same exam.

We rebel against that because we see that as standardized testing with common exams. We see that as not only an infringement on our freedom in the classroom but also the source of our students being untrained and even frightened to think on their own after 12 years of similar standardized testing. We teach in the humanities. Education in the humanities cannot be quantified in the same way as, say, business productivity. Yet, the way that the Borg describes their ideal education, you and I and the professors at Harvard or even the Sorbonne should all be giving the same exam with the same exact rubric so that that THEY can prove that education is happening. In fact, I often wonder if they expect the students to turn in the same exact answers.

This is a sick, cynical exercise.  Universities have been around for oh, going on 700 years or so.  We educate people, and at this point in history, we’re the institution that decides who gets to be middle-class in this country.  That’s a lot of power, a lot of power that politicians and “business leaders” want to get their mitts on and try to exert some control.  The way they do this is through “instruments” that they claim will quantify the “value” of what we do. 

Clio B. sounds the alarm ringing in the night:  “This is, at our college at least, essentially a very obvious creep toward No Child Left Behind at the college level because that worked so well at K-12.”  And what has been the dividend of No Child Left Behind, my fellow college and university professors?  According to Tenured Radical,

Higher education needs to start paying more attention to what is going on at the secondary level and vigorously fight unnecessary mandates, particularly the testing mandate associated with No Child Left Behind. Right now we in higher ed pretend this has nothing to do with us, but we are wrong. . . . [T]esting is homogenizing education, and that homogeneity is creeping upward: more and more students expect content in a class, as opposed to debate about ideas. Fewer students feel confident that they can write an academic essay without a structured “prompt,” although curiously, at my school, they write blogs, songs, plays, and film scripts with ease and creativity. What else can explain this gap between their unoriginal, often openly craven, papers and the vigor of their creative labors than the fact that they have learned to suppress critical thought in their academic work in order to score well on tests?

I have seen this too, in the past few years:  perfectly bright students are utterly paralyzed at the prospect of taking a bunch of primary sources, reading them through, and making an evidence-based argument in a 5-8 page essay.  This didn’t use to be the case at my uni.  When I came here in 2001, I was impressed that 1) all of my students at Baa Ram U. had the skills they need to succeed in college, and 2) many, if not all, were eager to be exposed to new ideas and to learn through reading and writing, and through experimentation and debate.  Many of my students are still like this, but I too have seen a kind of caution and even fear in my students at the prospect of having to complete an assignment that requires creativity or initiative.  Is anyone else seeing this, too?  What’s the NCLB generation like at your uni?

I’ll let Clio B. have the last word–well, almost the last word:

We [in Clio B.'s department] see a huge difference between feeding numbers to the Borg and education. We test education with our own assignments and exams, which are based on writing and through which we can see if students are improving their thought processes. We feed numbers to the Borg with this “instrument” thing and evaluate our students based on our own thing.

What’s that mantra people use when talking about cleaning up wasteful spending and redundancies in government?  Waste, fraud, and abuse?  How much is it really worth to give these people the illusion that they’re doing something useful?  (Let’s have faculty assess the “Outcomes Assessment” people–we could grade them on the value they bring to our unis!)  Hey, I’m a compassionate liberal–I don’t want anyone thrown out on the street in this economy.  Let’s give “Assessment Outcomes” people the opportunity to teach 3 or 4 classes a semester at our unis.  That’s probably the best way they can contribute to the education of our students–if in fact they care about that at all.

67 Comments »

67 Responses to “Are you part of the solution, or part of the problem?”

  1. Sympathetic Ear on 27 Aug 2009 at 10:20 am #

    I understand why you’re cranky, Historiann. In my corner of the woods, we’re also talking about how to head off the tendency towards standardized tests as the answer to “outcomes assessment.” No clear solution is forthcoming because the alternatives seem to be either 1) an essentially corporate test scored by machines or 2 )lots more uncompensated labor from a faculty that has seen its salaries frozen and research support slashed.

    I’m hoping that at least in my history department, we can head off the worst by creating our own “instrument,”one that can demonstrate that our students have mastered some historical thinking, research and writing skills by analyzing their capstone senior projects. Perhaps naive of me.

    The worst possible quagmire for us to get into in my opinion — as Clio B’s experience seems to suggest — is anything where we canonize specific knowledge content. Once we assign specifics to “demonstrating understanding of the causes of the American Revolution, ” we turn college history into group-think and indoctrination, don’t we? How dreary. And I can only imagine (with dread) what this might mean for the history of women and gender.

    So here’s a question you don’t address. What’s driving the OA Borg? I don’t think it’s malice. I don’t think it’s driven by the federal government. I think it’s about marketing and rankings.

  2. Indyanna on 27 Aug 2009 at 10:26 am #

    Oh My God, I’m so not ready to begin thinking about this before the year ever starts. I’ve been saying this for years, although far more foamingly, and thus less eloquently, than Clio B., or than Historiann in her annotations on the text. Our department has been saying all of the same surrenderist things reported above. Let’s just give them what they want and they’ll go away. The only thing approaching a solution would require confrontation on a level probably not even imaginable in this generation of faculty. It would involve refusing, and if necessary, physically barring the Borg from the self-denominated Humanities part of the campus.

    But, I’ll get eye-rolled enough as soon as I start this rant in our first actual meeting on this subject; no sense getting eye-rolled on-blog. I can’t even grow out a respectable gray “faded-hippie” ponytail anymore.

    Just a couple of thematics, though, under which resistance could be arranged, at least rhetorically:

    Epistemological: There ARE no outcomes to the pedagogical process, nor should there be. “Outcomes” happen when the “content” we teach interacts with that midnight phone call from the police a decade later, saying “we have your [child] down here and need you to come down and talk.” Or that conversation with your family M.D. on “sure, you can get a second opinion, or a tenth opinion, but this is what these pictures are showing…” Then you find out what your education amounted to. As I’ve said before, the obituary is the only good assessment instrument available out there.

    Ethical: Our national association, the [Your Letters Here], doesn’t allow us to manufacture data in this way. If a student did this, they’d be out of the program– maybe in front of a grand jury if funding was involved. So no, we can’t just fill in your wackdoodle matrix here and hit send. Put that in your “Electronic Evidence Room,” and no, tomorrow doesn’t look like a good day for us to meet with the Assessment Team either… This would require the national associations to take stands, though, and they’ve been co-opted.

  3. Historiann on 27 Aug 2009 at 10:29 am #

    Sympathetic Ear–my department assesses the rough and final drafts of both undergrad and grad papers. It’s a hassle, but it’s better than content-specific quizzes, as you note.

    In my state, I think the OA borg is something dreamed up by right-wing state pols who demand “accountability” for the miserly 11% of my uni’s budget they contribute. Or rather, it was dreamed up by my uni’s administrators as a strategy for keeping the pols at bay, but at what price? Why not just say, “Our professors and instructors are well-trained and highly-skilled professionals whose students earn X number of degrees per year, and who are now working in Y countries in Z different professions.” End of story.

    It may also be about marketing and rankings. But more to the point–as both Clio B. and TR suggest–it’s all a part of de-skilling the teaching professions, and whose interests does that serve?

  4. Historiann on 27 Aug 2009 at 10:32 am #

    Indyanna–I thought this one might get your juices flowing. Maybe we should propose “incomes assessment.” Who’s coming into our schools? Why do they think getting a college degree is important? Why are they taking out loans to get an education?

    It’s not like we’re hurting for “customers,” now, is it? And in this country, isn’t the “customer” always right? (N.B. I do not endorse the “customer service” model of education, but I just want to note that my classes are over-subscribed this year, and we’ve raised our class caps by 2 across the History department. We are not a subsidized entity that’s producing nothing–we’re a dramatically undersubsidized, underfunded institution that’s being asked to do more for less.)

  5. perpetua on 27 Aug 2009 at 11:12 am #

    I don’t mean to take the conversation in a different direction, but it does pertain in some ways to “outcomes” and the market-customer-consumer driven education that we’re all being forced into. . . Have you seen this recent gem at NYT.com?

    http://thechoice.blogs.nytimes.com/2009/08/27/raising-questions-about-why-college-is-so-costly/?hp

  6. SouthernProf on 27 Aug 2009 at 11:34 am #

    The source of this outcomes assessment “stuff” is partly all of the things described in the comments. It is overseen by regional accrediting agencies (SACS where I teach) who get their marching orders from several sources, including the federal government, other funding agencies (such as state governments), and the general business-sponsored “need” for accountability. Accrediting agencies are the centralized entities that enable those worried about educational accountability to put real pressure on colleges and universities through the back door. NCLB is a similar model of enforcing educational accountability that comes with real teeth in the form of levels of competency designations and threats to funding (our future?). My view is that at the university level assessment is a combination of good intentions, bad mechanisms, and corporate models masking a right-wing attempt to alter curriculums without having to publicly debate the merits of doing so. See also state-level pressures to change and homogenize core curriculums at all of their state campuses.

  7. Historiann on 27 Aug 2009 at 11:38 am #

    perpetua–I hadn’t seen that. Bauerlein is a well-known academic crankypuss who serves as a somewhat more openly-aligned-with-the-right-wing and smaller-time version of Stanley Fish. If only liberal arts professors were the source of all evil in the world!

    The reason that college and university education costs so much more than it used to certainly isn’t because universities expect their liberal arts faculty to conduct research. (We’re the best bargain going in terms of bang for the research buck.) The biggest reason is that state and the federal governments no longer subsidize universities at the levels they did back in the 1940s-1970s. Part of the cost rise is due to the fact that colleges and unis are expected to DO so much more for our students. Who do you think pays the bill for those vast, vast banks and endless labs of new computers, maintained and refreshed with the very latest in software? Dorms are also much more de-luxe, and more expensive to maintain (since EVERYONE has to have cable TV in their dorms, of course! etc.) Not to mention, the proliferation of administrators at every level–administrators who make more money than the teaching faculty. (Administrators like our friends in “Outcomes Assessment!”)

    But, the NYT blog readers should know that there are colleges in the USA where the faculty are paid just to teach. They are called community colleges. Somehow, I don’t think the average NYT reader is going to like that as a solution to all of those faculty fatcats with their *hyooooge* research budgets of $450 annually!

  8. Historiann on 27 Aug 2009 at 11:39 am #

    Southernprof: right on. “See also state-level pressures to change and homogenize core curriculums at all of their state campuses.”

    Been there, done that, got the tee-shirt!

  9. Cassandra on 27 Aug 2009 at 12:07 pm #

    Historiann said, “When I came here in 2001, I was impressed that 1) all of my students at Baa Ram U. had the skills they need to succeed in college, and 2) many, if not all, were eager to be exposed to new ideas and to learn through reading and writing, and through experimentation and debate. Many of my students are still like this, but I too have seen a kind of caution and even fear in my students at the prospect of having to complete an assignment that requires creativity or initiative. Is anyone else seeing this, too? ”

    Yes…and it’s connected to these questions:

    “Maybe we should propose ‘incomes assessment.’ Who’s coming into our schools? Why do they think getting a college degree is important? Why are they taking out loans to get an education?”

    There are too many undergraduates who are not capable of doing college-level work flooding some universities. I received my first GA teaching assignment in 2001, and I quit grad school in 2007 because the quality of students dropped so severely that I wore myself out psychically trying to deal with their inability to do the same sort of work I was asked to do in high school.

    Reading 2 chapters of a pre-digested textbook (complete with bold-faced key words and on-page definitions!) was too much. Writing a 2-3 page paper was too much. Doing library research was too much. Showing up to class regularly was too much. And the post-semester customer service surveys they filled out always blamed my instruction for their trials and tribulations after receiving their first C ever. It had nothing to do with the fact that they didn’t pay attention or take notes or follow directions. I’m just a big, prejudiced meanie-pants! And I smell too!

    What got me, though, was how poorly so many of them did on multiple choice exams! One would think, with all that standardized testing they were tormented with, that they’d know how to take a MC exam without throwing a conniption. Oh, but they *LOVED* essay exams…and fully expected partial credit just for writing even the most inane, inaccurate tripe for a response. “But I used that one word right!” was screeched at me at least once a semester.

    This is clearly an effect of No Child Left Behind, which is only a pointer to trends that had started before it was enacted.

  10. Kathleen Lowrey on 27 Aug 2009 at 12:27 pm #

    One of the best things I’ve read lately is Marc Bousquet’s book _How the University Works_. He’s really good on this kind of administrative parasitism and how it’s slowly killing (well, actually not so slowly — according to him it only got rolling the late 60s/70s) the teaching, learning, and research that properly make up universities. How much does this OA stuff cost? How much do OA consultants make? What if all that cash went into hiring more actual full-time profs to teach in actual classrooms?

  11. Comrade PhysioProf on 27 Aug 2009 at 12:31 pm #

    We educate people, and at this point in history, we’re the institution that decides who gets to be middle-class in this country. That’s a lot of power, a lot of power that politicians and “business leaders” want to get their mitts on and try to exert some control. The way they do this is through “instruments” that they claim will quantify the “value” of what we do.

    Yep. And these greedy sick corporate fascist motherfuckers want to turn the universities into yet another instrumentality of their depraved authoritarian fantasy of turning everyfuckingone in the entire country–other than them, of course–into obedient, compliant, corporate drones happy to work for minimum wage at shitfuck Walmart jobs. GO USA!!

  12. GayProf on 27 Aug 2009 at 12:32 pm #

    This is a bit off topic, but I have recently been surprised by the number of colleagues who have advised me (and other untenured profs) to cut the amount of reading that we assign. Students, after all, give lower evaluations when they have to worker harder. Really, if we want our teaching record to look great, we should bow to their shorter attention spans.

  13. Historiann on 27 Aug 2009 at 12:34 pm #

    Kathleen, maybe that’s the question faculty should ask instead of meekly complying with “OA” demands for data: How much money do you make? How is your work “mission-critical” to this university? Please explain in 10 words or less how your work improves the work that faculty and students do here?

    Cassandra, how appropriate is your name! And, in the words of History’s Greatest Monster, “I feel your pain.” I’m sorry you were driven from grad school because of this. One of the great things about Baa Ram U. is that faculty are not required to work harder on a student’s education than the student is willing to work. Yes, it means that a solid 20% of my students get Ds or Fs in my freshman surveys–but hey, as someone (Clio B.?) once said, you can lead a horse to education…

  14. Historiann on 27 Aug 2009 at 12:38 pm #

    GayProf–ugh. Maybe they just want company in catering to the short attention spans so that they don’t look like captitulators?

    I’m so, SO glad I work in a department that keeps teaching evals. in their (extremely limited) place. I once taught for a semester in a really, totally, terminally screwed-up department, but the dept. chair there gave me some incredibly useful wisdom and insight into student evaluations. She said, “Oh, well–they’re useful for diagnosing a colleague with a serious drug or alcohol problem who isn’t showing up to teach hir classes, but other than that, they’re not terribly important.”

  15. Indyanna on 27 Aug 2009 at 12:39 pm #

    The “Outcome” they want is that the school year starts, the faculty meets, some guy-in-a-tie with the requisite MBA/EdD/K-Street policy shop credentials is introduced as the new (or maybe the “Founding”) Associate Deputy Provost for Curriculum Meddling, who claps hir [oh, wait, I already said "guy-in-a-tie"] hands and says: o.k., we’ve acquired this new curriculum initiative package from Vendor A, LLC. This is what I need you to do, this is what I need you to do, this is what I need you to do (each “this” being accompanied by a finger pointed to a valued “team” member). O.K? O Kay!! (clap!!) let’s have a great year… and with that, the staff disperses to the classrooms. That’s what the draft “Vision Statement” says if somebody thinks to stab it in the backside with a syringe of truth syrum.

    Just like back at More Science High, in Anytown, USA. Are we in Goshen yet?

  16. John S. on 27 Aug 2009 at 12:43 pm #

    I am handing in my teaching portfolio today, so that my department can assess whether or not I have met the teaching standards we expect to tenure an assistant professor. Huzzah! What a post to read! Seriously, though, one of the things that I most appreciate about the process at my Uni. is the level of delegation. It is certainly appropriate for our College Personnel Committee and our Dean to insist on high quality teaching. But it also seems very appropriate to me that it’s people in my own field judging whether or not I haz skillz in the classroom. The physicists trust the historians to be able to teach their field, and vice versa.

    But Historiann–I’ve been at my current U since 2001 as well, and I have seen the same kind of caution you describe in my students the whole time, though it is increasing. I’ve wondered if it is a function of the distribution of majors, or even the way that the University allocates financial aid. The number of history majors we have has gone up, but I still have many science majors in my classes. And I have found that they are often very loathe to step outside the box, so to speak, if it might harm their grade.

    The increased caution may be money-related. I have had students (mostly non-majors) tell me that as our fees have gone up and up (more than doubled since I got here), they are more reliant on financial assistance. Since the most generous assistance is merit based, they focus more and more on keeping their GPA over a certain line. I think that this can breed caution as well. To some extent, I can’t blame them. Stretching intellectually does sometimes lead to biting off more than you can chew. It’s inherently risky (as I learned in that Physical Chemistry class in college! I still can’t calculate the entropy of Carbon Dioxide at absolute zero.). And if trying and failing (or, more specifically, trying and getting a B- in the class) might disqualify you from the aid package you have, why do it? This is a peril of a system of financial aid that is so heavily merit-based, instead of need-based.

  17. Kathleen Lowrey on 27 Aug 2009 at 12:52 pm #

    It sounds like Clio isn’t in a position to challenge this stuff, because she doesn’t have tenure; and I think it is sneakily hard to challenge alone because any individual objection can sound like defensiveness: “oh-ho, why don’t YOU want your outcomes assessed?”. That’s where I think something like Bousquet’s book is really helpful and important — it contextualizes this kind of thing in terms of a pernicious trend across North American universities, where more and more money is being spent on administrative costs and less and less on classroom teaching. Bousquet suggests that the AAUP is kind of “on” this, which is great, but I don’t know how widespread that knowledge (it’s not just you, it’s not just your university, it’s a structural pressure with a 40 year history and a continent-wide reach) is among faculty generally. I for sure didn’t know all the stuff in his book until I read it.

  18. Ignatz on 27 Aug 2009 at 1:27 pm #

    This is totally off topic–maybe for a new thread. But I’m curious, professors and instructors: do you think the AAUP responds to your needs? Do you think it’s DOING (forgive the shouting) anything? If not, is it because it’s not a union?

    The organization formed over the free speech issue; tenured professors got fired for speaking out against World War One. Firings over free speech happen rarely now, and perhaps the AAUP deserves the credit.

    However, speaking as someone who’s never gotten tenure, I find the AAUP’s real-life impact nil. I think faculty need unions to combat OA Borgs and the other ish foisted upon them. Especially non-tenured, non-tenure track, and adjunct instructors.

  19. Indyanna on 27 Aug 2009 at 1:48 pm #

    Our union hasn’t said boo about assessment, or basically about any of the status of the faculty and professional prerogative issues that are implicitly in play here. Anything that can be structuralized, mechanized, modified, managed, etc., will find constituents on the labor as well as the management side of things. Again, it’s the national associations that should move. But they all have big townhouses on The Hill too, and move in the same wingtip circles that the Ed.docracy does. So they end up speaking the same jargon and sitting through the same subcommittee “markup” meatgrinders that create products that explode in your suite in the form of so many administrative hand grenades. Sad but true.

  20. Dr. Crazy on 27 Aug 2009 at 2:01 pm #

    There is a way to do outcomes assessment that doesn’t involve standardized tests, but the problem is that it requires a huge influx of cash (for the computer software/server space/ etc. to make the process reasonable, for support staff to manage the data, etc.) and a willingness on the part of f-t faculty (whose numbers are shrinking every day) to take on a fair amount of uncompensated labor (with the understanding that research expectations will be just as high, they’ll still have just as much teaching and service, etc.).

    So I’m willing to believe that assessment – done right – isn’t actually the enemy. The problem is assessment without money (one piece of the broader calls for excellence without money). You can’t expect something (all of this extra work) for nothing. And yet, that’s exactly what state legislatures, accreditors, taxpayers, etc. do all. the. time.

  21. Historiann on 27 Aug 2009 at 2:20 pm #

    Dr. C.–I think that assessment *is* inherently opposed to actual education, but I think you make a great point about the injury of uncompensated labor heaped up on the insult of having to engage in OA in the first place.

    Assessment without money! Awesome.

    Kathleen–yes, Clio B. is very vulnerable. I think any individual who stuck hir neck out would be, but esp. of course the untenured. It seems like fighting this at the departmental or college level is the only way to go.

    As for John S.: I’m sure u haz mad t-ching skilz! Interesting point about caution being money related–I’m sure you’re right, but I’m talking about caution on a more micro-level. As in, being paralyzed by the prospect of an argument-driven short essay. I’m not necessarily looking for conceptual or analytical brilliance–I’m just looking for someone to write something in which the answer is not implied in the starting question. (As opposed to the weekly short essays my students write on their reading assignments, which are very limited and highly structured yay or nay questions or summaries.)

  22. Dr. Crazy on 27 Aug 2009 at 2:48 pm #

    Here’s the thing: assessment such as what you’re discussing in this post isn’t actually *about* education. It’s about determining whether *programs* are performing adequately, succeeding, whatever. It’s not about students, except in a tangential sort of way. So the sort of assessment stuff this post discusses (and that I have become WAY too knowledgeable about this summer) is more about the efficiency of the organization and how the different programs within it fit together – and ultimately it’s supposed to be useful in order to decide on areas toward which resources should be directed, whether for innovation or improvement.

    All of that’s fine and good, but it’s totally not about educating actual students. It’s about organizational units. It has nothing to do with ideas, thinking, knowledge, or action. The problem is, if people who are actually doing the educating of students aren’t the ones driving assessment, then INSANE things happen, such as a state-mandated standardized test in math that was administered to random sections at my university last winter. And no, these were not sections of math courses where the test was administered, and yes, this interrupted classroom instruction time for something nuts like 500 students.

    I don’t think that the assessment borgs will leave us alone if we play nicely, but I don’t think opting out is an answer either. Opting out can be just as or more detrimental to students as opting in (as in the above example). I’m also not hopeful that there is any reasonable way to resist the drive toward assessment, even if at the department or college level. At my institution, we’ll lose accreditation (just went through the process and we’re on probation) if we don’t get on board with this assessment stuff. That’s the bottom line. How does losing accreditation serve students? Or serve our interests as educators?

    Ok, now I’m depressed. I’m going to go read some theory or something :)

  23. Nikki on 27 Aug 2009 at 3:01 pm #

    There are multiple things that bother me about assessment. One thing that really gets me is the same problem I have with semester teaching evaluations—the students are not held accountable for their own learning. Assessment (or at least the way it is done at my uni)implies that if x% of your students are not learning a goal then the fault is with the teacher. There is no way to assess the amount of time or effort the student did or did not put into the class. So, if my student cannot articulate the causes of the American Revolution that must be my fault, not the fault of the student who spent his or her semester getting drunk and skipping class.

  24. Ignatz on 27 Aug 2009 at 3:22 pm #

    In my field–composition–I think there may need to be some sort of assessment at some schools–depending on institutional values.
    As Stanley Fish wrote in the NY Times the other day, students need to come out of first-year comp with a set of skills. He even had them right: imho, comp should teach the ability to write a clear and grammatically correct argument. (Actually I don’t think grammar matters much in “the real world,” but it does to some other college teachers.) When I taught comp as a TA, we had to lead discussion sections for 2 years before we got our own class;received intensive instruction in how to teach argument; had to teach argument; eventually got to do it however we wanted, but had our course description and syllabi scrutinized; got observed by faculty. I think most students exited our courses knowing how to write a real, clear,and mechanically correct argument.
    Then I taught at a university where TAs– new MA students– got one week of pedagogy training before being thrown into 2 sections of comp teaching. I seriously doubt these newbies benefited from the structured, vigorous supervision I’d had. Ok, these students came from a lower-tier school than where I’d TAed. But geez–when I taught them in upper-division courses, I was usually unpleasantly surprised by their argumentation, clarity, grammar…
    A stronger commitment to comp on the department’s part would’ve helped. But assessment could have either a) showed the department that it needed a stronger commitment or b) failing that, identified students who needed more help learning to write.

  25. Buzz on 27 Aug 2009 at 3:40 pm #

    So far, I have found assessment to be a fairly useful undertaking. A couple years ago, our university decided that we were simply not doing enough assessment to satisfy the SACS accreditors. However, the job of assessment was not assigned to some bureaucrats in the central administration; it was parceled out to the departments. We have designed and implemented our own assessment standards. These are then evaluated by a university-wide faculty committee, but they have not been at all intrusive. The committee just seems to be interested in ensuring that we have all the required procedures in place in time for our re-accreditation.

    In our department, we have two separate committees handling our assessment; one covers undergraduate education, one graduate. Myself and one colleague have basically run the graduate student assessment committee. We looked at the structure of our master’s and doctoral programs and what we wanted to the students to be learning. And we discovered that, honestly, a lot of information we would like to have about our graduate students wasn’t being collected in a useful fashion. So we started collecting it, and that has led us to identify several aspects of the program that weren’t performing the way we had hoped. For example, we had an oral exam as part of the doctoral qualifiers. We found that the students hated it and that it just wasn’t that useful for pinpointing what students did and didn’t know. So we got rid of it and otherwise re-organized the program, in ways we hope will improve it.

    This worked because the job of assessment was assigned to the departmental faculty, who were overseen at the university level by other faculty. That last point is probably very important; we didn’t have to waste time justifying ourselves to any borg drones. We did have to do some cosmetic things, like include learning outcomes on our syllabi, but nobody outside the department policies the content of the outcomes, nor do they show up to check on whether what we are teaching matches what we wrote down. (I heard a horror story about that kind of thing at the secondary school level from my wife yesterday; she has a fellowship assisting middle school science instruction this year.) And in areas where things didn’t seem to be too wrong, nobody’s been upset that we haven’t done much. Our undergraduate program has grown to be the largest its even been over the last five years; the undergraduate assessment committee basically decided that we were doing the right things and hasn’t been that active.

  26. Kathleen Lowrey on 27 Aug 2009 at 4:11 pm #

    Buzz — I think your story really gets at the key issue; your assessment was done by faculty, for faculty. I don’t mind “unpaid” service work (actually, I do get paid — I get an annual salary compensating my research, teaching, and service duties that adjuncts would love to have) if it keeps administrative powers in faculty hands. Again, I know I am a broken record here, but Bousquet’s book is just so good on how universities have fewer and fewer f/t faculty per student, more and more adjunct faculty who are forbidden any administrative say in anything, and then lots and lots more f/t, highly compensated, fully pensioned administrative staff running things. Clio’s story sounded like the worst-case scenario of a consulting team being hired by administration and paid to come in and bully temporary faculty for ends that are lucrative for the consulting team (and which justify the existence of university admin staff — “look! we organized and ran an OA exercise!”) but that will actively degrade classroom practice for students and profs alike.

  27. Indyanna on 27 Aug 2009 at 4:42 pm #

    On the politics of assessment, i.e., what to do about the accreditors? It seems to me that in many if not most political societies, when fundamental interests of groups and institutions are at stake in zero-sum ways, what you do is…. try to set big scary fires in the adversary’s rear areas, i.e. raise fundamental questions about their legitimacy. Who are the accreditors, what do they do when they’re not coming around on per diems pouncing on programs, whence derives their authority? Americans have never been fond of murky bureaucratic entities that don’t have storefront offices in their towns, and whose authority seems to strike randomly from the blue, like lightning. Except for these eerie transnational “anti-doping agencies” that are gaining a stranglehold on global athletic competition, it would seem hard to find any entities that would be more vulnerable to exploiting that kind of populist aversion than regional alphabet-soup accreditors. Anything for which the cuddliest reference available is something like “Middle States” ought to be a big fat target.

    So: “black helicopter” analogies, allusions to “intellectual death panels,” investigatory probes– blog or print–about conflict of interest issues, whatever. The health care crazies have been recently showing us how you can thwart or at least hobble supposedly inevitable change in pluralist societies. Make the flannel suits play dee-fense for a change, which they probably aren’t good at. Profs. are supposed to be the smart ones, right? Is there really a grass roots constituency out there, ordinary people who
    wake up in towns like yours thinking: “how do we know that colleges and universities are really educating people when they award these degrees? What do tests and grades really mean?” Without push-polling, I doubt whether you could find any. There is some measure of parody herein, but really, who accredits the accreditors?

    There also seems to be a lot of epistemological confusion across these comments about what assessment really is, but this comment is too long already.

  28. Digger on 27 Aug 2009 at 5:39 pm #

    Ignatz, wow, you got training before thrown into a classroom? The place I got my MA had us submit syllabi before we arrived, and then boom, teaching. Fortunately, my mom is a teacher and I had several good instructors to model my course after (and to ask for help), so I was more in the swimming ledger than the sinking ledger. Others of my cohort… not so much. Some actually “taught” some very, very frightening things to their Anthro 101 students. Racist things and things about looting sites, yikes.

    This drive to remove all subjectivity from people’s interactions, to standardize everything, makes me bananas. The assumption seems to be that we are really idiots, and incapable of deciding for ourselves what is appropriate. I see it as part of the same trend from the previous post, that has parents drive their kids to school because some people are bad people; where doctors prescribe a bazillion tests to cover their butts; etc. Brave New World, anyone?

  29. James Stripes on 27 Aug 2009 at 6:23 pm #

    How about a mass resignation with the instructions that the OA Borg team can teach the now professor-less courses? In a few weeks they will be begging the resignees back with pay increases.

  30. Dr. Crazy on 27 Aug 2009 at 7:01 pm #

    What Buzz describes is assessment working as it should – and I HOPE that it’s how it goes at my institution (there are some good signs). My concerns about faculty workload and how assessment mandates affect that are at least some extent institution-specific. Our university keeps asking faculty to do more and more, particularly in arts and sciences where we have the highest teaching load on campus, while at the same time full-time positions are being slashed from A&S. Research expectations have increased, there is an expectation that faculty do both public engagement as well as university and department service, and faculty are being expected to roll out new graduate programs and to supervise more and more independent student projects (both in grad programs and in undergrad contexts). In that case, something has to give (or is going to give, even if it’s not acknowledged) in order for people to take on more. What concerns me is that what faculty have most control over is what happens in the classroom, and thus what “gives” is what happens there. Not because that’s what *should* give, but because it’s the one thing that the bean counters (at least at my institution) seem often to acknowledge the least.

  31. Another Damned Medievalist on 27 Aug 2009 at 7:40 pm #

    I’m going to have to respond in a longer post at my place this weekend, I think, because, well, Clio Bluestocking’s post just pisses me right off. I have to deal with exactly this attitude from one of my colleagues every freakin’ meeting we have to discuss our program and our curriculum, and it is impossible to move forward. Assessment can work very well, if faculty are willing to not only buy into it, btu do the damned work. (The fact that we aren’t paid to do extra work is a different issue)

    I used to work in manufacturing. We had CE and ISO-2001 certification. Basically, these are quality control standards, and are an industrial equivalent to outcomes assessment. Here’s how it works, basically: company sets procedures for doing stuff, and documents them, and agrees to make all employees responsible for knowing and following the procedures. If that doesn’t get done, then you lose certification. Who sets the procedures? the company, usually based on what is already being done. What if a procedure needs changing? Fine, it gets documented, and employees learn and follow the new procedure.

    The same is true of good assessment, at least on a student- and departmental/program level. The department agrees on what things they think are important for students to learn. This looks upwards to what the institution as a whole things important (generally found in a Mission Statement, Strategic Plan, and Gen Eds) and downwards to individual faculty members and courses. So, for example, one of our outcomes has to do with differentiating primary and secondary sources, and analyzing them to use as evidence in creating historical arguments and narratives. Outcome easy. Assessment hard. Why? because of that one colleague, who claims that, like pornography, we all know good analysis when we see it. But for at least half of the department, we don’t feel at all threatened by having to sit down and articulate some of the things that characterize good analysis and putting them into a basic rubric that can be used for papers and exams that require the use of primary sources.

    We don’t teach the same areas, but we find it useful to *have* to articulate clearly what it is we historians do. And doing that often helps us to craft better assignments that focus on teaching what we say is important to us.

    Maybe this is in part because I have a lot of friends who teach in Europe and are used to external examiners, group grading, and double-blind grading, but I honestly don’t find any of this a threat to my teaching OR my freedom in the classroom. I find it a way to constantly re-evaluate and refine my teaching. But then, I have always done that with my exam results anyway. If *all* the students do badly on an exam, I tend to think it was a badly crafted exam that didn’t reflect what I taught, or was unclear to the students, and try to figure out ways of correcting in the future …

  32. Another Damned Medievalist on 27 Aug 2009 at 7:56 pm #

    And I should make it clear that I have nothing against Clio Bluestocking — I’m just fed up with the attitude that often seems to be promoted by a section of older faculty who are so hidebound that they can’t manage a little self-reflection. When I see junior faculty espousing the same attitude, it’s usually at places where the senior faculty have it very good, and have historically had no obligation to make sure that course X as taught by faculty member 1 bears any resemblance to exactly the same course X taught by faculty member 2. That makes no sense. Our colleagues in English can teach courses that use entirely different materials, yet are functionally the same and have the same results. Psych departments can give joint exams and have students pass, even though the individual faculty members may spend more time on one issue and less on others.

    Sorry — I’m just ranting now, aren’t I?

    I just hate when a perfectly good concept with a proven track record gets trashed because it’s been badly executed.

  33. Historiann on 27 Aug 2009 at 8:05 pm #

    But, ADM: what’s the rationale behind “Outcomes Assessment?” Was there any proof that students weren’t being educated in American universities before OA administrators and specialists came along? Do Buzz’s students really learn THAT much more because he now includes a “course objectives” thingy on page 1, with a bulleted list? (Do mine?)

    I think Clio’s post illustrated quite nicely the gap between OA and actual education. And, I stand by my prescription that OA administrators should instead teach 3 or 4 classes, if they want to improve the quality of education so desperately. But from where most of us sit–at least, from where most people who have commented on this thread sit–it’s pointless makework that isn’t rooted in any evidence that the makework will improve the REAL work we’re supposed to be doing.

    As Fratguy said in a comment on an assessment post last year (and please excuse the sciency talk, humanities folks):

    Any exercise in quantification and prediction runs into the dichotomy of applicability vs generizability. A hypothesis is tested to generate data, for example how efective a particular teaching technique is. If more a priori stipulations and specifications are placed on how a particular technique is tested (ie this particular technique, by professors from this particular school of thought, to this set of students with a minimum level of education etc etc etc) the data that is generated will more accurately reflect expected student outcomes WITHIN THE DEFINED CONFINES OF THE EXPERIMENT. In order to turn the experiment from a mere parlor trick into something that is generalizable, that can be picked up off the shelf and used universally, preconditions need to be lifted and the data becomes subsequently much less robust.

    By the descriptions in this post is appears that there are no experimental conditions imposed on the data that is gathered. It is infinitely generalizable and therefore nothing more that a numeric description of what has happened. Without preconditions or descriptions of the experiment the “data” is utterly meaningless. It cannot tell you how to improve your your teaching, it does not even define “improve” other than as a change in the data, maybe. People who traffic in tautologies should be called out for the BS artists that they are.

    The resulting data, or rather descriptive numbers, though meaningless are nonetheless very potent in the wrong hands. It is clear that management loves this stuff as a means of justification of self and of predetermined ends. If the data is meaningless it can be made to mean anything.

    This is why so few of us have faith in the value of the exercise.

    And, I endorse Kathleen’s endorsement of Marc Bosquet’s How the University Works. Great stuff.

  34. Rad Readr on 27 Aug 2009 at 8:09 pm #

    DATELINE: The Front Lines. As the administrative lackey responsible for promoting the assessment process in my college, I say. Thank you, thank you, thank you. And I’m embarrassed (seriously) to admit that I am part of the problem.

    As I followed the thread of discussion, my view after doing this stuff and serving on committees comes closest to the following:
    Clio is right that there is a zombie quality to the assessors — it keeps coming back. Southernprof zeroes in on the entity promiting this: regional accreditation agendies. Kathleen L. then asks the key question: where’s they money and who is making it? These three are connected because the agencies promote a multi-year process that keeps their people employed.

    At my university, entire positions have been created for assessment (this during a time period when we are laying off people), and I believe we have several staff working on it full time, and a couple of administrators devoting a good chunk of time and energy to this — in part because of accreditation.

    A little assessment is good: departments should articulate what they expect graduates to take away from a major and evaluate whether the students are getting it, but this is different. We are talking about an insidious bureaucracy that makes its way from one university to another.

  35. Historiann on 27 Aug 2009 at 8:13 pm #

    Rad–thanks for chiming in, and I’m sorry that you’re part of the evil borg!

    Maybe your analysis of self-designed and for internal purposes only OA versus educrat-or-politican-imposed-from-the-outside OA explains the different opinions that have emerged on this thread. I’ve only seen the latter kind, of which you write. I’ve never heard of a department developing assessment from within, so that may explain my dim view of OA, i.e. that it is a fraud that feathers a lot of nests rather than giving departments the real resources they need to improve the quality of education: more faculty for lower faculty-to-student ratios.

    OA feels like the “clicker:” seems like a nifty idea, but then you realize it’s just a gimmick to distract you from the fact that you don’t have the resources (in my case, smaller classes at the intro level) to teach in the most effective manner.

  36. Clio Bluestocking on 27 Aug 2009 at 8:41 pm #

    A few points, since I seem to have started this discussion.

    First, I’m at a place that does not have tenure at all. The most we get are 6 year contracts, and that is after nine years of shorter-term contracts.

    Second, we do have a very active AAUP chapter and a lot of long-time activist on our campuses. Sadly, the AAUP is a bit preoccupied by a rather huge problem that may end up in a major newspaper very soon. Still, should they address this whole OA thing, I’m not sure that they wouldn’t get, “well, it’s coming so we might as well accept it.” When you talk to the people who say that, ask a few more questions about what “it” is, they do mean standardized testing. Also, because this demand for the OA business does come from accrediting agencies — there was a resistance to it, but the accrediting agency said they wouldn’t accredit — one measly AAUP chapter in protest is not going to make a dent in the larger problem. It would have to be a national push back.

    Third, and this is meant to address those who are describing a somewhat saner process or at least the the good points of the spirit of the process, particularly Another Damned Medivalist. My — and my colleagues’ (and we are all around the same age) — reactions are not against trying to become better teachers, or set standards for our own department. The problem is that WE ALREADY DO THAT. We already set up detailed course descriptions that include outcomes. We already are evaluated within an inch of our lives from student, peers, chairs, and deans. We already meet individually with our chairs and deans about our teaching, after both have observed us and looked at our syllabi, tests, assignments and grades in our courses.

    This is not a resistance to self-reflection and improvement. The resistance is to the assumption that we aren’t doing that already, and that what we are already doing is not good enough because it doesn’t collect the exact same, quantifiable data in the exact same way, from the exact same tests and the exact same assignments.

    If there is a flaw in our existing process, if we should make it more rigorous, then fine. But, what’s the point of maintaining the existing process of official evaluation if it means nothing, if the only thing that has any meaning for this OA borg — and the accreditors behind it — is this “instrument.” That is what is so infuriating and insulting.

  37. Another Damned Medievalist on 27 Aug 2009 at 8:45 pm #

    Historiann, I agree that OA is definitely part of a system that came in with people who seek to see academia as a business, and that’s problematic. But I honestly don’t see that it’s at all unreasonable to expect ourselves and our colleagues to reflect on what it is we do and why we do it, and whether we do those things well. But then I’m a medievalist, and we and the classicists are pretty used to being asked to justify our existence to the world, not to mention our departmental colleagues who often want to cut the single pre-modernist in order to hire in whatever field is hot in a given year (they never seem to want to give up a modern line or an Americanist line — those are ‘relevant’)

    I also used to work in a state where CCs were required to provide biannual program reviews. Some of it *was* bullshit, but you know? Sitting down with the other faculty in my program (History, econ, poli sci, international studies, and geography) and looking at where students took classes, when they took them, which majors were doing well, who was carrying the most FTEs — all part of the assessment process, although not at course level — made it easier for us to justify a request for another FT line in history, and to see whether or not we were really offering courses that were on a level with those at Flagship U. As it turned out, they were. Or at least, by looking at the numbers, we could see that our transfer students were doing as well as or better than native students at Flagship U and the two highest ranked State Us. It was a pain in the butt, but those things were also important when it came to funding and to arguing that the transfer programs should not be cut in favor of vocational programs.

    As for the course-level OA, I worked on a couple of Gen Ed OA task forces. A cross-curricular committee worked on things like the nebulous ‘critical thinking’ outcome, and how it should be assessed. Turns out that, no matter what the field, no matter whose rubric we used, once we had agreed on what demonstrating critical thinking looked like in a very general sense, we all marked essays within a couple of points of each other. Content is a different issue, obviously. But consider how many historians bewail the data dump essay that doesn’t answer the question. That implies that we think arguing a point well as a part of historical analysis or choice of narrative is important to us. So those are the skills we assess as a whole, and then leave the appropriate content to the experts — ourselves and our colleagues.

    Mostly, though I think what it does is get us to reflect on what we are doing, and how to do it better. We talk about how students have changed, but has our teaching changed with them? How can we meet the challenges of ensuring rigor while teaching people who have entirely different learning styles, unless we constantly re-examine what we are doing?

    I’m not saying that we have to bend over backwards because our students are not prepared. But even our best students access information in ways that didn’t even exist when we were at school. In many fundamental ways, they don’t have the same understanding of the educational process that we were raised with, and I’m not just talking about standardized tests. They have a lot of breadth in some ways, and very little depth. They aren’t used to committing things to memory, because they’re jacked in most of the time.

    It’s not necessarily wrong, but it is very different. If teaching is at all connected to learning, then it’s incumbent on the teachers to make what we do accessible.

  38. Cassandra on 27 Aug 2009 at 8:48 pm #

    I wonder:

    Does Another Damned Medievalist have tenure? Tenure-track?

    If either, then the self-righteous chuffing is problematic with regard to understanding Clio’s OA-at-a-school-without-tenure context.

    When one has some security that one HAS CONTROL over the nature of the OA, then perhaps one won’t feel so manipulated by the people demanding it.

    Unless, of course, lots of Kool Aid has been imbibed.

    One point AMD made that I doubt few of us disagree with is that some sort of “quality control” is useful for syllabi. But a list of “cover these 12 topics at some point during the semester, however you see fit, along with whatever extra you wish to add” is a bit different from “here’s your pre-planned syllabus, do not deviate from it, nor add anything not pre-approved.”

    OA seems to often be a good cover for political wonks to do the latter while pretending it’s the former.

  39. Another Damned Medievalist on 27 Aug 2009 at 8:56 pm #

    Oh — as an added fun piece, I had to input our assessment numbers a couple of weeks ago. One of the things I had to add was a comment that some of the numbers were meaningless, and that we’d be reviewing that and making changes. Why? Because it turns out that one colleague called all hir assessments X, and the entire course grade was based on assessment X times 10, while a couple of others used something they called assessment X only as a minimal part of the grade, more as an incentive to do the reading, and in an entirely different format. One colleague’s X was pretty much what everybody else called Y — or even Y1. So we weren’t comparing apples to apples. Some students blew off X, because they could still get an A without X in some classes. Others didn’t, because it was the entire grade.

    This has led to our examining what it is we want overall. We have a capstone course, but there’s no way of making sure that the students have actually been introduced to all the components of the capstone before they get there. So, it’s called a capstone, but isn’t. OA, and being forced to sit down and talk with each other, has meant that we are now doing curriculum revisions to make sure that there is no way a student can get to a thesis stage without being exposed to all the things we say the capstone is supposed demonstrate competency in.

  40. Historiann on 27 Aug 2009 at 8:58 pm #

    I see where you’re coming from, ADM, but I’m more with Clio B. I think thinking about our teaching, reflecting on our goals, talking about our teaching (on blogs, with colleagues, with friends, with students), revising syllabi, revising lectures, experimenting with new ideas and technologies, etc.–the vast majority of us do this all of the time. And the assumption of Outcomes Assessment is that we don’t do any of this. Instead, it assumes that we need to make “instruments” up to retrieve certain “data points” and report them to someone else, and somehow–that’s supposed to make us more effective?

    I have never seen anyone come back to my department with any “data” about what we need to do better and how we need to do it. So what’s the point? (I suppose I should be grateful that we’re being left alone–but again, I ask: what’s the point?)

    Outcomes Assessment is an expensive redundancy that infantilizes us and is aimed at de-skilling what we do. I think if we talk to the K-12 teachers, they’ll back me up on the effects of NCLB and the fad for schools with highly directed curricula (i.e. no need for teacher creativity or responsiveness to individual student needs!) and requirements for 2-hour uninterrupted blocks of literacy instruction for 6, 7, and 8 year-olds. No recess! Awesome. Adderall and ritalin all round, then.

    I don’t hear anyone here resisting OA because it infringes on their academic freedom, although that is a concern that Clio B. articulated in her original post as she sees a standardized, multiple-choice test on the horizon. It just sounds to me like people think it’s a waste of time and money, when the “instruments” (our tests and essays) and the “data” (grades, graduation rates, etc.) are right there for the taking, if anyone cares.

  41. Historiann on 27 Aug 2009 at 9:04 pm #

    Cassandra–please stay away from the name-calling and the motive-impugning (i.e. “chuffing” and “kool-aid.”) We can have legitimate disagreement without getting personal, and when you write things like that, it sounds very personal.

  42. Another Damned Medievalist on 27 Aug 2009 at 9:12 pm #

    I started working on assessment in committees when I was contingent faculty — no tenure, no long-term contract. I don’t have tenure now, because my campus doesn’t do tenure, but I have the closest thing there is to it — but I only got that this last year. I also served in faculty governance as contingent faculty, at a time when the faculty were enmeshed in a fight for a vote of no confidence in our college president.

    My own experience is that faculty often *lose* the chance to have control because at first, the administration doesn’t take it seriously enough, so the faculty don’t, and then, the administration realize they *have* to do something, and order the faculty to miraculously come up with something that, because they’ve never been asked to do things properly or had the system and its purpose explained to them, just see it as more bs. And then it becomes some deep philosophical argument, rather than an opportunity to make institutional changes.

    I think OA catches on most slowly in places where administration is normally top down, because the faculty have often either given up, or simply don’t trust the administration in terms of OA needing to be faculty driven. But I think that in many such places, this is the one place where accreditation agencies work in favor of faculty regaining more control, because they actually check, and campuses get dinged when the faculty haven’t bought in.

  43. Cassandra on 27 Aug 2009 at 9:15 pm #

    I was already thinking about this — and then Historiann quoted that apt entry from Fratguy — but would the OA people be satisfied by a simple pretest-posttest experimental situation?

    Give an exam in the 1st week to assess how much students know at the time, their skill level, etc., then give a final exam assessing the exact same “knowledge” supposedly gained over a semester’s instruction. Then, compare data sets.

    If I ever went back to teaching college courses, I have seriously considered doing that, especially in light of the common student slander … er, complaint … that I hadn’t taught them what they needed to know to do well in the course, or the related claim that they learned nothing in my courses. (Really? Nothing??).

    This way, they had clear warning what to watch for over the coming weeks, and if they missed it, it’s their fault, not mine. ;-)

  44. Another Damned Medievalist on 27 Aug 2009 at 9:28 pm #

    eep — posted at the same time!

    Historiann, I honestly don’t think it’s a strain to go from looking at a class and saying, “wow, I use essay exams to judge whether my students are getting this, and most of them didn’t” to “wow, ‘most’ equals 80% — maybe there’s something wrong, and here’s how I’m going to address it,” or from, “you know, the average grade for the capstone is 73% — that’s embarrassing, especially when the students have mostly As and Bs going in … hmmm” to “if good students aren’t able to do well on the capstone, maybe we need to look at it and ask why, and put down ways we’re going to try to fix that.”

    I think in lots of ways, it IS a PITA and a waste of time. But it also is a way of maintaining continuity and departmental goals when faculty come and go (remember, accreditation cycles are 10 years — it’s entirely possible that a department could undergo a complete change of staff in that time), and it helps to ensure that no matter the course, students can expect some sort of consistency in expectations at a particular level.

    I agree that good faculty and good departments do this already. I agree that they should. But honestly, I have more discussions with my online colleagues and foreign colleagues about expectations than I do in my own department. Why? because lots of us are raised with the idea that we don’t ask our colleagues about teaching because it makes us look like we don’t know what we’re doing. I used to go and talk to one of my colleagues about an assignment, and whether ze thought it appropriate for the course I was teaching (my own, or one that ze and I alternated, it made no difference), and the colleague acted as though I were asking permission and/or didn’t know what I was doing. No. I know what I’m doing, but I wanted to get a feel for our students and the department, so I could gauge my assignments appropriately. I can’t lower my standards, but I can teach in ways that help the students achieve them.

    I know plenty of people who work in departments where each faculty person is in hir own little world. If it takes OA to get them to work together, I’m sorry that’s what it takes, but I’m all for it.

  45. Another Damned Medievalist on 27 Aug 2009 at 9:33 pm #

    Cassandra — from what I know of SACS, yes, pre-and post-tests are perfectly valid, as long as you test for what you say are the outcomes. But you can’t use a test that focuses purely on content, for example, if you say you are looking for analytical skills and clear written expression.

    But yes, if you gave an essay exam that required using certain sorts of evidence effectively at the beginning and at the end, and in between showed assignments designed to teach those skills adn tracked how the students did? That’s pretty much what OA asks for in a nutshell.

  46. Another Damned Medievalist on 27 Aug 2009 at 10:08 pm #

    Clio — I just saw your last comment, and I wholeheartedly agree that it really can feel as if our abilities are being questioned. But I also think that this can be a really pernicious red herring thrown out by our lazier colleagues. My department has been asked over and over again if there is not some standardized test we can use — by the accreditors, by the administration, and by our colleagues in the health professions and business. And every time, we say no. Not even the GRE, because it’s heavily weighted towards modern history, and doesn’t require research or analysis.

    And then we point to our outcomes, and remind the person asking the question that history is a discipline, and thus more a sort of training than a mastery of any particular content. The outcomes we’ve been working on (still in progress) reflect that. The response has always been, “ok — make sure you are assessing those things and tracking them.”

    So for me (as I think I’ve said — sorry for the hijacking), it’s a PITA to track all this stuff, but the by-products of the process are mostly positive.

  47. Susan on 27 Aug 2009 at 10:31 pm #

    Oh, gosh. This on a day when I started with a meeting about assessment… Like Buzz, I think that there is a role for program level assessment, but naturally, I don’t think it needs to be a test. Nor do you actually need a process: we looked at our senior theses last year and realized that we needed to think seriously about what skills we want to teach in lower division courses so students in upper division courses know what to do. . . So we’re thinking about things like making sure that lower division courses do basic stuff that we sort of take for granted, and our mostly first generation students do not know.
    The point of this is that there *is* a difference between a program assessment — what do we want history majors to know, and do they know it — and the assessment that we do in our courses.

    The big fight we’re having now is that WASC (which is driving the whole discussion) requires not just program level assessment, but institutional level sutff.

    And the other fight is not just about staff, but who controls the staff. Do they work for the faculty, institutional research, etc.

  48. Another Damned Medievalist on 28 Aug 2009 at 5:54 am #

    I think institutional level assessment can be good, Susan — again, if done well. At present, we are doing the same thing, and I have to say, it can be awkward. Getting admissions to understand that retention also has to do with the people they admit in the first place is … interesting, to say the least!

  49. Clio Bluestocking on 28 Aug 2009 at 6:34 am #

    ADM, I see what you are say — truly! I do — but that’s not what is going on at our place. Like I said in my last comment: we are already creating outcomes, reviewing our teaching, and reviewing the student sucess or lack of success.(And, really there does need to be more done on the students’ side, asking “why do students fail?”) But rather than say, “let’s how we can take your own process of quality control and make it more effective,” they come in and say “your more rigorous and complicated process? That means nothing. You all have to give the same 15 question test to demonstrate that education is taking place.” That is a joke, and that is also why — at our place — we do see standardized testing on the horizon.

    Hearing other people’s stories made me wonder about who was saying “this is how OA should work.” Obviously, other people in the domain of our same accrediting agency (and who are they and where did they a come from?) do the thing differently. It turns out that our administration — I’m not yet sure exactly how, but I think that it involved a 1/2 million dollar consultant because these things always do — dictated to us. Our administration is, right now, is very very very big trouble for many many many bad things, which is another story but which also gives you an idea of how we ended up with something so idiotic and unpopular.

  50. Historiann on 28 Aug 2009 at 6:41 am #

    ADM wrote, Historiann, I honestly don’t think it’s a strain to go from looking at a class and saying, “wow, I use essay exams to judge whether my students are getting this, and most of them didn’t” to “wow, ‘most’ equals 80% — maybe there’s something wrong, and here’s how I’m going to address it,” or from, “you know, the average grade for the capstone is 73% — that’s embarrassing, especially when the students have mostly As and Bs going in … hmmm” to “if good students aren’t able to do well on the capstone, maybe we need to look at it and ask why, and put down ways we’re going to try to fix that.”

    These are reasonable questions–has anyone *seriously* ever marked a set of papers or exams that 80% of their students failed?–but it doesn’t take into account student effort. One of the things that reassures me about my “instruments” is that there are always some As and lots of Bs. Success is possible–as I tell my students, the differences I see in student achievement have more to do with the amount of perspiration, not the quality of individual inspiration. Assessment exercises that judge only the faculty without taking into account student effort are like judging a physician by the outcomes of her patients without checking to see if her patients filled their prescriptions and took her advice. And I have never met an Assessment “instrument” or data point that did that.

    This is why grades are such useful information. They judge student achievement and reward it accordingly. It seems to me that looking at the spread of grades assigned by faculty would be a reasonable way of identifying problems with teaching: I myself would be suspicious of a colleague whose grade distribution was very low, as well as of a colleague whose grade distribution was extraordinarily high. That’s something that’s easily done without putting the faculty to all sorts of busywork generating more fake data.

  51. Historiann on 28 Aug 2009 at 6:46 am #

    Clio B., send me a link to that story you alluded to earlier that may soon appear in a major Metro newspaper. I’m crossing my fingers that your uni is in for major humiliation–it may relieve you from dealing with some of those toads for a while. (Here’s hoping. Even if it only sets some wheels in motion for the administration of karmic justice, that will be good enough.)

    Once again: waste, fraud, and abuse. No one likes those–and how many stories do we read in the newspaper in which an indidivual faculty member is accused of having hir hand in the till or of shirking her work responsibilities, versus the newspaper stories about this or that major boondoggle or expensive NCAA violation or covered-up rape scandal? Seems to me all of this stuff flows from administrators, not from the faculty. So who’s really not doing hir job well? Srsly?

  52. Clio Bluestocking on 28 Aug 2009 at 7:34 am #

    I will do that. It just hit the paper this morning.

  53. Tom on 28 Aug 2009 at 8:04 am #

    Historiann–I couldn’t quite read my way through all the comments here: too depressing. But one wonders: if the tenure process works, one can theoretically expect tenured teachers to be effective teachers. Does the move towards assessment echo the move towards the de-tenuring of faculty teaching? That is, are universities actually saying: “Since fewer and fewer faculty are going through the tenure process, we need to keep a better eye on what kind of education they/we are delivering!” If so, of course, that would only support the claim that what might be just as effective is to funnel money into tenure-tracking more faculty.

    One effect of the de-tenuring of faculty, of course, is that the “average” full time teacher at most schools now teaches more sections than in the past. Administrators who promote OA may be implicitly acknowledging that they need/want to oversee teaching because they’ve been making hiring choices that are financially good, but bad for educating students–because increasing the number of sections and students that the average faculty member teaches is bound to lessen the quality of instruction.

  54. Indyanna on 28 Aug 2009 at 9:37 am #

    Historiann pointed out yesterday, I think, how universities have been around for 700 years or so, most of them before there were accreditation agencies, and one could imagine them being around 700 more after those shops have been dismantled and their “story” relegated to the larger one of American bureaucratic excess and overreach. We’re trained to abhor reifying extant relations of power in the pasts that we study, and we should at a minimum be more skeptical of them on the playing fields we compete on. Assessment is mostly a new weapon system in the ongoing game of struggling over relations of power in the academy. Figuring out how to “make it work for us” is, I think, an exercise in surrender. And it isn’t just accreditation agencies. It’s NCATE and the whole alphabet soup of professional (b)orgs that would love to put faculties in blue smocks with “How Can I Help You?” embroidery on the back. One of the best books I ever read in graduate school, wholly out of my field now, was Robert Wiebe’s _The Search for Order_. By the interpretive terms of which, modern liberal arts disciplines and faculties are what he called the “Old Middle Class.” Affable, open-minded, individualistic, reasonable to a fault, always looking to compromise, and gettin’ our lunch money stolen every day on the fields of academic battle by the lightweight “diss-a-plines” that field better armies of lobbyists and strategists at the seats of state power. College is the new high school.

  55. Kathleen Lowrey on 28 Aug 2009 at 11:51 am #

    Tom — bing bing bing bing bing! So exactly right on.

    ADM, actually I think your line about “lazy colleagues” is the real red herring here. This whole issue makes me use everything I learned from feminism — it’s not a few shrill malcontents who are afraid of the Unobjectionable Goodness that is OA. OA is a leading indicator of a huge structural shift in how universities operate that is *bad*, not good.

    Diverting debate toward carictures of individuals (but what about the lazy prof who’s been reading the same boring lecture notes in a monotone since 1988?) distracts from the fact that there is a ginormous (sorry for the technical language, but you know, I think it is warranted) problem of structural power in operation here. We need more tenured and tenure-track profs in the saddle making the key decisions about education, not fewer with less and less decision-making power. OA (which is damnably different than, “hey, let’s work together with our colleague/peers to make sure teaching and learning is all it can be at our institution”) actively moves in the opposite direction.

  56. Kathleen Lowrey on 28 Aug 2009 at 11:52 am #

    Clio B — I love scandale. What is it?

  57. Kathleen Lowrey on 28 Aug 2009 at 11:59 am #

    oh wait, it occurred to me mentioning it here might “out” you. I withdraw the query.

  58. Emma on 28 Aug 2009 at 1:41 pm #

    I dunno about any of this from a prof’s perspective. But, from a student’s perspective, I’d much rather have my prof paying attention to me as a student rather than as a product to shape up and get out the door with the proper stuff stuffed into my head. It seems to me that these metrics or functions or assessments or whatever they’re called have the potential to fundamentally change the nature of the prof/student relationship for the worse, thus making college education less useful for students. That’s what it all feels like to me.

    Disclaimer: I graduated from a VERY small liberal arts college nearly 20 years ago. So my knowledge of academia is very limited in all sorts of ways. But I do get that this issue is tied into other issues such as larger class size, declining tenure positions, etc. etc.

  59. Emma on 28 Aug 2009 at 1:45 pm #

    Which is to say, I thought college wasn’t so much about what you know when you leave as it’s supposed to be about the process of learning it, expressing it, and defending it. How does any assessment tool, other than grades and exams over the course of semesters and years, really measure that? Isn’t that why you get grades in college? Otherwise, why not just stamp everybody’s forehead with “OA Approved” upon graduation and leave it at that?

  60. Susan on 28 Aug 2009 at 6:38 pm #

    I think doing outcomes assessment in a reasonable way (which is what my experience has been) can be useful. What I worry about is that we have to do it each year. And my guess is that the payoff will diminish. We’ll learn a lot initially, make changes, and then it will just be tinkering around the edges. And then it will be a massive time sink. . .

  61. Earnest English on 29 Aug 2009 at 3:38 am #

    I just wanted to say how much I appreciate this entire discussion, especially ADM’s continued engagement and Historiann’s stepping in when the conversation seemed to start seeming a bit acrimonious.

    One thing that ADM has mentioned but has not really gotten that much attention is when faculty are reluctant to talk to their colleagues about their teaching, their goals, student performance, etc. I trust that faculty are working on their own professional development as teachers, but if we don’t talk and really study what’s going on across the sections of a course, then I’m afraid we’re just not going to know. I’ve taught at some places where there was a huge range and variety in core classes taught by a number of different faculty paired with a real reluctance to share one’s teaching. (In one place, there were many efforts to create a teaching culture, but they seemed ineffective for whole groups of the department; some teachers seemed to react with hostility.) While I would always say there are many legitimate ways of achieving a course’s Aims and Scope or learning objectives, I think it’s also important that faculty do have a sense of the range of teaching and especially expectations and outcomes of a given course, especially a core or gen ed course.

    I realize that OA makes teachers want nothing more than to cover our butts, not delve into their own and others’ teaching. I’m not saying that top-down sponsors anything conducive to education. Yet at the same time there are places where teaching is not discussed or shared in a systematic way for curricular improvement, for example. What do we do when assessment really is necessary, not by the muckity mucks, but by us? And how do we do that? And if we’re doing it anyway, how can we use those processes to show the muckity mucks that we’re already doing it and to get off our backs?

  62. Historiann has a man-date. . . : Historiann : History and sexual politics, 1492 to the present on 29 Aug 2009 at 7:42 am #

    [...] I thought I’d let you know that the discussion about “Outcomes Assessment” inspired by Clio Bluestocking has spurred some ot….  Sisyphus at Academic Cog questions the value of quantitative information in evaluating [...]

  63. Historiann on 29 Aug 2009 at 7:53 am #

    Thanks for your comment, Earnest English, and welcome. You anticipate what I would say exactly: “I’m not saying that top-down sponsors anything conducive to education. Yet at the same time there are places where teaching is not discussed or shared in a systematic way for curricular improvement.”

    Well–if you want to talk about teaching, TALK ABOUT TEACHING! I’ve never been in an environment where it was actively discouraged, and in fact, I’ve usually been in departments and had supportive friendships in which we talk about it constantly–not just bitching, but question-asking and problem-solving. We don’t need the OA fraudsters to do this. Ceding ground to them on this just “proves” their fraudulent point, which is that “we” don’t know what the hell we’re doing and we have to be led by educrats to talk about the activity most of us are engaged in for at least half of our professional time.

    I really like Tom’s comment from yesterday morning, in which he said:

    [O]ne wonders: if the tenure process works, one can theoretically expect tenured teachers to be effective teachers. Does the move towards assessment echo the move towards the de-tenuring of faculty teaching? That is, are universities actually saying: “Since fewer and fewer faculty are going through the tenure process, we need to keep a better eye on what kind of education they/we are delivering!” If so, of course, that would only support the claim that what might be just as effective is to funnel money into tenure-tracking more faculty.

    Yeppers. (See also my comments on this post above about the de-skilling of teaching that OA pushes.) In my department, probational regular faculty AND all adjunct faculty are observed in the classroom annually and receive a letter in their file reviewing their teaching. We talk about these letters in Executive Committee and in T & P meetings. This is how it’s done, folks. We do this work already.

  64. dance on 31 Aug 2009 at 8:40 am #

    Taken me a while to come back and read all the comments, but I just wanted to pull out this bit from ADM’s comment above, and it happens to respond to Historiann’s recent comment:

    I used to go and talk to one of my colleagues about an assignment, and whether ze thought it appropriate for the course I was teaching (my own, or one that ze and I alternated, it made no difference), and the colleague acted as though I were asking permission and/or didn’t know what I was doing. No. I know what I’m doing, but I wanted to get a feel for our students and the department, so I could gauge my assignments appropriately.

    YES. I got exactly the same reaction when I came in as a new prof, assigned to teach 1 of 6 sections of our fundamental yearlong core course which all of us teach, and asked for more information. I have great conversations about how quizzes work, about how various assignments played out—but generally with people outside my dept, or outside my school. Rarely with my dept colleagues, all of whom are great and friendly people, caring teachers, whom I have no problem with. In a lovely, functional dept, our culture does not work that way. And since I can’t even do it one-on-one, god knows trying to make it happen on the dept level has been my forlorn hope for the last four years.

  65. quixote on 31 Aug 2009 at 9:53 am #

    My apologies for not reading all of the comments. I just wanted to add that the “Outcomes Assessment” disease is not limited to humanities. It’s equally damaging in biology, and, I’d be willing to bet, all the sciences.

    Critical thinking is essential to discovery in all knowledge, and what this crap does is kill it.

  66. Acid Test » War on Teachers I: GIGO on 29 Oct 2010 at 3:57 pm #

    [...] People began to get desperate. Tests were the only real lever they had, and they weren’t working so they did the usual thing. They tried them some more. “No Child Left Behind” became the mad aunt in the attic that people try to forget, but her nephew “Race To The Top” is now messing about in the kitchen. Educrats began talking about doing testing right by using assessments and rubrics and Student Learning Outcomes. [...]

  67. War on Teachers I: GIGO « The Confluence on 29 Oct 2010 at 4:00 pm #

    [...] People began to get desperate. Tests were the only real lever they had, and they weren’t working so they did the usual thing. They tried them some more. “No Child Left Behind” became the mad aunt in the attic that people try to forget, but her nephew “Race To The Top” is now messing about in the kitchen. Educrats began talking about doing testing right by using assessments and rubrics and Student Learning Outcomes. [...]

Trackback URI | Comments RSS

Leave a Reply