Any exercise in quantification and prediction runs into the dichotomy of applicability vs generizability. A hypothesis is tested to generate data, for example how efective a particular teaching technique is. If more a priori stipulations and specifications are placed on how a particular technique is tested (ie this particular technique, by professors from this particular school of thought, to this set of students with a minimum level of education etc etc etc) the data that is generated will more accurately reflect expected student outcomes WITHIN THE DEFINED CONFINES OF THE EXPERIMENT. In order to turn the experiment from a mere parlor trick into something that is generalizable, that can be picked up off the shelf and used universally, preconditions need to be lifted and the data becomes subsequently much less robust.

By the descriptions in this post is appears that there are no experimental conditions imposed on the data that is gathered. It is infinitely generalizable and therefore nothing more that a numeric description of what has happened. Without preconditions or descriptions of the experiment the “data” is utterly meaningless. It cannot tell you how to improve your your teaching, it does not even define “improve” other than as a change in the data, maybe. People who traffic in tautologies should be called out for the BS artists that they are.

The resulting data, or rather descriptive numbers, though meaningless are nonetheless very potent in the wrong hands. It is clear that management loves this stuff as a means of justification of self and of predetermined ends. If the data is meaningless it can be made to mean anything. In my field this “data” is used as a bludgeon to alter behavior (how the hell else are you going to get through the skulls of a generally male and upper middle class workforce ?) Sometimes this is a good and necessary thing, when the desired outcome is demonstrably desireable and measureable. We have all been conditioned to slobber when the grade bell is rung. On the other hand when the ends are not as neatly defineable, or demostrably good, I sit in meeetings, let the drone of numbers wash over me and pretend that I am the HUD secretary under LBJ. The droning CEO becomes Secretary McNamara talking about kill ratios. I can only hope these people will be judged as poorly by history.

Oh, yeah, and forget about that part of coming back from summer in the archives. Just joking. G.S. prominently features reams of the newly obligatory boilerplate whine about the regrettable cult of institutional reward for research and publication over continuing alignment studies. So we’re going to cut that part out too…

I’m so sorry that you’ve been put through this pointless exercise in pointless hurdle-clearing. I take pleasure in the pretty certain knowledge that if anyone on my department’s T & P committee complained that a junior colleague didn’t have a statement of ELO, that that comment would be roundly derided as pointless nitpicking and that the commenter would be told “DUH, read the frackin’ syllabus.”

Again, I ask: when did we consent to this self-neutering and second-guessing as a way of life? Sometimes I think that many of our colleagues don’t see the value in their work. (How adding an ELO statement addresses that, I have no idea. I think therapy would be a better investment of one’s time and money.)

I like PZ’s suggestion, although it sounds suspiciously non-quantitative and so I don’t know if it would fly. Indyanna’s “Electronic Evidence Room” numbers generator approach might be the way to go. Who would notice, I wonder?

I also like ej’s notion of billable hours, but you’re right: that’s not the kind of data “assessment” demands. “Assessment” is an exercise in which we’re supposed to identify “areas for improvement” instead of congratulate ourselves on a job well done. Erica’s stories about “assessment” among engineers was depressing, but it highlights the arbitrary (if not totally perverse) nature of the data we’re supposedly collecting.

Deliliah, thanks for your insider perspective on assessment. I especially liked your diagnosis as to why assessment is a particular disease of the Educrats:

Perhaps Indyanna’s prankish suggestion that we submit obituaries of majors is the way to go. (But, the problem with that approach is that today obituaries are, God willing, an assessment of a department at work 50 years ago…)

I guess the question is, when historically did people in higher education feel the need to explain why and how specficially education is of itself a good thing? (This is a trend that goes farther back than Republican takeovers of state legislatures, although that’s been a potent recent reality for many of us in public higher ed.) Why did we neuter ourselves politically by deigning to answer questions like that? That’s like asking farmers to prove that food is good for human growth. Just because some of us make our livings providing food and education doesn’t make the endeavor suspicious or corrupt.

My guess is that this has something to do with the professionalization of education in the twentieth century, as well as feminism. In many ways, the democratization of education in the 20th C relied on the underpaid or free labor (of nuns) of women teachers. That is, around the turn of the 20th C as women moved into education and men moved out of it, it became cheaper and therefore feasable to offer to more people, thus making secondary school standard instead of just elementary ed. for all. But, by the 1960s and 1970s, when other professional options besides “teacher” or “nurse” were open to bright young women, those bright young women abandoned education as a career for higher-paying jobs. This put public school districts and private schools in a position of having to pay higher wages, thus the demand for “accountability,” “assessment,” etc. No one much cared about education until teachers had a modicum of leverage to insist on better wages. The fact that many of those bright women went to graduate school, got Ph.D.s, and to some extent, feminized (some of) the humanities is part of the bigger picture for the advent of “assessment” in colleges and universities.

But, this is all just a guess. Those of you who are twentieth century historians and/or historians of education may disagree and will perhaps offer a more compelling explanation.