Friday, March 09, 2007

More on the Pre-Test at SIU thread - funny observation from a friend

There have been two Pre-Test entries so far here and there.

What we can know after reading the back and forth about this post is that SIU professors, in large numbers, are giving pre-tests. Many of them feel great about this, after all the students might not pass without them. Others feel it is cheating to tell the students the test questions a couple of days before the test. I don't have to think hard on the ethics of this one.

Some professors say that having management give raises based on student evaluations doesn't effect them or management isn't doing it. Others say this is silly. Now if you go and talk to the people who study the methods of teaching at SIU, they will tell you that SIU does give raises based on student evaluations and everyone knows it. They say, that anyone who doesn't think the management initiatives like this work are just fooling themselves.

I say that pushing for higher student evaluations and backing it with money is like beating a child. Everyone knows that beating a child is going to warp the child right? The child often doesn't realize their behavior is tied to being abused, they block that part out. This is why child molesters are often raped and killed in prison, some studies indicate that a majority of prisoners in the US were abused as children. Think about that for a while.

Now for the funny part. In the first thread there are a series of comments between Don Mills a former SIU professor and some random PoliSci professor. The first thing that struck me was how much better Don writes, but we have seen Don's writings here before and knew he was good. They go back and forth about student evaluations with claims and counter claims, finally I thought there was some clarity about how the pandering was happening at the entry level classes, the upper level classes are better because the losers have been weeded out.

What was interesting was how clearly different Don and the other guy reacted to their 100 level classes. I suspect that both are really good teachers and good people. But, the other day someone said to me, "you know that Polisci class the professor is so proud of?" "Yes," I replied. "It is multiple choice"

We are comparing a real math class with a class where you fill in the answers in bubbles? I looked in Google for a definition that would fit this and didn't find one. Let me do a new quote, never been heard before, "if you are doing multiple choice, it shouldn't be a college class."

My friend who pointed this out, he said to ask the PolySci guy, "if he would be so proud if the students were writing essays like they do a real universities."

I don't know the answer myself. Is the class multiple choice? If it is, all arguments that you are getting better teaching evaluations are BS. If you did Math 150 multiple choice, the math professors would get great evaluations too. Multiple choice is another version of pre-testing.

Your comments are welcome.

12 comments:

Scott McClurg said...

"We are comparing a real math class with a class where you fill in the answers in bubbles?"

Your facts are wrong. Graded evaluation in my class covers a mix of writing and active learning with some multiple choice. Of the 700 points awarded in the class, only 25-percent of the grade is based on the multiple choice component. This type of differential evaluation is consistent with what the psychology of learning tells us about students and learning styles. The students tend to do better on the essays than the multiple choices (the average percent correct hovers in the 60% range); and for what its worth, the essays are built on game theoretic concepts so they are are hardly a walk in the park. The way I do evaluation is based on a sound science of learning and isn't developed (obviously given the grades) to get everyone to pass. So carp if you like, but do not to distort the facts please.

And let's not be so quick to pretend you can't use multiple choice exams in mathematics classes to good end. We've been using them on the SAT, GRE, ACT, MCAT, and so on for years for just that purpose. Are they perfect? No. But then again, essays and open ended forms of evaluation aren't perfect either.

"Multiple choice is another version of pre-testing."

This doesn't even make sense. Pre-test means to test before with the same instrument (Campbell and Stanley); multiple choice is to provide a fixed set of items relative to a question-stimulus (Schuman and Presser). They are in no shape or form the same thing.

I agree that MC can be riduclously easy, but that need not be the case. Just like some mathematics problems have a degree of difficulty, so do MC. Just like mathematics problems can be used to gauge mastery of material, so can MC questions if thoughtfully designed. (Thoughtfully designed = don't use the ones provided by the publisher.)

Again, quibble if you like, but this reasoning is based on sound science as to how people learn and how they are capable of conveying knowledge. I know you guys just think "math is harder" and "polysci why try," but that's your prejudice and I'll close with an anecdote -- a social scientist I know always tells his students that it ain't rocket science....that its much harder. How does he know? He quit NASA because he found the problems too easy, while human behavior was a much more difficult nut to crack. So the smartest guy I know doesn't see much of a difference in the two; I'll trust him.

Peter in Carbondale said...

Since pre-testing works to pump student evaluations, even when it doesn't improve their grades, it makes sense that multiple choice does the same.

If you don't think that MC is baby stuff compared to essay or math problems, you are smoking.

One primary difference is how much the student need to know coming into the class to survive it. If you don't know algebra, you aren't passing Calculus. If you don't know how to write essays, you will not pass a 300 level English class. If you are fed answers, anyone can pass a MC test. Today's students like this better.

We will have to agree to disagree, I'm afraid.

I am happy you guys are working hard and doing your research, but it might be time for you learn about how humans really interact with management. Also, what tools management can use to influence behavior. You seem woefully ignorant on these facts. This must influence the political world you study?

Anonymous said...

Peter,

What do you gain by being insulting to those who post here? Are you interested in having a dialogue and exchanging ideas or simply ridiculing those who post responses?

Scott McClurg said...

Let me be blunt...I do not pre-test. I don't know where this myth that everyone pre-tests came from, but its certainly not true in my case. So please stop suggesting that it is.

As to my understanding of the political world, my record speaks for itself.

Anonymous said...

Could it be, and I'm just speculating here, that perhaps the level of instruction in classes where pre-tests are given, just isn't that good? If the prof can't get it across to his students during regular class time, maybe he, or she, should take a look at their teaching methods. If the only way a prof can get good evaluations from their students is by ensuring a good grade, perhaps there is an issue with the educator. Shoot, I'm sure we've all experienced different levels of instruction and different grades from each of these teachers. Myself, I really like the "hard ass" teachers. Makes me work harder, it's tougher to get a good grade. I swear under my breath at the work load, but in the end, I feel like I've accomplished something. Oh, and because I worked hard, I got the grade I wanted. The teachers who seem to not care just don't inspire me, I get lazy and, well, you know what happens then.

Peter in Carbondale said...

To Anon #3 - I think you sound like the perfect SIU employee. If you find a discussion about how to improve SIU, that addresses real problems insulting, that seems kind of strange. I wrote what I thought was true and now I'm just responding to comments. Generally, I try to match my response to the stupidity or brilliance of the commenter's writings.

What I hope to gain is a general improvement of SIU and Southern Illinois by education of the ignorant. I'm very happy with Wendler and Sue Davis being gone, I think I helped there. Now I'm off to other topics.

Thanks for asking.

Peter in Carbondale said...

Scott -

You wrote here that SIU decline is students was because of bad luck. You wrote that management input doesn't effect you. This means you don't have a clue about what is happening outside of your Silo.

I'm glad you don't pre-test. I never wrote or implied you did. I think you write well and hope you have a long, happy and productive career. It is your empathy to the problems of others at SIU you might consider thinking about.

Your experiences are your experiences, they don't have to be the norm. The problem is if 20% of SIU professors are doing it and passing 40% of the students who are not learning, there is a real and long lasting problem.

One of the real problem with a majority of university professors is they have no idea what is going on outside of their office.

Peter in Carbondale said...

Scott -

You wrote here that SIU decline in students was because of bad luck. You wrote that management input doesn't effect you. This means you don't have a clue about what is happening outside of your Silo.

I'm glad you don't pre-test. I never wrote or implied you did. I think you write well and hope you have a long, happy and productive career. It is your empathy to the problems of others at SIU you might consider thinking about.

Your experiences are your experiences, they don't have to be the norm. The problem is if 20% of SIU professors are doing it and passing 40% of the students who are not learning, there is a real and long lasting problem.

One of the real problem with a majority of university professors is they have no idea what is going on outside of their office. It is a problem of any system that doesn't reward employees based on the organization's success.

Scott McClurg said...

"You wrote here that SIU decline is students was because of bad luck."

I'd like you to repost the point where I said that. I ask because I know I didn't write that at all. In fact, I've only restricted myself to the hard data I have in front of me rather than throwing out conjectures. If you want my opinion about why SIU enrollment's decline then I'll give it, but I doubt it has very much to do with any of this and nearly everything to do with rising costs for the students.

"You wrote that management input doesn't effect you."

It doesn't. But what's important is that this is an actual data point and I'm not trying to generalize past it. So far, we have two data points. You can't draw conclusions from that. Don and you are.

But my case is an interesting one because I apparently meet the two conditions Don and you have laid out for being identified for management pressure -- I teach a large 100 level class and I give out a lot of failing grades. If management really wants to penalize people for handing out low grades, then I'd like an explanation as to why I've somehow missed out on the experience.

"This means you don't have a clue about what is happening outside of your Silo."

No, it means I disagree with your opinion. You have this very white-black view of things that just doesn't comport with the way the world works. I don't doubt that some pressure gets applied on faculty from time to time; I do doubt that it is centralized policy. I have friends in a wide variety of departments and we've been discussing this issue and the vast majority of them have an opinion close to mine. Its not systematic data but again I find it suspicious that my other hard-on-the-student colleagues also seem to escape managements eye.

Plus, grade inflation is a problem at more than SIU. Indeed, there was a huge to-do about this at one of the Ivy's just a couple years ago because basically everyone was getting an A. The fact that this is a systemic problem leads me to believe that particularlistic explanations (i.e., SIU is a bad, bad place) don't fit the evidence. You're entitled to your opinion and its not as if I think there aren't problems at SIU, but the evidence for this argument is terribly thin. Vague declarations that I don't ever leave my office (which isn't true, but whatever) aren't the same thing as clear and convincing facts.

Don Mills said...

There are a couple of points worth making in this ongoing discussion.

One, the commonality shared by a pre-test question and a multiple choice question is the ‘educated guess’ option. If one takes a pre-test, then encounters similar questions on the exam itself, one can present correct answers without necessarily demonstrating that they have mastered the concept in question, by observing that, for instance, Question 1 on the exam looks just like Question 3 on the pre-test, with the numbers changed – that is, they make an ‘educated guess’, and proceed accordingly. Likewise, for a multiple-choice question, a person possessing at least some acquaintance with the material can make educated guesses as to what is the correct answer, via the process of elimination. On a 20-question multiple-choice exam, a person might score a 70%, even though they really only had a 50% mastery of the material, by guessing correctly on four of the ten questions. Fill-in-the-blank questions would seem to be an acceptable alternative, as it removes the guesswork component.

Also, when one is grading essays, I would think that one would have to watch out for a student trying to b.s. his way into getting a higher score than he deserves. A lot of students don’t like math because they know that there is no b.s. component of which they can take advantage – the answer is right or it isn’t, the method is correct or it is not.

Scott McClurg said...

"On a 20-question multiple-choice exam, a person might score a 70%, even though they really only had a 50% mastery of the material, by guessing correctly on four of the ten questions. Fill-in-the-blank questions would seem to be an acceptable alternative, as it removes the guesswork component."

What this boils down to is that you believe there is error in measuring grades. Well...duh. Measuring any human anything involves a tremendous amount of error.

Fraydog said...

I've had Dr. McClurg's class. He's one of the better professors I've had at SIU, certainly one of the most demanding. As far as pretesting, I can verify that, while it goes on, he certainly isn't part of the group of people here who do that.

I do respect both sides in this discussion. I think good points are made on both sides.