Interesting article/research on professors and course evals

[Luckymama]

I just came across this during my morning meeting and thought some of you might find it interesting :)

 

http://www.npr.org/blogs/ed/2014/09/26/345515451/student-course-evaluations-get-an-f

[kiana]

No surprises there. 

[Bootsie]

This is no surprise.  I know the head of a physics department who had to argue with his dean that a faculty member who had a 3.72 on a faculty evaluation was not a better instructor than one who had a 3.71; these were not statistically different results, especially given the small sample size, but at that university salary increases were based upon these evaluations.  

 

There are also many ways in which these evaluations can be manipulated.  I know a professor who wrote a big "5" on the board before he handed the evaluations out.  (A "5" was the highest rating.)  His evaluations were higher that semester, he claims, due to the suggestion of a giving him a 5.  The next semester he did the same thing one one class, but put donuts out for the students to pick up when they completed the evaluations.  His evaluations skyrocketed.  

 

It isn't clear how to interpret the answers to many of the questions asked--we have questions at our university like "The work in this course was more than average," and "The professor has high expectations for student performance."  Is it good or bad to get students to "strongly agree" with those statements?

 

The university where I teach has spent a fortune in the last few years on developing and evaluating this forms (these are the types of things causing college costs to rise), but have simply generated a lot of meaningless data.  

[regentrude]

 

 

The better the professors were, as measured by their students' grades in later classes, the lower their ratings from students.

"If you make your students do well in their academic career, you get worse evaluations from your students," Pellizzari said. Students, by and large, don't enjoy learning from a taskmaster, even if it does them some good.

There's an intriguing exception to the pattern: Classes full of highly skilled students do give highly skilled teachers high marks. Perhaps the smartest kids do see the benefit of being pushed.

 

This.

 

[regentrude]

I have taught multiple sections of recitations for the same course, with identical homework assignments, tests, syllabi and class content.

My student ratings between different sections during the same semester differed by as much as 0.6 (on a scale from 0 to 4).

 

OTOH, I have to admit that there is some merit to the evaluations if you look at large numbers of students, compare similar classes, and look at general trends. It is, of course, nonsense to compare a small upper level elective for majors with a large mandatory course for non-majors - but if you look, for example, at different instructors teaching the same mandatory course for non-majors and see large differences in their ratings that remain consistent over several semesters, it does allow some conclusions about the quality of their teaching.

 

Usually, the department chairs and the other faculty members in a department know very well who is a good teacher and who is not - without sitting in on the classes or seeing the ratings. They know because they are talking to students. Which gives a more nuanced picture than a bunch of number grades on prefab questions.

ETA: And this is also how the students on campus know about a professor's reputation: they talk to each other. They know very well which instructors to avoid and which to seek out - completely without access to the ratings (they may see the small subset of rating info that is required to be published)

[Matryoshka]

They should be adding another question to the questionnaire.

 

Which better describes you?

- I like to be challenged in my coursework

- I like an Easy A

 

That would really help sort out those kids who give rave reviews to crappy, content-less courses and dis professors who actually try to make them think and learn.  The prof who was running the Honors Comp class using I Am a Magical Teenage Princess got rave reviews.  Yep.  Be interesting to see if her Honors class also gives rave reviews, as in the past she's taught regular-level, and it doesn't appear she's doing anything to make this class "Honors" except asking them to read 15 more pages a week.  Woo-hoo.

[Bootsie]

My university has been encourage faculty to increase the student response rate on evaluations.  One suggestion has been for faculty to give students extra credit when they complete the evaluations.  Needless to say, faculty who give extra credit for completing the evaluations receive how evaluations, on average.  

 

This raises major ethical issues.  I teach finance where we emphasize conflict of interest of bond rating agencies and how that can lead to overly high ratings for bond issues, but then the university is placing strong pressure on faculty to do this.

[regentrude]

My university has been encourage faculty to increase the student response rate on evaluations.  One suggestion has been for faculty to give students extra credit when they complete the evaluations.  Needless to say, faculty who give extra credit for completing the evaluations receive how evaluations, on average.  

This raises major ethical issues.  I teach finance where we emphasize conflict of interest of bond rating agencies and how that can lead to overly high ratings for bond issues, but then the university is placing strong pressure on faculty to do this.

 

Aside from the ethical issue (about which I fully agree), there may be absolutely nothing a faculty member can do to increase response rates when a school has switched to an anonymous electronic rating system. When they were still filling out paper in the classroom, profs could give them points  (which I always refused to do) because they saw who was there. Now the students fill out an electronic evaluation, and we have absolutely no information which of our students participate. We just get the results after finals are over.

I wish we got better participation rates, because they would give a more complete picture - but other than explaining to the students beforehand  WHY it is important that they take the time, there is really nothing we can do.

[Robin in DFW]

nm

[G5052]

Yes, I've said the same thing over and over:

 

Show me your stuff," Stark says. "Syllabi, handouts, exams, video recordings of class, samples of students' work. Let me know how your students do when they graduate. That seems like a much more holistic appraisal than simply asking students what they think."

 

The daytime professors complain that the evening professors get better scores because we tend to have older students who are more reasonable about it and serious about their work.

 

It can become a popularity game.  One of the professors in my department gives all multiple choice exams, and he's higher rated than I am because he's considered an easier professor by the student body.

 

My last evaluation by my dean consisted of looking at how many grievances/grade challenges I had, my average student evaluations, and sitting in one of my classes for all of 15 minutes.  My previous one was done by my department head, and she got in trouble because she did more than expected.  She looked at course materials and interviewed both past and present students on top of the other things.  Guess which one I found more meaningful?

 

But they do it over and over.  Ours are now electronic with a link in Blackboard.  We let them know about it, and then make sure that there's time one day in class for the professor to scoot out early while those who haven't done it yet complete it.  They can do it on their phones, or many of the rooms have a wall or more of computers already.  So the completion rate is usually pretty good, but I still think it is all a game.

[Bootsie]

We have a section where students can write comments on the evaluations.  Last semester I had a colleague who received the comment:

 

"The professor is old-fashioned.  He expects you to remember things for the exam."

[regentrude]

We have a section where students can write comments on the evaluations.  Last semester I had a colleague who received the comment:

 

"The professor is old-fashioned.  He expects you to remember things for the exam."

 

 I got a nasty evaluation one year, with the main complaint being : "She expects us to read the book".

 

I do indeed.

[G5052]

Similar here.  My favorite negatives (I'm an IT/CSC professor):

 

She expected me to figure out a way to get my work done even when my computer was stolen because I left it in the student center.

 

She gets frustrated with computers.

 

 

 

[Caroline]

Similar here.  My favorite negatives (I'm an IT/CSC professor):

 

She expected me to figure out a way to get my work done even when my computer was stolen because I left it in the student center.

 

She gets frustrated with computers.

 

I always tell my students that computer issues is the new "My dog ate my homework" excuse. I sometimes teach math to math ed majors. 

 

Comments I have received include--Her tests required thinking in new ways. She expected us to show up for every class. She expected us to do our own research. 

[G5052]

I always tell my students that computer issues is the new "My dog ate my homework" excuse.

 

Exactly.  The list of lost computers, stolen computers, computers with the wrong software, power problems, no internet, poor internet, etc. would amaze you.

 

I tell them to make contingency plans upon contingency plans. Figure out how you can use someone else's computer, come to the college, go to Panera.

 

Unless the college website is down and/or there are widespread power outages, no extensions and no late work accepted.

[creekland]

Some of these recent posts are a wee bit scary...

 

I hope these students will be as blunt on job applications - though then they might make the news for having a gazillion dollars in college debt and somehow being unable to find a job.

[kiana]

Some of these recent posts are a wee bit scary...

 

I hope these students will be as blunt on job applications - though then they might make the news for having a gazillion dollars in college debt and somehow being unable to find a job.

 

Less likely -- those aren't anonymous -- but people do do those things on job applications.

 

[Sebastian (a lady)]

On the other hand of the class eval debate is the question of whether schools do anything with the information.  I'll confess my experience is several decades old.

 

I won't go into details, but I had an instructor try to get me thrown out of school a year after I submitted a detailed negative evaluation (which included my name and phone number if the department wanted more information.  FWIW, I had a B in the class and was one of the higher scoring students.)  Thankfully his accusation was clearly in retribution for the eval and it didn't go anywhere.  But it left me with quite bitter.

 

One of my post graduation roommates had a notorious professor for one of her intro engineering courses.  He made open comments about how women shouldn't be/couldn't be engineers.  But then he belittled most of the students - male and female.  She failed the class, as did many other students.  When she retook it the next semester, she was initially assigned to the same prof.  She immediately went to the department to transfer to another section.  In the subsequent discussion of why she wanted to change sections, it became apparent that there hadn't been course evaluations submitted on this prof for many semesters.  Even though they had been dutifully completed by the students.  (This was a prof for which there were tshirts made, with boxes to check for which of his classes you'd passed or failed.)

 

I do usually complete evals.  I try to give specifics.  Not that I had to do the readings.  But I have commented that the discussion in class was more about a teacher's politics than the course material.  Or that the instructor didn't adequately cover the material (my Intro to Special Education instructor was particularly bad about this).  But I also give high praise and details about the good instructors.

 

 

[Sebastian (a lady)]

I should add that the stupid comments pp's have shared from student evals are stupid and should be taken as such by the readers of the evaluations.  Of course students have to read the books and do their assignments and pay attention in class.

 

[kiana]

I should add that the stupid comments pp's have shared from student evals are stupid and should be taken as such by the readers of the evaluations.  Of course students have to read the books and do their assignments and pay attention in class.

 

The biggest problem is with administrators who just want to look at the numbers and ignore the comments.

 

I know very few people who want to do away with evaluations completely, but many who feel that the numbers are ridiculous. 

[regentrude]

The biggest problem is with administrators who just want to look at the numbers and ignore the comments.

 

I know very few people who want to do away with evaluations completely, but many who feel that the numbers are ridiculous. 

 

At our school, the verbal comments are only seen by the instructor. The department chair and the administration only see the numbers.

 

[Sebastian (a lady)]

The biggest problem is with administrators who just want to look at the numbers and ignore the comments.

 

I know very few people who want to do away with evaluations completely, but many who feel that the numbers are ridiculous. 

 

I do understand that.  And I agree.  An instructor with a 4.5 average isn't significantly different than one with a 4.6 or a 4.4.  Or probably a 4 for that matter.

[Bootsie]

I should add that the stupid comments pp's have shared from student evals are stupid and should be taken as such by the readers of the evaluations.  Of course students have to read the books and do their assignments and pay attention in class.

 

Yes, good evaluators know this and will ignore those comments.  However, these same people are ranking the professor with numerical evaluations; it is unlikely that their ability to provide a meaningful evaluation with a numerical scale is any better.  The evaluator doesn't know which numerical evaluations to throw out because they are from students with unreasonable expectations. In a small class, these can greatly influence the professor's numerical evaluation score.