I do give instructions and the questions themselves are fairly specific. I even give them examples in the discussion. How much did their discussion indicate that they read the assignment? How much did their discussion indicate that they prepared for the IRQ’s? How much were they able to explain concepts? How much did they contribute to the TRQ? They give them a rating of 1, 2, 3 (3 being the highest; 0 if they were absent or didn't participate). Even with that, there is NO EXCUSE for giving someone a rating of 3 who wasn't even there. There no excuse for rating someone who wasn't there a higher rating than someone who was. This is what bothers me the most. Combined with the gaming the system and the implicit bias, I have to question are peer reviews doing what they're supposed to do? Are they really just a popularity contest? Do students even pay attention to what they're supposed to be rating? Don't get me wrong, I know some students do take it seriously and carefully consider but since these ratings have real (grade related) consequences is it fair to others when their team members don't take it seriously or other teams game the system? On Sun, Mar 22, 2020 at 10:49 AM Jennifer Imazeki <[log in to unmask]> wrote: > Like Molly, I have not had any issues with peer evaluations (not sure if > this says more about economists than TBL :-)) but I did wonder what kind of > instructions you all are giving your students for thinking about what the > quantitative score represents? That is, what is your rubric? I give my > students a fairly detailed rubric that lists the kinds of things that they > should be using to make their evaluation (e.g., what good and less good > participation looks like) and expect the qualitative explanation they give > for their score to reflect what is in the rubric. That also gives students > language to use when they do give lower scores and I think that may help. > > On Sat, Mar 21, 2020 at 6:07 PM Molly Espey <[log in to unmask]> wrote: > >> I have some ongoing quantitative research on peer evaluations, hoping to >> send out for review in early summer. >> >> >> >> While I don’t have enough racial diversity to include in the analysis, I >> have found females get higher peer evaluations from both males and females >> in my introductory level microeconomics courses (with over 4000 >> peer-to-peer evaluations) but this difference diminishes as the students >> advance in economics courses. That is, females get higher evaluations than >> males in my intermediate microeconomics but not by as much, and there is no >> gender difference in evaluations in my environmental economics course (that >> requires the intermediate as a prerequisite). >> >> >> >> I give midsemester evaluations that do not count toward the students’ >> final grade, as well as end-of-semester evaluations that do. I use the >> midsemester evaluations to give a warning/feedback to students who do not >> get good ratings, or for whom the peer evaluation weighted team portion of >> their grade is lower than their individual performance. Sometimes good >> students don’t get “bad” peer evaluations but could still step up more to >> help a team do better. >> >> >> >> I think a problem some encounter is with asking for peer evaluations too >> often. Students should be reminded that their interactions can affect >> their grade but they shouldn’t be asked for evaluations after every >> interaction (in my humble opinion). Mid-semester provides the opportunity >> to give feedback (anonymous), allowing students to adjust their behavior >> for the better. >> >> >> >> I have not found a problem with students not differentiating evaluations >> in a way that does not reflect contributions fairly accurately, for the >> most part. I tend to see pretty consistent evaluations across students for >> a given teammate, that is, if one is rated lower than the others by one >> teammate, that student tends to be rated lower by the rest as well. I also >> rely on qualitative comments and my own subjective observations, which I >> know instructors of very large classes cannot do. >> >> >> >> I remain a proponent of both quantitative and qualitative evaluations. >> >> >> >> >> >> Molly Espey, Professor >> >> John E. Walker Dept. of Economics >> >> 247 Sirrine >> >> Clemson University >> >> Clemson, SC 29634 >> >> (864) 656-6401 >> >> >> >> [image: signature_745619304] >> >> >> >> >> >> *From: *Team-Based Learning <[log in to unmask]> on behalf of >> "Amy E. Hughes" <[log in to unmask]> >> *Reply-To: *"Amy E. Hughes" <[log in to unmask]> >> *Date: *Friday, March 20, 2020 at 12:50 PM >> *To: *"[log in to unmask]" <[log in to unmask]> >> *Subject: *Re: Questioning the value of quantitative peer ratings >> >> >> >> Thanks very much for bringing this up, Herbert— >> >> >> >> I, too, have these concerns, after roughly 7 years of using TBL in a >> Humanities classroom. I’ve struggled with peer evals again and again, >> particularly when students object to giving differentiated scores to their >> teammates. But I kept doing them because I was taught it is a crucial part >> of the pedagogy. But Herbert's post — especially the observations about >> African American students getting consistently lower evals — makes me want >> to abandon the quantitative peer eval as well. >> >> >> >> During this semester in particular -- when I've had to move my class >> online five weeks before the semester is over (I'm SO grateful to everyone >> on this list who provided help and guidance about that! Thank you!) -- I >> don't know if it is a good idea to ask students to evaluate each other, >> when so many of them are grappling with unusual problems that cannot be >> "fixed" (e.g., uneven internet access, missing class in order to travel >> home, mental health challenges, etc). >> >> >> >> I know reflection and constructive feedback is valuable. But in light of >> the challenges and drawbacks that many of us experience while using peer >> evaluations, not to mention the potential for someone’s bias to impact a >> student’s grade, are the cons and risks outweighing the pros and benefits? >> >> >> >> How are TBL teachers accounting for and/or offsetting the potential for >> bias in peer evals? Has any research has been done about how race, gender, >> disability, etc. affects students’ peer evaluations in TBL classrooms? As >> we know, there is ample research that faculty who identify as POC, women, >> etc. tend to get lower ratings and comments from students, sometimes laced >> with biased language. Therefore, it makes sense (unfortunately) that this >> might happen when students evaluate each other. >> >> >> >> I'd welcome any references, guidance, and strategies you can share. >> >> >> >> Thank you, >> >> Amy Hughes >> >> >> >> Dr. Amy E. Hughes *(she/her)* >> Associate Professor of Theatre & Drama >> >> Head, Bachelor of Theatre Arts (BTA) >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__smtd.umich.edu_programs-2Ddegrees_degree-2Dprograms_undergraduate_theatre-2Ddrama_bachelor-2Dof-2Dtheatre-2Darts-2Din-2Dtheatre_&d=DwMFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=45N1t8RJ-DEz4bMRs7eKAdJIqhP2zj0fY5HI0SWfkfI&m=WmsigKmiX_69L-B_OGXIRrj7uCUw8RTo8w5q01RXtU8&s=_FL9VDKNtJ-9jHEhAsOWmTx1acWaCVsCFgP0JFRMGiE&e=> >> University of Michigan, Ann Arbor >> [log in to unmask] >> >> Bio: http://bit.ly/AEHughesUM >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__bit.ly_AEHughesUM&d=DwMFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=45N1t8RJ-DEz4bMRs7eKAdJIqhP2zj0fY5HI0SWfkfI&m=WmsigKmiX_69L-B_OGXIRrj7uCUw8RTo8w5q01RXtU8&s=GF7ZE_H4G6NblB13YJxPquHdt1mI9K5FhGQxX1sbtrQ&e=> >> Coeditor, *A Player and a Gentleman: The Diary of Harry Watkins, >> Nineteenth-Century US American Actor*: http://bit.ly/PlayerGentleman >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__bit.ly_PlayerGentleman&d=DwMFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=45N1t8RJ-DEz4bMRs7eKAdJIqhP2zj0fY5HI0SWfkfI&m=WmsigKmiX_69L-B_OGXIRrj7uCUw8RTo8w5q01RXtU8&s=p_tXRD77Tr_SrxmFiF7TOCYV1FIizJSqMwk6CXIHjCI&e=> >> and http://harrywatkinsdiary.org >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__harrywatkinsdiary.org&d=DwMFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=45N1t8RJ-DEz4bMRs7eKAdJIqhP2zj0fY5HI0SWfkfI&m=WmsigKmiX_69L-B_OGXIRrj7uCUw8RTo8w5q01RXtU8&s=8BcOgXd0VzK11POPj0_5CRhFfBItzX6_JDJtZYGZllc&e=> >> Author, *Spectacles of Reform: Theater and Activism in >> Nineteenth-Century America*: http://bit.ly/HughesSpectacles >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__bit.ly_HughesSpectacles&d=DwMFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=45N1t8RJ-DEz4bMRs7eKAdJIqhP2zj0fY5HI0SWfkfI&m=WmsigKmiX_69L-B_OGXIRrj7uCUw8RTo8w5q01RXtU8&s=WHHvDBxiahy8csHyQyh0a9bFc8aIOZ-svOWeu6KfWfo&e=> >> >> >> >> >> >> >> >> On Thu, Mar 19, 2020 at 1:30 PM Herbert Coleman <[log in to unmask]> >> wrote: >> >> Over the years my students and I have struggled with peer ratings. >> Originally, they had difficulty dividing points between members so I >> changed the format. Then some teams kept giving all top scores to every >> member. When I put in a penalty for that a few teams collaborated and >> decided to gage so that they each varied their ratings and each gave a >> different person a bonus. >> >> >> >> Two years ago, I noticed that African-American students were >> consistently getting lower ratings. This was the case even when I observed >> one who was clearly the team leader and the qualitative feedback says they >> doing great. Now for the second time, I have students giving ratings to a >> student who had dopped the class or was absent on the date of the TRQ or >> even the whole unit. This last case enraged me because the team members who >> completed the ratings gave the person who was absent a higher rating than >> someone who was actually there! >> >> >> >> If students are resiting completing these faithfully, then what value are >> they, really? I use a qualitative rating but that doesn't figure into the >> score for the one being rated just the one doing the rating. >> >> >> >> I've had the discussions and explained the value but still, this is the >> kind responses I'm getting. >> >> >> >> I guess I'm asking how do we get students to faithfully hold each other >> accountable? I should mention that they also do a self-evaluation for each >> unit. >> >> -- >> >> >> Herb Coleman, Ph.D >> Adjunct Professor of Psychology and Student Development >> *RETIRED-*-Dir. Campus Technology Services >> Austin Community College >> [log in to unmask] >> (512) 223-1790 ext. 22162 >> >> ************************************************************************************************************ >> >> “Keep working, keep striving, never give up. Fall down 7 times get up 8. >> >> >> Without commitment, you’ll never start. But more importantly, without >> consistency, you’ll never finish. >> >> Ease is a greater threat to progress than hardship. So, keep moving, >> keep growing, keep learning. >> >> See you at work. >> >> >> *” ― Denzel Washington, * >> ************************************************************************************************************ >> >> >> ------------------------------ >> >> To unsubscribe from the TEAMLEARNING-L list, please click here. >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ubc.ca_scripts_wa.exe-3FSUBED1-3DTEAMLEARNING-2DL-26A-3D1&d=DwMFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=45N1t8RJ-DEz4bMRs7eKAdJIqhP2zj0fY5HI0SWfkfI&m=WmsigKmiX_69L-B_OGXIRrj7uCUw8RTo8w5q01RXtU8&s=0pEYA9AdojjUjG1ZD_Io8N85yi0O_PQ5jHPca40Eey4&e=> >> >> Further information about the UBC Mailing Lists service can be found on >> the UBC IT website. >> >> >> ------------------------------ >> >> To unsubscribe from the TEAMLEARNING-L list, please click here. >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.ubc.ca_scripts_wa.exe-3FSUBED1-3DTEAMLEARNING-2DL-26A-3D1&d=DwMFaQ&c=Ngd-ta5yRYsqeUsEDgxhcqsYYY1Xs5ogLxWPA_2Wlc4&r=45N1t8RJ-DEz4bMRs7eKAdJIqhP2zj0fY5HI0SWfkfI&m=WmsigKmiX_69L-B_OGXIRrj7uCUw8RTo8w5q01RXtU8&s=0pEYA9AdojjUjG1ZD_Io8N85yi0O_PQ5jHPca40Eey4&e=> >> >> Further information about the UBC Mailing Lists service can be found on >> the UBC IT website. >> >> ------------------------------ >> >> To unsubscribe from the TEAMLEARNING-L list, please click here. >> <https://lists.ubc.ca/scripts/wa.exe?SUBED1=TEAMLEARNING-L&A=1> >> >> Further information about the UBC Mailing Lists service can be found on >> the UBC IT website. >> > > ------------------------------ > > To unsubscribe from the TEAMLEARNING-L list, please click here. > <https://lists.ubc.ca/scripts/wa.exe?SUBED1=TEAMLEARNING-L&A=1> > > Further information about the UBC Mailing Lists service can be found on > the UBC IT website. > -- Herb Coleman, Ph.D Adjunct Professor of Psychology and Student Development *RETIRED-*-Dir. Campus Technology Services Austin Community College [log in to unmask] (512) 223-1790 ext. 22162 ************************************************************************************************************ “Keep working, keep striving, never give up. Fall down 7 times get up 8. Without commitment, you’ll never start. But more importantly, without consistency, you’ll never finish. Ease is a greater threat to progress than hardship. So, keep moving, keep growing, keep learning. See you at work. *”― Denzel Washington,* ************************************************************************************************************ ######################################################################## To unsubscribe from the TEAMLEARNING-L list, please click the following link: https://lists.ubc.ca/scripts/wa.exe?SUBED1=TEAMLEARNING-L&A=1 Further information about the UBC Mailing Lists service can be found on the UBC IT website.