Jorden,
Jim
Sibley has provided excellent advice about using a multiplier to prevent
students from “gaming” the peer evaluation process. Let me
add another suggestion or two. I apologize in advance for the length of
my post. .
One
strategy (of which I am not particularly fond) is to treat a composite mean helping
behavior score of 10 as a “C” letter grade (75% of the helping
behavior points).
C
is average. Only those individuals scoring above 13 receive an “A”
(100% of the helping behavior points) for their helping behavior.
Individuals with a composite mean helping behavior score between 11-12 will
receive a “B” (85% of the helping behavior points) Individuals
scoring between 7 and 8 will receive a D (60% of the helping behavior
points). This strategy prevents grade inflation and discourages student “gaming.”
Unfortunately, it may also serve to weaken positive student attitudes about
working in teams.
Many
faculty who use TBL for the first time implement a summative peer
evaluation process (what I will refer to as team “helping behavior”)
but fail to provide opportunities for the team members to provide corrective formative
peer evaluations to allow for behavior modification over the course of the
semester that will improve individual and team learning. As a result,
many students are not completely honest in their responses (they attempt
to game the process). Therefore, I have added a paragraph to my TBL syllabus
which serves to reduce this tendency, and to eliminate the possibility of
grade inflation.
FROM MY SYLLABUS
HELPING BEHAVIOR
In
addition to Peer Evaluations done periodically throughout the semester, each
individual will rate the helpfulness of all of the other team members prior to
the final exam.
Individual helping behavior scores will be calculated using the mean of two
scores: a) the average of the points you receive from the members of your
group; and b) the participation score you receive from me.
Assuming arbitrarily that: 1) helping behavior is worth 10 points, and 2) that
there are five members in a team, an example of this grading procedure is as
follows: Each individual must assign a total of 40 points to the other
four members in your group without giving the same score to every member.
Not all group members contribute equally. Some members are more motivated
or more communicative than others. For this reason raters must
differentiate their ratings. This means that each rater would have to
give at least one score of 11 or higher (with a maximum of 15) and at least one
score of 9 or lower. The
instructor will assign a participation grade for each of the members and this
score will be added to the average score from the group members to derive a
composite helping behavior score. Helping behavior scores
produce differences in grades only within a team. As a result, group
members cannot help everyone in their group to earn an A by giving them a high
peer score. The only way for everyone in a group to earn an A is by doing
an outstanding job on the individual and team exams and projects.
Although rare, should a problem arise with anyone’s group participation,
we will address the problem discreetly, but directly.
In
include another paragraph on the Helping Behavior Form (when students provide
their final “summative” helping behavior grade to each of their
team members):
Teams work most effectively when every person
on the team is accountable for their own behavior. Throughout the
semester you have frequently given and received feedback as a member of your team
using a formative feedback process created by your team. Now it is time
to assign a summative helping behavior score to each group member to indicate
the extent to which individuals contributed to your team success.
As you assign your helping behavior scores
below they should be representative of how HELPFUL each member of
your group has been throughout the entire year. Helping behavior scores
do not come from one isolated instance but rather as a cumulative measure of
individual accountability to the group. Please be HONEST in
your evaluation of your other group members. I understand that some
students may be hesitant to be truthful about the performance of others, but
individual accountability is paramount to successful teams.
If you do not submit BOTH
written and numerical peer feedback for EACH member of your team you
will receive a zero for your helping behavior score. YOUR FEEDBACK is
CONFIDENTIAL. Team members will receive feedback but NOT WHO provided
it.
Helping
behavior (summative evaluation) provides a means to assess individual accountability
across a semester and insures that the students take the formative process
feedback seriously. It is very important that the final helping behavior
score account for no less than 5-10% (up to approximately one letter grade) of
the final course grade. Unlike the formative peer evaluations, the
helping behavior scores need to be confidential and shared only with the
professor. The summative helping behavior process should include
parameters that allow for complete and honest disclosure.
Feedback—both
process feedback and summative helping behavior feedback should be direct,
clear, and ensure that group members are accountable to the rest of the
team. The value of the process feedback is that all of the members of the
team have the opportunity to discuss the feedback and make performance
improvements over the course of the semester.
Based
on interviews with over 6,000 team members and leaders, LaFasto and Larson
(2001) identified eight characteristics of high performance teams: a
clear elevating goal, a results-driven structure, competent team members,
unified commitment, collaborative climate, standards of excellence, external
support and recognition, and principled leadership. The eight characteristics
are similar to Katzenbach and Smith’s (1999) six team basics that define
the discipline required for team performance: small number, complementary
skills, common purpose, common set of specific performance goals, commonly
agreed upon working approach, and mutual accountability. Peer evaluation
procedures could potentially provide a results-driven structure while the
specific criteria would provide the standards of excellence and help to ensure
mutual accountability.
The
importance of process feedback can be illustrated with a simple example.
If a person boards an airplane in San Francisco and wants to fly to Washington
D.C. they can expect to see a mountain range within the first 15 or 20 minutes
of their flight. If instead all they see is ocean, then they are
obviously going in the wrong direction because there is no ocean between San
Francisco and Washington D.C. For students experiencing Team-Based
Learning, this is what process feedback in the form of peer evaluation ought to
be. It should establish benchmarks for them to figure out, given their
assessment, whether or not their strategy is taking them to their goal.
Evaluation is something every group member should do in order to make sure that
the group is successful.
Students
will support that which they help to create. I have students design their
own rubric and process for measuring individual accountability. Before
students begin to develop a rubric for measuring each of the team criteria,
they are required to read a short four-page article written by Michaelsen and
Schultheiss (1988) that clearly outlines seven characteristics of helpful
feedback:
1.descriptive,
not evaluative, and is “owned” by the sender
2.specific,
not general
3.honest
and sincere
4.expressed
in terms relevant to the self-perceived needs of the receiver
5.timely
and in context
6.desired
by the receiver, not imposed on him or her
7.usable;
concerned with behavior over which the receiver has control.
Michaelsen, L. K., & Schultheiss, E. E. (1988). Making
feedback helpful. The Organizational Behavior Teaching Review, 13,
109-112.
When
student TBL teams implement effective peer evaluation strategies for gathering
feedback about how well their team is functioning they can make the necessary
corrections that will ultimately enhance their overall productivity. The
optimal condition occurs when team members share equally in all tasks, employ
emergent leadership that shifts as a function of specific tasks and expertise
alignment, and work together to produce learning that is greater than the sum
of what could have been learned working independently. When TBL is
working well, individual and team learning scores are high. As a result,
helping behavior scores should also be high.
I’ve
provided detailed information about the peer evaluation process (formative
feedback) here: http://www.uky.edu/~drlane/groups/peer.html
There
will be a special issue of “New Directions for Teaching and Learning”
dedicated to Team-Based Learning published in January 2009 that should serve to
clarify additional issues related to peer evaluation procedures and
criteria. In the interim, I trust several of you will continue to find this
TBL listserv useful.
Have a great week!
-Derek
Derek R. Lane, Ph.D.
Associate Dean for Graduate Programs
in Communication
College of Communications &
Information Studies
133 Grehan Building
University of
Kentucky
Lexington, KY 40506-0042
Tel: (859) 257-7805
Fax: (859) 323-9879
Email: [log in to unmask]
Faculty website: http://www.uky.edu/~drlane
IMPORTANT DISCLAIMER REGARDING THIS TRANSMISSION:
The contents of this email message and any attachments are
confidential and intended solely for the addressee(s). The information may
also be confidential and legally privileged. This transmission is sent in
trust, for the sole purpose of delivery to the intended recipient(s). If
you have received this transmission in error, any use, reproduction, or
dissemination of this transmission is strictly prohibited. Neither the
transmission of this email message and any attachments nor any error in
transmission or misdelivery shall constitute waiver of any applicable legal
privilege. If you are not the intended recipient, please immediately
notify the sender by reply email and delete this message and any attachments.
From: Team Learning
Discussion List [mailto:[log in to unmask]] On Behalf Of Jorden
C. Sahl
Sent: Tuesday, March 11, 2008 10:32 AM
To: [log in to unmask]
Subject: the math of peer evals
Hi
Everyone,
I recently finished my second Introductory Psychology course using TBL. I have
a question that is not that complicated but for some reason I can’t get
myself around the math and I want to double check with other TBL users before I
teach it again.
I used the Peer Evaluation form from the TBL book that assigns an average of 10
points, with one 9 and at least one 11 (max 15). A student asked me – so
what does my average have to be to get “perfect” on the peer
evaluations? And I was stumped (I had just never thought of it that way)
The max anyone can get would be 15, so I would assume in the rare case scenario
of one person getting all 15s from their peers that would be
“perfect”. But then, if 10 is average, is 10 “perfect”
(I.e., if someone gets an average of 10, do they get 100% on peer evaluations,
with the possibility for students to get higher than 100%)? (My class picked
30% for the peer evaluations grade weight). Calculating it this way (10 is
100%) increases everyone’s grade quite a bit (about half a letter grade),
which I don’t have a problem with, as long as it is correct!
I am second guessing myself with the math and want some input before I use this
method again.
Thanks so much.
Jorden
Cummings Sahl, M.A.
Graduate
Student, Clinical Psychology
Depression
& Wellness Research Lab (302.831.2215)
University
of Delaware
Newark,
DE 19716