I have greatly enjoyed this dialogue--it is well-informed, thoughtful, and exploratory. As a social scientist, let me just chime in and lend my support to Larry's concern about treatment fidelity.
As we all learned in graduate school, the conclusions one draws from research are only as valuable as they are:
1. reliable--i.e., reproducible, and
2. valid--that the measurements we took do indeed measure what we think we're measuring.
You will often hear me say that all teachers SHOULD teach differently because every teacher is a unique individual and will be better able to bring out the best in their students in some ways more than others. I am not a "fundamentalist" when it comes to rubber-meets-the-road, moment-to-moment experiences in the classroom.
I am, however, a bit more uptight about the names one gives to what one does in reporting one's experience. Teaching is a mystifyingly complex activity, so the labels we put on how we do it matter a great deal.
If what you decide to research is not Team-Based Learning as it is understood to be best-practiced by the members of the community that has collected around it, then please just take care to make it explicitly clear where you deviate from our understanding of the practice and why you chose to do so.
:-)
-M
Michael Sweet, Ph.D.
Director of Instructional Development, Center for Teaching and Learning
MAI 2206 | Mail Stop G2100 | (512) 232-1775 | http://ctl.utexas.edu
-----Original Message-----
From: Team-Based Learning [mailto:[log in to unmask]] On Behalf Of Larry Michaelsen
Sent: Wednesday, December 14, 2011 11:39 AM
To: [log in to unmask]
Subject: Re: Research concept: Assessing team learning with remedial learnersat a Japanese university
Brent,
Sorry to be so slow in responding. The crunch of finals got in the way.
Also, sorry for the lengthy post. Once I got going it was hard to wind down.
I'm writing for three reasons: 1) to answer a couple of questions that are asked and/or implied by your description of the TBL "treatment"
you are using, 2) to suggest some concerns and offer some ideas for things that I think are important to keep in mind as you move forward and, 3) to ask for some more detail on what you are doing and how you are assessing the outcomes.
1) Based on what you've said (below), you are unsure about whether or not the Readiness Assurance Process is effective with mature/adult students and whether or not TBL is capable of developing self-managed teams of Japanese students because of their cultural heritage and/or lack of experience with any type of active learning. My answer to both questions is, IT DEPENDS ON WHETHER OR NOT YOUR RATS AND APPLICATIONS ARE WELL DESIGNED AND IMPLEMENTED. Based on lots of personal experience with executive development and executive MBA classes, mature students REALLY appreciate the value the RATs and absolutely love the immediate feedback they get from using the IFAT answer sheets. Also, based on TBL applications in many cultures where passivity is the norm, (e.g. Chinese students and students from former Soviet Union countries (see chapters
12 and 13 of the 2004 TBL book and recent listserv posts by Peter Balan regarding Chinese business students in Australia), good TBL practice is VERY successful in overcoming passivity with undergraduate students.
2) From what you've said, I have two concerns about what appears to be your view about the function of the RAP in TBL. One is that, although, I can tell that you understand one of the two key functions of the RATs*creating individual accountability*I'm concerned that you may not see the critical importance of the RATs for promoting team development, which I think is absolutely critical in developing students motivation to help each other assert themselves. The other is that, I'm concerned that you have the impression that in TBL, the RATs also ensure accountability for the team projects. Indirectly, that might be true to some extent because they DO build team. However, the key to accountability for the team projects (i.e., the applications) is making sure they conform to the 4S's (discussed below).
I do agree with the Phoenix practice of having students work on "real" problems has advantages over hypothetical ones (e.g.
cases). That's because the more real the problem, the greater the probability that the assignment will meet the first 4S criterion (Significant to at least the student(s) for whom the problem is real).
In, addition, I agree that Phoenix practices such as having teams establish a charter, submit team evaluations for each project and keep logs of their activities are likely to both reduce social loafing and largely eliminate complaints. In fact, I even agree that these practices will reduce social loafing (and complaints about it) in situations in which students choose to "divide-and-conquer" to complete poorly-designed assignments. However, in my opinion, the fact that you need take these steps at all should sound a warning that something
isn't quite right.
I do NOT agree, however, with what seems to be a Phoenix assumption that the absence of social loafing will result in both learning and team development. For example, even if you are dealing with an interesting and real-world case requiring teams to produce an extensive document you are asking for problems. That's because the only way to accomplish the task is by dividing up the writing task and spending a great deal of time working INDIVIDUALLY on an assigned part. Further, it is very likely that, if everyone does their part so that all the pieces and the overall document are pretty good (which sometimes does happen), no one will get a bad peer evaluation and there will be no complaints about social loafing. In addition, the team leader that pulls the different component parts of the project together is likely to be frustrated by editing individual writing styles into one voice, dealing with students that have different time management skills and/or expectations of quality. Team leaders usually have very little power to enforce team norms or the charter, except use the threat of no or low team grade.
Social loafers usually don't care about their grade strongly enough for losing points to be a credible power base for peers to use to change
the loafer's behavior. In addition, I am not sure from your email
whether the team evaluation has a strong enough percentage of the grade to be truly motivating.
Unfortunately, however, neither the production of a worthy product nor the absence of complaints social loafing ensures either deep learning or team development. When students use the divide-and-conquer approach; there will be a minimum of content-related discussion either within or between the teams for two reasons. One is that, from a practical standpoint, much of the discussion that does occur will be about how to get the document produced (i.e., who does what and about making transitions between the pieces). The other is that, since much of the work will be done while individual team members are sitting in front of their computer, there will be a minimum of discussion within the teams and little, if any, opportunity for teams to learn from interaction with other teams*which is key to both learning and team development.
By contrast, the natural way to complete 4S team assignment is to engage in content related discussions in two different settings. If all of the teams are asked to make a Specific decision about the Same problem (assuming that it is Significant to students*irrespective of whether the problem is based on a real or a case problem) and Simultaneously report their choices, students will deepen their understanding by engaging in and learning from discussions first within and then between the teams. In addition, the teams will become more cohesive because they will feel the need to pull together to be able to meet the challenges from other teams. Further, although direct observation does have a positive effect with smaller classes the, irrespective of class size. THE PRIMARY TOOL FOR REDUCING SOCIAL LOAFING IS CONSISTENTLY PROVIDING IMMEDIATE PERFORMANCE FEEDBACK. This is achieved by using the IFATs (on the tRATs) and inter-team discussions that are always part of well-designed 4S team application activities, projects and exams.
How does this relate to your "experiment" with TBL? I have two concerns. One is that you may be managing the RAP in a way that promotes pre-class study but, may not do much for developing team cohesiveness.
The other concern is about using student presentations in a spot where a 4S application would seem to be far more appropriate for both learning and building team cohesiveness. Unless your assignment is different than any that I've ever seen, requiring students to make presentations (either individual or group) is likely to either delay or interfere with both learning and team development.
3) The issue here is TREATMENT FIDELITY. Given your desire to test TBL for promoting more active engagement by Japanese students, I'm concerned that, irrespective of the outcomes (good or bad), you won't know how much the Phoenix adaptations (and potential TBL omissions) affected (positively or negatively) the impact of the TBL practices that you actually DID employ.
Thus, you will need to be very aware of (and report) the details of what you actually do. For example, the description of your first attempt at implementing TBL doesn't give any information about a number of important issues. First, how did students receive feedback on the RATs?
If you didn't provide real-time, choice-by-choice feedback (by using the IFATs or something else), you haven't used the single most powerful tool available for both overcoming passivity and developing self-managed teams. Second, you don't say anything about the difficulty of the questions. One of the most common mistakes is using questions that are too easy. If your average individual score is too high, there is little need for discussion and minimal opportunity to develop team skills or team cohesiveness. Finally, I'd also like to know about what you had the students do after the RAT. Ordinarily, it would be some sort of application (designed around the 4-S's).
Further, the fact that you had students do presentations (individual or
group?) makes me very nervous. Individual presentations have the same effect as lectures (team interaction would be rude) and, while there MAY be some team development during the preparation phase (assuming that it is in-class), team presentations are no different than individual presentations*except for the presenter the rest of the class is sitting quietly*which is just the opposite of what I think you want to accomplish.
-----
Larry K. Michaelsen
Professor of Management
University of Central Missouri
Dockery 400G
Warrensburg, MO 64093
[log in to unmask]
660/429-9873 voice/cell phone
660/543-8465 fax
>>> Brent Duncan <[log in to unmask]> 12/13/11 2:56 AM >>>
Thanks for your thoughts on the Hawthorne Effect, Bill. Like it came
straight
from one of my workshops.
I fully agree; and, as you point out, so does most of the research. My
point
might not have been clear; I was not introducing the Hawthorne effect
to
advocate for it, but to point out that constant faculty oversight is
like an
application of the Hawthorne effect.
To avoid further confusion, I think it is important to differentiate
between the
Hawthorne study conclusions and the Hawthorne effect. Management gurus
and practitioners still hold tight to the "happy employees are
productive
employees" conclusion of the Hawthorne studies. However, this
conclusion is
essentially "cow psychology"; as in, "a happy cow produces more
milk." Last
time I checked, most humans are a bit more complex than cows. Contrary
to
the conclusions of the Hawthorne studies and the assertions of some
organizational psychologists, most research shows that performance
precedes
satisfaction, not the other way around. in other words, employees who
perform tend to be more satisfied on the job.
Rather than finding a connection between satisfaction and performance
as
many assert, the Hawthorne studies showed that paying attention to
employees can result in temporary increases of productivity. MBWA
serves as
an example of how a manager can use the Hawthorne effect to elicit
temporary increases in performance; "the boss is coming, look
busy." Faculty
oversight in TBL also serves as an example of how teachers use
constant
oversight in attempt maintain performance levels in the classroom.
A problem with relying on attention events to motivate performance is
that
employees performance becomes dependent on extrinsic motivational
forces.
This is why the Hawthorne Effect is the enemy of trainers; performance
will
usually increase through an attention event, but it is difficult to
determine the
degree to which performance increases can be attributed to the training
or to
the attention. If training is not reinforced or if it does not provide
subjects
with substantial new tools for sustained performance improvements,
performance will likely drop in direct correlation to the performance
that was
motivated by the attention event. Homeostasis at work; it all balances
out.
However, this does not mean that there is anything wrong with the
extrinsic
motivators; we just need to be aware of the psychological forces at
work, and
make sure we are productively using the attention events to foster
development rather than dependence.
Regarding faculty oversight as an application of the Hawthorne effect,
we
have to apply a certain degree of micromanagement when we are working
with
students who have no experience in applying teamwork to learning, and
with
students who lack the maturity for self-direction.
However, is there ever a point at which students can develop
sufficient
capacity and intrinsic motivation to collaborate without faculty
oversight?
In professional and adult development environments, managers and
teachers
attempt to facilitate individuals toward independence and teams toward
interdependence by helping them gain skills and motivation for
perpetual
development beyond the classroom. Fully delegating responsibility and
authority without any oversight is usually a mistake; but, we might
find that
continuous attention events can ultimately hinder development of
intrinsic
skills and motivation.
So, the question becomes, does TBL allow teachers to scaffold students
and
teams toward self-direction in learning or must the faculty keep
constant
vigilance regardless of the context?
Regards,
Brent
|
