Dear Sabine,

I’m writing to give you a heads up about a potential problem with using
ONLY the Readiness Assurance Tests and not following them up with
applications. The single most common mistake that faculty make in
attempting to implement TBL, and the one that causes the most student
push-back, is using too many RAT’s. The push-back arises for two reasons.
One reason is that when you implement TBL you are, in effect, making a
implicit bargain that, “If you will do the pre-class preparation, I’ll make
sure that you are rewarded by having the opportunity to see why the ideas
are important to you.” The real payoff for students comes from
opportunities to practice using the content on well-designed (i.e., “4-S”)
assignments. That is how you pay them for their effort to prepare for the
RAP.  In fact, having challenging 4-S applications is ultimately the single
most important aspect of successfully implementing TBL.  If students are
able to see why the material is important, everything pretty much falls
into place. If you don't do  the applications, they may not see the
payoff--even if there is one.

The other reason that too many RAT’s produces student push-back is that one
of the benefits of the RAP is the single most powerful activity that I know
for building team cohesiveness. If, however, the teams are not sold on the
value of the RAT’s (which naturally happens *WHEN/IF* the RAT’s are
followed by good 4-S applications) the cohesiveness can work AGAINST you.
That happens when the cohesive groups are united in a belief that you
aren’t doing what they think (and what normally) is the teacher’s
job--delivering the content. Instead, you are making them do all the work.

That said, I’d strongly recommend selling the value of the individual->team
testing sequence ON ITS OWN. In part, that would mean being very clear
about the fact that you think they will learn more through talking with
each other than the ever would simply listening to you (which I clearly
think IS true) and provide them with the rationale and evidence for your
belief. In addition, If you mention TBL at all, I would be very clear that
you are only using part of TBL and specifically NOT call the tests
Readiness Assurance Tests (because that’s not why you are doing it anyway).
(Note: In one of the early articles on TBL I called them Informative
tests—because they were more formative than summative in nature and because
the team tests produced so much peer teaching—Michaelsen, L. K. Watson, W.
E.  & Schraeder, C. B. (1985). Informative Testing: A Practical Approach
for Tutoring with Groups.  *The Organizational Behavior Teaching Review*, *9
*, 18-33). Finally, I think you are going to be really careful about how
much they will affect students’ grades. On one hand, you need to ask pretty
difficult questions or they will simply defer to a really talented member
instead of talking things through—which is what facilitates peer teaching.
On the other hand, low individual scores make them really nervous. As a
result, you will have to tread a very fine line between counting them too
much (which, to some will seem to be unfair—i.e., testing before lectures
is NOT what they are used to) and counting them too little (or you’ll risk
having too many individuals fail to do the necessary preparation).

Larry


On Tue, Jul 2, 2013 at 9:48 AM, Peters, Sabine <[log in to unmask]> wrote:

> Good Morning,
>
> Thank you so much for your reply. You're absolutely correct, at this point
> it's "only" the RAP (and IF ATs) I'm focusing on, and it will certainly be
> laid-out that way. Given my time-constraints and the organizational
> factors, along with all the logistics involved in making it happen, this
> seems to the best course of action for the time being. It may never come to
> full-up TBL implementation there, but the way I look at it, it would be a
> step in the right direction to facilitate learning and (hopefully) improve
> their students' performance.
>
> Sincerely,
> Sabine
>
> ________________________________________
> From: Sandy Cook [[log in to unmask]]
> Sent: Monday, July 01, 2013 7:03 PM
> To: Peters, Sabine; [log in to unmask]
> Subject: RE: Testing is Learning
>
> Dear Sabine,
>
> It is exciting when people are using elements of TBL and find that it
> brings energy and excitement to the learning process (for both students and
> faculty).  I don't see in your description any mention of you using all
> components of TBL (RAP, Application, Peer eval, etc).  If not -  you are
> missing some of the strengths of complete TBL.  As you write this up - if
> you are only using the RAP component of TBL, I would suggest that you
> highlight that it is only a portion of the whole process - not the full TBL
> process, so that people do not get confused and think that TBL is just RAP.
>
> ********************************************************
> Sandy COOK, PhD | Senior Associate Dean, Curriculum Development |
> Medical Education, Research, and Evaluation (MERE) |
> W: (65) 6516 8722| F: (65) 6227 2698 |
>
> Assistant Manager: Belinda Yeo | [log in to unmask] | 6516-8511
>  Important:  This email is confidential and may be privileged.  If you are
> not the intended recipient, please delete it and notify us immediately; you
> should not copy or use it for any purpose, nor disclose its contents to any
> other person.  Thank you.
>
> -----Original Message-----
> From: Team-Based Learning [mailto:[log in to unmask]] On
> Behalf Of Peters, Sabine
> Sent: Monday, July 1, 2013 10:59 PM
> To: [log in to unmask]
> Subject: Re: Testing is Learning
>
> I must say the timing of this post could not have been more perfect. I'm
> in the beginning stages of setting up my dissertation prospectus and the
> idea behind RP is exactly what I want to focus my research on. Up until a
> few days ago, I didn't realize that there was an actual term for this
> approach, so thank you for that. I intend to follow the RAP with the IF ATs
> (as a review tool, administered twice a week over a four-week data
> collection period) to determine if student performance (along with
> confidence and level of test anxiety) on standardized tests in the foreign
> language learning arena (EFL) will improve. From there, depending on the
> outcome, I hope to encourage the institution to look at implementing TBL as
> a possible instructional strategy, at least on a trial basis. If I can
> demonstrate that RAP can be effective in this instructional environment,
> maybe the leadership will be more inclined to allow further
> "experimentation" in this direction.
>
> I have conducted a small pilot study (N = 16, 4 teams of 4 students in two
> different language classes) at the school back in March using just RAP (and
> IF ATs) and it was very exciting to see this "energy" during team
> discussions first hand. I can't wait to go full-up with it in the fall.
>
> Introducing the TBL concept gradually seems to be the way to go. Nobody
> there had ever seen or even heard of TBL, the RAP and IF ATs but the
> leadership seemed intrigued, so I'm optimistic.
>
> I found a few recent studies that were conducted using RP (Pilotti, M.,
> Chodorow, M., Petrov, R., 2009; Tullis, J., Finley, J., Benjamin, A., 2012)
> and I'm wondering if there are any that were done in the language learning
> arena. Are you aware of any?
>
> It's so ironic, I used to make little review quizzes for my language
> students all the time (I've taught German and ESL in the past, and my
> students loved them in both settings) to help with content acquisition and
> exam prep without any thought of possible research implications. Now it's
> going to turn into my life's work in academia. Life is full of surprises
> ....
>
> Thank you so much for your post!
>
> Sabine Peters, Major, USAF
> PhD Candidate, The Florida State University
>
> ________________________________________
> From: Team-Based Learning [[log in to unmask]] on behalf of
> Michael Kramer [[log in to unmask]]
> Sent: Saturday, June 29, 2013 7:44 PM
> To: [log in to unmask]
> Subject: Testing is Learning
>
> There is an ongoing research program on something called Retrieval
> Practice (RP), which could have profound effects on pedagogy, and which I
> am attempting to integrate into my new TBL course that begins in September
> 2013.
>
> I have done a search of this listserv and found no references to RP, so I
> would like to open a new thread on it.
>
> Here are some links on RP:
>
>
> http://blog.questionmark.com/professor-roddy-roediger-on-applying-the-retrieval-practice-effect-to-creating-and-administering-assessments
>
>
> http://www.purdue.edu/newsroom/research/2011/110120KarpickeScience.html
>
>
> http://scholar.google.com/scholar_url?hl=en&q=http://testology.co.il/articles/retrieval_practice.pdf&sa=X&scisig=AAGBfm0RNTTRVFy6ve0yvTKPXhLfIzWw9w&oi=scholarr
>
> The key insight of RP research is summarized in the first paragraph from
> the Karpicke-Blunt article, above. Assessments do not merely inform
> instructors and students of how much  knowledge the students acquired
> during previous learning sessions. Rather, assessments are themselves
> learning experiences.
> Not only that, there is a growing body of evidence supporting the claim
> that assessments are one of the most powerful tools available for creating
> learning: more powerful than repeated studying or creating concept maps.
>
> The term "assessment" in the context of the RP research program comprises
> any attempt to recall information as long as THE INFORMATION IS NOT
> AVAILABLE DURING THE ASSESSMENT.  In the context of a typical classroom,
> that means that books and notes cannot be used. The essence of an
> assessment is that the student is recalling information from memory.
>
> Note that the RP concept of assessment differs fundamentally from the
> well-known concept of "formative assessment." Formative assessments are
> administered for the purpose of determining how much test-takers have
> learned. They are not administered because the test-giver believes that the
> test, in itself, will increase what test-takers have learned.
>
> In the context of TBL, iRATs are assessments. It is important to note,
> however, that if someone were to give "iRATs" online, the likelihood is
> high that students would use books, notes, internet searches, etc. Thus,
> they would not be retrieving from memory. Accordingly, learning from online
> "iRATs" is likely to be significantly attenuated compared to closed-book,
> in-class iRATs. The tRATs, especially when using IF-ATs, enhance the iRAT
> process, because feedback enhances learning outcomes.
>
> The RP research program does not address group learning, so I can't draw
> any conclusions about group activities, per se.
>
> However, memory research in general, and RP research in particular,
> demonstrate that the typical classroom structure leads to learning outcomes
> that fall significantly short of their potential.
>
> If we take a paradigmatic TBL course, the semester is divided into units.
> Once a unit is complete, the likelihood that assessments and application
> exercises will refer to prior units is low. In order to learn, it is
> essential that learners be prompted to recall previously learned material
> repeatedly over extended (days, weeks) periods of time.
>
> This suggests that RAPs and assessment activities should on week 8, say,
> should be referring to material covered in weeks 1-7.
>
> Another possible implication of RP research for TBL is that the fear of
> overtesting is misplaced. Virtually all human beings, if RP research is
> confirmed, lack a proper understanding of the best learning strategies. We
> all tend to overestimate how much we understand after we've heard or read
> something. We all tend to test our recall less than would be ideal to
> optimize our learning.
>
> For the last few weeks, I have been attempting to use RP learning
> strategies in my own studies. Instead of taking notes as I read a text, I
> read a few pages or a passage in a text, then I put the text away. I then
> write what I recalled of the text. Then I review the text and write some
> more.
>
> What I have not done is repeat these exercises days and weeks later.
> Changing habits ingrained over decades is not easy. But if the result is
> more efficient learning, it would be worth the effort.
>
> As I plan for my new September TBL course, here are some modifications
> that I am considering:
>
> 1. Give RAPs weekly. These are closed-book quizzes on the reading, which
> have been demonstrated to improve learning.
> 2. Include questions on RAPs that cover prior weeks' content. I did not do
> this in my Fall 2012 TBL course, which was my first attempt at TBL.
> 3. Have students do individual freewriting activities in class, in order
> to increase the number of retrieval practice sessions. This means that
> students write continuously for a number of minutes on a topic without
> editing or attending to penmanship, grammar, or, even, correctness of
> information.
> 4. Have students pair off and "freespeak" for two minutes on a focus
> question. This mean one student speaks continuously and the other student
> merely listens and nods. Then the students switch roles.
> 5. Make some application exercises closed-book
>
> I'll end this too-long post with a personal anecdote. For the past 10 or
> so years, I begin my classes by playing the "name game." My classes are
> designated "writing intensive" so they are capped at 25 students. I
> recognize that many here teach classes 10 times as large as mine, so this
> activity would not be possible for them.
>
> In the name game, we all stand up (if we are physically able) and form a
> circle. The first person states her name. Then the next person states the
> first person's name followed by her own name. The third person states the
> first person's name, the second person's name, and then his own name.
>
> I got this idea when my son started preschool and the preschool teachers
> began a parents' meeting by doing this name game.
>
> It is only since I started researching RP did I realize that this name
> game is a powerful form of retrieval practice and has been researched. It
> turns out that it is more effective to run it in reverse. That is, if I am
> the seventh person in the circle, I announce my name, then I state the name
> of the sixth person, followed by the fifth person, fourth person, etc.
>
> In the past, I merely used the name game to send a message that the focus
> of this class would not be the "sage on the stage" but the "scaturient
> student."
>
> Having studied RP I now realize that I need to state explicitly that the
> point of the name game is that we learn best by forcing ourselves to recall
> repeatedly the information we have been exposed to. In a matter of minutes,
> almost everyone learns almost everyone's name. The lesson is clear and it
> is a message I will need to hammer home every class: repeated recall of
> studied material is an effective method for deepening learning.
>



-- 
*******************************
Larry K. Michaelsen, Professor of Management
Dockery 400G, University of Central Missouri
Warrensburg, MO 64093
660/543-4315 voice, 660/543-8465 fax
For info on:
Team-Based Learning (TBL) <www.teambasedlearning.org>
Integrative Business Experience (IBE)
<http://ucmo.edu/IBEl<http://faculty.ucmo.edu/ibe/home.html>
>
*******************************