Hey all,

 

Over the last couple of years, I've taught an Introduction to Data and Databases course. For the first two semesters, I taught face-to-face using team-based learning methodology. Over the next two semesters, I revised the teaching methodology for hybrid-online and fully online. I was pleasantly surprised by how well TBL adapted to the online environment. I'd like to share some of my approaches and experiences. I'll start with the Reading Assessment Process (RAP).

 

Background

 

I am a part-time instructor in the Informatics Department at the University at Albany (SUNY). I spent 10 years as a lawyer, 15 more years as a game designer, and have been working in process improvement and business intelligence in the manufacturing sector for the last 4 years. As you can imagine, I'm fairly process oriented -- forgive my obsession with "rules".

 

Philosophy

 

In designing any process, the objectives and purposes are the best starting point. My thinking about the RAP has evolved over the years. The online processes that I developed are hugely dependent on my take about the goals of the RAP.

 

When I started, I viewed the RATs as pure and simple tests. The students were to read (or view) the materials, take the tests, and be graded. That way I would know if they had done the reading well or poorly, and would have something to add to their cumm grade. I designed questions by focusing heavily on the materials. One answer was the "correct" one (often quoted directly from the source materials) and the others were not. To make the questions more challenging, I spent a fair amount of time devising plausible, but incorrect answers. When it came time to review the answers, the students simply accepted my "correct" answer or were annoyed at the way the other answers were tricky or vague.

 

I realized I was putting a lot of effort into deceiving (or confusing) the students. The better I got at creating plausible but incorrect answers, the greater the deception. That didn't sit well with me. I know the material far better than they do. The fact that I could deceive one or more of them each test accomplished nothing . . . and was downright mean.

 

I also found that the few questions that I seeded with more than one correct answer (in an effort to create "appealable" issues) produced the best discussions and the most meaningful appeals. In those cases, the "correct" answer, as indicated by the answer key, was merely a starting point for a larger discussion. The more questions that had multiple correct answers, I more I encouraged the students to "buck the system", discount the "correct" answer, stick to their guns, and support their answer. Capping the exercise and reinforcing the point, I gave them full credit as long they could give me a reasonable argument for their answer. It was a tough road, however. Students are used to seeing tests has teacher-controlled exercises with one right answer and a bunch of wrong answers.

 

Fairly quickly, I started eliminating questions with plausible but incorrect answers. I started using, as a general course, questions with at least two correct answers (or at least two justifiable answers). The more I shared appeals presenting alternative answers, drew out explanations, balanced them against the "correct" answer, and liberally awarded full points to the appealing groups, the more the student realized that RAT questions were a starting point for discussion, not a black-and-white evaluation of their preparation. The more correct answers I seeded the questions with, the more robust the discussion. The RAP became a process to engage the students with the materials, not an end-point testing the students' mastery of the materials.

 

In devising multiple correct answer questions, I found myself naturally pulling back from the materials. I could cover more material in one question if more than one answer was correct. I found it easier to create application-, implementation-, synthesis-, or analysis-oriented questions, using the materials as a starting point for novel situations. That too created more robust discussions. There were fewer and fewer easy questions and lots and lots of justifiable answers.

 

A happy side-effect was that I could draw in the students who did not do the reading. As long as they read the question carefully during either the iRAT or tRAT, listened to their team members during the tRAT, and contributed (as part of the group) to the appeal discussion, they were exploring the ideas and could achieve a decent grade.

 

Looked at as engagement and discussion seeds, the components of the RAP needed to be re-weighed. The iRAT is least important. It's primary purpose is to introduce the students to the questions. Whether they get the answer right is far less important than their review of the possible answers. The tRAT is more important but not much. It's an opportunity for the students to share their ideas, take a stab at a correct answer, and discuss possible rationales. The most important, by far, aspect of the RAP is the appeal process. That's where the students justify their answers and receive feedback.

 

With a grading structure of 25% for the iRAT and 75% for the tRAT, as modified by the appeals rationales, the purpose is reinforced. Also, I make at least one appeal mandatory for all teams. This reinforces the notions that (1) questions have more than one potential "correct" answer, and (2) only if the team probes the alternative answers through the appeal process can they benefit from these correct answers. As the semester goes on, more and more teams appeal more and more questions. Some teams catch on quickly and create an appeal from every question, because . . . you never know.

 

Process

 

My process relies on tools available in Blackboard. Frankly, I've never used another CMS so I can't say if similar tools exist elsewhere.

 

First, I create a pool of 10 RAT questions, each with five different answers. I use "all of the above" and "none of the above" liberally. I also use "some of the above" to further encourage thinking about alternative correct answers.

 

Using that pool of questions, I create the iRAT using the test tool in Blackboard. I set the question order to be random and the answer order (within that question) to be random. I allow the students to take the iRAT as many times as they like with two conditions: (1) they don't know their iRAT results until after the tRAT answer sheet (see below) has been submitted, and (2) they cannot start the iRAT after a certain deadline. I'm perfectly happy to have the students review the test more than once. That furthers engagement with the materials.

 

Here are the iRAT assignment instructions:

 

"The following test has 10 questions, each worth 10 points. Choose the best answer for each.

Make a note on the full text of your answers (or enough of it to remind you which one you choose) so you have a record of your choices to reference during the tRAT. Noting down just the letter (A., B., C., etc.) of your answer will not be sufficient as answers are scrambled for each test.

You have 30 minutes to submit your answers. The test will time out after that period of time and auto-submit.

You will not be notified of your score on this test until after submission of the tRAT for your team.

You can retake the test as often as you like (prior to the due date), but your final iRAT score will be based on your latest submission."

 

Once the deadline for the iRAT passes, I open up the tRAT assignment in Blackboard. The tRAT has two parts: the test and the answer sheet assignment. Unlike the iRAT, the tRAT has a set order for the questions and a set order for the answers to ease grading. I ask that the students gather in some synchronous environment (chat, Skype, Google Hangout, etc.) and take the tRAT together. Again, the students can open and run through the test as often as they like. No results are provided for the test so repeat review is not a problem.

 

Once the students have had a chance to review and discuss the iRAT, one of them submits an answer sheet to me. That sheet lists the questions in order with a first, second, and third best answer to each question. The answer sheet submission is open to any member of the group, but only one member can submit the sheet and it can only be submitted once.

 

Here are the tRAT assignment instructions:

 

"Complete this test and the RAT Team Answers assignment at the same time. The RAT Team Answers assignment can be found listed in your group area. Do the following:

 

Schedule, then gather your team at one time, communicating in person, via chat, using Google Hangouts, Facetime, Skype, or another means.

Once your team has gathered, discuss each question and choose the best answer for each.

One (or more) team members should take notes on which answers the team favors. Pick a first, second, and third choice for each question.

Once your team has decided its 3 choices per question, one person should submit the RAT Team Answers assignment, listing the three choices per question, as well as the names of the team members who participated in the team test (team mates who don’t participate get a 0 on the tRAT). List the text of the answers as well as the letter choices, to make sure your grade is accurately calculated.

 

The following test has 10 questions. Getting the correct answer on the first choice is worth 10 points; getting the correct answer on the second choice is worth 5 points; getting the correct answer on the third choice is worth 3 points; getting none of the choices correct is worth 0 points.

You have 40 minutes to submit your answers. The test will time out after that period of time and auto-submit.

You can view the test as many times as you like. You can submit your RAT 2 Team Answers assignment only once.

You will be notified of your score on this test shortly after you submit the RAT 2 Team Answers assignment.

Your team's appeal assignment is based on the results of the team RAT (see separate RAT Team Appeal assignment)."

 

I then grade the tRAT answer sheet. This is a relatively quick and easy process because the question order is always the same and the correct answer key (a, b, c, d, or e) is always the same. If the team gets a question "wrong", I provide the correct answer key when I respond to their iRAT answer sheet assignment in Blackboard. Because the tRAT answer sheet is a team assignment in Blackboard, I can input one grade result and it flows down to each member of the team. I then simply have to modify the grade for those team member who didn’t participate to 0.

 

Once I've finished grading the tRAT answer sheets, I open the appeals process. Again, the students gather to discuss their answers, create rationales for them, and write up the appeal document. Again, any of them can submit the appeal document but only one of them can submit it and only once.

 

Here are the RAT Appeals instructions:

 

"Once you receive your grade on the RAT Team Answers assignments, you have the opportunity to appeal the results. You must discuss and appeal as a team. Follow this process:

 

As a team, discuss, either at the same time (as you did for your team test) or using your team RAT Appeals discussion forum (appeals discussed on the full-class discussion boards will result in 0 points on the appeal), any incorrect answers that you believe were as good as the correct ones. You can appeal as many question results as you wish, but must appeal at least one question.

Draft and agree on a statement for each appealed result specifically explaining the ground for your appeal and citing any support from the readings or from other sources.

Submit all appeal statements using this assignment. This assignment can be submitted only once.

 

If any of your appeals are approved, you will gain points on both your team test and any individual tests that picked the appealed answers (instead of the "correct" one). Note that you can take an appeal from a RAT question that you got right on the tRAT, but one or more team members got wrong on the iRAT. You just have to get your team to agree to submit the RAT Appeal."

 

In my last class, I responded directly to the groups on their appeals, replying to any questions or points made through Blackboard. A better method would have me setting up a discussion forum for the entire class labeled RAT Appeals. I would create a new thread in that discussion area for each RAP. In that thread, I would present a long entry setting out each question and its answers, the various appeals taken from that question, my response to the appeal arguments, then an grant/rejection of the appeal. Students could review that thread to discover which appeals were made, how they were argued, and which of those were granted and which were denied. Also, students could reply to the thread, furthering the discussion if they like.

 

Finally, at the end of the semester, I create a fifth, final RAP which has only an individual test. That test is made up of a random assortment of the questions from the prior four RAPs, in a random order with the answers randomized. That encourages the students to re-engage with all their prior RATs at the end of the semester.

 

One result I found occurring quickly in the RAP process. The students would skip the simultaneous gathering portion and simply exchange their answers and rationales asynchronously via email or IM. That does undermine the give and take of the group discussion, but I decided that, if that's how the group wanted to handle their work, that's fine. They are still engaging with the materials.

 

Another repercussion (not unique to this process) was that some student contributed more and some less, particularly if they dropped into an asynchronous communication pattern. So be it. Absent an in-class environment, I can't control how much they participate in their learning. Even with in-class activities, a student can always mail it in or sleep through it.

 

The solution to that problem is not a better RAP. The solution to this and all other group contribution issues is the peer evaluations. As long as peer evaluations are worth a large portion of their grades (20-25%), and are conducted regularly throughout the semester so the non-participants have notice, active group participation is incentivized nicely. I'll discuss more about that later.

 

I appreciate your patience as I rambled on. Hopefully this is of some use to some of you,

 

M Alexander Jurkat

INF 202 Team Lead

Informatics Department

University at Albany