Research design meets actual people: 7 Generation Games

Today was my most recent experience in the clash of commercial and academic cultures. For seven years, I was an assistant and then associate professor, teaching statistics and research methods, writing articles for academic journals. For five years before that, I was a graduate student at the University of California. I even did a post-doc on an NIH fellowship. All the research things. So, I am well aware, as my colleague at lunch today was telling me, that the National Science Foundation prefers studies where subjects are randomly assigned to experimental and control groups. Randomized controlled trials are the gold standard of research, as the textbook I’m using to teach biostatistics says about every third page.

“Yeah, we’re not doing that.”

In response to her shocked look, I explained,

“Last year, when we did the pilot study for 7 Generation Games, we were able to get a control group but that was before we had our data in that showed the children at the two schools that played our game did significantly better in math. Now, the superintendent of schools is telling me that they want to be in the experimental group, too, because he is not going to face down parents and tell them that their children did not get to use a program that he believes would have helped their children do better in math because — well, what reason could he give them that would be satisfactory? Because it would help us provide more credible evidence to the Institute for Education Sciences? Seriously, why the hell would that parent on an American Indian reservation chance that their child would perform worse in math so that we could get better data for our study? Does that make sense to you? As for randomly assigning them to be in the group to play the game – how can we do that? We can offer it to low-income schools at no cost but if they are over-worked with a hundred competing responsibilities (as many, many teachers are), already have their curriculum planned for the year, are part of some reform that doesn’t allow them to do anything but a specific lesson plan or unwilling/uninterested for any other reason, we cannot MAKE them use the game.”

She nodded that she saw my point but then suggested hopefully that we could do a random assignment by classroom, within school.  I told her that we had that idea, too, but the teachers who used the game and found their students doing better and enjoying math class more shared the link to download the game, and the teacher resource site with other teachers at their school. My colleague exclaimed,

“That’s terrible! You have contamination of implementation!”

I corrected her,

“Or, as the teachers called it, not being an asshole. Look, say you’re a teacher in a school without a lot of resources where children are generally performing below grade level. You get this new game that your kids love to play and they are doing better on their math tests. The teacher next door asks can she get a copy and you say, “No”, because a bunch of researchers want to see how much worse the kids in the class next door will be doing by the end of the year. Of course you don’t do that. You share it with the other teacher because you care about students in your school and you also don’t want her throwing an eraser at you.”

================================================================

You can buy a copy of Spirit Lake: The Game for $9.99 and we’ll even give a free copy to a Title I school to boot

====================================================================

What are we doing to solve our research dilemma? Well, since it is a computer game backed up with a  database of student performance data, we can track how long and how many times each student plays our game. In the pilot study, we found that not only did the intervention classrooms perform better than the controls, but that students who played the game more out-performed those who played less.

An additional possibility is to do a pre-test at our same intervention schools next fall, then a post-test after eight weeks. Start the game in the ninth week of school and then test again after another eight weeks. We do have some schools we could use as control groups but they are so different to not really be helpful – our intervention schools are primarily on or near American Indian reservations, so using control groups in downtown Los Angeles would not be that informative, I don’t think.

A cross-over design has been suggested, but there are those teachers again, who I have to meet with and say,

“Look, I know that we asked you to use this game because we thought it would help your students do better in math and they would enjoy it. Now that it seems to be working, we want you to stop and see if your students do worse for the next couple of months. What do you say, because, you know, science.”