Brain Training Doesn’t Work? A Critique.

You’ve probably seen the newest big headline in gaming: Brain Training Games Don’t Work. I haven’t seen too much of the discussion on this, but I’m sure there are people on both sides of the aisle. People who adored their Brain Age and Brain Age 2 games and played them constantly. Perhaps fans of Professor Layton teaching that “every puzzle has an answer” would disagree. On the other hand, I’m sure many people can’t possibly see how adding numbers really fast changes anything about your intelligence. Well, I see something like this, and suddenly the scientist in me jumps out and wants to read the research and find out what it actually says before making a decision. Well, I did that, and I’ve got my decision here for you all to read.

First of all, let me state some minor credentials. I have a PhD in aquatic ecology. I have learned to critically read and review primary literature. I took three separate statistics courses and have a good understanding of most statistical tests and understand the concepts of study design. I am not a sociologist nor a neurologist. I have taken basic anatomy and social psychology. I am definitely a nerd who has been doing logic puzzles as long as she can remember and does them to relax. Seriously, nothing like a good sudoku to chill out and not think too hard. Okay, now you get me, onto the article.

The article was in the journal Nature as a letter. Nature is one of the two top rated journals in the world. Science is the other one. This suggests that it is cutting edge research and has been peer reviewed by the best scientists in the field.

After reading this article, I lost a little bit of respect for Nature.

Overall, the research is okay, but has issues. The problem lies in the authors ignoring many of these issues or glossing over them with many references stating why they aren’t really issues. So, here are my major issues:

– Sample population.

Their sample population came from people who volunteered, many of which were watching the British TV programme ‘Bang Goes the Theory’. This population was pretty well educated, with more than 50% having college or higher degrees. This is already a biased population. I am very curious how their “benchmark” scores would compare to the average of the population in general. It might be difficult to greatly increase the performance of a group of people who are already relatively intelligent.

– Controls.

I think this study was lacking in an actual zero activity control group. What they were able to show here is that doing brain games does not increase these scores any more than doing another activity that involves reasoning and critical thinking. How do playing games compare to something that does not require processed thought? They didn’t test that. I would contend that if someone is playing a brain game versus watching a sitcom, there might be a bigger difference than playing a brain game versus googling a specific task. I am a google master; it takes thought to do that effectively.
– Time Spent Training

The participants were asked to train for 10 minute sessions, three times a week, for 6 weeks. What? I’ve had to learn a lot of things in my time as a student. I’ve studied for huge standardized tests and a doctoral candidacy exam where they could ask me anything about biology, anything. I’m also a musician and have learned vocal and piano pieces to perform for an audience. In my experience, if I had only accomplished these tasks for 10 minutes, three days a week, for 6 weeks, I wouldn’t have learned anything either. That’s a total of 180 minutes minimum, which was the control group average. Now, the experimental groups completed an average of 28 and 24 sessions, but that is still only a maximum average of 280 minutes, or about 4 and a half hours over 6 weeks. Now, the researchers say that the number of sessions did not correlate with any of the changes over time in the intelligence tests, therefore the number of sessions did not matter. I believe their training times were absolutely too low to even begin to call these results acceptable. When I was playing piano, I was practicing a minimum of an hour a day, and that was only my minor! Clearly, this needs to be redone with more training to be credible.

Also, I would like to note that the people playing games, either type of games, put in more sessions than the control group. That suggests people enjoyed them. Again, if the option is doing nothing or playing games, and if people are more likely to play the games, that has to be better than doing nothing or doing something truly mindless. Apparently, most people don’t enjoy doing logic puzzles for fun under normal, non-gaming circumstances (who knew!).
Now, I don’t hate everything about this study. I tried out some of the tests they use in the benchmarking. They are a good variety of different types of tests. They do mention the possibilities that more training could change the results. They try to hide it, but it is there. I also cannot fault their numbers. The numbers say there was no effect.

Now, I’m going to teach you a little something about statistics and science here to make my final point. In statistics, we have a null hypothesis which is simply: no effect seen. Then we have an alternative one: effect seen. The null hypothesis is the default, if the data cannot give good enough evidence for the alternative hypothesis, we accept the null. That does not mean we’ve proven the null hypothesis or no effect hypothesis true. Not at all. All this paper did was be unable to show an effect. That doesn’t mean an effect wasn’t there, or wouldn’t be there in another study that ran longer and had more training sessions. That is why I can’t believe this is a Nature paper. Nature is for something groundbreaking. This study was not that. There are other journals for papers like this, not ones that run national headlines that people without a statistics background will read.

On that note, statistics should be a required part of all high school education. Okay, that’s another rant for another time.

I’m sure there are people out there who agree or disagree with my review of this study. I am a scientist, I am fallible, and I am open to peer review of my own review. This is written in my, somewhat educated, opinion. My final opinion on the matter: clearly, using your brain is better than not, and therefore if Brain Age is an alternative to lounging on a couch watching Food Revolution (or Fat City for you Joystiq Podcast fans), then go for it. Don’t quit based on one, somewhat flawed, study.

Let me know what you think in the comments!


3 Comments (+add yours?)

  1. Aradia
    Apr 21, 2010 @ 21:07:42

    I think, regardless of length of time spent at a task, playing logic games such as sudoku teaches you to look at things from a different perspective. I don’t know if it will make you smarter, but I do feel that logic puzzles challenge you to approach problems form multiple angles, not just the same way you have in the past, which means you are using your brain in a way you aren’t accustomed to. And isn’t that the very definition of exercise?


  2. martin
    May 03, 2010 @ 07:27:44

    Unfortunately, I think that in addition to the flaws you mention the training regime was designed to fail. The scientists involved must have been aware that other studies had succeeded in showing transfer from brain training to general cognitive ability. And yet they ignored the stringent training requirements of those studies and implement something that would be far less likely to work.

    Why? To debunk populist brain game claims.

    Too bad.

    Martin Walker


  3. Revel Mob
    May 28, 2010 @ 04:24:36

    Great post, thank you! Since we are on the topic of Brain Training, have you tried this recently released – free – BrainAgeTest iPhone app:

    What’s your mental age?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

April 2010
« Mar   May »
%d bloggers like this: