The Reproducibility Project is a cover band, and like all good cover bands, they have a great name and a charismatic front man. Rather than reproducing classic rock singles, however, this band covers research projects.
Dr. Brian Nosek led a group of 270 psychology researchers in “covering” 100 classic research hits. Their album–called Estimating the Reproducibility of Psychological Science– just dropped in the journal Science and it’s a big hit. Seriously. It came out two days ago and there have already been nearly 300 articles about in the press.
Covering other people’s research projects–that is, conducting replication studies–is a core part of science. At least, it’s supposed to be. That’s why just about every research article ends with something along the lines of, “Hey, guys, don’t get carried away. This was just one study. We’ll need to do more studies before we know for sure that these results will stick.”
The problem is, at least in psychology, these follow-up studies almost never actually happen. It’s not that psychologists are stupid or lazy. It’s a question of motivation. Just as cover bands never get the same kind of cash, fame, or groupies as the original bands they’re covering, psychologists who conduct replication studies don’t get published in top journals.
Getting published in a big-time journal is to academic types what getting on the cover the Rolling Stone was to rock bands back when people paid attention to the Rolling Stone. Getting a credit in a top journal is the surest route to tenure–which, for eggheads, is like sex, drugs, and rock’n’roll in one package. Tenure means you have a rock solid job with great benefits, lots of time off, and, most importantly, a parking space close to your building.
So publishing a sexy new finding gets you published in a top journal and one step closer to tenure. Publishing a replication gets you published in a lower-tier journal (if you’re lucky) and one step closer to losing your teaching gig and starting over again with a part-time job at a college halfway across the country. In summary, replication studies are good for science and bad for your career.
The Replication Project is a big deal because they did cover band work but got rock star attention for it. They replicated 100 studies published in top journals and got their results published in a top journal. What they found, in a nutshell, is that many of these influential studies didn’t replicate very well.
The media angle has been that this is bad news for psychology. On the contrary, it’s great news. A hundred studies is just the tip of the iceberg and we haven’t even started looking at the quality of the replications themselves, but the support for the Reproducability Project suggests that we may finally be turning a corner on replications.
We all want progress. But progress means getting nearer to the place where you want to be. And if you have taken a wrong turning, then to go forward does not get you any nearer. If you are on the wrong road, progress means doing an about-turn and walking back to the right road; and in that case the man who turns back soonest is the most progressive man. -C.S. Lewis
Consider this my standing ovation for the 270 researchers who had the courage to join Dr. Nosek’s cover band and the 97 authors of the original studies who assisted them in their replications.