Evolutionary Psychology

Evolutionary Psychology is moving to SAGE. The new address is evp.sagepub.com. Submissions here.
Robert Kurzban

The Evolutionary Psychology Blog

By Robert Kurzban

Robert Kurzban is an Associate Professor at the University of Pennsylvania and author of Why Everyone (Else) Is A Hypocrite. Follow him on Twitter: @rkurzban

What We Have Here is a Failure to Replicate

Published 19 April, 2013

On Monday of this past week, Hal Pashler gave a talk as part of the Psychology Department’s colloquium series at my home institution, the University of Pennsylvania. His talk focused on the issue of replication in certain areas of experimental psychology. Some readers might recall something of a stir in the blogosphere when Doyen et al. reported a failure to replicate work that looked at whether priming people with words related to old age, such as “bingo,” and “Florida,” caused people to walk slower than those not primed with such words. Pashler discussed some work that he and his colleagues also published in PLoS reporting their attempts to replicate related studies showing that subjects who plotted points closer together experienced feelings of greater closeness to their families relative to subjects who plotted points further apart. I’ve put the results of the closeness study here in the Figure below. In his talk, Pashler discussed a number of other attempts to reproduce results of this general type, all with the same result: a failure to replicate.

Figure from Pashler et al. (2012). Ratings of closeness by condition.

Figure from Pashler et al. (2012). Ratings of closeness by condition.

The topic of replications has been discussed at some length, and I’m not in a good position to contribute anything substantive to this discussion, but I thought I would spend a few moments musing about the topic for a few reasons. First, there’s a new paper (paywall) by Ioannidis & Doucouliagos “What’s To Know About The Credibility Of Empirical Economics?” Holding aside their rather dismal evaluation of the dismal science — “the credibility of the economics literature is likely to be modest or even low” – I particularly liked the way the authors quite dryly expressed the general problem: “Replication is a public good and hence prone to market failure.” The footnote this with a reference to Dewald et al. (1986), who wrote: “A single researcher faces high costs in time and money from undertaking replication of a study and finds no ready marketplace which correctly prices the social and individual value of the good.”

Pashler had a great slide, a picture of a passage from a second grade textbook, telling the (young) reader that a cornerstone of science is replication. Given how rarely research is, in fact, replicated in many areas of science, the point is well taken. Which is not to say that there aren’t efforts being made to try to address the problem, including, for instance, the Reproducibility Project and Psych FileDrawer.

Discussions of why replications aren’t more common – including Pashler’s remarks – focus extensively (but not exclusively) on incentives. If a researcher attempts to do an exact replication of published work, there are two possible results. If the result replicates successfully, it is likely to be difficult to publish because journals tend not to publish replications, though this is changing. Last month, for example, Bobbie Spellman announced an initiative at her journal, Perspectives on Psychological Science,  providing an interesting mechanism for publishing replications. Other journals are proving more receptive to publishing replications – and failures to replicate – which will probably have some beneficial effect. In any case, my guess, though I don’t know, is that replications of results are cited relatively infrequently, especially compared to the original results. Publishing failures to replicate is likely no easier than publishing successes.

The issue of incentives does not, of course, end with authors. One issue that the editorial team at Evolution and Human Behavior is discussing is what our policy ought to be in this regard. While I myself feel that the sort of Registered Reports that PoPS is soliciting have tremendous value, what will the effect be on the journal? There is little use denying that in the present era, journals – and their editors – are judged on quantitative metrics, especially citation counts. To the extent that replications, successful or not, draw fewer citations than new research, publishing replications entails a cost to the journal, exactly along the lines of the Dewald et al. quotation above: publishing such papers is enduring a cost to produce a public good.

If its’ true that publishing replications reduces the infamous impact factor – as well as other metrics – authors are affected as well. At many institutions, departments and personnel committees use metrics such as impact factor to evaluate the quality of the journal that candidates up for promotion are publishing in. Would contributors to particular journals be willing to pay the price of the reduced impact factor to support a policy of publishing replications?

This is not, exactly, a rhetorical question. I’m interested in the question of whether members of the evolutionary psychology community believe that E&HB should encourage/tolerate/permit the publication of replications and failures to replicate. (Please feel free to contact me offline. No need to make your thoughts public unless you want to.) I should note a couple of points. First, the journal doesn’t receive many replications, successful or otherwise. Second, I recently green-lighted a paper that was as close to a replication as one can do, given that the study was executed in a very different context from the initial study. So, there is a sense in which the journal is already in the business of publishing replications. Should it be?

Citations

Dewald, W.G., Thursby, J.G. and Anderson, R.G. (1986) Replication in empirical economics. The Journal of Money, Credit and Banking Project. American Economic Review 76: 587–603.

Doyen, S., Klein, O., Pichon, C. L., & Cleeremans, A. (2012). Behavioral priming: it’s all in the mind, but whose mind?. PLoS One7(1), e29081.

Pashler, H., Coburn, N., & Harris, C. R. (2012). Priming of social distance? Failure to replicate effects on social and food judgments. PloS one7(8), e42510.

Ioannidis, J., & Doucouliagos, C. (2013). What’s to know about the credibility of empirical economics?  Journal of Economic Surveys.

  • http://twitter.com/asehelene Åse Kvist Innes-Ker

    I’ve been thinking about these questions for a while (read – followed on twitter, read blogs, gone to conferences, and blogged myself about it). I’m for replications, registration, changes that makes science more reliable and robust. Chris Chambers has started an interesting initiative at Cortex, that I think people are watching. Overall, the deluge of iffy (but p<.05) papers in the name of impact factor (wasn't that devised so that the libraries would not end up looking like a hoarders home?) is kind of depressing. I would think, also, that for Evolutionary Psychology, it is actually important to make relatively close replications in different populations. So, I vote for more of this.

    • rkurzban

      Thanks for your comment. Cortex is one of the models we’re looking at. Anyway, thanks for the vote…

  • http://popsych.org/ Jesse Marczyk

    Replication is no doubt an important issue in research, psychological or otherwise. I would not like it to take center stage as THE issue, though. That is, I can’t help but wonder whether a lot of bad work could be be avoided without the all-important impact factor concerns if journals required researchers to actually have theory present in their research. I can’t count the number of papers and presentations I’ve read/attended where the sole purpose for the research being conducted – the reason the researcher undertook the project – was that the authors had some personal hunch they would find an effect (that is assuming the researchers even provide a reason they conducted the research which, in many cases, they do not).

    I’m obviously biased in how important I feel theory is, and I’m not saying that i think requiring the inclusion of a functional level of analysis to all papers would solve all the problems, but I think it’s important enough to give serious consideration to.

    • rkurzban

      Thanks for your remarks, Jesse. When Pashler was giving his talk, he presented the “theory” that motivated the work in the papers he was trying to replicate, and in several cases the audience laughed. My sense, however, is that not only would those theories have been taken seriously by other audiences, the theories at stake would have been considered to be at the cutting edge. So, it’s not that the work in question wasn’t guided by theory… it’s just that one person’s theory is another person’s punch line.

      • http://popsych.org/ Jesse Marczyk

        I’ve always enjoyed this quote from Pinker:

        [Social Psychology] has been self-handicapped with a relentless
        insistence on theoretical shallowness: on endless demonstrations that
        People are Really Bad at X, which are then “explained” by an
        ever-lengthening list of Biases, Fallacies, Illusions, Neglects,
        Blindnesses, and Fundamental Errors, each of which restates the finding
        that people are really bad at X. Wilson, for example, defines
        “self-affirmation theory” as “the idea that when we feel a threat to our
        self-esteem that’s difficult to deal with, sometimes the best thing we
        can do is to affirm ourselves in some completely different domain.” Most
        scientists would not call this a “theory.” It’s a redescription of a
        phenomenon, which needs a theory to explain it.

  • TimothyBates

    For the next couple of years, failures to replicate may be _more_ citable than average papers. After that, hopefully editors raise standards regarding internal replication, so that the number of failures to replicate will drop off. For direct replications that succeed, you might adopt the rule of three (not accepted where three studies in the literature already show the effect, otherwise accepted). So, you might raise the IF of journals taking on fake findings. But that said, publishing failures saves the field so many wasted years of researcher time following fake leads and bogus claims, that it’s worth any effect on factors like IF. Researchers might both cite from, and submit their better work to, journals that support taking on null findings. As Darwin said “I have no faith in anything short of actual measurement and rule of three”.

  • David Frederick

    My vote is for papers with multiple attempts at replications + extensions. For example, replicating the original study twice with new and improved measures, especially if it was published in EHB, as a way of continuing the dialogue about the topic in the journal.

    Or replicating the study directly through Study 1, and then extending it in Study 2.

    Direct replications + extensions could be encouraged to be substantially shorter than the original in terms of intro/methods/discussion if the key theory and methods are very similar to the original. This helps minimize the trade-off between between journal space and impact of replications. I like the short-report format that Body Image has, which limits text to 2500 words + 25 references number of references (though I don’t like the limit on figures and tables).

    I am sure I am not alone in attempting replicate widely cited papers that have one experiment with a small sample, and then fail to find the purported effect despite a larger sample size (or what have you), and then debating whether it is worth conducting two more studies to show that the effect isn’t popping up versus focusing my time on projects that “work” and writing those up so that I have a better shot at a journal like EHB.

  • JaziZilber

    Here is a great effort i think is useful on replications.

    http://blog.dansimons.com/2013/03/registered-replication-reports-at-aps.html

    Dan Simons organized a system for publishing replications.

    1) The protocol for replications will be published in as much details in advance (original authors can commetn then, and will not be able to excuse later on)

    2) multiple replications will be aggregated.

    3) The publication venue (a highly rated journal, i think) will have a defined corner for replications. I have no idea how this would affect the various reputations coins.

    We gain many things here

    1) no longer hazy arguments which replication was done “right” and which replication counts etc.

    2) I believe a dedicated journal for replications is the best way to handle the interests issues.

    PS. isnt it possible to have a dedicated corner for replications that will count separately in the metrics? (are letters to the editor and 2 lines pieces counted always to the stats??)

  • http://www.facebook.com/greengross Gil Greengross

    I think that the benefits of replications outweigh the possible costs that you mentioned. One way to overcome the IF trap, is by asking writers that mention a study that was replicated to include the replication citation in the manuscript. This is something that reviewers and editors can emphasize and would help to change the culture of what is published. I often see a paper citing a results that was later not replicated without mentioning it at all. Journals can also devote relatively little space to the replication papers, after all the results are the most important part here, and maybe have a full description of the protocols online.

    It is also worth mentioning that many researchers do try to replicate other studies conceptually, which is more likely to be published as a novel result, though the value of it as a true replication (as oppose to a new study) is questionable.

    • rkurzban

      Thanks for your comment. One point that Pashler made was the
      difference between conceptual replications – which get published if ‘successful’
      and put in a file drawer if not – and genuine replications, which are unlikely
      to get published either way. This is, of course, part of the problem…

  • Pingback: What We Have Here is a Failure to Replicate | E...

  • http://www.facebook.com/profile.php?id=705091948 Ian Weiss

    Here’s a bold, perhaps terribly naive, but hopefully provocative perspective:

    If the public is funding research through government grants, then researchers and editors should have no right to make decisions about what to do with the data based on their own career or business interests. They certainly should not be able to hide data just because it makes them look bad, or because it’s boring. Taxpayers are not paying for the opportunity to help certain scholars gain prestige. They are paying for the discovery of the correct answers to important questions. So what we really need is a much more transparent system that treats data like it belongs to the people who paid for its collection: Everybody. If every lab’s raw data is put out in the open – say, in a massive online archive – we can see who’s contributing to a compelling, impressive body of research (that includes replications), and who’s p-hacking their way to plausible-looking false positives. I know that probably sounds crazy, but how else are we supposed to align the way scholars advance their careers with the way science advances our understanding of the world?

  • http://psychologydegreesx.com/ psychology degrees online

    Fantastic post

  • http://psychologydegreesx.com/ psychology degrees online

    I like this blog

Copyright 2013 Robert Kurzban, all rights reserved.

Opinions expressed in this blog do not reflect the opinions of the editorial staff of the journal.