Estimating the reproducibility of psychological science

Alexander A. Aarts; Joanna E. Anderson; Christopher J. Anderson; Peter R. Attridge; Angela Attwood; Jordan Axt; Molly Babel; Štěpán Bahník; Erica Baranski; Michael Barnett-Cowan; Elizabeth Bartmess; Jennifer Beer; Raoul Bell; Heather Bentley; Leah Beyan; Grace Binion; Denny Borsboom; Annick Bosch; Frank A. Bosco; Sara D. Bowman; Mark J. Brandt; Erin Braswell; Hilmar Brohmer; Benjamin T. Brown; Kristina Brown; Jovita Brüning; Ann Calhoun-Sauls; Shannon P. Callahan; Elizabeth Chagnon; Jesse Chandler; Christopher R. Chartier; Felix Cheung; Cody D. Christopherson; Linda Cillessen; Russ Clay; Hayley Cleary; Mark D. Cloud; Michael Conn; Johanna Cohoon; Simon Columbus; Andreas Cordes; Giulio Costantini; Leslie D.Cramblet Alvarez; Ed Cremata; Jan Crusius; Jamie DeCoster; Michelle A. DeGaetano; Nicolás Delia Penna; Bobby Den Bezemer; Marie K. Deserno; Olivia Devitt; Laura Dewitte; David G. Dobolyi; Geneva T. Dodson; M. Brent Donnellan; Ryan Donohue; Rebecca A. Dore; Angela Dorrough; Anna Dreber; Michelle Dugas; Elizabeth W. Dunn; Kayleigh Easey; Sylvia Eboigbe; Casey Eggleston; Jo Embley; Sacha Epskamp; Timothy M. Errington; Vivien Estel; Frank J. Farach; Jenelle Feather; Anna Fedor; Belén Fernández-Castilla; Susann Fiedler; James G. Field; Stanka A. Fitneva; Taru Flagan; Amanda L. Forest; Eskil Forsell; Joshua D. Foster; Michael C. Frank; Rebecca S. Frazier; Heather Fuchs; Philip Gable; Jeff Galak; Elisa Maria Galliani; Anup Gampa; Sara Garcia; Douglas Gazarian; Elizabeth Gilbert; Roger Giner-Sorolla; Andreas Glöckner; Lars Goellner; Jin X. Goh; Rebecca Goldberg; Patrick T. Goodbourn; Shauna Gordon-McKeon; Bryan Gorges; Jessie Gorges; Justin Goss; Jesse Graham; James A. Grange; Jeremy Gray; Chris Hartgerink; Joshua Hartshorne; Fred Hasselman; Timothy Hayes; Emma Heikensten; Felix Henninger; John Hodsoll; Taylor Holubar; Gea Hoogendoorn; Denise J. Humphries; Cathy O.Y. Hung; Nathali Immelman; Vanessa C. Irsik; Georg Jahn; Frank Jäkel; Marc Jekel; Magnus Johannesson; Larissa G. Johnson; David J. Johnson; Kate M. Johnson; William J. Johnston; Kai Jonas; Jennifer A. Joy-Gaba; Heather Barry Kappes; Kim Kelso; Mallory C. Kidwell; Seung Kyung Kim; Matthew Kirkhart; Bennett Kleinberg; Goran Knežević; Franziska Maria Kolorz; Jolanda J. Kossakowski; Robert Wilhelm Krause; Job Krijnen; Tim Kuhlmann; Yoram K. Kunkels; Megan M. Kyc; Calvin K. Lai; Aamir Laique; Daniël Lakens; Kristin A. Lane; Bethany Lassetter; Ljiljana B. Lazarević; Etienne P. Le Bel; Key Jung Lee; Minha Lee; Kristi Lemm; Carmel A. Levitan; Melissa Lewis; Lin Lin; Stephanie Lin; Matthias Lippold; Darren Loureiro; Ilse Luteijn; Sean MacKinnon; Heather N. Mainard; Denise C. Marigold; Daniel P. Martin; Tylar Martinez; E. J. Masicampo; Josh Matacotta; Maya Mathur; Michael May; Nicole Mechin; Pranjal Mehta; Johannes Meixner; Alissa Melinger; Jeremy K. Miller; Mallorie Miller; Katherine Moore; Marcus Möschl; Matt Motyl; Stephanie M. Müller; Marcus Munafo; Koen I. Neijenhuijs; Taylor Nervi; Gandalf Nicolas; Gustav Nilsonne; Brian A. Nosek; Michèle B. Nuijten; Catherine Olsson; Colleen Osborne; Lutz Ostkamp; Misha Pavel; Ian S. Penton-Voak; Olivia Perna; Cyril Pernet; Marco Perugini; R. Nathan Pipitone; Michael Pitts; Franziska Plessow; Jason M. Prenoveau; Rima Maria Rahal; Kate A. Ratliff; David Reinhard; Frank Renkewitz; Ashley A. Ricker; Anastasia Rigney; Andrew M. Rivers; Mark Roebke; Abraham M. Rutchick; Robert S. Ryan; Onur Sahin; Anondah Saide; Gillian M. Sandstrom; David Santos; Rebecca Saxe; René Schlegelmilch; Kathleen Schmidt; Sabine Scholz; Larissa Seibel; Dylan Faulkner Selterman; Samuel Shaki; William B. Simpson; H. Colleen Sinclair; Jeanine L.M. Skorinko; Agnieszka Slowik; Joel S. Snyder; Courtney Soderberg; Carina Sonnleitner; Nick Spencer; Jeffrey R. Spies; Sara Steegen; Stefan Stieger; Nina Strohminger; Gavin B. Sullivan; Thomas Talhelm; Megan Tapia; Anniek Te Dorsthorst; Manuela Thomae; Sarah L. Thomas; Pia Tio; Frits Traets; Steve Tsang; Francis Tuerlinckx; Paul Turchan; Milan Valášek; Anna E. Van't Veer; Robbie Van Aert; Marcel Van Assen; Riet Van Bork; Mathijs Van De Ven; Don Van Den Bergh; Marije Van Der Hulst; Roel Van Dooren; Johnny Van Doorn; Daan R. Van Renswoude; Hedderik Van Rijn; Wolf Vanpaemel; Alejandro Vásquez Echeverría; Melissa Vazquez; Natalia Velez; Marieke Vermue; Mark Verschoor; Michelangelo Vianello; Martin Voracek; Gina Vuu; Eric Jan Wagenmakers; Joanneke Weerdmeester; Ashlee Welsh; Erin C. Westgate; Joeri Wissink; Michael Wood; Andy Woods; Emily Wright; Sining Wu; Marcel Zeelenberg; Kellylynn Zuni

doi:10.1126/science.aac4716

Estimating the reproducibility of psychological science

Alexander A. Aarts, Joanna E. Anderson, Christopher J. Anderson, Peter R. Attridge, Angela Attwood, Jordan Axt, Molly Babel, Štěpán Bahník, Erica Baranski, Michael Barnett-Cowan, Elizabeth Bartmess, Jennifer Beer, Raoul Bell, Heather Bentley, Leah Beyan, Grace Binion, Denny Borsboom, Annick Bosch, Frank A. Bosco, Sara D. BowmanMark J. Brandt, Erin Braswell, Hilmar Brohmer, Benjamin T. Brown, Kristina Brown, Jovita Brüning, Ann Calhoun-Sauls, Shannon P. Callahan, Elizabeth Chagnon, Jesse Chandler, Christopher R. Chartier, Felix Cheung, Cody D. Christopherson, Linda Cillessen, Russ Clay, Hayley Cleary, Mark D. Cloud, Michael Conn, Johanna Cohoon, Simon Columbus, Andreas Cordes, Giulio Costantini, Leslie D.Cramblet Alvarez, Ed Cremata, Jan Crusius, Jamie DeCoster, Michelle A. DeGaetano, Nicolás Delia Penna, Bobby Den Bezemer, Marie K. Deserno, Olivia Devitt, Laura Dewitte, David G. Dobolyi, Geneva T. Dodson, M. Brent Donnellan, Ryan Donohue, Rebecca A. Dore, Angela Dorrough, Anna Dreber, Michelle Dugas, Elizabeth W. Dunn, Kayleigh Easey, Sylvia Eboigbe, Casey Eggleston, Jo Embley, Sacha Epskamp, Timothy M. Errington, Vivien Estel, Frank J. Farach, Jenelle Feather, Anna Fedor, Belén Fernández-Castilla, Susann Fiedler, James G. Field, Stanka A. Fitneva, Taru Flagan, Amanda L. Forest, Eskil Forsell, Joshua D. Foster, Michael C. Frank, Rebecca S. Frazier, Heather Fuchs, Philip Gable, Jeff Galak, Elisa Maria Galliani, Anup Gampa, Sara Garcia, Douglas Gazarian, Elizabeth Gilbert, Roger Giner-Sorolla, Andreas Glöckner, Lars Goellner, Jin X. Goh, Rebecca Goldberg, Patrick T. Goodbourn, Shauna Gordon-McKeon, Bryan Gorges, Jessie Gorges, Justin Goss, Jesse Graham, James A. Grange, Jeremy Gray, Chris Hartgerink, Joshua Hartshorne, Fred Hasselman, Timothy Hayes, Emma Heikensten, Felix Henninger, John Hodsoll, Taylor Holubar, Gea Hoogendoorn, Denise J. Humphries, Cathy O.Y. Hung, Nathali Immelman, Vanessa C. Irsik, Georg Jahn, Frank Jäkel, Marc Jekel, Magnus Johannesson, Larissa G. Johnson, David J. Johnson, Kate M. Johnson, William J. Johnston, Kai Jonas, Jennifer A. Joy-Gaba, Heather Barry Kappes, Kim Kelso, Mallory C. Kidwell, Seung Kyung Kim, Matthew Kirkhart, Bennett Kleinberg, Goran Knežević, Franziska Maria Kolorz, Jolanda J. Kossakowski, Robert Wilhelm Krause, Job Krijnen, Tim Kuhlmann, Yoram K. Kunkels, Megan M. Kyc, Calvin K. Lai, Aamir Laique, Daniël Lakens, Kristin A. Lane, Bethany Lassetter, Ljiljana B. Lazarević, Etienne P. Le Bel, Key Jung Lee, Minha Lee, Kristi Lemm, Carmel A. Levitan, Melissa Lewis, Lin Lin, Stephanie Lin, Matthias Lippold, Darren Loureiro, Ilse Luteijn, Sean MacKinnon, Heather N. Mainard, Denise C. Marigold, Daniel P. Martin, Tylar Martinez, E. J. Masicampo, Josh Matacotta, Maya Mathur, Michael May, Nicole Mechin, Pranjal Mehta, Johannes Meixner, Alissa Melinger, Jeremy K. Miller, Mallorie Miller, Katherine Moore, Marcus Möschl, Matt Motyl, Stephanie M. Müller, Marcus Munafo, Koen I. Neijenhuijs, Taylor Nervi, Gandalf Nicolas, Gustav Nilsonne, Brian A. Nosek, Michèle B. Nuijten, Catherine Olsson, Colleen Osborne, Lutz Ostkamp, Misha Pavel, Ian S. Penton-Voak, Olivia Perna, Cyril Pernet, Marco Perugini, R. Nathan Pipitone, Michael Pitts, Franziska Plessow, Jason M. Prenoveau, Rima Maria Rahal, Kate A. Ratliff, David Reinhard, Frank Renkewitz, Ashley A. Ricker, Anastasia Rigney, Andrew M. Rivers, Mark Roebke, Abraham M. Rutchick, Robert S. Ryan, Onur Sahin, Anondah Saide, Gillian M. Sandstrom, David Santos, Rebecca Saxe, René Schlegelmilch, Kathleen Schmidt, Sabine Scholz, Larissa Seibel, Dylan Faulkner Selterman, Samuel Shaki, William B. Simpson, H. Colleen Sinclair, Jeanine L.M. Skorinko, Agnieszka Slowik, Joel S. Snyder, Courtney Soderberg, Carina Sonnleitner, Nick Spencer, Jeffrey R. Spies, Sara Steegen, Stefan Stieger, Nina Strohminger, Gavin B. Sullivan, Thomas Talhelm, Megan Tapia, Anniek Te Dorsthorst, Manuela Thomae, Sarah L. Thomas, Pia Tio, Frits Traets, Steve Tsang, Francis Tuerlinckx, Paul Turchan, Milan Valášek, Anna E. Van't Veer, Robbie Van Aert, Marcel Van Assen, Riet Van Bork, Mathijs Van De Ven, Don Van Den Bergh, Marije Van Der Hulst, Roel Van Dooren, Johnny Van Doorn, Daan R. Van Renswoude, Hedderik Van Rijn, Wolf Vanpaemel, Alejandro Vásquez Echeverría, Melissa Vazquez, Natalia Velez, Marieke Vermue, Mark Verschoor, Michelangelo Vianello, Martin Voracek, Gina Vuu, Eric Jan Wagenmakers, Joanneke Weerdmeester, Ashlee Welsh, Erin C. Westgate, Joeri Wissink, Michael Wood, Andy Woods, Emily Wright, Sining Wu, Marcel Zeelenberg, Kellylynn Zuni

Research output: Contribution to journal › Article › peer-review

5512 Scopus citations

Abstract

Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.

Original language	English
Article number	aac4716
Journal	Science
Volume	349
Issue number	6251
DOIs	https://doi.org/10.1126/science.aac4716
State	Published - Aug 28 2015

Bibliographical note

Funding Information:
A.H. is supported by the Earth and Natural Sciences Doctoral Studies Programme, which is funded by the Higher Education Authority (HEA) through the Programme for Research at Third Level Institutions, Cycle 5 (PRTLI-5) and is cofunded by the European Regional Development Fund (ERDF)". M.K. is supported by the Enterprise Ireland Competence Centre for Biorefining & Biofuels (CC/2009/1305A) and INTERREG IVB NEW REsource Innovation Network for European Waste Project No. 317J-RENEW. S.D. is supported by Science Foundation Ireland, under Grant No. 799 06/CP/E007. Experimental campaign was financially supported by the EU project Biomass Research Infrastructure for Sharing Knowledge (BRISK).

ASJC Scopus subject areas

General

Access to Document

10.1126/science.aac4716

Cite this

Aarts, A. A., Anderson, J. E., Anderson, C. J., Attridge, P. R., Attwood, A., Axt, J., Babel, M., Bahník, Š., Baranski, E., Barnett-Cowan, M., Bartmess, E., Beer, J., Bell, R., Bentley, H., Beyan, L., Binion, G., Borsboom, D., Bosch, A., Bosco, F. A., ... Zuni, K. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716

@article{5ddd0408f415431b8ad9809c47f90cf0,

title = "Estimating the reproducibility of psychological science",

abstract = "Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.",

author = "Aarts, {Alexander A.} and Anderson, {Joanna E.} and Anderson, {Christopher J.} and Attridge, {Peter R.} and Angela Attwood and Jordan Axt and Molly Babel and {\v S}t{\v e}p{\'a}n Bahn{\'i}k and Erica Baranski and Michael Barnett-Cowan and Elizabeth Bartmess and Jennifer Beer and Raoul Bell and Heather Bentley and Leah Beyan and Grace Binion and Denny Borsboom and Annick Bosch and Bosco, {Frank A.} and Bowman, {Sara D.} and Brandt, {Mark J.} and Erin Braswell and Hilmar Brohmer and Brown, {Benjamin T.} and Kristina Brown and Jovita Br{\"u}ning and Ann Calhoun-Sauls and Callahan, {Shannon P.} and Elizabeth Chagnon and Jesse Chandler and Chartier, {Christopher R.} and Felix Cheung and Christopherson, {Cody D.} and Linda Cillessen and Russ Clay and Hayley Cleary and Cloud, {Mark D.} and Michael Conn and Johanna Cohoon and Simon Columbus and Andreas Cordes and Giulio Costantini and Alvarez, {Leslie D.Cramblet} and Ed Cremata and Jan Crusius and Jamie DeCoster and DeGaetano, {Michelle A.} and Penna, {Nicol{\'a}s Delia} and {Den Bezemer}, Bobby and Deserno, {Marie K.} and Olivia Devitt and Laura Dewitte and Dobolyi, {David G.} and Dodson, {Geneva T.} and Donnellan, {M. Brent} and Ryan Donohue and Dore, {Rebecca A.} and Angela Dorrough and Anna Dreber and Michelle Dugas and Dunn, {Elizabeth W.} and Kayleigh Easey and Sylvia Eboigbe and Casey Eggleston and Jo Embley and Sacha Epskamp and Errington, {Timothy M.} and Vivien Estel and Farach, {Frank J.} and Jenelle Feather and Anna Fedor and Bel{\'e}n Fern{\'a}ndez-Castilla and Susann Fiedler and Field, {James G.} and Fitneva, {Stanka A.} and Taru Flagan and Forest, {Amanda L.} and Eskil Forsell and Foster, {Joshua D.} and Frank, {Michael C.} and Frazier, {Rebecca S.} and Heather Fuchs and Philip Gable and Jeff Galak and Galliani, {Elisa Maria} and Anup Gampa and Sara Garcia and Douglas Gazarian and Elizabeth Gilbert and Roger Giner-Sorolla and Andreas Gl{\"o}ckner and Lars Goellner and Goh, {Jin X.} and Rebecca Goldberg and Goodbourn, {Patrick T.} and Shauna Gordon-McKeon and Bryan Gorges and Jessie Gorges and Justin Goss and Jesse Graham and Grange, {James A.} and Jeremy Gray and Chris Hartgerink and Joshua Hartshorne and Fred Hasselman and Timothy Hayes and Emma Heikensten and Felix Henninger and John Hodsoll and Taylor Holubar and Gea Hoogendoorn and Humphries, {Denise J.} and Hung, {Cathy O.Y.} and Nathali Immelman and Irsik, {Vanessa C.} and Georg Jahn and Frank J{\"a}kel and Marc Jekel and Magnus Johannesson and Johnson, {Larissa G.} and Johnson, {David J.} and Johnson, {Kate M.} and Johnston, {William J.} and Kai Jonas and Joy-Gaba, {Jennifer A.} and Kappes, {Heather Barry} and Kim Kelso and Kidwell, {Mallory C.} and Kim, {Seung Kyung} and Matthew Kirkhart and Bennett Kleinberg and Goran Kne{\v z}evi{\'c} and Kolorz, {Franziska Maria} and Kossakowski, {Jolanda J.} and Krause, {Robert Wilhelm} and Job Krijnen and Tim Kuhlmann and Kunkels, {Yoram K.} and Kyc, {Megan M.} and Lai, {Calvin K.} and Aamir Laique and Dani{\"e}l Lakens and Lane, {Kristin A.} and Bethany Lassetter and Lazarevi{\'c}, {Ljiljana B.} and {Le Bel}, {Etienne P.} and Lee, {Key Jung} and Minha Lee and Kristi Lemm and Levitan, {Carmel A.} and Melissa Lewis and Lin Lin and Stephanie Lin and Matthias Lippold and Darren Loureiro and Ilse Luteijn and Sean MacKinnon and Mainard, {Heather N.} and Marigold, {Denise C.} and Martin, {Daniel P.} and Tylar Martinez and Masicampo, {E. J.} and Josh Matacotta and Maya Mathur and Michael May and Nicole Mechin and Pranjal Mehta and Johannes Meixner and Alissa Melinger and Miller, {Jeremy K.} and Mallorie Miller and Katherine Moore and Marcus M{\"o}schl and Matt Motyl and M{\"u}ller, {Stephanie M.} and Marcus Munafo and Neijenhuijs, {Koen I.} and Taylor Nervi and Gandalf Nicolas and Gustav Nilsonne and Nosek, {Brian A.} and Nuijten, {Mich{\`e}le B.} and Catherine Olsson and Colleen Osborne and Lutz Ostkamp and Misha Pavel and Penton-Voak, {Ian S.} and Olivia Perna and Cyril Pernet and Marco Perugini and Pipitone, {R. Nathan} and Michael Pitts and Franziska Plessow and Prenoveau, {Jason M.} and Rahal, {Rima Maria} and Ratliff, {Kate A.} and David Reinhard and Frank Renkewitz and Ricker, {Ashley A.} and Anastasia Rigney and Rivers, {Andrew M.} and Mark Roebke and Rutchick, {Abraham M.} and Ryan, {Robert S.} and Onur Sahin and Anondah Saide and Sandstrom, {Gillian M.} and David Santos and Rebecca Saxe and Ren{\'e} Schlegelmilch and Kathleen Schmidt and Sabine Scholz and Larissa Seibel and Selterman, {Dylan Faulkner} and Samuel Shaki and Simpson, {William B.} and Sinclair, {H. Colleen} and Skorinko, {Jeanine L.M.} and Agnieszka Slowik and Snyder, {Joel S.} and Courtney Soderberg and Carina Sonnleitner and Nick Spencer and Spies, {Jeffrey R.} and Sara Steegen and Stefan Stieger and Nina Strohminger and Sullivan, {Gavin B.} and Thomas Talhelm and Megan Tapia and {Te Dorsthorst}, Anniek and Manuela Thomae and Thomas, {Sarah L.} and Pia Tio and Frits Traets and Steve Tsang and Francis Tuerlinckx and Paul Turchan and Milan Val{\'a}{\v s}ek and {Van't Veer}, {Anna E.} and {Van Aert}, Robbie and {Van Assen}, Marcel and {Van Bork}, Riet and {Van De Ven}, Mathijs and {Van Den Bergh}, Don and {Van Der Hulst}, Marije and {Van Dooren}, Roel and {Van Doorn}, Johnny and {Van Renswoude}, {Daan R.} and {Van Rijn}, Hedderik and Wolf Vanpaemel and Echeverr{\'i}a, {Alejandro V{\'a}squez} and Melissa Vazquez and Natalia Velez and Marieke Vermue and Mark Verschoor and Michelangelo Vianello and Martin Voracek and Gina Vuu and Wagenmakers, {Eric Jan} and Joanneke Weerdmeester and Ashlee Welsh and Westgate, {Erin C.} and Joeri Wissink and Michael Wood and Andy Woods and Emily Wright and Sining Wu and Marcel Zeelenberg and Kellylynn Zuni",

note = "Funding Information: A.H. is supported by the Earth and Natural Sciences Doctoral Studies Programme, which is funded by the Higher Education Authority (HEA) through the Programme for Research at Third Level Institutions, Cycle 5 (PRTLI-5) and is cofunded by the European Regional Development Fund (ERDF){"}. M.K. is supported by the Enterprise Ireland Competence Centre for Biorefining & Biofuels (CC/2009/1305A) and INTERREG IVB NEW REsource Innovation Network for European Waste Project No. 317J-RENEW. S.D. is supported by Science Foundation Ireland, under Grant No. 799 06/CP/E007. Experimental campaign was financially supported by the EU project Biomass Research Infrastructure for Sharing Knowledge (BRISK).",

year = "2015",

month = aug,

day = "28",

doi = "10.1126/science.aac4716",

language = "English",

volume = "349",

number = "6251",

}

Aarts, AA, Anderson, JE, Anderson, CJ, Attridge, PR, Attwood, A, Axt, J, Babel, M, Bahník, Š, Baranski, E, Barnett-Cowan, M, Bartmess, E, Beer, J, Bell, R, Bentley, H, Beyan, L, Binion, G, Borsboom, D, Bosch, A, Bosco, FA, Bowman, SD, Brandt, MJ, Braswell, E, Brohmer, H, Brown, BT, Brown, K, Brüning, J, Calhoun-Sauls, A, Callahan, SP, Chagnon, E, Chandler, J, Chartier, CR, Cheung, F, Christopherson, CD, Cillessen, L, Clay, R, Cleary, H, Cloud, MD, Conn, M, Cohoon, J, Columbus, S, Cordes, A, Costantini, G, Alvarez, LDC, Cremata, E, Crusius, J, DeCoster, J, DeGaetano, MA, Penna, ND, Den Bezemer, B, Deserno, MK, Devitt, O, Dewitte, L, Dobolyi, DG, Dodson, GT, Donnellan, MB, Donohue, R, Dore, RA, Dorrough, A, Dreber, A, Dugas, M, Dunn, EW, Easey, K, Eboigbe, S, Eggleston, C, Embley, J, Epskamp, S, Errington, TM, Estel, V, Farach, FJ, Feather, J, Fedor, A, Fernández-Castilla, B, Fiedler, S, Field, JG, Fitneva, SA, Flagan, T, Forest, AL, Forsell, E, Foster, JD, Frank, MC, Frazier, RS, Fuchs, H, Gable, P, Galak, J, Galliani, EM, Gampa, A, Garcia, S, Gazarian, D, Gilbert, E, Giner-Sorolla, R, Glöckner, A, Goellner, L, Goh, JX, Goldberg, R, Goodbourn, PT, Gordon-McKeon, S, Gorges, B, Gorges, J, Goss, J, Graham, J, Grange, JA, Gray, J, Hartgerink, C, Hartshorne, J, Hasselman, F, Hayes, T, Heikensten, E, Henninger, F, Hodsoll, J, Holubar, T, Hoogendoorn, G, Humphries, DJ, Hung, COY, Immelman, N, Irsik, VC, Jahn, G, Jäkel, F, Jekel, M, Johannesson, M, Johnson, LG, Johnson, DJ, Johnson, KM, Johnston, WJ, Jonas, K, Joy-Gaba, JA, Kappes, HB, Kelso, K, Kidwell, MC, Kim, SK, Kirkhart, M, Kleinberg, B, Knežević, G, Kolorz, FM, Kossakowski, JJ, Krause, RW, Krijnen, J, Kuhlmann, T, Kunkels, YK, Kyc, MM, Lai, CK, Laique, A, Lakens, D, Lane, KA, Lassetter, B, Lazarević, LB, Le Bel, EP, Lee, KJ, Lee, M, Lemm, K, Levitan, CA, Lewis, M, Lin, L, Lin, S, Lippold, M, Loureiro, D, Luteijn, I, MacKinnon, S, Mainard, HN, Marigold, DC, Martin, DP, Martinez, T, Masicampo, EJ, Matacotta, J, Mathur, M, May, M, Mechin, N, Mehta, P, Meixner, J, Melinger, A, Miller, JK, Miller, M, Moore, K, Möschl, M, Motyl, M, Müller, SM, Munafo, M, Neijenhuijs, KI, Nervi, T, Nicolas, G, Nilsonne, G, Nosek, BA, Nuijten, MB, Olsson, C, Osborne, C, Ostkamp, L, Pavel, M, Penton-Voak, IS, Perna, O, Pernet, C, Perugini, M, Pipitone, RN, Pitts, M, Plessow, F, Prenoveau, JM, Rahal, RM, Ratliff, KA, Reinhard, D, Renkewitz, F, Ricker, AA, Rigney, A, Rivers, AM, Roebke, M, Rutchick, AM, Ryan, RS, Sahin, O, Saide, A, Sandstrom, GM, Santos, D, Saxe, R, Schlegelmilch, R, Schmidt, K, Scholz, S, Seibel, L, Selterman, DF, Shaki, S, Simpson, WB, Sinclair, HC, Skorinko, JLM, Slowik, A, Snyder, JS, Soderberg, C, Sonnleitner, C, Spencer, N, Spies, JR, Steegen, S, Stieger, S, Strohminger, N, Sullivan, GB, Talhelm, T, Tapia, M, Te Dorsthorst, A, Thomae, M, Thomas, SL, Tio, P, Traets, F, Tsang, S, Tuerlinckx, F, Turchan, P, Valášek, M, Van't Veer, AE, Van Aert, R, Van Assen, M, Van Bork, R, Van De Ven, M, Van Den Bergh, D, Van Der Hulst, M, Van Dooren, R, Van Doorn, J, Van Renswoude, DR, Van Rijn, H, Vanpaemel, W, Echeverría, AV, Vazquez, M, Velez, N, Vermue, M, Verschoor, M, Vianello, M, Voracek, M, Vuu, G, Wagenmakers, EJ, Weerdmeester, J, Welsh, A, Westgate, EC, Wissink, J, Wood, M, Woods, A, Wright, E, Wu, S, Zeelenberg, M & Zuni, K 2015, 'Estimating the reproducibility of psychological science', Science, vol. 349, no. 6251, aac4716. https://doi.org/10.1126/science.aac4716

TY - JOUR

T1 - Estimating the reproducibility of psychological science

AU - Aarts, Alexander A.

AU - Anderson, Joanna E.

AU - Anderson, Christopher J.

AU - Attridge, Peter R.

AU - Attwood, Angela

AU - Axt, Jordan

AU - Babel, Molly

AU - Bahník, Štěpán

AU - Baranski, Erica

AU - Barnett-Cowan, Michael

AU - Bartmess, Elizabeth

AU - Beer, Jennifer

AU - Bell, Raoul

AU - Bentley, Heather

AU - Beyan, Leah

AU - Binion, Grace

AU - Borsboom, Denny

AU - Bosch, Annick

AU - Bosco, Frank A.

AU - Bowman, Sara D.

AU - Brandt, Mark J.

AU - Braswell, Erin

AU - Brohmer, Hilmar

AU - Brown, Benjamin T.

AU - Brown, Kristina

AU - Brüning, Jovita

AU - Calhoun-Sauls, Ann

AU - Callahan, Shannon P.

AU - Chagnon, Elizabeth

AU - Chandler, Jesse

AU - Chartier, Christopher R.

AU - Cheung, Felix

AU - Christopherson, Cody D.

AU - Cillessen, Linda

AU - Clay, Russ

AU - Cleary, Hayley

AU - Cloud, Mark D.

AU - Conn, Michael

AU - Cohoon, Johanna

AU - Columbus, Simon

AU - Cordes, Andreas

AU - Costantini, Giulio

AU - Alvarez, Leslie D.Cramblet

AU - Cremata, Ed

AU - Crusius, Jan

AU - DeCoster, Jamie

AU - DeGaetano, Michelle A.

AU - Penna, Nicolás Delia

AU - Den Bezemer, Bobby

AU - Deserno, Marie K.

AU - Devitt, Olivia

AU - Dewitte, Laura

AU - Dobolyi, David G.

AU - Dodson, Geneva T.

AU - Donnellan, M. Brent

AU - Donohue, Ryan

AU - Dore, Rebecca A.

AU - Dorrough, Angela

AU - Dreber, Anna

AU - Dugas, Michelle

AU - Dunn, Elizabeth W.

AU - Easey, Kayleigh

AU - Eboigbe, Sylvia

AU - Eggleston, Casey

AU - Embley, Jo

AU - Epskamp, Sacha

AU - Errington, Timothy M.

AU - Estel, Vivien

AU - Farach, Frank J.

AU - Feather, Jenelle

AU - Fedor, Anna

AU - Fernández-Castilla, Belén

AU - Fiedler, Susann

AU - Field, James G.

AU - Fitneva, Stanka A.

AU - Flagan, Taru

AU - Forest, Amanda L.

AU - Forsell, Eskil

AU - Foster, Joshua D.

AU - Frank, Michael C.

AU - Frazier, Rebecca S.

AU - Fuchs, Heather

AU - Gable, Philip

AU - Galak, Jeff

AU - Galliani, Elisa Maria

AU - Gampa, Anup

AU - Garcia, Sara

AU - Gazarian, Douglas

AU - Gilbert, Elizabeth

AU - Giner-Sorolla, Roger

AU - Glöckner, Andreas

AU - Goellner, Lars

AU - Goh, Jin X.

AU - Goldberg, Rebecca

AU - Goodbourn, Patrick T.

AU - Gordon-McKeon, Shauna

AU - Gorges, Bryan

AU - Gorges, Jessie

AU - Goss, Justin

AU - Graham, Jesse

AU - Grange, James A.

AU - Gray, Jeremy

AU - Hartgerink, Chris

AU - Hartshorne, Joshua

AU - Hasselman, Fred

AU - Hayes, Timothy

AU - Heikensten, Emma

AU - Henninger, Felix

AU - Hodsoll, John

AU - Holubar, Taylor

AU - Hoogendoorn, Gea

AU - Humphries, Denise J.

AU - Hung, Cathy O.Y.

AU - Immelman, Nathali

AU - Irsik, Vanessa C.

AU - Jahn, Georg

AU - Jäkel, Frank

AU - Jekel, Marc

AU - Johannesson, Magnus

AU - Johnson, Larissa G.

AU - Johnson, David J.

AU - Johnson, Kate M.

AU - Johnston, William J.

AU - Jonas, Kai

AU - Joy-Gaba, Jennifer A.

AU - Kappes, Heather Barry

AU - Kelso, Kim

AU - Kidwell, Mallory C.

AU - Kim, Seung Kyung

AU - Kirkhart, Matthew

AU - Kleinberg, Bennett

AU - Knežević, Goran

AU - Kolorz, Franziska Maria

AU - Kossakowski, Jolanda J.

AU - Krause, Robert Wilhelm

AU - Krijnen, Job

AU - Kuhlmann, Tim

AU - Kunkels, Yoram K.

AU - Kyc, Megan M.

AU - Lai, Calvin K.

AU - Laique, Aamir

AU - Lakens, Daniël

AU - Lane, Kristin A.

AU - Lassetter, Bethany

AU - Lazarević, Ljiljana B.

AU - Le Bel, Etienne P.

AU - Lee, Key Jung

AU - Lee, Minha

AU - Lemm, Kristi

AU - Levitan, Carmel A.

AU - Lewis, Melissa

AU - Lin, Lin

AU - Lin, Stephanie

AU - Lippold, Matthias

AU - Loureiro, Darren

AU - Luteijn, Ilse

AU - MacKinnon, Sean

AU - Mainard, Heather N.

AU - Marigold, Denise C.

AU - Martin, Daniel P.

AU - Martinez, Tylar

AU - Masicampo, E. J.

AU - Matacotta, Josh

AU - Mathur, Maya

AU - May, Michael

AU - Mechin, Nicole

AU - Mehta, Pranjal

AU - Meixner, Johannes

AU - Melinger, Alissa

AU - Miller, Jeremy K.

AU - Miller, Mallorie

AU - Moore, Katherine

AU - Möschl, Marcus

AU - Motyl, Matt

AU - Müller, Stephanie M.

AU - Munafo, Marcus

AU - Neijenhuijs, Koen I.

AU - Nervi, Taylor

AU - Nicolas, Gandalf

AU - Nilsonne, Gustav

AU - Nosek, Brian A.

AU - Nuijten, Michèle B.

AU - Olsson, Catherine

AU - Osborne, Colleen

AU - Ostkamp, Lutz

AU - Pavel, Misha

AU - Penton-Voak, Ian S.

AU - Perna, Olivia

AU - Pernet, Cyril

AU - Perugini, Marco

AU - Pipitone, R. Nathan

AU - Pitts, Michael

AU - Plessow, Franziska

AU - Prenoveau, Jason M.

AU - Rahal, Rima Maria

AU - Ratliff, Kate A.

AU - Reinhard, David

AU - Renkewitz, Frank

AU - Ricker, Ashley A.

AU - Rigney, Anastasia

AU - Rivers, Andrew M.

AU - Roebke, Mark

AU - Rutchick, Abraham M.

AU - Ryan, Robert S.

AU - Sahin, Onur

AU - Saide, Anondah

AU - Sandstrom, Gillian M.

AU - Santos, David

AU - Saxe, Rebecca

AU - Schlegelmilch, René

AU - Schmidt, Kathleen

AU - Scholz, Sabine

AU - Seibel, Larissa

AU - Selterman, Dylan Faulkner

AU - Shaki, Samuel

AU - Simpson, William B.

AU - Sinclair, H. Colleen

AU - Skorinko, Jeanine L.M.

AU - Slowik, Agnieszka

AU - Snyder, Joel S.

AU - Soderberg, Courtney

AU - Sonnleitner, Carina

AU - Spencer, Nick

AU - Spies, Jeffrey R.

AU - Steegen, Sara

AU - Stieger, Stefan

AU - Strohminger, Nina

AU - Sullivan, Gavin B.

AU - Talhelm, Thomas

AU - Tapia, Megan

AU - Te Dorsthorst, Anniek

AU - Thomae, Manuela

AU - Thomas, Sarah L.

AU - Tio, Pia

AU - Traets, Frits

AU - Tsang, Steve

AU - Tuerlinckx, Francis

AU - Turchan, Paul

AU - Valášek, Milan

AU - Van't Veer, Anna E.

AU - Van Aert, Robbie

AU - Van Assen, Marcel

AU - Van Bork, Riet

AU - Van De Ven, Mathijs

AU - Van Den Bergh, Don

AU - Van Der Hulst, Marije

AU - Van Dooren, Roel

AU - Van Doorn, Johnny

AU - Van Renswoude, Daan R.

AU - Van Rijn, Hedderik

AU - Vanpaemel, Wolf

AU - Echeverría, Alejandro Vásquez

AU - Vazquez, Melissa

AU - Velez, Natalia

AU - Vermue, Marieke

AU - Verschoor, Mark

AU - Vianello, Michelangelo

AU - Voracek, Martin

AU - Vuu, Gina

AU - Wagenmakers, Eric Jan

AU - Weerdmeester, Joanneke

AU - Welsh, Ashlee

AU - Westgate, Erin C.

AU - Wissink, Joeri

AU - Wood, Michael

AU - Woods, Andy

AU - Wright, Emily

AU - Wu, Sining

AU - Zeelenberg, Marcel

AU - Zuni, Kellylynn

N1 - Funding Information: A.H. is supported by the Earth and Natural Sciences Doctoral Studies Programme, which is funded by the Higher Education Authority (HEA) through the Programme for Research at Third Level Institutions, Cycle 5 (PRTLI-5) and is cofunded by the European Regional Development Fund (ERDF)". M.K. is supported by the Enterprise Ireland Competence Centre for Biorefining & Biofuels (CC/2009/1305A) and INTERREG IVB NEW REsource Innovation Network for European Waste Project No. 317J-RENEW. S.D. is supported by Science Foundation Ireland, under Grant No. 799 06/CP/E007. Experimental campaign was financially supported by the EU project Biomass Research Infrastructure for Sharing Knowledge (BRISK).

PY - 2015/8/28

Y1 - 2015/8/28

N2 - Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.

AB - Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.

UR - http://www.scopus.com/inward/record.url?scp=84940513037&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84940513037&partnerID=8YFLogxK

U2 - 10.1126/science.aac4716

DO - 10.1126/science.aac4716

M3 - Article

C2 - 26315443

AN - SCOPUS:84940513037

SN - 0036-8075

VL - 349

JO - Science

JF - Science

IS - 6251

M1 - aac4716

ER -

Estimating the reproducibility of psychological science

Abstract

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this