16/12/2015

Overhalfofpsychologystudiesfailreproducibilitytest:NatureNews&Comment

NATURE |NEWS

Overhalfofpsychologystudiesfailreproducibilitytest
Largestreplicationstudytodatecastsdoubtonmanypublishedpositiveresults.
MonyaBaker
27August2015
Donttrusteverythingyoureadinthepsychology
literature.Infact,twothirdsofitshouldprobablybe
distrusted.
Inthebiggestprojectofitskind,BrianNosek,asocial
psychologistandheadoftheCenterforOpenSciencein
Charlottesville,Virginia,and269coauthorsrepeated
workreportedin98originalpapersfromthreepsychology
journals,toseeiftheyindependentlycameupwiththe
sameresults.
Thestudiestheytookonrangedfromwhetherexpressing
insecuritiesperpetuatesthemtodifferencesinhow

BrianNosek'steamsetouttoreplicatescoresof
studies.

childrenandadultsrespondtofearstimuli,toeffective
waystoteacharithmetic.
Accordingtothereplicators'qualitativeassessments,aspreviouslyreportedbyNature,only39ofthe100
replicationattemptsweresuccessful.(Therewere100completedreplicationattemptsonthe98papers,as
intwocasesreplicationeffortswereduplicatedbyseparateteams.)Butwhetherareplicationattemptis
consideredsuccessfulisnotstraightforward.TodayinScience,theteamreportthemultipledifferent
measurestheyusedtoanswerthisquestion 1.
The39%figurederivesfromtheteam'ssubjectiveassessmentsofsuccessorfailure(seegraphic,
'Reliabilitytest').Anothermethodassessedwhetherastatisticallysignificanteffectcouldbefound,and
producedanevenbleakerresult.Whereas97%oftheoriginalstudiesfoundasignificanteffect,only36%
ofreplicationstudiesfoundsignificantresults.Theteamalsofoundthattheaveragesizeoftheeffects
foundinthereplicatedstudieswasonlyhalfthatreportedintheoriginalstudies.
Thereisnowayofknowingwhetheranyindividualpaperistrueorfalsefromthiswork,saysNosek.Either
theoriginalorthereplicationworkcouldbeflawed,orcrucialdifferencesbetweenthetwomightbe
http://www.nature.com/news/overhalfofpsychologystudiesfailreproducibilitytest1.18248

1/7

16/12/2015

Overhalfofpsychologystudiesfailreproducibilitytest:NatureNews&Comment

unappreciated.Overall,however,theprojectpointsto
widespreadpublicationofworkthatdoesnotstandupto
scrutiny.
AlthoughNosekisquicktosaythatmostresourcesshould
befunnelledtowardsnewresearch,hesuggeststhata
mere3%ofscientificfundingdevotedtoreplicationcould
makeabigdifference.Thecurrentamount,hesays,is
nearzero.
Replicationfailure
TheworkispartoftheReproducibilityProject,launchedin
2011amidhighprofilereportsoffraudandfaulty
statisticalanalysisthatledtoanidentitycrisisin
psychology.
JohnIoannidis,an
epidemiologistat
StanfordUniversityin
California,saysthatthetruereplicationfailureratecouldexceed80%,
evenhigherthanNosek'sstudysuggests.Thisisbecausethe

Relatedstories
Firstresultsfrom
psychologyslargest
reproducibilitytest

ReproducibilityProjecttargetedworkinhighlyrespectedjournals,the

Replicationstudies:Bad

originalscientistsworkedcloselywiththereplicators,andreplicating

copy

teamsgenerallyoptedforpapersemployingrelativelyeasymethodsall

ReproducibilityProject:

thingsthatshouldhavemadereplicationeasier.

Psychology

But,headds,Wecanreallyuseittoimprovethesituationratherthan
justlamentthesituation.Themerefactthatthatcollaborationhappenedatsuchalargescalesuggeststhat
scientistsarewillingtomoveinthedirectionofimproving.
TheworkpublishedinScienceisdifferentfrompreviouspapersonreplicationbecausetheteamactually
replicatedsuchalargeswatheofexperiments,saysAndrewGelman,astatisticianatColumbiaUniversity
inNewYork.Inthepast,someresearchersdismissedindicationsofwidespreadproblemsbecausethey
involvedsmallreplicationeffortsorwerebasedonstatisticalsimulations.
Buttheywillhaveahardertimeshruggingoffthelateststudy,saysGelman.Thisisempiricalevidence,
notatheoreticalargument.Thevalueofthisprojectisthathopefullypeoplewillbelessconfidentabout
theirclaims.

http://www.nature.com/news/overhalfofpsychologystudiesfailreproducibilitytest1.18248

2/7

16/12/2015

Overhalfofpsychologystudiesfailreproducibilitytest:NatureNews&Comment

Publicationbias
Thepoint,saysNosek,isnottocritiqueindividualpapersbuttogaugejusthowmuchbiasdrives
publicationinpsychology.Forinstance,boringbutaccuratestudiesmaynevergetpublished,or
researchersmayachieveintriguingresultslessbydocumentingtrueeffectsthanbyhittingthestatistical
jackpotfindingasignificantresultbysheerluckortryingvariousanalyticalmethodsuntilsomethingpans
out.
Nosekbelievesthatotherscientificfieldsarelikelytohavemuchincommonwithpsychology.Oneanalysis
foundthatonly6of53highprofilepapersincancerbiologycouldbereproduced2andarelated
reproducibilityprojectincancerbiologyiscurrentlyunderway.Theincentivestofindresultsworthyofhigh
profilepublicationsareverystronginallfields,andcanspurpeopletoloseobjectivity.Ifthisoccursona
broadscale,thenthepublishedliteraturemaybemorebeautifulthanreality,"saysNosek.
Theresultspublishedtodayshouldsparkabroaderdebateaboutoptimalscientificpracticeandpublishing,
saysBetsyLevyPaluck,asocialpsychologistatPrincetonUniversityinNewJersey.Itsayswedon'tknow
thebalancebetweeninnovationandreplication.
Thefactthatthestudywaspublishedinaprestigiousjournalwillencouragefurtherscholarship,shesays,
andshowsthatnowreplicationisbeingpromotedasaresponsibleandinterestinglineofenquiry.
Nature doi:10.1038/nature.2015.18248
Tweet

Follow@NatureNews

References
1. OpenScienceCollaboration.Sciencehttp://dx.doi.org/10.1126/science.aac4716(2015).
Showcontext
2. Begley,C.G.&Ellis,L.M.Nature483,531533(2012)
Showcontext

Article PubMed ChemPort

Relatedstoriesandlinks
Fromnature.com
Firstresultsfrompsychologyslargestreproducibilitytest
30April2015
Replicationstudies:Badcopy
http://www.nature.com/news/overhalfofpsychologystudiesfailreproducibilitytest1.18248

3/7

16/12/2015

Overhalfofpsychologystudiesfailreproducibilitytest:NatureNews&Comment

16May2012
Fromelsewhere
ReproducibilityProject:Psychology

Forthebestcommentingexperience,pleaseloginorregisterasauserandagreetoourCommunity
Guidelines.Youwillberedirectedbacktothispagewhereyouwillseecommentsupdatinginrealtimeand
havetheabilitytorecommendcommentstootherusers.

12comments

Subscribetocomments

MarinPanovic 2015111311:24AM

Dr.Nosekworkdefinitelygoesto61%resultinhisresearchbeforereadinghismethodologyand
hypothesis,nogreatscientist,noaverageornomentallyretardedscientistbelievesthatherorhis
wordisfinal,sotheoreticallyDr.Nosekcanprove100%researchunreliable,orunconservativeor
Dr.Nosekcanprovethatweliveinyear7525ofByzantinecalendar,thatresearchis100%reliable,
nobelieverdoubts,inscienceontheotherhandthequestionisimportant,ifthefirstansweris
incorrectwe'llgetcorrectanswerlater,thereisnocorrectanswertoquestionnotmade,whatishe
tryingtosayisthatheiswhiteanglosaxonprotestantmanandthosewhowanttoprovehim
evolutionarynotsuperiorshouldn'tevendare

jackguy 2015091712:46AM

Thestudyhasproblems.ReadthisresponsebyDr.JennyDavis.
http://thesocietypages.org/cyborgology/2015/09/08/thereproducibilityprojectsfataldesignflaw/

BorisShmagin 2015090309:26PM

Thecoordinatesystemsarethepointtostartandthendiscussreproducibility.Mathematics,
technologyandnaturalscienceshavedifferentcoordinatesystems.Mathematicshasthemost
logicalandreproduciblecases.Theyareabstractandexistasculturalevents.Technologycreates
sophisticatedobjects,reproducibilityofwhichisagoalandthedifferenceintheirproperties(errors)
mightbeverysmall.Thisisnotthecasefornaturalobjectlikehuman.Thistopicwasspecial
consideredfornaturalobjectlikeriverwatershed:

https://www.researchgate.net/publication/268334171_Modeling_the_Nature_System_Analysis_for_Knowle
https://www.researchgate.net/publication/264555209_Hydrology_Modeling_an_Uncertainty

http://www.nature.com/news/overhalfofpsychologystudiesfailreproducibilitytest1.18248

4/7

16/12/2015

Overhalfofpsychologystudiesfailreproducibilitytest:NatureNews&Comment

PeterMetaSkeptic 2015090405:24AM

readingyourcomment,Ican'thelpmyselfthinkingaboutSokal&Bricmont'sbook"Intellectual
Impostures".

BorisShmagin 2015090908:32PM

PeterIputmynamebecausemycommentbasedofmyresults

PeterMetaSkeptic 2015091207:20PM

Puttingitsownnamemeansthatyou'rereadytodefendyourviewandIrespect
that.Howeveritdoesn'tmeanthatIhavetoagreewithyourpointofview.Obvious
statement,isn'tit.wewon'tsettleourargumenthere.That'sthepitfallofcomments.
Iwishyouthebestinyourresearch.Sincerely,Peter.

Thiscommentwasdeleted.

PeterMetaSkeptic 2015090705:41AM

Argumentumadhominem.Whatasurprise!YoucouldhaveaskmewhyIfoundthe
lackofclarityofthecommentabovemisleading,butthatoptionhasnotcrossed
yourmind.Clearexpressionofideas,concepts,theories,solutions,problems,...is
requiredinscienceandmostscientistsaretryingtodojustthat,becausethereisa
linkwithintellectualhonesty.

AnnaNeumann 2015090304:15PM

"ButcontrarytotheimplicationoftheReproducibilityProject,thereisnoreplicationcrisisin
psychology.Thecrisismaysimplybetheresultofamisunderstandingofwhatscienceis."Dr.Lisa
FeldmanBarrettoffersasoundresponsetosaid"crisis"inaNYTimesopedthisweek
http://www.nytimes.com/2015/09/01/opinion/psychologyisnotincrisis.html?_r=1

PeterMetaSkeptic 2015090405:14AM

Itremindsmeofanoldphilosophicaltrick.Whenrealityisn'tonyourside,redefineituntilthe
http://www.nature.com/news/overhalfofpsychologystudiesfailreproducibilitytest1.18248

5/7

16/12/2015

Overhalfofpsychologystudiesfailreproducibilitytest:NatureNews&Comment

newrealityyouinventcancopewithyourtheories.AsRichardFeynmansaidwhatweforgotto
teachexplicitlyinscienceisakindofutterhonesty

RichardPlant 2015090303:20PM

WevebeensayingthisforyearsinrelationtoPsychologyexperimentsadministeredusing
computers.Putsimplyresearchersmaynotbedoingwhattheythinktheyaredoingwhenthey
presentastimulussynchronisewithotherequipment,e.g.fMRI,EEG,eyetrackersandrecord
ReactionTimes.Wedliketoseeresearchersactuallypublishtimingvalidationdatawiththeir
paperstoprovethefigurestheyquoteareaccurate.Themajorityofresearcherssimplydonthave
anyinsightintothisorhowtheirequipmentreallyworksandthatsbeforeyougetontothestatistics!
Sometrainingcouldcertainlyhelphere.Atthemomenttheresalotoffocusonnewtechnologyand
flashyexperimentsorrunninglargenumbersofparticipantsontheweb.Itsalmostasthoughsome
researchershaveforgottenthebasicsoftheScientificMethodandconstructingPsychology
experimentsonacomputerusinganexperimentgeneratoristooeasy?Wedontcarehow
researchersdothis,justthatthattheyshould.Aquicklookatacoupleofourrecentpapersmight
scareinstitutionsandtheresearchersthemselvesintodoingsomething?Wethinkthatfundersand
publishersshouldplayabiggerrole.Atthemomentresearchersinanydisciplinewontcareunless
therearesolidconsequencesintermsofreducedfundingorhigherqualitythresholdsfor
publications.8xCouldmillisecondtimingerrorsincommonlyused
equipmentbeacauseofreplicationfailureinsomeneurosciencestudies?Areminderon
millisecondtimingaccuracyandpotentialreplicationfailureincomputerbasedpsychology
experiments:Anopenletter

DjordjeVilimanovic 2015083108:40PM

Sothereisa61%chancethatthistoocan'tbereplicated:)

phoebemoon 2015082802:40AM

The"Truth"WearsOff.http://www.newyorker.com/magazine/2010/12/13/thetruthwearsoff

SeeotherNews&CommentarticlesfromNature
Nature

ISSN00280836

EISSN14764687

http://www.nature.com/news/overhalfofpsychologystudiesfailreproducibilitytest1.18248

6/7

16/12/2015

Overhalfofpsychologystudiesfailreproducibilitytest:NatureNews&Comment

2015NaturePublishingGroup,adivisionofMacmillanPublishersLimited.AllRightsReserved.
partnerofAGORA,HINARI,OARE,INASP,CrossRefandCOUNTER

http://www.nature.com/news/overhalfofpsychologystudiesfailreproducibilitytest1.18248

7/7