Variability in the analysis of a single neuroimaging dataset by many teams

Botvinik-Nezer, Rotem; Holzmeister, Felix; Camerer, Colin F.; Dreber, Anna; Huber, Juergen; Johannesson, Magnus; Kirchler, Michael; Iwanir, Roni; Mumford, Jeanette A.; Adcock, R. Alison; Avesani, Paolo; Baczkowski, Blazej; Bajracharya, Aahana; Bakst, Leah; Ball, Sheryl; Barilari, Marco; Bault, Nadège; Beaton, Derek; Beitner, Julia; Benoit, Roland G.; Berkers, Ruud; Bhanji, Jamil P.; Biswal, Bharat B.; Bobadilla-Suarez, Sebastian; Bortolini, Tiago; Bottenhorn, Katherine L.; Bowring, Alexander; Braem, Senne; Brooks, Hayley R.; Brudner, Emily G.; Calderon, Cristian B.; Camilleri, Julia A.; Castrellon, Jaime J.; Cecchetti, Luca; Cieslik, Edna C.; Cole, Zachary J.; Collignon, Olivier; Cox, Robert W.; Cunningham, William A.; Czoschke, Stefan; Dadi, Kamalaker; Davis, Charles P.; Luca, Alberto De; Delgado, Mauricio R.; Demetriou, Lysia; Dennison, Jeffrey B.; Di, Xin; Dickie, Erin W.; Dobryakova, Ekaterina; Donnat, Claire L.; Dukart, Juergen; Duncan, Niall W.; Durnez, Joke; Eed, Amr; Eickhoff, Simon B.; Erhart, Andrew; Fontanesi, Laura; Fricke, G. Matthew; Fu, Shiguang; Galván, Adriana; Gau, Remi; Genon, Sarah; Glatard, Tristan; Glerean, Enrico; Goeman, Jelle J.; Golowin, Sergej A. E.; González-García, Carlos; Gorgolewski, Krzysztof J.; Grady, Cheryl L.; Green, Mikella A.; Moreira, João F. Guassi; Guest, Olivia; Hakimi, Shabnam; Hamilton, J. Paul; Hancock, Roeland; Handjaras, Giacomo; Harry, Bronson B.; Hawco, Colin; Herholz, Peer; Herman, Gabrielle; Heunis, Stephan; Hoffstaedter, Felix; Hogeveen, Jeremy; Holmes, Susan; Hu, Chuan-Peng; Huettel, Scott A.; Hughes, Matthew E.; Iacovella, Vittorio; Iordan, Alexandru D.; Isager, Peder M.; Isik, Ayse I.; Jahn, Andrew; Johnson, Matthew R.; Johnstone, Tom; Joseph, Michael J. E.; Juliano, Anthony C.; Kable, Joseph W.; Kassinopoulos, Michalis; Koba, Cemal; Kong, Xiang-Zhen; Koscik, Timothy R.; Kucukboyaci, Nuri Erkut; Kuhl, Brice A.; Kupek, Sebastian; Laird, Angela R.; Lamm, Claus; Langner, Robert; Lauharatanahirun, Nina; Lee, Hongmi; Lee, Sangil; Leemans, Alexander; Leo, Andrea; Lesage, Elise; Li, Flora; Li, Monica Y. C.; Lim, Phui Cheng; Lintz, Evan N.; Liphardt, Schuyler W.; Vermeer, Annabel B. Losecaat; Love, Bradley C.; Mack, Michael L.; Malpica, Norberto; Marins, Theo; Maumet, Camille; McDonald, Kelsey; McGuire, Joseph T.; Melero, Helena; Leal, Adriana S. Méndez; Meyer, Benjamin; Meyer, Kristin N.; Mihai, Paul Glad; Mitsis, Georgios D.; Moll, Jorge; Nielson, Dylan M.; Nilsonne, Gustav; Notter, Michael P.; Olivetti, Emanuele; Onicas, Adrian I.; Papale, Paolo; Patil, Kaustubh R.; Peelle, Jonathan E.; Pérez, Alexandre; Pischedda, Doris; Poline, Jean-Baptiste; Prystauka, Yanina; Ray, Shruti; Reuter-Lorenz, Patricia A.; Reynolds, Richard C.; Ricciardi, Emiliano; Rieck, Jenny R.; Rodriguez-Thompson, Anais M.; Romyn, Anthony; Salo, Taylor; Samanez-Larkin, Gregory R.; Sanz-Morales, Emilio; Schlichting, Margaret L.; Schultz, Douglas H.; Shen, Qiang; Sheridan, Margaret A.; Silvers, Jennifer A.; Skagerlund, Kenny; Smith, Alec; Smith, David V.; Sokol-Hessner, Peter; Steinkamp, Simon R.; Tashjian, Sarah M.; Thirion, Bertrand; Thorp, John N.; Tinghög, Gustav; Tisdall, Loreen; Tompson, Steven H.; Toro-Serey, Claudio; Tresols, Juan Jesus Torre; Tozzi, Leonardo; Truong, Vuong; Turella, Luca; van Veer, Anna E. ‘t; Verguts, Tom; Vettel, Jean M.; Vijayarajah, Sagana; Vo, Khoi; Wall, Matthew B.; Weeda, Wouter D.; Weis, Susanne; White, David J.; Wisniewski, David; Xifra-Porxas, Alba; Yearling, Emily A.; Yoon, Sangsuk; Yuan, Rui; Yuen, Kenneth S. L.; Zhang, Lei; Zhang, Xu; Zosky, Joshua E.; Nichols, Thomas E.; Poldrack, Russell A.; Schonberg, Tom

doi:10.1038/s41586-020-2314-9

DetailsÜbersicht

Variability in the analysis of a single neuroimaging dataset by many teams

Botvinik-Nezer, R., Holzmeister, F., Camerer, C. F., Dreber, A., Huber, J., Johannesson, M., et al. (2020). Variability in the analysis of a single neuroimaging dataset by many teams. Nature, 582(7810), 84-88. doi:10.1038/s41586-020-2314-9.

Item is Freigegeben

einblenden: alle ausblenden: alle

Basisdaten

einblenden: ausblenden:

Datensatz-Permalink: https://hdl.handle.net/21.11116/0000-0006-715A-6 Versions-Permalink: https://hdl.handle.net/21.11116/0000-000C-DC62-E

Genre: Zeitschriftenartikel

Dateien

einblenden: Dateien

ausblenden: Dateien

:

Botvinik-Nezer_2020.pdf (Preprint), 2MB

Öffnen Speichern

Datei-Permalink:
https://hdl.handle.net/21.11116/0000-000C-CF88-2

Name:
Botvinik-Nezer_2020.pdf

Beschreibung:
-

OA-Status:
Grün

Sichtbarkeit:
Öffentlich

MIME-Typ / Prüfsumme:
application/pdf / [MD5]

Technische Metadaten:

Öffnen

Copyright Datum:
-

Copyright Info:
-

Lizenz:
https://creativecommons.org/licenses/by-nc-nd/4.0/

Externe Referenzen

einblenden:

Urheber

einblenden:

ausblenden:

Urheber:
Botvinik-Nezer, Rotem, Autor
Holzmeister, Felix, Autor
Camerer, Colin F., Autor
Dreber, Anna, Autor
Huber, Juergen, Autor
Johannesson, Magnus, Autor
Kirchler, Michael, Autor
Iwanir, Roni, Autor
Mumford, Jeanette A., Autor
Adcock, R. Alison, Autor
Avesani, Paolo, Autor
Baczkowski, Blazej¹, Autor
Bajracharya, Aahana, Autor
Bakst, Leah, Autor
Ball, Sheryl, Autor
Barilari, Marco, Autor
Bault, Nadège, Autor
Beaton, Derek, Autor
Beitner, Julia, Autor
Benoit, Roland G.², Autor
Berkers, Ruud², Autor         Bhanji, Jamil P., AutorBiswal, Bharat B., AutorBobadilla-Suarez, Sebastian, AutorBortolini, Tiago, AutorBottenhorn, Katherine L., AutorBowring, Alexander, AutorBraem, Senne, AutorBrooks, Hayley R., AutorBrudner, Emily G., AutorCalderon, Cristian B., AutorCamilleri, Julia A., AutorCastrellon, Jaime J., AutorCecchetti, Luca, AutorCieslik, Edna C., AutorCole, Zachary J., AutorCollignon, Olivier, AutorCox, Robert W., AutorCunningham, William A., AutorCzoschke, Stefan, AutorDadi, Kamalaker, AutorDavis, Charles P., AutorLuca, Alberto De, AutorDelgado, Mauricio R., AutorDemetriou, Lysia, AutorDennison, Jeffrey B., AutorDi, Xin, AutorDickie, Erin W., AutorDobryakova, Ekaterina, AutorDonnat, Claire L., AutorDukart, Juergen, AutorDuncan, Niall W., AutorDurnez, Joke, AutorEed, Amr, AutorEickhoff, Simon B., AutorErhart, Andrew, AutorFontanesi, Laura, AutorFricke, G. Matthew, AutorFu, Shiguang, AutorGalván, Adriana, AutorGau, Remi, AutorGenon, Sarah, AutorGlatard, Tristan, AutorGlerean, Enrico, AutorGoeman, Jelle J., AutorGolowin, Sergej A. E., AutorGonzález-García, Carlos, AutorGorgolewski, Krzysztof J., AutorGrady, Cheryl L., AutorGreen, Mikella A., AutorMoreira, João F. Guassi, AutorGuest, Olivia, AutorHakimi, Shabnam, AutorHamilton, J. Paul, AutorHancock, Roeland, AutorHandjaras, Giacomo, AutorHarry, Bronson B., AutorHawco, Colin, AutorHerholz, Peer, AutorHerman, Gabrielle, AutorHeunis, Stephan, AutorHoffstaedter, Felix, AutorHogeveen, Jeremy, AutorHolmes, Susan, AutorHu, Chuan-Peng, AutorHuettel, Scott A., AutorHughes, Matthew E., AutorIacovella, Vittorio, AutorIordan, Alexandru D., AutorIsager, Peder M., AutorIsik, Ayse I., AutorJahn, Andrew, AutorJohnson, Matthew R., AutorJohnstone, Tom, AutorJoseph, Michael J. E., AutorJuliano, Anthony C., AutorKable, Joseph W., AutorKassinopoulos, Michalis, AutorKoba, Cemal, AutorKong, Xiang-Zhen, AutorKoscik, Timothy R., AutorKucukboyaci, Nuri Erkut, AutorKuhl, Brice A., AutorKupek, Sebastian, AutorLaird, Angela R., AutorLamm, Claus, AutorLangner, Robert, AutorLauharatanahirun, Nina, AutorLee, Hongmi, AutorLee, Sangil, AutorLeemans, Alexander, AutorLeo, Andrea, AutorLesage, Elise, AutorLi, Flora, AutorLi, Monica Y. C., AutorLim, Phui Cheng, AutorLintz, Evan N., AutorLiphardt, Schuyler W., AutorVermeer, Annabel B. Losecaat, AutorLove, Bradley C., AutorMack, Michael L., AutorMalpica, Norberto, AutorMarins, Theo, AutorMaumet, Camille, AutorMcDonald, Kelsey, AutorMcGuire, Joseph T., AutorMelero, Helena, AutorLeal, Adriana S. Méndez, AutorMeyer, Benjamin, AutorMeyer, Kristin N., AutorMihai, Paul Glad³, Autor         Mitsis, Georgios D., AutorMoll, Jorge, AutorNielson, Dylan M., AutorNilsonne, Gustav, AutorNotter, Michael P., AutorOlivetti, Emanuele, AutorOnicas, Adrian I., AutorPapale, Paolo, AutorPatil, Kaustubh R., AutorPeelle, Jonathan E., AutorPérez, Alexandre, AutorPischedda, Doris, AutorPoline, Jean-Baptiste, AutorPrystauka, Yanina, AutorRay, Shruti, AutorReuter-Lorenz, Patricia A., AutorReynolds, Richard C., AutorRicciardi, Emiliano, AutorRieck, Jenny R., AutorRodriguez-Thompson, Anais M., AutorRomyn, Anthony, AutorSalo, Taylor, AutorSamanez-Larkin, Gregory R., AutorSanz-Morales, Emilio, AutorSchlichting, Margaret L., AutorSchultz, Douglas H., AutorShen, Qiang, AutorSheridan, Margaret A., AutorSilvers, Jennifer A., AutorSkagerlund, Kenny, AutorSmith, Alec, AutorSmith, David V., AutorSokol-Hessner, Peter, AutorSteinkamp, Simon R., AutorTashjian, Sarah M., AutorThirion, Bertrand, AutorThorp, John N., AutorTinghög, Gustav, AutorTisdall, Loreen, AutorTompson, Steven H., AutorToro-Serey, Claudio, AutorTresols, Juan Jesus Torre, AutorTozzi, Leonardo, AutorTruong, Vuong, AutorTurella, Luca, Autorvan Veer, Anna E. ‘t, AutorVerguts, Tom, AutorVettel, Jean M., AutorVijayarajah, Sagana, AutorVo, Khoi, AutorWall, Matthew B., AutorWeeda, Wouter D., AutorWeis, Susanne, AutorWhite, David J., AutorWisniewski, David, AutorXifra-Porxas, Alba, AutorYearling, Emily A., AutorYoon, Sangsuk, AutorYuan, Rui, AutorYuen, Kenneth S. L., AutorZhang, Lei, AutorZhang, Xu, AutorZosky, Joshua E., AutorNichols, Thomas E., AutorPoldrack, Russell A., AutorSchonberg, Tom, Autor mehr..

Affiliations:
1International Max Planck Research School on Neuroscience of Communication: Function, Structure, and Plasticity, MPI for Human Cognitive and Brain Sciences, Max Planck Society, ou_2616696
2Max Planck Research Group Adaptive Memory, MPI for Human Cognitive and Brain Sciences, Max Planck Society, ou_2295691
3Max Planck Research Group Neural Mechanisms of Human Communication, MPI for Human Cognitive and Brain Sciences, Max Planck Society, ou_634556

Inhalt

einblenden:

ausblenden:

Schlagwörter: Decision; Decision making; Human behaviour; Scientific community

Zusammenfassung: Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2,3,4,5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

Details

einblenden:

ausblenden:

Sprache(n): eng - English

Datum: Eingereicht: 2019-11-14Angenommen: 2020-04-07Online veröffentlicht: 2020-05-20Erschienen: 2020-06-04

Publikationsstatus: Erschienen

Seiten: -

Ort, Verlag, Ausgabe: -

Inhaltsverzeichnis: -

Art der Begutachtung: -

Identifikatoren: DOI: 10.1038/s41586-020-2314-9
Anderer: epub 2020
PMID: 32483374

Art des Abschluß: -

Projektname : -

Grant ID : P29362-G27; SFB F63; P 32686

Förderprogramm : Austrian Science Fund (FWF) (FWF)

Förderorganisation : -

Projektname : -

Grant ID : ISF 2004/15

Förderprogramm : -

Förderorganisation : Israel Science Foundation (ISF)

Projektname : -

Grant ID : NHS14-1719:1

Förderprogramm : -

Förderorganisation : Swedish Foundation for Humanities and Social Sciences

Projektname : -

Grant ID : R24MH117179; NIH-NIBIB P41 EB019936; NIH-NIMH R01 MH083320; NIH RF1 MH120021; R01 DA041353; ZICMH002888; ZICMH002960

Förderprogramm : -

Förderorganisation : United States Department of Health & Human Services

Projektname : -

Grant ID : 12T2517N

Förderprogramm : -

Förderorganisation : Research Foundation Flanders (FWO)

Projektname : -

Grant ID : 665501

Förderprogramm : Marie Skodowska-Curie Actions under COFUND

Förderorganisation : European Commission (EC) (EC)

Projektname : -

Grant ID : -

Förderprogramm : University of Basel Research Fund

Förderorganisation : -

Projektname : -

Grant ID : -

Förderprogramm : -

Förderorganisation : La Caixa Foundation

Projektname : -

Grant ID : WWTF VRG13-007

Förderprogramm : Vienna Science and Technology Fund

Förderorganisation : -

Projektname : -

Grant ID : 71801110; 71971199; 71602175; 71942004

Förderprogramm : -

Förderorganisation : National Natural Science Foundation of China (NSFC) (NSFC)

Projektname : -

Grant ID : 18YJC630268

Förderprogramm : -

Förderorganisation : Ministry of Education in China (MOE)

Projektname : -

Grant ID : 2018M633270

Förderprogramm : -

Förderorganisation : China Postdoctoral Science Foundation

Projektname : -

Grant ID : 390523135; CRC1193

Förderprogramm : -

Förderorganisation : German Research Foundation (DFG)

Projektname : -

Grant ID : -

Förderprogramm : -

Förderorganisation : Brain Canada

Projektname : -

Grant ID : -

Förderprogramm : -

Förderorganisation : Health Canada

Projektname : -

Grant ID : R01MH096906

Förderprogramm : -

Förderorganisation : National Institute of Mental Health of the NIH (NIMH)

Projektname : -

Grant ID : -

Förderprogramm : Canada First Research Excellence Fund

Förderorganisation : -

Projektname : -

Grant ID : 785907

Förderprogramm : Horizon 2020 Research and Innovation Programme

Förderorganisation : European Commission (EC) (EC)

Projektname : -

Grant ID : -

Förderprogramm : Foundation CELLEX

Förderorganisation : Max Planck Society

Projektname : -

Grant ID : LSHM16053-SGF

Förderprogramm : -

Förderorganisation : Dutch foundation LSH-TKI

Projektname : -

Grant ID : -

Förderprogramm : -

Förderorganisation : National Science Foundation (NSF)

Projektname : -

Grant ID : -

Förderprogramm : -

Förderorganisation : Australian National Imaging Facility (NIF)

Projektname : -

Grant ID : -

Förderprogramm : National Collaborative Research Infrastructure Strategy (NCRIS)

Förderorganisation : -

Projektname : -

Grant ID : 452-17-013

Förderprogramm : VIDI

Förderorganisation : Netherlands Organisation for Scientific Research (NWO)

Projektname : -

Grant ID : -

Förderprogramm : -

Förderorganisation : Swedish Research Council

Projektname : -

Grant ID : -

Förderprogramm : -

Förderorganisation : Tianqiao and Chrissy Center for Social and Decision Neuroscience Center Leadership Chair

Projektname : -

Grant ID : 30991544

Förderprogramm : Belgian Excellence of Science program (EOS)

Förderorganisation : FNRS-Belgium

Projektname : -

Grant ID : R4195

Förderprogramm : -

Förderorganisation : Era-NET Neuron

Projektname : -

Grant ID : 16YJC630103

Förderprogramm : -

Förderorganisation : Ministry of Education of Humanities and Social Science

Projektname : -

Grant ID : 100309/Z/12/Z

Förderprogramm : -

Förderorganisation : Wellcome Trust

Quelle 1

einblenden:

ausblenden:

Titel: Nature

Kurztitel : Nature

Genre der Quelle: Zeitschrift

Urheber:

Affiliations:

Ort, Verlag, Ausgabe: London : Nature Publishing Group

Seiten: - Band / Heft: 582 (7810) Artikelnummer: - Start- / Endseite: 84 - 88 Identifikator: ISSN: 0028-0836
CoNE: https://pure.mpg.de/cone/journals/resource/954925427238

Datensatz

Basisdaten

Dateien

Externe Referenzen

Urheber

Inhalt

Details

Veranstaltung

Entscheidung

Projektinformation

Quelle 1