Empirical evidence of a replicability crisis in linking psychological variables to brain structure: causes, consequences and recommendations

1. BackgroundLinking interindividual differences in psychological phenotype to interindividual variations in brain structure stands as an old dream for psychology and a crucial question for cognitive neurosciences. Evidence of covariations with brain structure, specifically gray matter volume (GMV), have been brought for a range of psychological measures. Nevertheless, in line with the recent concerns regarding reproducibility of findingsin social and psychological sciences 13, studies raised questions regarding the replicability of the findings in the so-called structural brain behavior (SBB)-association studies 4,5. Here we address the question of replicability of SBB-associations among heathy adults using an empirical investigation. 2. MethodParticipants consisted of 466 healthy adults from the enhanced NKI Rockland cohort 6 with structural MRI and scores on broad range of psychological measurements (cognitive and personality scores). All T1- weighted scans were processed with CAT12 7 (modulated nonlinear only). Replicability of association of each psychological score with GMV is assessed using two approaches. 1) 100 random subsamples were drawnfrom the main cohort (discovery samples). In each sample, structural associations of each psychological score were assessed using GLM. Inference was made at cluster level using TFCE 8 (P ¡ 0.05; ¿ 100 voxels). Spatial overlap of these findings over 100 subsamples, characterizes replicability of morphological associations of the respective score using the exploratory approach. 2) For each of the 100 discovery subsamples, an ageandsex-matched test sample was generated from the main cohort. For each psychological variable, the significant clusters from the exploratory approach were used as a-priori ROIs. Association between the score and mean GMV within each ROI was compared between each discovery and its paired test sample, providing means to assess ROI-based SBB replicability rates. To study the influence of sample size, replicability ratesare also compared between subsamples of three different size. We further compared SBB-findings in healthy adults with SBB-associations in a clinical cohort. 3. Resultswhen probing a range of psychometric variables with an exploratory approach, significant associations between psychological phenotype and GMV are found rarely within healthy cohort. Where significant associations were found, these associations showed a poor replicability and high spatial variability (Fig1), which is further negatively influenced by decreased sample size. ROI-based approach also confirmed low rate of replicated as- sociations in the independent samples and highlighted over-estimation of effect size in the exploratory analyses (Fig2). Within the clinical cohort more stable SBB-associations were detected (Fig3). 5. References 1. Open Science Collaboration OS. Estimating the reproducibility of psychological science [Internet]. Science 2015;349(6251):aac4716.[cited 2018 Jun 25 ] Available from: http://www.ncbi.nlm.nih.gov/ pubmed/263154432. Camerer CF, Dreber A, Holzmeister F, et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015 [Internet]. Nat. Hum. Behav. 2018;1.[cited 2018 Aug 29 ] Available from: http://www.nature.com/articles/s41562-018-0399-z3. Ioannidis JPA. Why most published research findings are false [Internet]. PLoS Med. 2005;2(8):06960701. [cited 2018 Jun 29] Available from: http://www.ncbi.nlm.nih.gov/pubmed/16060722 4. Boekel W, Wagenmakers EJ, Belay L, et al. A purely confirmatory replication study of structural brainbehavior correlations [Internet]. Cortex 2015;66:115133.[cited 2017 Oct 24 ] Available from: http://www. sciencedirect.c 5. Genon S, Wensing T, Reid A, et al. Searching for behavior relating to grey matter volume in a-priori defined right dorsal premotor regions: Lessons learned [Internet]. Neuroimage 2017;157:144156.[cited 2017 Oct 23 ] Available from: http://www.ncbi.nlm.nih.gov/pubmed/28552730 6. Nooner KB, Colcombe SJ, Tobe RH, et al. The NKI-Rockland Sample: A Model for Accelerating the Pace of Discovery Science in Psychiatry. [Internet]. Front. Neurosci. 2012;6:152.[cited 2017 Dec 11 ] Available from: http://www.ncbi.nlm.nih.gov/pubmed/23087608 7. Gaser C, Dahnke R. CAT - A Computational Anatomy Toolbox for the Analysis of Structural MRI Data. HBM Conf. 2016 2016;32(7):7743.8. Smith SM, Nichols TE. Threshold-free cluster enhancement: Addressing problems of smoothing, thresh- old dependence and localisation in cluster inference [Internet]. Neuroimage 2009;44(1):83-98.[cited 2018 May 16 ] Available from: https://ac.els-cdn.com/S1053811908002978/1-s2.0-S1053811908002978-main.pdf tid=bb431556-fec9-4fc2-b24f-946f4a006dd1&acdnat=1526466648 d7bb1193e109afcba5d4bb2048e54b599. Dwan K, Gamble C, Williamson PR, Kirkham JJ. Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias - An Updated Review [Internet]. PLoS One 2013;8(7):e66844. [cited 2018 Jun 5 ] Available from: http://dx.plos.org/10.1371/journal.pone.0066844