Ulrich Schimmack | How robust are published results?

Z-curve: How robust are published results in scientific literatures? 

Professor Ulrich Schimmack of the University of Toronto will present this online luncheon talk on evaluating publication bias and false-positive rates in the scientific literature. Through analysis of clinical trials in leading medical journals, his work demonstrates that the conventional significance criterion (α = .05) yields a false positive risk of 13%, providing empirical evidence that contradicts claims about widespread false positives in published research. Further investigation of 65,970 test statistics from emotion research journals reveals comparable findings, with false positive risks estimated between 5% and 33%. The research indicates that adjusting α to .01 effectively reduces false positive risk below 5% across both domains.

While these findings suggest more modest false positive rates than previously postulated, the analyses reveal persistent publication bias affecting effect size estimates. The research also identifies substantial variation in statistical power, with predicted replication rates ranging from 15% to 70% in psychological studies. These findings offer methodological implications for statistical threshold selection and sample size determination in replication studies.

Recommended Pre-Readings:

• Estimating the false discovery risk of (randomized) clinical trials in medical journals based on published p-values
• Credibility of results in emotion science: a Z-curve analysis of results in the journals Cognition & Emotion and Emotion

If you are interested in attending or joining virtually, please fill out the expression of interest form below.

EOI Form

Ulrich Schimmack is a Professor of Psychology at the University of Toronto. His research focuses on the scientific understanding of happiness, formally known as subjective well-being (SWB). His work examines both cognitive and affective components of well-being, investigating how life-satisfaction judgments and the balance of pleasant and unpleasant experiences contribute to overall happiness.

Learn more about him here.