The Evidence Layer for Scientific AI

 Quick Analysis Plan Copied

 Long Analysis Plan

Evidence Battle-Test: Do BWAS Replications Require Thousands of Participants?

Yes — and the support is unusually rigorous. The core evidence comes from a Nature paper (2022) that resampled three large-scale datasets (ABCD, HCP, UK Biobank) to model how sample size affects brain-wide association study (BWAS) effect sizes and replication.

Reported observations across ABCD (3,928 youth split evenly into discovery/replication), HCP (900 adults), and UK Biobank (900 subsamples from 32,572 middle-aged adults):

• Univariate BWAS effect sizes were tiny — median |r| ≈ 0.01; top 1% exceeded |r| > 0.06; largest replicated out-of-sample correlation was |r| = 0.16.
• Simulation showed substantial inflation, false negatives, and sign errors in smaller samples; replication probability was low even with rigorous denoising.
• Multivariate SVR on RSFC predicted cognitive phenotypes with r_pred up to 0.39 out-of-sample — stronger than univariate, but still with in-sample inflation.
• Effect size distributions were similar across ABCD, HCP, and UKB when size-matched, supporting the generality of the finding across age ranges (9–69 years).

BGPT inference: The authors' interpretation — that thousands are needed — is well-supported for univariate RSFC/cortical-thickness BWAS targeting cognitive and psychopathology phenotypes. However, the claim's strongest form rests on the assumption that typical BWAS effect sizes reflect biological reality rather than measurement noise. If behavioral measurement reliability improves substantially (e.g., reliable task-based phenotypes), effect sizes might rise, reducing the required n below the 'thousands' threshold — this remains untested in these data.

Blind spots and limitations:

• Observational, not causal — the resampling logic assumes that in the very large ABCD sample the observed effect sizes converge on the true population values, which is an assumption, not a proof.
• Focus limited to RSFC and cortical thickness; task fMRI, DTI, or multimodal measures are not addressed, and may have different effect size distributions.
• Site/scanner harmonization reduces but does not eliminate residual confounds; demographic covariate adjustment choices affect results.
• Potential for pipeline selection bias (a specific denoising pipeline was used); alternative pipelines could yield different effect size distributions.
• The conclusion is about current BWAS practices; it does not preclude methodological innovations that shift the sample-size requirement downward.

Falsification test: The authors specify what would overturn this finding: a new large-scale BWAS where small-sample results reliably replicate across independent datasets with stable effect sizes matching in-sample estimates, or where reliable replication is achieved with far fewer than thousands of participants.

Confidence level: High for the descriptive claim that current BWAS practices yield tiny effects requiring large samples; moderate for the prescriptive claim that 'thousands' is a fixed universal requirement, since it depends on measurement reliability and pipeline choices that could evolve.