Review Papers Against Their Own Results

 Quick Explanation Copied

 Long Explanation

Core Argument, Verified Against Its Own Logic

Rubin's central device is the formula 1 − (1 − α)^k, where k is the number of tests formally associated with a reported statistical inference — not the number of tests a researcher performed k-based error rate argument. This redefinition does the heavy lifting: once unreported tests are excluded from the reference set, most p-hacking scenarios leave each individual inference at α_Individual = .050. The logic is internally consistent — the 20-sided-dice example for optional stopping is arithmetically sound — but note the hidden assumption: that researchers' inferences genuinely refer to study-specific null populations, not to broader substantive claims. If a p-hacked significant result is presented as evidence for a substantive hypothesis ("listening to 'When I'm Sixty-Four' makes people younger"), Rubin concedes this is a theoretical error — meaning the practice is still damaging, just re-labeled. That concession substantially dilutes the practical bite of the title.

The formula only produces inflation when a researcher makes a joint inference from k>1 tests without adjusting α_Constituent — e.g., the .143 rate for k=3 shown above. Rubin's claim that this is "rare and easily resolved" depends on García-Pérez's finding that all observed multiple testing in 109 articles from two psychology journals (2021–June 2022) consisted of single tests of individual hypotheses García-Pérez 2023 empirical audit — a convenience sample from two selective journals that may under-represent forking-path inference.

Where the Convergent Counter-Evidence Bites

Multiple independent simulations show systematic, sometimes large inflation — seemingly contradicting the title — though each involves outcome-dependent selection the paper would classify as theoretical-error territory: omission of random effects in mixed models raises average fixed-effects Type I error to 0.078 (max 0.274) LMM misspecification meta-analysis; selecting the smallest p-value across 72 analysis configurations inflates Type I error to ~0.23 in a trial-simulation study RCT design-change simulation; aggressive outlier filtering (MAD 2.0) yields FPR ≈ 0.151 across 1,000,000 simulated RT analyses RT multiverse simulation. Rubin's rebuttal would be that each involves rejection of a joint studywise null or theoretical misinterpretation — but that is precisely the move critics will find circular: any demonstrated inflation is re-classified as "irrelevant" because the affected inference is deemed atheoretical. The paper offers no prevalence data showing that theory-restricted inferences are what researchers actually make in p-hacking contexts Author-reported limitation.

Blind Spots and What Would Change the Verdict

Self-referential framing: Rubin cites his own work repeatedly (Rubin 2017a/b, 2020a/b, 2021a-c, 2022, 2024) as support, and his disclosed prior stance against preregistration creates an interpretive conflict-of-interest risk worth noting alongside the declared "no conflict." Citation imbalance: Simmons et al. (2011), the load-bearing target, is critiqued extensively but the abundant simulation literature documenting real-world inflation in genetics, diagnostics, and clinical trials is largely unaddressed — e.g., sex-stratified genetic meta-analysis shows cryptic-relatedness inflation Genetic meta-analysis inflation study, and response-rate misspecification in Simon designs inflates α from 0.0948 to 0.284–0.369 Simon design misspecification study. What would falsify Rubin: a corpus-level study showing that reported inferences in p-hacked or outcome-selected contexts typically function as joint inferences over the unreported analysis family (e.g., via multiverse/specification-curve reanalysis of published papers showing the "selected" result is the only significant one among many plausible specifications). The multiverse-simulation literature already gestures in this direction.

Confidence: moderate — the logical analysis is sound; the empirical generalization ("not usually inflated") is untested. Check Prevalence Evidence Author Review: Mark Rubin

 BGPT Paper Review

Study Novelty

Challenges the dominant post-2011 consensus (Simmons et al.) with a coherent minority position tracing to Rothman/Fisher traditions; the statistical/theoretical-error distinction framing is genuinely useful even if components are not new.

Scientific Quality

Logically rigorous conceptual work, but purely argumentative: no simulation, meta-analysis, or corpus data tests whether theory-restricted inferences are common. Heavy self-citation; counter-evidence literatures (genetics, trials, multiverse) unengaged.

Study Generality

Applies broadly across significance-testing disciplines, but the argument is anchored in psychology's replication crisis and hinges on contested reference-class choices.

Study Usefulness

Useful corrective vocabulary (statistical vs. theoretical errors; k formally associated with an inference) for methods debates, though its practical guidance ("inflation is transparent and resolvable") lacks empirical grounding.

Study Reproducibility

No original data, code, or simulation; arguments are reconstructible only by re-deriving the examples, and key premises (prevalence of inference types) are not testable from the paper itself.

Explanatory Depth

Deep engagement with Neyman-Pearson vs Fisher foundations and the Bayesian inversion fallacy, but the mechanistic account of why replication failures occur remains asserted rather than demonstrated.