BGPT: Paper Review: Challenges for assessing replicability in preclinical cancer biology

Explore by Goal

Quick Explanation Copied

Core message

This paper documents why assessing replicability in preclinical cancer biology is often blocked before experiments even begin: key statistics/data are missing for most experiments, protocols lack sufficient detail, and essential materials require protracted clarification/sharing—leading to large attrition (193 targeted experiments → 50 completed).

Long Explanation

Paper Review: Challenges for assessing replicability in preclinical cancer biology

Citation: https://doi.org/10.7554/eLife.67995

Registered-report replication project in preclinical cancer biology (selection: 2010–2012 high-impact papers; replication lifecycle: design → protocol review → preparation → conduct → replication-study analysis/writing → peer review).

1) Visual overview: what fraction of the targeted work got completed?

The project targeted 193 experiments across 53 papers, but only 50 experiments across 23 papers were completed.

2) What information was missing? (data availability & statistical computability)

The authors report that raw data suitable for computing effect sizes and conducting power analyses were publicly accessible for only 4/193 (2%); additionally, for 131/193 (68%) experiments they received nothing despite requests.

They also state that 53/193 (27%) relied on only representative images/graphs, and that inferential outcomes were reported but the specific statistical test was unclear for 16/78 (21%) experiments where statistical analysis outcomes were reported.

3) Can you even design a protocol without clarifications?

The paper reports that 0/193 (0%) experiments could be fully replicated without clarifications; 17/193 (9%) needed few clarifications, 77/193 (40%) needed some, 60/193 (31%) moderate, 29/193 (15%) strong, and 10/193 (5%) extreme.

4) Time/effort: how long does “replicability assessment” actually take?

The project reports mean durations (weeks) for phases: design (mean 30), protocol peer review (mean 19), preparation (mean 12), conducted (mean 90), analysis/writing (mean 24), results peer review (mean 22), for an overall mean length of 197 weeks.

Skeptical interpretation: these durations are an “upper envelope” for what replicability assessment can cost in practice, but the authors also note unexpected barriers and material/protocol issues can shift scheduling/cost estimates during the lifecycle.

5) What kinds of fixes do the authors actually propose?

The paper emphasizes improvements across multiple “verification gates”:

Documentation/reporting: better descriptive and inferential statistics, avoiding only representative images/graphs, and providing sufficient methodological detail for protocol design.
Data/code/material transparency: avoid “upon request” policies as the practical mechanism for access; archive underlying data, code, and materials in repositories where feasible.
Pre-registration / registered reports: separate confirmatory from exploratory work and reduce outcome-contingent publication incentives during replication.

Key skeptical caveat: this paper primarily quantifies barriers to verification (information access + protocol sufficiency), not the biological effect-size distribution of the underlying original claims. Therefore, it does not by itself prove “low efficacy” of preclinical cancer biology—rather, it shows that credibility assessment is frequently underdetermined because verification is infeasible.

6) Critical appraisal: what’s strong, what’s uncertain, what could mislead?

Strengths

Large-scale, lifecycle-wide audit: the project tracks barriers across design, protocol peer review, materials access, experiment conduct, analysis/writing, and replication-study peer review—so it identifies failure modes beyond “just experimental execution.”
Registered Reports framing: the replication protocols and analysis plans had peer review before experiments began, reducing some common publication-bias mechanisms in outcomes of the replication study.

Uncertainties / limitations

Original-work selection: papers were selected by impact/citation and by exclusions (e.g., not exclusively genomics/proteomics/high-throughput). That sampling frame may not represent all preclinical cancer biology subfields.
Protocol drift & feasibility constraints: some replications required modifications and some experiments were stopped. Such process changes can blur whether a null/inconsistent replication reflects (i) an original false positive, (ii) unmeasured methodological dependencies, or (iii) irreducible biological variability. The authors discuss the modification burden and that failures may reflect unrecognized omissions or unidentified methodological factors.
Non-response variability: helpfulness/responsiveness varied and had at most modest associations with the extent of clarification needed, so “author non-response” may not cleanly map to “lack of underlying data quality.”

What would most strongly change the conclusions?

If it were shown that, despite missing reporting in the paper, independent teams could reconstruct full methodology and obtain data/materials at high rates in most cases (i.e., the barriers were specific to this cohort/time period/sampling frame), then the urgency of transparency mandates would be reduced.
If later analyses (including their companion meta-analysis) showed that most replications with full protocol/data access still yield consistent effect sizes, then the main driver of uncertainty would shift from “missing verification assets” to “biological heterogeneity/methodological subtlety.”

Author-specific deep dives on BGPT

Feedback:

Updated: April 12, 2026