BGPT: Paper Review: Ten simple rules for neuroimaging meta-analysis

Explore by Goal

Quick Explanation Copied

Ten rules that mainly target bias control in neuroimaging meta-analysis

This paper is a best-practice + reporting checklist focused on coordinate-based neuroimaging meta-analysis: define the question/criteria up front, keep search coverage and reference space consistent, handle repeated contrasts, double-check extracted coordinates, correct for multiple comparisons, run diagnostics, and report enough detail to reproduce the pipeline and inclusion decisions.

Long Explanation

Paper review (methodology checklist): Ten simple rules for neuroimaging meta-analysis

DOI: 10.1016/j.neubiorev.2017.11.012

Core thesis: reliable neuroimaging meta-analytic conclusions require explicit control of (i) what is pooled, (ii) whether inference spaces/coverage are comparable, (iii) dependency created by multiple contrasts from the same subjects, (iv) extraction errors, (v) multiplicity and error-rate reporting, and (vi) diagnostics + traceable reporting.

Paper metadata snapshot

Publication: 2017 (accepted Nov 2017; published 24 Nov 2017)
Format: methodological guidelines + checklist
Primary target: coordinate-based CBMA; many rules extend to image-based

VISUAL 1 — “Score-bar” view of the paper-review rubric you provided

VISUAL 2 — Ten rules as a “quality-control pipeline”

Note: the numeric weights in this figure are not from the paper; they are a visualization-only heuristic to help you see the intended emphasis across rules. The underlying rule content is from the paper’s ten-rule structure.

VISUAL 3 — Checklist “coverage map” (rule-to-item index)

This heatmap is again a visual aid, not an encoded fact from the paper. The conceptual targets (question definition, inclusion criteria, space consistency, contrast dependence, QC, preregistration/planning, multiplicity/error control, diagnostics, transparent reporting) come directly from the paper’s ten-rule text.

What the paper gets right (skeptical but favorable)

It attacks the core reproducibility failure modes specific to coordinate-based CBMA: inconsistent inference spaces (ROI/SVC vs whole-brain), reference-space mismatch (MNI vs Talairach), dependence from multiple contrasts, and transcription/sign errors in extracted coordinates.
It clarifies an interpretation trap: in coordinate-based meta-analysis, significance reflects spatial convergence in a null space, not the magnitude/direction of activation/deactivation (contrast sign information is lost in typical CBMA).
It incentivizes preregistration and “analysis-before-seeing” discipline to combat multiple testing degrees of freedom and p-hacking-like behavior in meta-analytic pipelines (including stopping rules like “repeat until significant”).
It demands diagnostics (contribution analysis, funnel plots, heterogeneity/meta-regression where appropriate) so readers can evaluate whether clusters are robust or driven by small/odd subgroups.

Critical blind spots & limitations (what this checklist does *not* fully solve)

No guarantee of empirical accuracy: the paper provides guidelines, but it does not itself demonstrate (within the paper) that adherence improves replication rates across many topics—so the checklist is strongest as a process quality control framework, not as a proven causal lever.
Publication bias is conceptually different in CBMA: coordinate-based methods test convergence and may be insensitive to some forms of non-significant-result suppression; thus “usual” bias checks may not work as directly as in effect-size meta-analysis.
“Same search coverage” can be fuzzy in practice: the checklist discusses excluding explicit ROI/SVC and assessing hidden/partial coverage via acquisition parameters, but real-world reporting is incomplete in many papers—so the meta-analyst may face uncertainty and have to contact authors or make judgment calls.
Degrees of freedom remain across “allowed” branches: even with preregistration, inclusion of different contrast handling, thresholds, null-space adjustments (e.g., ROI masks), and sensitivity analyses can still create flexibility; the paper warns about this generally by recommending planning and transparency, but it can’t eliminate it.

VISUAL 4 — “Rule 4” risk: what changes if coverage/reference space differs?

The plot illustrates the paper’s logic: coordinate-based CBMA often tests convergence against a null assuming whole-brain uniform opportunity; including ROI/SVC or mismatched search coverage/reference space can inflate apparent significance by violating the null model.

Rule-by-rule critique (skimmable)

Rule	Main purpose	Most likely failure mode if ignored
1	Specify research question, paradigms/contrasts, inclusion/exclusion criteria	Pooling mismatched cognitive processes → weak/heterogeneous convergence interpretation
2	Consider power: enough experiments but also balance heterogeneity vs specificity	Small-N meta-analysis driven by few studies → poor generalization
3	Collect/organize data with tracking; extract required metadata (coords, N, space, inference)	Ineligible studies included; missing space/threshold metadata; extraction errors
4	Same search coverage + align reference space (MNI/Talairach); exclude/adapt ROI/SVC/partial coverage	Null-space violation → inflated false convergence
5	Adjust for multiple contrasts (dependence within subject group)	Over-counting dependent contrasts → invalid uncertainty
6	Double-check extracted coordinates/spaces; QC diagnostics like map/peak plausibility	Sign errors, wrong space labels, out-of-mask coordinates silently contaminating results
7	Plan analyses beforehand; preregister protocol (mark deviations)	Degrees-of-freedom exploitation → distorted p-values & selective reporting of analyses
8	Balance sensitivity and false positives with multiple-comparison control & thresholds	Chance findings dominate future literature → unreliable follow-ups
9	Show diagnostics: contributions, funnel plots, heterogeneity/robustness	Clusters driven by one subgroup/small studies go undetected
10	Be transparent: report flow chart, experiment-level table, included contrasts, QC/adjustments	Others cannot reconstruct inclusion decisions or verify coordinate extraction

Each row is a compact restatement of the paper’s ten-rule intent.

What would change my mind? (falsification-oriented reflection)

If future empirical studies show that strict adherence to these rules does not improve reproducibility/traceability (e.g., replication rates and external validation fail to improve), then the checklist would be less impactful as a process prescription.
If coordinate-based convergence meta-analysis (under well-specified null spaces) is shown to permit more direct inference about effect magnitude/direction than the paper claims, then the interpretational caution would require revision.

Author reviews (bespoke BGPT links)

Feedback:

Updated: March 29, 2026