Review Claim by Claim

Concise appraisal

The article provides a clear, practical, and up-to-date roadmap for planning and conducting systematic reviews in biology emphasizing protocol registration, comprehensive search strategies, independent dual screening, formal risk-of-bias assessment, appropriate meta-analysis choices, narrative synthesis guidance (SWiM), and certainty assessment with GRADE —all adapted to biology-specific resources and heterogeneity concerns. Key methodological recommendations and limitations are documented below with direct source citations.

Core source
De Cassai et al 2025 Methods Guide

In-depth critical review of Methodological Standards for Conducting High-Quality Systematic Reviews

Executive summary

The authors present a comprehensive, practice-oriented synthesis of internationally recognized guidance (Cochrane, PRISMA, GRADE, SWiM) and translate these principles to biological research contexts (ecology, molecular biology, evolutionary studies). They emphasize pre-registration, exhaustive search strategies including domain repositories, independent dual screening and extraction, formal risk-of-bias assessments, careful model selection in meta-analysis, sensitivity analyses, and transparent certainty appraisal with GRADE. The manuscript acknowledges AI tools for screening/extraction but flags transparency and algorithmic bias concerns and calls for explicit reporting of automated steps.

Primary evidence basis

The central recommendations are explicitly stated and drawn from the article itself:

• Protocol development and preregistration and required contents for protocols (objectives, eligibility, detailed searches, risk-of-bias plans) —recommended to reduce selective inclusion and outcome reporting bias Protocol and preregistration guidance
• Comprehensive searches across general databases (MEDLINE, EMBASE, CENTRAL) and biology-specific resources (GenBank, GBIF, Dryad, BioStudies, Zenodo), and explicit reporting of search strings and dates to maximize transparency and reproducibility Search strategy and domain databases
• Dual independent screening and extraction, use of standardized screening tools (Rayyan, Covidence, EPPI-Reviewer) to reduce omission of eligible studies and human error Double screening and screening tools
• Risk-of-bias assessment using appropriate tools matched to design (ROBINS-I/ROBINS-E, QUADAS-2, SYRCLE, AMSTAR 2 for reviews) with domain-level judgments and transparent justifications provided per study and domain Risk of bias assessment practice
• Meta-analysis guidance: choose effect measure suited to data (RR, OR, HR, MD, SMD), quantify heterogeneity (I2) and prefer fixed-effects only when homogeneity is certain; use random-effects to account for between-study variation and conduct sensitivity/subgroup analyses to explore heterogeneity Meta-analysis and heterogeneity guidance
• Narrative synthesis and SWiM: when pooling is inappropriate, use SWiM to transparently report grouping, criteria for comparisons, and certainty assessments SWiM for narrative synthesis
• Certainty of evidence: apply GRADE across five domains (risk of bias, inconsistency, indirectness, imprecision, publication bias) and report certainty even when meta-analysis is not performed GRADE for certainty assessment

Strengths of the paper

• Comprehensive synthesis of cross-disciplinary guidelines and explicit translation to biology, including domain repositories and examples — increases applicability across ecological, molecular, and evolutionary subfields Biology domain adaptation
• Clear, operational recommendations (protocol fields, search reporting, risk-of-bias justification) that are implementable and aligned with PRISMA/PRISMA-P — improves reproducibility and transparency Protocol required elements
• Balanced view on AI: acknowledges automation benefits for scale while cautioning about transparency, accountability, and bias — appropriate and timely given evolving literature AI in SR workflows

Limitations, blindspots, and critical caveats

1. Overreliance on clinical frameworks may under-appreciate biology-specific biases: tools and risk-of-bias instruments originating in clinical research (eg ROBINS, GRADE) may not fully capture ecological sampling biases, taxonomic misidentification, spatial/temporal autocorrelation, or nonindependence of observations common in ecology and evolutionary biology. The authors acknowledge this but provide limited new, concrete tool adaptations for those domain-specific biases Clinical to biology translation limitations
2. Practical implementation resource gaps: recommending searches of many domain repositories is sound, but the paper does not provide annotated example search strings or repository-specific syntax examples which substantially aid reproducibility across teams of different skills; the authors say search strings should be reported but stop short of giving templates for biology resources Search strings and reproducibility
3. AI transparency and validation: authors correctly flag concerns about algorithmic bias and accountability but do not provide reproducible auditing steps or validation benchmarks for AI tools used in screening/extraction; best practice would include recommending publicly shared training/validation datasets and performance metrics for any AI-assisted step AI limitations and ethics
4. Heterogeneity quantification nuance: the paper correctly recommends I2 and sensitivity analyses, but practical guidance on meta-analytic model choice when the number of studies is small (where I2 uncertainty is large) could be expanded; pragmatic decision rules or Bayesian alternatives could be discussed to help reviewers with sparse evidence bases I2 limitations when small study numbers
5. Data sharing and reproducible code: the article notes repositories (Dryad, Zenodo) but does not insist on sharing analysis code (eg meta-analysis scripts, risk-of-bias coding sheets) as a default practice; requiring and exemplifying code sharing would materially improve reproducibility and error detection in SRs Data and code sharing gap

Concrete, prioritized recommendations to improve the article and downstream practice

1. Provide concrete appendix templates: example search strategies and exact query strings for PubMed, EMBASE, GenBank, GBIF, Dryad, and Zenodo; include repository-specific syntax variants and example date-stamped searches (improves immediate reproducibility).
2. Suggest domain-adapted bias tools: expand or propose modifications to existing risk-of-bias tools to capture ecological nonindependence, taxonomic misidentification, spatial/temporal autocorrelation, and sampling frame biases (would close a critical biology blindspot).
3. Mandate analytic transparency: recommend deposition of extraction sheets, risk-of-bias justification tables, and full analysis scripts (eg R markdown or Python notebooks) in a public repository by default for any review claiming to follow these standards.
4. Operationalize AI oversight: require authors using AI to report tool name/version, training/validation metrics on held-out labeled screening datasets, and to provide an audit trail of AI decisions alongside human corrections.
5. Guide small-study meta-analysis: include decision-flow for model selection when studies are few (eg prefer predictive distributions, Bayesian random-effects with weakly informative priors, or report multiple model results) and provide reproducible code examples.

Reproducibility and usability scoring (critical assessment)

Visualization: recommended workflow diagram

Confidence and falsifiability

Confidence in the paper's central claims (that applying these standards improves SR quality and transparency) is high given alignment with established guidance, but empiric falsification is possible: a large-scale empirical evaluation comparing reviews that implement these steps versus those that do not (on reporting completeness, bias indicators, reproducibility of results) could disprove the asserted practical superiority. The authors themselves note how poor protocol transparency undermines trust and recommend pre-registration as a solution Protocol transparency and falsifiability

Minimal checklist for authors (ready-to-use)

• Register protocol (PROSPERO, OSF) and include objectives, eligibility, databases, full search strings, selection process, extraction plan, risk-of-bias tools, and synthesis strategy.
• Run searches across at least 3 general databases + 1 domain repository and report full strings and dates.
• Dual independent title/abstract and full-text screening; report conflicts and resolution mechanism.
• Use validated risk-of-bias tools appropriate to design; provide domain-level justifications in supplement.
• Choose effect measures justified to outcomes; report heterogeneity (I2) and run sensitivity analyses; prefer random-effects unless homogeneity is certain.
• Apply GRADE for certainty assessment and report in summary tables regardless of meta-analysis.
• Deposit extraction sheets, bias tables, and analysis scripts to a public repository (Dryad, Zenodo, GitHub) and include DOI.

Next steps and offer to help

If you want, I can:

• Generate repository-specific example search strings for PubMed, EMBASE, GenBank, GBIF, Dryad (customized to your PICOS) and provide them as ready-to-run queries.
• Create Rmarkdown templates for meta-analysis with example code for fixed/random effects and Bayesian alternatives for small-study meta-analysis.
• Produce a bias-tool adaptation matrix mapping ecology/evolutionary study features to ROBINS/QUADAS domains and suggested extra items to capture spatial/temporal nonindependence.