Review Papers Claim by Claim

 Long Explanation Copied

Scope and Design

Guéguen et al. benchmarked 38 combinations of sequence-resolution algorithms (DADA2 standalone; LotuS3 with DADA2 or UPARSE), taxonomic classifiers (Kraken2, DECIPHER, RDP, MMseqs2, Lambda, Metaxa2), and databases (SILVA, GreenGenes2, RDP, RefSeq, Metaxa2) across seven 16S hypervariable regions and three mock communities (Zymo 8 genera, ZIEL1 13, ZIEL2 18; 29 unique genera), evaluated by F1, AUPRC, and CLR-based Euclidean distance. Notably, this is the first benchmark to include the 2024 updates of RDP v2.14, GreenGenes2, and LotuS3 16S benchmark preprint.

Key Findings

• V1-V3 gave the best medians (F1 0.737, L2 29.0, AUPRC 0.383); V4 was notably weaker despite its historical popularity (F1 0.56, AUPRC 0.193) Region metrics table.
• Four pipelines dominated: DADA2+MMseqs2+SILVA, DADA2+Metaxa2, LotuS3(DADA2)+RDP, LotuS3(UPARSE)+RDP; on the combined 3Z mock, LotuS3(DADA2)+RDP had the highest F1 (0.931) and LotuS3 was ~9.5-fold faster than DADA2-based pipelines Top pipelines and runtime.
• Metrics alone hide failures: dominant gut genera Clostridium and Ruminococcus were frequently missed, and Lactobacillus was detected only by MMseqs2, Metaxa2, and LotuS3+RDP Elusive genera analysis.
• Quantification was broadly inaccurate regardless of pipeline; Alistipes and Atopobium were over-quantified by all four top pipelines Quantification deviation.

Critical Appraisal

Strengths: systematic coverage of the full workflow, transparent code and data (GitHub/Zenodo; PRJNA674596), and honest reporting that identification and quantification diverge. The GC2 underperformance and the L2-pseudo-count caveat are candidly discussed. Limitations the authors acknowledge: only three mocks with one species per genus, a single primer pair, incomplete combination space, and the Escherichia-Shigella merge potentially favoring Metaxa2. BGPT adds: external evidence supports region-dependence of results (V1-V2 vs V3-V4 biases for taxa like Akkermansia and Bifidobacterium in human gut profiling V1-V2 vs V3-V4 primer bias), and complex 227-strain mocks showed substantial reference-miss rates even for top denoisers Complex mock denoiser benchmark, so the 29-genus mocks here likely overstate real-world accuracy. GreenGenes2's failure is plausibly an artifact of its V4-heavy 23M-sequence expansion, which the authors infer but do not test directly. What would falsify the headline: an independent rerun on PRJNA674596 or additional mocks where V4 or a GG2 combination beats V1-V3 DADA2+MMseqs2+SILVA.

Recommendations (authors')

Default: V1-V3 + DADA2+MMseqs2+SILVA (pooled or per-sample) or LotuS3+DADA2+RDP; DADA2+Metaxa2 only when quantification of genera stable since the 2012 SILVA 111 reference matters. Future work should control 16S copy-number variation for quantification.

 BGPT Paper Review

Study Novelty

Incremental update of a well-established benchmark genre; novelty lies in first evaluation of LotuS3, RDP v2.14, and GreenGenes2 in combination, plus genus-level detection/quantification analysis beyond aggregate metrics.

Scientific Quality

Systematic design, transparent metrics, honest caveat discovery (DECIPHER L2 artifact), code/data release. Weaknesses: tiny sample count, low mock complexity, one primer pair, incomplete combination space, Metaxa2-favoring E. coli-Shigella merge. L'Oréal funding disclosed as non-influential but is a monitorable sponsor context.

Study Generality

Directly useful for any 16S user, but conclusions rest on three simple mocks with one species per genus; generalization to natural communities, other primers, and other platforms is not demonstrated.

Study Usefulness

Gives concrete, current pipeline recommendations (V1-V3 + DADA2+MMseqs2+SILVA or LotuS3+DADA2+RDP) and warns against blind trust in metrics, directly guiding experimental design and tool choice.

Study Reproducibility

Public data (PRJNA674596), scripts on GitHub and Zenodo, versioned tools and databases reported in detail; independent reexecution is straightforward.

Explanatory Depth

Descriptive benchmarking with post-hoc explanations (GG2 V4-sequence dilution of k-mers, primer bias, copy-number variation) that are plausible but largely untested mechanistically.