Review papers with raw data transparency

Visual, evidence‑first paper analysis — Shi et al., 2022 (Microbiome)

Visualizations below reproduce key numeric summaries from the paper and demonstrate the mechanistic points the authors make about Bray–Curtis and unweighted UniFrac failure modes. Every factual claim below is tied to the original paper (Shi et al. 2022) via inline citation.

Dataset summary — high‑abundance OTU signal (paper-extracted numbers)

Data values are taken directly from the paper's Table 1 / extracted data summary and illustrate the very low summed high‑abundance signal in the Schnorr dataset (0.058), which the authors link to Bray–Curtis underperformance in that dataset.
Shi et al. dataset summary

Adjusted Rand index (high-level reproduction) — methods vs datasets

This plotted pattern reproduces the core qualitative results: UU does very well in several geographically separated datasets (including perfect for Schnorr) but fails on Smits (seasonal, many shared low‑abundance OTUs); Bray–Curtis fails when high‑abundance signal is scarce (Schnorr) but is good when high‑abundance OTUs drive separation (Martínez). The combined metric shows consistent mid‑to‑high performance across datasets in the authors' results.
Shi et al. Rand index findings

Interpretation — mechanism & recommendations (evidence tied to paper)

• Why Bray–Curtis fails: BC uses raw counts/abundances and therefore is dominated by differences in high‑abundance OTUs; if a dataset lacks OTUs with high mean abundance, BC has low discriminating power — demonstrated for Schnorr and recovered by the authors when they merged distal phylogenetic tips (trimming) to increase tip mean abundance and observed improved BC Rand indices (then worsening after over‑trimming). Shi et al. Bray–Curtis trimming experiments
• Why unweighted UniFrac fails: UU is presence/absence and therefore sensitive to the total number of non-zero entries across samples; in seasonal Hadza (Smits), many low‑abundance taxa are shared across seasons and UU loses signal — the authors simulate converting low counts to zeros (thresholding) and show UU performance improves as total Shannon diversity declines (fewer non-zero entries). Shi et al. UU thresholding experiments
• Combined metric usefulness: A normalized linear mixture (α d_UU + (1−α) d_BC) with α = 0.5 gives consistently good Rand indices across tested datasets and often outperforms generalized UniFrac in their comparisons; code is provided for reproducible adoption. Shi et al. combined metric

Limitations, potential biases, and missing tests (critical)

1. Dataset choice: only five datasets (four geographic/seasonal + one clinical) — selection may bias observed patterns; broader set (different body sites, shotgun metagenomes, environmental microbiomes) could change conclusions.
2. Preprocessing dependencies: authors reprocessed 16S data using VSEARCH and UNOISE2; OTU‑level choices (OTU vs ASV, clustering thresholds) can alter the high/low‑abundance balance that drives metric behavior.
3. Two‑group focus: experiments assume two clusters (adjusted Rand index to ground truth); many real problems have continuous gradients, nested structure, or more than two groups where these findings may not directly generalize.
4. Trimming/binarizing are interventionist: while useful as sensitivity analyses, trimming branches or thresholding counts may remove biologically meaningful microdiversity — risk of overfitting to clustering metrics rather than preserving ecological signal.
5. Statistical uncertainty: although authors used simulations and repeated resampling for some perturbations, full parameter sensitivity (e.g., α tuning across many datasets) and external validation cohorts are limited; claims that α=0.5 is generally robust should be treated as provisional pending larger benchmarks.

All of these caveats are discussed by the authors; see the Availability/Methods and Supplement for their R package that enables re‑analysis. Shi et al. limitations & package

Actionable recommendations for practitioners

• Inspect dataset-level summaries (total Shannon, sum of high‑abundance OTU means). If high‑abundance signal is low, suspect Bray–Curtis to underperform; consider phylogenetic-aware metrics or binning strategies (Shi et al. suggestion).
• If many low‑abundance taxa are ubiquitous (high total Shannon driven by many nonzeros), unweighted UniFrac can be misled; either threshold small counts to zeros, use weighted/phylogenetic metrics, or combine UU with abundance information (combined metric suggested by authors).
• For clinical/weak‑signal problems where clusters are not well separated, test multiple metrics and consider the combined metric as a stable default; use package MicrobiomeCluster to reproduce authors’ combined metric and tune α on held‑out datasets (Shi et al. R package).

Primary source (all claims above are anchored to this paper): Shi et al. 2022 (Microbiome)