Review Papers Claim by Claim

Concise critique

The paper proposes a sampling based divisive hierarchical spectral clustering pipeline and two stability metrics (normalized Misclassification Error Distance MED and Cluster Misclassification Error Rate CMER) to identify replicable clusterings in UMI count single cell RNAseq data; it emphasizes kNN inverse-distance affinities, outlier exclusion by kNN distances, and mapping a binary spectral clustering tree to nested clusterings to select stable partitions across samples Paper main method and abstract

Bottom line

• Clear, pragmatic pipeline with helpful stability metrics and reproducible code availability claim (GitHub) but sensitivity to preprocessing hyperparameters (kNN k, outlier thresholds, gene selection) is a key limitation and can materially change which clusterings are declared stable Paper limitations summary

Detailed review and critique of Finding stable clusterings of single-cell RNA-seq data

1) Paper summary (what they did)

The authors describe a sampling based, divisive hierarchical spectral clustering pipeline for UMI single-cell RNAseq data. The processing pipeline follows canonical steps—filtering, selection of highly variable genes via Pearson residual metrics, dimensionality reduction using SVD with rank estimated by optht/Gavish Donoho, outlier removal by examining kNN distance distributions (k up to 64), and divisive spectral clustering using Ng Jordan Weiss applied recursively to build a binary clustering tree. They map tree nodes to nested clusterings using a branch length measure H (an F statistic analogue) and evaluate clustering stability by two sampling-based metrics: the normalized Misclassification Error Distance (MED) and per-cluster Cluster Misclassification Error Rate (CMER) computed across multiple random half-samples (and normalized against random labelings) Paper abstract and method outline

2) Strengths

• Stability-focused framework: Explicitly quantifying clustering replicability using MED and CMER across independent random samples addresses an important reproducibility gap in scRNA-seq analysis where many methods produce unstable results depending on stochastic seeds and hyperparameters Stability evaluation rationale
• Practical pipeline and code availability: The authors applied the pipeline to seven public datasets (small synthetic mixes and large 10x datasets such as PBMC68k, retinal, lung, breast cancer) and provide a GitHub repository that contains scripts and RDS files for reproducibility which increases transparency and reuse potential Code availability statement
• Large scale tests: The manuscript examines both small ground truth mixes (Zhengmix4eq/8eq) and multiple large real datasets including the 68k PBMC and 100k breast datasets, allowing evaluation of how stability behaves with scale and biological complexity Data sets used

3) Weaknesses, limitations, and critical issues

1. Sensitivity to preprocessing hyperparameters: The approach depends on multiple, influential choices—definition of analysis genes (top 2000 by Pearson residual metrics across full data and per-sample), SVD rank estimation, the k parameter for kNN affinities (authors typically used k=64), and the outlier threshold (3 SDs beyond mean kNN distance). The paper acknowledges dependency but provides limited principled guidance for choosing these values; small changes can change MED/CMER distributions and the clustering declared «stable» kNN parameter dependence
2. Potential removal of biologically meaningful rare cells: Their outlier exclusion rule (kNN distances > mean+3SD across multiple k) may exclude real rare subpopulations whose local density legitimately differs; the authors note they may 'err on the side of excluding too many cells' to favor stability—this is an explicit tradeoff (stability vs sensitivity) that risks discarding rare but meaningful biology Outlier removal tradeoffs
3. Metric choices have limited external biological validation: MED and CMER quantify reproducibility across random half-samples—useful for technical stability—but biological validity still requires orthogonal checks (differential expression, marker overlap, external annotation). The authors recommend DE as follow-up but present limited systematic evaluation showing that low MED/CMER correlate with biologically meaningful marker separation across all datasets. Thus stability may not equal biological correctness Differential expression as biological corroboration
4. Computational cost and scalability caveats: Calculating all pairwise distances (for kNN distributions and outlier detection) and multiple sampling repeats is computationally heavy for datasets with 50k–100k cells; the authors note run times can be large though they used sampling and choose sparse kNN affinities to reduce cost. The paper lacks a quantitative runtime/memory benchmark against modern scalable alternatives (e.g., Leiden/Leiden-sparse graphs, approximate nearest neighbors, scalable spectral approximations) Computational cost note
5. Limited comparison to alternate stability/consensus frameworks: Ensemble and consensus clustering approaches (SAFE-clustering, scEVE, scICE, etc.) and recent GNN and contrastive methods are active; the paper cites related work but does not compare head-to-head using common benchmarks (ARI/NMI, run time, stability metrics) which would clarify practical advantages and tradeoffs Comparison limitations

4) Reproducibility and data/code availability

The authors state that data sources are public (Bioconductor DuoClustering2018 mixes, 10x PBMC68k, and other literatures) and point to their GitHub repo containing pipelines and RDS files for each dataset Data and code availability

Practical reproducibility will depend on whether the repository includes versioned environments, parameter configs, and example outputs; the paper mentions scripts and RDS files but the reviewer recommends a containerized workflow (Docker/Singularity) or pinned conda environment to guarantee replicability on diverse compute platforms.

5) Concrete suggestions to improve the work

• Provide a sensitivity analysis showing how MED/CMER and final chosen stable clusterings change as you vary key hyperparameters (kNN k, number of HVGs, outlier SD threshold, SVD rank perturbations). This would quantify robustness and guide users.
• Include systematic biological validation: compute marker gene separation, ARI/NMI versus published annotations where available, and show whether low MED/CMER correlates with higher biological interpretability across datasets.
• Benchmark runtime and memory against scalable alternatives (approximate kNN libraries such as Faiss/Annoy/HNSW, Leiden on sparse graphs, scEVE/scICE ensemble methods) and provide recommendations for large (>50k cell) data.
• Replace full pairwise distance calculations by approximate nearest neighbor-search for the kNN distance distribution and evaluate whether outlier calls are preserved.
• Provide options for soft cluster assignments or probabilistic measures to avoid throwing away rare subpopulations by outlier filtering—e.g., flag isolated points but forbid deletion unless supported by multiple criteria (marker absence, low UMI counts, batch-specificity).

6) Specific technical comments and blindspots

• The use of inverse-distance affinities restricted to kNN is reasonable, but the symmetry enforcement and effect of non-uniform kNN distances (variation across three orders of magnitude in one dataset) can distort spectral embeddings; authors note this but do not provide a remedial normalization (e.g., mutual kNN graphs or adaptive kernel width) kNN distance variation
• The paper estimates SVD rank via optht/Gavish Donoho; this is a defensible choice yet rank estimation in noisy sparse count matrices can be unstable—adding cross-validated reconstruction or permutation tests would strengthen rank choices.
• Normalizing MED against random labelings is sensible, but details of the randomization procedure, number of repeats, and confidence intervals for MED/CMER should be explicit to enable statistical interpretation (e.g., p-values or bootstrap CIs).

7) Practical takeaways for practitioners

1. Use MED and CMER to quantify reproducibility when comparing clusterings across samples or preprocessing settings; they provide objective metrics beyond visual inspection of UMAP/tSNE.
2. Be cautious: stable clusters per MED/CMER are not automatically biologically correct—confirm with marker genes or DE analysis across samples as the authors recommend DE as corroboration
3. When applying the pipeline to large datasets, prefer approximate kNN and profile runtime; consider avoiding overly aggressive outlier deletion if rare cell discovery is a goal.

8) Confidence and final assessment

Overall the paper makes a useful methodological contribution by formalizing sampling-based stability metrics and proposing a practical divisive spectral approach with nested clustering mapping. The approach is moderately novel and useful, particularly for teams seeking to prioritize reproducible clusterings. Its limitations largely concern sensitivity to preprocessing choices, potential loss of rare biology via outlier filtering, and missing head-to-head benchmarking versus alternative stability or ensemble methods.

Key citations

• Paper main method and abstract