See what computational analyses actually reported, with provenance for every claim.Know what the science actually supports before you trust the answer.
Press Enter β΅ to search claims
Explore by Goal
"Science is organized knowledge. Wisdom is organized life."
- Immanuel Kant
Quick Analysis Plan
Copied
Cluster stability workflow using ARI/NMI (agreement with reference labels and/or across resamples), plus stability-matrix/bootstrapping logic (BASC) for replication-aware stability. See the full plan below with Plotly summaries of reported ARI/NMI improvements from raw full-text sources.
Your goal is to quantify whether clusterings are reproducible under resampling / perturbations, and to summarize stability using ARI and NMI (and optionally bootstrap-ensemble stability matrices).
1) What ARI & NMI are measuring (make the βstabilityβ definition explicit)
ARI: measures agreement between two partitions adjusted for chance; it is appropriate when you can compare a clustering to a reference or between two runs with aligned cluster labels.
NMI: measures mutual information between cluster assignments (often robust to some label permutations) and is also used to compare partitions.
In practice, βstabilityβ can mean at least two different things:
Reference stability: clustering is consistent with known labels/ground truth (when available). ARI/NMI are then direct agreement metrics.
Replication stability: clustering assignments agree across resampled subsets / perturbations (no ground truth required). Here, ARI/NMI are computed pairwise between clusterings from resamples.
Cluster quality metrics such as ARI and NMI are commonly reported for clustering performance in single-cell and chromatin-contact settings (e.g., GNODEVAE uses ARI/NMI in large multi-dataset benchmarking; Hi-Cformer uses NMI/ARI to evaluate clustering after embedding/imputation).
2) Stability strategy that works with/without ground truth
A. If you have ground truth labels
Split into resamples: bootstrap or subsample cells (or samples) using a resampling scheme consistent with your data dependencies.
Run the full clustering pipeline on each resample with identical hyperparameters.
Compute ARI/NMI per resample versus reference labels.
Summarize distributions: report mean/median + confidence intervals (e.g., bootstrap CI over resamples), not only a single point estimate.
Inspect failure modes: e.g., which clusters destabilize first; whether instability concentrates in rare subpopulations (important in single-cell).
B. If you do NOT have ground truth labels
Generate clustering ensemble: multiple runs on resampled/perturbed data.
Pairwise compare partitions using ARI/NMI between resample clusterings (pairwise ARI/NMI matrix).
Cluster stability matrix: convert pairwise similarity into a stability matrix; optionally use an explicit bootstrap framework like BASC to quantify stability from ensemble replications and choose stable numbers of clusters.
Summarize stability:
Heatmap of pairwise ARI/NMI stability.
Histogram/violin of ARI and NMI across pairs.
Per-cluster stability (if you can align clusters across runs).
BASC (βbootstrap analysis of stable clustersβ) constructs a stability matrix from multiple clustering replications, uses bootstrap schemes appropriate for time series dependencies, and applies hierarchical clustering on the stability matrix to identify robust group-level clusters.
C. Particular warning: stability depends on preprocessing/graph construction
Stability metrics can change when you vary neighbor graph parameters, outlier removal, or the choice of how you subselect highly variable genes.
3) Visual: reported ARI/NMI improvements (useful sanity anchor for your interpretation)
These are not stability estimates from your data; they show how ARI/NMI are used as clustering quality scores in large full-text benchmarks.
Cell-by-feature matrix (or embedding) and any graph/neighbor definition parameters.
Clustering method + its hyperparameters (store all configs).
Labels for reference comparison (optional; only if you have ground truth).
Resampling scheme (bootstrap/subsample) and number of resamples B.
Distance metric/affinity graph settings (kNN, outlier removal), because stability can be sensitive to them.
Step-by-step pipeline
Set up configuration registry: single source of truth for preprocessing + clustering + resampling settings (for reproducibility audits).
Generate resamples:
For i.i.d. observations: bootstrap with replacement.
For time- or batch-dependent structures: use a block/bootstrap logic appropriate to your dependencies (BASC uses circular block bootstrap for fMRI time series).
Run clustering for each resample using the full pipeline (do not compute embeddings once and reuse if your pipeline is supposed to reflect uncertainty).
Compute ARI/NMI pairwise between all resampled clusterings:
Store a pairwise matrix of ARI values and a pairwise matrix of NMI values.
Distribution plot (hist/violin) of ARI/NMI across pairs or across resamples vs ground truth.
Cluster-number sensitivity: repeat for multiple K values if your method supports K (BASC includes explicit cluster-number selection logic from stability contrast).
Hard checks (skeptical):
Verify that βstabilityβ is not coming from trivial solutions (e.g., always predicting one dominant cluster).
Check whether instability concentrates in rare types/outlier-prone regions (ties to the scRNA stability paper warning).
Ensure you havenβt accidentally leaked label information into preprocessing.
For stability estimation frameworks, BASC is an example of explicitly quantifying stability from bootstrap ensembles via a stability matrix and hierarchical processing.
For single-cell, stability depends on neighborhood/outlier/preprocessing choices; therefore, your stability conclusions should be conditional on your pipeline.
Resample agreement vs ground truth (if available) with ARI/NMI distributions.
Cluster-number vs stability curve (if you sweep K), using stability contrast logic akin to BASC.
Stability-annotated cluster memberships (identify which clusters are consistently recovered).
5) Limitations & blind spots to actively guard against
Metric β biological validity: ARI/NMI measure agreement, not whether clusters correspond to meaningful cell states. You still need biological validation.
Preprocessing sensitivity: stability can change with kNN affinity, outlier removal, and feature selectionβso document pipeline exactly.
Chance agreement & label alignment: ARI is chance-adjusted, but NMI can still be affected by cluster granularity; ensure consistent K across comparisons or use appropriate alignment strategies.
Dataset dependence: stability may not generalize; single-cell stability has been reported as dataset-dependent in multi-dataset evaluations.
6) Click-to-run: AI Scientist agent (iterative clustering stability computation)
If you upload your clustering outputs (labels per run) and/or the underlying embeddings, the agent can compute pairwise ARI/NMI matrices, stability distributions, and (if requested) BASC-style stability-matrix logic.
Optional next links (bespoke queries to go deeper)
It computes pairwise ARI/NMI across clustering runs from bootstrap resamples, then plots stability heatmaps and distributions; finally, it summarizes reference ARI/NMI if ground-truth labels exist.
Get emailed when your analysis is done!
We'll email you the results when your analysis is finished.
Hypothesis Graveyard
A single scalar βstability scoreβ computed from one random split will generalize across datasets and preprocessing pipelines; likely wrong because stability is reported as dataset- and pipeline-dependent in multi-dataset single-cell analyses.
If ARI is high, NMI must be high (and vice versa) with equal meaning; likely too strong because they weight different aspects of partition similarity (granularity vs information content), especially across different K values.
Explore by Goal
Quick Analysis Plan
Long Analysis Plan
Top Data Sources
Ask a Follow-Up
Key Insight
Keep Exploring
Analysis 
Hypothesis Graveyard
Potential Experiments
Discussion