The Evidence Layer for Scientific AI

 Quick Analysis Plan Copied

 Long Analysis Plan

Evidence baseline: what ProTokens has already shown

Reported reconstruction across held-out validation sets is high: median rTM-score 0.97 on RCSB/CAMEO (n=513), 0.98 on CASP14 (n=87), 0.97 on CASP15 (n=45), and 0.98 on 33,842 AFDB dark clusters; token-space Spearman correlation with local environments is 0.62, and PT-score tracks TM-score at r≈0.94 [1,2]. Known blindspots: ProToken degeneracy (many-to-one token maps), training bias toward well-characterized folds, and reliance on AlphaFold2/ESMFold embeddings [2]. OmniScore provides an external sanity check: ProTokens rank behind OmniScore-84M (AUROC 71.76%) and AminoAseed (72.43%) on StructTokenBench functional-site classification [3]. These numbers anchor the ablation targets below.

Cross-fold generalization ablation design

A1 — Split logic (the critical confound). Random splits inflate generalization estimates because remote homologs straddle train/test. Build folds via Foldseek/PT-score clustering at TM-score thresholds 0.5/0.3/0.2, holding out entire fold-clusters (cluster-holdout), plus a random-split arm as the optimistic ceiling. Use the 33,842 AFDB dark clusters as the pool so held-out folds are genuinely underrepresented in training [1,2].
A2 — Ablated components. Retrain the Distiller (VQ-VAE) with (a) full model; (b) SE(3)-invariant encoder removed; (c) alignment/uniformity regularization losses removed; (d) codebook size halved/doubled; (e) FAPE loss swapped for RMSD-only; (f) sequence-embedding branch (PCA-reduced AF2/ESM) ablated from the twin-tower latent [1,2].
A3 — Readouts per fold-holdout. Reconstruction (median rTM, rLDDT>0.90 fraction), latent quality (token-vs-environment Spearman; PT-score↔TM-score correlation), downstream: PT-DiT de novo scTM-scores (baseline 0.82–0.94) and the 12-DMS directed-evolution benchmark (baseline top-1: unified 75%, PT-DiT-only 33%, ESM2-15B 9%) [1,2].
A4 — External comparator. Evaluate frozen ablated embeddings through StructTokenBench adapters alongside OmniScore-84M (71.76% AUROC) to detect whether fold-holdout erodes functional-site signal below the OmniScore floor [3].

Analysis plan and falsification

Report mean±SD over ≥5 cluster-holdout folds; use paired bootstrap across folds; pre-register the falsification criterion: if median rTM falls below 0.7 on held-out novel folds, or unified-embedding top-1 on DMS drops to ESM2-15B-level (≤9%), cross-fold generalization claim fails [2]. Known degeneracy means token-level metrics should be complemented with decoder-based rTM to avoid penalizing many-to-one mappings spuriously [2].

Missing controls to flag: the source papers report no random-vs-cluster split comparison, no variance over folds, and no fully novel-fold test — this ablation plan fills exactly that gap (BGPT inference from reported limitations) [2,3].

Analyze Ablation Power Build Ablation Matrix Run AI Scientist Analysis