ModelDetector pipeline

1. Input: a protein multiple-sequence alignment (MSA). Inputs: protein MSAs
2. Summary-statistics engineering:
   • Pairwise extraction: uses amino-acid pair substitutions to populate a 20×20 frequency matrix F2, then derives 400 relative-rate-change features for CNN input. Pairwise summary statistics (F2 -> 400 features)
   • Triplet extraction: adds a second frequency matrix F3 from three-sequence substitutions via an inferred “common ancestor” (parsimony-based), again resulting in additional 400 features (total 800 when combined). Triplet summary statistics (F3 -> +400 features)
3. Classifier: ResNet-18. ResNet-18 used as CNN backbone ResNet foundation paper
4. Training labels: the “true” substitution model is known for simulated alignments generated by AliSim. AliSim simulator supports model-conditioned alignment generation Labels come from simulator ground truth

Scientific distinction (known vs inferred): In this work, the mapping from summary stats → model class is inferred by supervised learning; correctness is demonstrated only on the simulated-data generating family. Generalization to real protein evolution regimes that violate simulation assumptions is still an open question. Known ground truth only in simulations

Model set (9 classes):

• Clade-specific: Q.plant, Q.bird, Q.yeast, Q.mammal, Q.insect (5)
• General: Q.pfam, LG, WAG, JTT (4)

These model families (e.g., JTT, WAG, LG) are classical empirical AA substitution models, while Q.pfam and clade-specific models are estimated via methods like QMaker (as referenced in the paper). JTT established empirical AA model WAG general empirical AA model LG general empirical AA model QMaker supports clade-specific models

Training data construction (simulation from real alignments)

• Real alignment sources: 1000 HSSP alignments per clade (plants/birds/yeast/mammal/insect) are sampled; each is required to have at least 50 variant sites. Real clade-specific alignment selection HSSP real alignment context HSSP database
• Simulator: AliSim generates new alignments using trees and site-rate parameters estimated from the real alignments; the study includes site rate heterogeneity (gamma + invariant) via simulator options. AliSim simulation configuration (+G,+I; parameters from real alignments) AliSim overview

Key implication: the deep model is effectively trained to invert a particular simulator+estimation pipeline, which may or may not match real complexities. Simulator-realism limitation

Reported average test accuracies (classification correctness): pModelDetector 96.78%, ModelDetector 97.45%, ModelFinder 97.88%. Average accuracy comparison

Runtime comparison for 50 taxa and varying alignment lengths (includes summary-statistics creation + prediction for DL). Table 3 runtime numbers

The paper reports high correlations between summary statistics computed from simulations and those computed from the real alignments they used for simulation parameterization. For example, it reports average correlation of F2 matrices ranging from 0.88 (yeast) to 0.94 (mammal), and >95% of alignments have correlation >0.8. Reported summary-statistics correlation checks

Skeptical interpretation: High correlations suggest that the specific simulator-derived summary statistics align well with statistics from the alignments used to parameterize simulation trees. But that does not guarantee that the learned mapping will hold under other sources of mismatch (e.g., different among-site rate distributions, non-stationarity, mixture models, alignment errors, compositional heterogeneity). The paper itself highlights simulator-realism concerns and the lack of real ground-truth labels. Correlation checks and generalization caveats

Major validity threats (what could break the method)

• Ground-truth supervision is simulation-only: the paper explicitly trains and tests on simulated alignments because the true underlying model of real alignments is unknown. This creates a “match the simulator” risk rather than a “match nature” guarantee. Simulation-only ground truth limitation
• Model-mixture complexity excluded: the paper notes that mixture models are not included and that real alignments may have regions following different models. Excluded mixture models + real-alignment complexity
• Identifiability & information-criterion caveats: the paper motivates ML model selection issues and cites concerns about AIC/BIC behavior in phylogenetics. However, the evaluation compares to ModelFinder, whose own approximation strategy matters—so “DL vs ML” still depends on the proxy baseline being well-aligned to the scientific goal of selecting the correct evolutionary model. Caution about information criteria behavior ModelFinder method and approximation context BIC statistical basis
• Classes limited to 9 models: accuracy is reported only within a fixed 9-class set; if the true model is not among these, predictions may be systematically biased. Fixed 9-model closed set evaluation
• Correlations don’t establish correct likelihood: high correlation of summary statistics is encouraging but does not directly measure how well the inferred model matches the likelihood under the original generative process on out-of-distribution real MSAs. Correlation diagnostics vs final accuracy

What would disprove the main claim?

• On independent real protein MSAs with careful evaluation proxies (e.g., out-of-sample likelihood comparisons, posterior predictive checks), ModelDetector would need to consistently underperform or fail to match ModelFinder’s choices. Falsification path implied by simulation-only evaluation
• Performance collapse when alignment length is short (the paper reports accuracy ~<90% for 100-site alignments). Short-alignment performance drop
• If mixture models or non-stationary regimes are common in real datasets, the closed-set classifier may systematically misclassify the nearest “single model” class. Mixture-model blind spot