Check Results Against Sources

Concise plan to validate Phold predictions against HMM and MSA methods

• Assemble benchmark sets: INPHARED1419 proteins and representative EnVhog/EFAM subsets from Phold and Empathi datasets Phold framework and datasets and Empathi embeddings and outputs.
• Compute baseline HMM annotations (PHROG/HHblits/HMMER/HHsearch), build MSAs (MMseqs2/ColabFoldv) and run Empathi embeddings to compare calls at matched confidence thresholds Empathi thresholds and outputs.
• Evaluate per-protein and per-cluster concordance (precision/recall/F1), disagreement types (general versus specific label mismatch), and length/MSA depth/pLDDT stratified performance with visualizations.

Goal

Produce a reproducible, quantitative validation comparing Phold structure-based functional calls to HMM/profile-based annotations and MSA/embedding-based classifiers (Empathi) to identify agreement, systematic disagreements, and the biological/technical drivers of differences.

Key motivation and sources

• Phold uses ProstT5 3Di tokens plus Foldseek against a 1.36M curated phage-structure DB and reports markedly higher annotation sensitivity than sequence-only methods Phold framework and datasets.
• Empathi is an embedding-based hierarchical classifier built from ProtT5 embeddings that doubles annotations in EnVhogDB and supplies embeddings and prediction CSVs for direct comparison Empathi embeddings and outputs.
• Phynteny integrates synteny/context to further annotate genes and has been benchmarked against Phold, suggesting complementary strengths of structure, synteny and embedding/HMM methods Phynteny synteny-model.

Datasets to use (concrete)

1. INPHARED 1419 benchmark set used in Phold: genomes, GenBank outputs, and Phold predictions (Zenodo record with ColabFold structures and phold outputs) INPHARED1419 phold benchmarking data.
2. Phold Search DB 1.36M structures (Foldseek-formatted DBs and predicted structures) to replicate Foldseek hits and thresholds Phold 1.36M structure DB.
3. Empathi public dataset package with embeddings and predictions (Zenodo HuggingFace) to compare embedding-based labels and confidence thresholds Empathi datasets on Zenodo.
4. enVhog database and HMMs for PHROG-based calls and mapping cluster IDs (envhog resource) enVhog data tables.

High-level analysis workflow

Step A: Data assembly and harmonization

• Download INPHARED1419 phold GenBank and phold_per_cds_predictions.tsv from Zenodo and Phold GitHub; parse CDS IDs, sequence, phold label, prostT5 mean confidence, Foldseek top-hit Evalue/TM-score and per-residue masking metrics Phold outputs and metrics.
• Obtain Empathi embeddings and Empathi predictions CSVs (same proteins or cluster representatives) to align indices; ensure mapping between PHROG IDs, enVhog IDs, and protein sequence identifiers for cross-method comparisons Empathi data artifacts.
• Generate or fetch MSAs and profile HMM hits: run MMseqs2/HHblits per protein or cluster to produce HMM-based annotations (PHROG pHMM hits via HHsearch/HMMER). Record top pHMM hit, E-value, coverage, and PHROG label. Use same PHROG database version cited in Phold/Empathi to avoid label drift PHROG pHMM baselines.

Step B: Define comparison tasks and thresholds

• Binary agreement: same top-level PHROG/Empathi/Phold category at high confidence thresholds (match vs mismatch).
• Multi-label concordance: allow Phold to propose multiple functions and check overlap (e.g., baseplate vs tail appendage confusion noted in Empathi/Phold papers) with taxonomy of disagreement (biologically plausible vs erroneous) Empathi notes on biologically similar label errors.
• Thresholds: use Phold Foldseek evalue/TMscore and ProstT5 mean confidence masking (e.g., Phold used default evalue 1e-3 and prostT5 mask threshold; Empathi used confidence cutoffs 0.95/0.5) and evaluate performance across deciles.

Step C: Metrics and statistical tests

• Per-protein precision, recall, F1 at multiple thresholds for Phold, HMM, and Empathi.
• Cluster-level concordance: fraction of clusters where all member proteins share identical predictions vs mixed predictions (Empathi reported low mixed-cluster fraction) Empathi cluster-level concordance.
• Stratified analyses: by protein length, MSA depth (neff), ProstT5 mean confidence, pLDDT (ColabFold pLDDT where available), and taxonomy (phage family) to find systematic biases (e.g., short proteins often uncharacterized by structure alone) Phynteny short protein limitation.
• Statistical tests: McNemar test for paired binary disagreements, bootstrap confidence intervals for F1, and permutation/randomization controls to test whether agreement exceeds that expected by cluster-level composition (use random shuffles preserving cluster sizes) Phold-analysis randomization and ablation scripts.

Step D: Disagreement triage and manual curation

• Classify disagreements into categories: (1) HMM high-confidence vs Phold low-confidence; (2) Phold high-confidence vs HMM negative; (3) Empathi assigns multi-function where others single; (4) biologically plausible swaps (e.g., tail-baseplate) vs contradictory enzymatic assignments.
• For a representative sample of disagreements (n=200 balanced across families and length bins), do manual inspection: alignment, examine Foldseek top hits (TMscore, coverage), visualize structures with 3Dmol for structural alignment, check MSA neff and HHsearch alignment quality, and consult PHROG/uniProt annotations where available.

Step E: Visualization and reporting

1. Interactive plots (Plotly): precision-recall curves per method; F1 vs confidence thresholds; heatmap of per-category agreement rates; violin plots of annotation rate vs protein length and MSA depth.
2. Sankey diagrams to show flows from HMM label -> Phold label -> Empathi label for the full dataset (use cluster-level aggregation first to reduce node count) Phold Sankey analysis code.
3. Embed representative 3Dmol viewers for paired structural comparisons of disputed predictions.

Practical computational plan and reproducible pipeline

Use Snakemake or Nextflow to orchestrate steps; containerize with Apptainer/Docker using Phold, Empathi, MMseqs2, HH-suite, Foldseek, and ColabFoldv as components. Key Python libraries: biopython, pandas, numpy, scikit-learn, plotly, seaborn; use phold and empathi packages where available to reuse provided outputs and reduce compute.

Resource and runtime estimates

• Phold prostT5 prediction step is GPU-accelerated; Phold reports single GPU runtime for INPHARED 182 of ~3942 seconds for ProstT5 predictions and ProstT5 is ~50-100x faster per 3Di token than ESMFold token generation, so use ProstT5 predictions where possible to scale Phold runtime and ProstT5 speed.

Concrete reproducible checklist and deliverables

1. Download required Zenodo records and Empathi HuggingFace model files. (Links in citations above.)
2. Make mapping table protein_id <-> enVhog_id <-> PHROG_id to harmonize names.
3. Run HMM-based annotation pipeline: build HHblits profiles and HHsearch/ HMMER annotation vs PHROG pHMMs; save per-protein best hit metrics.
4. Fetch Empathi predictions and ProtT5 embeddings for same protein set; if missing, compute ProtT5 embeddings for proteins using provided Empathi container or HuggingFace weights.
5. Extract Phold phold_per_cds_predictions.tsv and ProstT5 confidence, Foldseek top-hit metrics for same proteins.
6. Compute metrics (precision/recall/F1/AUPR) across thresholds; save results to a reproducible report (notebook + HTML outputs).
7. Create figures: PR curves, F1 vs threshold, Sankey of label flows, violin plots stratified by length/MSA neff/pLDDT; embed interactive 3D viewers for curated disagreements.
8. Produce an annotation concordance table at cluster-level and per-category confusion matrices and a triage spreadsheet for curated disagreements including links to structure files and MSA views.

Potential pitfalls, limitations and how to control for them

• Label bias from PHROG: many methods use PHROG labels as ground truth; to avoid circularity, run holdout experiments where clusters used for training Empathi/PHROG derivation are removed when comparing Phold (Phold authors also did similar holdouts) Empathi robustness holdouts.
• Functional label granularity: structural similarity may map to broader molecular functions while HMM hits may be more specific; define hierarchical mapping to compare at both coarse and fine-grained levels.
• Short proteins and low-confidence structures: structure-based methods may struggle with small proteins; stratify analyses by length and pLDDT and consider combining Phold + Phynteny for short proteins Phynteny complementarity for short proteins.
• Annotation propagation errors: when Phold maps to PHROG-labeled structural homologs, confirm that the propagated PHROG label is not itself a predicted/uncertain label; flag cases where PHROG annotations are hypothetical.

Practical example commands and code fragments

Use phold and empathi containers where possible. Example high-level commands (to be wrapped in Snakemake):

# 1 fetch phold predictions (assumes phold installed) phold compare -i inphared_gbk -o phold_out -t 16 --foldseek_gpu
# 2 run HHblits/HHsearch against PHROG pHMMs for HMM baseline hhblits -i protein.fasta -d phrog_db -o protein.hhr
# 3 collect Empathi predictions (or compute embeddings) empathi input_embeddings.pkl name --confidence 0.95

Visualization plan (Plotly + 3Dmol)

• PR curves and threshold sweeps (Plotly interactive) per method and per PHROG category.
• Sankey: HMM -> Phold -> Empathi flows at PHROG-category level (aggregate low-count nodes).
• Scatter matrix: ProstT5 confidence vs ColabFold pLDDT vs MSA neff colored by agreement/mismatch.
• Interactive 3Dmol panels for curated disagreements, loading Phold top hit PDB and Phold query structure for side-by-side comparisons (3Dmol viewer supports direct PDB loading) Phold plot and 3D visualization.

Deliverables and reproducibility

• An open Snakemake/Nextflow pipeline that: downloads Zenodo assets, harmonizes identifiers, runs HMM/MMseqs2/HHsearch, collects Phold and Empathi outputs, computes metrics, creates interactive plots, and exports a triage spreadsheet for curated disagreements.
• A Jupyter/Colab notebook that reproduces key figures and includes links to Zenodo assets and GitHub code, using provided Phold and Empathi containers to ensure portability.

What would disprove Phold claims in this validation?

If HMM/profile-based annotations (PHROG pHMMs) consistently and significantly outperform Phold on an independent holdout of deeply novel proteins (not represented in Phold DB or PHROG label set) at matched error rates, or if Phold high-confidence predictions have high false-positive rates when compared to experimental labels or well-curated UniProt/SwissProt annotations, then Phold's claims of greater sensitivity without sacrificing precision would be challenged Phold falsifiability conditions.

Immediate next steps for you

1. Confirm compute availability (GPU for ProstT5; CPU+Foldseek for comparisons).
2. Decide evaluation scope: full INPHARED1419 (recommended) OR representative stratified sample (faster).
3. Authorize pipeline run: fetch Zenodo records (Phold DB and INPHARED files) and Empathi embeddings; I can prepare a runnable Snakemake workflow and a prefilled Colab notebook.