Review Papers Claim by Claim

What the paper does

• Defines a new hierarchical scheme of 44 molecularly coherent functional categories for phage proteins rederived from PHROG terms and organized parent to child so specific models are trained inside general parents Empathi functional category design
• Computes ProtTrans 1024 dimension embeddings for 904k deduplicated proteins from 18.5k INPHARED genomes and trains binary SVM RBF classifiers (one per category) on clustered non overlapping MMseqs2 clusters (198k clusters) to reduce data leakage Dataset and modelling setup
• Demonstrates extensive annotation expansion across metavirome databases (EnVhogDB and EFAM) and claims superior sensitivity over homology and prior embedding-based tools (VPF-PLM) while maintaining precision at chosen confidence thresholds Annotation expansion results

Performance claims and robustness

Reported model-level metrics are strong: most binary models achieve F1 >= 88% and 75% of models reach F1 >= 95% (collar model removed for low data). Holdout robustness where entire PHROGs and their similar PHROGs are removed from training still yields >=75% F1 in 51/60 replicates, interpreted as generalization beyond detectable sequence homology Model metrics and robustness

Critical appraisal and limitations

1. Dependence on PHROG-derived labels Empathi training labels originate from PHROG pHMM matches; authors acknowledge these are secondary, in silico annotations not experimentally validated and can propagate PHROG biases into models. This is an important caveat because embedding models will learn any label biases present in PHROG assignments PHROG label dependence and caution
2. Possible circularity and hidden homology Although MMseqs2 clustering and PHROG-based holdouts reduce direct leakage, remote homologs undetectable by sequence searches could remain in both train and test sets; reported holdout experiments removed PHROGs and similar PHROGs but this relies on PHROG similarity measures which themselves can miss deep relations. The paper notes this limitation and suggests PLM fine tuning on metagenomic viral proteins as future work Hidden remote homology and PLM fine tuning
3. Evaluation balance precision vs sensitivity Many reported gains (e.g., doubling EnVhogDB annotation from 16% to 33%) depend on stringent confidence thresholds (95%); lower thresholds increase sensitivity but risk false positives. The authors present full precision sensitivity curves and stress the user choice tradeoff, which is appropriate, but downstream users must treat low confidence predictions cautiously Confidence threshold tradeoff
4. Lack of experimental validation Apart from leveraging proteomic-derived PVP annotations for sensitivity checks, the manuscript does not provide new wet lab validation of novel predictions; the authors acknowledge that many labels remain in silico and that experimental work is needed to fully validate novel functional assignments Need for experimental validation
5. Possible class imbalance and small class limits Authors note some categories have limited examples (collar proteins) leading to model removal; for rare functions, embedding-based classifiers may underperform or overfit unless augmented with additional curated examples or domain-specific fine-tuning Class imbalance and collar model

Reproducibility and data availability

The authors provide code and precomputed embeddings plus predictions via a HuggingFace repository and a Zenodo dataset (3.7 GB embeddings dedup pickle) that enables reproduction and reannotation of new datasets; an Apptainer image and PyPI package are also available, which strongly supports reproducibility and practical reuse Empathi data and embeddings on Zenodo

EnVhogDB and EFAM resources used for benchmarking are publicly accessible which allows independent reanalysis of claimed annotation expansion Benchmark datasets used

How to strengthen and extend Empathi

• Fine tune ProtTrans or newer PLMs (ProtT5 variants/E SM3) on metavirome proteins to better represent viral sequence diversity and reduce domain shift; retrain classifiers after fine tuning (authors similarly suggest PLM fine tuning) PLM fine tuning suggestion
• Introduce calibration/evaluation on independent experimental datasets (e.g., metaproteomics verified PVP sets or curated enzymatic assays) to estimate real-world false positive rates across confidence cutoffs.
• Make model explainability outputs available such as embedding saliency, nearest neighbor exemplars in embedding space, or motif/attention maps to help users interpret predictions and triage novel assignments.
• Develop a continuous retraining pipeline that automatically incorporates newly annotated phage genomes and experimentally validated sequences to limit model drift.

Key numeric summary (extracted from paper and datasets)

Run further analyses or evolve this review

Final assessment summary (concise)

Empathi represents a significant, reproducible advance in scalable phage protein annotation by using PLM embeddings and task specific hierarchical binary classifiers, substantially increasing annotation coverage in metavirome resources while providing confidence-scored predictions; its main limitations are the reliance on PHROG-derived training labels and the need for experimental validation and PLM fine-tuning to fully close domain shift gaps.

Author reviews