Review papers with raw data transparency

What the paper claims (and what is actually measured)

• Core modeling move: treat DTI as heterogeneous graph link prediction using a graph-transformer “embedding trunk” plus a link-prediction MLP. PIGLET architecture and link prediction head
• Representation move: build proteome-wide pocket similarity edges using HOTPocket/BioLiP2 pocket sets and ESM2 residue embeddings (cosine similarity threshold > 0.95). Pocket similarity construction
• Key evaluation claim: on the benchmark “Human” dataset, PIGLET is similar to SOTA on random splits but wins clearly on a drug-similarity split. Random vs drug-split AUROC
• Leakage/inductive-bias test (ablation): DrugBank-derived message-passing edges materially help on the drug-similarity split (AUROC 0.873 with vs 0.720 without), but not on random splits. DrugBank message-passing ablation

• On random splits, all models cluster near AUROC ~0.97–0.98, suggesting the task may be comparatively “easy” under that split protocol (consistent with the paper’s leakage concern). Random split AUROC clustering
• Under the drug-similarity split, performance degrades sharply for sequence/structure-based models (e.g., TransformerCPI test AUROC ~0.542; FragXsiteDTI ~0.531). PIGLET retains a substantially higher score (~0.873). Drug-similarity split degradation
• However, AUROC alone doesn’t guarantee calibration, nor does it reveal which chemical–structural factors drive the scores; AUROC can be sensitive to class balance and score monotonicity but not necessarily to error rates at decision thresholds. (The paper focuses on AUROC and does not report calibration metrics in the provided text.) AUROC-only evaluation emphasis

• The large AUROC drop on the drug-similarity split without DrugBank MP suggests that DrugBank edges add useful inductive bias in the “novel drug” regime. Drug-split sensitivity to DrugBank MP
• The fact that random split performance is ~unchanged is consistent with the leakage issue being already present (or with message passing edges not being the limiting factor under random splits). Random split insensitivity to DrugBank MP
• Blind spot: the provided text emphasizes avoiding leakage by using fingerprint-cluster medioids for DrugBank assignment; the robustness of this leakage mitigation depends on the specific clustering + similarity thresholds and the underlying relationship between “drug similarity” clusters and actual binding behavior. DrugBank leakage mitigation method

• Reported scale: full graph has 25,630 nodes and 3,604,984 edges; largest connected component has 22,895 nodes and 3,604,206 edges. Graph scale (Table 2)
• This density suggests many short multi-hop paths exist; thus evaluation must robustly sever those shortcuts (the paper tries via drug-similarity splitting). Rationale for drug-similarity split

• The extracted target-level table indicates heterogeneous outcomes: e.g., Aceclidine’s M1–M5 ground-truth targets show very low PIGLET scores (<0.01) but extremely high percentiles (e.g., 0.986405). Aceclidine high percentiles with low scores (Table 6)
• Blind spot: the paper’s percentile definition suggests relative ranking across all proteins, but we don’t see the exact score distribution or thresholding scheme; thus, interpreting “0.9 or higher” depends on how scores map to ranking. Case study recovery thresholds

5.1 Heterogeneous graph design choices

• Pocket similarity is built from ESM2 residue embeddings with a fixed cosine similarity threshold of 0.95, applied via HOTPocket predicted pockets + BioLiP2 known pockets. Fixed pocket similarity thresholding
• Drug similarity edges rely on ChemBERTa embeddings and a cosine threshold of 0.8. Fixed drug similarity threshold
• The paper claims these similarity edges should help a guilt-by-association mechanism: similar pockets tend to bind similar ligands. That biological premise is plausible but not proven within the provided text; the key uncertainty is how well the thresholds and embedding space align with binding specificity rather than generic structural similarity. Biological premise behind similarity edges

5.2 Split protocol realism & residual leakage

• The drug-based split uses RDKit Morgan fingerprints, hierarchical clustering by Tanimoto similarity, then cutting into 100 clusters, assigning clusters to 5 folds and a held-out test set such that all binding interactions for a given drug remain in its split. Drug-based split construction
• Residual leakage risk can still occur if the graph contains informational shortcuts not severed by the drug-similarity split (e.g., via pocket similarity edges that may correlate with drug similarity, or via DrugBank-derived message-passing edges that were restricted using a fingerprint cluster assignment to fold medioids). DrugBank edge restriction mechanism
• Because the paper selects hyperparameters using this drug-based split (with cross-validation) but never uses the testing set for tuning, it reduces classic test leakage; nonetheless, repeated cross-validation + epoch selection can still lead to variance and implicit overfitting to the specific split protocol. Hyperparameter tuning and epoch selection protocol

7.1 Strengths

• Addresses a known pitfall: random splits may inflate DTI performance via leakage; uses a drug-similarity split to stress generalization. Leakage-mitigation motivation
• Performs a graph-structure ablation isolating DrugBank message passing effect, showing where the additional graph signal matters. Ablation supports inductive bias claim

7.2 Likely blind spots / underreported uncertainty

• Calibration/thresholding: AUROC improvement does not directly show improved precision at clinically relevant score thresholds; the case study uses “score ≥ 0.9” but the mapping from score to probability is not shown in the excerpt. Score threshold interpretability gap
• Dataset dependence: performance is only benchmarked on the Human dataset, so it’s unknown whether the graph construction and split protocol improvements translate to other DTI benchmarks (Davis/KIBA/BindingDB/C. elegans). Cross-dataset generality unknown
• Biological validation: the case study includes computational recovery of known targets but does not include orthogonal biochemical/structural validation in the provided text. Thus “real-world utility” remains a hypothesis. Computational-only deployment validation

7.3 What would change the conclusion?

• If drug-similarity splits are further refined (e.g., stronger severing of pocket similarity shortcuts) and PIGLET no longer maintains AUROC superiority, then the advantage likely came from representational shortcuts rather than true generalization. Dependence on split/threshold design
• If external benchmarking on independent datasets (other DTI benchmarks) shows no advantage over strong sequence/structure models under similar “novel drug” regimes, the generality claim weakens. Need for external dataset benchmarking