Paper Review — Verify Claims Fast

What the model is (visual summary)

• Representation: three heterogeneous graphs — ligand atom graph, protein atomic graph (distance-based edges), and a residue-level pocket graph (8 Å pocket) (paper text) IGModel representation details
• Encoder: Edge‑aware Graph Attention (EdgeGAT/EGATConv) message passing, producing 1024‑dim atom/pocket embeddings.
• Decoders: shared latent space → VRMSD and Vpkd (128‑dim each); Vpkd augmented by an RMSD‑mapped vector and a learned decay weight W (sigmoid) that rescales native pKd -> docking‑pose pKd labels.

Key reported strengths (paper claims with citation)

1. Very high docking power (Top‑1 success rates ≈95–97% reported for RMSD model) on CASF‑2016 redocking and when native poses were excluded IGModel docking power numbers
2. Balanced scoring and ranking power: PCC ≈0.83 for pKd predictions on CASF‑2016 core (authors compare to recent GNN/transformer baselines) IGModel scoring power PCC
3. Generalization tests: authors evaluated on cross‑docking (PDBbind-CrossDocked-Core), DISCO, and AF2‑derived test sets and on an "unbiased" unbias‑v2019 split; IGModel reportedly retains substantial performance on AF2 structures (tables in paper) IGModel generalization claims

Critical issues, blindspots and methodological cautions

• Preprint status — not peer reviewed (bioRxiv), so claims require independent confirmation IGModel preprint status
• Training/test overlap risk & dataset leakage: Authors state they used PDBbind v2019 and an "unbias‑v2019" test; however the preprint lacks full, machine‑readable splits, the exact redundancy-removal thresholds and sequence/ligand‑similarity cutoffs are not reproducibly detailed in the main text (supplement may contain details). Without full split files/code, performance can be inflated by hidden similarity between train and test — a recurring concern in ML SF literature IGModel dataset split description
• Label assumptions (pKd decay with RMSD): The model assumes binding strength decays linearly (scaled by learned W) with RMSD from native and uses native pKd to generate pose pKd labels. This is a modelling choice with physical simplifications: RMSD is not a perfect surrogate for free energy; some non‑native but low‑RMSD poses may still have different energetics and orientation-dependent solvation/entropy effects. The assumption requires empirical validation vs. experimental affinity/pose ensembles or free-energy calculations IGModel RMSD→pKd label assumption
• Evaluation metrics & baselines: Comparisons are largely to recent DL SFs (e.g. RTMScore, DeepDock, GT_ft1.0, GatedGCN_ft1.0) — but the preprint sometimes cites baseline numbers from other papers rather than re‑evaluating all methods identically (same docking generators, same pose sets, identical filtering). Cross‑paper comparisons risk methodological mismatch; best practice requires re-running baselines on identical pose sets and splits (or releasing evaluation code/data) IGModel baseline comparisons
• Explainability claims: Attention/importance ranking highlights polar atoms and hydrogen bonds in examples — informative but not definitive: attention does not prove causal mechanistic physics; independent perturbation tests (masking interactions, energy decomposition, free energy changes) are needed to confirm the model's physicochemical interpretability IGModel attention explainability
• Missing reproducibility artifacts in main text: the preprint does not clearly link to public code, trained weights, or exact split files in the main article body (some supplementary parts referenced). Reproducible scoring-function papers typically release code + data partitions to allow independent re-evaluation (e.g., RTMScore, PocketGen, MaSIF) — release status must be checked and is essential for acceptance and community uptake IGModel code/data availability

Practical implications & where this fits in the field

If the reported numbers survive independent re-evaluation (identical splits, re-run baselines and AF2 tests) IGModel would be a significant step toward integrated pose‑and‑affinity predictors that yield interpretable per‑pose binding strength estimates — useful for pose selection after docking and for guided lead optimization. However, the key next steps are:

1. Full public release of code, split files, and model weights so other groups can reproduce results.
2. Independent re-evaluation on community benchmarks with identical docking poses (e.g. CASF-2016 pose sets, CrossDocked, DISCO) and on prospective blind tests.
3. Targeted ablations to test the pKd←RMSD label assumption (compare against experimental per‑pose free energy changes or FEP calculations where available).

What would change my assessment (falsifiability)

Findings that would reduce confidence: (a) re‑running baselines and IGModel on identical pose sets showing smaller or no advantage; (b) proving the RMSD→pKd label heuristic is inconsistent with experimental pose energetics; (c) demonstration that results depend on training/test leakage. Conversely, public release and independent re-runs confirming reported metrics would substantially increase confidence.

Selected primary citation

Quick actionable recommendations for authors & adopters

• Release code, exact split files, trained checkpoints and inference scripts for reproducibility.
• Provide a controlled re-run of baselines on identical pose sets (same docking outputs) to eliminate cross‑paper comparison bias.
• Provide ablation showing sensitivity of pKd predictions to the RMSD→pKd labeling assumption (e.g., train with alternate label mappings; compare to FEP on a subset).
• Publish uncertainty estimates for pKd (epistemic + aleatoric), especially when applying to AF2 structures or low-confidence pockets.