Review Papers Claim by Claim

 Quick Explanation Copied

 Long Explanation

Benchmark evidence for LLM superiority over curated databases

The paper's central empirical claim rests on a manually curated, disagreement-enriched benchmark (SurfaceBench, n=147). Sonnet 4.6 with gene-symbol-only input classified 95.7% of proteins correctly, nearly matching Opus 4.8 (95.9%) and far ahead of Haiku 4.5 (81.6%); the best database under recalibrated inclusion rules was UniProt at 90%SurfaceBench accuracy comparison. The benchmark was deliberately biased toward database disagreement, so accuracy on the full proteome is likely higher (easy multi-pass proteins were under-represented), but it also means the 95.7% figure is not directly comparable to a random sample.

Three orthogonal positive-control sets (ADC targets n=234, TCE n=30, viral receptors n=62) recovered 324/326 (98-100%) by the agent versus 92-97% for UniProt — but the authors explicitly note these sets test recall only, not specificityPositive-control target recovery.

The deep dive on 5,332 candidates produced 1,783 canonical and 1,252 likely surface proteins; 50-protein full re-runs gave five-tier κ=0.85 (95% CI 0.72-0.93), rising to 0.92 when evidence was held fixed — most run-to-run variance came from which papers were retrieved, not interpretationDeep-dive reproducibility kappa. This κ=0.85 exceeds inter-curator agreement reported for professional GO curation (~κ 0.4).

Critical blind spots

• No proteome-wide ground truth. Database-vs-agent agreement (75-90%) was scored against the agent's own calls, which is circular; only SurfaceBench (147 proteins, hand-curated by a single author) provides independent truthProteome-wide agreement analysis.
• Corpus access bias. 89% of targeted-discovery papers were readable (88% via PMC); broader OpenAlex search found 3.6x more papers but only 49% were bot-reachable — the database reflects what machines can read, not the state of science. Literature itself is positively-biased and may rest on poorly validated antibodies.
• Conflict of interest. Authors are employees of Deliverome Bio (whose internalization/deliverome data this resource advertises) and received Anthropic API credits through the Claude AI for Science program — the paper both builds on and promotes Claude models.
• Only two benchmark errors on the deep dive (ATG9A, RPN1) are disclosed; the deep dive also misclassified secreted ADC antigens COLEC10, FMOD, KLK3 as non-surface — arguably defensible, but shows boundary cases persist.

Verdict: A well-executed, unusually transparent resource study (all code, records, revisions, and per-claim verbatim quotes archived on Zenodo/GitHub with a public API). Claims are properly hedged by the authors themselves. What would falsify the central claim: a pre-registered, randomly-sampled, multi-curator benchmark showing optimized databases matching or exceeding the LLM pipeline.

 BGPT Paper Review

Study Novelty

First proteome-scale, evidence-traceable LLM-curated accessible surfaceome with a disagreement-enriched benchmark; builds on established surfaceome databases (SURFY, CSPA, HPA) but the agentic curation approach is genuinely new.

Scientific Quality

Rigorous benchmarking with recalibrated database cutoffs, reproducibility kappa, positive controls, and disclosed limitations. Weaknesses: single-curator benchmark, circularity of proteome-wide agreement metrics, recall-only positive controls, and industry/Anthropic conflicts.

Study Generality

Method generalizes to any literature-scattered annotation problem (authors note ligand-receptor pairing, localization); surfaceome focus is specific but the pipeline template is broadly transferable.

Study Usefulness

Directly useful for therapeutic target selection: 3,035 canonical/likely proteins with per-claim verbatim evidence, public API, orthology, ECD length, and paralog data enable antibody/ADC/CAR-T target triage.

Study Reproducibility

All code MIT-licensed on GitHub, records on Zenodo with numbered revisions, prompts archived, three-replicate benchmark calls, and explicit reproducibility kappa — but exact model versions pin reproducibility to a proprietary LLM.

Explanatory Depth