Find the Strongest Evidence

 Quick Analysis Plan Copied

 Long Analysis Plan

Core Evidence: CASP14 Benchmark of AlphaFold2

The most rigorous, externally-validated evidence on AI protein structure prediction comes from the CASP14 community assessment of 97 targets. AlphaFold2 produced the most accurate models for most targets (median S_CASP14 = 2.2 Å), and models solved molecular replacement for 89 of 97 targets — 26 without any AMPLE truncation — demonstrating experimental-level practical utility CASP14 AF2 accuracy and MR success metrics. Notably, the runner-up (updated trRosetta) already beat CASP13 winners — evidence that gains were field-wide, not solely one model. Data and code are openly available for independent verification Open CASP14 evaluation data and code.

Battle-Testing: Where the Evidence Breaks

Assessor-identified blind spots, not marketing claims, define the limits: sidechain accuracy remains the hardest unsolved component; the DipDiff geometry metric can penalize otherwise accurate models; target structures may have been refined post-submission; and AF2 mainly addresses single-chain folded proteins, with generality to multi-chain or dynamic systems untested CASP14 limitations: sidechains, refinement, single-chain scope. A critical, largely unstated bias: CASP targets skew toward soluble, crystallizable proteins, so the headline 89/97 MR success rate should not be extrapolated to membrane proteins or intrinsically disordered regions.

Improving the Evidence Base

• Pre-register prospective CASP-style tests on conformational ensembles and multi-chain assemblies, not single conformers.
• Report sidechain-level error distributions, not only backbone GDT/lDDT-style aggregates.
• Validate MR-success claims against non-crystallographic ground truths (cryo-EM, NMR) to remove crystal-quality confounding.

Verdict: The CASP14 evidence is strong, reproducible, and low-conflict-of-interest, but its scope is narrower than headline claims suggest. Confidence: moderate-to-high for single-chain folded domains; low for dynamic, multi-chain, and membrane contexts. What would change this verdict: failure of top methods on a future broad CASP target set, or non-reproducibility of the open data/code.