Review papers with raw data transparency

Quantitative highlights (from paper)

Amino‑acid diversity per query: beam vs multinomial (paper claims)

Novel NNAA output (reported)

Interpretation (concise, evidence‑first)

• Representation and model: The authors use CHUCKLES to encode amino acids at atomic SMILES granularity and train a Transformer conditional generator on a masked infill task (900k training pairs; model architecture/optimizer/hyperparameters match REINVENT transformer implementation) — this supports chemistry-aware generation beyond enumerated AA lists (PepINVENT methods).
• Data foundations: Semi-synthetic peptides are drawn from natural AA + large enumerated NNAA pool derived from reaction-based enumeration (Amarasinghe et al.), which gives access to many synthetically plausible sidechains but is not the same as experimentally measured NNAA space (NNAA enumeration).
• Performance claims: High per-sample validity (>98%), strong uniqueness, and large AA diversity expansion per query are reported (Table 1 and figures). The model can produce topology-aware outputs and be steered via RL for ring-size/topology objectives and for an MPO run optimizing CAMSOL-PTM solubility and an XGBoost permeability predictor (balanced accuracy 0.78 reported for permeability model) — all are internally consistent with the methods and code published in the GitHub repo (PepINVENT results).

Critical appraisal — strengths

• Clear, well-documented pipeline and open code/data (Zenodo + GitHub) increases transparency and reproducibility potential (Data & code availability).
• Technical novelty: applying conditional text-infill at atomic-level AA tokens (CHUCKLES) to permit stereochemistry/backbone modifications is conceptually new for peptide generative models.
• Useful demonstration of RL steering for topology and MPO (practical use-case: Rev-binding peptide optimization) showcases modularity with peptide-specific scorers (CAMSOL-PTM and an XGBoost PAMPA model).

Critical appraisal — limitations, blindspots, and risks

1. Semi-synthetic training data ≠ experimental reality. The NNAA pool stems from reaction-based enumeration (synthetically plausible precursors) not measured biological/pharmacokinetic data. Generated novel NNAAs therefore require synthetic follow-up and ADME/toxicity testing before claims about improved properties are actionable (NNAA enumeration limitations).
2. Lack of wet‑lab validation. The paper demonstrates in silico MPO but does not present experimental synthesis/biophysical or cellular validation of generated peptides. Without experimental verification, improvements in predicted permeability/solubility remain hypothetical.
3. Sampling bias and meaning of 'novelty'. The paper shows beam search tends to propose likely, repeatable sequences (oversamples natural motifs), while multinomial broadens exploration; the large counts of "novel" NNAAs (>91k) should be interpreted with caution because many novel SMILES can be stereochemical variants or minor side-chain permutations with uncertain synthetic feasibility or biological utility (Sampling bias & novelty).
4. Synthetic feasibility & SPPS compatibility not enforced. The NNAA enumeration is carefully assembled, but PepINVENT does not (yet) integrate an explicit synthetic-feasibility scorer or NNAA protection/compatibility filter (e.g., NNAA-Synth style retrosynthetic/protection check), which would reduce false leads (NNAA-Synth (relevant)).
5. Predictor limits & domain shift risk. The CAMSOL-PTM and XGBoost permeability predictor were trained on datasets of (mostly) known chemistries; their reliability on radically novel NNAA chemistries, bicyclic motifs, or backbone heterocycles may be limited—distributional shift risk is not quantified.
6. Reproducibility caveats. While code/data are available, reproducing the enormous semi-synthetic generation (1M peptides) and retraining a transformer at scale requires compute; hyperparameter sensitivity and random seeds for multinomial sampling can change diversity statistics. Authors provide reasonable details (epochs, lr, warmups), but full seeds/checkpoints should be archived to maximize reproducibility (Reproducibility details).

How robust are the claims?

The computational claims (validity, uniqueness, chemical-space exploration) are supported by the internal statistics and visualizations; however, claims that generated peptides will improve properties in wet‑lab settings remain hypotheses until synthesis and experimental assays validate them. Confidence in the generative capability is moderate–high (model behavior is plausible given transformer text infill training), but translational confidence (synthesis, ADME, bioactivity) is low without experimental follow-up (Translational caveat).

Concrete recommendations (to authors / users)

• Integrate a synthetic-feasibility / protected-NNAA scorer (NNAA-Synth style) into RL scoring to penalize infeasible NNAAs and prioritize SPPS-compatible candidates (NNAA-Synth recommendation).
• Publish model checkpoints and random seeds (and a lighter-weight distilled model) to enable exact reproduction of diversity/novelty statistics by other groups.
• Report additional external validations: synthesize a small set (<10) of high-MPO scored peptides (diverse NNAA chemistries) and measure solubility, PAMPA permeability, and proteolytic stability to directly confirm predicted gains.
• Quantify domain shift: test CAMSOL-PTM/XGBoost predictor performance on NNAA-containing peptides and report confidence intervals or applicability domain metrics to avoid overtrusting predictors on out-of-distribution chemistries.

What would falsify the key claims?

• Failing to synthesize a representative set of proposed NNAAs or repeatedly obtaining low yields would falsify the practical applicability of many generated candidates.
• Experimental assays showing no improvement (or worsening) of solubility/permeability for multiple high‑scoring MPO outputs would falsify the claim that RL‑guided generation produced useful property optimization.

Confidence & brief scorecard

Below are my structured judgments (see numeric fields returned separately). Supporting evidence includes the paper text, tables, and public repo; main uncertainty derives from limited experimental validation and feasibility scoring absent from the RL loop.

References used in this review (high‑value, inline)

1. PepINVENT paper
2. Amarasinghe NNAA enumeration