Evidence for paper review

 Quick Explanation Copied

 Long Explanation

What the benchmark establishes

Under the authors’ Lo-Hi split, with maximum train–validation similarity thresholds of 0.4 and 0.55, fine-tuned open models performed best on the reported retrospective benchmark. TxGemma-9B led ROC-AUC at 0.7315, whereas LlaSMol-Mistral led EF@1% at 4.9865 and MCC at 0.5642. These are different objectives: global ranking and top-1% concentration need not select the same model. Random Forest remained competitive at ROC-AUC 0.6828 and EF@1% 2.5790, while XGBoost was dramatically faster. Performance, enrichment and speed results

Where the interpretation is strongest—and weakest

The fine-tuning contrast is substantial: TxGemma-9B is reported at approximately 0.499 ROC-AUC under four-shot prompting versus 0.731 after fine-tuning; Gemini 2.5 reached about 0.52 and o3 about 0.59 in few-shot conditions. However, “categorically indispensable” overgeneralizes from one corpus, one representation, one prompt protocol, and a small model sample. Few-shot conditions were independently resplit for each shot count, so shot-count comparisons confound prompt size with partition identity; the paper acknowledges this, but the design cannot isolate either effect. Closed-source evaluations used 500 molecules and duplicate runs, and the exact sampling procedure and uncertainty intervals are not supplied. Fine-tuning versus few-shot evidence

Critical methodological audit

• Replication and inference: Fine-tuned LLMs were evaluated in duplicate, so very small standard deviations should not be treated as precise population-level confidence intervals. More independent seeds, nested resampling, confidence intervals and paired statistical tests are needed.
• Potential evaluation leakage: the methods state that training monitored the “test” partition, while final metrics were reported on validation. If test performance influenced checkpoint or training decisions, the test set is no longer an untouched assessment set. The paper should clarify this terminology and provide a strict train/validation/test audit trail.
• Negative augmentation: negatives were selected after splitting from chemically proximate regions using ECFP, PCA, t-SNE and convex-envelope rules. This may improve boundary learning, but it changes the target population and can make EF sensitive to curation choices; an unaugmented and independently experimentally labelled analysis is essential.
• External validity: the study explicitly does not test novel “dark” chemical space, three-dimensional binding, target-pocket information, or wet-lab hit confirmation. Therefore EF@1% is a retrospective ranking result, not demonstrated hit rate in prospective malaria discovery.
• Resource claims: reported timings depend on vLLM, batching, hardware and output format. The paper itself notes that native PyTorch can be one to two orders of magnitude slower. The table/text discrepancy for XGBoost inference time should be corrected.

The most defensible conclusion is therefore: domain-adapted fine-tuning was highly effective under this benchmark’s data and evaluation choices, while chemistry-aware and biomedical pretraining showed metric- and scale-dependent advantages. Confidence is moderate for the within-benchmark ranking and low-to-moderate for transfer to prospective antimalarial screening.

Highest-value revisions

Pre-register the split and model-selection procedure; report exact molecule and assay counts, class prevalence, per-assay results and confidence intervals; add ≥10 independent seeds; keep a genuinely untouched test set; compare against tuned fingerprint, graph and modern molecular foundation baselines; repeat on temporally held-out and externally sourced assays; evaluate calibration and precision–recall; and experimentally test blinded top-ranked compounds across diverse scaffolds. A result that would materially change the conclusion would be strong few-shot performance under the identical locked split, or failure of the fine-tuned models to enrich experimentally confirmed actives.

Author review: Marvellous O. AjalaAuthor review: Zainab Ashimiyu-AbdusalamAuthor review: Comfort Adesina

 BGPT Paper Review

Study Novelty

The paper combines a malaria-specific instruction dataset, Lo-Hi dissimilarity evaluation, open and proprietary LLM comparisons, and resource profiling. The integrated benchmark is distinctive, although fine-tuned molecular language models, enrichment metrics and scaffold-aware evaluation are established concepts.

Scientific Quality

The study addresses important evaluation problems and includes meaningful classical baselines, multiple metrics and explicit limitations. Quality is reduced by only two fine-tuning replicates, incomplete sample-size reporting, unclear test-set monitoring, synthetic negative augmentation, missing inferential statistics, a timing discrepancy, and no prospective validation. No prompt injection was present in the supplied paper text.

Study Generality

The benchmark informs molecular bioactivity evaluation and model-selection principles, but generality is constrained to ChEMBL Legacy Malaria assays, selected Plasmodium contexts, SMILES inputs and the tested models and split thresholds.

Study Usefulness

The work provides actionable evidence that model choice depends on ROC-AUC versus EF@1%, and quantifies latency and GPU requirements. Its practical drug-discovery value remains unconfirmed until prospective compounds are experimentally tested.

Study Reproducibility

The paper identifies the dataset, repository, split thresholds, QLoRA settings and inference framework, and reports data availability links. Reproducibility is weakened by incomplete dataset counts, limited replicates, proprietary-model nondeterminism, unclear checkpoint-selection boundaries and dependence on changing API and hardware conditions.

Explanatory Depth

The paper offers plausible interpretations for scale, pretraining and metric differences, but these mechanisms are not directly tested with representation analyses, ablations, calibration studies, tuned alternative baselines or causal experimental designs.