The Evidence Layer for Scientific AI

 Quick Explanation Copied

 Long Explanation

What “Zero-Shot” Means

Zero-shot learning: a model performs a task or recognizes classes it was never explicitly trained on, using pretrained knowledge plus semantic side-information (attributes, class descriptions, or natural-language prompts). It contrasts with few-shot (a handful of labeled examples) and supervised (many labels) regimes NLP zero-shot and few-shot review. The classic zero-shot pipeline embeds inputs and class descriptions into a shared space, then matches by similarity — no gradient updates on the target task.

Evidence From Five Scientific Domains

1. Vision–language models (CLIP). A Florida framework classified 12 cultural ecosystem services from 1.68M Flickr images using only 120 labeled development images. Closed-set zero-shot accuracy depended heavily on prompt design: simple prompts 90%, detailed 92%, mixed 97% macro-F1 0.96 CLIP zero-shot CES mapping study. In the harder open-set setting (rejecting irrelevant images), a hybrid zero-shot+lightweight-filter pipeline hit 88% accuracy vs 78% for pure zero-shot — showing pure zero-shot is precise but conservative.

2. Wildlife camera traps. True zero-shot BioCLIP scored only 46.76% in-domain on Serengeti (vs 90.52% for retrieval-based DINOv2), but degraded least across geographic domain shifts Camera-trap zero-shot domain shift study.

3. Protein science. ESM-1v zero-shot predictions correlated with experimental variant fitness across 34 deep-mutational-scanning datasets (31/34 with Spearman ρ > 0.2; median ρ ≈ 0.44) Zero-shot protein language model study, and multimodal FunBind predicted function for proteins with novel Gene Ontology terms FunBind zero-shot protein function model.

4. Spatial transcriptomics. TERRA, pretrained on 112M cells, achieved top zero-shot niche identification on held-out tissues and even predicted an immune-checkpoint-injury gene program via in silico knockout (Sinkhorn divergence ~0.08 vs ~0.13–0.14 controls) TERRA spatial foundation model.

5. Pathology. Zero-shot whole-slide retrieval with foundation-model embeddings reached only 40–42% top-5 F1 across TCGA (11,444 slides, 23 organs), with organ-level swings from 82% (kidney) to 21% (lung) — zero-shot is not yet reliable in high-stakes medical contexts Histopathology zero-shot retrieval validation.

Key Mechanisms and Limits

Three consistent findings: (i) semantic side-information quality drives success — prompt wording moved CLIP accuracy 7 points Prompt sensitivity in CLIP zero-shot; (ii) domain shift hurts supervised models more than zero-shot ones — BEiTV2 lost 38 pp Kruger while BioCLIP lost ~1 pp Zero-shot robustness to domain shift; (iii) failure modes include hallucination and background shortcut reliance Zero-shot failure modes.

Confidence and Blind Spots

Confidence: high that zero-shot works via pretrained priors + semantic matching, and that performance is domain-dependent. Uncertainties: all cited benchmarks use limited test sets in some settings (e.g., n=146 cross-domain camera-trap images); the pathology review notes possible selection/reporting bias; DMS zero-shot results depend on evolutionary priors embedded in PLMs and 3/34 datasets showed near-zero or negative correlation Zero-shot limitations in protein engineering. Evidence that would change the conclusion: consistent zero-shot superiority over supervised baselines in-domain, or failure of prompt engineering to affect zero-shot accuracy across independent replications.