Find the Strongest Evidence

Key empirical takeaways (evidence-first, with inline source extracts)

1. Shapley values quantify marginal contribution of each eligibility rule to population size and estimated treatment effect. Trial Pathfinder computes Monte‑Carlo Shapley estimates to rank criteria by their impact on eligibility and HR; relaxing many lab/prior‑therapy rules increased eligible pools ~53–107% while producing small average HR changes Trial Pathfinder results and methods
2. Outputs are only as good as the data and causal adjustment: Trial Pathfinder depends on Flatiron EHR captures, which omit some trial criteria and contain missingness; causal inference used IPTW but residual confounding remains plausible — authors list these as explicit limitations Trial Pathfinder limitations
3. Computation & approximation issues matter: Exact Shapley calculation is exponential; Trial Pathfinder uses Monte‑Carlo sampling and logistic/IPTW modeling which introduce approximation variance and dependence on model specification. This computational challenge and statistical-comparison methods have been previously highlighted in the literature on coalitional games in medicine Coalitional games & Shapley limitations (2013)
4. Methodological generality and alternative domains: Shapley-derived fields have been extended successfully beyond clinical eligibility (e.g., sensory neuroscience) showing feature-agnostic mapping but also small-sample and feature-selection sensitivity — illustrates both power and fragility of Shapley-style attributions Shapley fields in neuroscience (preprint 2025)

Practical, evidence-based critique (what to trust, what to question)

• Trust: Relative ranking of criteria is informative for prioritizing which criteria to reconsider; observed large eligibility expansions are robust across multiple trials in the dataset Trial Pathfinder general findings
• Question: Causality and safety. Shapley values here measure change in HR when criteria are removed in the emulated dataset; they do not guarantee that prospective relaxation will preserve safety or efficacy because (a) unmeasured confounding may bias HRs, (b) adverse-event capture differs between EHR and trial settings, and (c) Monte‑Carlo Shapley estimates have sampling error Trial Pathfinder cautions on causality/safety
• Bias sources to watch: Flatiron is US community oncology data — healthy‑user bias, treatment selection, and geographic/provider heterogeneity can shift estimates; publication/sponsor ties (industry co‑authors) may shape framing — verify with independent datasets and pre-registered analytic plans Data provenance and COI statement

Concrete recommendations to improve/battle-test Shapley use in trial design

1. Prospective pilot studies: implement a prespecified, limited eligibility relaxation (guided by top Shapley-ranked criteria) with preplanned safety/adverse-event monitoring and an independent DSMB to test observational predictions in a randomized or stepped-wedge design (discussed below as a novel experiment).
2. Triangulate with multiple datasets: re-run Shapley analyses on other EHR/claims registries (international sites) to test sensitivity to population and capture differences Need for external validation
3. Quantify estimation uncertainty: report Monte‑Carlo SEs/confidence intervals for Shapley contributions and run sensitivity analyses varying IPTW models, propensity covariates, and missing-data handling to show robustness.
4. Use negative‑control outcomes and falsification tests: demonstrate that relaxing neutral criteria does not change unrelated outcomes (reduces risk of confounding-driven spurious Shapley signals).
5. Regulatory engagement & transparency: pre-register analytic pipelines and share code/data licenses (Trial Pathfinder provided code; data licensing remains necessary) to reduce reproducibility/selection bias concerns Code sharing and data licensing

Novel experiment (specific, testable)

Where new evidence would overturn the main conclusion

• If prospective pilots show increased severe adverse events or higher withdrawal rates among newly eligible participants, that would falsify the safety inference from EHR emulation.
• If re-analyses on external registries produce systematically different Shapley rankings (i.e., top-ranked criteria differ), it would indicate high sensitivity to data source and limit generalizability.

Selected citations

• Trial Pathfinder (Nature 2021)
• Shapley fields (neuroscience preprint)
• Coalitional games & Shapley limitations (2013)