Review Papers Claim by Claim

Visual paper analysis — "Temporally‑Informed Random Forests for Suicide Risk Prediction" (Bayramli et al., 2021)

What the paper did — succinct (paper-sourced)

• Assembled structured EHR features + NLP-extracted concepts from Mass General Brigham RPDR (1998–2018) and trained predictive models for suicide attempts; applied feature selection to arrive at ~16,131 features for RFs and used a previously validated Naive Bayes Classifier (NBC) as baseline.
  Dataset and features (paper)
• Defined three summary temporal features (period covered in years, visit number, visit rate = visits/period) and tested three BRF variants: NoTime (no temporal features), Base-BRF (temporal treated like other features), and OT-BRF (require temporal features present in every tree).
  Temporal variables and model variants

Key strengths (evidence-based)

• Simple, interpretable modification to RFs that explicitly encodes coarse temporal context — computationally cheap and easily portable to tree-based pipelines used in health systems (OT-BRF implementability).
• Empirical improvement at high-specificity operating points (95% specificity) — clinically relevant because suicide risk tools need high specificity to avoid overwhelming false positives; OT-BRF increased sensitivity at 95% specificity relative to alternatives per reported test metrics (Sensitivity improvement (paper)).
• Inclusion of NLP-derived features from notes increases clinical signal breadth — the methodology bridges structured and unstructured data sources used in modern EHR research (NLP features included).

Major limitations and blindspots (critical)

• Non-equivalent training sets between NBC and RFs: NBC trained on full cohort (1.6M patients) while RFs trained on sampled subsets (52k patients, with visit sampling). This complicates direct comparison: apparent RF superiority may partly reflect cross-sample design differences rather than purely algorithmic gains. Evidence: paper describes full NBC training vs sub-sampling for RFs to contain dataset size (~citation doi="10.1101/2021.06.01.21258179" title="Temporally-Informed Random Forests for Suicide Risk Prediction [2021]" research-extract="NBC: 1,625,350 patients used; RFs trained on sampled subsets (training: 52,006 patients) with per-visit sampling (0.05 inclusion probability) and upsampling/undersampling strategies to get bootstrap-case prevalence ~12% for training." url="https://dx.doi.org/10.1101/2021.06.01.21258179" number-citations="1" descriptive-anchor-text="Different training data sizes" evidence-strength="moderate">)
• Potential overstatement of temporal-encoding benefit because temporal variables are coarse summaries. OT-BRF forces three low-dimensional summary features but does not explicitly model fine-grained temporal sequences, diminishing gains when features are effectively binarized (paper notes many features split at low counts). The authors themselves point out modest AUC differences (~1.8% AUC drop when removing temporal features) and hypothesise binary occurrences reduce scale problems. (Authors’ own limitation on effect size)
• Use of impurity-based Gini importance is known to bias toward variables with many categories or larger scales; authors mention this and do not present permutation importance or SHAP analyses to validate importance rankings. This weakens claims that temporal features were truly the key drivers — alternative importance measures are standard best practice but absent (Gini importance limitation (paper)).
• Generalisability risk: single health system (Mass General Brigham) over 1998–2018; medical practice, coding, and healthcare-seeking patterns vary across systems and time. External validation in independent institutions or modern datasets is absent — a critical next step before clinical translation. (See broader reviews noting pervasive lack of robust external validation across suicide prediction models.) Systematic review: need for external validation
• Sampling & preprocessing opacity: the paper subsampled visits (0.05 inclusion probability) to control table size but did not make full code or exact sampled data available; authors state data cannot be shared due to PHI. This prevents independent reproduction of their specific training/test splits and evaluation. The methods describe hyperparameter grid search but no public code or seeds are provided (data availability statement: restricted). (Data/code unavailability (paper))

Technical critique & suggestions (practical)

1. Validate temporal effect using permutation and conditional feature importances (or SHAP) computed on held-out test data to reduce Gini bias and confirm that temporal variables causally help split thresholds rather than proxying visit-volume artifacts. (Paper used only impurity-based importances.)
   Need for alternative importance measures
2. Provide external validation on at least one independent health system (transferability tests). The literature repeatedly flags that promising models often fail external transport without recalibration (Need for external validation (systematic review)).
3. Compare OT‑BRF to temporally-explicit sequence models (e.g., Transformer/LSTM or neural ODE/GRU-ODE hybrids) on identical training/test splits. The preprint cites prior work using temporal neural ODEs and sequential models that can capture fine temporal dynamics; a direct benchmark would clarify whether expensive sequence models substantially outperform the simple OT‑BRF trick given the same data (Neural ODEs for temporal EHR modeling).
4. Report calibration (e.g., calibration curves, Brier score) and clinical utility measures (decision curve analysis, net benefit) in addition to AUC/PPV/sensitivity, since high discrimination alone does not guarantee clinical usefulness in low-base-rate outcomes like suicide. The field review highlights missing calibration reporting as pervasive shortcoming (Calibration reporting needed).
5. Clarify sampling/splitting reproducibility: publish code (model training, subsampling, feature selection, seeds) and, if possible, synthetic-but-true-to-distribution artifacts or model weights to allow external groups to reproduce model behaviour without PHI disclosure.

Bottom line (balanced judgment)

What would change the conclusion?

• Strong evidence that OT-BRF maintains performance (AUC and sensitivity) in at least one independent health system and is well-calibrated would increase confidence substantially.
• If permutation/SHAP analyses show temporal features contribute large, independent predictive value (not merely proxies for visit volume), the mechanistic claim about scale-adjustment would be supported.
• If temporally-explicit sequence models (trained and tested on the same splits) substantially exceed OT‑BRF performance, it would suggest coarse summaries are insufficient and that sequence modeling is necessary; conversely, parity would support the pragmatic OT‑BRF approach.

Quick action checklist for authors / implementers

1. Compute permutation importance and SHAP; publish these and show temporal features’ independent contributions.
2. Report calibration plots, Brier score, decision curve analysis, and stratified performance by demographic subgroups to check fairness.
3. Run external validation on at least one independent RPDR-like dataset (or share a secured evaluation endpoint for external teams), and compare OT-BRF to neural ODE/GRU/LSTM baselines on identical splits.
4. Open-source code for reproducibility (training pipeline, sampling seeds, hyperparameter grid) and provide pre-trained model artifacts without PHI when possible.