The Evidence Layer for Scientific AI

 Quick Analysis Plan Copied

 Long Analysis Plan

Replication & Extension Plan for MAPS Ensemble Boundary Uncertainty

The paper trains ten independent MAPS LSTM forced aligners, applies order-statistic confidence intervals (2nd–9th order statistics ≈ 97.85% nominal CI), and evaluates gradient boundary regions on TIMIT and Buckeye American English corpora. Code and data are openly archived on GitHub (github.com/maetshju/maps_ci_analysis) and Zenodo (10.5281/zenodo.15571293) (MAPS ensemble CI code availability).

Step 1 — Reproduce ensemble alignment. Load the ten pretrained MAPS LSTM models from the Zenodo archive. For each utterance in TIMIT and Buckeye test splits, run forced alignment independently in each model, linearly interpolate boundaries within each model, then take the per-boundary median across ten models as the point estimate and the 2nd/9th order statistics as the CI bounds. Report mean and median absolute error against reference transcriptions (Ensemble vs single-model boundary error).

Step 2 — Segment-ambiguity analysis. Group boundary-region widths by bisegment category pairs (e.g., vowel–vowel, affricate–fricative, stop–vowel). Compute median widths and empirical CDFs per category to confirm that ambiguous sequences yield wider uncertainty bands (Widths track segment ambiguity).

Step 3 — Calibration audit. The reported containment of only 34.06% against a nominal 97.85% CI reveals severe miscalibration — the ensemble measures model-function uncertainty, not phonetically true boundary ranges. A key extension is to recalibrate via conformal prediction split by corpus to achieve valid empirical coverage (Containment miscalibration finding).

Step 4 — Cross-corpus generalization. TIMIT (read speech) yields lower errors than Buckeye (conversational speech), confirming domain sensitivity. Extend by testing on non-American-English corpora (e.g., Mandarin, Dutch child speech) to probe generalizability beyond the training distribution (TIMIT vs Buckeye error breakdown).

Known blindspots: No statistical significance tests for ensemble improvement; no comparison with human annotator inter-rater uncertainty; Praat TextGrid format cannot natively encode overlapping or gradient boundaries; ensemble inference is ~10× slower; single-author study without independent replication.