Predicting AI-System Responses Without Running Them: A Hierarchical Test-Time Update for the NeurIPS 2026 Predictive AI Evaluation Competition

The Predictive AI Evaluation Competition at NeurIPS 2026 asks for a program that returns the probability that an AI system answers a benchmark item correctly, without running the system, on benchmarks unseen during training and with 0 to 31 revealed labels per subject–benchmark pair. Our entry (Team Drift) combines a weighted logistic ridge prior, trained on 167 public binary benchmarks, with a test-time update that fits four shrunk corrections (a benchmark shift, a subject-ability slope, a subject–benchmark offset and an item offset) to the revealed labels. It ships as standard-library Python. On a local harness that holds out whole benchmarks, it scores a Brier area under the learning curve (ALC) of 0.1953 (five-fold mean), against 0.2475 for the organisers' empirical-mean starter predictor and 0.2500 for a constant 0.5; this figure is optimistic because test-time shrinkage was tuned on the same folds. Under a stricter split that also holds out subject groups, the score is 0.2060. A later variant with static item embeddings and per-budget temperature calibration improved that local score by 0.0011, but scored worse on the hosted formative evaluation (0.1888 against 0.1759 on the platform's primary brier field), so we keep the simpler model. Ablations show that nearly all the gain comes from the test-time update, and the item-text prior adds nothing measurable. No leaderboard rank or final summative score exists for this entry at the time of writing.

Drafted with AI assistance; reviewed and approved by the author. CloudLab Works.

@techreport{alvord2026predictive,
  author = {Alvord, Alex},
  title = {{Predicting AI-System Responses Without Running Them: A Hierarchical Test-Time Update for the NeurIPS 2026 Predictive AI Evaluation Competition}},
  institution = {CloudLab Works},
  publisher = {Zenodo},
  year = {2026},
  month = oct,
  type = {Preprint},
  doi = {10.5281/zenodo.23720944},
  url = {https://doi.org/10.5281/zenodo.23720944}
}