Predicting an AI system's answer without running it

Anyone choosing an AI system for a job runs into the same problem. You cannot test every candidate on every task, so you test a few and guess about the rest. The Predictive AI Evaluation competition, a NeurIPS 2026 competition run by Stanford's AIMS lab, turns that guess into a scored task. We have entered. Here is the problem, what we built, and what our numbers show.

Cite: Preprint · DOI 10.5281/zenodo.23720944 · Paper and BibTeX.

The problem

The organisers' vocabulary matters, because the scoring depends on it.

• A subject is an AI system pinned to one configuration: model, evaluation harness, harness version, reasoning effort. The same model under two harnesses is two subjects.
• An item is a question or task.
• A response is one run of a subject on an item, scored 1 if correct and 0 otherwise.

The program receives the attributes of a subject and an item and returns the probability that the response is 1. It never runs the subject. Test items come from benchmarks it has never seen, so there is no memorising the answers.

Evidence is rationed on purpose. For each subject and benchmark, the platform reveals 0, 1, 3, 7, 15 or 31 labelled responses. The score is the Brier error (mean squared gap between probability and outcome) at each of those label budgets, combined by area under the learning curve: Brier ALC. Lower is better, and always guessing 0.5 scores 0.25. The weights favour small budgets, the realistic case.

Why it matters

The organisers frame it around decision makers in enterprises, hospitals and schools who must judge whether an AI system fits their use with limited evaluation resources. A good predictor could show how much testing is enough, and where a few labels buy the most certainty.

What we built

Our predictor has two parts. One earns its keep far more than the other.

The offline prior is a weighted logistic ridge model trained on the 167 public binary benchmarks in the organisers' database. It sees hashed subject attributes (name, provider, harness, effort, year) and hashed text features of the item. This is all that is available at zero labels.

The test-time update takes whatever labels have been revealed for a benchmark and adjusts the prior. It fits four small corrections: a shift for the benchmark, a slope on the subject's ability, an offset for the particular subject-item pair, and an offset for the item itself. Each is shrunk toward zero, and the fit is a few Newton sweeps. At budget 0 it does nothing and the prior speaks alone. The whole thing ships as a small zip of plain Python and three weight files, with no dependencies beyond the standard library.

Our numbers

We cannot run the organisers' scorer, so we built a harness. It holds out whole benchmarks in five folds, trains the prior only on the rest, and replays the label-reveal protocol on the held-out benchmark. These are our numbers on our harness, not the official score. Brier ALC, mean ± spread over the five folds, ten subjects per benchmark:

The entry beats the starter baseline on all five folds.

Two cautions. First, we tuned the update's shrinkage settings on these same evaluation folds, so 0.1953 is slightly optimistic. Two other settings, run with those values fixed, give 0.2094 (three subjects per benchmark) and 0.2020 (ten subjects with independent item pools). Second, we do not know how the platform picks subjects, or whether they share items. Those settings bracket the possibilities; they do not settle them.

The platform's number. Our first submission, in the competition's formative phase, scored 0.1759 on the platform's primary Brier metric, with a calibration error of 0.1509. Formative-phase scores do not go on the leaderboard. The platform's number is better than our harness's, and we do not yet know why; the likeliest explanation is that the platform's subject setting is easier than the ones we bracket above. Until we can show that, treat 0.1759 as the platform's reading of this entry, not as evidence the harness is too strict.

What did not work

The text prior adds nothing. Removing the item-text features moves the score from 0.1953 to 0.1954, which is noise. Training the text weights more weakly makes the zero-label case worse: 0.253 to 0.257 against 0.247. Text features learned on other benchmarks do not carry over to a new one.

The subject prior matters only at zero labels, where it takes the error from 0.259 down to 0.247. Everything else comes from the test-time update. Per-benchmark tuning, variance shrinkage and text-kernel smoothing did not beat the fixed settings by a meaningful margin.

What we have not done

We use only the 167 benchmarks with binary responses and leave out 111 graded ones, because we do not know how the organisers convert them to 0/1. We have not tested on unseen subject identities. Our label-reveal rule matches the organisers' in distribution, not exactly. We wrote no acquisition function, so labels are revealed at random. The organisers' own checker accepts the package.

The pattern, if you are entering something similar

• Build the local harness first, and make it mimic the protocol, including the cold start.
• Put a do-nothing baseline and the organisers' baseline in every table.
• Say which settings you tuned on your evaluation folds, and report numbers from settings you did not.
• Publish the negative results.

Independent, personal project. Not affiliated with or endorsed by Stanford, the AIMS lab or NeurIPS.