A model that checks “done”, and the 140 lines that keep it honest

The most expensive sentence an agent can write is "done." It costs nothing to type, and if it is wrong, somebody finds out later, usually at a worse time.

So I built a small thing that reads a claim ("deployed the stack, tests pass") and the evidence pasted under it, and returns one of three verdicts: supported, partial, or unsupported. It runs on one Bedrock model behind one Lambda function. It is open source under MIT: cloudlabworks/bedrock-done-claim-checker.

The model call itself is six lines. Everything else is harness.

What the harness does before the model sees anything

It refuses to ask the model questions a regex can answer. If the evidence is empty, the verdict is "unsupported" and the model is never called. If the evidence has no concrete artifact (a commit hash, a URL, a timestamp, command output, a number), that gets noted. If most of the words in the claim never appear in the evidence, that gets noted too. These checks are cheap, deterministic, and impossible to talk out of their answer.

It counts before it spends. A single counter per day, with a hard cap. The check and the increment happen in one atomic update, so two requests at the same moment can't both slip under the line. Past the cap, the answer is "try tomorrow," and again, the model is never called. Concurrency is pinned to one. Input sizes are capped, and so is output length.

It uses one model, chosen at deploy time. No routing, no fallback chain, no "try the bigger one if the small one is unsure." Swapping the model is a redeploy with a different parameter. That makes every result traceable to exactly one model, which is the property you want when you are trying to learn whether a model is good enough.

What the harness does after the model answers

It accepts only the shape it asked for. The prompt asks for JSON with a verdict from a fixed list, a list of missing items, and one sentence of reason. If the reply is anything else (prose, a made-up verdict like "probably," a list of numbers where strings belong), the result is "unparseable." It is never passed along as a verdict. A friendly "Looks done to me!" is the most dangerous reply a checker can give, so it gets treated as no answer at all.

It lets the cheap checks overrule the expensive one. If the model says "supported" but the deterministic checks flagged the evidence, the verdict becomes "partial," and the reason says why. The model can make a verdict weaker. It cannot make one stronger than the evidence allows.

What it is not

It is not a judge of truth. It reads text someone pasted. If the pasted evidence is fabricated, the checker will grade the fabrication. What it does is make the cheap failure modes (no evidence, vague evidence, evidence about something else) visible every time, without a person reading every report.

It is also small on purpose. One template, one handler, ten unit tests with fake AWS clients so they run anywhere without an account. Teardown is a script, and it confirms the stack and its log group are gone rather than assuming so.

Status, plainly

The offline tests pass. The template lints clean. It has not run against a live AWS account yet. A live run is planned, privately, with the URL locked to signed requests. When it happens, this post gets the results: which model, how many claims, how often the deterministic checks and the model disagreed.

Why bother

Because the interesting question about a model isn't "is it smart?" It's "what is it allowed to decide, and what catches it when it is wrong?" Those answers live in the harness. Here, the model decides one thing: whether evidence that already passed the cheap checks actually supports the claim. Everything around that decision is ordinary code that you can read in one sitting.

Independent project by CloudLab Works; not affiliated with or endorsed by AWS.