An open-weight model topped a hacking leaderboard. The harness is why.

This week Tenzai published a quarter of results from their AI security agent on HackerOne. For the whole quarter they ran it on GLM 5.2, an open-weight model from Z.ai. The quarter before, they ran the same agent on Claude Opus 4.6. Their claim: the same mix of severities, the same variety of attack types, the same share of false positives, and roughly 30% cheaper per report. Over 300 reports to 94 organizations, 3.5% false positives, 91% fully validated. (Their write-up.)

Those are a vendor's numbers about its own product, and the method isn't published. Read them that way. But the sentence that matters in the post isn't a number:

"Once open weight models clear some threshold, the rest of the harness comes into play."

I agree, and I can show why from my own lab, without a leaderboard.

What a harness is

The model proposes. The harness decides what the model is allowed to touch, checks what actually happened, and decides what a person must approve. Tenzai's version, as they describe it:

• Scope becomes code. Program rules are turned into deterministic guardrails before any scan starts.
• Every finding is validated. Reports come with reproduction scripts, deduplicated and prioritized.
• A human gates the submission. The agent researches on its own. A person decides what goes out.

None of those three depend on which model sits in the middle. That's the point. If the model is a part you can swap, the harness is the product.

Four mistakes, one morning

I run an agent that does real operations work in my lab: mail, DNS, cloud accounts, browser tasks. This morning it made four mistakes worth writing down. The model didn't catch any of them. The harness caught all four.

1. The change that left mail with nowhere to go. The agent moved a domain's mail to a new provider. Its script removed the old mail records and then failed to create the new ones, a plain bug in how it looped over the values. For about 45 seconds the domain had no mail route at all. What caught it was a rule, not intelligence: after every DNS change, read the live records back from a public resolver and compare. The read-back showed zero mail records, a fix script ran, and the records were confirmed on two public resolvers. The fix script also carried a fallback: if the new records failed again, it would switch the old provider back on. A backup of every record had been taken before the first change.

The model wrote the bug. The harness made it a 45-second event instead of a lost afternoon.

2. The form that filled in its own name. Creating a mailbox, the agent typed a name and a username. The web form helpfully rewrote the username from the name fields before the agent clicked Add. The account came out with the wrong address. The fix was easy. The lesson went straight into the harness: read every field back right before submitting, and refuse to submit if it doesn't match.

3. Two hands on one browser. I was signed in to a cloud console, and the agent was working in the same window. We were both clicking. A password got set twice on my side before the agent's step ran, and for a few minutes nobody knew which value was live. The harness rule now: one writer per session, and anything unproven gets proven by signing in, not by trusting that a step "ran".

4. The reports that arrived late. Several background jobs reported "fail" well after their work had been checked and confirmed done. Others reported "done" for work that didn't happen. The harness ignores both. A job's report is never the evidence. The evidence is the system: the DNS record on a public resolver, the user in the admin console, the email in the inbox.

The parts that matter

Strip it down, and a harness is five habits that any model can sit inside:

The last row is where most setups go wrong in one of two directions. Gate everything, and the human starts clicking yes without reading. Gate nothing, and the first bad day is very bad. In my lab the line is two questions: can it be undone, and does it leave the lab? Only a "no" and a "yes" together wait for me.

Why this matters more as models get cheaper

Tenzai's broader point is uncomfortable and, I think, right: if an open-weight model you can run on private hardware is good enough to find serious bugs, the people attacking your systems have it too. A third of their findings had already been reported by someone else, often months earlier, and were still open.

The same shift applies to anyone putting agents into operations. The model is becoming a commodity you can swap on price. What you can't buy off the shelf is the layer that knows your systems: what "done" means for a DNS change, which steps can be undone, which actions need a human. That layer is where the reliability lives, and it is where the work is.

I've published pieces of mine as open source: prove-it-done checks an agent's completion claims against evidence, Agent Trace Clinic reads what an agent actually did, and MCP Failure Lab breaks tool calls on purpose to see what the agent does next. The newest, the Bedrock Done-Claim Checker, wraps a single cloud model in exactly this kind of harness.

Swap the model whenever a better one shows up. Keep the harness.