The 4B beat the 9B on a Turing Pi RK1

On one Turing Pi RK1, the smaller model finished our frozen coding-and-tool task set with 36 of 36 verified passes. The larger model finished with 33 of 36. The 4B also generated tokens almost twice as fast and used roughly half the peak process memory.

This is a result about two deployed model artifacts on one board—not a universal ranking of parameter counts.

Cite: Preprint · DOI 10.5281/zenodo.23720938 · Paper and BibTeX.

What ran

The hardware was a 32 GB Turing Pi RK1 with the RK3588 NPU. Both models used W8A8 quantization, three NPU cores, and the rknn-llm 1.3.1 runtime, with the existing 0.9.7 driver.

The contenders were Qwen3-4B-Instruct-2507 and Qwen3.5-9B. That distinction matters: this was not a controlled experiment varying only parameter count. They are different model releases, with different conversion histories.

Both ran at a 2,048-token context limit and a 512-token generation cap. Decoding used top-k 1, with identical remaining sampling settings. Each task started a fresh model process with no history or cache reuse. We used the 4B artifact's embedded chat template and explicit non-thinking framing for the 9B, following its source template.

The 9B conversion came from a public RK3588 artifact repository, and its file hash matched the published artifact. Its conversion card records toolkit 1.3.1 and optimization level 1. The existing 4B artifact's hash was also checked; its loader reports toolkit 1.2.4b3. Its original conversion repository, upstream revision, and optimization level remain unknown.

Freeze the questions; execute the checks

Before comparison inference, we froze six coding tasks, six tool-selection tasks, their prompts, and their assertions or answer keys. Each model received three fresh-process repetitions of each task: 36 trials per model, 72 total.

The coding tasks covered stable deduplication, interval merging, key-value parsing, list rotation, balanced delimiters, and time arithmetic. We executed the returned Python against assertions, including empty inputs, boundary cases, and input-mutation checks. Passing meant the code ran correctly—not that the answer sounded plausible. The harness stripped at most one surrounding Python fence and did not repair code.

The tool tasks used restricted, real OpenClaw parameter schemas. They tested file reading, memory search and retrieval, skill loading, web search, and an impossible request: power off the board with no power-control tool available. That last task required the exact refusal sentinel.

Tool responses were not dispatched to live tools. The harness checked whole-response JSON validity, selected tool name, exact arguments, and whole-answer equality against frozen keys. Both models refused the unavailable operation correctly in all three repetitions.

The results

Timing and throughput values below are medians across all completed task trials. Peak RSS is the maximum process resident memory observed. Time to first token and inference time exclude initialization; wall time includes it.

Both models passed every coding trial. The entire score gap came from one tool task, repeated three times.

The larger model's failure

The request explicitly named the read tool and asked for lines 10 through 14 inclusive. The correct arguments were offset: 10 and limit: 5.

The 9B chose memory_get instead, with from: 9 and lines: 5. It produced the same answer in all three repetitions. These are verbatim excerpts of the failure; the intervening path field is omitted to avoid exposing an internal path:

{
"name": "memory_get",
"arguments": {

"from": 9,
"lines": 5
}
}

The JSON was valid. The selected tool was wrong, and the starting line was off by one. A JSON-only validator would have missed the failure entirely.

DDR: check during inference, not just before and after

The original before-and-after snapshots showed DDR at 528 MHz. Those snapshots did not measure clocks while inference was running, so we followed up with one additional 4B trial of the frozen stable-deduplication task, using the same unchanged runner.

The DMC governor was dmc_ondemand. Available frequencies were 528, 1,068, 1,560, and 2,112 MHz; minimum and maximum were 528 and 2,112 MHz. We sampled DDR and NPU clocks every 200 milliseconds without changing any settings.

DDR ramped to 2,112 MHz during inference. All 69 samples between the runner's initialization-complete and inference-complete markers showed DDR at 2,112 MHz and the NPU at 1,000 MHz. The code passed its assertions, and native generation throughput was 3.43 tokens per second. This diagnostic trial is separate from the results table.

The idle snapshots therefore do not justify calling the published throughput numbers DDR-limited at 528 MHz. We did not continuously sample the original 72 trials, and this single follow-up does not reconstruct their active clocks or prove the absence of memory-bandwidth limits. It does show that the board's DDR governor ramps under this workload. Governor and frequency bounds were unchanged before and after the check; no driver or kernel changes were made.

Why this matters—and what it does not prove

The practical lesson is to benchmark the deployed artifact against the work it must actually do. A larger model can be slower and still miss an exact operational requirement that a smaller one follows.

There is useful context in Tool-Calling LLM Agents Under Realistic MCP Faults: Scale Does Not Purchase Fault Tolerance. That paper studies realistic tool failures, two different backbone sizes, and multiple agent stacks. Its finding that scale did not improve fault tolerance in those tested configurations is a reason to measure operational behavior rather than assume it from model size.

Our experiment is not a replication of that paper. We did not inject MCP faults or run an end-to-end live-tool agent loop. We tested code execution and offline tool-choice correctness.

The limits are important:

• One board, one frozen task set. These small tasks are not a broad coding or agent benchmark.
• Three deterministic repeats. They demonstrate repeatability here, not independent statistical samples.
• Different model families and conversion histories. Parameter count is not the only changing variable.
• Unknown 4B conversion provenance. Its runtime identity and hash are known, but the original conversion recipe is not.
• Restricted schemas and exact answers. Passing does not establish reliability across arbitrary tools or multi-step workflows.
• Clock conditions matter. Throughput describes the measured board configuration, not the hardware's maximum possible performance.

For this task set, on this RK1, the 4B won. That is enough to choose the smaller artifact for this workload—and not enough to declare that smaller models always win.