Harness engineering · Part 1

Your tool exited 0. It did nothing.

What a misbehaving RGB controller taught me about designing tools for language models.

Last month I spent an evening debugging RGB lighting on my desktop. This is not a serious problem, and I knew that at the time. But the bug turned out to be one I keep meeting in production LLM systems wearing a different hat, and it explained something I’d been struggling to articulate about tool design for agents.

The setup: a workstation with addressable RGB across the motherboard headers, RAM, and an AIO cooler. I wanted the colors driven from my wallpaper, so a script derives a palette and pushes it to every controller. Straightforward.

Except the lighting kept coming out wrong, and every command I ran reported success.

The CLI that always succeeded

Here is the comment I eventually left at the top of the rewritten script, once I understood what had been happening:

Replaces the old apply.sh, which could never have worked: it drove zone 3 (the WiFi antenna, not a fan header), in ‘static’ mode (single-colour-only), at 40 LEDs (the chain is 45), through the CLI (which exits 0 without doing anything). All four were wrong, and none of them reported an error.

Four independent bugs. Zero error messages. The script had been “working” for weeks, in the sense that it ran to completion every time.

The root cause is a tool contract problem. The openrgb CLI exits 0 whether or not it did anything — an unknown device name, a zone that never got resized, a mode that doesn’t support per-LED color, all of these “succeed” silently. The return value carried no information about whether the tool had done its job.

There was a second, nastier layer underneath. OpenRGB’s zone map for this motherboard is wrong: in its Gigabyte driver, the fourth addressable header and the single-LED WiFi antenna share a header id — a genuine upstream collision. So writing per-LED color to the zone named ARGB_V2_4 lights up the WiFi antenna and never touches a fan. The name describes something that isn’t there.

The zone table in the rewrite carries that as a standing warning, because it is not the kind of thing you rediscover cheaply:

# The WiFi antenna is header id 8 -- the SAME id OpenRGB gives to ARGB_V2_4.
# So we drive the antenna AS zone 3, because that is physically what it is.
MOBO_FANS = {
    0: (3 * FAN_LEDS, "bottom + side trios (mirrored)"),   # ARGB_V2_1 / D_LED1
    1: (RADIATOR_FANS * FAN_LEDS, "radiator fans"),        # ARGB_V2_2 / D_LED2
    2: (2 * FAN_LEDS, "rear fans"),                        # ARGB_V2_3 / D_LED3
}

A tool that lies about what it did, on top of a map that lies about what exists. I was reasoning confidently from both.

The same bug, one abstraction up

Swap the actors and this is a story about an agent.

A language model calling a tool is in a strictly worse epistemic position than I was that evening. It can’t see the fans. It has no peripheral vision, no “hm, the case still looks blue.” All it has is the string your tool returned. If that string says success, the model will believe it, fold it into its state, and build every subsequent decision on top of it.

This is why the most dangerous tool in a harness isn’t the one that errors. It’s the one that returns success without doing anything. An error is information — the model can retry, escalate, route around it. A false success is corruption, and it propagates: the model reports completion to the user, moves to the next step, and the transcript now contains a confident claim that something happened when it did not.

I’ve seen the production version of this more than once. A search tool that returns an empty result set identically whether the index is empty, the query was malformed, or the backend timed out. A file writer that silently no-ops on a path outside its sandbox. A database tool that swallows a constraint violation and returns {"status": "ok"}. In each case the model does exactly what I did with apply.sh — reasons fluently, and at length, from a premise that was never true.

The most expensive version I’ve run into had no model in it at all, which is rather the point: the pattern is older than agents, and agents just remove the last human who might have noticed. A retention sweep — a legal deletion obligation, on a clock — was implemented as a workflow on an orchestrator that, it turned out, had never been deployed to any environment. The code existed. The schedule existed. Every review that asked “is retention handled?” found an implementation and said yes. Nothing ever ran it. The gap between implemented and executing had no signal attached to it, so it stayed open until someone went looking for the wrong reason.

What the rewrite actually changed

The fix wasn’t cleverer code. It was refusing to trust the tool’s own account of itself. Four changes, each with a direct analog in harness design.

Speak the real protocol, not the convenience wrapper. I dropped the CLI and talked to the OpenRGB SDK socket directly, because the socket protocol lets you read state back and the CLI doesn’t. In harness terms: prefer the interface that can tell you what happened over the one that’s easier to call. Ergonomics that cost you observability are a bad trade.

Verify preconditions before acting. Every device is now checked present before anything is written to it. A tool that assumes its target exists will report success against a target that doesn’t. For an agent, this means tools should fail loudly on a missing resource rather than treating absence as a no-op.

Read the state back. apply.py --verify applies the palette, then asks the hardware what it actually holds:

if args.verify:
    print("\nreadback:")
    for i in range(c.count()):
        name, modes, active, zones, _ = c.controller(i)
        print(f"  [{i}] {name:<34} mode={modes[active]:<8} zones={zones}")

Eight lines, and they are the difference between a tool that reports intent and one that reports fact. The return value is derived from observed state, not from the fact that a write didn’t throw. This is the single highest-leverage change you can make to an agent tool: return what is now true, not what you attempted.

Ground the map in measurement. The zone map in hardware.py carries a warning to future me: every number was established empirically, and none of them should be “fixed” from a datasheet. The datasheet describes what the hardware was supposed to be. The agent equivalent is that your tool descriptions should describe what the system actually does, including its known lies — not what the API documentation claims.

Reconcile, don’t apply deltas. The whole thing is idempotent: it declares the full target state and drives the hardware to it, so recovering from suspend is just running it again. Agent harnesses need this badly, because retries are the normal case rather than the exception. A model will call your tool twice. It will call it again after a timeout that actually succeeded. If your tool reconciles toward a declared state, none of that matters. If it appends, increments, or sends, all of it matters a great deal.

The whole thing is about six hundred lines of Python driving one specific motherboard. It will not run on your machine — the zone counts are this board’s, established empirically — but the shape is the point.

Where this series is going

I’ve spent the last two years building evaluation infrastructure and data pipelines for LLM systems, and the failure modes that actually cost me time were almost never in the model. They were in the layer around it: what the model can see, what it’s allowed to do, what it’s told happened, and what it does when something goes wrong.

That layer doesn’t have a settled name yet. I’ve been calling it the harness. It’s the agent loop and its termination conditions, the tool schemas, the context that survives compaction, the sandbox, the evals that tell you whether a change helped, and the telemetry that lets you reconstruct a run after it goes sideways.

I want to work through it the way you’d work through any other system: bottom-up, one layer at a time, paying attention to what breaks. Next: the bare API call, and the surprising number of things you have to decide before you have a loop at all.