How Harness² works
How an agent can improve its own setup on a task it has never seen, with no answer key and without retraining the model. Built up one step at a time.
Harness² ships with a tiny offline demo, and this page follows it. One agent has one job: total a ledger.
- A fixed modelIts weights never change.
- A hidden bugThe starting setup quietly drops negative amounts.
- No answer keyNothing will say which total is right.
- More tasks comingA second ledger follows. Lessons should carry over.
Each layer is a short animation. Use Next, or click a figure and press ← →.
A model turns text into more text
Start at the bottom. A language model reads some text and writes what should come next. That's all it does.
Give it text, get text back. That's the whole interface.
Ask it to total our ledger and it gets stuck. It can't open a file or run code. All it can do is write.
Harness² never changes the model. Its weights stay frozen the whole time. Every improvement on this page comes from changing what's wrapped around it.
An agent is a model in a loop with tools
Wrap the model in a small program. When the model writes a command, the program runs it and hands the result back.
The loop shows the model the task and waits for its reply.
The model replies with an action: run the summary script on the ledger. The loop runs it in a shell.
The shell prints a result. The loop adds it to the model's input, and the model reads it.
The model decides it's done and writes the answer file.
The log of actions and results is the trajectory. The file is the deliverable. One full run, start to finish, is a rollout.
The harness is a folder of plain files
Why did the model run summarize.py? Because a file told it to. Apart from the task itself, everything the loop gives the model comes from one folder.
systemprompt.mdsays to use the script. The loop adds this file to the model's input on every call.The script is a file in the same folder. Read it closely. It matters soon.
A full harness has eight kinds of file. Four go into the model's input on every call. Skills load when relevant, subagents are helper agents, scripts run in the shell, and plugins wrap every action the agent takes.
This folder is the harness. Change a file and the same frozen model behaves differently. Improving an agent by editing this folder is called harnessing.
On a new task, nobody hands you the answer
The usual way to tune a harness: try an edit, score it on tasks with known answers, keep whatever scores higher. Harness² is built for tasks nobody prepared answers for. So run the starting folder, the base harness, and look at what comes back.
The base harness reports Total: 105.25.
Check by hand: 100.25 − 20.10 + 5.00 = 85.15. The script's
if a > 0line skipped the −20.10 credit.We only know that because we can see everything. The process that improves the harness has no answer key and no score. The benchmark's grader exists, but it stays locked away until every decision is made.
What it does have: the task, the source file, and every run's log and output. A careful reader can check a total against three rows without any key.
So Harness² makes more runs on purpose, each with an edited folder, and learns from how they differ.
Propose: two editors each change a copy
An editor is another model session, set up to improve a harness instead of doing the task. Harness² runs two of them, each with its own job.
Each editor gets a workspace: the task, the source file, the base run's full log and output, and a copy of the harness to edit in place. Its
library/folder is empty for now.INV owns how the agent works: steps, tools, scripts, checks. DOM owns what an expert in the field knows. Each gets its own copy and stays out of the other's job.
INV reads the log. The script printed 105.25, but the rows add up to 85.15. Cheapest fix: delete the filter. Its note: “Keep signed amounts: the base total omitted credits.”
DOM writes down what an accountant knows, as a skill the agent loads when it's relevant: credits are negative, and a total must match its rows.
Every edit must point to a moment in a log. If an editor can't name the moment, the edit is a guess. A broken edit gets one retry, then falls back to the unchanged folder. Changing nothing is always allowed.
Probe: run each copy on the same task
Each edited folder is a candidate. Each one gets a single run, a probe, on the same ledger. The model, the task and the file are identical. Only the folder differs.
Three rollouts side by side: the base folder and the two candidates.
base and inv1 do exactly the same thing until the script prints its total. That's where they diverge.
The two folders differ by one deleted line, so that line caused the different total. A side-by-side difference like this is a behavioral contrast. It's evidence about what the edit did, and it needed no answer key.
dom1 still has the buggy script, so it sees 105.25 too. Its new skill makes it check the rows, catch the mismatch and fix the total. Same answer, longer path.
Compose: keep what the runs support
One more editor session, the composer, reads everything so far and writes a single final folder.
The composer reads all three runs, both edited folders, and the editors' notes.
It has three legal moves: merge edits from several candidates, adopt one candidate as-is, or keep the base unchanged. An edit the evidence doesn't support only adds clutter.
Both edits earned their place, so it merges them. Every decision gets a note that points at a run.
The merged folder is the final, called
agg1. It gets its own rollout: Total: 85.15.With a budget of k = 1 there's one final, so it's delivered. Only now does the benchmark's grader score it, to measure the method. Nothing the grader says flows back.
Spend more: go wider or go deeper
Propose, probe and compose together make one step, and one step is a budget of k = 1. A bigger k buys more, in one of two shapes.
Parallel. Run the base folder twice. The model is random, so a failure in both runs is a real flaw, while a failure in only one might be bad luck.
Two INV and two DOM editors work from the same evidence without seeing each other. Probe all four. Here
dom2's rule never kicked in, so it still says 105.25.Two composers each read all four candidates and write a final, working separately. Each final gets a rollout.
A selector picks which output to deliver. The outputs are renamed A and B in shuffled order, with folders and logs hidden. It checks each one against the request and the source file.
Both totals match the ledger, but A adds a line the request didn't ask for. The selector picks B and writes down why.
Cost: 2 base + 4 probes + 2 finals = 8 rollouts, and 4 editors + 2 composers = 6 editor sessions. It's always 4k and 3k, plus the selector.
Sequential. Round 1 is the k = 1 step you already know. It ends in a checkpoint,
c1.Round 2 starts from
c1, not the original folder. Its editors also see everything round 1 tried, includingc1's own run.Round 2's composer writes
c2. It can build onc1, bring back an edit dropped earlier, or keepc1if round 2 made things worse.The finals are
c1andc2. The selector picks one blind, as before.Same budget: 8 rollouts and 6 editor sessions. Parallel tries more ideas at once. Sequential keeps building on its own last attempt.
Remember: carry one lesson to the next task
Everything so far happened inside one task. The paper calls that task-level recursion. The second loop runs across tasks: after each one, Harness² writes down what the evidence showed, for the editors of the next task.
A reflector reads the full record of the task: the base folder, every candidate, every run, and the selector's reason.
It compares each candidate's folder with the base and writes one row per changed file: what changed, and what that did to the run.
Each row gets a verdict. “Confounded” means several changes were tangled together, so no run isolates one. Every candidate gets rows, not just the winner: the selector judged outputs with the folders hidden, so its pick doesn't say which edit worked.
Then at most one lesson that should transfer: when it applies, what to do, and the evidence. If the rows don't support one, it writes
NO_ENTRYinstead.The lesson and the full adopted folder go into the library for this domain, accounting.
Task 2 is a new ledger. It starts from the original base folder, bug included. The base run says 35.50. The right total is 20.25.
This time the editors' workspace has
library/filled in: an index of earlier accounting tasks, each with its folder and lesson.The INV editor finds the cue, a ledger with credits and debits, and applies the sign fix right away. It still checks the fix against this task's own run: an old lesson is a hypothesis here, not a fact.
Its probe totals 20.25. The right fix arrived on the first try.
That's the second loop, harnessing-level recursion. The folder the agent runs with is rebuilt for every task. The library the editors read, the improvement harness, keeps growing. The model never changed.
The whole loop
One task from start to finish, then the hand-off to the next.
Base runs. Run the unchanged folder k times on the new task.
Propose. INV and DOM editors each edit a copy, citing moments in the runs and reading the library.
Probe. Run each candidate once on the same task, and compare the runs.
Compose. Merge, adopt one candidate, or keep the base, then run the final.
Pick. With k > 1, choose one final's output blind. Only then does the grader score it, to measure.
Remember. Verdicts and one lesson go into the library, and the next task begins.
You can't score an edit without an answer key. You can still watch what it changes, and keep what the runs support.
Does it work?
In the paper, a single step (k = 1) improved GLM-5 and Gemini 3.5 Flash over their own base harnesses on three benchmarks of professional work: up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 percentage points on LAB's partial pass rate. A bigger k brought further gains.
Who sees what
Nobody inside the method ever sees a score. The grader runs last, outside it.
| Role | Folders | Run logs | Outputs | Answer key |
|---|---|---|---|---|
| INV and DOM editorsEdit a copy of the folder. | yes | yes | yes | no |
| ComposerMerges candidates into a final. | all of them | yes | yes | no |
| SelectorPicks one final's output. | hidden | hidden | renamed A, B… | no |
| ReflectorWrites verdicts and a lesson. | yes | yes | yes | no |
| GraderMeasures the delivered output, after everything. | no | no | the chosen one | yes |
Run the ledger yourself
The demo runs both modes on the two ledgers with scripted editors, so it needs no model or API key. From a clone of the repo:
$ uv venv --python 3.12 .venv && source .venv/bin/activate $ uv pip install -e '.[render]' $ harness2-demo --output runs/demo --k 2
Glossary
- harness
- The folder of files around a model: prompts, memory, skills, scripts, subagents and plugins.
- harnessing
- Improving an agent by editing its harness instead of its model.
- rollout
- One full run of the agent on a task.
- trajectory
- The log of a rollout's actions and results.
- deliverable
- The file a rollout produces.
- base harness
- The starting folder, unchanged.
- candidate
- An edited copy of the folder.
- probe
- One rollout of a candidate on the same task.
- INV
- Edits to how the agent works: steps, tools, scripts, checks.
- DOM
- Edits to what an expert in the field knows: rules, required parts, checks.
- composer
- The session that turns candidates into one final folder.
- selector
- Picks one final's output, blind to how it was made.
- reflector
- Writes per-edit verdicts and one lesson after each task.
- library
- The per-domain store of adopted folders and lessons. Also called the improvement harness.
- k
- The budget: 4k rollouts and 3k editor sessions per task.
- KEEP_BASE
- Choosing to leave the folder unchanged.