← Explainers
Explainer · AI agents

How Harness² works

How an agent can improve its own setup on a task it has never seen, with no answer key and without retraining the model. Built up one step at a time.

The running example

Harness² ships with a tiny offline demo, and this page follows it. One agent has one job: total a ledger.

TASK.MD Sum every signed amount in ledger.csv. Return Total: with two decimal places. LEDGER.CSV row 1100.25 row 2−20.10 row 35.00 Total: ?
  • A fixed modelIts weights never change.
  • A hidden bugThe starting setup quietly drops negative amounts.
  • No answer keyNothing will say which total is right.
  • More tasks comingA second ledger follows. Lessons should carry over.

Each layer is a short animation. Use Next, or click a figure and press ← →.

Layer 0 · the model

A model turns text into more text

Start at the bottom. A language model reads some text and writes what should come next. That's all it does.

FIG 0 · text in, text outStep 1 of 3
TEXT IN TEXT OUT model What is 2 + 2? 4 Total the amountsin ledger.csv I can't openfiles. Paste thenumbers here? WEIGHTS FROZEN
  1. Give it text, get text back. That's the whole interface.

  2. Ask it to total our ledger and it gets stuck. It can't open a file or run code. All it can do is write.

  3. Harness² never changes the model. Its weights stay frozen the whole time. Every improvement on this page comes from changing what's wrapped around it.

Layer 1 · the agent loop

An agent is a model in a loop with tools

Wrap the model in a small program. When the model writes a command, the program runs it and hands the result back.

FIG 1 · one run on the ledgerStep 1 of 5
THE LOOP model SHELL $ _ $ python … Total: 105.25 action result action result TRAJECTORY [ACTION] python scripts/summarize.py ledger.csv [OBSERVATION] Total: 105.25 [ACTION] write answer.txt DELIVERABLE answer.txt Total: 105.25 trajectory + deliverable = one rollout
  1. The loop shows the model the task and waits for its reply.

  2. The model replies with an action: run the summary script on the ledger. The loop runs it in a shell.

  3. The shell prints a result. The loop adds it to the model's input, and the model reads it.

  4. The model decides it's done and writes the answer file.

  5. The log of actions and results is the trajectory. The file is the deliverable. One full run, start to finish, is a rollout.

Layer 2 · the harness

The harness is a folder of plain files

Why did the model run summarize.py? Because a file told it to. Apart from the task itself, everything the loop gives the model comes from one folder.

FIG 2 · inside harness/Step 1 of 4
HARNESS/ systemprompt.mdevery call guardrails.mdevery call LongTermMEMORY.mdevery call tool_guidance.mdevery call skills/<n>/SKILL.mdon demand sub_agents/<n>/AGENT.mddelegated scripts/summarize.pyvia shell plugin/<n>.tsinside the loop SYSTEMPROMPT.MD Reconcile the ledger using python scripts/summarize.py. SCRIPTS/SUMMARIZE.PY total = 0 for a in amounts: if a > 0: total += a print("Total:", total) simplified from demo.py WHEN IT'S READ every callalways in the model's input on demandloaded when relevant delegateda helper agent, own context via shellcode the agent runs inside the loopwraps every action it takes HARNESSING Edit any file here and the same frozen model acts differently. Editing this folder to improve the agent is harnessing.
  1. systemprompt.md says to use the script. The loop adds this file to the model's input on every call.

  2. The script is a file in the same folder. Read it closely. It matters soon.

  3. A full harness has eight kinds of file. Four go into the model's input on every call. Skills load when relevant, subagents are helper agents, scripts run in the shell, and plugins wrap every action the agent takes.

  4. This folder is the harness. Change a file and the same frozen model behaves differently. Improving an agent by editing this folder is called harnessing.

Layer 3 · the problem

On a new task, nobody hands you the answer

The usual way to tune a harness: try an edit, score it on tasks with known answers, keep whatever scores higher. Harness² is built for tasks nobody prepared answers for. So run the starting folder, the base harness, and look at what comes back.

FIG 3 · the base runStep 1 of 5
LEDGER.CSV 100.25 −20.10 skipped 5.00 base harness ANSWER.TXT Total: 105.25 100.25 − 20.10 + 5.00 = 85.15 THE IMPROVER CAN SEE ✓ the task ✓ the source file ✓ every run's log and output IT CAN'T SEE ✕ an answer key ✕ a score ✕ the grader's verdict
  1. The base harness reports Total: 105.25.

  2. Check by hand: 100.25 − 20.10 + 5.00 = 85.15. The script's if a > 0 line skipped the −20.10 credit.

  3. We only know that because we can see everything. The process that improves the harness has no answer key and no score. The benchmark's grader exists, but it stays locked away until every decision is made.

  4. What it does have: the task, the source file, and every run's log and output. A careful reader can check a total against three rows without any key.

  5. So Harness² makes more runs on purpose, each with an edited folder, and learns from how they differ.

Layer 4 · propose

Propose: two editors each change a copy

An editor is another model session, set up to improve a harness instead of doing the task. Harness² runs two of them, each with its own job.

FIG 4 · INV and DOM editorsStep 1 of 5
WORKSPACE/ task.md source/ledger.csv rollouts/base1.md deliverables/base1.md harness/edit in place library/empty for now MISSION.md INV · how the agent works DOM · what an expert knows harness/ harness/ scripts/summarize.py for a in amounts: −if a > 0: −total += a +total += a + skills/ledger_check/SKILL.md Credits are negative. Before finishing, check the total equals the signed sum of every row.
  1. Each editor gets a workspace: the task, the source file, the base run's full log and output, and a copy of the harness to edit in place. Its library/ folder is empty for now.

  2. INV owns how the agent works: steps, tools, scripts, checks. DOM owns what an expert in the field knows. Each gets its own copy and stays out of the other's job.

  3. INV reads the log. The script printed 105.25, but the rows add up to 85.15. Cheapest fix: delete the filter. Its note: “Keep signed amounts: the base total omitted credits.”

  4. DOM writes down what an accountant knows, as a skill the agent loads when it's relevant: credits are negative, and a total must match its rows.

  5. Every edit must point to a moment in a log. If an editor can't name the moment, the edit is a guess. A broken edit gets one retry, then falls back to the unchanged folder. Changing nothing is always allowed.

Layer 5 · probe

Probe: run each copy on the same task

Each edited folder is a candidate. Each one gets a single run, a probe, on the same ledger. The model, the task and the file are identical. Only the folder differs.

FIG 5 · base vs inv1 vs dom1Step 1 of 4
base run script 105.25 write file folder: original inv1 run script 85.15 write file folder: 1 line removed ≠ DIVERGES HERE dom1 run script 105.25 check rows 85.15 write
  1. Three rollouts side by side: the base folder and the two candidates.

  2. base and inv1 do exactly the same thing until the script prints its total. That's where they diverge.

  3. The two folders differ by one deleted line, so that line caused the different total. A side-by-side difference like this is a behavioral contrast. It's evidence about what the edit did, and it needed no answer key.

  4. dom1 still has the buggy script, so it sees 105.25 too. Its new skill makes it check the rows, catch the mismatch and fix the total. Same answer, longer path.

Layer 6 · compose

Compose: keep what the runs support

One more editor session, the composer, reads everything so far and writes a single final folder.

FIG 6 · three runs in, one folder outStep 1 of 5
base105.25 inv185.15 dom185.15 composer reads all of it MERGE ADOPT ONE KEEP_BASE NOTES kept → scripts/summarize.py → inv1 counted the −20.10 credit kept → skills/ledger_check → dom1 caught 105.25 ≠ the rows FINAL · AGG1 scripts/summarize.py sign fix, from inv1 skills/ledger_check/ row check, from dom1 Total: 85.15 DELIVERED
  1. The composer reads all three runs, both edited folders, and the editors' notes.

  2. It has three legal moves: merge edits from several candidates, adopt one candidate as-is, or keep the base unchanged. An edit the evidence doesn't support only adds clutter.

  3. Both edits earned their place, so it merges them. Every decision gets a note that points at a run.

  4. The merged folder is the final, called agg1. It gets its own rollout: Total: 85.15.

  5. With a budget of k = 1 there's one final, so it's delivered. Only now does the benchmark's grader score it, to measure the method. Nothing the grader says flows back.

Layer 7 · more budget

Spend more: go wider or go deeper

Propose, probe and compose together make one step, and one step is a budget of k = 1. A bigger k buys more, in one of two shapes.

FIG 7a · parallel, k = 2Step 1 of 6
BASE PROBES FINALS SELECTOR base1105.25 base2105.25 inv185.15 dom185.15 inv285.15 dom2105.25 agg185.15 agg285.15 A Total: 85.15 Credits: 1 B Total: 85.15 PICK B B is exactly what was asked; A adds a line. 8 rollouts (4k) · 6 editor sessions (3k)
  1. Parallel. Run the base folder twice. The model is random, so a failure in both runs is a real flaw, while a failure in only one might be bad luck.

  2. Two INV and two DOM editors work from the same evidence without seeing each other. Probe all four. Here dom2's rule never kicked in, so it still says 105.25.

  3. Two composers each read all four candidates and write a final, working separately. Each final gets a rollout.

  4. A selector picks which output to deliver. The outputs are renamed A and B in shuffled order, with folders and logs hidden. It checks each one against the request and the source file.

  5. Both totals match the ledger, but A adds a line the request didn't ask for. The selector picks B and writes down why.

  6. Cost: 2 base + 4 probes + 2 finals = 8 rollouts, and 4 editors + 2 composers = 6 editor sessions. It's always 4k and 3k, plus the selector.

FIG 7b · sequential, k = 2Step 1 of 5
ROUND 1 base2 runs inv1 dom1 c185.15 ROUND 2 c1checkpoint + history inv2 dom2 c285.15 SELECTOR ATotal: 85.15 BTotal: 85.15 PICK A c2, this time 8 rollouts (4k) · 6 editor sessions (3k)
  1. Sequential. Round 1 is the k = 1 step you already know. It ends in a checkpoint, c1.

  2. Round 2 starts from c1, not the original folder. Its editors also see everything round 1 tried, including c1's own run.

  3. Round 2's composer writes c2. It can build on c1, bring back an edit dropped earlier, or keep c1 if round 2 made things worse.

  4. The finals are c1 and c2. The selector picks one blind, as before.

  5. Same budget: 8 rollouts and 6 editor sessions. Parallel tries more ideas at once. Sequential keeps building on its own last attempt.

Layer 8 · remember

Remember: carry one lesson to the next task

Everything so far happened inside one task. The paper calls that task-level recursion. The second loop runs across tasks: after each one, Harness² writes down what the evidence showed, for the editors of the next task.

FIG 8a · after task 1Step 1 of 5
reflector reads the base folder, every candidate, every run, and the selector's reason scripts/summarize.py · inv1 dropped the positive-only filter → the probe's total included the credit skills/ledger_check · dom1 added a row-sum check → the probe caught 105.25 ≠ the rows and fixed it helped helped VERDICTS helped hurt no observed effect confounded ONE LESSON cue: a ledger with credits and debits lesson: include both signs when summing in scripts/summarize.py evidence: the summarize.py row · confidence: high library/ accounting 1 entry lesson + folder
  1. A reflector reads the full record of the task: the base folder, every candidate, every run, and the selector's reason.

  2. It compares each candidate's folder with the base and writes one row per changed file: what changed, and what that did to the run.

  3. Each row gets a verdict. “Confounded” means several changes were tangled together, so no run isolates one. Every candidate gets rows, not just the winner: the selector judged outputs with the folders hidden, so its pick doesn't say which edit worked.

  4. Then at most one lesson that should transfer: when it applies, what to do, and the evidence. If the rows don't support one, it writes NO_ENTRY instead.

  5. The lesson and the full adopted folder go into the library for this domain, accounting.

FIG 8b · task 2Step 1 of 5
LEDGER-2.CSV 35.50 −10.25 −5.00 BASE RUN Total: 35.50 library/ accounting · 1 entry INV editor reads library/INDEX.md cue matches applies the sign fix PROBE Total: 20.25 Execution harness: rebuilt from the base for every task Improvement harness (the library): grows after every task Model: never changes
  1. Task 2 is a new ledger. It starts from the original base folder, bug included. The base run says 35.50. The right total is 20.25.

  2. This time the editors' workspace has library/ filled in: an index of earlier accounting tasks, each with its folder and lesson.

  3. The INV editor finds the cue, a ledger with credits and debits, and applies the sign fix right away. It still checks the fix against this task's own run: an old lesson is a hypothesis here, not a fact.

  4. Its probe totals 20.25. The right fix arrived on the first try.

  5. That's the second loop, harnessing-level recursion. The folder the agent runs with is rebuilt for every task. The library the editors read, the improvement harness, keeps growing. The model never changed.

Layer 9 · all together

The whole loop

One task from start to finish, then the hand-off to the next.

FIG 9 · one task, then the nextStep 1 of 6
1 · Base runsoriginal folder 2 · ProposeINV + DOM edits 3 · Probe1 run per candidate 4 · Composemerge, adopt, keep 5 · Pickblind choice 6 · Rememberverdicts + lesson library read by editors NEXT TASK Steps 1–5: task-level recursion, inside one task Step 6: harnessing-level recursion, across tasks k: how many times steps 2–4 run
  1. Base runs. Run the unchanged folder k times on the new task.

  2. Propose. INV and DOM editors each edit a copy, citing moments in the runs and reading the library.

  3. Probe. Run each candidate once on the same task, and compare the runs.

  4. Compose. Merge, adopt one candidate, or keep the base, then run the final.

  5. Pick. With k > 1, choose one final's output blind. Only then does the grader score it, to measure.

  6. Remember. Verdicts and one lesson go into the library, and the next task begins.

The takeaway

You can't score an edit without an answer key. You can still watch what it changes, and keep what the runs support.

Does it work?

In the paper, a single step (k = 1) improved GLM-5 and Gemini 3.5 Flash over their own base harnesses on three benchmarks of professional work: up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 percentage points on LAB's partial pass rate. A bigger k brought further gains.

Who sees what

Nobody inside the method ever sees a score. The grader runs last, outside it.

RoleFoldersRun logsOutputsAnswer key
INV and DOM editorsEdit a copy of the folder.yesyesyesno
ComposerMerges candidates into a final.all of themyesyesno
SelectorPicks one final's output.hiddenhiddenrenamed A, B…no
ReflectorWrites verdicts and a lesson.yesyesyesno
GraderMeasures the delivered output, after everything.nonothe chosen oneyes

Run the ledger yourself

The demo runs both modes on the two ledgers with scripted editors, so it needs no model or API key. From a clone of the repo:

$ uv venv --python 3.12 .venv && source .venv/bin/activate
$ uv pip install -e '.[render]'
$ harness2-demo --output runs/demo --k 2
Recap

Glossary

harness
The folder of files around a model: prompts, memory, skills, scripts, subagents and plugins.
harnessing
Improving an agent by editing its harness instead of its model.
rollout
One full run of the agent on a task.
trajectory
The log of a rollout's actions and results.
deliverable
The file a rollout produces.
base harness
The starting folder, unchanged.
candidate
An edited copy of the folder.
probe
One rollout of a candidate on the same task.
INV
Edits to how the agent works: steps, tools, scripts, checks.
DOM
Edits to what an expert in the field knows: rules, required parts, checks.
composer
The session that turns candidates into one final folder.
selector
Picks one final's output, blind to how it was made.
reflector
Writes per-edit verdicts and one lesson after each task.
library
The per-domain store of adopted folders and lessons. Also called the improvement harness.
k
The budget: 4k rollouts and 3k editor sessions per task.
KEEP_BASE
Choosing to leave the folder unchanged.