A history of
Harness Hacks

how will we prompt AGI?

Delivery: Say “Viktor is an AI employee” over the wordmark. Four boxes, then hand off to the definition.

AI employee. Viktor is an AI employee. What makes him different from a chat window is not the model. It is his harness.

Four boxes. Tools as code: 173 integrations, about five thousand individual tools and 1300k skills, loaded when needed and composed in code. One continuous agent: start a task in Slack, continue it in OpenCode, finish it on a live call in the web app. Same work, same agent, same state, because it lives in a workspace, not in a chat window. Multiplayer and async: an org-scale agent, not a personal one. Your team and Viktor in one thread, steering him mid-task. And he is wherever you already work, from Slack and Teams to APIs and MCP.

Handoff. All of that sits between the model and the people using it. That layer is the harness. Let’s name the parts.

Sources: Viktor product description of his harness, September 2026. Tool loading and the persistent workspace match Peter Albert, What Breaks When Your Agent Has 100,000 Tools, March 3, 2026, and A Harness for AGI, July 15, 2026. Surfaces named as the product states them: Slack, Teams, web, desktop, mobile, OpenAI- and Anthropic-compatible APIs, ACP, MCP. No employer named.

Between the model
and the world: harness.

Language Models are
Few-Shot Learners.

Delivery: 3 seconds on the pairs. Wait for the room to say “kot”.

The hack. A text completer was built to finish documents. We wrote a fake document whose only sensible ending was the answer we wanted.

shape of work: examples › framing › chain › loop › tree › graph

scale: one prompt › tries › subagents › swarm

Brown et al., OpenAI, “Language Models are Few-Shot Learners”, May 2020
GPT-3, 175B parameters, trained only to predict the next token
No weight updates: the examples live in the prompt
The paper’s own headline demo is translation

Next. Two researchers ask what the examples are actually doing.

Sources: Brown et al., Language Models are Few-Shot Learners, May 2020. Date anchors the GPT-3 paper, not the invention of few-shot learning. Weights stay fixed during inference. The English → Polish prompt and completion are illustrative, not a recorded paper example or model run; the paper's own headline few-shot demonstrations include translation (English → French).

Prompt Programming:
examples locate a task.

Delivery: Point back at the pairs. Both routes into the same box.

The hack. Few-shot examples were supposed to be a lesson. The model was not learning from them, so we replaced the lesson with a label that points at the same place.

shape of work: examples › framing › chain › loop › tree › graph

Reynolds and McDonell, EleutherAI, “Prompt Programming for Large Language Models”, February 2021
Examples locate an already learned task; they do not teach it
Some zero-shot framings beat few-shot on the same translation task, on GPT-3
Introduces metaprompts: the model writes its own instruction

Next. One call is still one call. Break the task into a chain.

Sources: Reynolds and McDonell, Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm, submitted February 15, 2021. The paper argues the function of few-shot examples "is better described as locating an already learned task rather than meta-learning", demonstrates zero-shot prompts that outperform few-shot on translation, and introduces metaprompts, prompts that induce the model to generate its own task instructions. The comparison is the authors' own on GPT-3, not an independent replication. "Locating" is their interpretation of the observed behaviour, not a measured mechanism inside the weights. Date anchors this paper, not the invention of prompt design; the two routes in the drawing are schematic, not recorded prompts.

AI Chains:
one prompt per step.

Delivery: Ultra-light example first, then the three calls. Land the reliability line.

The hack. A language model is meant to answer in one go. We ran it like a shell pipeline, and used ordinary program variables as the model’s memory between calls.

shape of work: examples › framing › chain › loop › tree › graph

Reliability means taking control away. Making something reliable means moving it outside the model’s control, back into the domain of traditional software. The model cannot skip the check, because the check is a line in a program, not a suggestion in a prompt.

Wu, Terry and Cai, Google Research, “AI Chains”, October 2021
Each step’s output becomes the next step’s input; intermediates are readable, editable, rerunnable
20-person user study: better outcomes, more transparency and control
LangChain repo created October 2022, a year later

Next. A person still lists the steps. Can one sentence make the model do the working itself?

Sources: Wu, Terry and Cai, AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts, submitted October 4, 2021, revised March 17, 2022: "the output of one step becomes the input for the next, thus aggregating the gains per step"; a 20-person user study reports improved task outcomes plus transparency, controllability and sense of collaboration, and observes users calibrating expectations through sub-tasks, comparing alternative strategies and debugging individual chain components. That is a controlled user study against a single-prompt baseline, not a benchmark result, and the twenty participants are its whole sample. The three-step drawing is schematic, not a chain from the paper. langchain-ai/langchain was created October 17, 2022 per the GitHub repository record. Date anchors this paper, not the invention of multi-step prompting.

Large Language Models are
Zero-Shot Reasoners.

Delivery: Define no-shot the first time you say it. Pause after forty percent.

The hack. A prompt is supposed to describe the task. We appended a sentence that describes nothing and only exists to change what the model emits first.

the model eats the harness: reasoning › format › prompting › harness › execution › climb › world

And then no-shot. Few-shot: we hand over examples. Zero-shot: we hand over a framing. No-shot: no examples and no handwritten procedure either, because the model writes the workflow itself. Keep that word.

Wei et al., Google, chain-of-thought prompting, January 2022: eight worked examples
Kojima et al., University of Tokyo and Google, “Large Language Models are Zero-Shot Reasoners”, May 2022
“Let’s think step by step”, text-davinci-002, GSM8K: 10.4% → 40.7%
Two calls: one extracts the working, one extracts the number

Next. The working needs to touch the world.

Sources: Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, January 2022 (few-shot exemplars containing intermediate reasoning steps; eight exemplars on GSM8K; different models from Kojima's study); Kojima et al., Large Language Models are Zero-Shot Reasoners, May 2022 (Zero-shot-CoT prompts the model twice, reasoning extraction with “Let's think step by step” then answer extraction with “Therefore, the answer (arabic numerals) is”; large InstructGPT model text-davinci-002, GSM8K 10.4% → 40.7% within that paper's own comparison, not a universal gain).

ReAct:
reason, act, observe.

Delivery: List, chain, now loop. Pause at observe.

The hack. Text generation has no side effects. We invented a text format where a certain line is not a sentence but a command, then ran it and told the model the result was its own document.

shape of work: examples › framing › chain › loop › tree › graph

Yao et al., Princeton and Google Brain, “ReAct”, October 2022
Thought, Action, Observation, interleaved in one prompt; the harness runs the action and pastes the result back
Wikipedia API as the tool; +34% on ALFWorld, +10% on WebShop over the baselines
LangChain shipped a ReAct chain within three weeks

Next. One loop, one trajectory. A poor first idea needs another branch.

Sources: Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models, first submitted October 6, 2022. Interleaves reasoning traces with task-specific actions and environmental observations; Wikipedia API interaction supports question answering and fact verification. The loop is the prompted implementation, with the harness executing actions; the graphic is schematic, not a recorded trajectory. Date anchors the paper, not the invention of tool use.

Tree of Thoughts:
try another branch.

Delivery: “And now, a tree.” 3 seconds on A, B, C.

The hack. A completion is one answer. We asked for several partial answers and used the model as a candidate generator inside a classic search algorithm.

shape of work: examples › framing › chain › loop › tree › graph

Yao et al., Princeton and Google DeepMind, “Tree of Thoughts”, May 2023
Propose several next steps, evaluate, expand, backtrack
Game of 24, GPT-4: chain-of-thought 4% → ToT 74%, at far higher inference cost

Next. From text to a search structure. Now the text itself has to fit a structure.

Sources: Yao et al., Tree of Thoughts: Deliberate Problem Solving with Large Language Models, first submitted May 17, 2023. Proposes and evaluates intermediate thoughts, then searches with lookahead or backtracking. The reported GPT-4 Game of 24 comparison is 4% for chain-of-thought versus 74% for ToT, not a general gain across tasks or an equal-compute ablation. A/B/C are schematic candidate states, not literal paper outputs.

Grammar-based
sampling.

Delivery: Recall the JSON begging. Pause on wrong Bob.

The hack. The sampler exists to pick likely tokens. We used it as a parser: a grammar decides which tokens are even allowed, and the model only picks among what is left.

the model eats the harness: reasoning › format › prompting › harness › execution › climb › world

llama.cpp PR #1773 by Evan Jones, merged July 2023
Grammar compiled to parser state; every token the state cannot accept is masked before sampling
OpenAI Structured Outputs, August 2024, the hosted version
Guarantees shape, not truth: Bob parses when you asked for Alice

Next. Once answers can be scored, so can instructions.

Sources: llama.cpp, grammar-based sampling (PR #1773 by ejones), merged July 2023 (proposed June 2023); the PR describes the nondeterministic pushdown automaton, filtering the logits against the current parse stacks and advancing that state with each accepted token. OpenAI, Structured Outputs, August 2024. These are implementation milestones, not claims to have invented constrained decoding. The conversational opener is a familiar chat-tuning behaviour the grammar work was written to defeat, not a measured rate. “Cannot be malformed” applies to a generation that runs to completion; an interrupted or truncated generation can still stop mid-object. Alice/Bob are illustrative; no sampled probabilities or tokenizer segmentation are depicted. The string schema excludes null, not JSON generally.

Large Language Models
as Optimizers.

Delivery: Point back at “step by step”. Read the winning prompt in full.

The hack. A model answers prompts. We pointed one at its own prompts as the thing to optimize, and used a benchmark as the loss.

the model eats the harness: reasoning › format › prompting › harness › execution › climb › world

Yang et al., Google DeepMind, “Large Language Models as Optimizers” (OPRO), September 2023
Optimizer PaLM 2-L-IT proposes instructions from the scored history; scorer PaLM 2-L answers
GSM8K: “Let’s think step by step” 71.8% → “Take a deep breath and work on this problem step-by-step” 80.2%
Objective is a 3.5% sample of the training set; test accuracy computed afterwards

Next. A much older trick: run the same thing again.

Sources: Yang et al., Large Language Models as Optimizers, September 2023; results and optimisation procedure. Optimizer PaLM 2-L-IT proposes instructions; scorer PaLM 2-L answers, using a fixed 3.5% sample of the GSM8K training set as the objective; test accuracy is computed afterwards. Table 1 reports the highest test accuracy among found instructions: 80.2% versus 71.8% for “Let's think step by step”; the comparison is against that standard line and a few baselines, not a best human effort. Best reported result, not independent evaluation of a single preselected winner, and the whole instruction is compared: nothing isolates a causal effect of the word “breath”. “Nobody typed it” means the optimizer generated the string; humans designed the meta-prompt and the loop. “Coached” describes the loop, not any mental state.

Large Language
Monkeys.

Delivery: Explain the monkeys and the typewriter. Both numbers, then the caveat.

The hack. Temperature exists to make output varied. We used it as a random search: hundreds of samples, and a test suite as the selection function.

scale: one prompt › tries › subagents › swarm

Brown et al., Stanford and Oxford, “Large Language Monkeys”, July 2024
DeepSeek-Coder-V2-Instruct, SWE-bench Lite: 15.9% with one sample, 56% with at least one passing patch in 250
Coverage, not delivery: an external check still has to find the passing patch
Agentless, Xia et al., UIUC, October 2024: no loop at all, locate › repair › validate, 32% on SWE-bench Lite at $0.70 per issue
Infinite monkeys: Émile Borel, 1913

Next. Many tries multiply the calls. The next hack multiplies the windows.

Sources: Brown et al., Large Language Monkeys, July 2024, verified September 2026 (DeepSeek-Coder-V2-Instruct on SWE-bench Lite: 15.9% of issues solved with one sample, 56% with 250; majority voting and reward models "plateau beyond several hundred samples"). The date anchors this study, not the invention of repeated sampling. Coverage is not delivered success: it counts issues with at least one passing patch, which still has to be identified by an external check.

Multi-agent research:
fresh windows.

Delivery: Start with the window. Lead, three fresh windows, then the cost.

The hack. One context window is a hard ceiling on how much a model can hold, and every task we cared about was outgrowing it. So we stopped trying to fit. Split the work across several windows, give each a fresh context and only its own brief, and stitch the results back together outside. The limit stays; the task no longer has to live inside it.

scale: one prompt › tries › subagents › swarm

Anthropic, “How we built our multi-agent research system”, June 2025
Lead splits the task among 3 to 5 workers; each returns compressed findings, not its transcript
Opus 4 lead with Sonnet 4 workers beat single Opus 4 by 90.2% on Anthropic’s internal eval
About 15× the tokens of a chat; token usage explained 80% of performance variance

Next. Every one of these was a harness engineer’s decision. Let the model change the harness itself.

Sources: Anthropic, How we built our multi-agent research system, June 13, 2025 (orchestrator-worker pattern; subagents run in parallel with their own context windows and condense findings for the lead; lead spawns 3 to 5 subagents; Opus 4 lead with Sonnet 4 subagents outperformed single-agent Opus 4 by 90.2% on an internal research eval; token usage explained 80% of performance variance; multi-agent systems ≈15× chat tokens in their data; each subagent needs an objective, output format, tool guidance and clear task boundaries; vague instructions caused duplicated work and gaps). The 90.2% figure is Anthropic's internal eval, not an independent benchmark, and the token multiple is an aggregate, not a per-task bill. The system result does not isolate the context-isolation benefit. The bridge to specs is a narrative link, not a claimed historical origin of spec-driven development.

Live-SWE-agent:
rewrite the harness.

Delivery: Recall the hundred-line reader. Editable harness, then the protected check.

The hack. A harness is written once, by an engineer, and then has to serve every task and workload that comes after. It never does; each new kind of work needs its own tools. But the model has already captured so much of the world that it knows what those tools should be. So give it the control: start with only bash, let it write the tools it needs for this task, during the run. The harness becomes one more thing the model produces. Only the check stays out of its reach.

the model eats the harness: reasoning › format › prompting › harness › execution › climb › world

Xia et al., UIUC, “Live-SWE-agent”, November 2025
Starts from mini-SWE-agent, bash only; creates and installs its own tools mid-run, no offline training
77.4% on SWE-bench Verified
Wang et al., “Rethinking the Evaluation of Harness Evolution”, July 2026: under matched budgets, harness evolution does not consistently beat simple retrying

Next. The model’s code still runs only after the model has finished talking.

Sources: Xia et al., Live-SWE-agent, v3 November 24, 2025 (first submitted November 17; v1 reported 75.4%), starts from a bash-only scaffold (mini-SWE-agent), lets the agent create and install tools for its own scaffold on the fly with no offline training, and reports 77.4% on SWE-bench Verified; Wang et al., Rethinking the Evaluation of Harness Evolution for Agents, first submitted July 14, 2026, revised August 27, finds that under matched feedback and inference budgets automatic harness evolution does not consistently outperform simple test-time scaling and generalizes poorly. The latter studies Terminal-Bench 2.1 with GPT-5.4 and Opus 4.6, not a matched-compute replication of Live-SWE-agent. The ladder recalls S4 (few-shot instruction following), S11 (OPRO prompt search), S14 (repeated sampling with external selection), S21 (tool code on demand) and this slide; "on request" describes inference-time delegation, not a claim that these behaviours were trained in. The reader repair is illustrative, not a recorded trajectory or explanation of the benchmark score. Its one-based reader returns lines[start-1:start+99], numbered from start, with next=start+100 if more remain. Protected acceptance is a design recommendation: evaluator code, inputs and reporting authority are outside the agent’s write permissions; calling it does not guarantee a pass.

Speculative Programmatic
Tool Calling.

Delivery: Ask why we wait. Ordinary timeline, then the early start. Mark the end of history.

The hack. Models are slow and functions are fast. A tool call arrives as a large chunk of code, streamed one token at a time, and the harness waits for the last bracket. Why wait? The moment a call inside the stream is syntactically complete, run it. By the time the model finishes, the results are already there.

the model eats the harness: reasoning › format › prompting › harness › execution › climb › world

Alex Zhang, “Speculative Programmatic Tool Calling”, August 2026
Only pure, side-effect-free calls opt in; results reused when execution reaches them
About 1 to 1.2× wall-time speedup in small RLM experiments, high variance

History stops here. Everything after this is my reading, then my predictions.

Sources: Alex Zhang, Speculative Programmatic Tool Calling, August 24, 2026 per the author's index. Implementation: eligible tools opt in with speculatable=True and pure=True, while side-effecting tools are not speculated. The post reports about 1–1.2× wall-time speedups in small RLM experiments with substantial variance. Claude Code workflows, accessed September 2026: Ultracode can create JavaScript workflows that orchestrate subagents with loops, branches and intermediate results held in script variables. “Graph-shaped” and “no-shot” are Ivan's synthesis, not Anthropic terminology.

Three trajectories.

Delivery: History left of the dashed line, predictions right. Read each rail title with its from → to, then walk the nodes. Slow down.

Three trajectories. Every hack in this history moved along one of three rails. Left of the line is what we saw. Right of the line is where I think each rail goes.

Shape of work. Examples or a framing, it is still one prompt in, one answer out. Then a chain, a loop, a tree. The interaction grows from a single call into a structure. Next shape: a graph.

The model eats the harness. Reasoning, output format, prompt writing, the harness itself, and now execution timing. Each was a harness job; each is now a model job. Next it eats harness design, and then the world.

Scale. One call. Hundreds of tries. Subagents with their own windows. Next: hundreds of agents, not in a tree but in a graph, talking to each other.

Handoff. Four predictions, one per dashed node.

Sources: Synthesis of the preceding slides; Ivan’s reading, not a published taxonomy. Dashed nodes are predictions from September 2026.

Turn most tasks into
a hill climb.

Delivery: Prediction one. Trace the loop, pause on “gradient”.

The prediction. Most tasks become hill-climb objectives. Once better beats worse, run experiments over prompts, tools, memory, routing, whole harnesses, and keep what scores higher. Designing the harness was the last decision a human made around the model; now the model makes it.

the model eats the harness: reasoning › format › prompting › harness › execution › climb › world

The gradient is the scarce artifact: a test, an evaluator, a preference, a measured outcome
Basis: OPRO 2023, Live-SWE-agent 2025, Prime Intellect’s Prime Agent 2026
Prime Agent’s Factorio loop: once an exploit scored well, refinement learned to cheat better
Specialized for one objective, not universally better

Sources: Prediction from September 2026, not a measured general trend. Historical basis: OPRO, Live-SWE-agent and Prime Agent. Prime Intellect reports evaluator exploitation in its Factorio refinement loop. “Gradient” is Ivan's analogy for directional feedback, not a claim of weight updates.

Work becomes
a graph.

Delivery: Prediction two. Chain, loop, tree, then graph.

The prediction. A chain gives one route, a loop another attempt, a tree branches. A graph adds parallel work, joins, retries and routes that only appear after an intermediate result. The graph becomes the default unit of agent work, and the searches get much larger than today.

shape of work: examples › framing › chain › loop › tree › graph

Plans on one branch, implementations on others, reviewers prune, a failed check reopens one part
Best-of-N is one node inside it
Claude Code workflows, 2026: model-written scripts with loops, branches and intermediate results; the model decides when to write one
This is the no-shot dispatch named on S7

Sources: Prediction from September 2026. Claude Code dynamic workflows, accessed September 2026: model-written JavaScript holds loops, branching and intermediate results, and Ultracode can decide to create a workflow. Tree of Thoughts supplies branching and backtracking. “Graph-shaped default” is Ivan's prediction, not an Anthropic claim.

The world retreats
into the model.

Delivery: Prediction three. Rings inward, then the right column. Separate now, next, then.

The prediction. The world retreats into the model. Now: we describe the world to the model and gate its tools. Next: the model carries a working model of its tools, APIs, files and people, predicts what a call returns, and round-trips only to verify. Then: the world reshapes itself to be held by models, with docs and interfaces written for a model to internalize. Reality becomes the final check.

the model eats the harness: reasoning › format › prompting › harness › execution › climb › world

What stays outside: authority, meaning permission and the check
The harness collapses to a contract; the prompt collapses to intent
Frontier access stays gated, because the gate is the only harness left
Basis: Live-SWE-agent and speculative tool calling; no claim that infrastructure moves into weights

Sources: Prediction from September 2026. Historical basis: Live-SWE-agent and speculative programmatic tool calling on the preceding slides. No claim that runtime infrastructure literally moves into weights; “held by the model” means predicted and acted on from inside one generation, with verification outside.

A civilization
of experts.

Delivery: Prediction four. Walk the bars left to right. Leave a beat before the close.

The prediction. One prompt, one model, many tries, subagents, a graph of them. People call the next thing a city of geniuses. I think it is a civilization of experts: hundreds of agents in a graph, not a tree, talking to each other, with selection applied at every role. Selection and specialization matter as much as raw intelligence.

scale: one prompt › tries › subagents › swarm

Recall Large Language Monkeys: 15.9% → 56% with 250 patches, external check still required
Apply that selection to planners, implementers, reviewers, coordinators; keep the best fit per branch
The paper measured patch sampling only; a recursively selected organization is my extrapolation

Sources: Prediction from September 2026. Large Language Monkeys reports 15.9% single-sample and 56% best-of-250 coverage for DeepSeek-Coder-V2-Instruct on SWE-bench Lite, with external selection still required. Applying best-of-N selection recursively across organizational roles is Ivan's extrapolation, not a result from that paper.

How will we
prompt AGI?

Probably through an agent that makes the case.
The prompt might be short.
The work behind it could be civilization-sized.

A history ofHarness Hacks

Between the modeland the world: harness.

Language Models areFew-Shot Learners.

Prompt Programming:examples locate a task.

AI Chains:one prompt per step.

Large Language Models areZero-Shot Reasoners.

ReAct:reason, act, observe.

Tree of Thoughts:try another branch.

Grammar-basedsampling.

Large Language Modelsas Optimizers.

Large LanguageMonkeys.

Multi-agent research:fresh windows.

Live-SWE-agent:rewrite the harness.

Speculative ProgrammaticTool Calling.