Context Engineering is Half The Harness

There is a pattern in computing that repeats roughly once a decade. A new capability appears and for the first year or two the entire conversation is about the capability itself, about benchmarks and demos and possibility space.

Then somewhere around year two or three the conversation undergoes a phase transition and starts being about how to make the thing work reliably in production, how to coordinate many instances of it, how to give it the kind of operational guarantees that real systems require. We are at that inflection point with AI agents right now, and the infrastructure conversation has started in earnest. But it has started lopsided, focused almost entirely on one half of the problem while largely ignoring the other half. And the half being ignored is, I think, where the more important breakthroughs will come from.

The term that has coalesced around this infrastructure layer is “harness.” An agent harness is the software that wraps around a language model to manage its lifecycle, context, and interactions with the outside world. It is not the brain that does the thinking but the environment that gives the brain tools, memories, and structure. Over the past year, a remarkable amount of excellent engineering work has gone into the harness layer, and I want to acknowledge that work specifically because it sets the stage for what I think is missing.

Anthropicpublished their research on effective harnesses for long-running agents, showing that even Opus 4.5 running on the Claude Agent SDK could not reliably build a production web app from a high-level prompt, and that the fix was not a smarter model but a structured two-agent architecture with feature lists, progress files, and session startup rituals that imposed the discipline of a good software engineer on a model that would otherwise try to do everything at once.

Manuspublished their context engineering lessons after rebuilding their agent framework four times, sharing that KV-cache hit rate is the single most important metric for production agents, that dynamically adding or removing tools destroys cache and confuses the model, that the file system is the ultimate context, and that leaving errors visible in the trace actually helps the agent learn rather than repeating the same mistakes. They called their iterative process of architecture searching and empirical guesswork “Stochastic Graduate Descent” and it is one of the most honest descriptions of applied AI engineering I have read.

OpenAIThe Codex team published their harness engineering experience, describing a repository of roughly a million lines of agent-generated code maintained by a small team, where they discovered that 20% of their engineering time was spent cleaning up “AI slop” until they started encoding “golden principles” directly into the repository and built recurring cleanup agents that scan for deviations and open targeted refactoring PRs.

LangChainpublished the State of Agent Engineering survey covering over 1,300 practitioners, finding that 57% of organizations deploy multi-step agent workflows in production, 32% cite quality as the top barrier, 89% have implemented observability, but only 52% have adopted evals, and that for large enterprises the biggest challenge remains hallucinations and consistency of outputs.

All of this work is genuinely excellent and I am not here to critique it. But if you look at the totality of what the harness conversation has been about, a pattern emerges. It is almost entirely about managing text. About what tokens go into the context window, how to compress them, how to persist them across sessions, how to route them to the right model, how to keep the KV-cache stable. The entire intellectual energy of the harness community has been poured into building a better library for agents, giving them better indexing, better shelving, better ways to find and organize the information they need before generating their next token.

What is almost completely absent from the conversation is the lab. Agents do not just read code and write code. They act on systems. They run code against real environments, observe the results, interact with services that have state and side effects, and make decisions based on what actually happened rather than what they predict would happen. And the harness infrastructure for this, for execution and verification and cross-system experimentation, barely exists.

Everyone built the library. Nobody (yet) built the lab.

A scientist with an excellent library but no laboratory can produce literature reviews but not discoveries. I believe the execution harness, the lab infrastructure that lets agents test hypotheses against real systems and produce knowledge that did not exist before anything was executed, is one of the largest gaps in the current agent infrastructure landscape, and the place where the most important work will happen over the next two years.

#Self-testing is not experimentation

Before going further I want to draw a distinction that might not be obvious, because the immediate objection to this framing is “agents already run code.” And that is true. Claude Code runs the code it writes and observes whether it works. Anthropic’s Puppeteer integration lets Claude test the web app it built by clicking through it in a real browser. Manus agents interact with real file systems and real APIs. These are all forms of execution.

But there is an important difference between an agent that tests its own output and an agent that designs and executes experiments against a system it did not build, probing for emergent behaviors that no one anticipated. The first is quality assurance: did the thing I just made work the way I intended? The second is research: what does this system actually do under conditions its builders never considered? Current harnesses support QA reasonably well. Almost none support research.

And the research mode is where the most valuable findings live, because the bugs that matter in production are not the bugs the developer thought to test for. They are the bugs that emerge from interactions, timing, and load patterns that nobody anticipated.

#This is not a new idea, the integration is

I want to be clear about intellectual lineage here, because the concept of execution as discovery is not new. The fuzzing community has been doing this for decades. AFL, libFuzzer, and Hypothesis represent sophisticated execution harnesses for finding bugs that cannot be found by reading code. The chaos engineering community, from Netflix’s Chaos Monkey to Gremlin to LitmusChaos, has been injecting failures at service boundaries and observing what happens for years. The security research community at places like Google Project Zero and Trail of Bits has been combining automated program analysis with execution-based verification since before large language models existed.

What is new is the opportunity to connect these execution capabilities to the LLM agent loop as a first-class harness primitive. Right now, fuzzing, chaos engineering, and automated testing are separate tools that happen to be invoked by the same developer. They are not integrated into the agent’s reasoning loop in a way that lets the agent form a hypothesis, design an experiment, observe the results, update its understanding, and design the next experiment in a continuous cycle. The innovation is not execution itself but the tight integration of execution into the agent’s cognitive loop, treating the lab as infrastructure that is as native to the agent’s workflow as the context window is.

#The difference between reading and running

Consider a scenario that I’ve deliberately simplified to make the argument clear, with the caveat that real cross-service bugs are messier and harder to trace than this.

readingAn agent is reviewing a web application that has a shopping cart, a coupon service, and a payment processor. These are three separate services maintained by different teams, deployed on different cycles, communicating through message queues. The agent reads the code and identifies a suspicious interaction: a coupon that grants a negative discount could, in theory, result in a negative total price, and the payment processor might interpret this as a credit rather than a charge. The agent writes up this finding and suggests it might be exploitable.

This is useful. But it is also fundamentally a literature review. A hypothesis generated by reading static code, subject to all the limitations of static analysis: maybe the payment processor validates for negative amounts and rejects them, maybe the message queue serialization truncates negative values, maybe there is middleware between the services that normalizes the price. The agent has no way to know without actually running the scenario.

runningNow consider what it would mean for the agent to have a lab. It spins up the three services in a sandboxed environment, generates a valid user session, applies a coupon that creates a negative discount, submits the order, and watches what actually happens at each service boundary. It discovers that the coupon service happily applies the negative discount, the cart service passes the negative total through to the payment processor, but the payment processor does in fact reject the charge, returning an error code. However, the error handling in the cart service interprets this specific error code as “retry with saved payment method,” which triggers a second charge attempt that succeeds because the retry path skips the validation. The agent has now found a real exploit, not by reading code but by running an experiment, and the finding includes the exact request sequence, the service-level responses, and the specific error handling path that enables it.

The difference between these two scenarios is the difference between a literature review and a lab result. Both have value, but the experimental result is categorically more useful, more trustworthy, and more actionable.

I should be honest about this particular example, though. The negative price exploit is deterministic. Given the same inputs, the same code always produces the same outputs. A sufficiently advanced model with a large enough context window could, in theory, trace the logic through all three services and predict the outcome without running anything: if the total is −$5, the payment processor returns NEG_AMT, the cart service maps NEG_AMT to the retry path, the retry path skips validation. That deduction is hard but it is not impossible through pure reasoning. I chose this example for pedagogical clarity, not because it represents the strongest case for the lab.

The strongest case is the class of bugs that cannot be hypothesized from reading code at all, regardless of how smart the reader is. A performance cliff that appears at exactly 487 concurrent connections because of a kernel TCP buffer size interaction that is nowhere in the application code. A memory leak that accumulates over six hours of normal operation and eventually causes an OOM kill that cascades through a service mesh, where the specific accumulation pattern depends on the specific sequence of real requests the system received.

None of these are deducible from static analysis regardless of how much code the agent reads, because the information that reveals them is produced by the physical act of running the system under specific conditions. The agent needs a lab not just to verify hypotheses, but to discover what the library could never predict.

#Where the library ends and the lab begins

I want to be precise about the relationship between context engineering and execution, because the wrong version of my argument is easy to attack and the right version is important to get right.

For deterministic logic errors that trace through predictable code paths, increasingly powerful models with larger context windows will eventually find them through reasoning alone, and it would be dishonest to claim otherwise. If a bug is caused by a specific sequence of function calls that always produces the same output given the same input, then a model that can read all the relevant code and reason carefully through the execution path can, in principle, find that bug without running anything. The negative price exploit is exactly this kind of bug. Context engineering is making models better at precisely this kind of multi-step deductive reasoning, and the trajectory is clearly toward more capability, not less.

I am not arguing that the library is insufficient for this class of problems. I am arguing that this class is not where the hardest production bugs live.

Many of the hardest production bugs are non-deterministic, load-dependent, or time-dependent. And for these specifically, context engineering is not currently insufficient but structurally unable to help, because the information that reveals them does not exist in the source code in any form.

#The Bitter Lesson, applied to the lab

Manus’s Peak Ji articulated the most important design principle in harness engineering: your harness should be the boat on a rising tide, not a pillar stuck to the seabed. Boris Cherny, the creator of Claude Code, made the same point through the lens of Richard Sutton’s Bitter Lesson, keeping Claude Code deliberately unopinionated so that model improvements translate directly into product improvements. The diagnostic they both point to is critical: if your agent’s performance doesn’t improve when you swap in a stronger model, your harness is hobbling the agent rather than helping it.

This principle applies to the lab as much as it applies to the library, but with an important asymmetry. A better library can become less necessary if models develop genuinely better memory, longer effective attention, or native persistence across sessions.

A better lab cannot become less necessary through model improvements because hypothesis and experiment are fundamentally different computational processes. The model does one. The execution environment does the other. No advance in the first eliminates the need for the second.

Now, there is a reasonable counterargument here: maybe the model will eventually generate its own lab on the fly. Claude Code already does a primitive version of this when it writes a test script, runs it, and interprets the results. Maybe in two years a frontier model can spin up its own docker-compose environments, write its own concurrent test harnesses, and interpret its own results without any purpose-built infrastructure. This is plausible and worth taking seriously.

But the value of purpose-built lab infrastructure is not in raw capability. It is in reliability and iteration speed. A model can probably cobble together a test environment from scratch using docker and bash scripts. But it will take fifteen minutes and the environment will be fragile and specific to that one experiment. Purpose-built lab infrastructure provides the same advantage that CI/CD provides over “the developer can run tests manually”: it is not that it does something impossible, it is that it does something necessary reliably, repeatedly, and fast enough that the agent can iterate at the speed of thought rather than the speed of infrastructure provisioning.

The difference between an agent that can run one experiment per hour and an agent that can run fifty experiments per hour is not quantitative. It is qualitative.

Because the agent that iterates fifty times can explore the hypothesis space in a way that the slow agent fundamentally cannot.

This makes the execution harness one of the most Bitter-Lesson-safe investments you can make in the agent stack. It is infrastructure that gets more valuable as models get smarter, because smarter models generate better hypotheses that are more worth testing, which creates more demand for lab infrastructure, not less. A model that can hypothesize about increasingly subtle race conditions or increasingly complex cross-service interactions needs a more sophisticated lab to verify those hypotheses, not a simpler one. The relationship between model intelligence and lab demand is not zero-sum. It is positively correlated.

Anthropic’s use of Puppeteer for browser-based testing in their long-running agent work is the closest existing example of execution as a harness primitive, and it is telling that they called it out as one of the biggest improvements to their agent’s reliability. When Claude could actually see and interact with the web application it was building, rather than just reasoning about the code, the quality of its output improved dramatically because it could verify its own work through experiment rather than introspection. But Puppeteer for web UI testing is QA, not research. It verifies that the agent’s own work is correct. The general form we need is broader: give the agent a laboratory where it can test any hypothesis about any system’s behavior by actually running the scenario, observing the result, and feeding structured findings back into its reasoning loop.

#What needs to be built

I want to be specific about what the lab requires, because staying at the level of “we need better execution infrastructure” is not useful to anyone.

01Fast environment provisioning

First, you need fast environment provisioning. An agent that has to wait five minutes for a test environment to spin up will not iterate fast enough to be useful. The lab needs to maintain warm pools of pre-configured environments that can be specialized for a specific experiment in seconds rather than minutes, in the same way that a JIT compiler maintains hot code paths. This is a hard infrastructure problem involving container orchestration, dependency caching, and state management, and it is entirely distinct from anything happening at the model layer.

02Intelligent observation design

Second, you need intelligent observation design. When an agent runs an experiment, the raw output, every log line, every network packet, every database query, could easily be tens of thousands of tokens. You cannot feed all of that into context. The lab needs to be opinionated about what gets captured and how it gets compressed, preserving the causal chain that matters (this request triggered this error which caused this retry which succeeded through this unexpected path) while discarding the noise. This is where the lab and the library actually meet: the observation compression layer is a library problem (context engineering) applied to lab outputs (execution results). It is one of the more interesting subproblems here precisely because it sits at the intersection of the two halves.

03Concurrent and stateful test orchestration

Third, you need concurrent and stateful test orchestration. The most important bugs in production systems are not reproducible with a single sequential request. They require concurrent load, specific timing conditions, or accumulated state from a sequence of operations. The lab needs to generate realistic workloads, not toy examples, and manage the complexity of multi-step experimental scenarios where the outcome of step three depends on the specific timing of steps one and two. This is where the connection to decades of fuzzing and chaos engineering research becomes directly relevant: those communities have developed sophisticated techniques for generating interesting inputs and managing stateful test campaigns that the agent harness community should be drawing on rather than reinventing.

04Topology-aware orchestration

Fourth, and this is the piece that connects to the cross-system problem, you need topology-aware orchestration. The lab needs to understand that service A talks to service B through a message queue and that service B talks to service C through a REST API, and it needs to be able to instrument each of these boundaries to capture the cross-service behavior that reveals emergent bugs. This requires a model of the system topology that lives outside any individual agent’s context and that the lab maintains as persistent infrastructure.

These are infrastructure problems that require sustained engineering investment, and they create durable value that compounds over time as the lab accumulates knowledge about how to test specific kinds of systems effectively.

#The convergence

The harness conversation is going to converge. The teams building the library will eventually need lab capabilities, and the teams building the lab will eventually need sophisticated context management. The complete agent harness is one that manages both what the agent knows and what the agent can prove, both its reading and its experimentation. Right now these are being built by different communities with different assumptions, but they are two halves of the same layer.

The teams that understand this convergence early and build for it will have a structural advantage, because they will be able to offer agents that do not just hypothesize about system behavior but prove it, that do not just suggest changes but verify them, and that do not just read code in isolation but test interactions across service boundaries in a real laboratory. That is a qualitatively different kind of agent from what exists today, and building the harness infrastructure that enables it is, I believe, some of the most important engineering work happening in AI right now.