Choosing Model Vs Harness

Model Is the Engine, Harness Is the Car

There is a habit that shows up in almost every team building AI. Something in the product is not good enough and the reflex is to reach for a bigger model. The next tier up is always one config change away and it usually does move the number. But it quietly hides where the problem actually lived, and it does so while raising the cost of every request from that point forward.

A result that Composio published recently makes the point better than any argument can. They took a single model, Kimi K3, and ran it through six different agent harnesses, among them Claude Code, Codex, OpenCode, Pi Agent and Hermes, across twenty-six agentic tasks. The outcomes were not close.

Codex ranked last on success despite sitting mid-pack on speed and cost, and Claude Code cost roughly four times what Hermes did to do comparable work. If you believed the model was the product, that spread should be impossible. It is not only possible, it is the normal state of affairs once you leave the chatbox and start building systems.

The cleanest way I have found to hold this is to treat an AI product’s performance as a product of three terms rather than a property of one:

System outcome = model capability × harness effectiveness × data quality

The model determines what is theoretically possible. The harness determines whether that possibility is reliably converted into something a user can actually use. Data quality decides whether the whole thing is being measured against reality or against a tidy fiction. The reason this is a product and not a sum matters: a strong model behind a broken harness is still a broken product, because a term near zero drags the whole thing down no matter how large the other terms are. The Composio numbers are that multiplication made visible. Kimi K3’s capability was fixed, so the spread came from the harness term.

What actually belongs to each layer

It is worth being precise about the split, because the fog between the two layers is where most wasted effort goes. The model is all about reasoning and instruction-following. The harness is everything your application does around the model. For an agent the harness often matters as much as the model, because it controls the three things that determine the outcome: what the model sees, what actions it is allowed to take, and what happens when it makes a mistake. A model does not retrieve its own context, retry its own failed tool calls, or decide on its own when to stop looping. The harness does all of that, and each of those decisions can rescue or wreck a run that the model itself would have handled fine.

The research that first made this legible was the work on the agent-computer interface. When the SWE-agent team studied why agents succeeded or failed on real GitHub issues, they found that the interface between the model and the environment (how files were shown, how edits were confirmed, how feedback was surfaced) moved the success rate dramatically without touching the model at all. A well-designed interface let the same model resolve substantially more issues on SWE-bench than a naive one. That is a harness result in the purest sense: the intelligence was constant, and the packaging around it decided whether that intelligence reached the task.

The hard part is attribution

Knowing the two layers exist is easy. The difficult work is deciding when an outcome disappoints which layer caused it because nearly every symptom has a plausible explanation on both sides. An incorrect answer can be weak reasoning in the model, or it can be missing, stale or irrelevant context that the harness failed to assemble. A tool-use failure can be the model choosing the wrong tool, or it can be an ambiguous description and a malformed schema that would trip up any model. High cost can be an expensive model reasoning too much, or it can be a bloated prompt shipping the same context on every turn inside a loop that never should have run that many times. A response that arrives slowly can be a model with low generation speed, or a harness making sequential calls it could have made in parallel. Inconsistent output can be model variance, or loose prompts with no validation. An agent that loops forever can have weak planning, or it can be missing the termination logic.

The pattern across that whole list is that the visible symptom does not name its own cause. This is why the model-upgrade reflex is so seductive. Swapping in a stronger model will sometimes paper over a harness defect well enough to move the metric, and now you are paying frontier prices forever to compensate for a bug you could have fixed once.

How to actually separate the layers

The only reliable way to attribute an outcome to a layer is to hold one of them still and vary the other, which is just controlled experimentation applied to a stack most people run as an uncontrolled whole. There are three experiments to run, and they answer three different questions.

The first holds the harness constant and varies the model. Fix the dataset, the prompts, the retrieved context, the tools, the sampling settings and the quality metric, then run several models through that identical rig. Whatever moves is the model effect, cleanly isolated. This is exactly the shape of the Composio study read from the other direction, and it is the honest version of “benchmarking.” The point is not a universal leaderboard that tells you which model is best in the abstract, but a measurement of which model is best for your workload, run against your data through your harness.

The second experiment inverts the first. Keep one model fixed and vary the harness: the prompt design, the retrieved context, the tool descriptions, the workflow structure, the memory strategy, the retry policy, the validation steps. Whatever moves now is the harness effect. This is the experiment teams skip most often and regret most, because it is where the counterintuitive wins hide. It is entirely common to discover that better retrieval lets a smaller, cheaper model outperform an expensive one that was running on poor context, which means the “obvious” model upgrade would have bought less than a retrieval fix at a fraction of the ongoing cost. The Minions work is a sharp instance of this: pairing a small local model with a frontier one, the researchers found that changing how the work was divided between them mattered more than which models were used, with a better decomposition recovering nearly all of the frontier quality at a fraction of the cost.

The third experiment is the one that stops you from treating the first two as independent. Models do not respond uniformly to the same harness. One model needs detailed tool instructions and another does better with terse ones; one benefits from an explicit planning step and another becomes less reliable when you push it to reason at length. This interaction is real and measurable, and it means the optimization target is not the best model in isolation or the best harness in isolation, but the best model–harness pair. A harness lovingly tuned for last quarter’s model can actively hurt this quarter’s, which is why “we upgraded the model and quality dropped” is a sentence you hear more often than intuition would predict.

“It’s bad” is not a diagnosis

Separating the layers experimentally only works if you can see inside a run, and that requires evaluating at more than one level. A single end-to-end score tells you the system is bad without telling you where it broke, which is precisely the information you need to avoid the upgrade reflex. The useful structure is three layers of evaluation stacked on top of each other.

At the bottom are component evals, which test the individual parts in isolation: retrieval relevance, context completeness, tool selection, argument generation, structured-output validity, response correctness. These answer where the failure happened.

Above them sit workflow evals, which ask whether the complete trajectory succeeded: did the agent choose the right sequence of tools, recover from an error, stop at the right time, respect its permissions, and produce a usable result.

At the top are product-outcome evals, which measure what users actually care about: task completion, time saved, human corrections required, escalation rate, resolution or conversion rate, satisfaction.

The gap between the second and third layers is where generic benchmarks quietly mislead. A model can post a strong number on a public benchmark and still produce poor product outcomes, because the benchmark does not resemble the real workflow. Benchmarks like agent leaderboard, which test agents in realistic tool-use and conversation settings rather than one-shot question answering, exist precisely because the older style of evaluation was measuring the wrong layer, scoring model knowledge when the product depended on multi-step reliability.

Cost belongs in the same frame

None of this is complete without money in the picture, because a configuration that is slightly more accurate and dramatically more expensive is not obviously the better choice, and the Composio spread, four times the cost for comparable work, is a reminder of how wide the economic gap between harnesses can be even at fixed capability. The metric that keeps you honest is cost per success rather than cost per token:

Cost per success = total workflow cost ÷ number of successful outcomes

Total tokens alone will lie to you, because they hide where the spend goes and why. What you want to see is tokens by workflow step, context growth over the length of a session, reasoning tokens separated from output tokens, etc.

How to pick between model and harness?

Start by defining a representative dataset built from real tasks, hard edge cases and actual production failures.

Define success next, with objective criteria where you can get them and human review where the task is subjective. Establish a system baseline across quality, cost, latency and reliability, so you have something to move against.

Fix the obvious harness problems first like the missing context, the ambiguous tools, the runaway loops, because these are cheap to fix and they change the baseline against which any model comparison is fair. Only then compare models under controlled conditions using the improved harness so the test is honest.

Tune the harness for each serious model candidate since the pair is the real unit. If quality is poor across every model you try, the problem is the harness, the data or the definition of success and no model will save you.

If one model wins consistently across a well-tuned harness, that is a real signal to switch. If an expensive model only slightly outperforms a cheap one, route most traffic to the cheap one and reserve the expensive model for the hard cases through confidence-based escalation.

If long sessions degrade, the answer is memory and context management. If tool calls are unreliable, the fix is schemas, examples, validation and retries. If your offline results are strong but production is weak, your dataset is not representative and your traces will tell you why. And if quality is fine but the economics are bad, the levers are compression, caching, routing and cutting loops, all of them on the harness side of the ledger.

Anyone can buy the same frontier weights, and the Composio result shows those weights performing wildly differently depending on who wrapped them. What is not rentable is a harness tuned to your workload, your data, your tools and your failure modes, with an eval stack that tells you which layer to touch when something breaks.

A strong model cannot rescue a weak harness any more than a strong engine can rescue a car with no steering. A better engine helps. But it will not compensate indefinitely for bad navigation(instructions), dead weight(context), or a broken transmission(routing).

Model Is the Engine, Harness Is the Car.

LinkedIn / @ptkbhv / GitHub / Books