21 Match models to the work

The cheapest model call is not necessarily the cheapest accepted change. A worker can generate a plausible patch quickly, then consume an hour of expert attention repairing its assumptions. A more capable model can spend unnecessary time exploring a task that a short deterministic procedure would have resolved. Choosing between them requires an account of the work, not just a ranking of models.

Cost pressure is a reason to evaluate allocation, not evidence that a particular routing policy is already effective. Model capabilities, prices, interfaces, and rate limits change. A durable method should help select the current arrangement without turning today's favorite model into an architectural dependency.

Describe the judgment the task requires

“Coding” contains several different activities. Locating the source of an error is not the same task as changing a known expression. Comparing two lifecycle designs is not the same as translating a stable interface. Reviewing a concurrency repair requires evidence and skepticism that a broad prose summary may not exercise.

For the resource application, consider three assignments:

1. Investigate whether the existing publication operation preserves the old active resource after a failed candidate
2. Implement a bounded acquisition adapter against a stable publication contract
3. Review a candidate for premature completion and resource-lifetime defects

The investigation needs relevant retrieval and calibrated uncertainty. The implementation needs reliable local editing and feedback. The review needs the ability to challenge a plausible explanation and construct a counterexample. A model that performs well on one may be poorly suited to another. The surrounding tools and supplied evidence can change the result as much as the model choice.

Some tasks should not be routed to open-ended judgment at all. A known formatting transformation or schema validation may already have a reliable tool. An agent can invoke it without being asked to reinvent its rules.

Build a small evaluation around acceptance

A useful first evaluation can be modest. Select several representative, already understood tasks from each class. Keep their desired outcomes and environments stable. Give each candidate the inputs it would receive in actual work. Save the model version, relevant settings, tool access, and supplied context so that a later result is interpretable.

The evaluation record might contain:

Task family: Bounded adapters for local resources.

Required result: The requested resource is acquired and checked; failed preparation preserves the active version; no unauthorized contract expansion occurs.

Candidate arrangements: One model with the existing tools; another with the same tools; a prescribed localization-and-edit procedure using a model only at designated steps.

Observations: Accepted or rejected result, reason for rejection, elapsed time, human intervention time, tool and inference cost, retries, and integration repairs.

Hard constraints: No unapproved dependencies, no changes outside the task's authority, and no acceptance based solely on the worker's report.

Decision: Prefer the least costly arrangement that meets the required quality and risk constraints for this task family; retain an escalation path for cases outside the tested range.

This is a proposed experiment, not a table of measured results. Its most important property is that all candidates must earn the same kind of acceptance. A cheap candidate should not receive easier tests, while an expensive one is charged for more demanding scrutiny.

The simpler arrangement deserves genuine consideration. Agentless studied a prescribed localization, repair, and validation pipeline and obtained competitive results on its historical benchmark without an open-ended agent workflow. It still used language models. Its relevance here is the existence of a credible procedural baseline, not a claim that interactive agents are unnecessary for current development. Agentless, version 2.

Count the expensive parts you can overlook

Inference spend is observable and easy to compare, which makes it tempting to treat it as the whole bill. Include repeated context, tool execution, test environments, unsuccessful attempts, merge repair, and human attention. Report elapsed delivery time separately: a background task can take longer on the clock while requiring less active supervision.

Avoid collapsing everything into one monetary number unless the conversions serve an actual decision. Ten minutes of scarce maintainer attention may matter more than a small cloud charge, but assigning an invented hourly value can hide that fact behind false precision. A small set of dimensions is often more honest: accepted quality, human minutes, elapsed time, and direct cost.

Sample size limits what the evaluation can settle. One success is a useful demonstration. It is weak evidence for routine reliability. One unusually difficult task can also distort a tiny comparison. Inspect failure patterns, repeat material cases when feasible, and report how much of the real workload the sample represents.

The evaluation can reveal that task design is the bottleneck. If every model misunderstands whether checking implies availability, routing among them may have little value until the contract is repaired. Likewise, if every candidate spends most of its time waiting for a build, a faster model may not shorten delivery.

Use escalation without surrendering authority

A practical routing policy can start bounded tasks with a less expensive arrangement and escalate on named signals: repeated failure at the same boundary, unresolved architectural uncertainty, or an inability to produce trustworthy evidence. Escalation should carry the current state and failed approaches, not ask a new model to reconstruct everything from a triumphant summary.

Changing models does not grant new permissions. A stronger reviewer is still a reviewer; a stronger implementer still owns the same task. Nor does provider diversity establish independent reasoning. Models can share missing context, common patterns, and the same mistaken oracle.

Test one allocation decision

Choose a task family you actually repeat. Compare two arrangements on a small set of representative cases and include at least one unsuccessful attempt in the record. Ask a reviewer who did not generate the patches to apply the same acceptance criteria.

The observable outcome is a bounded rule such as “use arrangement A for these stable adapters, escalate when publication semantics change,” with the evidence and uncertainty attached. If the sample cannot distinguish the candidates, say so. An inconclusive comparison is more useful than a permanent model hierarchy built from impression.