26 Measure before optimizing

An agent notices that the test suite repeatedly constructs a temporary repository. It proposes sharing a fixture. That may be an excellent optimization. It may also allow one test's mutations to affect another, or save little time because compilation dominates the run. The first task is to discover which situation applies.

Optimization begins with a desired consequence. “Fewer subprocesses” is a mechanism. “A developer gets reliable feedback soon enough to stay in the task” is an outcome. Intermediate measures can explain the outcome, but they should not quietly replace it.

The same applies to agent workflows. Fewer tokens can mean less irrelevant context or less necessary information. More parallel workers can reduce waiting or increase integration repair. A faster first patch can still take longer to accept. Measurement should follow the result the team actually values.

Establish the unit of comparison

Consider an illustrative investigation of a slow standards-engine test suite. The hypothesis is that repeated fixture construction, rather than the assertions themselves, dominates execution. No performance result is asserted here; this is a worked measurement design.

An investigation record could be:

Outcome: Reduce the time to reliable suite feedback without changing what the suite checks or introducing order dependence.

Baseline: Record the exact revision, test selection, command, machine, process settings, and cache conditions. Separate setup time, test-body time, and teardown where feasible.

Hypothesis: Repeated construction of an equivalent immutable corpus accounts for enough of the run to justify reuse.

Candidate change: Reuse only the immutable source fixture. Give each mutating test its own derived state.

Comparison: Run the same workload before and after under comparable conditions. Include repeated runs, changed test order where relevant, and checks that a mutation cannot leak between tests.

Decision: Retain the optimization only if the observed improvement matters and the isolation contract remains credible.

The record prevents an easy mistake: changing both the fixture and the test workload, then attributing the whole difference to reuse. It also keeps correctness in view. A fast suite that occasionally inherits a previous test's state is worse feedback, even if its median runtime improves.

Measure enough to answer the decision. A small isolated optimization may need simple timings and a clear distribution of observed variation. A noisy shared runner may require more repetitions or a more controlled comparison. Sophisticated statistics cannot rescue a workload that does not represent the actual use.

Interpret intermediate measures carefully

Suppose instrumentation confirms that the candidate launches fewer processes. That supports a mechanism claim. It does not yet establish shorter end-to-end execution, lower memory use, or faster development. Those are separate observations.

Likewise, a smaller agent-tool catalog can reduce payload bytes while increasing discovery calls. A narrower task can reduce context while shifting specification work to the coordinator. These tradeoffs may be worthwhile. They need to be counted where they occur.

For the running resource application, total acquisition latency can be decomposed into locating the source, transferring bytes, checking, publishing, and making the result visible to the consumer. If checking dominates a particular workload, improving connection setup may not change the experience materially. If publication blocks all readers, average transfer speed may be the wrong priority altogether.

Use representative sizes and failure cases. A tiny cached fixture may hide the path that matters. An enormous synthetic input may exaggerate a bottleneck absent from normal use. Report what the workload represents and what it excludes.

Measure development without confusing its clocks

Active human time, elapsed delivery time, and machine cost are different quantities. A background agent can require little attention while taking hours. Several simultaneous tasks can overlap elapsed time while competing for the same review capacity. Summing their wall-clock durations does not yield the person's working time.

METR's early-2025 randomized study found a slowdown for experienced contributors using the evaluated tools on familiar mature repositories. That is a result in a particular population and period, not a permanent verdict on coding assistance. Its 2026 continuation warns that selection effects and measuring concurrent-agent work make newer estimates unreliable. The useful response is careful local measurement, not choosing whichever headline confirms an existing preference. Early-2025 trial, 2026 update.

A local comparison should include specification, supervision, unsuccessful attempts, review, repair, and integration. Keep quality criteria stable. If one workflow produces more ambitious results, record that change in scope rather than calling the longer duration a simple slowdown or the extra output a simple speedup.

Subjective experience matters too. A developer may find a workflow less tiring or more enjoyable without finishing faster. That can be valuable, but it is a different claim. Record it in its own terms rather than forcing it into a productivity number.

Look for the receiving bottleneck

Daniel Stenberg's April 2026 curl account describes a shift from low-quality report overload toward useful findings arriving faster than the project could comfortably remediate and distribute fixes. The changing report mix and operating conditions prevent a simple causal attribution to AI. The maintenance lesson is nevertheless concrete: better discovery can move the limiting work downstream. High-Quality Chaos.

For agent-assisted development, the receiver may be the reviewer, integrator, release engineer, or user who must evaluate changed behavior. Improving generation without considering that receiver can enlarge a queue of plausible unfinished work. A useful optimization may therefore reduce batch size or improve evidence rather than increase patch output.

Once the relevant checks pass and the decision is supported, stop measuring for this change. More instrumentation has its own maintenance cost. Keep enough evidence to understand the result and revisit it when the workload or environment materially changes.

Test one suspected bottleneck

Choose a delay you routinely notice. State its consequence, measure the baseline, and identify one change that could affect it. Hold the workload and acceptance criteria steady while comparing the candidate. Include an unsuccessful or noisy observation rather than retaining only the best run.

The exercise succeeds when you can say what improved, by how much under the observed conditions, and what did not follow from the measurement. Discovering that the suspected bottleneck is unimportant is a useful result: it prevents a confident optimization of the wrong thing.