Coding-agent productivity needs a systems denominator

The coding-agent industry needs to stop treating activity as an adequate proxy for productivity. Generating more patches can be useful, but the benefit is not established until those patches become dependable changes at a defensible cost. Review, testing and infrastructure are part of that cost, even when they belong to someone else’s budget.

OpenAI’s November 2 Pando engineering account is a useful prompt for this argument. It describes an internal Git backend adapted to agent-heavy repository demand. The company reports a substantial reduction in median push latency. That is an infrastructure achievement, with a narrower denominator than software delivery.

The distinction should become standard in AI coverage. A model can improve, an engineering system can improve and a team can become more productive, but these are three claims requiring different evidence. This editorial argues for measuring the conversion between them. The Frontier Wire has not benchmarked Pando or run a new developer-productivity study.

The argument

• Generated output is an intermediate artifact, not a complete productivity measure.
• Review effort, shared infrastructure and correction costs belong in the accounting.
• More parallel work is valuable when it increases accepted, useful delivery under stable standards.
• Teams should publish enough of the task mix and measurement boundary to make the result interpretable.

The easiest metric is often the wrong incentive

Lines of code, commits, model calls and patches are easy to count. That makes them attractive dashboard material. It does not make them reliable measures of value. A small deletion can remove a dangerous behavior; a large generated change can create work without solving a relevant problem.

The problem is not that activity metrics are useless. They can diagnose adoption, capacity and workflow changes. The mistake is promoting them into an outcome measure without showing how they relate to what the organization needed to accomplish.

Once a metric becomes a target, the workflow can adapt around it. Rewarding patch count encourages smaller patches or more proposals, including proposals that would never have been worth making manually. Rewarding generated lines can discourage simplification. Rewarding quick local success can move verification work downstream.

A serious agent evaluation should therefore start with the intended contribution. Is the goal to clear a backlog, investigate an unfamiliar system, improve reliability or produce a new feature? The answer determines which outcomes should be counted and which forms of activity are merely supporting observations.

Infrastructure absorbs the new demand

An agent does not work in isolation from the development system. Its environment reads code, installs dependencies, executes tests and stores artifacts. A proposed change then interacts with source control, continuous integration and the people responsible for accepting it.

Parallel execution amplifies that shared demand. A hundred agents may create many more than a hundred repository operations or automated jobs. A single task can generate several attempts, and each attempt can repeat expensive setup before producing anything a reviewer would consider useful.

Git’s push reference concerns updating remote references and associated objects. Its pack-objects documentation describes the packed data used in repository transfer. These are concrete units of infrastructure work. Neither operation tells us whether the proposed software behavior is desirable.

Pando’s significance is that a familiar workflow needed different backend machinery under a new demand pattern. That is a valuable systems story in its own right. It becomes a productivity story only when evidence connects the infrastructure improvement to the delivery of useful software.

A local speedup is not a team-wide multiplier

Suppose a team makes one stage of its workflow much faster. The total improvement depends on how much that stage constrained the complete process. If review dominates the elapsed time, faster repository writes may remove frustration without multiplying delivery at the same rate.

That is not a criticism of the optimization. Engineers should improve specific bottlenecks. It is a criticism of claims that transfer a percentage from one operation to an entire organization without measuring the connection.

The same caution applies to coding agents. A tool might make a first draft faster while leaving debugging unchanged. It might reduce typing but increase time spent checking unfamiliar suggestions. It might improve routine tasks while adding overhead on work where the developer already knows the relevant code exceptionally well.

These possibilities can coexist. A useful study should reveal the conditions under which the tool helps rather than search for a single multiplier that describes every engineer, task and repository. The strongest result is a qualified comparison that another team can recognize as relevant to its own work.

The historical evidence argues for measurement

The METR study of early-2025 AI tools reported a slowdown in its randomized study of experienced open-source developers working in familiar repositories. The tools, tasks and population were specific to that earlier setting. The result should not be presented as a verdict on every current coding agent.

Its enduring lesson is that perceived assistance and measured completion time can diverge. That makes direct observation valuable. A team cannot establish productivity merely by asking whether developers feel faster, just as it cannot establish usefulness by counting how often a tool was invoked.

The 2024 DORA research overview also describes tradeoffs between individual experience and delivery outcomes. It is observational research with its own scope, not a randomized causal estimate for today’s products. Its relevance is that improvements at one level need not move every organizational metric in the same direction.

Together, these sources justify a modest standard: measure the thing being claimed. A claim about developer experience needs experience evidence. A claim about delivery needs delivery evidence. A claim about infrastructure needs infrastructure evidence. Keeping them separate makes genuine gains easier to trust.

The accepted change is a better starting unit

An accepted change has crossed a meaningful boundary. Someone or some process has judged that it meets the project’s requirements well enough to integrate. That makes it a more useful starting denominator than an unreviewed suggestion, although it is still not a complete measure of business value.

The acceptance rule must be explicit. Passing a small test suite is not identical to maintainer approval. Maintainer approval is not identical to a successful rollout. A benchmark can legitimately choose one of these boundaries, but should identify the one it uses.

The SWE-bench paper supplies a research precedent for evaluating repository-level issue resolution. Such tasks are closer to practical engineering than isolated snippets, while remaining defined experiments. Their resolution criteria should not be casually equated with every organization’s entire release process.

For team measurement, count accepted changes by task class and include the effort required to obtain acceptance. This does not require inventing a perfect universal value score. It requires moving the main denominator past the stage where producing output is mistaken for completing useful work.

Review labor cannot disappear from the bill

A developer who checks an agent’s patch is doing engineering work. The time may be short or substantial. It may involve understanding the change, detecting hidden assumptions, asking for revisions or rebuilding confidence in a part of the system the agent touched.

If that effort rises while generation becomes cheaper, a model-only cost calculation can improve even when the team’s total effort does not. The accounting is not wrong because the model bill is inaccurate. It is wrong because the boundary excludes a resource necessary to obtain the claimed result.

Review also has an opportunity cost. An experienced maintainer examining low-value proposals is unavailable for architectural decisions or difficult incidents. That does not mean every review minute should be priced with an elaborate economic model. It means the work should be visible before a cheap-patch claim becomes a cheap-delivery claim.

A practical report can begin simply: active review time, number of revision cycles and reasons for rejection. The purpose is diagnostic as well as evaluative. If the main burden is checking tests the agent could have run, the next improvement differs from one aimed at reducing subtle specification errors.

Quality standards must remain stable

A productivity comparison becomes difficult to interpret if acceptance standards change between conditions. A team may accept more agent-generated work because the backlog feels urgent, or scrutinize it more because the tool is unfamiliar. Either response can affect the observed conversion rate.

A good study should define the relevant quality threshold before results are known. Where judgment is unavoidable, use a consistent rubric and record disagreements. If reviewers can be blinded to the generation method without harming the task, that may reduce one source of bias.

The standard should include preservation of existing behavior where required. A patch that fixes the stated issue by removing a useful feature is not equivalent to one that meets the requirement without that loss. Tests and review need to reflect the actual contract rather than only the easiest visible outcome.

Stable standards also protect the interpretation of progress over time. If a later system solves more difficult tasks, that is important and should be reported. If it solves easier tasks under looser review, a higher acceptance rate does not establish the same advance.

A systems scorecard

The following is an editorial proposal for reporting agent-assisted engineering. It is not a result from OpenAI, METR or a new Frontier Wire experiment.

The value of this scorecard is the relationship between its rows. More candidates can be useful when acceptance grows without an unacceptable increase in review. Faster tests can be useful when they reduce elapsed delivery time or reveal more defects. Each connection needs evidence.

A single aggregate score may be appropriate for an internal decision, but publish the components when making a broader claim. Otherwise, readers cannot tell whether an improvement reflects stronger models, easier task selection, more infrastructure or a changed quality threshold.

Queues connect the stages

Workflow stages do not merely consume service time. Work can also wait between them. A fast agent may finish a patch and then spend most of the day waiting for CI capacity or review. Increasing parallelism can make that waiting more visible.

Google’s SRE chapter on monitoring distributed systems provides background for distinguishing internal activity from user-visible behavior and for examining latency distributions. The analogous engineering question is how long useful work takes from request to accepted outcome, including the gaps between stages.

A team should retain start and completion timestamps for the main transitions and count work that remains unfinished at the end of the measurement window. Looking only at completed tasks can make a congested process look healthy by excluding its growing backlog.

The median is useful, but the tail matters too. Occasional long waits can disrupt a workflow or create repeated retries. A system with a fast typical operation and unreliable bursts may impose more total work than its central statistic suggests.

More attempts have to earn their cost

Parallel agents make it tempting to launch several approaches and keep the best one. That can improve success. It also changes the resource budget and can create duplicate work. A comparison needs to preserve both sides of the tradeoff.

A useful report would separate first-attempt success from eventual success under a declared retry policy. It would count all attempts, not only the winning trajectory. The same applies to human assistance: a solution obtained after a maintainer supplies the key insight differs from one obtained without that intervention.

There is nothing inherently illegitimate about search. Engineering often involves trying alternatives. The problem is presenting a result that benefits from many attempts as if it came from a single inexpensive pass, or comparing it with a baseline that received fewer opportunities.

This is another reason to measure accepted work per total declared resource. The team can then decide whether additional parallelism is worthwhile for its task mix. A system may justify expensive search on difficult work while being wasteful on routine changes a developer could make directly.

A concrete accounting example

Imagine a fictional team comparing two workflows on a matched set of maintenance issues. One produces many candidate patches quickly, but reviewers reject a large fraction and request several rounds of repair. The other produces fewer candidates, with a higher share accepted after straightforward review.

A patch-count dashboard would favor the first. A model-bill-per-candidate calculation might also favor it. A delivery measure could favor either, depending on accepted outcomes, task difficulty, human effort and infrastructure demand. There is no contradiction; the metrics describe different stages.

The team should not resolve that ambiguity by choosing whichever denominator makes its preferred product look best. It should publish the transition counts and resource boundary. If the first workflow ultimately delivers more accepted work, that is a real benefit. If it merely creates a larger review queue, that is a different result.

Now add a post-release observation period. Some accepted changes need corrective work. Record that work without pretending every follow-up is a failure: requirements evolve and ordinary software also needs maintenance. The purpose is to compare the nature and burden of corrections under consistent rules, not to demand impossible perfection.

Task mix determines where agents help

A team clearing repetitive dependency updates faces a different problem from a team changing a subtle concurrency invariant. Both produce code. Their verification needs, uncertainty and failure costs differ, so pooling them into one average can hide the most useful pattern.

Classify tasks before interpreting the result. Relevant dimensions might include familiarity with the repository, clarity of the specification, availability of tests and interaction with other changes. The classification need not be elaborate, but it should explain why the sample represents the work being claimed.

This matters for external comparisons. A high resolution rate on curated tasks does not directly establish the same rate on a team’s incoming backlog. A low rate on unusually difficult tasks does not mean the system has no practical use. The evaluation needs a bridge between the test population and the intended workflow.

HELM supplies a broader precedent for exposing performance across scenarios instead of relying on one aggregate. For coding agents, the corresponding obligation is to show which kinds of engineering work improved and which remained expensive or unreliable.

Coordination is part of capability at scale

Several agents can work on separate branches while making semantically interacting changes. File isolation prevents some collisions; it does not establish that the resulting behaviors fit together. Integration is a reasoning and coordination problem as well as a source-control operation.

A useful study should include tasks with different relationships: independent work, overlapping files and changes that interact through a shared interface. Increasing concurrency on only the first category can show capacity without testing the coordination demands of a more coupled backlog.

The complete system also includes how tasks are assigned and how agents receive feedback from one another. If one agent’s discovery helps several others, that is a legitimate systems benefit. If several agents independently rediscover the same failure, that is a cost. Both should be visible in the evidence.

This is where a systems denominator becomes more than accounting. It identifies the capability under study. A model’s ability to write a patch is different from an agent system’s ability to coordinate many patches into a coherent release. Claims about the latter need experiments at that level.

Teams need experiments, not surveillance dashboards

The measurement standard proposed here should not become an excuse to rank developers by activity. Individual output is entangled with task allocation, maintenance responsibility and work that prevents future problems. A dashboard can easily punish the person doing the difficult review that makes everyone else’s apparent productivity possible.

Use the metrics to evaluate workflow changes and diagnose bottlenecks. Compare conditions with a clear hypothesis, preserve context and inspect failures. A small, credible experiment is more useful than an elaborate dashboard that assigns false precision to incomparable work.

A practical rollout can begin with a limited task class and a stated observation period. Ask whether the agent reduces total effort or improves useful delivery under the existing standard. Then expand the task mix only when the evidence supports the next claim.

The outcome may be mixed. That is normal and actionable. A tool may be excellent for investigation but weak for final implementation, or helpful to newcomers but distracting for maintainers who know the repository deeply. Reporting those differences improves product decisions rather than weakening the case for AI.

Avoided work deserves an explicit place

Some of the most useful engineering produces no patch at all. An investigation may show that a proposed change is unnecessary, that an apparent defect is a documentation problem or that a risky migration should wait. A productivity system organized only around merged changes would miss those outcomes too.

The answer is to define the task outcome at the beginning. If the requested work is an investigation, an accepted decision record can be the artifact. If the work is a correction, the artifact may be a tested patch. The measurement should follow the intended contribution rather than force every activity into the same visible-output category.

This matters for agent evaluation because a system can appear productive by implementing work that a better investigation would have ruled out. The implementation may be technically competent while the project is worse off. A complete evaluation needs a way to credit discovering that the premise is wrong and explaining the evidence clearly.

These outcomes require judgment, which is why they should be documented rather than reduced to arbitrary points. Record what decision changed, what evidence supported it and who accepted the result. That creates a reviewable contribution without pretending that every useful insight has a universally comparable numerical value.

A systems denominator should therefore be understood as accepted useful outcomes, with accepted changes as one important class. That refinement keeps the argument from replacing one simplistic metric with another. The purpose is to make value and its required work more visible, not to insist that all engineering looks like a commit.

The publication standard should change

A credible productivity announcement should identify the unit of useful work, the population, the task mix, the comparison condition and the accounting boundary. It should distinguish model measurements from infrastructure measurements and connect either to delivery only when that connection was observed.

This standard does not require every engineering blog to become a randomized trial. A descriptive deployment report can be valuable on its own terms. It should simply avoid borrowing the language of causal productivity gains when it reports a narrower operational improvement.

Our account of benchmark sample sizes explains why small differences can be difficult to rank. Our comparison of coding agents examines the workflow context around model capability. Both point toward the same editorial discipline: state the system and the evidence behind the conclusion.

OpenAI’s Pando report is a useful example of infrastructure responding to a change in demand. The broader obligation belongs to the industry: show what that demand produces. Coding-agent progress becomes economically meaningful when more dependable software reaches users with an acceptable total burden. Until then, a larger stream of patches is evidence of activity, and the productivity question remains open.