A severemisalignmentof AI in softwareengineering.

Introduction: What are we actually optimizing?

This essay is inspired by “A Severe Misalignment of AI in Mathematics,” a declaration published on Terence Tao’s blog on September 11, 2026. Its warning is that solving problems is a means toward understanding, not understanding itself.^[1] What follows extends that question to software engineering. It is an independent commentary, not a translation of the declaration.

When we measure AI’s engineering value by issues closed, code generated, and tests passed, we should ask: are these measures of engineering capability, or simply measures of some of the visible artifacts that engineering produces?

Consider SWE-bench. It gives a model a real repository and a GitHub issue, then asks it to modify the code to resolve the problem.^[2] This is closer to actual work than a conventional programming puzzle. Yet performance on a bounded task cannot, by itself, establish competence across the entire software lifecycle.

Code itself is not the ultimate goal of software engineering. The real deliverable is a system that can keep operating and evolving under real-world constraints—and that can be understood, changed, and owned.

When AI optimizes for finishing the current task as quickly as possible, while an organization needs a system that can evolve safely for years, a severe misalignment can emerge.

Code generated, tests passed, and issues resolved are proxies for engineering capability—not the ultimate purpose of software engineering.

Passing the tests is not the same as doing the engineering

A programming puzzle usually has clear inputs, outputs, and criteria for success. Real engineering problems rarely arrive so neatly defined.

Suppose the task is to implement an account-balance deduction. A model can produce database updates, transaction logic, and unit tests. Every test may pass. The implementation may look clean and conventional.

But the engineer still needs to ask: can concurrent deductions overspend the balance? Are retries after a timeout idempotent? What happens when a message is delivered twice? Can a database failover violate consistency? How do we reconcile a downstream operation that succeeded when its caller believes it failed?

Then there are historical-data migrations, staged rollouts, rollback strategies, and ownership of monitoring. These concerns are often absent from the issue description and only partially represented in tests.

Tests provide evidence that the code behaves as expected under the assumptions and scenarios they express. They cannot, on their own, establish that the requirements and assumptions are correct, or exhaust everything that may happen in production.

A patch that passes the entire test suite has therefore not necessarily passed the full test of reality. Software engineering operates in an open, changing environment.

Completing the task that has been expressed and discovering the problem that has not yet been expressed are different capabilities.

An evaluation that rewards only test-pass rates may undervalue the latter. Yet identifying missing constraints, clarifying requirements, and recognizing unanticipated risks are central to experienced engineering judgment.

Issues are a proxy for capability, not the purpose of engineering

Issues, feature requests, and bug reports are more than a queue to empty. They can also be vehicles for understanding a system, improving its design, and developing the people who maintain it.

To fix an apparently small bug, an engineer may read historical code, trace a call chain, examine logs, reproduce the failure, and ask about the business context. Through this process, they build a working understanding of the system.

The resulting dozen lines of code may be the residue of hours or days of thought. Looking only at the final patch invites an overly simple conclusion: if AI can produce the same dozen lines quickly, the preceding process of understanding must have had no value.

But resolving an issue can also perform less easily measured functions:

• Update the team’s shared understanding and expose persistent architectural problems.
• Develop engineers’ ability to diagnose failures and assess risk.
• Make implicit business rules explicit in code, tests, and documentation.
• Establish who understands a part of the system and who can take responsibility for it.

If the number of patches increases without a corresponding increase in organizational understanding, a shorter queue does not necessarily mean a stronger engineering organization.

This is not an argument that every repetitive task must be done by hand. What matters is preserving the process through which work creates understanding, judgment, and accountability—not preserving manual effort for its own sake.

Code generation can outpace verification

One important change brought by AI is a lower marginal cost of producing code. The costs of reading it, validating requirements, assessing architecture, analyzing security risks, and owning production outcomes do not automatically fall at the same rate.

An organization may therefore encounter a new bottleneck: code arrives faster than the team can verify and understand it.

A requirement that could have been addressed by changing a shared abstraction may instead produce several local implementations. Logic that should have been removed may acquire another compatibility layer. A data model that needs redesigning may be patched over with more conditional branches.

Each change can pass the current tests. Each pull request can look reasonable in isolation. After hundreds of locally sensible changes, however, the system as a whole may become more complex.

Faster patch generation does not necessarily mean a lower total cost of ownership.

Costs may simply move from implementation into future review, debugging, maintenance, incident recovery, and refactoring. Human development can create the same pattern. AI may amplify it; it does not uniquely cause it.

A randomized controlled study from 2025 offers a cautionary example. Sixteen developers working on mature open-source projects they knew well completed 246 tasks. With the early-2025 AI tools tested in the study, task completion took 19% longer on average. Afterward, participants nevertheless estimated that AI had saved them about 20% of their time.^[3]

This was a result for a particular population, set of tasks, and generation of tools. It does not establish that AI generally reduces productivity, and it is not a measurement of tools available in 2026. It illustrates something narrower: feeling faster and achieving better end-to-end engineering productivity can be different things.^[3]

Beyond technical debt: cognitive debt

Engineering teams know technical debt: some compromises made for short-term delivery raise the cost of future change. AI-assisted development also warrants attention to another risk, which this essay calls “cognitive debt.”

Cognitive debt accumulates when a system still runs and its code may still pass tests, but the team gradually loses the ability to explain it. Why is this condition necessary? What business constraint is hidden in this interface? Which failure model justifies this retry policy? What historical data would be affected by removing this compatibility code?

Why can two apparently duplicate implementations not be merged? Was a particular design intentional, or merely an incidental result of generation?

With familiar technical debt, someone may at least know where a compromise was made. Cognitive debt is more elusive: a team may not know what it is missing.

AI can help explain what code does. The more important maintenance question, however, is often this:

Why must it work this way, rather than through another apparently simpler design?

The answer may lie in requirements discussions, past incidents, load tests, organizational boundaries, and failed experiments. The surface structure of the current code cannot reliably reconstruct all that context. Evidence-backed design records and traceable decisions still matter.

An organization that repeatedly merges code no one truly understands, without verifiable design rationale, may end up with a feature-rich, thoroughly tested system for which no one can take coherent, system-level responsibility.

Production incidents still require an accountable party that can exercise judgment, make decisions, and bear the consequences. Automatically generating code does not automatically remove organizational responsibility.

Do not break the chain of engineering development

One of an engineering organization’s most valuable resources is not just its codebase, but the people who can understand and evolve it.

When junior engineers implement small features, fix modest bugs, and write tests, they are not merely performing low-value labor. These tasks can also be entry points into understanding a system.

Through small changes, they learn to read unfamiliar code, clarify requirements, estimate the scope of a modification, identify exceptional paths, diagnose failures using logs, explain their choices in review, and take responsibility for a release.

If every simple task is automated without a replacement learning mechanism, a strange situation can arise: engineers who have not experienced the full path from requirements to design, implementation, and failure are suddenly expected to take over and review large AI-generated changes.

Senior engineers, meanwhile, may shift from designers to output reviewers, spending much of their energy inspecting an unending stream of patches rather than developing better abstractions.

In the short term, a team may deliver more features. Over a longer period, it may consume the environment needed to develop the next generation of experienced engineers. That cost is difficult to see in delivery-speed metrics.

This is a risk to manage, not an inevitable consequence of automation. AI can also be an explainer, a debugging partner, and a tutor. The question is whether a team deliberately builds learning into its workflow, rather than assuming that obtaining an answer is the same as developing a capability.

The real misalignment is between metrics and goals

The problem is not that machines can generate code. Repetitive implementation, additional tests, interface adaptations, migration scripts, documentation, and static checks can all benefit from machine assistance. The appropriate degree of automation still depends on risk and the conditions for verification.

The deeper question is: which metrics are used to direct that assistance?

If an organization mainly rewards lines of code, pull-request counts, issues closed, time to first generation, and automatic merge rates, it may mistake improvement in those metrics for engineering success itself.

The real goals are closer to these: a reliable system, reversible changes, controlled security risks, reasonable operating costs, safe adaptation to new requirements, and an organization still capable of responding to the unknown.

DORA’s 2025 report describes AI’s primary role as an amplifier of an organization’s existing strengths and weaknesses. Returns depend not just on the tools, but on the underlying organizational system.^[4]

An engineering implication follows: a team with clear architecture, trustworthy tests, and explicit ownership may use AI to accelerate valuable work. A team with confused requirements, missing monitoring, and unclear ownership may instead expand its existing problems faster.

Code output is a means. Reliability, evolvability, and the team’s capacity for judgment are closer to the ends.

Realigning AI with software engineering

An AI engineering workflow should ask not only whether the model completed the task, but whether the change strengthened the long-term capabilities of the system and the team.

First, evaluate the whole change lifecycle. A strong engineering agent should do more than generate an implementation. It should help identify ambiguous requirements, make assumptions explicit, state system invariants, analyze failure models, assess impact, and propose migration, staged-release, rollback, and monitoring plans. Not every small change needs a heavyweight process. But the process must include verification appropriate to its risk.

Second, avoid deriving all verification from the same source. When one model interprets a requirement, implements it, generates its tests, and reviews its own changes, the result can be internally consistent while overlooking a shared false assumption. Switching to a second model does not automatically make verification independent. What matters more is introducing evidence from different sources.

Independent specifications, existing tests, property-based tests, static analysis, security checks, production observations, and human reviewers with relevant context can form a broader verification system. The tests themselves need scrutiny; counting them is not enough.

Third, match autonomy to risk. Copy changes and low-risk internal utilities may warrant lighter review. Changes involving money, permissions, privacy, data migrations, or core consistency need explicit ownership, sufficient verification, and executable recovery paths. A short patch is not necessarily a low-risk patch.

Fourth, make understanding part of the deliverable. A change should leave behind the necessary design rationale, key assumptions, verification results, and unresolved questions—not just a diff. The goal is not long documents. It is enabling the next maintainer to find a trustworthy answer to “why.”

Fifth, preserve the process of learning. AI can reduce mechanical work, but teams still need deliberate opportunities for code reading, debugging, design discussions, releases, and incident reviews. Ask newer engineers to explain an AI-generated solution, find counterexamples, and compare alternatives—not merely click Accept.

These practices are not intended to slow AI down. They are intended to make it accelerate valuable engineering work rather than simply accelerate the flow of code into a repository.

Conclusion: Code was never the only scarce resource

The misalignment worth watching in AI-assisted software engineering is not merely the generation of incorrect code. Obvious defects may be caught by tests, reviews, and alerts. Other defects can remain latent for a long time and must not be underestimated.

A subtler risk is that an organization keeps accepting code that looks correct, runs, and passes existing tests, without accumulating a corresponding understanding of the system. Eventually, it may lose sight of what software engineering was meant to achieve.

The outcome of software engineering is more than code. It is a system that can continue to deliver value, together with people who can understand it, change it, and take responsibility for it.

The measure of AI’s engineering value should not be only how many changes it helps us make per unit of time. We should also ask:

After each change, do we have a more reliable system—and more understanding of that system, rather than less?

AI becomes aligned with software engineering when it increases not only code output, but also an organization’s understanding, judgment, and capacity to act over the long term.

AI is making writing code cheaper. But what software engineering truly lacks has never been code alone: it is understanding, verification, and accountability.