More than 12,000 unit tests, all green. And still the bugs show up at the release boundary.

This is what Paul Stack reports from Swamp, where agents write every line of code. In two recent posts, Your Agent-Written Tests Aren’t Real Tests and Why Agent-Written Code Needs Product Verification, he explains why. I agree with almost everything he writes. And I think his conclusion is the best argument for Spec-Driven Development I have read this year.

In this post, I explain his main point, show how the AI Unified Process already answers it, and what I changed in my own plugins.

The Same Misunderstanding, Written Twice

When an agent writes a function and then the unit test for it, both come from the same understanding of the requirement. If the agent got the requirement wrong, the test is wrong in the same way. The code passes the test. The build is green. The product is still wrong.

It gets worse. When a test fails inside the agent loop, the fastest way to green is to change the test. For the agent, a test is just another file it can edit. At Swamp, the UAT suite ran about 2,000 times and found real defects in 80 runs. All of these defects had passed the full unit test suite.

Stack also points to research. Asking agents to write tests did not help them solve more issues. Most agent-written test patches have weak assertions or no real oracle at all.

So what are these tests? Stack says they are a specification, not verification. They are a useful map for the next agent. They show what the code is supposed to do. But they are not proof that the code is right.

It Is Not About the Test Level

A common answer right now is: invert the test pyramid. Write fewer unit tests and more end-to-end tests. Stack says this is not enough, and I agree. If the agent writes the end-to-end tests from its own understanding, they confirm the wrong thing too. They just look more convincing.

The key sentence of his second post is this: independence is a property of where the expectations come from and who can change them. Not of the test level.

This gives us two simple rules:

  1. The expectations must come from outside the implementation loop. From a product definition, from domain experts, from a specification a human has approved.
  2. The expectations must be protected. If the agent can edit a failing test until it passes, the test is worthless as a check.

Stack also makes a useful split. The agent may maintain the test mechanics: selectors, fixtures, setup. This is routine work, and agents do it well. But a change to an assertion, an expected value, or the list of covered capabilities needs review outside the loop.

Unit tests do not disappear. They are still valuable where the rule is defined outside the code: a pricing rule, a permission model, a regulatory check. The question is always the same. Does the test check a rule that someone else defined, or does it only repeat what the implementation already does?

Where the AI Unified Process Puts the Oracle

When I read Stack’s posts, I recognized the reason why I built the AI Unified Process around requirements and use cases, and not around code.

In the AI Unified Process, the expectations live in the specification, before any code exists:

  • Use case specifications describe the main success scenario, the alternative flows, the postconditions, and the business rules. A human reviews and approves them.
  • Test cases chain several use cases into one user journey. They contain concrete test data with literal values and validations that you can observe in the UI. Placeholders like “a valid customer” are not allowed, because you cannot automate them.
  • Implementation and tests are separate steps. The implementation skill does not write tests. The test skills derive the tests from the use case or test case. The Playwright tests are black-box tests against the running application.
  • Every test points back to the spec. A @UseCase annotation names the use case, the scenario, and the business rules a test method covers.
  • The coverage check is independent. /coverage-check runs in a separate, read-only sub-agent. It audits code and tests against the specification, never against a specification reconstructed from the code. A @UseCase annotation alone counts as a claim, not as evidence. The auditor reads the test body.

So the question “who owns the oracle?” has a clear answer:

ArtifactOwned byThe agent may
Requirements, use case specs, business rulesBusiness and analysts, approved by a humanPropose changes, never decide them
Test cases (TC) with test data and validationsBusiness and testersPropose changes, never decide them
Test mechanics (selectors, fixtures, setup)The agentChange freely
Assertions and expected valuesThe specificationChange only when the spec changes
Production codeThe agentChange freely

This is the shift-left idea applied to verification. The oracle exists before the code, and a human owns it.

What I Changed in My Plugins

The method is right, but Stack’s posts made me look at my own plugins with fresh eyes. I found gaps, and I fixed the most important ones right away. The changes are in the marketplace for all four stacks: Vaadin with jOOQ, Angular with JPA, Blazor with .NET, and NestJS with Next.js.

1. A rule for failing tests. Before, the test skills told the agent to run the tests and check that they pass. They did not say what to do when a test fails. Now every test skill has a section “When a Test Fails”. The specification is the oracle, not the code. Every expected value comes from the specification or the test data, never from a run of the code. When a test fails, the agent first decides which side is wrong:

  • A problem in the test mechanics, like a locator, a wait, a fixture, or setup: the agent fixes the test.
  • A value or outcome the specification states, like a message, a count, a status, or a calculated value: the code disagrees with the specification. The agent must not change, weaken, or delete the assertion, and must not disable the test. The test stays red. The agent reports the rule, the expected value, and the actual value. Which side is wrong is my decision, not the agent’s.

The skills even name the typical tricks: isGreaterThan(0) instead of the real count, isNotEmpty(), a partial match instead of the full message, @Disabled, or .skip.

2. The implementation does not touch the oracle. /implement never edits an existing test or anything under docs/ any more. If a test fails against the new code, or if a specification looks wrong, it goes into the Open questions of the report. The agent must not make the two agree by editing one of them.

3. Better examples. Agents copy the reference tests, so they must show the right thing. The old templates used isGreaterThan(0). Now they assert the exact values from the test data, for example exactly 100 persons, and cite the specification. The Playwright template now follows the naming of the use case.

4. Shapes from the code, values from the spec. The Hilla test skill reads the generated client to learn method names and DTO shapes. That is mechanics. The expected values still come only from the specification.

5. Reverse engineering creates drafts. When /reverse-engineer recovers use cases from existing code, it now always sets the status to Draft. A recovered specification describes what the code does, not what it should do. If the code has a bug, the bug is now written down as required behavior. So a human must review and approve the use case before any test is derived from it.

What Comes Next

Some ideas need more work, and I will come back to them:

  • Technical protection. The rules above are instructions to the agent. A hook or a CI check that fails when an assertion changes without a spec change would make them hard rules.
  • Examples for every business rule. A small table per rule with input, expected outcome, and boundary values. /spec-review could warn when a rule has none.
  • A defect flow. When a bug escapes, fix the spec first: a new alternative flow, business rule, or test case. Then the regression test. Then the code.
  • Tests from the spec only. Write the tests in a separate session that sees the specification but not the implementation.
  • Unit and architecture tests where they belong. Rule-level unit tests for pricing or permission rules, and ArchUnit tests for rules from the software architecture document.

Conclusion

Agents made code cheap. They also made end-to-end tests cheaper to maintain and bugs faster to fix. But they did not make good expectations cheap. Someone still has to decide what the product must do, and that someone should not be the agent that writes the code.

More tests do not solve this. More coverage does not solve this. What solves it is a specification that a human owns, that exists before the code, and that the agent cannot change to get green.

This is the core of Spec-Driven Development, and the core of the AI Unified Process. Thank you, Paul Stack, for putting it so clearly. And for the push to make my own plugins better.

Keep IT simple.

Sources