AI wrote the tests. We made them count.

The team had wired a model into test authoring and the case count climbed fast. Six weeks later the escape rate had not moved, and a spot check found that about a third of the generated cases asserted the implementation back at itself — they would pass against a bug as readily as against a fix.

In a regulated domain that is worse than no coverage, because it looks like coverage. An audit asks which requirement a given test proves, and the model wrote it from the diff is not an answer anyone can file.

Generated from requirements, not from the diff

The generator's input became the requirement and its acceptance criteria. A case that cannot name the requirement it covers does not merge — which, as a side effect, produced exactly the traceability the compliance team had been asking for by hand.

Put a human gate on the first run

Every generated case is reviewed by the engineer who owns the requirement. The review turned out to be where most of the value sat: a model is good at enumerating cases and poor at knowing which of them matter.

Made each case prove it earns its place

Candidates run against a deliberately broken build before a human ever reads them. A test that passes while the feature is broken is discarded automatically. That single rule removed most of the tautologies without anyone arguing about them.

Attached a cost ceiling per requirement

Generation spend is tracked per requirement and per accepted case, not per call. Two teams were spending more on regenerating cases than those cases could plausibly be worth, and that now shows up in the same review as everything else.

Traceable requirement coverage went from 46% to 81%. The compliance lead's measure is the traceability rather than the percentage — every case now points at the requirement it exists for.

Authoring throughput settled at roughly three times the pre-AI rate, at 34% lower spend per accepted case than the unmanaged approach it replaced. The suite grew more slowly than before, and started catching things.

The mutation gate we are proudest of nearly sank the engagement. On its first full run it discarded 60% of the team's existing hand-written tests as well — true, and deeply unwelcome in week four. We should have scoped it to new cases first and presented the legacy result as a finding rather than as a deletion.