4,100 tests nobody trusted, cut to 900

Every pull request ran a four-hour suite, and roughly two runs in five failed for a reason unrelated to the change. The team's documented procedure for a red build was to rerun it — twice — before anyone investigated.

That habit is the real cost. A suite that cries wolf does not merely waste compute; it teaches an organisation to ignore the one failure a quarter that was telling the truth. Two of the three production incidents that year had been caught by the suite and reran away.

Measured every test before touching one

Ninety days of CI history, per test: pass rate, runtime, flake count, and whether it had ever failed on its own against a change that turned out to be genuinely broken. That last column is the only one that matters, and the overwhelming majority of tests scored zero on it.

Deleted, merged, or moved 3,200 of them

1,900 duplicated a unit test one layer down. 800 were UI tests asserting something an API test could assert in a hundredth of the time. 500 had not failed meaningfully in three years and were retired.

Fixed flake at the source instead of retrying it

Nearly all of it came down to three patterns: shared test data, implicit waits, and a seeded clock nobody had actually seeded. Retry logic came out of CI once the causes were gone — it had been hiding the evidence that would have found them.

Split the suite by what it protects

A 22-minute gate on every pull request, a fuller pass on merge, and a nightly run for the slow end-to-end journeys that genuinely earn their runtime. The gate is small enough that a red build is a signal again.

Pull-request feedback fell from three hours forty to twenty-two minutes, and flake settled at 2% — low enough that the team's rule is now investigate first, rerun never.

CI compute came down to roughly a third of the previous spend, which covered the engagement inside the year. The change the team actually talks about is quieter: people started reading the failures again.

Cutting 3,200 tests was easy to defend on a slide and hard to live through. We under-invested early in showing why each one went, and for about three weeks the deletions read as an outsider throwing away someone's work. A shared, per-test rationale from day one would have bought the same result with a fraction of the friction.