Six-week release trains, cut to one

A payments team shipping on a six-week train. Every train ended in a nine-day regression pass, run by four people against a spreadsheet of roughly 900 manual cases, and the go/no-go was a two-hour meeting where the most confident voice usually won.

The suite was not failing the way everyone assumed. Coverage looked healthy on paper, but almost all of it sat on the code that changed least — the ledger and the reconciliation jobs — while the newest surfaces, the ones the last three incidents came out of, were checked by hand or not at all.

Mapped coverage against risk, not against code

Eighteen months of incidents, support escalations and hotfixes, laid over the suite. Six areas carried most of the failure history. Two of them had no automated coverage whatsoever, and one had 140 cases guarding a screen that had not changed in two years.

Retired what nobody read

310 manual cases were deleted outright: duplicates, checks on removed features, and cases whose failure had never once changed a decision. A test is a product with running costs — these were being paid for on every train and returning nothing.

Made the release decision a checklist, not a meeting

Four gates, each with a named owner and a number behind it: risk coverage on the areas that changed, suite reliability across the last ten runs, open Sev-1 and Sev-2 count, and a rehearsed rollback. A gate is met or it isn't, and both answers are visible before anyone joins a call.

Ran the first three trains alongside the team

We did not hand over a document. Two of our engineers sat in the release calls until the team was running them without us, then left the runbook behind and stepped out.

Regression settled at two days, most of it unattended. The weekly train held through a quarter that included a payment-provider migration, and the release call now takes about fifteen minutes — the gates are already green or already red by the time it starts.

The figure leadership quotes is escaped defects per quarter, down 71% against the same period a year earlier. The one the team quotes is smaller and better: they stopped scheduling their work around the regression window.

We spent the first three weeks building a coverage dashboard nobody used. This team did not need another view of the problem — they needed permission to delete tests and a rule for when the train leaves. We would start with the gates now and let the reporting follow them.