Peak season, without the war room

Two peak seasons in a row, checkout had degraded within an hour of the traffic spike and stayed degraded. Both times the team scaled the web tier. Both times it made no measurable difference, and both postmortems concluded "more capacity next year".

Nobody had load-tested the path end to end. The tests that existed hit the storefront, which cached beautifully and told them nothing about the two synchronous calls checkout made to an inventory service that cached not at all.

Rehearsed the real journey, not the landing page

A load profile built from the previous year's actual traffic shape — including the ten-minute cliff when the campaign email went out — driving browse, cart, checkout and payment against a production-shaped environment.

Followed the first thing that broke

At three times normal load the failure was not the web tier at all. Inventory reservation held a database row lock for the duration of an external payment call, so checkout throughput was capped by a third party's latency and no amount of web capacity could move it.

Fixed the mechanism, then re-ran the same profile

Reservation moved off the synchronous path with a compensating release, and the payment call got a time budget and a fallback. Every change was re-run against the identical profile, so each figure on this page has a chart behind it rather than an argument.

Made the rehearsal a habit

The profile now runs monthly in CI against a scaled-down environment, with the thresholds that mattered wired in as pass/fail gates. Peak readiness stopped being an annual project.

The rehearsal held at eleven times normal peak — comfortably past the previous year's high-water mark — with p95 checkout latency at 780ms against the 4.2 seconds measured on day one.

Peak week ran with no Sev-1 incidents and no war room. The on-call log for the busiest evening of the year has a single entry in it, and it is about a dashboard.

We nearly missed it. The first two weeks went into the web tier, because that is where the previous postmortems pointed — and the previous postmortems were wrong. Load-testing the journey rather than the endpoint should have been week one; we let an existing narrative choose our starting point.