Surviving a sale day at 9x normal traffic

The situation#

The previous festive sale produced a forty-minute checkout outage at peak. Autoscaling was configured and had never been tested under a realistic traffic shape. Support volume tripled and first response times went past a day.

We had five weeks.

What we did#

Weeks 1–2 — rehearsed peak. We built a traffic model from the previous year's access logs and load tested a copy of production at 10x. Three failure modes surfaced: autoscaling that took eleven minutes to add capacity, a database connection pool that saturated well before compute did, and a third-party recommendation call with no timeout that blocked checkout rendering.

Weeks 2–3 — fixed and re-tested. Pre-warmed capacity ahead of the announced start time rather than relying on reactive scaling, connection pooling reconfigured, and every third-party call given a timeout and a fallback. Then load tested again — twice.

Weeks 3–4 — support automation. Order status, returns initiation and address changes automated end to end against their commerce platform, with an escalation path for anything below the confidence threshold.

Week 5 — a rehearsed degradation plan. An agreed sequence for shedding non-essential features under extreme load, with the toggles tested. Recommendations and personalisation go before checkout does.

Sale day#

Peak reached 9.2x baseline. Checkout availability was 99.99% with no customer-visible incident. Support automation resolved 65% of inbound volume without human involvement; first response on the remainder stayed under fifteen minutes.

The degradation plan was not needed, which is the correct outcome for a plan of that kind.

After the sale#

They moved to a NoOps retainer. Right-sizing back down outside peak reduced the run-rate 38% against pre-engagement spend — the platform is now both more reliable and cheaper than the version that fell over.