An on-sale that stopped falling over in the first two minutes

The situation#

Every significant on-sale followed the same script. The announced time arrived, traffic went up fourteenfold in ninety seconds, checkout slowed, everybody refreshed, and the platform was effectively unavailable for the first three to five minutes.

By the time it recovered, a visible share of the good inventory had gone to buyers who were clearly not fans, and social media had already formed a view.

Behind that, the operational surface was manual. Transfers, name changes, partial refunds and waitlist promotions were handled by two people working through a shared inbox, which was manageable between events and completely overwhelmed for the seventy-two hours after each on-sale.

What we did#

Weeks 1–3 — modelled the cliff and tested against it. Not sustained load — the actual shape, with a browse-heavy first thirty seconds and a checkout-heavy next two minutes, against production-scale inventory. Contention on a limited allocation behaves nothing like contention on an open one, and testing against a large synthetic allocation would have hidden the real failure.

Weeks 4–9 — queue and read path. Server-side queue tokens issued centrally, a cache-first read path for the browse phase, and inventory holds moved to a design that does not serialise every purchase through one row. Clear feedback in the queue, because an honest wait with a position indicator stops the retry loop and an ambiguous spinner does not.

Weeks 10–13 — access controls. Velocity limits per identity and payment instrument, anomaly detection on purchase patterns, and payment-level checks that make disposable identities expensive. Layered, server-enforced, and scoped as raising reseller cost rather than eliminating them.

Weeks 14–16 — ticket lifecycle automation. Transfers, name changes, in-policy refunds and waitlist promotion handled by the system within configured authority limits, with contested and out-of-policy cases routed to a person with the case assembled.

The queue argument#

The operator resisted queueing for three weeks. The concern was reasonable — making customers wait feels like losing them.

The load testing settled it. Without a queue, every user who experienced slowness retried, and the retries were generating more load than the original demand. The platform was not failing because of the number of people who wanted tickets; it was failing because of the number of times each of them asked.

A queue with a visible position converted an unbounded retry storm into an orderly sequence, and the measured completion rate went up rather than down.

What we would do differently#

We built the reseller controls before instrumenting what resellers were actually doing.

The first set of controls was designed against the general pattern — velocity, identity reuse, payment instrument reuse. It caught a reasonable share. Once we added proper instrumentation two months later, the dominant pattern turned out to be something we had not designed for, and a control targeted at it outperformed everything built earlier.

Measurement should have come first. We now instrument the abuse before designing the defence, on the same principle we apply to performance work — you cannot fix what you have not characterised, and reasoning from the general case produces generic controls.

Where it went next#

The operator moved to a NoOps retainer covering on-sale operations, with a rehearsal before each major event rather than a standing assumption that last time's configuration still holds.

Attendee data unification is in progress — the operator knew who bought and not who attended, which meant every campaign started from an undercount. That is data platform work and it compounds across events rather than being spent on one.