Shipping a stalled AI feature in nine weeks

The situation#

The prototype was built in a fortnight by a strong engineer and demoed to the board. Seven months later it was still behind a feature flag for eleven internal users.

The blockers were not technical mysteries. They were unanswered questions: what is the accuracy, what does it cost per active user, what happens at 3am when the provider returns 529s, and what stops a customer pasting instructions that leak the system prompt.

What we did#

Weeks 1–2 — built the baseline. We took 400 real documents from production traffic, had their subject-matter experts agree correct summaries, and scored the existing prototype. It was substantially worse than the demo suggested, mostly because the demo corpus was small and clean.

Weeks 2–5 — fixed retrieval, not the model. Chunking strategy, hybrid keyword and vector search, and a reranking step, tuned against a labelled query set. Retrieval precision improved 3.1x. Most of the perceived "model quality" problem disappeared with it.

Weeks 5–7 — cost and resilience. Simple classification steps routed to a smaller model, stable prompt prefixes cached, context trimmed of material that was never load-bearing. Provider abstraction with automatic fallback. Per-tenant cost attribution so their pricing team had real numbers.

Weeks 7–9 — hardening and rollout. Prompt injection defences, PII handling, request-level audit logging, per-feature rate limits, and a staged rollout from 5% to 100% with the evaluation suite gating each step.

The cost work#

Net effect: 47% cheaper per request with quality held flat on the evaluation suite. The suite is what made the trade-off measurable rather than a gamble.