AI feature you can't trust?

Your AI feature is impressive when it works. You can't put it in front of a customer.

The strange thing about an unreliable AI feature is that the model call itself is usually fine. The tool wired it correctly, the prompt is reasonable, and on a good run the output is impressive. What is missing is everything around the call: a contract on what comes back, a plan for when the provider is slow or down, a limit on what one user can spend, and any record of what actually happened. A demo needs one good run. Production needs the ten thousandth, and that is a different engineering job.

What actually breaks

• The output is parsed with string matching, and the model does not always cooperateThe generated code asks for prose, then digs the useful part out with regular expressions, split calls, or JSON.parse on whatever came back. Models drift: markdown fences around the JSON, a friendly preamble, a renamed field, and the parse throws or returns nothing. Providers can enforce a schema at the API level so the reply matches the declared shape (OpenAI calls this structured outputs). You still handle refusals and replies cut short by the token cap, but you stop doing string surgery on prose.
• Provider errors are treated as impossibleOpenAI and Anthropic return 429 when you hit a rate limit and 5xx when they are overloaded, and both happen routinely, usually at the moment you care most: launch day, a busy afternoon, the demo. The correct response is a retry with exponential backoff, honouring the retry delay the provider sends back, and a fallback when retries run out. The generated code tends to make one call and render whatever happened, so a single transient provider error reads to your customer as your product being broken.
• No timeout, so one slow call hangs the whole featureModel calls are slow at the best of times and occasionally hang outright. Official SDKs ship with generous defaults, up to ten minutes in the OpenAI Python client, on the assumption that you will tune them; generated code rarely does. Meanwhile your hosting platform enforces its own much shorter limit on request duration, so the platform kills the request first. The user watches a spinner, then gets a generic error, and no code you own ever decided what should happen.
• User content goes into the same prompt as your instructionsA language model cannot reliably tell your instructions apart from text a user pasted in; it is all one context window. This is prompt injection, the top entry in the OWASP risk list for LLM applications. Anything the feature reads, a pasted document, a scraped page, a support ticket, can tell the model to ignore your rules, reveal your system prompt, or produce output you would never sign off, published under your product's name. The hostile text does not need to be visible: white on white and odd Unicode work.
• One user can spend your whole model budgetYou pay per token, input and output, and the model decides output length at run time. With no maximum tokens set, no per-user cap, and retries multiplying calls, your cost has no upper bound by construction. OpenAI and Anthropic both offer budget alerts and spend limits, but they sit off until someone configures them, and on some plans the limit warns rather than stops. Cost scales with usage, so your biggest traffic night is your biggest bill, and an unconfigured account finds out from the invoice.
• No evaluation set and no logging, so nobody knows what is trueWhen the model gives a wrong answer, nothing crashes. There is no exception, no stack trace, no entry in your error tracker: the feature confidently returns something wrong and the only witness is the user. Without logs of inputs and outputs you cannot reproduce a complaint, and without a fixed set of test cases you cannot tell whether the prompt change you just made fixed one case and quietly broke three others. Reliability you cannot measure is reliability you do not have.

How I make an AI feature reliable

The first day is measurement, not fixes. I put logging around every model call: input, output, latency, token cost, and whether the parse succeeded, so failures stop being invisible. From real traffic and your complaints I build a small evaluation set: real inputs with agreed correct behaviour, including every case that has embarrassed you. Then I harden the call path: schema-enforced output, validation before anything touches your data or your users, retries with backoff, a hard timeout with a sensible fallback, and caps on tokens per request and spend per user. Every change afterwards is judged by rerunning the evaluation set, not by trying it once and feeling good. You keep the logs and the set, so the next change can be judged the same way.

What “done” looks like

We agree one of these in writing before you pay anything. If it passes, the sprint passed. If it doesn't, you get a full refund and keep the work.

• ✓The feature produces an acceptable answer on an agreed set of real inputs, including the exact ones that failed before, and the run is repeatable on demand.
• ✓When the provider is slow, down, or rate limiting, the user sees a clear message or a fallback within an agreed number of seconds, never an endless spinner.
• ✓Pasting hostile instructions into the feature does not make it ignore its rules, reveal its prompt, or act outside its scope, demonstrated against a set of hostile inputs agreed up front.
• ✓Every model call is logged with its cost, a hard spend cap is enforced, and you can see yesterday's total without opening the provider's dashboard.

Other things that block launches

• Your Lovable app works in the preview. It breaks in production.
• Stripe took the money. Your customer got nothing.
• Your Bolt.new app runs fine in the preview. The published version doesn't.
• Your Replit app works in the workspace. The published version doesn't.
• Your v0 app looks ready to ship. The part that breaks is the part you can't see.
• Your Base44 app works when you use it. It breaks when your customers do.
• Your Cursor app runs on your machine. That is the only place it has ever been tested.
• Your customers can see each other's data. It is never just the one screen you found.
• Users can't sign in. It worked every time you tested it.
• It runs on your laptop. It will not deploy anywhere else.
• The full The Launch-Blocker Sprint