What an LLM Can and Can't Do for Your Business

LLM features look extraordinary in demos because demos use carefully chosen prompts and curated inputs. You see the system summarize a complex document perfectly, extract key data points without error, draft a response that sounds exactly right. Then you ship it to real users with real inputs and the failure rate is nothing like the demo suggested. The model confabulates facts. It misses edge cases that seem obvious. It's inconsistent across runs on identical inputs. This isn't a bug you can fix — it's the fundamental architecture of the system.

LLMs are next-token predictors. They generate the most statistically likely continuation of a sequence of text, given their training. They are not reasoning engines, they do not have knowledge in the way a database has knowledge, and they don't maintain a consistent internal model of the world across sessions. This architecture makes them extraordinarily good at certain tasks — tasks where “generate plausible-sounding text” is exactly what you need. It makes them unreliable or dangerous for tasks where correctness and consistency are load-bearing.

The roadmap risk is real: founders are committing engineering resources to LLM-based features based on demo performance, not production performance. The features ship late, require ongoing prompt engineering to maintain, fail unpredictably in ways that are hard to debug, and sometimes have unit economics that don't survive contact with real call volume. Understanding where LLMs actually perform reliably — and where they don't — is the prerequisite for making good build decisions.

The most important mental model is the difference between a chatbot, an LLM-assisted workflow, and an LLM-as-judge system. A chatbot exposes an LLM directly to users and lets them have a conversation. An LLM-assisted workflow uses an LLM as one step in a larger process — for example, extracting structured data from unstructured text before it enters a database. An LLM-as-judge system uses an LLM to evaluate or score outputs from other systems. These have dramatically different failure profiles and acceptable failure rates. Getting the category right before you architect the feature is important.

Cost structure at scale is the other thing you need to model upfront. LLM inference costs are proportional to token volume, which is a function of prompt length times call volume. A feature that looks cheap in development can become expensive at scale in non-obvious ways — especially if the prompt includes long context (document text, conversation history, retrieval results). Before committing to an architecture, sketch the cost curve at 10x and 100x your current volume and ask whether the unit economics hold. Many features that work fine at small scale require a re-architecture at medium scale.

Before committing to any LLM-based feature, define the acceptable failure rate. Not in the abstract — specifically. What percentage of outputs can be wrong before users stop trusting the feature? What happens when it's wrong? If a wrong output means a user gets a mildly inconvenient suggestion, that's different from a wrong output meaning a user gets incorrect medical or financial information. The acceptable failure rate determines whether you need a human in the loop, what the review interface looks like, and whether the LLM is even the right tool for the task.

Run the feature on 100 real examples from your actual data before you build the full implementation. Not synthetic examples, not curated test cases — real production-like inputs that reflect the variety and messiness of what you'll actually process. Measure the failure rate manually on those 100 examples. If it's higher than your acceptable threshold at that sample size, the failure rate in production will be at least as high. This test takes a day. Building the full feature and then discovering the failure rate takes months.

Finally, model the cost at 10x and 100x your current volume before finalizing the architecture. Identify the levers available to reduce cost at scale — shorter prompts, smaller models, caching common responses, batching calls, moving to a self-hosted model if volumes justify the infrastructure. Having the cost conversation before the feature is built is far easier than having it after you have user dependencies on the architecture.

LLMs do well at tasks where generating plausible, fluent text is the goal: summarization, drafting, classification, extraction from unstructured text, and code generation for boilerplate. These tasks have in common that “pretty good most of the time” is genuinely useful, and errors are recoverable. A summarization that omits a detail can be improved. A drafted email that needs editing is still faster than starting from scratch. These are the use cases where LLMs create real value and where the failure mode is manageable.

LLMs perform unreliably on multi-step reasoning, exact numerical computation, anything that requires real-time information, and anything that requires consistent behavior across identical runs. If your feature depends on the model applying a rule correctly every time, or performing a calculation reliably, or knowing something that happened after its training cutoff — you're in the unreliable zone. These tasks require either a different architecture (retrieval for real-time information, a calculator for math, structured logic for rule application) or a human verification step.

There are categories where LLMs should not be trusted as the final output without a human in the loop: legal or medical advice, financial decisions, and anything where errors compound. A wrong legal interpretation compounded across hundreds of users isn't just a feature bug — it's a liability. A wrong financial calculation that users rely on without knowing the model can be wrong is a product design failure. These aren't arguments against using LLMs in high-stakes domains; they're arguments for building human review into the workflow before any output reaches a user who will act on it.