A different kind of dependency
Coming from a background of biometric and OCR pipelines on the e-KYC work at CPA — where a computer-vision model's output is at least a well-defined vector or bounding box — wiring an LLM into a product forces a mental adjustment. A database query is deterministic: same input, same rows, every time. An LLM call is not. The same prompt can come back phrased two different ways, occasionally with a different answer entirely, and your system has to be built assuming that from day one instead of discovering it in production.
That single property — non-determinism — is the root of almost every LLM engineering problem worth talking about. It's not that the models are unreliable in some vague sense; it's that a huge amount of standard backend practice quietly assumes determinism, and every one of those assumptions needs to be re-examined.
Prompts are an API contract you can't type-check
A REST endpoint has a schema. A prompt has… whatever text you wrote, and a hope. I've settled into treating prompts exactly like code: versioned in the repo, reviewed in pull requests, and never edited directly in a dashboard where the change isn't tracked. The moment a prompt lives outside version control, you lose the ability to answer 'what changed between the version that worked and the version that doesn't' — which is the first question you'll ask the day output quality regresses.
Structured output — asking the model to return JSON against an explicit schema, and validating that response the same way you'd validate any external input — closes most of the gap between 'an LLM' and 'a typed API.' It doesn't make the model deterministic, but it makes the failure mode a parseable, catchable error instead of a silently malformed string flowing downstream.
Where retrieval earns its complexity (and where it doesn't)
Retrieval-augmented generation gets reached for by default in a lot of teams, and it's genuinely the right tool when the answer depends on information that changes after the model's training cutoff, or that's private to your system — a user's own documents, a company's internal knowledge base. It is not the right tool for grounding general reasoning, and it adds a whole new failure surface: chunking strategy, embedding drift, retrieval ranking, and a second place non-determinism can sneak in before the model even sees a prompt.
The question I ask before reaching for RAG is simpler than the architecture: does the fact I need actually exist somewhere retrievable, or am I hoping retrieval will paper over a reasoning gap the model doesn't have? Those are different problems, and only one of them is solved by adding a vector database.
Latency, cost, and the fallback path
Every LLM call is a network round-trip to a service you don't control, with latency and cost that scale with output length — which is a very different budget from a database call. Streaming responses back to the client hides a lot of that latency perceptually, the same trick that made the e-KYC capture flow feel instant despite real model inference happening underneath. What streaming doesn't hide is cost, so caching identical or near-identical requests, and setting hard ceilings on output length for anything running in a loop, both stop being optional past a certain scale.
And because the dependency is genuinely less reliable than a database — rate limits, provider outages, occasional malformed output — every LLM-backed feature needs an explicit degraded path: a cached answer, a simpler rule-based fallback, or an honest 'try again' state, decided at design time rather than discovered the first time the provider has a bad day.
Evaluation is the part nobody enjoys
The unglamorous truth: getting a prompt to work once is easy, and knowing whether your last change made things better or worse across hundreds of real inputs is the actual engineering problem. That needs the same discipline as any other regression suite — a fixed set of representative inputs with known-good expectations, run automatically before a prompt or model change ships, not eyeballed on three examples in a notebook. It's less interesting than writing the prompt, and it's the difference between a demo and something you can keep shipping changes to without quietly degrading it.