Field notes from shipping production AI — what works, what breaks, and what we'd do differently. Practical writing from the senior engineers doing the work, on getting AI out of the pilot stage and into systems you can trust.
Evaluate a RAG system in two halves: did retrieval fetch the right context, and did the model answer faithfully from it? Measure retrieval with context precision and recall, generation with faithfulness and answer relevancy — against a fixed set of test cases.
You cannot fully eliminate hallucinations, but you can drive them down with layers: ground the model in retrieved facts, constrain it with low temperature and structured output, validate with guardrails and an LLM judge, and measure the rate with evals.
How a small, senior team using AI agents ships what used to take a team three to four times its size — and keeps it running.
Most production LLM bills can be cut 60–80% without hurting quality, because most requests are easy and do not need your most expensive model. The big levers: route to smaller models, cache repeated prompts, right-size, and trim context.
