How to cut LLM costs in production

Most production LLM bills can be cut by 60–80% without hurting quality. The reason is simple: most requests are easy, and easy requests do not need your most expensive model. The big levers are routing simple work to cheaper models, caching repeated and near-duplicate prompts, right-sizing the model to each task, and trimming context — then measuring cost, latency, and quality so the savings are deliberate, not accidental.

A demo costs nothing. The bill arrives in production, where the same call runs thousands or millions of times, and a team that reached for the strongest model everywhere suddenly has a cost — and often a latency — problem. Here is how to bring it down, lever by lever.

Why your bill is mostly waste

The default mistake is using one frontier model for everything. But in a typical production workload, 60–70% of requests are simple — classification, extraction, short Q&A, formatting — and a small, cheap model handles them with no perceptible quality loss. Paying frontier prices for that majority is the single biggest source of waste. Almost every optimization below is a way to stop doing that.

Lever 1 — route requests to the right-sized model

Model routing sends each request to the cheapest model that can handle it, and escalates only the hard ones. A small model takes classification, retrieval, and simple Q&A; a frontier model is reserved for genuine reasoning. This is the highest-impact lever — routing alone commonly delivers a 60–75% cost reduction, because the cheapest models can be roughly 20× cheaper than frontier ones, and most of your traffic never needs to touch the expensive path.

The trick is a fast, cheap classifier (or simple heuristics) deciding the route, so the routing itself adds negligible cost and latency.

Lever 2 — cache aggressively

Repeated work should not be paid for twice:

• Prompt caching lets you reuse the processed input tokens of a repeated prompt prefix — a long system prompt, a fixed instruction block, shared context. Every major provider now supports it, and it can cut input-token costs by up to ~90% and latency by up to ~85% on the cached portion.
• Semantic caching goes further: instead of matching identical requests, it recognizes semantically similar ones and serves a cached answer. On typical chatbot workloads it lands 25–35% cache-hit rates — a quarter to a third of requests answered with no model call at all.

Lever 3 — right-size the model to the task

Match the model to the job rather than defaulting to the biggest. A classification step does not need a reasoning model. A one-line extraction does not need a long-context model. A short rewrite does not need your flagship. Picking the smallest model that clears the quality bar for that specific task compounds with routing — many sub-tasks inside one feature can each run on a different, cheaper model.

Lever 4 — trim the prompt

Every token in the prompt is billed on every call, so tighten it: shorten verbose system prompts, include only the few-shot examples you actually need, and retrieve only the most relevant context instead of stuffing everything in. Leaner prompts cut cost and latency at once — and often improve quality, because the model is not distracted by irrelevant context.

The tradeoff you are actually managing

Underneath all of this is a three-way tension: cost, latency, and quality. You cannot maximize all three — the biggest model with the longest context gives the best answers, slowly and expensively. So set the priority per interaction. A real-time autocomplete lives or dies on latency, so it should use a small fast model. An overnight batch summarization has no latency constraint, so it can optimize for quality and cost. Naming the priority for each feature turns a hidden default into a design decision.

A worked example of where the money goes

Picture a support assistant handling 100,000 requests a month, all on a frontier model. Look at the traffic and a pattern appears: ~65,000 are simple (greetings, order-status, FAQ-style), ~25,000 are moderate, and ~10,000 genuinely need deep reasoning.

• Route the 65,000 simple ones to a model ~20× cheaper — quality is indistinguishable on those, and the bulk of your volume just got an order of magnitude cheaper.
• Cache the repeats: a big share of those simple questions are near-duplicates ("where's my order?"), so a 25–35% semantic-cache hit rate removes tens of thousands of model calls entirely.
• Reserve the frontier model for the ~10,000 hard cases, where it actually earns its price.

You did not lower quality anywhere a user would feel it — you stopped overpaying for the easy 90%. That is how the headline 60–80% savings is realized in practice.

The order to do this in

If you are starting from "one big model for everything," tackle the levers in impact order:

1. Caching first — it is the least invasive change and an immediate win; turn on prompt caching, then add semantic caching.
2. Routing second — the biggest structural saving, but it needs a classifier and testing to make sure quality holds on the cheap path.
3. Right-sizing and prompt trimming — ongoing hygiene as you build, tuning each sub-task to the smallest model and prompt that clears the bar.

Put it together — and measure

The levers stack. Combine routing, caching, and right-sizing and most production apps land in that 60–80% reduction range — with the savings spent back on the genuinely hard cases where the best model is worth it.

But none of this is safe blind. Instrument all three axes — cost and latency per request in production, quality via an eval suite — so you catch the prompt change that doubled spend, or the model swap that quietly added two seconds, before a user does. The goal is not the cheapest possible answer; it is the right answer, fast enough, at a cost that survives scale.