The Hidden Lever in LLM Costs: Prompt Caching and Why Prompt Order Matters

Why does it feel like the big LLM providers are quietly hiding this one?

Prompt caching can completely change the cost profile of an AI product. Yet it's still treated like a small pricing detail — a single greyed-out line under "Input" that most teams scroll right past.

Most teams already know LLMs are expensive. Far fewer understand that the way you order your prompt can be the difference between paying full price and getting a large part of your input cached at a fraction of the cost.

That's the crazy part: two prompts that look almost identical to a developer can behave completely differently in production.

How prompt caching actually works

Under the hood, prompt caching is a prefix match. The provider hashes your prompt from the very beginning and reuses the computation for as long as the leading bytes are identical to a request it has seen recently. The moment something changes, the cache stops matching from that point onward — everything after the first difference has to be processed at full price again.

One rule explains all of it: caching matches a prefix. Any change anywhere in the prefix invalidates everything after it.

This is why prompt order matters so much. If the dynamic part of your prompt (the user's message, a timestamp, a per-request ID, a variable) appears too early, it breaks the stable prefix and shrinks the cached portion to almost nothing. But if the static parts come first, the prompt becomes dramatically more cache-friendly.

The one ordering rule

Put everything that doesn't change at the top, and everything that changes per request at the bottom:

The model, the task, and the logic are all identical, but the cost is not.

The savings aren't small — and every major provider offers them

This isn't a rounding error. Across providers, cached input typically costs around one-tenth of fresh input — a ~90% discount on the part of your prompt that repeats. Look at how each provider prices it (we've highlighted the line everyone scrolls past):

Anthropic lands in the same place: cache reads on Claude cost about 0.1× the normal input price. The first request that writes the cache costs a little more (about 1.25× for the default 5-minute window, 2× if you opt into a 1-hour window), so caching pays for itself after just a couple of reads — and on a busy endpoint, it pays for itself almost immediately.

So why does it feel hidden?

None of this is secret, exactly. It's all in the docs. But it's buried: one line on a pricing page, a few paragraphs deep in a technical guide, for a decision that has more effect on cost than almost anything else in how you call these APIs. For a product that sends the same long system prompt, tool definitions, and few-shot examples on every single call, the difference between a cache-friendly and a cache-hostile prompt layout can move the monthly bill by an order of magnitude.

It deserves to be front and center, not a footnote.

The takeaway

If you're building on top of LLMs, treat prompt structure as a cost decision, not just a quality one:

• Front-load everything static — system instructions, rules, examples, tool/function definitions, and schemas.
• Put the volatile stuff last — the user's message, retrieved context that changes per request, timestamps, and IDs.
• Keep the prefix byte-for-byte stable. A dynamic timestamp or an unsorted JSON blob near the top silently destroys your cache hit rate.
• Watch the cached-token counters in each provider's usage response. If cached reads stay at zero across repeated calls, something early in your prompt is changing.

Two prompts can be functionally identical and cost wildly different amounts. The providers could be a lot clearer about that — but until they are, the order of your prompt is one of the easiest wins you have.

How we optimize LLM usage — and how you can too

Prompt ordering is one lever. Picking the right model for each request is another — and that one we open-sourced. TryAii-RE is our free, open-source AI model routing engine: instead of hard-wiring every call to a single expensive flagship, it routes each request to the model that actually fits the task, balancing cost, latency, and quality. Pair that routing with a cache-friendly prompt layout and you're optimizing on both axes at once — paying less per call and caching more of every call.

It installs in one line for Python or Node, runs anywhere, and is free to use. Grab it below: