Session 14: LLM Prompt Cache — From Principles to Practice

Date: August 21, 2026
Duration: 45 minutes
Presenter: Han Zhao
Host: Aimee Li
Participants: 65+ attendees visible during the live session

Session Recording

🎥 Watch Full Session

---

📋 Overview

Session 14 marked the first anniversary of Crossroad AI. Aimee Li thanked the volunteer presenters and the audience who have kept the series going since August 2025, and invited the next volunteer to share any AI practice, tool, or even failed experiment.

Han Zhao then presented LLM Prompt Cache. Every model call re-sends the whole conversation, so a provider that can skip recomputing an identical prefix saves both money and latency. Han covered three parts:

1. Why LLMs have prompt caches: what is cached, when it is hit, and how it is billed.
2. Using the cache in coding agents: how model switches, new sessions, side chats, and forks affect cache reuse.
3. Using the cache in our own products: how to structure prompts, test cache behaviour, and benchmark the real cost.

The one-line summary: optimise for fewer expensive tokens, not fewer tokens.

---

🎯 Key Topics Covered

1. Why LLMs Have a Prompt Cache

A chat with a model is not "one question, one answer." Each round re-sends everything that came before and appends the new content at the end. By the fourth call, the first 90% of the prompt has already been processed in earlier calls; only the newest block is genuinely new.

A cache hit does not shorten what you send. The full prompt is still sent on every request. The provider skips recomputation on the part of the prefix that matches exactly and processes only the new tail.

What this means in practice:

• The prefix must match exactly, token for token.
• The earlier a change appears in the prompt, the less of the prefix can be reused.
• The benefits are lower input cost, lower latency, and less repeated prefill work.

---

2. KV Cache vs. Prompt Cache

Han separated two terms that are often mixed up:

	KV cache	Prompt cache (also called context cache or prefix cache)
Level	Model-internal	Provider-level product feature
What it holds	The attention key/value tensors for tokens already processed, kept in GPU memory	An exact-prefix match on your request that lets the provider skip recomputation
Who can use it	The model engine; we cannot control it	Developers and users, through how prompts are built
Effect	Avoids recomputing earlier tokens while generating	Skips recomputation across requests and bills reused tokens at a lower rate

KV cache is the engine. Prompt cache is the product built on it.

Quadratic vs. linear

In a toy agent loop with initial context B where each step adds Δ, recomputing the whole prompt every step costs roughly O(n²). With a prefix cache, only the new part is recomputed, which approaches O(n).

The tokens do not disappear from your bill. What is avoided is the recomputation, and the reused tokens are priced differently.

---

3. How Caching Is Billed

Han used the GitHub Copilot model pricing table to explain the four price columns: input, output, cache read, and cache write.

• Cache read and cache write are both input-side concepts. Output is never cached.
• A cache write costs more than normal input, roughly 1.25×. One Claude Opus example in the table showed 625 versus 500 credits per million tokens.
• A cache read is far cheaper, at about 10% of normal input price (90% off).
• In a given round, each token is charged in only one category. It is never billed twice.

Worked example: three turns, one growing cache

Round	What happens	Billing
1. Cold start	Send 10,000 tokens. The model replies with a 200-token answer A.	10,000 tokens → cache write. Cache pool: 10,000.
2. Below threshold	Add a 100-token follow-up. New text is 200 + 100 = 300 tokens, under the 1,024 example threshold.	10,000 → cache read (90% off). 300 → uncached input at full price. Cache pool unchanged.
3. Threshold crossed	Add 2,000 tokens of new material. Leftover 300 + answer B (100) + 2,000 = 2,400 tokens, above the threshold.	10,000 → cache read. 2,400 → cache write. Cache pool: 12,400.

Rule of thumb: reads never stop covering a matching prefix. Writes fire only when the unwritten tail is long enough to be worth storing.

The exact trigger is a black box

Whether a write is triggered depends on the provider, the model, and even the thinking effort. The 1,024-token threshold is an illustration, not a guarantee. The concepts matter more than the precise numbers.

---

4. Coding Agents Amplify the Effect

In a web chat we often ask one question and leave. A coding agent behaves differently. A single user message triggers many model calls, because the agent plans, reads files, searches, edits code, and runs tests. Every tool result is appended to the prompt, so a single message can grow the prompt by thousands of tokens. Without prefix reuse, the quadratic cost would be unaffordable for the provider and slow for the user. That is why caching matters so much for coding agents.

---

5. Choose Model and Effort Early

Recent versions of GitHub Copilot in VS Code warn that changing the model or the thinking effort mid-session may reset the prompt cache and increase cost. Caches are not shared across models, and they are not shared across thinking-effort levels.

Principles:

1. Choose the model and effort early.
2. Keep them stable for one coherent task.
3. Treat the product's UI as provider-specific evidence of how the cache behaves.

The cheap-model trap: you have built up about 50K tokens of warm context on a hard problem, then ask a quick "what does this regex do?" on a lighter model. It feels thriftier, but the switch resets the cache and forces an expensive rebuild of the whole prefix. Switch only when quality, latency, or policy genuinely requires it, such as when the current model is not smart enough for the problem.

---

6. Small Steps Are Not Cold Starts

Han responded to a colleague's advice to "start a new session with each step and do not carry context across." He agreed with breaking work into small steps, but not with automatically opening a new session for each one.

If step 2 depends on step 1, a fresh session must rediscover the repository structure, prior decisions, tests and edge cases, and tool output that were already gathered. This costs you again and loses the warm cache.

• Modern agents compact the context automatically when it fills up, so writing a manual hand-over document is usually unnecessary.
• A new session makes sense when the work is genuinely unrelated, or when you must move to another agent.

Before opening a new session, ask whether the repository structure, prior decisions, tests and edge cases, and tool output are still useful. If they are, keep the thread.

---

7. Choose the Right Chat Boundary

Instead of choosing only between "continue" and "new session," Han described four boundaries, from most context reuse to cleanest isolation:

Boundary	When to use it
Continue	Same task and same direction. The next step builds on earlier work.
Side chat	A quick question using the current context, without derailing the main thread. In Claude Code you can highlight text and start a side chat, or use /btw ("by the way") for an aside that should not pollute the main context.
Fork	Same starting context, but a different follow-up task or alternate path. Several independent follow-ups can fork from one warm context.
New session	A genuinely different problem, or the old context is no longer useful.

The rule: same task, same thread, same model, same effort. Each boundary trades a little cache reuse for the right amount of isolation. Pick the cheapest one that still keeps the task honest.

---

8. Stable First, Dynamic Last

For products that call an LLM API, the guiding rule is "Stable first. Dynamic last. The prefix is the asset."

Reusable prefix (put first)	Request-specific suffix (put last)
1. System / developer instructions	5. Dynamic context
2. Tool definitions and schemas	6. User input
3. Stable reference context
4. Few-shot examples

If a prompt interleaves stable and variable segments, reorder it so all the stable content comes before any variable content. That saves money and makes the response faster.

System prompts: useful instructions, not padding

• The system prompt always sits at the head of the request, so reusable policies in it are cached naturally.
• A longer prefix can actually help, because a short prefix may never cross the cache-write threshold. Do not worry about a long system prompt.
• Do not pad it either. Keep useful instructions, then measure cached_tokens and cache_write_tokens in the API response.

Few-shot beats verbose instructions

Instead of a pile of "when X, except when Y, never Z" patch rules, write a simple instruction and add example user and assistant messages as if they were earlier conversation turns. The model treats them as demonstrations. This is cleaner, often uses fewer tokens, and, because the examples sit at the head of the prompt, it is cacheable as long as the examples stay stable.

In C#, this means keeping the example inputs and expected outputs in a JSON file, loading them, and converting them into a list of chat messages. The same fixed set in the same order on every call keeps this block a cache hit.

---

9. Test Cache Behaviour in Your Own Environment

Shared routing can make or break reuse

The company is moving toward a shared LLM gateway, where teams call a common API pool instead of running their own deployments. In such a setup, the same prefix may land on:

• different workers,
• different regions,
• different model versions,
• or different cache namespaces,

so it may miss the cache. The same concern applies to third-party aggregator APIs. Provider support does not equal your observed cache hit. A cache hit seen on Azure Microsoft Foundry does not guarantee one on a different provider or route.

Verify with the usage data in each response, such as cached_tokens, cache_write_tokens, cache key or namespace, model and region, and the resulting latency and effective cost.

Benchmark cache behaviour when you change models

Each model has its own caching strategy. Han swapped GPT-5.4 nano for GPT-5.6 Luna in one of his own projects and saw the cache hit rate drop. A model with a 30% lower token price can still produce a higher bill under a different cache strategy. When migrating, benchmark more than latency, speed, and correctness. Benchmark the cache behaviour as well:

• Quality: task success and correctness on representative workloads.
• Effective cost: uncached input + cache writes + cache reads + output, not the headline price alone.
• Cache hit rate: the same prompt structure, route, and effort, measured before and after migration.

---

10. Don't Optimise for Fewer Tokens

Cost isn't token count. It's uncached token count. The takeaways were:

1. Keep the right thread. Use side chat and fork, and choose the model and effort up front.
2. Choose stable prefixes and useful context.
3. Test cache reads, cache writes, latency, and effective cost every time.

---

💬 Q&A Highlights

Q (Ben Hao): Claude Code reportedly cut its system prompt by about 80%. Does that make the "stable system prompt" advice obsolete on the latest models?

A (Han): That change is in the Claude Code harness, not in the Claude model itself. Even a cache read is not free, since it is billed at about 10%. A system prompt also occupies the context window, so a shorter one leaves more room for the user. The bigger reason is that modern models have internalised much of what used to need explicit harness instructions. Ben added that harnesses are likely an intermediate stage whose features will gradually move into the models. Han agreed, but noted that for domain-specific tasks, such as those in the logistics industry, explicit instructions are still needed.

Q (Quinn Jiao): How long does a cache last? Claude Code runs often take a long time, and an AI assistant said the TTL is about 5 minutes. Does a cache still help in unattended long runs?

A (Han): The lifetime is a black box and may even vary with server load. It was around 5 minutes a few months ago, but his own tests show it is now longer. He suggested checking the per-request usage in GitHub Copilot after an hour to see the effect. Quinn observed that for long Claude Code runs started in the morning and reviewed in the afternoon, the cache may not help much, while short, fast question-and-answer use in VS Code Copilot benefits more. Han agreed and thanked Quinn for the addition: a cache does have a lifetime, and after days the whole prompt has to be recomputed.

---

📊 Presentations & Materials

Session Recording

🎥 Watch the August 21, 2026 recording

The slide deck was not supplied separately. The slide images on this page were captured from the recording.

Recording highlights:

• First-anniversary thanks from the host
• Why LLMs have prompt caches, and KV cache vs. prompt cache
• Cache read, cache write, and the worked three-turn billing example
• Keeping caches warm in coding agents: model and effort, small steps, side chat, fork
• Prompt structure for our own products: stable first, dynamic last, few-shot examples
• Cache behaviour behind shared gateways and across model migrations
• Live Q&A on system prompts and cache lifetime
• A light-hearted AI video and a preview of the September team activity

---

🔗 Featured Resources

1. OpenAI Prompt Caching - Provider documentation on cache behaviour and the cached_tokens usage field
2. Anthropic Prompt Caching - Provider documentation on cache writes, reads, and lifetime

---

🎮 Quiz Activity

No quiz activity was included in this session's recording. After the talk, the host led a Q&A segment, shared a short humorous video about AI as a worker, and announced an upcoming team activity.

---

🏆 Quiz Results & Winners

Status	Details
No quiz held	The recording did not include a quiz or winner announcement.

---

🔑 Key Insights

1. Caching saves recomputation, not tokens sent: the full prompt is still sent; the provider skips the matching prefix.
2. Exact prefix match decides everything: the earlier a change appears, the less can be reused.
3. Reads are cheap, writes cost a little extra: about 10% of input price for reads and about 125% for writes, so a warm cache pays back quickly.
4. Don't change model or thinking effort mid-task: caches are not shared across them.
5. Small steps do not require cold starts: use continue, side chat, or fork before opening a new session.
6. Stable first, dynamic last: put system prompts, tool schemas, reference context, and few-shot examples before the variable content.
7. Provider support is not an observed hit: shared gateways and aggregators can route requests so the cache misses.
8. Benchmark cache behaviour with every model change: a cheaper token price can still mean a costlier bill.

---

📚 Further Learning

For Developers

• Log cached_tokens and cache_write_tokens for every LLM call and track the cache hit rate over time.
• Move stable instructions, tool definitions, reference material, and few-shot examples to the start of every prompt.
• Replace long rule lists with a short instruction plus a few example turns where it fits.

For Teams

• Include cache hit rate and effective cost, not just headline token price, when comparing models or gateways.
• Verify cache behaviour behind any shared API pool or aggregator before depending on it.
• Agree on a team habit of choosing the model and effort at the start of a task.

---

📣 Announcements

• First anniversary: Crossroad AI reached one year, and the host invited volunteers for future sessions on AI practices, tools, and lessons learned.
• September team activity: the Nanjing office will host a cross-team activity of about 1.5 hours on a September afternoon, with September 10 and 17 as the tentative dates. No prior experience is needed, and prizes and small gifts are planned. Invitations were to be sent out over the following two weeks.

---

🙏 Acknowledgments

Special thanks to:

• Han Zhao for presenting LLM Prompt Cache
• Aimee Li for hosting and facilitating the session
• Ben Hao and Quinn Jiao for their questions and additions during the Q&A
• All participants for joining and keeping the series going for a full year

---

Stable first, dynamic last

Put everything that stays the same at the start of the prompt, and everything that changes at the end. The prefix is the asset.

Measure, don't assume

Cache behaviour depends on the provider, model, thinking effort, route, and time. Check cached_tokens and cache_write_tokens in your own environment rather than trusting the headline price.

---

Session 14 | August 21, 2026 | LLM Prompt Cache: From Principles to Practice | Han Zhao