Definition: Prompt caching is a technique where a model provider or serving system stores the processed form of repeated parts of a prompt, such as long instructions or documents, and reuses it on later requests.Why It Matters: Many applications send the same long instructions or documents with each request. Reprocessing them wastes time and money. Caching can lower latency and cost substantially for such workloads. Benefits depend on how much of the prompt repeats and on provider-specific rules.Key Characteristics: Caching typically applies to the stable beginning of a prompt, so structure matters: put static content first and variable content last. Caches have time limits, and exact matching of the repeated prefix is required. Pricing and rules differ by provider.
When a request arrives, the system checks whether the start of the prompt matches content processed recently. If so, it reuses the stored internal state and only processes the new part. Cached entries expire after a period, and any change in the prefix causes a miss. Developers order prompts with system instructions, tool definitions and reference documents first, followed by the user's question. Teams measure hit rates and cost savings, and adjust prompts to increase reuse.ProsCaching reduces cost and latency for repeated context.ConsSavings depend on hit rates, and small prompt changes can cause misses.
Document Q&A: A long manual is cached so each question only pays for the new part.Agent Loops: Tool definitions and instructions are cached across many agent steps.Coding Assistants: Repository context is cached across a session.
Response Caching (2000s-2010s): Web systems cached full responses.KV Cache in Serving (2020-2022): Inference engines reused attention state within a request.Provider Prompt Caching (2024): Model providers began offering cross-request caching.Current Practice and Trends (2024-Present): Prompt design increasingly targets cache efficiency.
When to Use: Use prompt caching when requests share long, stable context such as instructions, tools or documents.Designing for Reliability: Order prompts for reuse and monitor hit rates.Operating at Scale: Track savings and expirations, and standardize shared prefixes.Governance and Risk: Check how providers store cached data, including retention and region.
