Prompt caching

Upstream prompt caching lets part of your input be served from a cache instead of being recomputed. When a cache hit happens, the cached portion of your prompt is billed at a discount rather than full price. This page explains how to structure your requests so the cache actually hits.

Caching is keyed on the prefix of your prompt. The upstream provider stores recently seen prompt prefixes and, when a later request starts with the same prefix, serves that portion from cache. The part that matches the cache is called a cache read; the part the provider has to compute from scratch is fresh input.

The single most important rule: put stable content at the start of your prompt and dynamic content at the end. Because caching matches on the prefix, any change at the beginning of the prompt invalidates the entire cache window.

• Keep your system prompt, shared instructions, and reference context stable and at the top.
• Put per-request data — timestamps, IDs, user queries, retrieved documents — at the bottom.
• Avoid injecting a timestamp or nonce near the start of the prompt.
• In multi-turn conversations, keep the conversation history ordered and append new turns rather than rewriting earlier ones.

On the Anthropic Messages endpoint, caching is opt-in: you mark breakpoints with a cache_control field. A breakpoint tells the provider where a cacheable prefix ends, so the prefix before it can be reused on the next request.

{
  "model": "<public-name>",
  "max_tokens": 1024,
  "system": [
    {
      "type": "text",
      "text": "You are a helpful assistant with a large, stable instruction block.",
      "cache_control": { "type": "ephemeral" }
    }
  ],
  "messages": [
    { "role": "user", "content": "Dynamic user question goes here." }
  ]
}

On OpenAI-compatible routes, caching is automatic — you do not send any opt-in header. The provider caches based on prefix stability alone, so the guidance above still applies: keep stable content first and dynamic content last.

Not every model or route supports prompt caching, and the details vary by provider. The cache time-to-live (TTL), the minimum token threshold, and which endpoints support caching are all provider-specific and can change without notice. Do not assume a given model caches — verify it for your workload.

To confirm caching is working, check the usage breakdown in your dashboard. If you need the exact TTL or minimum thresholds for a specific model, ask your reseller for the current values.