Caching & Cost Optimization

What Is Prompt Caching?

When you send a request containing a large amount of static context (such as a system prompt or knowledge base), the model caches that content. Subsequent requests that reuse the same context can read from the cache instead of reprocessing it from scratch.

Cache hit rate = cached tokens read / total input tokens x 100%

Benefits of cache hits:

• Lower latency: Skips processing of cached portions, resulting in faster time-to-first-token
• Lower cost: Cached tokens are billed at a reduced rate (typically 10%-25% of the standard price)

Which Models Support Prompt Caching?

Support for Prompt Caching varies across models. We recommend verifying through the console, actual response data, and the model vendor's documentation.

Current guidance:

How to Improve Cache Hit Rates

1. Keep Prefixes Fixed; Place Dynamic Content at the End

{
  "messages": [
    {"role": "system", "content": "[Your fixed system prompt, ~2000 words...]"},
    {"role": "user", "content": "[User's dynamic question]"}
  ]
}

The system prompt stays constant (cache hit); the user message varies each time (billed normally).

2. Set Cache Breakpoints Strategically

Place large blocks of static content in the first few messages, with dynamic content at the end. The model matches the cache from the beginning and stops at the first point of difference.

3. Manage Request Intervals

Caches have a TTL (typically 5-10 minutes). If too much time elapses between requests, the cache may expire. High-frequency workloads naturally achieve higher hit rates.

4. Standardize Templates; Avoid Minor Variations

These two requests will not share a cache:

• "You are a professional assistant."
• "You are a professional assistant " (trailing space)

Even near-identical content will cause a cache miss if there is any character-level difference. Use a standardized prompt template.

5. Reuse the Same Model and API Key

Caches are not shared between different models. Same model + same prefix = highest hit rate.

How to Verify a Cache Hit

OpenAI-compatible endpoint (/v1/chat/completions): read usage.prompt_tokens_details.cached_tokens — the subset of prompt_tokens served from cache. The field is always present (0 on a miss). usage.provider_usage.raw carries the model vendor's usage in its native vocabulary (including cache-write counts) for reconciliation.

Anthropic-native endpoint (/v1/messages): read usage.cache_creation_input_tokens (written to cache this request) and usage.cache_read_input_tokens (served from cache). Explicit cache_control supports both the 5-minute and 1-hour TTL tiers; streaming and non-streaming requests share the same cache.

Two things to note:

1. Claude caching is always explicit: both /v1/messages and /v1/chat/completions support it, but you must set cache_control breakpoints — on the OpenAI-compatible endpoint, write the system message as a block array: {"role":"system","content":[{"type":"text","text":"<big prefix>","cache_control":{"type":"ephemeral"}}]}. Requests without markers are never cached automatically.
2. The cache is bound to the serving route. In rare cases (route switch) a request lands on a backup route; that single request shows a cache miss plus a re-write. This is expected — the next request hits the cache again.

Additional Cost Optimization Tips

Choose the Right Model

Not every task requires the most powerful model:

Control max_tokens

Set a reasonable max_tokens value to prevent the model from generating unnecessarily long output. For example, if you only need a "yes/no" judgment, max_tokens: 10 is sufficient.

Keep System Prompts Concise

An overly long system prompt increases the input token cost for every request. Keep prompts lean and remove unnecessary descriptions.