Put the stable part of the prompt first

How to do it

Providers can reuse the work on a prompt prefix they have seen recently, which lowers cost and latency on long prompts. A cache only matches an identical prefix, so a date or user name near the top breaks it. Sort keys when you serialise JSON into a prompt, and keep the tool list in a fixed order. OpenAI caches prompts of 1,024 tokens or more by default. Anthropic needs cache_control, either on the request or on the last block of the shared part. Put the marker at the end of the shared part when each request ends in unique text, such as retrieved chunks. Anthropic bills a cache write at 1.25 times the input price, so a prefix used only once costs more. Read the cached token count from the usage fields and chart it per feature. That is input_tokens_details.cached_tokens at OpenAI and cache_read_input_tokens at Anthropic.

Example

const response = await anthropic.messages.create({
  model: config.model,
  max_tokens: 4096,
  system: [{ type: 'text', text: STATIC_INSTRUCTIONS + '\n\n' + productCatalogue, cache_control: { type: 'ephemeral' } }],
  messages: [{ role: 'user', content: `Today is ${today}. Customer question:\n${question}` }],
});
const { input_tokens, cache_read_input_tokens, cache_creation_input_tokens } = response.usage;
const totalInput = input_tokens + (cache_read_input_tokens ?? 0) + (cache_creation_input_tokens ?? 0);
metrics.gauge('ai.cached_input_share', (cache_read_input_tokens ?? 0) / totalInput, { feature });

response = openai_client.responses.create(
    model=settings.model,
    instructions=STATIC_INSTRUCTIONS + "\n\n" + product_catalogue,
    input=f"Today is {today}. Customer question:\n{question}",
)
usage = response.usage
metrics.gauge("ai.cached_input_share", usage.input_tokens_details.cached_tokens / usage.input_tokens, tags={"feature": feature})

How to check

The share of input tokens served from cache is on the dashboard per feature. For a feature with a system prompt over 2,000 tokens, it is above half during busy hours. A test renders the prompt twice for two users and finds the same bytes up to the first user-specific text.