Cache whole answers for repeated inputs

How to do it

Only cache answers that do not depend on who asks or on data that changes. Include every input that shapes the answer in the key, so a new prompt or model never serves an old answer. Set a time limit on each entry, such as 7 days. Never cache answers built from one user's private data for another user.

Example

const key = createHash('sha256').update(JSON.stringify([prompt.id, prompt.version, config.model, input])).digest('hex');
const cached = await cache.get(key);
if (cached) return JSON.parse(cached);
const answer = await ai.generate({ promptId: prompt.id, vars: { input }, schema: Category, userId });
await cache.set(key, JSON.stringify(answer), { ttlSeconds: 7 * 24 * 3600 });

key = hashlib.sha256(json.dumps([prompt.id, prompt.version, settings.model, text]).encode()).hexdigest()
if (cached := cache.get(key)) is not None:
    return Category.model_validate_json(cached)
answer = ai.generate(prompt.id, {"input": text}, Category, user_id)
cache.set(key, answer.model_dump_json(), ex=7 * 24 * 3600)

How to check

A test calls the feature twice with the same input. The provider is called once. After the prompt version changes, the provider is called again.