Stream long answers

How to do it

Use the provider's streaming mode and pass the text on to the browser as it arrives, for example with server-sent events. Show a working state at once. Stream plain text answers only. Structured output that code acts on is validated as a whole, so wait for it or stream only a progress note. Run output checks on the finished text, and replace it if they fail. Record time to first token and total time per feature.

Example

app.get('/api/assistant', async (req, res) => {
  res.set({ 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-store' });
  const started = performance.now();
  let first = true;
  for await (const text of ai.stream({ promptId: 'help-chat', vars: { question: String(req.query.q) }, userId: req.user.id })) {
    if (first) { metrics.timing('ai.first_token_ms', performance.now() - started); first = false; }
    res.write(`data: ${JSON.stringify({ text })}\n\n`);
  }
  res.end();
});

@app.get("/api/assistant")
async def assistant(q: str, user: User = Depends(current_user)):
    async def events():
        started, first = time.perf_counter(), True
        async for text in ai.stream(prompt_id="help-chat", variables={"question": q}, user_id=user.id):
            if first:
                metrics.timing("ai.first_token_ms", (time.perf_counter() - started) * 1000)
                first = False
            yield f"data: {json.dumps({'text': text})}\n\n"
    return StreamingResponse(events(), media_type="text/event-stream", headers={"Cache-Control": "no-store"})

How to check

The latency dashboard shows time to first token per feature. Its 95th percentile is under the first-token target in the feature's spec, such as 1.5 seconds.