Record every step of every run

How to do it

Give each run an ID and write one row per step. Mask personal data in arguments and results, as for prompt text. Show users a plain list of the actions taken for them, with undo. The method is in log agent actions and let users undo them. Tracing standards, such as the OpenTelemetry semantic conventions for generative AI, let common tools display the steps.

Example

await db.agentSteps.insert({ runId: run.id, step, tool: call.name, args: maskPersonalData(JSON.stringify(call.args)),
  resultSummary: maskPersonalData(summarise(result)).slice(0, 500), costUsd, at: new Date() });

db.agent_steps.insert(run_id=run.id, step=step, tool=call.name, args=mask_personal_data(json.dumps(call.args)),
                      result_summary=mask_personal_data(summarise(result))[:500], cost_usd=cost_usd, at=datetime.now(UTC))

How to check

Run a test task with three tool calls. The run's trace has three step rows with tool, arguments, result, time and cost. The user's activity page lists the same actions.