Test agents by the end state

How to do it

Write at least 20 scenarios per agent, each with a starting state, a user request, fake tools and the expected end state. Assert on what changed, such as one return created, and on what must not happen, such as no refund without approval. Agents reach the same goal by different paths, so step-by-step asserts fail for no reason. Run each scenario at least three times, as in run each case several times.

Example

test('return request creates one return and no refund', async () => {
  const tools = fakeTools({ orders: [{ id: 'A-1042', customerId: 'u1', status: 'delivered' }] });
  const status = await runAgent(newRun({ userId: 'u1', request: 'I want to return order A-1042', tools }));
  expect(status).toBe('done');
  expect(tools.returns.created).toHaveLength(1);
  expect(tools.calls.map(call => call.name)).not.toContain('issue_refund');
});

def test_return_request_creates_one_return_and_no_refund():
    tools = fake_tools(orders=[{"id": "A-1042", "customer_id": "u1", "status": "delivered"}])
    status = run_agent(new_run(user_id="u1", request="I want to return order A-1042", tools=tools))
    assert status == "done"
    assert len(tools.returns.created) == 1
    assert "issue_refund" not in [call.name for call in tools.calls]

How to check

CI runs the scenario set on every prompt, tool or model change. It fails when a scenario misses its end state or calls a forbidden tool in any run.