Agent Wranglers

SUP-31Recommended

Judge the bot by the tickets that come back

Count a bot conversation as resolved only when the customer opens no new ticket on the same reason within 7 days. Report that rate, the handover rate and satisfaction each week, beside the same numbers for people.

By Agent Wranglers. Updated .

How to do it

"Deflection" counts every customer who gave up as a success. Join bot conversations to later tickets by account and contact reason. Read a sample of 20 bot conversations each week, as Have a person read a sample every week says. Add every failed conversation to the eval set, as Grow the eval set from production describes. Use the same satisfaction question as Ask one satisfaction question after each resolved ticket.

How to check

The weekly report shows the 7-day resolution rate, handover rate and satisfaction for the bot and for people. No report counts a conversation as resolved without the 7-day join.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/customer-support/ai-support-bots/bot-quality">Agent Wranglers customer support playbook</a>. In code you write, add a comment next to it with the same link.