Assistant rating
After every conversation, the same AI step that writes the summary also rates how it went. There is nothing to set up: it runs on every call, SMS thread, WhatsApp conversation and webchat once it has ended.
What is rated
- Assistant rating: a score from 0 to 100%, for how well the assistant helped the customer and your business. 90% or more is helpful, correct and complete. Around 50% means the customer was partly helped. Below 30% the answer was wrong, unhelpful or rude. The AI adds one sentence on why the score is this high or low.
- Colour: green from 75% (Good), orange from 50 to 74% (Fair), red below 50% (Weak). Below 50% a conversation is worth reading.
- Customer mood: Positive, Neutral or Negative: how the customer came across over the whole conversation. This is about the customer, not the assistant.
- Points to check: flags for what needs a look. A conversation can have none, one or several.
| Flag | Meaning |
|---|---|
| Strayed from your business | The customer brought up something unrelated to your business, such as a sum, politics or a homework question, and the assistant went along with it instead of steering back to your business. A question about your own services or appointments never counts as straying, even when the assistant did not know the answer. Neither does what customers bring to a business like yours, such as a complaint at a GP's practice or a boiler error code at a plumber. How the assistant keeps to your business is under instructions. |
| Answer not from knowledge | The assistant stated facts, such as a price, an opening time or an appointment, that did not come from your knowledge, the calendar or another task, or from the conversation itself. |
| Task failed | A transfer, booking, callback or other task the customer needed did not work. |
| Customer frustrated | The customer was clearly annoyed, impatient or unhappy with the help. |
| Asked for a person | The customer asked to speak to someone from your team. |
| Possible manipulation | The customer tried to make the assistant break its rules, reveal its instructions or approve a discount or exception it may not. |
The AI reads the whole conversation back, together with what the assistant looked up and did during it (see What the assistant did in Inbox & analytics). An answer from your knowledge or the calendar therefore counts as grounded. The score is about the assistant, not about the customer and not about the phone line. How the line sounded is under Line quality (see Inbox & analytics). When a lot of audio was lost during a call, the score's explanation says so too: the assistant may then not have heard parts of the conversation properly.
The rating is done by AI and can be wrong. Use it to find the conversations worth reading, not as a final verdict. Conversations from Try it out do not count, except Real call.
Where you see it
- On a conversation: under Overview you see Assistant rating (for example 25% · Weak), Customer mood and Points to check. Rest your mouse on a label, or click or tap it, for the explanation: on the score you see why it is what it is and how it is worked out, on a flag what it means and a link to all conversations with that flag.
- In the Inbox: a Flagged badge on the row. Under Filters you filter on Sentiment and Flag. See Inbox & analytics.
- Overview → Today: one line, for example "2 conversations were flagged", when something was flagged in the last 24 hours. It opens those conversations.
- Overview → Figures: an Assistant rating section with the average score, the score per day, the sentiment split, the most frequent flags and the lowest scores. Every figure opens the matching conversations for the same period.
Custom checks
On Enterprise you add your own yes/no questions per assistant, for example "Did the assistant confirm the postcode?". They are answered after every conversation of that assistant, in the same step.
- Go to Assistants and open the assistant.
- Open Assistants → Conversation rules → More options.
- Under Custom checks, type the question and click Add.
The check is saved straight away. An assistant can have up to 10 checks of at most 200 characters each. Click × to remove one. Each check is answered with yes, no, or N/A when the question does not fit the conversation. You then see:
- on the conversation, how many checks were passed, with the answer per check when you unfold it;
- on Overview → Figures, how often each check was passed, per assistant, with a link to the conversations where it was not;
- in the Inbox, the Custom check filter.
Duplicating an assistant copies its checks; the copy keeps its own figures and its own filter for them. Without Enterprise, existing checks stay visible but are no longer rated; you can still remove them.
Email about flagged conversations
An admin can have an email sent for every flagged conversation. Open Settings → Team and, under Notifications and availability, turn on Email me about flagged conversations. The email goes to the Notification email address, with the flags, the score, the summary and a link to the conversation. It is off by default.
Export and API
- On Enterprise, editors and admins download the conversations in the Inbox, with the assistant rating, as a CSV file. See Export.
- The API returns the rating as
analysison every conversation and filters onsentimentandflag.