Know which chatbot answers to check,
and which questions went unanswered
Your Heeya agent answers from your documents, pages and Q&As. The answer reliability score checks every reply against the passages it was written from, and flags it in Analytics. You read the risky ones first, and you turn each unanswered question into a Q&A without leaving the conversation.
Included from the Standard plan, free trial included. 1 message credit per analysed conversation.
Analytics, conversation of today
Illustrative conversation on a heating engineer's website
The annual gas boiler service is €120 including VAT, call-out included.
Probably reliable answer · 96%
I could not find this information. Would you like to leave your details so we can get back to you?
+ Add as Q&AInformation probably not found
What the score checks
A retrieval-based chatbot searches your content for the passages closest to the question, then writes a reply from them. It can still go wrong: it may add a detail the passage does not contain, miss an exception, or answer from general knowledge when your content says nothing. If you want the mechanics behind those failures, the guide to AI chatbot hallucinations covers them.
Right after each reply is sent, a second model reads the visitor's question with the last messages of the conversation, the passages that were retrieved, your instructions and the reply. It does not write anything: it returns a probability for each possible diagnosis, and Heeya turns them into one result per reply.
Four results, worded with caution
The score is an estimate, so every label says "probably". It shows under each reply in Analytics, and only there.
Probably reliable answer
Above 70%
The reply is based on your content. Nothing to do.
Answer to check
From 50 to 70%
The reply may be only partly based on your content. Read it over.
Risk of being incorrect
Below 50%
The reply is probably not based on your content. Check it, and correct it with a Q&A if needed.
Information probably not found
An unanswered question
The agent did not find the answer in your content. Add the information as a Q&A.
Greetings and thank-you messages get no result. At the top of each conversation, a summary gives the totals, such as "2 reliable answers, 1 unanswered question", and each total scrolls to the replies concerned.
From unanswered question to Q&A
"Information probably not found" is often the most useful result: it is the list of questions your visitors actually ask and your content does not answer. Under each reply, "Add as Q&A" opens a form with the visitor's question already filled in. You rephrase it if it depends on the rest of the conversation, write the answer, and the agent uses it a few seconds later.
The share of unanswered questions over time is also a running measure of intent coverage, one of the metrics in the chatbot KPIs guide.
What the score does not do
It is not a filter
- The reply has already been sent when the score arrives. Nothing is blocked or rewritten.
- Visitors never see the score. It only shows in your dashboard.
- If the check fails, the reply simply has no score, and the conversation is not counted.
It is not proof
- A reply flagged as risky can be right, and the reverse happens too. Read before you correct.
- It checks grounding, not truth: a reply faithful to a wrong document is scored as reliable.
- Only new conversations are scored, from the moment you turn the tool on.
Plans and pricing
The reliability score is included in the Standard plan at €19/month, the Premium plan at €99/month and the Enterprise plan. You turn it on per agent in the Tools page.
Each analysed conversation uses 1 message credit, once per conversation. Greetings and your own tests from the dashboard are free. With no credit left, replies are no longer scored and your chatbot keeps answering. Details are on the pricing page.
Start your 7-day free trialFAQ about the answer reliability score
What is a chatbot answer reliability score?
It is an automatic estimate of whether a chatbot reply is based on your own content. In Heeya, each reply is checked after it is sent and flagged in Analytics as probably reliable, to check, risky, or information probably not found. It tells you which replies to read first. It is not proof that a reply is right or wrong.
How is the score computed?
A second model reads the visitor's question with the last messages of the conversation, the passages the chatbot retrieved from your content, your instructions and the reply. It does not write text: it returns a probability for each possible diagnosis. The reliability percentage compares the probability that the reply is grounded with the probability that it is not.
Does the score slow down the chatbot or show up for visitors?
No. The score is computed right after the reply is sent, so the visitor never waits for it, and it only appears in your dashboard. If the check fails, the reply simply has no score and the conversation goes on as usual.
What does it cost?
Each analysed conversation uses 1 message credit, once per conversation, however many replies are scored. Simple greetings and your own tests from the dashboard are free. With no credit left, replies are no longer scored and the chatbot keeps answering normally.
Which plans include the reliability score?
The Standard plan at 19 euros per month, including its 7-day free trial without a credit card, the Premium plan at 99 euros per month and the Enterprise plan. You turn it on per agent in the Tools page.
What should I do with an unanswered question?
Read the reply first. If the information is missing from your content, click Add as Q&A under the reply: a form opens with the visitor's question already filled in, you write the right answer, and the agent uses it for the next visitors. Adding a Q&A from Analytics is available on all plans.
Related reading
Why chatbots get answers wrong, how to measure them, and how retrieval keeps answers grounded.
Guide
AI chatbot hallucinations
Why chatbots invent answers, and the guardrails that reduce it.
Metrics
Chatbot KPIs: 15 metrics
Definitions, formulas and benchmarks, including intent coverage.
Architecture
RAG for customer service
How retrieval-backed support works, and where it still needs supervision.
Read the answers that need it, fill the gaps your visitors find
Create your agent, load your documents and turn on the reliability score during the free trial.
