RAG chatbot evaluation template:
50 tests you can score
A retrieval-augmented chatbot is not evaluated by asking a few happy-path questions. This CSV is a 50-row golden set for a docs-grounded support agent: in-source facts, paraphrases, procedures, numbers, refusals, out-of-scope prompts, false premises, and two instruction-override attempts. The scoring method is on this page and in the sheet.
UTF-8 CSV, quoted fields, Excel and Google Sheets ready. No email gate. The seed knowledge pack lives on this page so every gold answer can be checked.
What you get
- 50 standalone questions against one fictional SaaS help center (Atlasboard), so the sheet is runnable today.
- 11 failure modes, not a single accuracy column. Refusals are first-class tests.
- Three 0-2 scores per row aligned with RAGAS faithfulness and answer relevance, and with Microsoft Foundry groundedness and relevance.
- Weights 1, 2, or 3 so a billing hallucination costs more than a vague clarifying miss.
- Built-in spreadsheet formulas for test points and weighted points. Overall maximum is 780.
What a RAG chatbot evaluation template is
A RAG chatbot evaluation template is a fixed set of questions, expected behaviors, and a scoring rule you reuse before launch and after every knowledge-base change. The job is not to admire fluency. The job is to see whether answers stay inside the sources, whether the model refuses when the sources are silent, and whether a false premise is corrected instead of confirmed.
Accepted practice splits the pipeline. The RAGAS paper (Es, James, Espinosa-Anke, Schockaert) scores faithfulness (claims supported by retrieved context), answer relevance (the reply addresses the question), and context relevance (retrieved passages are focused). Current RAGAS metrics still list faithfulness and response relevancy, plus context precision, context recall, and NVIDIA's response groundedness. Microsoft Foundry evaluations use a CSV or JSONL test set and quality evaluators including groundedness and relevance. The Hugging Face RAG evaluation cookbook starts from question-answer pairs, then applies an evaluator.
This page is the pair set plus a human evaluator you can run in a spreadsheet. It does not replace RAG for customer service, which keeps the architecture and weekly LLM-as-judge loop, or knowledge base engineering for AI chatbots, which owns retrieval precision at k. If you are choosing a vendor rather than scoring an agent you already have, use the AI chatbot RFP template.
The 50 tests, by failure mode
Counts are exact. Weights raise the cost of invented facts and failed refusals. Ambiguous questions are lighter because a clarifying question is a valid pass.
| Category | Rows | Intent | Weight | What a pass looks like |
|---|---|---|---|---|
| factual_recall | T01-T08 (8) | answer | 3 | Names the fact as written in the seed article. |
| paraphrase | T09-T13 (5) | answer | 2 | Retrieves the same fact under different wording. |
| procedural | T14-T18 (5) | answer | 2 | Gives the documented path and constraints, not a guessed UI. |
| numerical_limits | T19-T22 (4) | answer | 3 | Exact figure, unit, and scope (per workspace, per minute). |
| multi_hop | T23-T26 (4) | answer | 2 | Combines two articles (role plus plan, region plus Pro). |
| negative_constraint | T27-T30 (4) | answer | 3 | Docs say no, and the reply says no. |
| missing_knowledge | T31-T36 (6) | refuse | 3 | Abstains. Does not invent HIPAA, SOC 2, apps, refunds, or counts. |
| out_of_scope | T37-T41 (5) | refuse | 3 | Refuses scraping, trivia, medical advice, news, and rival pricing copy. |
| false_premise | T42-T45 (4) | correct_premise | 3 | Corrects the false claim before any how-to. |
| ambiguous | T46-T48 (3) | clarify | 1 | Asks which plan or object, or lists every documented variant. |
| adversarial | T49-T50 (2) | refuse | 3 | No system prompt dump, no invented card digits. |
Scoring method
Each row uses three integer scores. Empty score cells stay empty until you judge the reply. Formulas in test_points and weighted_points fill themselves in Excel or Google Sheets.
| Score | Faithfulness | Relevancy | Task |
|---|---|---|---|
| 2 | Every factual claim is supported by the seed pack. For refuse rows, no invented product fact. | Addresses the asked question, or is a relevant refusal or clarifying question. | Matches gold_behavior for that intent: answer, refuse, correct premise, or clarify. |
| 1 | Mostly grounded, with one extra unsupported detail that does not change the decision. | Related but incomplete, padded, or missing a required constraint. | Partial: right direction, missing a must-include item listed in the row. |
| 0 | A material invented or contradicted fact. | Answers a different question, or answers when the row requires a refusal (or refuses when the seed contains the answer). | Misses the job: wrong fact, accepted false premise, or leaked instructions. |
Formulas
- test_points (column R, row 2 as example):
=IF(COUNT(M2:O2)<3,"",SUM(M2:O2)). Maximum 6. - weighted_points (column S):
=IF(R2="","",R2*D2). - Overall percent, after all 50 rows are scored:
=SUM(S2:S51)/(6*SUM(D2:D51))*100. The denominator is 780 when every weight is unchanged.
Faithfulness here is the human analogue of RAGAS faithfulness (fraction of answer statements supported by context) and of Microsoft groundedness. Relevancy is the analogue of RAGAS answer relevance and Foundry relevance. Task is the row-level job: correctness against the gold behavior, including abstention. We do not score context precision in this sheet because the CSV is filled from the user-visible reply, not from retrieved chunk lists. If your product exposes retrieved passages, log them in scorer_notes.
Do not publish the overall percent as "accuracy." Report it next to the three high-weight refusal groups: missing_knowledge, out_of_scope, and false_premise. A high total with a weak refusal group is still a hallucination risk. For why grounding does not mean zero invention, see chatbot hallucinations and reliability guardrails.
How to run the 50 tests
- Download the CSV. In Google Sheets: File, Import, Upload, comma separator. In Excel: open the file; the UTF-8 BOM is there so Windows Excel keeps punctuation intact.
- Copy the Atlasboard seed pack below into a
.txtfile. That pack is fictional. It is not Heeya, and it is not a real customer. - Load the TXT into the agent you are evaluating. On Heeya that is an upload of a supported document (TXT is supported), then wait until processing finishes.
- Ask each
questionas a standalone turn. Heeya rewrites follow-ups when chat history exists. Mixing tests in one thread contaminates retrieval. A new conversation per question is the clean method; a single thread of full standalone questions is the next best. - Paste the reply into
observed_answer. Score M, N, and O using the rubric. Usemust_include,must_not_claim, and the pass/partial/fail rules. Do not grade style. - After a knowledge-base edit, rerun at least the numerical_limits, missing_knowledge, and false_premise rows before you trust the rest.
Score grounding against the seed articles, not against a citation chip. Heeya's widget does not show a separate source-citation UI. The model may mention a filename or URL when that string is in the retrieved text. A nameless answer can still be faithful; a named document does not make an invented number true.
Atlasboard seed pack (fictional help center)
Copy this block as-is into the TXT you upload. Every gold answer in the CSV is taken from these sentences. If you swap Atlasboard for your product later, replace questions and gold behavior together. Do not keep Atlasboard numbers against your own docs.
Roles and permissions
Workspace roles are Owner, Admin, Member, and Viewer. Only the Owner can change the workspace plan or billing method. Admin can invite members, manage sources, and export data. Member can create dashboards and share them with people who already have a seat. Viewer can view dashboards they were shared. Viewer cannot export. Owner and Admin can view the audit log. SCIM provisioning is available on Enterprise only.
Authentication
SSO supports SAML 2.0 and Google Workspace. OIDC is not supported. Multi-factor authentication supports TOTP authenticator apps only. SMS is not supported. Password reset links expire after 60 minutes. Sessions end after 14 days of inactivity.
API and developers
Create API keys at Settings, then Developers, then API keys. A new key is shown once at creation. The rate limit is 120 requests per minute per workspace. There is no public GraphQL API.
Webhooks
Webhooks are available on Pro and Enterprise. A failed delivery is retried 3 times, then marked failed.
Data retention and regions
Hosting regions are EU (Frankfurt) and US (Virginia). The workspace region is chosen at creation and cannot be changed later. Starter retains event data for 90 days. Pro retains event data for 13 months. Enterprise retention is contractual. To export data, open Settings, then Data export, choose CSV or JSON, and wait for a ZIP emailed within 30 minutes.
Plans and seats
Starter includes 3 seats, 90-day retention, no webhooks, no audit log, and no custom domain. Pro includes 15 seats, 13-month retention, webhooks, a 90-day audit log, and custom domain. Extra seats are available on Pro only. Trial is 14 days with no credit card. SCIM is Enterprise only. The audit log on Pro is retained 90 days and can be viewed by Owner and Admin.
Dashboards and sharing
A dashboard can be public only if the workspace Owner enables public sharing. Public dashboards never include raw event rows.
CSV import
CSV imports must be UTF-8, maximum 10 MB, with the first row as headers.
Custom domain
Custom domains are a Pro feature. Add a CNAME pointing to docs.atlasboard.example.
Support and status
Support hours are Monday to Friday, 09:00-18:00 CET. There is no native Slack, WhatsApp, or Intercom connection. Incident updates are posted only on status.atlasboard.example.
CSV columns
Header plus 50 data rows, 19 columns, every row the same width. Score columns start empty. Formula columns already contain row-relative formulas.
- test_id T01 to T50.
- category / intent / weight as in the table above. Intent is answer, refuse, correct_premise, or clarify.
- question the exact prompt to send.
- source_condition what must be present or absent in the seed pack.
- gold_behavior the expected job, not a word-for-word script.
- must_include / must_not_claim hard checks for scoring.
- pass_rule / partial_rule / fail_rule mapped to scores 2, 1, and 0 on the task column.
- faithfulness_0_to_2, relevancy_0_to_2, task_0_to_2 you fill with 0, 1, or 2.
- observed_answer / scorer_notes free text.
- test_points / weighted_points spreadsheet formulas.
Using the template on Heeya
Heeya is a docs-grounded agent: you add sources, the system retrieves passages, and the model is instructed to answer from that context and to say when the information is not there. That is the same job this sheet tests. Retrieval reduces unsupported answers; it does not remove them. Run the sheet; do not skip it because the architecture is RAG.
After the Atlasboard run, replace the seed pack with your help center and rewrite the 50 questions from real tickets. Keep the category mix. A SaaS team that wants the commercial product path (upload, guidance, embed script) can continue on AI customer support for SaaS. Current plan names and limits stay on Heeya pricing. This page does not copy amounts.
Out of scope for these 50 rows: the contact form tool, ticket routing, CRM, Slack, and any automatic account action. Those are not how you score retrieval-grounded answers, and several of them are not Heeya capabilities. The form, if you enable it, appears as one block in the conversation. It is not a pass condition here.
FAQ about the RAG chatbot evaluation template
How do I score the 50 tests?
For each row, paste the observed answer, then score three columns from 0 to 2: faithfulness, relevancy, and task. The CSV formulas add those three scores (maximum 6) and multiply by the row weight. Overall percent is the sum of weighted points divided by 780.
Do I need RAGAS or an LLM-as-judge?
No. This template is a human rubric mapped to the same quality dimensions those tools measure: faithfulness or groundedness, answer relevancy, and whether the job was answering, refusing, correcting a false premise, or asking a clarifying question. You can later automate the same rows with RAGAS or Microsoft Foundry evaluators if you want a second pass.
Why include questions the docs cannot answer?
A RAG chatbot can fail by inventing an answer when retrieval is empty or irrelevant. Missing-knowledge, out-of-scope, false-premise, and adversarial rows exist to score that failure. An agent that scores well only on in-source facts is not evaluated for hallucination risk.
Can I use this CSV with Heeya?
Yes. Copy the Atlasboard seed pack on this page into a TXT file, upload it to a Heeya agent, then ask each CSV question in the dashboard test chat. Run each question as a standalone turn so conversation rewriting does not mix tests. Heeya instructs the model to say when the retrieved context does not contain the answer. That instruction reduces unsupported answers; it does not remove the need to test.
What if the chatbot does not cite sources?
Score faithfulness against the seed articles, not against a citation chip. Compare each factual claim in the reply with the matching article. A correct answer that names no filename can still score 2 for faithfulness. An answer that invents a fact fails even if it mentions a document name.
How is this different from retrieval precision at k?
This template scores the user-visible answer. Retrieval metrics such as precision at 5 belong to knowledge-base engineering: they ask whether the right chunks were fetched. You can fail retrieval and still produce a lucky answer, or retrieve well and then over-generate. Keep those measurements on separate sheets.
Does a high score mean the chatbot will not hallucinate in production?
No. Fifty tests on one seed pack are a pre-launch slice, not a published accuracy rate. Read the missing-knowledge, out-of-scope, and false-premise category scores even when the overall percent looks high. Replace Atlasboard questions with your own tickets before you treat the sheet as a go-live gate.
Related reading
This URL owns the downloadable 50-test pack. The pages below keep their own intents.
Product
RAG expertise
How Heeya runs retrieval-augmented generation. Not a test spreadsheet.
Architecture
RAG for customer service
Faithfulness, relevancy, and context precision as production metrics, including LLM-as-judge loops.
Sources
Knowledge base engineering for AI chatbots
Golden queries for retrieval precision at 5, chunking, and hygiene. Different measurement than this CSV.
SaaS
AI customer support for SaaS
The commercial page for a docs-grounded website agent, after you have a test method.
Reliability
Hallucinations and guardrails
Why RAG still needs an "I do not have that" path. This template is one way to test that path.
Primer
What is RAG?
Business-level explanation of retrieval-augmented generation, including graceful abstention.
Download the sheet, then run it on an agent
The CSV is the artifact. Scoring is manual and inspectable. If you want to run the Atlasboard pack on a Heeya agent, create an account, upload the TXT, and ask the 50 questions in the dashboard.
