Free diagnostic

RAG knowledge base readiness assessment

A RAG knowledge base readiness assessment scores whether your documentation is safe and useful to retrieve from, before you index it for a chatbot. This page is an 18-question diagnostic. It does not upload files. The same answers always produce the same score.

Use it when you are about to ground an agent on a help center, product docs, or a drive of PDFs. It is not a live-answer evaluator. After you have an agent, measure retrieval with a golden question set, as described in the knowledge base engineering guide.

Start the assessment

What this diagnostic measures, and what it does not

Retrieval-augmented generation answers from chunks of your sources. If those sources are unstructured, stale, duplicated, incomplete, or confidential, the model will retrieve that. Comparable tools split into two jobs. This page does the first job only.

This page: source readiness

Help-center health dashboards typically flag unused, outdated, and duplicate articles after a corpus is published. Knowledge-management maturity models (ownership, lifecycle, access) describe the same operational gaps. This diagnostic asks those questions before ingest, including confidentiality.

The six dimensions are structure, freshness, coverage, consistency, confidentiality, and governance. They match the failure modes that show up first in production RAG: mixed topics in one file, retired policies still indexed, tickets with no article, two versions of the same rule, and internal notes leaking into a customer chat.

Not this page: live RAG evaluation

RAGAS scores a running system: context precision and recall, faithfulness, response relevancy. Those metrics need retrieved passages and generated answers. They cannot tell you whether last year's refund PDF should have been indexed at all.

Pinecone's chunking guidance states the practical test: if a chunk makes sense to a human without surrounding context, it will make sense to the model. Headings, topic-scoped files, and extractable text are how you get those chunks. This form asks about that source shape, not about token windows.

Anthropic's Contextual Retrieval write-up (19 September 2024) showed that stripping context at chunk time is a retrieval failure mode. In their experiments, adding chunk-specific context plus BM25 reduced top-20 retrieval failures from 5.7% to 2.9%. That is their lab result, not a Heeya outcome. It is cited here because it is a documented reason to score source structure before you embed. The NIST AI Risk Management Framework is a voluntary governance reference for data practices; this form is not a conformity assessment against it.

How the score is calculated

The method is published so two people with the same corpus get the same number. There is no hidden weight, no machine-learning judge, and no sampling.

  1. 1. Eighteen ordinal questions. Six dimensions, three questions each. Every answer is 0, 1, 2, or 3. Higher is better. Options are mutually exclusive.
  2. 2. Equal dimension weight. Each dimension totals 0 to 9. The raw score is the sum of all six, so 0 to 54. No dimension is silently boosted.
  3. 3. Integer percent. Displayed score = round(100 × raw / 54).
  4. 4. Bands on that percent. 0-39 Not ready. 40-69 Partial. 70-84 Ready with conditions. 85-100 Ready to test.
  5. 5. Two gates that can override the percent. A 0 on personal data (p-pii) or on secrets (p-secrets) sets the band to Do not ingest yet, even at 90+. A 0 on canonical sources (k-canonical) caps Ready or Ready with conditions down to Partial. Gates never raise a score.

The percent is a readiness index for ingest, not an accuracy rate, not a ticket-deflection forecast, and not a compliance certificate. Self-reporting can be optimistic. If two people disagree on a question, use the lower score.

For how retrieval itself works in Heeya, see RAG expertise and RAG for customer service. This URL stays the diagnostic.

Score your knowledge base

Answer for the set of files and URLs you would actually ingest, not for the documentation you wish you had. Required: every question. JavaScript scores in this browser.

0 of 18 answered

0 of 18
Structure

Whether a retriever can split sources into passages that still make sense alone.

Headings and sections

Can someone find a single topic from headings or a table of contents, without reading the whole file?

One topic per source

Would a 300-page mixed PDF be the unit of ingest, or a page per policy or feature?

Extractable text

Can the text be copied from the file, or is it a photograph of a page?

Freshness

Whether retrieval would surface the current rule, not last year's copy.

Review or effective dates

Can you tell when each source was last checked?

Retired versions

If v1 and v2 of a policy both get indexed, retrieval cannot know which one won.

Update cadence

When the product or a policy changes, how do the sources change?

Coverage

Whether real questions have a source, and whether the agent's job has a written edge.

Real questions mapped to sources

Have you taken actual user phrasing (tickets, chat, search logs) and pointed each one at a document?

In-scope and out-of-scope

Does the team agree what the agent must refuse (account actions, legal advice, other products)?

Known missing articles

Coverage work is a list, not a feeling that "the drive is full."

Consistency

Whether two retrieved passages can disagree, and whether names match the product.

One canonical source per policy

A 0 here caps a high overall score at Partial. Retrieval cannot pick a winner among contradictions.

Duplicates

Near-duplicate chunks waste context and can outrank a more specific page.

Names and terms

Users search the names they see in the product. Docs that still say the old name retrieve poorly.

Confidentiality

Whether indexing would make personal data or internal-only material retrievable in chat. A 0 on the first two questions blocks ingest. This is not a legal audit. For data-protection architecture, see the GDPR-compliant AI chatbot guide.

Personal data in the ingest set

Ticket dumps, invoices, and CRM exports do not belong in a customer-facing knowledge base.

Secrets and admin-only runbooks

If you would not quote it to a visitor, it should not be in the index.

Audience labels

Internal and public material in one pile is how the wrong passage gets retrieved.

Governance

Whether anyone is accountable after the first ingest. A knowledge base without owners drifts.

Named owner per source

Who is wrong if this page is stale next quarter?

Ship-to-docs handshake

A feature that shipped without a docs update is a retrieval bug waiting for the first user.

Who may add or remove sources

An open shared drive is not an ingest policy.

FAQ about RAG knowledge base readiness

What is a RAG knowledge base readiness assessment?

It is a scored review of the documents you would retrieve from, before you index them. It measures structure, freshness, coverage, consistency, confidentiality, and governance. It does not measure a live chatbot, embedding quality, or answer faithfulness.

How is the score calculated?

Eighteen questions, three per dimension, each worth 0, 1, 2, or 3 points. The raw total is out of 54. The displayed score is that total divided by 54, multiplied by 100, and rounded to an integer. Dimensions are equally weighted. A score of 0 on personal data or secrets blocks ingest. Known contradictions with no canonical source cap the band at Partial.

Is this the same as RAGAS or faithfulness scoring?

No. RAGAS and similar evaluators score a running retrieval system: whether retrieved passages support an answer, and whether the model stayed inside those passages. This diagnostic scores the corpus you would feed that system. Use this first. Use retrieval metrics after you have an agent and a golden question set.

Do my answers leave this browser?

No. Scoring runs in JavaScript on this page. There is no form post, no account requirement, and no upload of your documents. Copying the report copies text locally. Reloading the page clears the answers.

Does a high score mean the chatbot will not invent answers?

No. A ready corpus makes grounded answers possible. Retrieval-augmented generation still reduces unsupported answers rather than removing them. Stale, contradictory, or missing sources still produce stale, contradictory, or incomplete replies.

Can I use this if I have not chosen a vendor?

Yes. The questions are about your sources, not about a product. The CTA after a high score points to Heeya because this page is published by Heeya. The numeric result is usable with any retrieval stack.

What should I do with a blocked or low score?

If the confidentiality gate failed, remove personal data and internal-only material before any ingest. If the score is Not ready or Partial, fix the weakest dimensions, then score again. The knowledge base engineering guide on this site covers inventory, duplicates, ownership, and drift.

After you have a score

If the band is Ready to test or Ready with conditions, you can try retrieval on a curated set. Heeya accepts PDF, DOCX, PPTX, and TXT uploads, selected website pages, and question-and-answer pairs. The agent searches that collection and, if you enable it, can show a contact form. It does not replace a helpdesk, take account actions, or certify that answers are complete.

Plans are listed on Heeya pricing. The SaaS product page is AI customer support for SaaS. For why unsupported answers still happen, see chatbot hallucinations and reliability guardrails.

Method and copy: Anas Rabhi.