Sister companyNBM Games: discover our game studio at nbmgames.co
All insightsAI agents

Before you build an internal knowledge assistant: a practical checklist

An assistant that answers from your own documents is only as good as those documents, and only as safe as its access rules. What to check before the first line of code.

· 5 min read

An internal knowledge assistant answers employees’ questions from the company’s own material: policies, product documentation, past tickets, process guides. Most are built with retrieval-augmented generation (RAG). For each question, the system searches your documents for the most relevant passages and hands them to a language model, which writes an answer based on them.

The language model gets most of the attention, but it is seldom the hard part. Whether people trust the assistant depends on the documents behind it, the access rules around it and the way its answers are checked. Go through this list before you start building.

1. Source quality and ownership

The assistant cannot be more accurate than its sources. Before indexing anything, look closely at what you are about to feed it.

  • List the candidate sources (wiki, shared drives, policy documents, ticket history, product manuals) and choose which ones are in scope for the first version.
  • Give every source an owner who can confirm it is correct and who will fix it when it is not.
  • Remove or mark outdated versions and duplicates. If two documents disagree, the assistant will disagree with itself.
  • Check formats. Scanned PDFs, tables inside images and slide decks need extra processing before their content can be searched reliably.

Starting with one team’s well-maintained documents is usually better than indexing an entire drive.

2. Permissions and access control

The basic rule: the assistant must never show anyone content they could not open themselves. Get this right first, because a single leak of salary data or a confidential contract can end the project.

  • Carry each document’s permissions into the search index, and filter results by the user’s identity at the moment of the search, not only in the interface.
  • Decide how permission changes reach the index, and how quickly. Someone who leaves a project should lose access in the assistant too.
  • Keep highly sensitive material, such as HR files, salaries or legal matters, out of scope until access control has been tested.
  • Log who asked what and which documents were used, with a retention period that fits your privacy policy.

3. Freshness

  • How often does each source change, and how often will it be re-indexed?
  • When a document is deleted or replaced, is the old version removed from the index as well?
  • Can users see the date of the source an answer is based on, so they can judge whether it is still current?

4. Citations

Every answer should show where it came from: the document, the section and a link to it. Citations let users check an answer in seconds, let source owners find and fix wrong content, and make it acceptable for the assistant to say it does not know.

  • Link to the specific section or passage, not just to a 200-page file.
  • If no relevant source is found, the assistant should say so rather than answer from the model’s general knowledge.
  • Make it easy to flag a wrong or outdated answer, and send that flag to the owner of the source.

5. An evaluation set

Before building, collect real questions from the people who will use the assistant: from support tickets, internal chat channels and onboarding sessions. For each one, write down the expected answer and the document it should come from. A few dozen well-chosen questions are enough to start, as long as they include the hard cases:

  • Questions whose answer is not in the documents. The right response is “I could not find this”, not a guess.
  • Questions where permissions matter. Ask them as different users and check that each one sees only what they should.
  • Questions phrased the way people actually ask them, with abbreviations, typos and internal jargon.

Run the set again after every change to sources, prompts, search settings or models, and compare the results. Without it, every change is a guess.

6. Failure modes to plan for

  • Confident wrong answers. Language models fill gaps fluently. Answering only from retrieved sources, showing citations and declining when no source is found all reduce this risk.
  • Retrieval misses. The right document exists but is not found, often because of different wording, long documents split in the wrong places or content locked inside tables.
  • Stale or conflicting sources. The answer is faithful to a document that is no longer true.
  • Prompt injection. Text inside a document or message tries to give the assistant instructions. Treat document content as data, not as commands, and limit what the assistant is allowed to do.
  • Over-trust. When answers are usually right, people stop checking. Keep sources visible, and make uncertainty visible too.

7. Cost

Plan for three kinds of cost. One-time work: preparing sources, building connectors, mapping permissions and creating the evaluation set. Running costs: model usage for every question, keeping the search index up to date, and hosting. And people’s time: source owners keeping content current, and someone reviewing feedback and quality every week.

Model usage is typically priced by the amount of text processed, so estimate it from the number of questions you expect per day and the length of the passages sent with each one. Set budget alerts from the start, and revisit the estimate once real usage data comes in.

A good first version

If you can answer every item above for one team and one set of sources, you are ready to build a first version people will trust. If several items are still open, closing them is the most valuable work you can do before choosing any technology.

Next step

Reading is good. Building is better.

Bring a problem. We will bring a plan.

Or write to us: hello@nbmcore.com