You built a chatbot for your class. You uploaded your slides, hit share, and a student asked about photosynthesis. The chatbot answered from a random internet source instead of your Unit 3 notes.
The chatbot worked. The knowledge base didn’t.
Most teachers who try building a knowledge base chatbot run into this exact problem. The tool is fine but what you feed it is what makes or breaks it. Upload the wrong files, skip the structure, or leave old materials in there, and the chatbot gives answers you would never approve.
This guide covers what to include in your knowledge base, what to leave out, how to organize it, and how to test that the chatbot actually pulls from your content instead of guessing.
What is a knowledge base chatbot?
A knowledge base chatbot is a chatbot that answers from a specific set of documents you provide, not from the open internet or its general training data.
In business, that usually means help docs and company policies. In education, it means your teaching materials: lesson notes, rubrics, study guides, and problem sets.
Here is how it works in plain terms. You upload files. When a student asks a question, the chatbot searches those files for the most relevant content and generates an answer based on what it finds.
If the answer is in your materials, the chatbot uses it. If it is not, the chatbot should say it does not know.
That last part is what makes a knowledge base chatbot different from a generic one like ChatGPT or Claude. A generic chatbot answers from everything it was trained on, which includes the entire internet. That is useful for general questions, but it is a problem in a classroom.
The chatbot might give an answer that contradicts your textbook, uses terminology you have not introduced yet, or pulls from a source you would never assign.
A study published in JMIR tested this directly. Chatbots without a curated knowledge base hallucinated in roughly 40% of responses. When grounded in a reliable, domain-specific knowledge base, the hallucination rate dropped to 0% for GPT-4 and 6% for GPT-3.5. The knowledge base was the only variable that changed.
For teachers, grounding a chatbot in your own materials means students get answers aligned to your curriculum, your grading criteria, and your expectations. Not a random summary from the internet.
If you want to learn more, this blog is part of our guide on AI Chatbots for Education, which covers how chatbots work in the classroom, what to look for when choosing one, and how to set guardrails that keep students on track.
What makes a good knowledge base for a chatbot?
The chatbot can only work with what you give it. A messy, bloated, or outdated knowledge base leads to wrong answers, no matter how good the AI model is. A study in JMIR Formative Research on medical chatbots found that simply adding more information to a chatbot’s knowledge base did not improve accuracy.
In some cases, it made things worse. What improved performance was filtering and structuring the content so the chatbot retrieved the right information, not just more of it.
The same principle applies in the classroom. What makes a knowledge base chatbot useful is not how much you upload. It is what you include, what you leave out, and how you organize it.
What to include and what to leave out
| Include | Leave out |
|---|---|
| Lesson notes and unit summaries | Full textbook PDFs (too long, too noisy) |
| Rubrics and grading criteria | Slide decks with no speaker notes |
| Problem sets with worked examples | Marketing materials or school brochures |
| Vocabulary lists and sentence frames | Outdated materials from previous semesters |
| FAQ-style documents (questions students always ask) | Answer keys or test banks |
What to include. Focus on materials that directly answer the questions students actually ask. Lesson notes and unit summaries are the backbone. Rubrics help the chatbot give feedback aligned to your standards.
Problem sets with worked examples let the chatbot walk students through solutions the way you would. And FAQ-style documents (the questions students ask every semester, like “What counts as a primary source?”) retrieve best because they already match the question-answer pattern.
What to leave out. More is not better. A 300-page textbook gives the chatbot too much to search. It pulls from the wrong chapter or mixes up concepts. Break it into chapters or extract only the relevant pages. Slide decks with no speaker notes are another problem.
A slide that says “Photosynthesis” next to a diagram gives the chatbot nothing to work with because there is no text to search. Export slides with speaker notes, or write a companion doc.
Remove outdated materials from previous semesters because the chatbot cannot tell what is current, and never upload answer keys or test banks. If it is in the knowledge base, the chatbot can surface it to students.
How to organize your materials
How you structure the knowledge base matters as much as what you put in it.
- Break large files into smaller, topic-focused ones. A 50-page curriculum guide is harder for the chatbot to search than five 10-page unit summaries. Smaller files retrieve better.
- Add clear headings inside your documents. The chatbot searches by meaning. “Unit 3: Cell Division” retrieves more accurately than “Chapter 3.” Label sections so the chatbot knows what each part is about.
- Name files descriptively. “grade-9-algebra-unit-4-quadratics.pdf” tells the chatbot (and you) what is inside. “doc_final_v2.pdf” does not.
- Keep a running list of what is in the knowledge base. When you start a new unit or semester, you need to know what to swap out. A simple spreadsheet or checklist works.
How to build a knowledge base chatbot with your materials
These are the five decisions that determine whether your knowledge base chatbot actually works.
1. Define the chatbot’s purpose first
A tutoring bot, a writing feedback bot, and an exam review bot need very different knowledge bases. Decide what the chatbot is for before you upload anything. The purpose determines what goes in.
2. Write clear instructions
The instructions tell the chatbot how to behave. The knowledge base tells it what to know. You need both. Tell it to answer only from the materials, to ask clarifying questions instead of guessing, and to admit when it does not have enough information.
In Edcafe AI, this is the Instructions tab, where you write or generate your chatbot’s behavior rules using the built-in Prompt Assistant.
For more on writing strong instructions, see our 5-point AI prompting framework.
3. Upload your materials
In Edcafe AI, the Knowledge tab lets you add knowledge in three ways: upload files (PDF, Word, PowerPoint, Excel, and images), add a web page URL, or write a text note directly.
Use whichever format fits your content. Follow the include/exclude guidelines from the previous section.
4. Set knowledge limits
This is the most important setting. Edcafe AI gives you two options: “All knowledge” (the chatbot uses your files plus its general knowledge) or “Files only” (the chatbot answers strictly from your uploaded materials).
For most classroom use, “Files only” is safer. It keeps the chatbot on-curriculum and reduces the risk of AI hallucinations.
5. Test before sharing
Ask questions you know the answer to and check if the chatbot gets them right. Then ask questions your knowledge base does not cover and see if the chatbot admits it does not know. If it makes something up instead, adjust your instructions or your materials. Ask a colleague to try to break it before your students do.
Looking for the full setup walkthrough? See our step-by-step guide on how to build an AI chatbot with no code.
Common mistakes that make a knowledge base chatbot fail
A knowledge base chatbot can have the right materials and still give bad answers. These are the mistakes teachers make most often.
Uploading everything at once
A 200-page textbook PDF gives the chatbot too much to search. It pulls from the wrong chapter or mixes up unrelated sections. Start with one unit at a time. You can always add more.
No instructions, just materials
A knowledge base without instructions is like handing a substitute teacher your filing cabinet and no lesson plan. The chatbot needs to know how to behave, not just what to reference.
Leaving old materials in the knowledge base
Last semester’s reading list is still in there. A student asks about the current assignment and gets last year’s answer. Clean your knowledge base when you change units.
Assuming the chatbot reads images
A slide with a diagram and no text is invisible to most chatbots. If there are no words to search, the chatbot skips it. Add captions, alt text, or a companion document that describes what the visual shows.
Never testing “I don’t know” answers
This is the most important test and the one most teachers skip. Ask a question your knowledge base cannot answer. If the chatbot makes something up instead of saying it does not know, your students will get wrong information and trust it. IBM researchers found that AI models are trained to always produce an answer, even when they should not. Testing this path is how you catch that before your students do.
How to keep your knowledge base updated
A knowledge base chatbot is not something you set up once and forget. It needs to change when your course does.
Review and swap materials when you start a new unit or semester. Remove outdated readings, old rubrics, and last year’s problem sets. If a document is no longer part of your course, it should not be in the knowledge base.
Add new materials as you create them. The knowledge base should grow with your course, not stay frozen from the first week.
Check student conversations periodically. If students keep asking about something the chatbot cannot answer, that is a signal. You probably need to add a document for it.
The questions students ask are the best guide for what your knowledge base is missing.