I Built a Q&A Bot for My Docs and Almost Gave Up (Here's What Worked)

zhongqiyue Posted on May 30 I Built a Q&A Bot for My Docs and Almost Gave Up (Here's What Worked) # python # ai # webdev # tutorial A few months ago, I decided to build a Q&A bot for my project’s documentation. You know the dream: users type a question, and the bot answers instantly from the docs. No more digging through pages. No more stale FAQs. I thought it would be straightforward. Slap an LLM on top of a text file and call it a day. Oh, how wrong I was. The Problem That Nearly Broke Me I had a bunch of Markdown files – about 50 pages of setup guides, API references, and troubleshooting. I wanted the bot to answer questions like “How do I configure authentication?” or “What’s the maximum payload size?” My first attempt: dump the entire documentation into a single prompt and ask GPT-4 to answer. It worked… for the first two questions. Then I hit the token limit. Then I realized I was spending $0.50 per query. Then I noticed the model hallucinating answers from unrelated sections. I needed a smarter approach. But every tutorial I found either oversimplified (“just use LangChain!”) or assumed I had a PhD in information retrieval. What I Tried That Didn’t Work 1. Fine-tuning a model I spent a weekend preparing a dataset of question-answer pairs from my docs. Fine-tuned a small LLaMA model. The result? It memorized exact phrases but couldn’t generalize to rephrased questions. Also, updating the docs meant retraining. Hard pass. 2. Raw vector search without an LLM I embedded all the doc chunks, stored them in Pinecone, and returned the top-3 chunks as the answer. Users got a wall of text. No summarization. No conversation. It felt like Google without the ranking. 3. Prompt engineering with sliding windows I tried to dynamically select relevant chunks and inject them into a prompt. But I kept running into context window issues. Plus, the model would sometimes ignore the provided context and make stuff up. What Eventually Worked: A Minimal RAG Pipeline After