Guide · RAG
How to build a RAG knowledge assistant: choosing your stack, layer by layer
Retrieval-augmented generation is the default way to put your own documents and data behind a chat interface — and the stack you choose decides whether that takes a weekend or a quarter. This guide walks the decision the way a senior engineer would: one fork at the top, then seven layers if you take the hard road. No rankings for hire, no vendor’s roadmap — just the trade-offs.
Choosing a RAG stack is hard because every layer has a dozen credible options, and almost none of them fail in the demo. You wire up a loader, a splitter, an embedding model, a vector store, and a prompt; you ask it three questions you already know the answers to; it works. Then real users ask real questions of real documents — a scanned contract, a table that matters, a term that means two different things in two departments — and the cracks show up everywhere at once. The tools won’t tell you which layer is failing, and most tutorials stop at the part that was never going to be the problem.
So resist the urge to start by picking a vector database. The decision that actually shapes your next three months is more basic: do you want to assemble and own the pipeline, or do you want a system that already works so you can stay out of the plumbing? Everything else follows from that answer.
The core decision: a platform, or compose it yourself
There are two honest ways to build a RAG assistant, and they lead to different jobs.
A batteries-included platform gives you ingestion, chunking, a vector store, retrieval, a chat UI, and usually an API — wired together and working — on day one. You upload documents, tune a few knobs, and ship. The platforms in our index differ mostly in shape: Dify is the broadest, a visual builder for RAG plus agents and workflows; RAGFlow leans hardest into deep document parsing — tables, layouts, scanned PDFs — which is where most RAG projects actually bleed; FastGPT and MaxKBcenter on knowledge-base Q&A with visual flow builders; WeKnora is a document-understanding knowledge framework in the same family; AnythingLLM is the fastest path to “chat with these files” for a team, self-hosted or on the desktop; and Khoj aims at personal and small-team knowledge over your notes and documents. What you give up is control at the seams — when retrieval is subtly wrong, you’re tuning inside someone else’s abstractions.
Composing it yourself means you own each layer: a scraper or loader, a parser, a vector store, a retriever with reranking, an orchestration framework, optional memory, and an eval harness. It’s more work up front and far more control when — not if — you need to fix retrieval quality, swap a model, or meet a latency or compliance constraint the platform won’t bend to.
Honestly: if your goal is to validate that a RAG assistant is useful at all, or to serve an internal team, start on a platform — you’ll learn what “good retrieval” means for your data in days instead of weeks, and you can graduate later. Choose the compose path when RAG quality is your product rather than a feature, when you have requirements a platform fights you on, or when you already know your documents are hard enough that generic parsing will lose. A good tell: prototype your actual hardest document — not a clean one — on a platform first. If it handles that, you may never need to compose.
The layers, if you compose
Seven layers, in the order they touch a query. For each: the decision, a recommended default, and the one mistake that quietly costs you.
1 · Ingestion & scraping
First question: is your source the web or your own documents? For web content, the job is turning messy HTML into clean, chunk-ready text — Firecrawl is the low-friction default (it renders JavaScript and returns markdown), Scrapling is the code-first choice when you’re scraping at volume and want to own the pipeline without per-page fees, and llm-scraper earns its keep when you need structured fields out of a page rather than prose. For PDFs, contracts, and other unstructured documents, unstract is built to pull structured data from the formats that break naive loaders.
— ingestion quality caps everything downstream — a table flattened into word soup, or a two-column PDF read straight across, quietly wrecks retrieval, and no reranker saves you from it.
2 · Chunking & parsing
This is the layer that most quietly decides answer quality, and the one people spend the least time on. The real choice is between naive fixed-size splitting and structure-aware chunking that respects headings, sections, tables, and lists — the latter is more work but keeps related ideas together, which is the whole game. Default to structure-aware chunks of a few hundred tokens with a little overlap, and use a layout-aware parser for anything with tables or columns instead of reading text straight off the page — exactly what RAGFlow does well if you’d rather not build it.
— the chunk size you copied from a tutorial is probably too big — oversized chunks dilute the embedding and bury the answer in surrounding noise. Evaluate retrieval on real questions before you trust any setting.
3 · Vector store
These aren’t in our index — they’re infrastructure, and the field’s standard picks are Pinecone (managed), Weaviate, Qdrant, and Milvus (self-hostable), plus pgvector if you’d rather keep vectors inside Postgres. The real decision is operational, not algorithmic: managed versus self-hosted, and whether you want one fewer moving part. Default to pgvector if you already run Postgres and your corpus is modest — you avoid standing up a new system — and reach for a dedicated store when scale, hybrid search, or heavy metadata filtering justifies it.
— this is the choice teams agonize over and it’s rarely the one that determines answer quality — don’t let vector-database benchmarks distract you from chunking and retrieval, which do.
4 · Retrieval + rerank
The upgrade that moves quality most is going from plain vector search to hybrid retrieval (keyword and vector together) followed by a reranker that reorders the top candidates for actual relevance. It costs you a little latency and one more component, and it’s almost always worth it — most “the model hallucinated” bugs are really “the right chunk was never retrieved.” TrustRAG is an in-index, RAG-focused toolkit if you’d rather adopt these patterns than hand-roll them, and for genuinely multi-hop questions an agentic retriever like deep-searcher can iterate where a single query can’t.
— measure retrieval on its own — did it recall the correct chunk? — before you blame the LLM, because you can’t prompt your way out of missing context.
5 · Orchestration / framework
This is the glue between retrieval, model calls, tools, and memory, and the market splits into RAG-and-pipeline frameworks and agent frameworks. For a document assistant specifically, Haystack offers a clean, production-minded pipeline built around retrieval; LangChain gives you the largest ecosystem and the most integrations at the cost of heavier abstraction; and Pydantic AI is the lightweight, type-safe pick when you want explicit control and less magic. When your assistant genuinely needs multiple cooperating agents — a researcher plus a writer, say — Agno and CrewAI are built for that, but don’t reach for them early.
— a framework you’re fighting costs more than the boilerplate it saved — prototype your hardest flow in two candidates and keep the one that gets out of your way.
6 · Memory
Decide honestly whether you need memory that persists across sessions or just conversation history within one. Plain chat history — the last few turns in the context window — is simpler, adds no infrastructure, and is all most document assistants ever need; it just forgets between sessions and eventually crowds the context on long chats. mem0 is the in-index option when you genuinely want a persistent memory layer that extracts and recalls salient facts about a user over time, which is powerful for personalization.
— persistent memory is itself a retrieval problem — stale or irrelevant memories resurface and degrade answers — and it stores user data, so don’t add it until the product needs it and you’ve thought about PII.
7 · Eval & observability
These aren’t in our index either — the field’s standard picks are Ragas and DeepEval for RAG-specific metrics (faithfulness, answer and context relevance) and Langfuse or Phoenix for tracing what actually happened inside a query. The decision isn’t which tool; it’s whether you’ll hold yourself to a labeled set of real questions with known-good answers, or tune on vibes. Default to building a small eval set — even a few dozen real questions — before you optimize anything, and wire in tracing so you can see whether a bad answer was a retrieval miss or a generation miss.
— this is the layer everyone skips and the one that separates a demo from a system — without it, every “improvement” is a guess.
The mistakes that sink RAG projects
- Launching on vibes. If you can’t point to a labeled eval set, you don’t know your assistant is good — you know it looked good the three times you tried it.
- Chunking too big. Oversized chunks are the most common quiet failure: the answer is technically “retrieved,” buried in hundreds of tokens of surrounding noise the model then talks past.
- Blaming the model. Most wrong answers are retrieval failures wearing a hallucination costume. Confirm the right chunk was even fetched before you touch the prompt.
- Skipping document parsing. Tables read across columns, scanned PDFs read as gibberish, headers merged into body text — bad parsing poisons every answer, and no downstream layer can undo it.
- Over-engineering early. Multi-agent orchestration, persistent memory, and graph RAG are answers to problems you should confirm you have. One well-retrieved, well-evaluated call beats an elaborate pipeline no one can debug.
- Agonizing over the vector database. It’s the most-debated choice and rarely the one that determines quality. Pick on operational fit and move on to chunking and retrieval, which actually move the needle.
Which should you pick?
- , limited engineering time → a batteries-included platform like Dify or AnythingLLM.
- (tables, scans, contracts) → RAGFlow for its deep parsing, or unstract inside a compose pipeline.
- , you need full control → compose on Haystack or LangChain with hybrid retrieval and reranking.
- , not just answers → GPT-Researcher or STORM, with deep-searcher for agentic search over private data.
- → Context7 as your documentation-retrieval layer.
Frequently asked
- Do I need a vector database?
- You need vector search; you don’t always need a separate vector-database product. On a platform, it’s handled for you. If you’re composing and already run Postgres, pgvector is often enough for a modest corpus — one fewer system to operate. Reach for a dedicated store (Qdrant, Weaviate, Milvus, or managed Pinecone — none of these are in our index; they’re the standard external options) when scale, hybrid search, or heavy metadata filtering demands it.
- LangChain vs Haystack for RAG?
- For a focused document assistant, Haystack’s pipeline model is cleaner and feels more production-ready out of the box, so many teams find it fights them less. LangChain has the larger ecosystem and more integrations, which pays off in a sprawling app that touches many services, but its abstraction is heavier. Prototype your hardest flow in both and keep the one that gets out of your way — the honest answer depends on your surrounding system, not a feature checklist.
- How much does a RAG stack cost to run?
- There’s no single number, but there are four cost drivers: embedding your documents (once, plus on updates), LLM generation per query (usually the largest and most variable line), optional reranker calls, and hosting for the vector store. Self-hosting open-source components trades dollars for engineering and ops time. Model your expected query volume against a specific model’s pricing before you commit — a stack that’s cheap at 100 queries a day can surprise you at 100,000.
- Self-host or managed?
- Managed — a platform, a managed vector store, hosted model APIs — gets you live faster and offloads operations. Self-hosting buys control, data residency, and lower marginal cost at scale, in exchange for engineering time and on-call responsibility. Start managed to learn what your data needs; self-host when compliance, cost-at-scale, or control genuinely demands it, not by default.
- Why does my RAG assistant give wrong or made-up answers?
- Usually retrieval, not the model. The right chunk wasn’t fetched — bad chunking, no reranking, a weak embedding — or the source was parsed badly to begin with. Add hybrid retrieval plus reranking, check how your documents are being parsed, and build a small eval set so you can localize the failure instead of guessing. If the correct context is in front of the model and it still answers wrong, then look at the prompt.