ChatGPT-Style Retrieval-Augmented Generation (RAG) with LangChain

A LangChain RAG chain has four pieces with distinct responsibilities: a document loader + splitter that turns a policy PDF or markdown file into chunks small enough to embed meaningfully; an embedding model that converts each chunk (and later, each query) into a vector; a vector store that indexes those vectors for fast similarity search; and a chat model that receives the retrieved chunks plus the question and composes an answer.

The critical constraint that trips up first attempts: the embedding model used to index documents and the one used to embed the query at request time must be the same model. Mixing embedding models (or even different versions of the same model) puts query and document vectors in incompatible spaces — cosine similarity between them becomes meaningless, and retrieval silently returns unrelated chunks with no error.

Split the refund policy into chunks before embedding — a whole multi-page document embedded as one vector loses the granularity to distinguish "damaged item" refunds from "changed my mind" refunds, because one vector can only represent the average of everything in it. A few hundred tokens per chunk, with modest overlap, keeps each chunk focused enough that its embedding actually represents one coherent idea.

Source metadata attached at chunking time (which document, which section) is what lets a later answer cite where it came from — that citation is the second half of the trust story: not just "answer grounded in the docs," but "answer traceable to a specific paragraph a human can go check."

The retrieval step embeds the incoming question with the same embedding model and finds the top-k most similar chunks. The generation step then "stuffs" those chunks into the prompt alongside the question and asks the chat model to answer using only that context. The instruction to use only the provided context is the entire mechanism that keeps the bot from quietly falling back to its general training when the retrieved chunks don't actually answer the question.

Without that instruction, a question outside the document's scope ("what's your policy on international shipping duties" when the refund policy never mentions duties) gets a fluent, confident, and completely made-up answer — the model fills the gap from general knowledge because nothing told it not to. The fix is a system prompt that explicitly permits "I don't know" as a valid answer.

RAG earns its complexity when the answer source changes independently of any model release — a refund policy edited next week, a product catalog updated daily — and when you need traceable citations a compliance or support team can verify. It's the wrong tool when the knowledge is small enough to fit permanently in a system prompt, or when the task needs reasoning over a huge document all at once rather than a handful of relevant chunks.

That second case is where long-context prompting can outperform chunked retrieval — feeding the model the whole document instead of a handful of matching passages. Treat the choice as a workload question, not a loyalty to one architecture: measure retrieval hit rate and answer faithfulness on your own labeled question set before assuming the more complex pipeline is earning its cost.