Baza znanja

Retrieval-Augmented Generation (RAG) combines a language model with a search step over your own documents, letting the model answer questions using specific, private, or up-to-date information it wasn't originally trained on. This guide covers building a basic RAG pipeline on a VPS.

How RAG Works

1. Your documents are split into chunks and converted into vector embeddings
2. Embeddings are stored in a vector database
3. When a user asks a question, it's also converted to an embedding and used to search for the most relevant document chunks
4. Those relevant chunks are included in the prompt sent to the language model, which generates an answer grounded in that specific content

Prerequisites

• Ollama installed and running (see How to Install Ollama and Run Local LLMs on a VPS)
• Qdrant installed and running (see How to Set Up a Vector Database (Qdrant) for AI Applications)
• Python installed

Step 1 — Install Required Python Libraries

pip install langchain langchain-community sentence-transformers qdrant-client --break-system-packages

Step 2 — Load and Split Your Documents

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.document_loaders import TextLoader

loader = TextLoader("my_document.txt")
documents = loader.load()

splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(documents)

Step 3 — Generate Embeddings and Store in Qdrant

from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Qdrant

embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")

vectorstore = Qdrant.from_documents(
    chunks,
    embeddings,
    url="http://localhost:6333",
    collection_name="my_documents"
)

Step 4 — Retrieve Relevant Chunks for a Query

query = "What does this document say about pricing?"
results = vectorstore.similarity_search(query, k=3)

Step 5 — Build the Final Prompt and Generate an Answer

import requests

context = "\n\n".join([doc.page_content for doc in results])
prompt = f"Answer based only on this context:\n{context}\n\nQuestion: {query}"

response = requests.post("http://localhost:11434/api/generate", json={
    "model": "llama3.2",
    "prompt": prompt,
    "stream": False
})
print(response.json()["response"])

Improving Answer Quality

• Experiment with chunk size — too small loses context, too large dilutes relevance
• Instruct the model explicitly to say "I don't know" if the retrieved context doesn't contain the answer, reducing fabricated responses
• Retrieve more chunks (k) for complex questions, fewer for simple ones

Wrapping This in an API

See How to Serve a Machine Learning Model with FastAPI to expose this pipeline as a proper HTTP API for use by a frontend application.

Common Errors

Model gives generic answers ignoring the provided context — make the prompt's instruction to use the context more explicit and directive; smaller models sometimes need more forceful prompting.

Retrieval returns irrelevant chunks — verify the embedding model used for querying matches the one used when originally storing documents; a mismatch produces meaningless similarity scores.

Continue Reading

• How to Set Up a Vector Database (Qdrant) for AI Applications
• How to Install Ollama and Run Local LLMs on a VPS
• How to Serve a Machine Learning Model with FastAPI

Browse more articles in AI & Machine Learning on a VPS.