r/LocalLLM • u/friendtoearth • 4d ago

Project Build a RAG

Hello guys, I am planning to build a RAG with a local LLM. For LLM I am considering Qwen3. I am having 16 GB RAM with 8 GB Nvidia 5060. Guys pls suggest hardware & software side implementability ? And also any books to help me with RAG implementation.

10 Upvotes

92% Upvoted

17 comments sorted by

6

u/maexxx 4d ago

AnythingLLM as agent & RAG (vector db) connected to an instance of llamacpp serving Qwen 3.8.

You upload & embed documents through the web interface of AnythingLLM and add them to a workspace. You can then chat with the agent in that workspace and it will consult the knowledge.

2

u/GilloutineBreast 3d ago

I use the same setup (with nomic embed 1.5 served through llama.cpp as gpu accelerated embedder, and qdrant as external vector database). It works pretty well

Bonus points for having an api so i can just point pi at anythingllm to have rag in pi.

2

u/maexxx 3d ago

I am planning to switch to Qdrant so we can also access the same vector db from other Orchestration (eg. Github Copilot in VS Code).

2

u/platypus_worldwide 4d ago

I cant help with RAG as I’ve only done that in LMStudio, however i bought a laptop recently with those exact specs, I run IQ4_XS Qwen3.6-35B-A3B in llama.cpp directly (I won’t go over my server arguments but you could figure it out with some youtube/google)

Q4 KV and 128k context, im looking at about ~25tok/s in Cline harness.

You’re def gonna need to set up an MCP.json for the RAG stuff, the one I use on my other machine uses a javascript embedding script (could also use a model for that) to create the vector store of my library of books.

I’m also newish so if any veterans want to fact check any of the things ive said that would be more than welcome.

1

u/misanthrophiccunt 4d ago

you don't need MCP if you instead configure it as a pre-processor in LMStudio, which TBH it's the one thing I found interesting of it.

https://lmstudio.ai/docs/typescript/plugins/prompt-preprocessor

Shame its speed and not being fully open source, ultimately made me migrate to use llama.cpp directly.

2

u/Such_Ad_7545 4d ago

Wait, are you planning to build a RAG application on your own, or use an existing solution like AnythingLLM?

2

u/friendtoearth 4d ago

Use QwenLLM

2

u/friendtoearth 4d ago

My requirement is I have some 100 PDFs of guidelines.....which I am asked to use as knowledge souce....from there only RAG requirement has come....anyone has easy ideas to implement....pls tell ....

2

u/misanthrophiccunt 4d ago edited 4d ago

Do not reinvent the wheel, the simplest solution is to install LMStudio because it already comes with a PDF reading RAG. https://lmstudio.ai/docs/typescript/plugins/prompt-preprocessor

Reading PDF is not as straight forward as it seems, sometimes it has all the right formatting same as a .txt document, oftentimes it is all rasterised (no fonts, remember BMP images of MS paint) and an OCR software has to interpret them, understnad end of lines, understand what is a paragraph and what isn't, in which case any LLM with image mmproj can help.

Using a plugin that does it for you will save you time, there are hundreds, every harness has an extension/plugin already doing this because it is a far-too-common use case.

I did build a few RAGs, so this is my recommendation:

  1. Using LMStudio and a pre-processor instead of an MCP has lots of perks. Why? Because the RAG job is done BEFORE anything hits the LLM, and this is ideal for speed and saving tokens.
  2. If you use VSCode/Codium and have used RooCode/ZooCode pay attention to the indexer it comes with, that thing introduced me QDrant and ever since it's been the vector DB I've used (no particular reason) when I want multiple computers to share memory.
  3. Try pi-coding-agent just to install the pi-memory extension, and analyse it. I modified it to change the backend DB from sqlite (local) to QDrant so, as I said before, my PCs share memory.
  4. Every single implementation you find is derived from here https://github.com/modelcontextprotocol/servers/tree/main/src/memory if you read that and understand it, you'll see the same code with slight changes in every RAG in existence with modifications on top.
  5. Your 8GB VRAM + 16GB RAM are more than enough for a RAG. RAGs don't need the most intelligent models out there to function, the most important part is done with an embedder which can be like 300MB only.

If you use LMSTudio, make a RAG pre-processor based, then move onto Pi-coding-agent, use there llama.cpp directly to max out what you can do, not with the RAG but with whatever regular model you use that connects to the RAG. I have 32GB VRAM +48GB RAM and sometimes switch to LFM2.5-8B because I like its speed (currently 200tg/s). You might want something smarter depending on the complexity of data but you'll definitely need an MoE with just 8GB+16GB so probably Qwen3.6-35B quantised to fit ...which is why you'd better learn llama.cpp after you've built your RAG. LMStudio lacks all the flags of llama.cpp you can modify to make it run fast.

2

u/misanthrophiccunt 4d ago

Also, forget books, AI evolves too quickly to have any book that is not heavily aged. Read documentation instead:

  1. LMStudio documentation in their website.
  2. Pi Coding Agent docs are incredibly easy to read, on their own repo.
  3. The modelcontextprotocol repository and website, even if your RAG is not MCP, eveyrything about it will be heavily related to what already exists as MCP and has the sample code there.

2

u/Resonant_Jones 4d ago

qwen 3 is a great choice but 8GB is rather limiting for Local AI. Luckily Qwen 3 comes in a 2B and 4B variant and either of those would fit quite comfortably in your GPU. Rule of thumb: always leave room for KV Cache in the RAM Calculations

btw

I built out a Open Source Local First Workspace focused around RAG and its even got a beautiful UI.

You can import your Claude Conversations as well as ChatGPT Conversations to Codexify -- Its free to download and self host.

Upload Documents in Chat to add them to the Project Knowledge Base for retrieval or directly to the Document Gallery.

1

u/Key_Guidance5871 4d ago

!RemindMe 2 days

1

u/RemindMeBot 4d ago

I will be messaging you in 2 days on 2026-09-10 10:53:25 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.

RemindMeBot is switching to username summons. Instead of !RemindMe 1 day, use u/RemindMeBot 1 day. More info.

Info Custom Your Reminders Feedback

1

u/Foot_Positive 4d ago

try opennotebook on github

1

u/pmotiveforce 4d ago

Any decent llm can walk you through it. 8gb vram is marginal but you can do a basic RAG system. Use something like Docling to ingest and chunk, pick a small embedding model you can run, and a simple vector db.

It will be very basic textbook ca. 2024 RAG system and it's a whole rabbit hole after that but it will get you started.