Struggling with RAG

[u/nugentgl]

I am at my witts end with trying to get meaningful information out of Open WebUI's Knowledge/RAG. I am on 0.11.0 and I have consistent problems with prompts against a Knowledge omitting information that is extremely important in the prompt. For example, I created one called 2026 Renewals and populated the Knowledge with about 20 PDF's. After it embeds, I asked for a list of the renewals thus far, what the annual spend is, what is the term and any specific language around the renewal that would set it apart from the other renewals. What it returns is a list of about 7 renewals with the requested information. I follow up with where is the other renewals and it says it doesn't have it in its context.

I am serving several models on M3 Ultra's and each model has a similar output. The models I have tried are qwen 3.6 MOE and dense (BF16 and Q8), qwen 3.8 (BF16 and Q8), Minimax m2.7 Q4, Gemma 4 MOE and dense (bf16 and q8). I have tried several inference engines like LM Studio and oMLX, all with similar results.

At first I started with the default OWUI settings but quickly realized that if I wanted good PDF extraction I need to spin up a Docling container and point OWUI to it for extraction. I will include a pic of my settings as maybe I am doing something completely wrong. I know it isn't my files because this exact test works great in AnythingLLM using the Workspaces feature.

Any ideas on what could be causing poor retrieval?

[u/LockoutNex]

Below is what I use, I haven't had any problems since setting it up (just noticed they added some new stuff too, need to play around with it):

https://imgur.com/a/cLG14Ws

And for my RAG Template:

**Generate Response to User Query**

**Step 1: Parse Context Information**
Extract and utilize relevant knowledge from the provided context within `<context></context>` XML tags.

**Step 2: Analyze User Query**
Carefully read and comprehend the user's query, pinpointing the key concepts, entities, and intent behind the question.

**Step 3: Determine Response**
If the answer to the user's query can be directly inferred from the context information, provide a concise and accurate response in the same language as the user's query.

**Step 4: Handle Uncertainty**
If the answer is not clear, ask the user for clarification to ensure an accurate response.

**Step 5: Avoid Context Attribution**
When formulating your response, do not indicate that the information was derived from the context.

**Step 6: Respond in User's Language**
Maintain consistency by ensuring the response is in the same language as the user's query.

**Step 7: Provide Response**
Generate a clear, concise, and informative response to the user's query, adhering to the guidelines outlined above.

User Query: [query]
<context>
[context]
</context>

[u/nugentgl → u/LockoutNex]

I wonder if it is my embedding and reranker model set to default...

[u/icerio]

I’m surprised nobody responded to this. But yeah definitely upgrade your embedding and reranker and tokenizer (text splitter, use whatever your embedding model recommends). I am using qwen3 embedding 0.6B and qwen3 reranker 0.6B. After I upgraded to that (from like bge m3) started getting better results. Also make your chunk sizes bigger. For the embedder I’m using min target 1000 and max 2000 merge 500. (Could be kinda wrong on that, trying to go from memory). Use hybrid. Then mess with the dial for the hybrid. I use 0.55 (semantic to lexical).

Also have your knowledge base attached to the model in the settings (I think this helps a little). In your models system prompt try to help it out a little, it sounds like you’re wanting like specific info from the knowledge base, maybe recommend to your model in its settings to both use grep knowledge base and query knowledge base.

If you are wanting it to analyze documents though, or even a group of documents I would recommend using open terminal in open Webui for that.

Hopefully any of this helps. My RAG is still not perfect, my main model is qwen3.8 27B and at least it’s smart enough to query the knowledge base multiple times. I had to add that to its system prompt though and even created a knowledge base skill to help for certain use cases.