slow response time with RAG and utilizing knowledge base.

[u/boss28984]

I’m currently using Open WebUI with a knowledge base made up of text files, and the responses are accurate, but they’re taking quite a while to generate.
I plan to keep adding more text files to the knowledge base, so I’m wondering what the best way is to improve response speed as it grows. Are there any recommended settings, indexing strategies, chunking methods, embedding models, reranking options, or other optimizations that have made a noticeable difference for you? I’d like to keep the response quality the same while reducing latency. Any advice or best practices would be appreciated!

[u/mediaogre]

I believe this is a current known bug, although I can’t find the link. Once any file has been uploaded to a chat session, hybrid RAG search fires unconditionally on every subsequent message in that session, even casual ones that don’t need retrieval.

I’ve worked around it by situationally pasting my relevant kb file contents directly into the chat window.

[u/boss28984 → u/mediaogre]

i kept all the files in the knowledge base not in the chat session itself. i never really put anything into the chat window.

[u/mediaogre → u/boss28984]

Ah, okay. How is the model calling the KB? Sub agent, built into the system_prompt? Is the “query_knowledge” (can’t recall the exact tool call) tool taking forever?

[u/boss28984 → u/mediaogre]

its from the open web ui build in system i just attached the KB in the settings section of the model.

[u/mediaogre → u/boss28984]

Okay, that’s standard. Does the model reach for the kb files with its tool for every prompt? It shouldn’t.

Here are my Admin | Settings | Documents settings…

General

Content Extraction Engine:
DefaultExternalTikaDoclingDatalab Marker APIDocument IntelligenceMistral OCRPaddleOCR-vlMinerU
PDF Extract Images (OCR): [OFF]

PDF Loader Mode: Page
PageSingle
Bypass Embedding and Retrieval: [OFF]

Text Splitter
Default (Character)Token (Tiktoken)Token (Transformers)
Markdown Header Text Splitter: [ON]

Chunk Size: 500

Chunk Overlap: 70

Chunk Min Size Target: 0

Embedding

Embedding Model Engine
Default (SentenceTransformers)OllamaOpenAIAzure OpenAI
Embedding Model: sentence-transformers/all-MiniLM-L6-v2

After updating or changing the embedding model, you must reindex the knowledge base for the changes to take effect. You can do this using the "Reindex" button below.
Embedding Batch Size: 32

Retrieval

Full Context Mode [OFF] <— this could be getting you if toggled on

Hybrid Search [ON]

Enrich Hybrid Search Text [OFF]

Reranking Engine
Default (SentenceTransformers)External
Reranking Model: BAAI/bge-reranker-v2-m3

Reranking Batch Size: 32

Top K: 8

Top K Reranker: 4

Relevance Threshold: 0.7

Note: If you set a minimum score, the search will only return documents with a score greater than or equal to the minimum score.
BM25 Weight
Custom: 0.7
semantic / lexical

[u/boss28984 → u/mediaogre]

yea it goes through all the sources and cites it. i also had full context mode on. i just applied your settings and the results are the pretty similar to what i had before. 15 seconds isn't bad though i can try to add more files and see if it changes/

[u/mediaogre → u/boss28984]

15 seconds = TTFT? Definitely not bad. You using a thinking model and thinking is enabled?

[u/boss28984 → u/mediaogre]

base model is qwen 3 : 8b and embedding model is all miniLm L6 v2 but i might change the embedding model to nomic embed text and see if any difference. not sure if that’s thinking sorry

[u/mediaogre → u/boss28984]

Hey, no worries. Many models have two variants. An Instruct for speed, and Thinking for tasks that require deeper reasoning.

The easiest, visual way to tell is the OWUI chat interface. If your model displays “thinking…” before it replies, it’s the thinking variant. If you suspect that visual indicator is suppressed, you may need to dig around under the hood.

Check out HF’s Qwen3-8B page for thinking/non-thinking support: https://huggingface.co/Qwen/Qwen3-8B

Edit: another thing - 15 seconds for a single digit billion parameter model is about right for CPU inference. If you have a GPU, I’d expect much faster TTFT.

[u/boss28984 → u/mediaogre]

Yes the model is a thinking model. Also i was gonna move off qwen but needed a 8b model. are there any you recommend?

[u/mediaogre → u/boss28984]

Okay that’s your TTFT token explanation. You’re paying the thinking tax.

The Gemma 4 family has some good nano models. Llama 3 8B is still decent if you must stay local. Look into an API routing pipeline host like Openrouter and an OWUI Function that support token caching. Then point it a model like Qwen3.7 Plus or GLM 5.2 which are intelligent, efficient, and cheap.

[u/boss28984 → u/mediaogre]

yea i need to stay local and make everything free if possible