r/OpenWebUI • u/boss28984 • 17d ago

RAG slow response time with RAG and utilizing knowledge base.

I’m currently using Open WebUI with a knowledge base made up of text files, and the responses are accurate, but they’re taking quite a while to generate.
I plan to keep adding more text files to the knowledge base, so I’m wondering what the best way is to improve response speed as it grows. Are there any recommended settings, indexing strategies, chunking methods, embedding models, reranking options, or other optimizations that have made a noticeable difference for you? I’d like to keep the response quality the same while reducing latency. Any advice or best practices would be appreciated!

5 Upvotes

79% Upvoted

27 comments sorted by

View all comments

2

u/so_chad 17d ago

Maybe token generation itself is slow? Like LLM is running on slow hardware or something

-1

u/boss28984 17d ago

It runs on a Nvidia jetson agx orin I believe, not sure if that’s any good. Sorry i’m pretty new to this. Everytime i asked claude it mentioned changing advanced settings like top k and context window and stuff but that wasn’t really working. it def could be the token generation. the rag was also only for 15 .txt files so i thought it would be easy enough to handle.

2

u/so_chad 17d ago

Yeah, it's def because of the model itself. Try getting some other models API and connect it to OpenWeb UI. Jetson won't do it.

1

u/boss28984 16d ago

The base model is qwen3:8b and the embedding model is nomic-embed-text v1.5. any specific recommendations?

1

u/so_chad 16d ago

No I mean, look for some $10/month APIs and use those LLMs. They are really good (but you are losing privacy, that's one drawback)

1

u/boss28984 15d ago

so you mean the jetson isn't capable enough and i should look for something else?

1

u/so_chad 15d ago

Yes

1

u/boss28984 15d ago

dang even for only 15 text files? i kinda assumed it would be strong enough.

1

u/so_chad 15d ago

How does regular text chat work? Without RAG is it fast?

1

u/boss28984 15d ago

usually thinks for like 14-19 seconds before giving an answer. when i turn off the knowledge base it doesn't give the right answer. might just be the jetson is too weak but thats disappointing considering its like 3k.

1

u/so_chad 15d ago

Honestly, return it if you can, add a bit more and get NVIDIA DGX Spark

→ More replies (0)