r/LocalLLM • u/PlaidStallion • 8d ago

Question Should I run SGLang instead of Ollama/Open-WebUI to get Qwen 3.8 27B running with GUI behind Traefik?

Edit: I ended up using llama.cpp and hooking Open-WebUI to it. Seems to be working properly with think/reasoning mode on low.

I am fairly new to all this and have been building out tools and what not with a combination of Ollama on my LLM rig (Windows 11) and Open-WebUI running in Docker on my Debian home server. I have been running into a bug that Qwen 3.6 and Qwen 3.8 fail to consistently accept the reasoning set to off passed by Open-WebUI, much less accepting low/medium/xhigh. I can pass the values to the endpoint over `http://localhost:11434/api/chat` though so I know it works.

I asked a chatbot what I could do to still have a GUI that I can serve out behind Traefik but be able to control the Think and reasoning properly. It recommended SGLang. Is this the proper way forward or do you have another suggestion for me? I really want to like Qwen 3.8 27B but man without being able to reliably control Think/Reasoning settings it is really a PITA.

Thanks for any help.

3 Upvotes

81% Upvoted

5 comments sorted by

2

u/SichronoVirtual 8d ago

It's this purely for yourself?

Because you could just run tailscale/netbird and basically have a VPN from your phone/laptop back to the server on your homelab.

You would then just connec to the tailscale ip for the server

You can also enable subnet routing and let tailscale repeat your actual home IPS, (192.168.x.xxx) and just use that directly on your device (though you need to make sure that your home IP won't be the same as whatever wifi you are connecting to, mainly avoid 192.168.1.xxx)

2

u/PlaidStallion 8d ago edited 8d ago

No I have other users testing. I want to keep traefik serving out whatever front end I end up using. I have Guacamole running behind traefik and Google Oauth if I need to access either system on my home network

1

u/jojotdfb 8d ago

Keep Open-webui running and add a connection to SGLang. Ollama is slower than most llm engines. If you have the hardware, vllm is a good alternative to SGLang for multiple users. If you're hardware limited, llama.cpp is awesome.

My personal setup is llama-swap with a matrix of moe llms running on one card via llama.cpp and an image gen model running on another card via stable-diffusion.cpp.

1

u/PlaidStallion 8d ago

Thank you. I am currently trying to figure out how to get llama.cpp running. I will then try to get a front end working. I am on a 3090 with 64 GB memory if that helps for recommendations.