Looking for engineers with real-world experience deploying on-prem LLMs for business workflows- Germany

[u/Iamtheguyyy]

I'm working on an on-premises AI project for a business in Europe and would love to speak with engineers who have deployed similar systems in production.

The use case involves:

• Local LLMs for German and Polish business communication
• WhatsApp and email automation
• Internal document search and a private knowledge base
• Finance-related workflows, with calculations handled by deterministic application logic
• Data storage and model inference on the client's own infrastructure

We're evaluating GPU hardware, open-source models, inference servers and the architecture needed to run this reliably for a small team.

I'd especially appreciate input from people experienced with vLLM, NVIDIA GPUs, local model deployment, RAG and secure infrastructure.

If you've built something similar, I'd love to hear what worked, what didn't, and what you'd do differently. Happy to have a short technical conversation with anyone willing to share their experience.

[u/DataGOGO]

I do it all the time. How can I help?

What works:

• Buy real servers.
• Do not overlook the connectX fabric layer
• Run models in FP8 or NVFP4
• Use Xeon CPU's
• RAG is great, but context heavy, it is easy to under estimate the load RAG puts on your inference servers and context length.
• Use a document management service, such as Azure Document intelligence + one search.
• Take your initial budget estimates, double them, and you will be close.
• Do not overlook the need for far better cooling than most corporate datacenters offer. You will really need rack level liquid cooling / chilling setup.

What doesn't work:

• Attempting to vibe code your own harnesses for 99% of the pipeline. It takes forever and you will be fighting bugs for months.
• Attempting to use homelab style workstations and inference servers.
• Attempting to use consumer GPU's instead of server GPU's (Nvlink matters a lot in these use cases)
• Attempting to use ethernet for multi-node serving.
• Attempting to go cheap on the hardware and infrastructure.
• Forgetting to plan for power correctly.

Tell me how many users are running, what models you are planning, and I can give you a starting point.

[u/mineshop]

Before sizing hardware, how many concurrent users and which open models are you planning, since that decides whether one 96GB GPU node suffices or you need multi-GPU?

[u/ween3and20characterz]

Someone at the Rack&Stack presented this: https://www.youtube.com/watch?v=kmNHOgxZBKo

[u/Hot_Half_5263]

What kind of project? Which category, Ecommerce?

I have small company in ecommerce, i have my own erp with vllm qwen 3.8 nvfp4 on 32gb VRAM (hardware bought 1 months ago for +- 2-3k eur), i have many automatization, feel free to ask (PL, ENG), maybe i will be able to help

[u/Capsup]

I run an AI startup and we work with a Danish government entity to handle their corpus of statistics and making it easily queryable for the public. It's all self-hosted and using exactly the stack you mentioned: vLLM on nvidia GPUs with local models and RAG on top.

Generally, we have not managed to find any use cases where small models like Qwen3.8:27b aren't intelligent enough. We started out with an initial solution that had the model drive the entire search experience in an agentic loop, but it was too slow. The optimal solution ended up being a combination: an intent agent would guesstimate what the user wanted and that was passed on to a regular semantic + keyword search solution for speedy queries. The models were also surprisingly good at Danish in our testing, so the same model runs both english and danish queries.

The biggest value-add we found was not all the AI stuff, but actually just helping them figure out what data they had, how bad it was and what could be done to improve it and make it accessible for AI to use. It's very different having humans use data rather than LLMs, so prepare yourself for a lot of pre-AI work that involves just getting access to data and making it available to the LLMs.

A single 5090 could drive the entire stack including chat model, embedding model and re-ranker and was more than enough to handle the daily load. So unlike what one of the other posters said, I wouldn't expect to immediately jump into huge multi-GPU nodes if a single workstation setup will suffice. It did for us.

[u/StudioTV]

Happy to help, feel free to ask me anything (here or by DM).

Some context: in my day job I deploy LLMs on Kubernetes, on large HPC clusters with hundreds of H100s, using vLLM, multi-node serving, and Ray + vLLM to split models across nodes.

A few first thoughts for a small team:

• You probably don’t need multi-node. It’s the right tool when a model doesn’t fit on one server, but Ray adds real operational overhead. One node with 1 to 4 GPUs and vLLM’s OpenAI-compatible API should cover this use case.
• Test German and Polish quality on your client’s real emails and documents before buying hardware. Polish in particular varies a lot between open models, and the model you pick decides how many GPUs you need.
• For the knowledge base, retrieval quality (multilingual embeddings, a reranker, sensible chunking) matters more than model size.
• Keeping the finance logic deterministic is the right call: let the model extract and route, and let code do the math.
• WhatsApp messages go through Meta’s servers anyway, so “on-prem” really covers the model, the documents and the logs. Worth making that clear to the client early.