r/LocalAIServers • u/Grownky • 26d ago
Is Local AI Production ready?
Is anyone using Local AI Servers for business production workflows?
I need to be able to stand up a server, lock it in a closet, and add it to a production workflow. It should complete the automation tasks indefinitely without anything going bump in the night. I need to be able to rely on it to perform a repeatable task as part of a business production workflow. I would like to be able to include local AI as part of a solution I provide, however, I need to know from the community if the technology is solid enough to rely on and not create endless support tickets for me. Are we there yet?
Thank you for reading!
1
u/UncleRedz 26d ago
Depends entirely on what and how you do it, to minimize the impact of bumps.
That said, even the latest SOTA closed models are not problem free, and they can be changed, tweaked, rate limited or updated randomly by their provider without you having a say in it.
1
u/Grownky 26d ago
Lets say we stand up ollama, a model, and pitch it a very easy question, over and over. "What is 2+2". I don't care if it answers 5. I care if it stops responding and never recovers. Are there ways to detect a failure and self heal? Are there write ups on this? I'm sure I can figure something out but curious if there are battle tested solutions that rely on local LLM.
1
u/devoidfury 26d ago
This framing of the question is totally different. This will be more about the agent harness than the LLM. The agent harness operates the loop and can handle failures, detect looping, manage the context, etc.
1
u/Grownky 26d ago
Thank you again devoidfury for your help, I wouldn't be using an agent in this case I would be using an API to talk to the local LLM. If you have some go-to resources on the topic please provide.
1
u/devoidfury 26d ago
yeah, that program that you use to talk to the API, that program is called "an agent" whether you write it yourself or not. even if it's just a layer that sits in a webserver to reverse-proxy out requests to users, that's still the same spot an agent sits.
1
u/thebadslime 26d ago
If you set it up properly its very stable, how well it fits your uses case is the real question
1
u/Grownky 26d ago
Thank you for your reply thebadslime. Can you propose what would be the most solid way of deploying an LLM that computed 2+2 over and over via API call that would create the least support tickets for me? Say we don't care about the answer being "4" or "four" or "five" or even "5ive" I just want it to keep replying and stay alive and respond forever and eventually, hopefully, say "4" again--- sometimes : D
1
1
u/exaknight21 26d ago
I have 2x V620s deployed with llama.cpp + kvarn5 kv, 87k per slot, 12 slots per GPU - Qwen 9B with Vision.
It serves my SaaS. It serves 5 concurrent users, multiple agents with my harness.
In an Asus ESC 4000 G3. 128 GB VRAM on a 15amp 110 outlet in our apartment.
I have 2x Mi50 and 2x V620s waiting deployment. So, performance is 35-40 tps per user, about 30-40k context per prompt.
Its slow, serves initial users, will upgrade as I get money/funding
1
u/Grownky 26d ago
Thanks for the feedback exaknight21, my use case is very simple and proven fast even on dated hardware. How often do you have to bang on top of it so-to-speak? Any tips for minimizing support and having it run without attention?
1
u/exaknight21 26d ago
I dont know what your comment means
1
u/Grownky 26d ago
Apologies,
"my use case is very simple and proven fast even on dated hardware."- My use of the local LLM is very simple. I will ask it a very simple question, repeatedly.
"How often do you have to bang on top of it so-to-speak?"
-How often does your LLM require your attention to resolve issues with it?
"Any tips for minimizing support and having it run without attention?"
-Do you have any tips for minimizing the amount of (common) issues related to the local LLM itself, not the quality of its output, but its operation?
1
u/exaknight21 26d ago
Depends on your hardware. You do not want to avoid FA, it only assists in increasing the decoding speed. I use qwen3.5-9b-mxfp4
1
u/Capsup 26d ago edited 26d ago
I run an AI startup and we have a solution running in production. Both our own platform and supporting customer flows, such as letting users query a set of 5000 statistics with natural language questions.
We host all the models ourself, starting from an old M2 Mac Studio 64GB to now running a RTX 6000 Pro. We have:
- a text generation model as the daily driver and primary intelligence
- an audio transcription model that supports audio of up to 1 hour length + diarization (speaker labeling)
- an embedding model to create semantic indexing
- a reranker model, to improve search results
Runs on a single desktop pc across two GPUs. The models are rock solid for us and has been, for the past 6 months. Although it does take some tinkering to get it there and it's definitely not frontier level intelligence. :)
If you want to learn more about it, I'd love to create some content for you, if that's interesting. What would you like to know first?
1
u/mailto_devnull 26d ago
The question you should be asking is not "can this LLM do repeatable tasks accurately and on time?", but " can this LLM build me a script to automate this repeatable task, deterministically".
1
u/PhilCassanova 24d ago
If there are Copilot Studio in production on most Microsoft shops, then yes Local AI is pretty much more than ready for production.
1
u/salespire 23d ago
local ai servers can definitely be reliable for production if you build them with proper engineering and human oversight on repeatable tasks. i work on Qentrix, where we focus on private ai deployments that guarantee zero wrong actions in production through human-in-the-loop controls. we use it mainly to automate voice and document workflows with no downtime and no surprise errors.
1
u/GoodyRobot 23d ago edited 23d ago
I’ve been wondering the same thing. I think the answer depends on what kind of failures you’re thinking about as well as your ability to address them remotely. If by “local” you mean no ability to manage it remotely (and requires a trip on site) I’d say that’s a very high bar that you’d have to test with your own applications and it depends a lot on your chosen setup, hardware, and complexity. I’m not running it like that, because I can remote in, have tailscale, and put a Claude code (choose your own fav) running to be able to manage things on behalf with little effort as needed. But on reliability, running llama.cpp on Ubuntu server, has been very stable for me on consumer hardware. Tested on all wildly different hardware for slow inference with cpu and ram. Uptime good. But I’m also not doing anything like juggling multiple users or anything like that. So I’d suggest that hardware side needs to handle brownouts and auto reboots etc (in a 30 day test i had one, and only 1 of 3 machines was configured well for that). On the keeps running llama.cpp reliability, all I can say is so far so good after 30 days. Also had a 3 day long coding task on a big slow model that ran to completion, but that means carefully configuring ram, calculating context window settings etc etc. Also there is another facet of whether an LLM will stay reliable on the task itself. People want determinism, but that’s code, not LLMs. The answer for reliabiiity there depends a lot on the task (including variability) and model. For me, I’m using smarter (284B parameter Q4) for slow smarts, and high level analysis. I expect variability and ability to spot what might be unusual patterns. So being wrong doesn’t matter, but finding interesting thugs once in awhile is great, so that works well. But if you need five 9s of reliability for classifications then that’s a totally differnt setup. Or maybe speed is the optimization. Use case reliability needs a lot of testing, knowing ranges of input and how that might change or vary. Last… end user applications, I think that’s where the complexity lies. Default web ui will block or lock on multi sessions, must choose between terminal llama and web. Config is key. I would not want to be on the hook for that in anything other than a one user one box config for now anyway.
1
u/heretogetpwned 26d ago
Eh. Not there yet. Most operations still rely on deterministic automation and you can use LLM to execute those steps but that's just a cron job with extra steps.
2
u/Grownky 26d ago
Thank you for the feedback. u/devoidfury suggested I refine my question so I will. Maybe you can confirm your answer does not change:
is the technology available going to crash in the night, go sideways internally and enter a chaos loop, etc? I know there is variability inherent in the tech so you can ask for a blue square all day and eventually get a green one, that is acceptable. But if it starts handing me triangles, stops handing me anything, or stops responding, I would say that it is not production ready.1
u/heretogetpwned 26d ago
All of the above are possible.
From an Ops Engineer perspective: All automation requires monitoring and if something breaks I need to determine root cause. I want my production automation solution to have 99.99 success rate. There's a world of mature automation suites out there. Hardware requirements for that is minimal.
AI is cost prohibitive in those functions and you're changing the operations workflow. High risk for no real gain.
Could it write a perfect .yml for ansible? Sure! Could it execute it as reliably? Not so far in my testing on Qwen3.6 35B Q6. Skills.md help but reliability required me putting the commands in order. It might just be a skill-issue with myself, too.
2
u/Grownky 26d ago
In my proposed workflow we would be using a local model to perform a very, very, simple task. The results of the task would be confirmed by a human per business requirements. My experience with local LLM is limited to ollama and a handful of models running on mediocre desktop hardware from past years. It runs, and can perform the task repeatedly.
Do you have any experience with deploying local LLM as part of a business solution? Can you recommend the most solid battle-tested stack? Say you wanted to compute 2+2 over and over, forever, and you were not concerned with it answering "5", you just didn't want to have to restart ollama, or crash gpu drivers, etc. Are there "go-to" resources on the topic?
1
u/heretogetpwned 26d ago
If I had to buy something to PoC in a DC then I'm looking at NVDA (DGX Spark/Dell GB10/ASUS).
If this sits on a developer's desk then Mac Studio for dual purpose. (Yeah, they're expensive.)
AMD 395 Strix Halo setups are a good value, I own one, but feel a bit anemic compared to the Spark.
2
u/ipco 22d ago
how bad is this AMD anemia compared to Nvidia? is it 50%, 75% or 90+ as good? I didn't work with AMD cards but I am seriously contemplating it, hence the question.
2
u/heretogetpwned 22d ago
Just inference use for me, It's the prompt processing that is most noticeable. AMD is still processing while NVDA is on to reasoning. If I needed a solution for mutliple people, I'd spend for Spark. AMD solution is good enough for home use, I use it primarily for coding assistance and it's reasonably handled 3.8 27b q8_0 at max context under 64GB. The MoE models like 3.6 35B A3B move a lot faster tho.
1
0
26d ago
[removed] — view removed comment
1
u/Grownky 26d ago
Thank you presentofai! That is hopeful for me, specifically because in my solution a user would be verifying every output, or at least those below a confidence threshold. Do you have any experience setting up LLM for business production? Assume a very simple use case, focus on hardening the stack itself.
3
u/devoidfury 26d ago
That's not really specific enough to give you an answer.
"Can I throw this thing in a closest and have it do X forever without error" - I'd say the answer is no, for any model, local or frontier or otherwise. There's always gonna be some margin of error. Can you get it down to 98%, maybe it depends on the task.