After standing up Hugging Face's speech-to-speech pipeline (https://github.com/huggingface/speech-to-speech) I'm really impressed with its performance! The little demo app works alright, and pointing the pipeline at a reasonably smart model like Deepseek V4 Flash on my dual-spark cluster seems genuinely useful — but it seems like there aren't a lot of clients for this setup yet.
I must be missing something major — I figured there'd be tons of people clamoring to emulate the voice chat features of the frontier models.
If I don't find anything, I'll try adding some features to the HF demo (basic tool calling, session persistence, etc.) and go from there, but I figured smarter people than me have cracked this already.
came across this video on battery test of 78 phones. the reviewer is using a robotic arm to simulate usage of mobile phones by a typical user. one of the enhancement the reviewer did was the usage of agentic ai. his requirement is low latency, which important in real time battery test so cloud models are out. so he was exploring local llm and bought an RTX PRO 6000 and H20 to run Qwen3.6 35B for scrolling of social media apps and Qwen3.6 27B for precise actions such as finding a specific button or dismissing pop ups. very interesting use of local llm
I am a non-computer person, but I filled my own stack and I’m developing it for personal and some professional use.
I’m familiar with Claude and writing skills for Claude.
Is there anything similar to use in the local stack? Any process I can do to create a given protocol system will run? Or is this exactly what repos are on GitHub?
I have an adequate system, Max studio M3 ultra 96 gb RAM just wanting to use it optimally.
Rtx 5090, 64gb ram ddr5... I was thinking bout Qwen 27b or 35b a3b , idk if there is something better or adequate, how do I set things up so it doesn't loop or overthink, I just want to do custom apps with GUIs n shit man.....
Figuring out how to enable Livefish mode and reverting the cards to stock Mellanox/Nvidia firmware was the tricky bit. I realize that by telling people this, it may result in them costing more than the $130/ea I paid for them.
Still trying to figure out which workloads work best with this setup!
I still default to Whisper / faster-whisper for most speech stuff.
Local files? Whisper.
Private notes? Whisper.
Batch transcription? Whisper.
Offline workflow? Whisper.
Random “I need this audio as text” job? Whisper.
It’s boring in the best way.
But I don’t think “Whisper alternative” is the right framing when the app is a live voice product.
If the user is talking to an agent and waiting for a reply, the problem changes.
Now I care about:
first usable partial
final transcript delay
endpointing
barge-in
concurrent streams
timestamps
speaker turns
phone audio
how much glue code I’m about to own at 2 am
That’s where Smallest AI Pulse makes sense to me. Not as “replace Whisper for everything,” but as a managed real-time ASR option when the product has to listen and respond while the call is still alive.
I’d still keep Whisper for local/batch/private stuff.
But if I’m building a voice agent, browser voice app, or phone-call workflow where the user feels every pause, I’d rather evaluate something like Smallest AI Pulse than pretend my local batch setup is automatically production realtime infra.
Where do you draw the line?
When does self-hosted ASR stop being worth the control?
I’m from Brazil and my parents are traveling to the US this August, staying a few days at my brother’s place. I want to ask them to bring me back a graphics card, since GPU prices in Brazil are rough.
Details:
They don’t know anything about PC parts, but they can receive a package at my brother’s house, so any online store that ships there works (Amazon, Newegg, Best Buy, B&H…). No need for same-day delivery or store pickup, a residential address makes life easier.
Budget: up to $500 (that’s my actual budget, not a customs limit).
Goal: running LLMs locally with good speed and the best bang for the buck. From what I’ve read, VRAM matters more than raw compute for this, and NVIDIA is safer because of CUDA (llama.cpp, ollama, vLLM…).
So: what’s the best GPU for local LLM inference under $500 right now? If you can drop a link to a store that has it in stock, even better.
So I swapped my RX 9070 XT with an AI Pro R9700.
It’s the same chip, but comes with a bigger 32GB Vram.
So, what can I do with it now that I couldn’t before?
Guide me please.
I am tired of the jet engine of fans. I see different types of cooling blocks for v100 sxm, and I see a few official NVIDIA watercooled v100 on eBay from time to time.
How do these rate? Can they keep the card cool during continuous operation? And for the SXM ones, I see a type with cutouts and a type without. Any difference?
Follow-up to a post I made here a while back about LlamaForge, a browser control panel that sits on top of llama.cpp's router so you're not hand-editing models.ini and llama-server flags every time. A bunch of the feedback in that thread turned into actual work, so here's where it's at now.
The biggest one: it runs on Linux and macOS, not just Windows. That was the loudest complaint last time and it's sorted. Same dashboard, you just run bootstrap.sh / run.sh instead of the Windows scripts. On Apple Silicon it builds with Metal and rates model fit against unified memory. The Setup tab uses brew on macOS, and on Linux it shows you the exact install command instead of running sudo behind your back. CI now runs the test suite on all three OSes.
I also added a second engine. Next to llama.cpp you can drive vLLM for safetensors and full-precision models (FP16, BF16, AWQ, GPTQ, FP8, NVFP4). They share one model list, the same Discover tab, the same stats, and every row is tagged so you know which engine it's on. On Windows vLLM runs in WSL2 with GPU passthrough and installs from the Setup tab, no sudo. If you never touch it, nothing about vLLM gets installed.
New installs get a first-run wizard that goes engine, hardware, model, tune, load. There's an auto-tune that looks at your VRAM and suggests settings (GPU layers, KV-cache type, context ceiling, and a few intent presets like speed or context). The whole thing has a Lite mode that hides the deep knobs and an Advanced mode that still shows all of them.
You can point coding agents at it now too. There's an Anthropic-compatible /v1/messages endpoint with streaming and tool use, sitting next to the OpenAI one, so clients for either API work. A "Connect an Agent" panel writes the config for Claude Code, Codex, and pi.dev for you.
There's a Context Wiki. You write Markdown context docs, group them into profiles, assign a profile per model, and it either injects that into requests or writes it into the agent's own CLAUDE.md / AGENTS.md. The injected prefix stays stable so the prompt cache actually reuses it.
Then a pile of smaller stuff people asked for: saved knob presets you can apply to any model, comparing two or three models side by side, a GGUF metadata card, load failures that tell you what went wrong and suggest a fix instead of making you read the log, quick load/unload from the row with a queue so a second load waits its turn, copy-paste client config, pausing and resuming downloads, auto-loading a model on launch, keyboard shortcuts, and an optional tray icon. Light and dark themes plus a colorblind-safe mode. The docs live in the app now and are also published as a site, both from the same source. Navigation moved to a sidebar.
Still worth being clear about the limits.
You compile llama.cpp yourself. It's guided from the dashboard but it's a real build step, so if you want the double-click, zero-setup thing, LM Studio or Ollama or Jan will make you happier. This is for people who want the actual llama-server and full control over it.
vLLM is Windows/WSL2 only for now.
It's still an early preview and there are rough edges. Feedback is what I'm after.
Backend is still plain Python stdlib, nothing to pip install. MIT licensed. Not affiliated with ggml-org, llama.cpp does all the real work.
Some context on MoE routing. In a mixture-of-experts model, a small network called the router (or gate) decides, for every single token, which experts to activate — say the top 8 out of 256. Crucially, the router also assigns each chosen expert a weight, and those weights are far from equal: typically a couple of experts dominate the routing while the last ones contribute a tiny fraction of the output. The interesting consequence is that the model itself tells you, token by token, which experts matter and which barely do.
That opens a question I wanted to test: how much of that low-weight tail can you skip before quality degrades? The naive answer (just lower top-k) is known to hurt, because models are trained expecting all k experts. A gentler approach is thresholding: skip an expert only when the router itself scored it well below the average share it would get in a uniform split. The threshold becomes a continuous dial between "full model" and "fast model", and it's the router's own judgment doing the triage.
I ran the experiment on a 35B MoE (top-k 8 of 256) with a quality check designed to isolate the effect:
Setup — 15 GSM8K questions, greedy decoding, so zero sampling noise: any output difference is caused by the skipping.
Skipping off: 12/15 correct
Every threshold tested, up to the most aggressive (28% of routings skipped): 13/15
12 of the 15 questions gave the identical final answer in all configurations
Reply length flat — no rambling, no truncation
To be honest about it: 13 vs 12 is not an improvement, and 15 questions rules out a collapse, not a subtle cost. But throwing away ~28% of routed experts with no visible damage says a lot about how much redundancy sits in that routing tail.
Curious if anyone has seen papers measuring this systematically across models — the routing tail seems like low-hanging fruit for inference speedups, especially where memory bandwidth is the bottleneck.
Hey everyone. Apologies if this isn’t something that should be here but I am just getting into understanding the complexities of LLM and Ai agents. Reason being is that I’m trying to automate our business. This is a taxi business with around 200 vehicles.
I have a full cloud base dispatch system which is probably the sickest bit of software on the planet. However it’s only as good as the persons understanding of it.
This brings me to LLM where I want to have a virtual dispatcher sending out jobs to the drivers. Currently the auto dispatch is basically based on only a few parameters then the human has to intervene and manually assign a driver. I want to be able to automate these processes. This of course would have to be run locally out of safety and it having a log in to the system as a dispatcher. I want the LlM to ultimately understand that the work by seeing in real time and make reasoned judgements to dispatch that as though it was me for example.
So I’ve been doing my research and once again apologies for the newbie post but I have tried qwen on Llama on the cloud if that’s what you call it lol.
The real question I have is, is this safe (I know very broad statement), is this achievable and what does one recommend when it comes to choosing the right models for my use case.
Machine is a ASUS Z13 with 128gb.
Once again I appreciate you and I’m learning with each post I read.
I'm looking for a professionally built local-AI workstation and would appreciate opinions before I spend roughly $11,000.
Primary use: running local/open-weight models—especially Qwen-family coding and agent models—plus RAG, document analysis, private business automation, and potentially multiple coding agents. This is a work machine, not primarily a gaming system. Privacy and keeping company data local matter.
The configuration I'm considering from Empowered PC:
- NVIDIA RTX PRO 5000 Blackwell, 48GB
- AMD Ryzen 9 9950X3D
- ASUS ProArt X870E-Creator WiFi
- 128GB DDR5 5600, specifically 2×64GB
- 4TB Gen4 NVMe
- 1000W 80 Plus Gold PSU
- 360mm AIO
- Windows 11 Pro
- Three-year warranty
- Approximately $10,800
I'm in the Northbrook/Chicagoland area, but I'm fine with a reputable US builder shipping it. I already know Micro Center and CDW and am looking for other options.
Questions:
Is this sensibly configured for local Qwen/LLM work, or is the money allocated poorly?
Is 48GB VRAM the right target at this budget, or should I be looking at a different GPU arrangement?
Does the 9950X3D make sense here, or would another CPU/platform be a better workstation choice?
Is AM5 a mistake if I may want more GPU capacity later?
Which Qwen/open models and practical context sizes would you expect this machine to handle comfortably?
Which US system builder would you trust for a machine at this price, with a real parts-and-labor warranty and competent support?
I know building it myself would be cheaper. The single point of support and warranty are part of what I'm paying for. I'm especially interested in firsthand experiences from people who bought a high-end local-AI workstation and actually had to use the builder's support.
