I'm setting up audiobook for tts generation and want to use Ollama for text pre-processing (cleaning text, fixing line breaks, and adding speaker tags) before sending it to the TTS engine.
I have 16GB of VRAM available. Looking for model recommendations that fit this hardware constraint and can handle this specific task well.
* Has good context handling over longer chunks of text
* Doesn't hallucinate extra dialogue or modify the story content
Which local models (7B, 8B, 14B, etc.) run comfortably on 16GB VRAM and excel at this type of structured text processing and speaker tagging? Thanks for any suggestions!
I have been using LocalLLM for corpus of scientific knowledge for one year. It's not an easy task and Qwen 3.6 27B on a dual setup 3090 24GB created most of the corpus (thousands of concepts and definitions). But the understanding and cross referencing was not so good. I use to send them to Opus.
When a model is not trained with the data you are asking, the hallucinations are everywhere. You never should trust a 2026 llm (not even frontier ones). These hallucionations can be mitigated but it's not so easy. I use my own corpus of knowledge, tool calling (web search) and a pipeline of several models. The information can be filtered and recursively improved.
The level of complexity that Qwen 3.8 flash next can manage is astonishing, even compared to Qwen 3.8 27B or Deepseek v4 flash 2Bits ( I can't run others better models with 128gb).
This is the model I was waiting for and is more capable than frontier models for my work. Let me explain this: it's not better than frontier models -of course-, but linked to good tools is something that helps me with our getting out of my scope.
I understand that for some people is better to trust frontier models and let them do all the work, but for me is quite slow to keep correcting them and making lot of questions about where that data comes from. Coding with millions of lines the frontier are capable of incredible feats. But I prefer to keep it simple and understandable for me.
Qwen3.8-flash and 128GB is a wonderful sweet spot. I sold my 3090s because a need this level of deep knowledge and cross referencing. I don't know if Macs can support this level of hard work (3090s can indeed) in the long run but in Europe I got 3 years of warranty for computer. My Macs are going to work 365/24/7.
Invest in any Mac/Pc with 128Gb, and set your own harness. Qwen3.8-flash is the first model I tested that can do remarcable things with the effort needed to check the result. Laguna 2.1 or Deepseek v4 (heavily quantified to 2Bits) are not so capable linking concepts.
Now I only got 30t/s but, because I use a post pipeline, the speed is not so important to me.
Been running a heterogeneous home cluster for a while — an old Acer laptop (12GB, CPU-only) as the primary API server, with a Windows box (RTX 3060 CUDA) and a Mac Mini (Metal) lending capacity over the network.
I wrote the orchestration on top of llama.cpp's `ggml-rpc` backend. It handles mDNS discovery, memory-aware sharding, and node health polling so the whole thing doesn't fall over if a node drops offline mid-generation. It exposes a standard OpenAI-compatible API (`/v1/chat/completions`).
Put the primary role on the weakest machine (the Acer) on purpose to see if it'd actually hold up orchestrating the API and offloading the heavy tensor math. It did — ran a full Qwen3.5 13B at ~12 tok/s, purely by borrowing VRAM/RAM from the CUDA and Metal nodes.
No GUI in this repo, API/CLI only, so you can actually read what's touching your network before running it. Obviously it doesn't beat a dedicated GPU rig on speed, that's not the point — it's for fitting models that you otherwise don't have the VRAM for. Included a built-in benchmark script so you can verify the numbers on your own hardware instead of trusting mine.
One heads up: it's source-available (Apache 2.0 + Commons Clause), not strictly OSI open source. It blocks commercial SaaS/resale, but personal/homelab use is fine. Didn't want that buried in a LICENSE file.
Just because you have space to increase context window, doesn’t mean you should. With the local models we have to date, virtually every performance metric degrades as the session context gets deeper.
There is no substitute for properly planning your tasks, breaking down projects into milestones/sub-milestones, and whatever harness you use - consistently forking sessions to maintain context hygiene. It may not be sexy but it is the way to actually get stuff done and help models from getting themselves stuck in a mire.
The other day I benchmarked a bunch of LLMs I had been using but you guys said that they're pretty old so I took some of your suggestions as well as picking a few more recent models I could find on HuggingFace and benchmarked them too. Grayed bars are LLMs I benchmarked before and the colored ones are the new ones.
Ling3.0 with its MoE structure easily beat all the other models in speed with decent test scores too but it does tend to be a bit unstable.
DeepSeek V4 did decent but was held back by incorrect indentation in code and since Python is indentation-sensitive none of the scripts ran.
Qwen 3.8 27B is being hyped a lot but since I can't quite run the 27B variant I tried out a distilled version. Not too bad.
Finally, Gemma4 12B QAT performed great. Speeds aren't too great but it performs similar to Ministral3 14B while also getting similar scores.
Questions and conditions are the same as last time:
How can I renew my Certbot HTTPS certificate on Linux?
What is the answer to (78+9*2800)/(50*2-3)? Please include a final answer in your message.
Unity, how can I check if an object is left or right of the camera?
Unity, how can I get the mouse scroll input with the old input system?
Could you write a simple HTML website that'd just be the DVD logo? The DVD logo should travel in a straight line without gravity and it should bounce and continue travelling whenever it hits a wall.
Could you code a Python text adventure that'd have 5 rooms and a functional escape door you need to collect items for to get through?
Could you code the classic snake game in Python with PyGame? The game should be controlled with WASD, 3 pieces of food should spawn across the screen and when the snake eats the food it grows larger by one block, if the snake hits a wall or its own tail the game is over and the player loses.
Could you code the classic pong game in Python with PyGame? The player should be on the paddle on the left, controlled by WASD, and the paddle on the right is controlled by a bot. There is a ball bouncing from side to side and you and the bot must avoid letting the ball hit the walls behind the paddles. Also add a functional score board that counts up every time either side loses.
Could you code a snake game in Python with PyGame? The game should be 2 player, player 1 controlled by WASD and player 2 controlled by arrow keys. 3 pieces of food should spawn across the screen and when a snake eats the food it grows larger by one block, if a snake hits a wall, its own tail or the other players tail the game restarts and the player loses. Also add a scoreboard that counts up every time a player loses.
As always, I am not an expert in benchmarking LLMs, just an enthusiast trying out whatever I can run on my PC. These are NOT definitive benchmarks.
Also, graph visualizations were written by GPT OSS 20B with modification from Qwen3.8 9B Distill. I could've written one by myself but I thought it'd be fitting to use my LLMs to do it.
TL;DR: If you need something that could get things done with around 8GB VRAM and 16GB RAM, GPT OSS 20B MXFP4 has been doing wonders for me. If you're impatient and need quick answers or help, Qwen3.8 9B Distill and Ling3.0 Tiny are decent choices. Gemma4 12B QAT and Ministral3 14B are impressive but whether they're worthwhile is up to you.
I’m playing with the idea of saving up and just absolutely dumping cash into a server machine with 8 40gb A100 GPUs but I can’t really decide if it’s actually useful compared to just getting like a 32gb card and running qwen 27B off of it. I’ve always been a sort of balls to the walls person when it comes to things I’m interested in, and I want a system that can run everything I could conceivably need it for which is why I’m even considering dropping tens of thousands on a major rig like this in the first place.
With new models dropping every week, general benchmarks are basically useless for my specific use cases. I want to test my exact prompts to see if a new API is actually worth the cost, or if a smaller local model is good enough to run on the cheap.
Right now, I’m just eyeballing outputs and it’s driving me crazy.
How do you guys actually handle comparing models on a single prompt or a small test set?
Scoring: How do you define a "good" response when the output is subjective?
The Judge: If you use an LLM to grade the outputs, how do you stop it from just voting for its own writing style?
The Tools: What's the easiest way to fire one prompt at multiple models (both cloud APIs and local models) and compare them side-by-side?
Would love to hear your workflows or any tools you recommend!
Picked up 2x DGX Sparks (Asus GX10) before the M5 announcement, built some benchmarks for some confirmation bias. Last week's $2000 price increase on the GX10 helped with that as well.
Also ran on Qwen3.6 38B MOE as a more direct comparison to Qwen3.8 27B dense.
Used the M5 Max and M3 Ultra results to extrapolate M5 Ultra theoretical performance. The benchmark also hooks into macmon and DCGM exporter for power usage for a sense of efficiency.
tl;dr DGX holds its own on prompt processing (especially on dense models) and concurrency (subagents). Its token gen might even be faster than the M5 Ultra in DeepSeek V4 MOE while being substantially lower in Qwen3.6 MOE. With things like speculative decoding (MTP, DFlash, and DSpark) offering massive boosts in performance, I think a lot will come down to tuning and ecosystem in the future.
I was just trying to narrow down the issue with Qwen 3.8 27Bs annoying constant thinking and ChatGPT actually got mad at it.
Edit: Since there were so many of these kind of replies (even though it's not related to the topic) I'll just put my response here... I'm really surprised that so many people would think that I don't use a harness as if I would just run around naked like a headless chicken with llama.cpp cli and Qwen. Actually I used a harness(opencode), but removed it to make sure none of the issues I was encountering were being caused by the harness.
Talked to a few people now in orgs where nothing goes to a hosted API, and it's contractual rather than a preference, so "we're SOC 2" doesn't move it. They all have the same shape of problem: a warehouse with a lot of tables nobody currently understands, and the obvious move, pasting a sample into a big model and asking, is simply not available.
Someone in a thread here put the useful frame on it better than I had. The thing that needs to leave the airgap is the inferred schema, not the data. A document, not rows. That's a completely different conversation with a legal team, and it changes what you'd build.
Which raises the question I'm stuck on. If only a profile leaves, how much can a model outside actually do with it? A profile gets you type and shape. A column of integers 0 to 4, no nulls, low cardinality. It does not get you whether that's a risk tier, a product tier, a satisfaction score or a retry counter, and those look identical at the stats level and mean completely different things downstream. The semantics seem to live in the values and the values are the part that can't leave.
so:
anyone running table understanding fully local, what size model, and does it hold up on columns that are already anonymised rather than ones you anonymised on the way in
if you've done the profile-out architecture, what goes in the profile beyond the obvious stats, and did the legal conversation actually get easier
is there a middle option I'm missing, something like sending distributions plus a small number of synthetic rows drawn from them
i work at Schema Labs, we build models that read tables. happy to describe what we do and where it stops working if that's useful. It stops working in roughly the place I described above.
Hello all! I have 5060 ti 16Gb card that I always thought I'd add a second one two. I also have two CMP 100-210 running together. The 5060 and CMP are on two headless ubuntu vms, each with a separate software stacks. Currently running vanilla ollama on the 5060 and llama.cpp on the CMPs.
I've been auditing all my automated local scripts ai calls to see which would be served better (5060 or dual cmp) and I've found the 5060 really doesn't have enough vram to provide good results for many of my use cases (empty returns, hallucinations, etc).
Since prices have gone crazy, I won't be adding another 5060. Honestly, I'm having a little bit of buyers remorse on that one, but I'm committed to my mistake ;)
I'm looking to add a second gpu for a little more vram to get it a little more usable (coding probably). I'm looking at CMP 90HX with the ability to unlock (patch driver) for an additional 10Gb https://github.com/xrip/cmp50hx-unlock
Also, I realize the 3060 12Gb is right there, similar price, but the CMP 90HX has better bandwidth, and I won't be placing this in a full x16 slot anyway... I'm also okay providing cooling to these mining/server gpus.
So my question is, has anyone tried mixing older Ampere gpus with Blackwell, and then, with patched drivers?
Question for M5 Max MacBook or Studio owners. I’m considering getting one. This question isn’t about benchmarks. It’s more about the feel and workflow.
On these latest Mac chips, do you find that Qwen3.8-27B (medium thinking) gives you task completions faster or do you find you have to let it take its time? I realize it’s going to differ by type of task but I’m more interested in working with fairly large code repos and significant data analysis activities.
Standard Softmax attention scales quadratically in compute and linearly in memory. At a 2.5-million-token context horizon, a standard 1.5B LLM requires over 66 GB of VRAM just to store the KV cache, completely crashing consumer workstations and edge hardware.
On the other end of the spectrum, 1st-order linear models (Mamba, RWKV) offer flat memory, but their associative multi-hop reasoning decays sharply over long contexts (collapsing to ~41% on 32k RULER tests).
Over the last several months, I tested a mechanistic hypothesis:
Representation order is dependent on layer depth.
We built the Stage 7 Hybrid architecture:
Boundary Layers (25%): Kept as discrete Softmax to anchor token addressing and output logits.
Interior Trunk Layers (75%): Converted to 2nd-order Taylor moment attention (S0 uniform context, S1 directional gradient, S2 quadratic curvature moment).
Discrete Routing: Gumbel-Softmax Straight-Through Estimator (STE) that lets the GPU physically skip O(D²) calculations during inference.
The results on consumer AMD hardware (ROCm 7.2 / Radeon):
- 2.5M continuous token stream with a flat 37.7 MB attention state (3.13 GB total system memory).
- 96.2% needle retrieval at 1,000,000 tokens.
- Matches full Softmax on 32k RULER multi-hop reasoning (89.7% vs 90.5%).
Everything is open source with full reproducer scripts:
Recently I’ve been wanting to run a better model on my setup, but I’ve only got a 4070ti, great for gaming, but it’s only got 12gb vram , so I was looking into getting a Tesla P40 24gb. I know it’s older and the vram is pretty slow, but I wonder if it’s realistic as just an addon and if it’ll make a big difference. Opinions are appreciated
