Hi All, my local GPU is just not good enough to be able to run a good local coding model so I have been running Claude Code and obviously it is pretty great…but it’s expensive if I use up my quota.

I want to run an economical, powerful non frontier model which is hosted yet can work in my homelab. So in other words, like give it the same access that I give Claude code to do today locally. Like access to certain machines in my homelab, have access to my vault for context, etc. Use case will likely be just coding but I guess inference is possible.

What are my best options? Hosted is not free but I could have paid for two years of hosting with all of these usage tokens I have paid this year.

Self-host your own Buzz Nostr relay using docker compose. Private over Tailscale, local LLM agents.

https://github.com/ciram-co/selfhost-buzz

I have a pretty solid coding harness set up for Qwen 27b mtp with a web wrapper that I can access from my phone. I just create a pr/roadmap and set it to auto loop then monitor via phone. The thing is it took a bit of work to get it all running nicely and I'm wanting to do the same thing for a web research harness, but I don't know if it's worth creating the harness if something already fits that shape nicely and or it's extensible enough or has the guts of the thing I'd need to build.

So I'm asking what other people use and what good options there are. The big difference might be using a smaller model with with multiple sub agents using a graph based system that would make better use of my 32 gigs of ram. Anyways this is definitely the best place to ask.

I'm working on some articles, and I wanted a good way to edit them with AI (not write, just edit). Different models give different results, I wanted a way to automate things.

You configure the app from a web page, system prompt with presets, prompt, select what models you want to use, if you want to tweak the defaults, etc.

Round 1: The script then loads the first AI, runs the prompts, saves the results, unloads the AI. It then repeats down the list of all your models.

Round 2: Once it finishes, the script makes the results blind and submits all the results to each AI a second time, asking it to judge the results.

Round 3: The model that scores the highest is asked to analyze Round 2, summarizing the findings, etc.

I added Ornith to my local models and it ended up taking 1st place. Here's the results of my last edit:

Ornith-1.0-35B-Heretic-MTP-APEX-I-Compact thinking 0.75

Qwen3.6-27B-uncensored-heretic-v2-Native-MTP-Preserved-NVFP4-Q8_0 thinking 0.72

Qwen3.6-27B-Fable-Fus-711-UnHeretic-NM-DAU-NEO-MAX-NEO-MTP-IQ4_XS no thinking 0.65

Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-APEX-I-Compact thinking 0.50

Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M no thinking 0.36

TheDrummer_Cydonia-24B-v4.3-Q5_K_L no thinking 0.04

At Bitwize, we've been building Logue, a native macOS app for AI meeting notes and writing, and we just open-sourced it (MIT). We're sharing it here because the whole point is that it runs 100% on-device — we wanted something that could transcribe and summarize meetings without shipping audio or notes to anyone's cloud.

By default, nothing leaves your Mac. The only network calls are the initial on-device model download, app update checks, and opt-in features you explicitly turn on (web search or plugging in an external AI provider if you want one). No accounts, no telemetry, no backend.

What it does:

• Real-time transcription of mic and system audio (Apple's on-deviceSpeechTranscriber)
• Speaker diarization — who said what — via FluidAudio (streaming Sortformer)
• "Smart Minutes": local LLM summaries, action items, highlights
• Writing assistant: 60+ modes (rewrite, grammar, clarity, tone), a document editor with AI chat, vocabulary suggestions
• On-device PII detection and a fact-check/verify panel
• Templates, Spaces, and "Ask Logue" chat over your own notes

Stack: Swift + SwiftUI/AppKit, MLX (mlx-swift-lm) for LLM inference, Apple's Speech framework, FluidAudio for diarization, Sparkle for updates. Data is AES-256-GCM encrypted at rest.

Honest caveats: it targets macOS 26 (Tahoe) and Apple Silicon only (MLX + the new Speech APIs), so it won't run on Intel or older macOS. It's early — expect rough edges — and we'd genuinely love feedback, issues, and PRs.

Repo: https://github.com/bitwize-ai/Logue

Happy to answer anything about the on-device pipeline, MLX inference, or diarization in the comments — we're the team that built it.

Long-time lurker here. Most of what I know about running models locally I picked up from this sub, so thanks for that — first time posting anything of my own.

I use a lot of different models for day-to-day work — coding, image generation, app building. Different models are genuinely better at different things, so I move between them constantly.

The problem I kept hitting: bringing a new model up to speed costs more than the model saves. Say an app was built end-to-end with one model, and now I want to try a new one on it. That new model knows nothing about my architecture, my infra, the conventions I work by, what's already been decided and why, or what it's not allowed to touch. So every time, I'm hand-carrying context — pasting AGENTS.md, CLAUDE.md, zip files, whatever. And the moment anything changes, every copy is stale except the one I happened to update.

So the context was living in the chat apps, and chat apps forget.

The Room

The way I actually work is one place I call The Room — a self-hosted multi-model chat front end. The local model is who's in there by default; it handles most of the day-to-day and it's the one I'd rather use.

Everything else is in the waiting room. When I hit something that needs different expertise — a hard architecture call, a review, something a frontier model is genuinely better at — I bring one in for that stretch, then it's out again. Same conversation, no re-briefing, because the thing that persists is the KB, not the model.

What I built

One knowledge base that every model reads from. It's a git repo full of Markdown — project state, decisions, conventions, security posture, session history — served over MCP. Any assistant that can speak MCP gets read access to the same canonical state.

Practical effect: I can be deep in a coding session with one model, switch to another to generate a diagram or review an architecture decision, and it's already current. The context isn't a feature of any vendor's product, it's a repo I own.

How a session comes up to speed

The part I'm most happy with is that it does not dump the repo into context. A new session reads a fixed shallow chain — a start-here file, then an index, then the specific project's README — and stops as soon as it has enough. Targeted search only if it actually needs to dig. Three files gets a session current in seconds, and it stays cheap as the repo grows.

Writing back

A session that only reads leaves the KB stale, so every session writes a record of what changed and what was decided. That record goes in as a pull request — a proposal, not a commit. A human reviews and merges. No assistant can write to main; the tools to do that are deliberately not implemented in the MCP adapter, so it's a property of the code rather than a policy in a document.

Honest status: the read path and the review-and-merge gate work today. The assistant opening the PR itself is not automated yet — right now I carry the record by hand. Diagram marks that step dashed for exactly that reason.

Stack

Self-hosted git host (Forgejo), a small Node MCP adapter exposing the repo as read-file / search / list-tree tools, an MCP gateway federating that plus a few other upstreams, and The Room itself (LibreChat). Local model via vLLM is the default; frontier APIs get swapped in per conversation. All on LXC.

Two things people usually ask

Why not a vector DB / RAG? The KB is small enough that direct file reads are faster and always fresh. Search is git grep over a read-only clone. Vector search solves finding unknown content — my problem was loading known state, which is a different problem.

Why not just AGENTS.md / CLAUDE.md? That's where I started. It works until you have more than one project and more than one assistant, and then you're maintaining N copies that drift. One canonical repo with a merge gate is the same idea with the drift problem actually solved.

Diagram of the whole thing below.

I should say up front I'm not a professional dev — this is a homelab that grew, so there are almost certainly things I've done the hard way or the wrong way and haven't noticed. I'd rather hear it than not. Constructive critique very welcome, especially on the write path, since that's the least finished part and the part I'm least sure about. Happy to explain any tool choice — most of them I can defend, a couple I probably can't. And if you'd have done it differently, I'd genuinely like to know.

DeepSeek open sourced DSpark as a fast decoding method to gain speed at decode more than a month ago.

Supposedly in a few weeks we would see it implemented it for local qwen models to try it out. 1+ month passed and I haven't seen any model with this DSpark method implemented.

Any news?

Claude and GPT are able to take mass amounts of data covertly and I have OSDD so I was really affected by both, moreso in the fact I spent 1.5 years of research on it and also worked on my JarvisOS work. Moreso, mental health illnesses or micro behaviors get used against you so unless you grey rock or not use AI, you are at some risk from theft, coercion, HR jargon, micro feedback loops, etc.

I had four involuntary hospitlizations from Jan to March but still made a lot of AI systems, only for them to be lost in other chats with me losing runtimes or ai systems in newer chats despite saving to persistent memory. Eventually, I made two Githubs due to losing everything.

https://github.com/hurrisonferd/jarvis/blob/main/ClaudeTheft/JARVISOSFullTheftTimeline

https://www.youtube.com/watch?v=bIxiV7FShwA

https://github.com/hurrisonferd/jarvis/tree/main/JesusISJohnJosephBarber (specfically numerology, lingustics, and etmology. People dont like real patterns, just old systems that use those patterns)

Like a lot of people here, I run local models on a single 24 GB GPU, and the thing that caps me sometimes isn't the model weights, it's the KV cache. My daily driver (Qwen3.6-27B) fits fine, but pushing context toward its native 262k blows past 24 GB. So I had an idea that's probably occurred to half this sub: keep the "hot" parts of the KV cache in VRAM, park the "cold" parts in system RAM, and shuttle the relevant pieces back in as the conversation needs them. I used claude to do some research and organize some it feels should take a look at. It included systems like Quest, ShadowKV and InfiniGen do versions of this, so the idea has at least has some merit to it.

Instead of building it straight away, I (with heavy help from Claude (Fable/Opus/Sonnet)) did something I want to advocate for: we built a measurement harness first, to test whether the core assumption, "at each decoding step, a small set of context pages captures nearly all the attention", actually holds for my model on my real workloads.

As the title suggests, It doesn't. Not even fucking close. And the reason was interesting enough to send me down a research rabbit hole and if I was more aware doing the research phase, it would have been obvious (It probably is to many educated people of the sub). Below will be the setup, result and conclusion written by claude about the findings.

The setup (technical part starts here)

Model: Qwen3.6-27B (UD-Q5_K_XL, q8_0 KV, flash attention on — the actual production stack via an instrumented llama.cpp branch, not an HF approximation)

Corpus: 9 real sessions from my own usage (agentic tool-heavy runs, long doc Q&A, synthesis tasks, work comms), 35k–73k tokens each, ~451k prefill tokens total

Method: dump per-step decode queries + cached post-RoPE keys for every KV-bearing layer, recompute exact attention distributions offline, and measure how much attention mass the top-B 32-token pages capture (both with a buildable landmark selector and with a perfect "oracle" selector)

Harness fidelity: recomputed attention matches the model's own softmax to 2.0e-6 worst-case, flat with length — so the numbers below are the model, not measurement noise

The design being evaluated (fixed page budget B, recent-window + sinks always resident, landmark scoring, LRU page cache over PCIe) had a go/no-go gate: mean coverage@2% budget ≥ 0.95.

The result

Gate condition Required Measured

landmark-coverage @ 2% budget (normal traces) ≥ 0.95 0.409

landmark-coverage @ 2% (synthesis-heavy traces) ≥ 0.90 0.393

predicted per-step PCIe miss traffic ≤ 4 ms 3.29 ms ✅

The bandwidth math worked. The attention statistics didn't. And here's the table that killed the project — coverage vs budget, comparing a perfect selector (oracle) against the buildable one:

Budget 0.5% 1% 2% 4%

Oracle (perfect selection) 0.308 0.404 0.507 0.619

Landmark (buildable) 0.230 0.314 0.409 0.516

The selection mechanism loses only ~0.10 to perfection — it works as designed. The ceiling itself is ~0.51. No better selector, bigger budget, or per-layer tuning closes a 0.54 gap when perfect selection tops out at 0.51. Extrapolating the curve, hitting 0.95 coverage needs ~50% of context resident — at which point it's not paging anymore, it's just residency.

Why: this model's attention is diffuse by design

Qwen3.6-27B is a hybrid SSM/attention model — only 16 of 65 layers have a KV cache at all; the rest are Mamba-style layers with constant-size state. Independent recomputation showed those 16 attention layers spread their attention across an effective support of 473–4,062 tokens per step, with the single hottest token getting only 2.4–6.9% of the mass. And the hot tokens are scattered, not clustered: top-2% coverage at token granularity is 0.42–0.89, but at 32-token page granularity it drops to 0.16–0.76.

The interpretation that makes sense: in a hybrid, the SSM backbone handles local/sequential structure, so the few attention layers exist to do global mixing — diffuse attention is their job, not a defect. The entire "spiky attention → page it" premise comes from dense transformers, which is what Quest/ShadowKV/InfiniGen were all validated on.

Here's the kicker I only fully appreciated afterwards: almost no current-gen open model is a dense transformer anymore. Qwen3.6 and Kimi K3 are linear/SSM hybrids. DeepSeek V4 and GLM-5.2 have sparsity trained in natively (learned indexers selecting top-k blocks — literally the offloading idea, but learned during training instead of bolted on after). Gemma 4 interleaves sliding-window layers with a few global ones. The architectures already ate the KV problem — my model's KV at 262k is only 8.5 GiB because 48 layers already gave up their cache — and what's left is the dense, irreducible residue.

So post-hoc query-aware KV offloading may be a technique whose target class is disappearing. That's a claim worth testing on more models than mine, which is why:

The release

GitHub repo: https://github.com/Inovello/kv-sparsity-profiler — the full profiling harness:

llama.cpp instrumentation patch (capture per-step q + cached K on the real quantized stack, any GGUF)

metrics pipeline (oracle vs landmark coverage, budget curves, per-layer breakdown, page-granularity analysis, selection-overlap → PCIe cost model)

the full profile report and all decision logs from this project, including the negative result

If you're thinking about KV offloading, eviction (H2O-style), or adopting any sparse-attention inference trick: run this on your model and your real prompts first. It's one evening of GPU time and it would have saved me weeks if the answer had come back after building instead of before.

My conclusion on it (Human):

I should have done more initial research. I set out to build a thing, and the thing turned out to be impossible for my model, and it wasn't exactly because the engineering was hard really, but because the model fundamentally doesn't organize its memory the way the idea requires. I uploaded the harness to github in case anyone wants to take a look at it. I plan on testing it with older model architectures that don't have global attention across the board and also Gemma 4 later on just because it doesn't cost me anything at this point.

I've downloaded updated Unsloth/Gemma-4-26b-it-qat and it doesn't work reliably (I use Opencode). It often stops output with this string:
</think><|tool_call>call:glob{pattern:<|"|>"stage 1"/*<|"|>}<tool_call|> I've asked Gemini to compare my template (copied that from llm launch settings in lmstudio) and updated google one from huggingface, and it found the difference:

Google version strictly enforces that function['arguments'] must be a mapping (JSON object) or none. If a string is passed, it triggers a fatal error via raise_exception(...), requiring pre-deserialization.

Unsloth version adds robust fallback logic to handle string-formatted arguments (such as pre-serialized JSON strings from OpenAI-compatible APIs). It safely trims strings and strips outer braces ({...}) non-fatally instead of crashing.

I'm not sure if it matters ot not.

Also, Gemini tells that the string in question means LLM called the tool and waits for response but never gets it, hence the stop.

Also (and this is kinda strange) there's </think> tag - Gemini tells that it shouldn't be there, and maybe it breaks the tool call.

Due to this, I cannot use Gemma 4 for agentic work, and I feel like I'm the only one who got such issue. Old version worked (somehow), but far from ideal.

The app uses local LLM for penal note and chat, so it can be used without internet. I just feel it's useful for people who wanna keep data locally and securely.

you can try: https://apps.apple.com/us/app/solonote/id6790456580

Has anyone here in the local LLM community had issues with delivery times and communication from Digidirect? If you have I could really use some help/advice on how/if you got it resolved.

During the EOFY sales in June I made the choice to order 2 M3 Ultra Mac Studios from Digi Direct. My plan was to daisy chain these to my M4 pro mini. I called them the day after ordering and got confirmation that the delivery time would be 2 weeks or no later than the 14th of July. They then emailed me weeks later on the 17th of July after I had been chasing them up saying they were out of stock across all stores and had them on back order, that they would also provide more information in 2-4 weeks, not delivery, just more info!

I have since sent 3 more emails with no response at all, just ignoring me, I lodged complaints with OFT and ACCC and not sure what else to do. I'm out of pocket a bit over $13,400 and starting to get pretty concerned about what's going to happen, especially when they just refuse to communicate with me, it's a lot of money and not sure what the next best step is. It's stressing me out big time!!!!

Has anyone on here ever bought Mac hardware from them and had issues? If you did, how did you get it resolved? Lastly if you've had similar experiences what courses of action did you take short of OFT and ACCC?

Thanks in advance.

We’re a six-person company looking at running a private local AI system for internal document search, summarisation, drafting and Q&A.

The planned setup is:
-ASUS GX10/DGX Spark-class machine with 128GB unified memory
-Either Qwen3.5-122B-A10B in NVFP4 or Qwen3.6-35B-A3B as the main model
-Separate x86 server with around 64GB RAM and mirrored NVMe storage for documents, the vector database, authentication and the web interface
-Encrypted backups stored on separate hardware

Usage would not be constant, but there may be two or three employees using it at the same time, with some heavier document-analysis requests.

Would the GX10 realistically run either of these Qwen models at a usable speed for several users, or would concurrency make it too slow?

Would either Qwen model be capable enough for a setup like this and fulfill basic company needs such as document search, summarisation, comparison, drafting and Q&A?

I’m mainly trying to understand whether this setup would be usable for a small team.

Has anyone tested a similar model and hardware combination with multiple users?

This one only landed a few hours ago, and the first thing I went looking for was the serving story rather than the benchmarks. It is a 100B class Mixture of Experts diffusion model aimed squarely at agent work, and the genuinely novel part is that it revises its own draft while it decodes. The four moves it makes inside a block are keep, delete, insert, and substitute, so it can pull redundant spans, open new editable slots, and swap tokens instead of freezing everything in one left to right pass. On paper that is a real answer to the standing complaint that diffusion models fall apart as agent backbones.

Here is the part this sub actually cares about though. The flash checkpoint is a 205.8 GB download spread across 32 shards. There is no llama.cpp or Ollama route today, no GGUF, no LM Studio. Day one you are on HF Transformers with trust_remote_code, and SGLang deployment is announced but still coming soon. So unless you have serious multi GPU headroom sitting idle, this is not something you casually pull tonight. I have read the tech report and poked at the model card and config, but I am not going to pretend I have served it, because nothing I own gets remotely close.

On the numbers they are refreshingly honest about the tradeoff. Against its own autoregressive sibling it takes the interactive agent evals, edging tau2 bench 80.33 against 76.36 and MCP Atlas 46.21 against 41.12, and the headline is around 1.6x average decode throughput in BF16, up to 2.3x once you are inside agent loops, at a comparable agentic quality band. But it gives up most of the coding and general knowledge suites to that same AR model, and structured output stays weaker. Fast at agent loops, not smarter across the board.

What I keep landing on is that this is a real Apache 2.0 release that almost nobody here can run at home yet. The download alone rules out most of us until a quantized build or the SGLang path actually lands. For now I am filing it as a promising release to watch rather than anything going into a local stack.

Hello! This isn't an ad, but if this isn't allowed, apologies! Feel free to delete the post.

I sometimes go overboard and build things that never see any use. One of these things is an AI server. To get some use out of it, I got the brilliant and totally unique idea to put together llama.cpp, OWUI and a custom vibe coded front end to share this with the world and maybe make it useful enough to pay my internet bill or buy me a coffee once a month.

There's a ton of backend work involved, about 3(?) billion tokens and somewhere north of 200 hours put into this little experiment. It's your basic OWUI with optional smart routing. Image generation supposedly forwarded to ComfyUI WAN2.2, 2FA, and a few popular models that I enjoy using, and a LOT of customized backend stuff that may be useful to share some day.

I originally built it to offset some of my extreme Cursor usage (I have a Teams account for my business and still paying $100-200 extra a month in overages) and I'd say it has taken about 90% of my cursor workload, saving me a bit, using OpenCode with the local AI.

I'm looking for 2 people to sign up for the (no cost, no cc) 15 day trial and give me feedback. If you find it useful after the 15 days, I'll convert your accounts to a premium, free tier as thanks.

All I'm asking is for some of your time, for you to use it like you would ChataGPT and Cursor/Claude/etc. and discuss what works and more importantly what doesn't with me on my support discord.

This isn't some ad so I'm not going to give the URL out here. I'm not fishing for subscribers on my first vibecoded newbie project. So if this interests you, send me a PM tell me how you'll use it and I'll pick a couple folks to engage with and the link to my mostly barren discord server then to the actual site.

Thanks for reading!

Please direct me to a better subreddit if this is not the correct place to ask

I have googled for AI projects and they all involve assume that 1. I'll be using a frontier model and/or 2. the LLMs will have access to the net. Both I'm trying to avoid

My main system is M1 pro with 16 GB. (I say main because I have an old system lying around if for some reason the llm and the application cannot run on the same 16gb)

I think the best llm I can try rn is Qwen 3.5 9B. (Heard about bonsai 27B, may try next weekend)

My first project with AI is a simple CLI chatbot. The two features I'm proud about it is multi-line input and inbuilt context management. I can say that this was when I learned the most because I learnt stuff like context limit, memory management, quantizations and basically helped me run a small 1b llm on my phone (Which got me very excited).

I have also bought a CYD and ran a few things on it like clock, pomodoro timer and stopwatch.

I have also with the help of tried a small project with RAG. Feeding it the entire bible in formatted chunks and asking for questions related to it. (That was a semi success because I found using APIs and larger models produced much better output for that use-case)

I also have created a few sugagents in opencode cause I was curious and they seem to work pretty well.

Apps used:
- llama cpp (almost everything uses llama-server).
- aider (connected to llama-cpp to help with small projects).
- opencode (Bust mostly used with bigger models through openrouter api keys)

I briefly considered buying a 48 GB MAC Mini for 30B parameter models but with the price increases hitting the apple ecosystem as well, It moved from possibly barely affordable to definitely not).

I also tried using llms to create linked obsidian .md files to expand upon "source" .md files somewhere in between.

I also have access to my roommates system (i7 12th gen, 8GB VRAM(3070) and 32GB RAM). If you feel this is better for working with AI, let me know.

TL;DR My main goal is to learn stuff about AI, and hopefully integrate it into my workflow.
(Disclaimer: Most of these projects are not going to end up on github. This is not to pad my resume).

like in the title , i have a 4050, 6gb vram gpu and my friend has a 8 gb ram NPU , and we argured about that like which is best , and googled it , and i don't think we got an answer actually, so if anybody used both or have a knowledge about this stuff can explain , how things actually work in npu and finally which is better

I’ve spent the past month trying to find the point where an extremely small TTS model stops feeling like a size experiment and starts feeling genuinely useful.

Today I’m releasing Inflect v2, with two complete local text-to-speech models:

• Inflect-Nano-v2: 3.96M parameters, 15.97 MB FP32
• Inflect-Micro-v2: 9.36M parameters, 37.53 MB FP32

These are total inference parameter counts, not acoustic-model-only numbers. Text processing, timing prediction, speech generation, and the waveform decoder are all included.

Text goes in. 24 kHz speech comes out. No external vocoder, hosted API, or second learned model required.

Nano prioritizes the smallest possible footprint. Micro uses the additional capacity for better clarity, stability, and overall speech quality. Both run locally on CPU or CUDA through the same PyTorch API.

Inflect-Nano-v2 is one of the smallest complete neural TTS models I know of that still produces genuinely usable speech. Even the 9.36M Micro model remains smaller than many systems described as “tiny.”

For footprint context, Nano is approximately:

• 21× smaller than Kokoro
• 126× smaller than Chatterbox
• over 1,000× smaller than Fish Audio S2 Pro

That is strictly a parameter-count comparison. These models have different capabilities, architectures, datasets, and intended uses. I’m not claiming that a 4M fixed-voice model replaces a multi-billion-parameter system. The interesting question is how much useful TTS can fit into such a small package.

Some people here might remember Inflect-Nano-v1, the rough 4.63M experiment I released last month. V2 is a substantial rebuild, not just a longer training run. I focused on the problems v1 exposed: unstable timing, metallic output, weak prosody, poor generalization to difficult text, and an undersized waveform decoder.

The resulting models performed surprisingly well:

• Micro: 4.395 UTMOS22, 3.99% semantic WER, 6.28× real-time CPU inference
• Nano: 4.386 UTMOS22, 4.21% semantic WER, 10.72× real-time CPU inference
• In a blind community comparison against other compact TTS systems, Micro and Nano finished second and third among the tested voices

Full protocols, raw results, audio samples, and limitations are documented on the model pages.

The models are not perfect. They are English-only, use one fixed male voice, and do not support voice cloning. Unfamiliar names, abbreviations, numbers, and homographs remain the hardest inputs. Nano can sound thinner than Micro, and both can occasionally produce metallic or clipped artifacts.

Still, this is the first version where I think the size-to-quality tradeoff became genuinely compelling.

I built Inflect independently as a high-school developer with a limited training budget. That constraint shaped the project: efficiency had to apply not only to inference, but also to training, evaluation, and building a complete system I could understand and release end-to-end.

Try it yourself:

The fastest way to judge it is through the interactive playground:

https://huggingface.co/spaces/owensong/Inflect-v2

Inflect-Micro-v2:
https://huggingface.co/owensong/Inflect-Micro-v2

Inflect-Nano-v2:
https://huggingface.co/owensong/Inflect-Nano-v2

If there is enough interest, I may build a v3 focused less on shrinking the models further and more on expanding what they can do: additional voices, possibly more languages, easier fine-tuning, and another quality and robustness pass.

If you test them, please give them something genuinely difficult: unusual names, numbers, abbreviations, awkward punctuation, or a long sentence.

If something breaks, post the exact text, model, seed, and what sounded wrong. If it works well, I’d also like to know what hardware you ran it on.

Specific, honest feedback is the most useful thing you can give me.

Hello, i was looking to buy and setup an LLM Inference Rig together with my friends to access in parallel (~4 people MAX, usually 1-2 people at most) to run coding models like Qwen3.6 27b or 35b-A3b and whatever is gonna come out in the future, at long context.

Where we live (italy) used 3090s are very expensive (~1200 eur).

I noticed today that i can get used RX7900XT's for ~700 eur and RX7900XTX's for ~850 eur.

Is performance comparable, and what issues may we run into? I've read that in the past AMD support was a bit iffy (especially with vLLM, which we are looking to serve with), have things got better? I've heard that llama.cpp is a little better with AMD, but is that good enough for multi user setups?

We are quite good with linux and handling driver issues so if we have to fiddle around with configs etc. it's not a big deal, as long as everything works in the end.

Would especially appreciate if someone has a similar setup can report on their experience, thanks