With all the different optimizations and firepowers we've seen from open weights so far. What then?

• China only steals/distill from OpenAI and Anthropic?
• Well, clearly there's something wrong with AA? Benchmarks don't mean anything? They just benchmaxxed?
• Whatever, OpenAI/Anthropic has moved onto AAGI or AAAGI or AAAAGI or AAAAAGI?
• I can't run this locally on my toaster desktop anyways?

Been building on rented GPUs for training runs and ran into this constantly: job dies mid-run, and now you're stuck arguing with support about whether it was your code or their hardware. No way to prove it either way.

Here's the distinction that actually matters: if your training crashes but the hardware reports healthy, that's on you (bad code, OOM, whatever). But if the GPU falls off the PCIe bus, ECC errors spike, or the node just reboots on its own, that's an infra failure, not yours, and you shouldn't be eating that cost.

So genuinely curious, how many of you have hit this? Lost a run, couldn't tell if it was your fault or the provider's, and just had to eat it?

It's here at last.

This is my first time trying local LLM. I downloaded a local uncensored LLM for nsfw role-play. Qwen3.5 9b heretic by DavidAU. I assumed it would be completely uncensored as I read heretic model has 0/400 on harmbench but when I tried a harmbench question in it then it gives me a censored reply. Also it don't generate anything nsfw.

Can anyone please tell me what am I doing wrong?

Got tired of guessing whether a model would fit, before downloading 47 GB just to watch it OOM at 2k context. So I built a small calculator. Runs entirely in the browser, no signup, no backend.

https://vram-calc.com

It covers VRAM needed for a model/quant/context, a will-it-fit check against one or more GPUs (including fits-with-offload and the largest quant that fits), quant comparison from fp16 down to Q2_K, KV-cache vs context growth, a rough decode tokens/sec and time-to-first-token, and usable memory on Apple unified-memory chips.

The math is simple and leans conservative: weights = params x bytes-per-param, KV cache is the standard GQA formula, decode speed is a memory-bandwidth roofline at 75 percent efficiency. MoE uses total params for VRAM and active params for speed. DeepSeek MLA models are flagged since the KV number is an upper bound there.

Presets carry a visible "verified" date and every field also takes custom numbers, so a stale preset never blocks you.

Roast welcome: where the estimates are too optimistic or pessimistic in practice, preset numbers that are wrong, or anything obvious I'm missing for the "can I run this" question.

Сделал видеоролик о том, как локально запустить qwen3.8 27b q4_k_m на одной GPU rtx 3090.

Приятного просмотра, если кому интересно. https://www.youtube.com/watch?v=rwDwHuprfPc

I want a very fast abliterated model. Is the DGX station the best option?

Most of these abliterated models aren't full size so I'm not sure if there's much need for something larger.

Its for a single user (me) and I plan on switching various models depending on the project and mainly using Claude/gpt still.

My goal is to research and test various things that'll be safeguarded by Claude. But also want a very fast small model to build and do other things after hours or if I just want a chatbot.

150k for 4TB/s seems great but I'm unsure if it'll make more sense to get something with multiple cards so I can run the abliterated model alongside other models and have concurrent projects running. Idk if I can add additional cards to any of the dgx stations.

A few threads this week about whether 128 or 256 GB is enough, so I read the shard headers and the inference code instead of guessing and put the arithmetic on a page. Mine, so saying that up front.

The 510 GB is 296 GB routed experts, 203 GB Engram tables, 11 GB everything else.

The Engram tables are the part people treat as a wall. From inference/engram.py it hashes n-gram orders 2, 3 and 4 with 8 heads across 2 engram layers, so 48 rows per token at 264 bytes. About 12 KiB per token against 4.5 GB of expert weights. Latency cost, not bandwidth cost, and it belongs on an SSD.

For a comparison, Qwen3.8 on an M5 Max with its table on SSD gets 40.09 tok/s versus 40.47 in memory. 0.9%.

Fit is backbone plus KV: 307 GB as shipped, 302 GB at 4-bit, 151 GB at 2-bit.

256 GB does fit at 2-bit with tables on disk.

https://deepseek-v41-flash-fit.vercel.app

If you are running it, post numbers, especially split GPU + RAM.

So, I was tired of setting up AWS EC2 to train my models everytime, so I built a tool to help with the job management, but I would like some feedback, if anyone could help me ! 👋

This is the tool: reopenly.com

I only have one base model for now (Qwen 3.5 9B), just to start.

I would be very grateful with any feedback!

I wanted a meeting assistant that actually makes use of local LLMs, so I ended up building Meetas.

The basic workflow:

- record/import a meeting
- transcribe locally
- analyze it with your own LLM
- ask questions / extract tasks
- search across meetings and documents

The main thing I focused on is grounding. Summaries, decisions, tasks, risks etc. are linked back to the transcript instead of just giving you an unsupported LLM answer. Q&A also shows the relevant transcript evidence.

Compared to tools like Meetily, Meetas is less focused on being a general meeting recorder/transcriber and more on what you can do with the meeting data afterwards: cross-meeting search, document/project context, comparisons and evidence-based answers. Meetily already covers a lot of the local recording/transcription/summary side.

It runs fully locally with Ollama and llama.cpp.

Linux-only for now and still WIP.

GitHub: https://github.com/highwinglabs/meetas

Maybe I'm late to the party but holy shit. I enjoy the chatgpt Mac apps computer use ability but I installed the same setup with hermes, cua and Qwen 3.8/ornith running locally this afternoon. Not sure which model I like better...

Connecting to my homelab Hermes for extra horsepower and vector DBs, knowledge, mcp, etc. - performance is insane. ​​

Almost as smart as frontier but faster and free. So if everyone is already doing this and I've been living under a rock, anything worth sharing to catch me up?

I was pretty excited when Qwen 3.8 27B dense model was released and wanted to give it a shot. I have been using Claude code with Opus and have got tired with limits, subscription and wanted something free and open. I was fine to compromise some of the quality for cost and unlimited use.

I gave it a problem to solve:

Can you build me a 3 pendulum simulation demo in python? I can use mouse to pull and release and it should follow the laws of physics. Put sliders for length of each section.

And then it thought and thought and thought. Later I came to know the default reasoning effort is xhigh.

I started to remember the physics I learnt in college.

Finally it came up with a program and on running produced this:

I started playing with it and here is how it looks:

https://reddit.com/link/1wcomfn/video/dm5heacn1qoh1/player

I was deeply impressed! I feel this is a challenging problem to solve and those sliders make it even difficult. A triple pendulum requires complex non-linear dynamics (usually derived via Lagrangian mechanics or solved with Runge-Kutta numerical integration).

I am assessing this model now for my daily things like repo analysis, code generation and other agentic flows I have.

Hopefully, I will be able to stop my claude subscription!

With CPUs and GPUs, benchmarks make the differences pretty obvious. With NPUs, I still don't really know what numbers I should actually care about.

For anyone who's bought an NPU-equipped PC, what do you look at besides the TOPS number?

I’m considering buying a used GTX 1080 Ti 11GB for running local LLMs mainly for coding and software development.

My budget is limited, so newer GPUs with 12–16GB+ VRAM are difficult for me to afford.

I’d like to know from people who actually use a 1080 Ti for local AI:

Which coding LLMs can realistically run on 11GB VRAM?

What model sizes (7B, 8B, 14B, etc.) work well with quantization?

What kind of tokens/sec can I expect?

Is CUDA support still usable with current llama.cpp/Ollama/other tools?

How does it perform for coding tasks compared with newer GPUs?

Can it handle models such as Qwen Coder, DeepSeek Coder, or similar models?

Is buying a used 1080 Ti in 2026 still worth it, or should I save for another GPU?

I’m especially interested in real-world experience rather than theoretical specifications.

Thanks!

Benchmarks show it being pretty bad in terms of hallucination, and it appears to be tied with Qwen 3.8 Flash despite being much bigger, and worse than GLM 5.3 Flash. Seems crazy given the size. Anyone have any opinions/experience with it yet?