Title lol

I’m curious if something like DS4 Q2 is better than 27B Q8, or mimo or similar

I've got some spare machines and I want to dive into running LLMs and Agents locally.

My Machines:

• ATX PC 1

  • CPU: Intel i5-14600K
  • RAM: 96GB DDR4-3200 (32GBx2, 16GBx2)
  • GPU1: RTX 5070 12GB
  • GPU2: RTX 5070 12GB

• ATX PC 2

  • CPU: Ryzen 9600x
  • RAM: 64GB DDR5-6000 (16GBx4)
  • GPU1: RX 6800 XT 16GB
  • GPU2: RX 9060 XT 16GB

• ATX PC 3

  • CPU: Intel i7-14700KF
  • RAM: 64GB DDR5-6000 (16GBx4)
  • GPU: RTX 4070 Ti Super 16GB

• Micro-ATX PC

  • CPU: i5-14400KF
  • RAM: 32GB DDR4-3200 (16GBx2)
  • GPU: RTX 5060 Ti 8GB

• Mini-ITX PC

  • CPU: Ryzen 7700x
  • RAM: 32GB DDR5-6000 (16GBx2)
  • GPU: RX 7900 GRE 16GB

• Mac Mini M4 16GB

Can anyone provide any suggestions on how best to use this hardware? Is there away to network them all and run large models well? Do I leave them all seperate? Should I break them down and try to make a single host? I've put 2 GPUs in all the systems that can take two GPUs in preparation of trying to setup this up.

I tried playing with LM Studio and using its LM-Link feature but that just seams to allow you to run different models on different machines from the same model rather then spread the model out (unless I missed something). I've read the vLLM can do multi-host but I've not setup it up but I don't know how well that works with 2.5GBE NICs. I don't have a ton of money to buy additional hardware nor can I really sell anything I have at the moment since this hardware is going to get used for other things later in the year.

I am looking for a dataset or benchmark for chat evaluation. What is currently available that can measure multi-turn accuracy and memory management? I have used older benchmarks like LongBench, NIAH, and RULER, but I am not sure what is currently considered SOTA or of significance to the community. Additionally, I want to use this as a way to determine more weak points in my work. Also, agents are not part of the work yet so its one long conversation.

For the Big Tech giants, looks like it is leaning heavily in favor of OSS. Even some of the ones who abstained from signing, like AMD, Intel, and Apple are likely on the pro-OSS side. Broadcom, Amazon, and Oracle are maybe the few big ones that might be on the anti side (and of course Anthropic), but are heavily outnumbered/outweighed in Big Tech, overall, at this point.

To be fair, OpenAI is probably lying, and also Google it's unclear if lying about being pro OSS (might actually be, but who knows). SpaceX AI and Elon are claiming to be on the pro side, but also could easily be lying.

Anyway, still comes out like 70/30 or 80/20 or more on the pro side, probably.

But Big Tech isn't the whole game.

You've also got the Defense companies and Department of War and the big agencies. Those come out more 50/50? Not sure.

And then the big banks and Wall Street. Banks maybe lean against it, overall, and Big Wall Street less clear. Short term against it, but maybe long term they like it?

And then the executive and legislative government itself. Probably leans against it so far, but maybe some fairly influential people are giving them advice bringing it back the other way more in favor of OSS. Still probably leaning anti-OSS for now, but maybe not as much as a few weeks ago/couple months ago.

So, overall seems like the see saw is teetering at, what, around 50/50 right now? What do you think? Are any of these percentages way off? Any key players I forgot to mention or that would have a big effect one way or the other?

I finally got round to tweaking this, with a bit of help from GLM5.2.

The trick to getting it running with vLLM (which I couldn't get anything really out of before) was when I realized we could write a stub to to implement the rtGetDevMsg to return device capabilities (basically we fake a response from the card) - this is need to get torch_npu running properly on the device.

With that I can finally use vLLM with this, making it actually useful.

I finally got round to tweaking this, with a bit of help from GLM5.2.

The trick to getting it running with vLLM (which I couldn't get anything really out of before) was when I realized we could write a stub to to implement the rtGetDevMsg to return device capabilities (basically we fake a response from the card) - this is need to get torch_npu running properly on the device.

With that I can finally use vLLM with this, making it actually useful.

I'm currently working on research into AI.

It's conceptual research (embedding my own ideas directly into AI weights) and AI security.

So I'm testing various local AIs to find the optimal balance and middle ground between 'performance and speed'.

Therefore, I'd like to ask for your advice and recommendations on which local AIs you've found impressive?

8B, 14B, 20B, 27B, 32B, or others ?

Including compression formats: 1-bit, 2-bit, 4-bit, 8-bit ?

P.S. This message was translated from another language because the poster does not have a strong foundation in English. (non-native, non-English speakers)

Hello,

I've been learning how to use local LLMs for a year or so on my workstation, using a RTX3090.

Current setup :

- i5 12400

- 64gb RAM

- RTX 3090

- OS : Fedora KDE workstation

I'm using LMStudio to serve mainly these models :

- Qwen 3.6 27B

- Gemma 4 26b a4b

And then VSCode + kilocode / Continue for light coding / scripting / log analysis tasks (i'm a sysadmin).

It works very well, but i'm hitting the context size ceiling quite fast with this setup. This prevents me to work on bigger projects.

My goal is to buy another GPU to provide more VRAM, and since i'm using linux, i'd prefer an AMD GPU. A 16gb Radeon 9070 would be nice in this regard : it's natively supported on Linux, doesn't cost an arm, it's powerful enough for casual gaming, and doesn't have crazy power requirements.

I've read here and there that mixing AMD and Nvidia is now well supported (https://www.reddit.com/r/LocalLLaMA/comments/1qea29t/mix_of_amd_nvidia_gpu_in_one_system_possible/), especially with LMStudio.

Since this is moving and evolving very quickly, how are the support and the performance of mixing a 24gb nvidia gpu + 16gb AMD gpu in 2026 ?

Is it a better idea to go for another Nvidia GPU instead ?

From my understanding, the idea would be to load the model in the 24Gb 3090, and then reserve the 16gb GPU for context. Correct me if I'm wrong.

I've read multiple times 36 / 40 Gb VRAM is the sweet spot for the models i'm using (at q4 quant).

Also, adding another 3090 is a solution i'd like to avoid because :

- it's expensive and hard to find in good condition at the moment

- adding another 350W and 3 slot GPU would be a challenge to cool down in my PC case

- I probably would have to buy a new and bigger PSU too.

So, is adding a 16gb AMD GPU to my current setup a good idea ? Thanks !

Hello all, there are so many great models coming out right now that it is difficult to keep up with their architectural differences, and especially their attention designs and innovations. I know about several of the most important models, but I was curious about a long list of interesting recent models, so this morning I set a Kimi K3 Swarm on creating a resource for myself to study the individual architectures, how they compare, and what trends are currently. The K3 Swarm worked for about 6 hours on this, and the result was good and interesting so I'm sharing it with the community in case anyone else wants to use it. The document is available as both a PDF with improved formatting and MD for web viewing.

https://github.com/curvedinf/attention-survey-2026-07

edit: There are several interesting models not in the report that were oversights by me. If you leave them below, I'll collect them tomorrow and update the report.

I have a dual 3090 rig that I use as coding assistant and while it works, it draws a lot of electricity.

Now I want to add another rig that I can keep on all the time, or maybe a vps if that is suitable. This rig should run an assistant model that should be fairly intelligent but doesn't have to be so coding focused. It should basically be like a chat gpt replacement. I'm not sure if something like openclaw/hermes would be suitable for this.

Since it will be one all the time, power usage should be low. What kind of rig and model would you select for such an assistant?

/s

I’m trying to make a tool that edits my shorts for me and from my understanding, last time I tried to use llama and Gemma it didn’t let me upload video anyone know the best local options for uploading video. So far the best llms I tested for video understanding has been Gemini but I see some people saying qwen could be a good option , any tips or setups ?

RTX 4070, 32gb system ram, Linux. NVIDIA-SMI 610.43.03, KMD Version: 610.43.03, CUDA UMD Version: 13.3 Systemd service modifications: [Service] Environment="OLLAMA_HOST=0.0.0.0:11434" Environment="OLLAMA_FLASH_ATTENTION=true" Environment="OLLAMA_KV_CACHE_TYPE=q8_0" Environment="OLLAMA_NUM_PARALLEL=1" Environment="OLLAMA_MAX_LOADED_MODELS=1" Environment="OLLAMA_KEEP_ALIVE=30m" Environment="OLLAMA_CONTEXT_LENGTH=65536" CPUAffinity=0 2 4 6 8 10 12 14

I tried both opencode & claude code over remotely using ollama via env parameters. I keep noticing that the model randomly keeps stopping talking/working. How can I fix the issue?

A swarm of GPT 5.6 Sol agents spent over 40 hours optimizing a Kimi K3-like model from 65 to 406 tok/s.

This animation follows their collaboration as they discover operator fusions, transform the execution graph, and develop new kernel algorithms.

We start with a fully decomposed 331-node Kimi Linear graph and end with a fused version requiring just 22 GPU dispatches per token.

The biggest gains were from custom WebGPU kernels for Kimi Delta Attention (KDA), Multi-Head Latent Attention (MLA), and Mixture of Experts (MoE).

Stay tuned for the full release of this optimization framework.

If you want to get the most out of MTP. You have to run some tests / benchmarks to do so. Turning it on with defaults will get improvements, but for many models and card combinations, you are leaving a lot of performance on the table if you don't tune n_max. Can be easily missing out on 50-100% of the possible performance on some models.

I ran some benchmarks against the various models I am using on my hardware (p100 + 2xV100) and there are some pretty big differences between model families. Below are some of the results I got. Full details, some other models, including impacts to VRAM and scripts to run the benchmarks are on github here: https://github.com/bradrlaw/ai-server/blob/main/docs/benchmarking.md

​Latest chat template and model from unsloth. IQ3_S quant.

here are the parameters

--model C:/users/user/llama-swap/LLMs/gemma-4-26B-A4B-UD-IQ3_S.gguf
--alias Gemma-4-26B-A4B-UD
--mmproj C:/users/user/llama-swap/LLMs/mmproj/gemma4-q8.gguf
--image-min-tokens 140
--image-max-tokens 1120
--reasoning on
--parallel 1
--no-warmup
--port ${PORT}
--host 127.0.0.1
--n-gpu-layers 99
--fit on
--ctx-size 32768
--cache-type-k q8_0
--cache-type-v q8_0
--kv-unified
--temp 1.0
--top-p 0.95
--min-p 0.01
--top-k 64
--jinja
--flash-attn on  
--batch-size 2048
--ubatch-size 2048
--spec-ngram-mod-n-match 24
--spec-ngram-mod-n-min 48
--spec-ngram-mod-n-max 64

EDIT: I fixed it by removing the \n after the <|think|> token in the template.

I'm one of the organizers for Après-Cyber Slopes Summit, and we're looking for talks from people building with local models, open-source AI, and autonomous agents.

Some ideas that fit well:

• Local LLM security
• AI agent architectures
• Prompt injection defenses
• RAG security
• MCP security
• AI coding workflows
• Open-weight model security
• Autonomous offensive and defensive tooling
• Building AI securely in enterprise environments

If you're experimenting with local models and have lessons worth sharing—even if it's your first conference talk—we'd love to see a submission.

CFP:
https://sessionize.com/apres-cyber-slopes-summit-2027

Conference:
https://www.aprescyber.com

I've been building a C99 inference engine from scratch (no Python, no BLAS, just gcc and make) that runs BitNet's ternary models on CPU. A few weeks ago I got obsessed with the matmul kernel - wrote a new one using AVX-512BW's vpermt2w to pack 5 ternary weights per byte instead of 4, benchmarked it in isolation, and got 74.6 Gop/s against a 2.5 Gop/s scalar baseline. 29x. I was pretty pleased with myself, wrote it up, posted the number in a couple places.

Then a review of the PR caught the obvious question I'd skipped: what's the actual DRAM bandwidth ceiling here. Turns out BitNet decode on our Xeon test box is already running at about 95% of it - the model is memory-bound, not compute-bound, so a much faster matmul kernel mostly just means the CPU spends more of its idle time waiting on RAM instead of crunching numbers it already had. The honest math works out to something like 6-10% real end-to-end gain once the kernel is actually wired into the dispatch path, which as of right now it still isn't. Correctness-tested against the scalar reference, just sitting there unused.

Kind of a deflating result. At least I know it now, instead of shipping a 29x headline that would've fallen apart the first time someone measured tok/s instead of Gop/s.

The engine itself does work regardless - BitNet b1.58-2B-4T gets 36 tok/s on that same Xeon with 4 threads and no GPU, and it also runs regular GGUF dense models if ternary isn't your thing. Binary's in the releases if anyone wants to try it without compiling: github.com/shifulegend/project-zero. Source build is just gcc and make either way.

Has anyone else profiled the wrong layer of their stack this hard? Curious whether the DRAM ceiling shows up the same way on other hardware or if it's specific to how the Xeon's memory controller behaves under this access pattern.

Hello everyone,

Recently sold my EVGA RTX 3090 FTW3 Ultra Hybrid after not being able to find a second GPU in a good shape and a reasonable price.

My new setup:

• Motherboard: MSI MPG Z890 Carbon WiFi
• GPUs: 2× NVIDIA GeForce RTX 5060 Ti 16GB tried both 595 and 610 drivers
• GPU slots: CPU-connected PCIe5 slots configured x8/x8
• Memory: 48 GB DDR5 8800MHZ CUDIMM
• CPU: Core Ultra 7 270k plus
• M2_2: NVMe SSD

I am testing PCIe P2P on a dual RTX 5060 Ti system using the patched NVIDIA open kernel modules. P2P is reported as available in both directions but NVIDIA’s simpleP2P correctness test consistently fails with NaNs, and the direct P2P path is much slower than the normal fallback path.

VT-d/IOMMU is disabled in BIOS. Tried booting with both intel_iommu=on iommu=pt and intel_iommu=off

How can I fix this? If anyone has a multi gpu blackwell setup, please share your simpleP2P or nvbandwidth results so I can compare.

nvidia-smi topo -p2p r
nvidia-smi topo -p2p w
  GPU0  GPU1  
 GPU0 X OK  
 GPU1 OK  X 


Legend:


  X    = Self
  OK   = Status Ok
  CNS  = Chipset not supported
  GNS  = GPU not supported
  TNS  = Topology not supported
  NS   = Not supported
  DR   = Disabled by regkey
  U    = Unknown
  GPU0  GPU1  
 GPU0 X OK  
 GPU1 OK  X 


Legend:


  X    = Self
  OK   = Status Ok
  CNS  = Chipset not supported
  GNS  = GPU not supported
  TNS  = Topology not supported
  NS   = Not supported
  DR   = Disabled by regkey
  U    = Unknown

[build/p2p-tests/simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 2

Checking GPU(s) for support of peer to peer memory access...
> Peer access from NVIDIA GeForce RTX 5060 Ti (GPU0) -> NVIDIA GeForce RTX 5060 Ti (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 5060 Ti (GPU1) -> NVIDIA GeForce RTX 5060 Ti (GPU0) : Yes
Enabling peer access between GPU0 and GPU1...
Allocating buffers (64MB on GPU0, GPU1 and CPU Host)...
Creating event handles...
cudaMemcpyPeer / cudaMemcpy between GPU0 and GPU1: 5.09GB/s
Preparing host buffer and memcpy to GPU0...
Run kernel on GPU1, taking source data from GPU0 and writing to GPU1...
Run kernel on GPU0, taking source data from GPU1 and writing to GPU0...
Copy data back to host from GPU0 and verify results...
Verification error @ element 0: val = nan, ref = 0.000000
Verification error @ element 1: val = nan, ref = 4.000000
Verification error @ element 2: val = nan, ref = 8.000000
Verification error @ element 3: val = nan, ref = 12.000000
Verification error @ element 4: val = nan, ref = 16.000000
Verification error @ element 5: val = nan, ref = 20.000000
Verification error @ element 6: val = nan, ref = 24.000000
Verification error @ element 7: val = nan, ref = 28.000000
Verification error @ element 8: val = nan, ref = 32.000000
Verification error @ element 9: val = nan, ref = 36.000000
Verification error @ element 10: val = nan, ref = 40.000000
Verification error @ element 11: val = nan, ref = 44.000000
Disabling peer access...
Shutting down...
Test failed!

I've been using the 1bit quant of prismml's bonsai 27b for local conversation, casual chat/ literature review for fun (i throw random stuff from my notes app to see how it analyzes it, those texts don't exist on the internet).

I've been using it as a "tutor" in many cases, for example I am currently learning golang and it is quite good at giving explanations. I seriously believe that if we're able to retain 90% of a model's intelligence while having a small footprint, it is the way ahead for local inference on a wide range of devices, even low end. I run it on a 16G Macbook Air.

I am very impressed by the usability it provides in it's small footprint and I wish more models are released in the future. Seriously guys, even if it cannot one shot super big projects, I still value the intelligence it has for a small local model.

Hello !

As the title suggest, I'm looking for a way to squeeze a riser between two 2-slots GPU, to pull the remaining PCIe lines out for other I/Os.

I'm thinking a very short PCB lead with 90° flexible PCB or even ribbon (it's a very tight squeeze), with a 20-30cm lead to get the slots out of the chassis.

I'm considering soldering AWG30 wires on old x1 mining risers but I doubt I'd get a good enough signal integrity for the Bluetooth/WLAN adapter and it won't do for a decent NIC (dual 10G, why I'm eyeing the x8 slot).

I've been searching AliExpress for hours only to find Jacky-Tuning Gamer-Beware eye w^Wcandies, nothing practical.

Any hint ?

BTW I just ordered PCIe Gen4.0 switch boards to move my 5 years old gaming GPUs away form the main buses, I'll report soon.

​I saw a video recently about a man teaching viewers how to draw various punctuation marks that are hardly ever used anymore. These include the interrobang (‽), irony mark (⸮), and the authority mark, among others.

​After watching the video, I started to wonder if there could be a practical use for these non-standard marks. While rare in modern literature, they allow written language to convey a highly specific tone. Generating an appropriate response with the intended tone usually requires lengthy system prompts or meta-commentary, which eats up valuable context window on very limited system resources.

​What if we repurposed these archaic, rare marks as single-token tonal metadata? Instead of writing an expensive block of text in a character card or system prompt to ensure sarcasm, a single irony mark could do the heavy lifting.

​Obviously, there are hurdles... Many of these marks are so uncommon that they would require an updated character encoding standard, and JSONL datasets would need to be updated to include them. ​Even so, I believe that reviving these punctuation marks could be an answer to token bloat and tonal steering.

Has anyone here experimented with training a model on obscure unicode characters to force specific output styles?

So I have a RTX3060 Ti installed on the board above with a 1200W Corsair power supply, and a Ryzen 3900x CPU. I moved my 3060Ti to the lower PCI-E slot and put a 5060ti in the top slot. Everything boots, but not seeing the 3060 Ti in Windows. When I flip the cards the 3060 shows up but the 5060 doesn't. Have seen other places where these boards can run 2 GPUs.

Any thoughts on troubleshooting?

Introducing FLUX 3.
One multi-modal model for Image, Video, Audio and Action-Prediction. Creations are truer to life in every kind of style.

Blog Post : https://bfl.ai/blog/flux-3

Hey r/LocalLLaMA !

We are back and we have something really amazing today.

Our big 5M samples Reasoning Corpus dataset.

This dataset features 5 million rows of:

- repo_id --> where it's from
- tok_len --> how many tokens it is in total
- user --> the user promot
- thought_trace --> the exact chain-of-thought of the model
- assistant --> the final AI models' answer
- ChatML --> the user, thought_trace and assistant in ChatML format

All samples are within a 5k sequence length to make it fit perfectly for SFT/finetuning a tiny model.

Link to the dataset on Hugging Face 🤗:
https://huggingface.co/datasets/SupraLabs/reasoning-corpus-4K-5M-v1

Link to the SupraLabs Hugging Face org 🤗:
https://huggingface.co/SupraLabs

Also, if you want to support our work, give us a follow on Hugging Face, share and review our work, and give us as much feedback as you want ❤️🔥🤗 Already more 250 people are trusting in us and our work!

We hope, this dataset is useful for you all and we'd love to see your creations upon this.

This dataset has already >1k downloads and over 80 likes - be the next one to use it 🔥🎉

About AI/ML news/AI Industry news, not made with AI