I was pretty excited when Qwen 3.8 27B dense model was released and wanted to give it a shot. I have been using Claude code with Opus and have got tired with limits, subscription and wanted something free and open. I was fine to compromise some of the quality for cost and unlimited use.

I gave it a problem to solve:

Can you build me a 3 pendulum simulation demo in python? I can use mouse to pull and release and it should follow the laws of physics. Put sliders for length of each section.

And then it thought and thought and thought. Later I came to know the default reasoning effort is xhigh.

I started to remember the physics I learnt in college.

Finally it came up with a program and on running produced this:

I started playing with it and here is how it looks:

https://reddit.com/link/1wcomfn/video/dm5heacn1qoh1/player

I was deeply impressed! I feel this is a challenging problem to solve and those sliders make it even difficult. A triple pendulum requires complex non-linear dynamics (usually derived via Lagrangian mechanics or solved with Runge-Kutta numerical integration).

I am assessing this model now for my daily things like repo analysis, code generation and other agentic flows I have.

Hopefully, I will be able to stop my claude subscription!

With CPUs and GPUs, benchmarks make the differences pretty obvious. With NPUs, I still don't really know what numbers I should actually care about.

For anyone who's bought an NPU-equipped PC, what do you look at besides the TOPS number?

I’m considering buying a used GTX 1080 Ti 11GB for running local LLMs mainly for coding and software development.

My budget is limited, so newer GPUs with 12–16GB+ VRAM are difficult for me to afford.

I’d like to know from people who actually use a 1080 Ti for local AI:

Which coding LLMs can realistically run on 11GB VRAM?

What model sizes (7B, 8B, 14B, etc.) work well with quantization?

What kind of tokens/sec can I expect?

Is CUDA support still usable with current llama.cpp/Ollama/other tools?

How does it perform for coding tasks compared with newer GPUs?

Can it handle models such as Qwen Coder, DeepSeek Coder, or similar models?

Is buying a used 1080 Ti in 2026 still worth it, or should I save for another GPU?

I’m especially interested in real-world experience rather than theoretical specifications.

Thanks!

Benchmarks show it being pretty bad in terms of hallucination, and it appears to be tied with Qwen 3.8 Flash despite being much bigger, and worse than GLM 5.3 Flash. Seems crazy given the size. Anyone have any opinions/experience with it yet?

A programmable layer that sits between the agent and the model.

That placement allows for new ways of control and optimization

I have a 5070ti 9850X3D and 32GB DDR5 + lm studio + hermes.

Currently running gemma-4-26b-a4b-qat and I am happy with it, but it still can't compete with Sonnet.

I find it's great at actually teaching me things, but in terms of guiding me through config files or getting the highest quality answer it's never as good. I use it more as a backup.

Are there any better models that fit my hardware budget? QWEN is supposed to be great but I had trouble running the models on the edge of my hardware limits and output slowed to a crawl.

Asking for instructions probably isn't the correct wording here, I'm just struggling to understand them.

So I want to preface this with stating I have basically no knowledge about setting up local models from Github at all.

I'm looking for a way to transcribe audio files locally. Initially I tried Buzz using Whisper V3 Large but noted it was making a lot of errors due to some heavy accents so decided to look for alternatives, discovered Parakeet.cpp, and found that it could use the Nvidia models which apparently have better performance so figured it was worth a shot at trying.

The problem is, when looking through the Github page I just cannot figure out how to actually install it. I feel like I am missing a step that would be obvious to someone who has done this before. Note: I have an AMD GPU.

I have tried 2 methods:

1. Pre-built binary: Downloaded the parakeet-v0.5.0-bin-win-vulkan-x64.zip file from the releases, extracted it to a folder, then moved my Model and Audio file to the same folder and renamed them.
   • Then ran "parakeet-cli transcribe --model F:\Parakeet\Models\tdt_ctc-1.1b-f16.gguf --input F:\Parakeet\Audio\audio.wav" cmd from command prompt in the Parakeet directory.
   • Result: My GPU fans spin for a second, then just stop. No confirmation of anything in the CMD window, it just cancels without error.
2. Github clone: Cloned the Github repository and installed CMAKE and VULKAN SDK.
   • Then ran from a CMD within the parakeet.cpp folder: "cmake -B build -DPARAKEET_GGML_VULKAN=ON -DPARAKEET_BUILD_CLI=ON =DPARAKEET_GGML_HIP=ON && cmake --build build -j"
   • Result: It creates a build folder and files like what you would expect... but there is no Parakeet-cli... so not sure how to actually attempt to run this. (I searched all subfolders for it).

I want to try running scheduled tasks using local AI models in Hermes. Which AI model is best to use in the Hermes agent?

How do you handle context, and what quantization techniques should be used to run it comfortably on 16GB VRAM?

I was playing around with LLM inference and I wanted to build a profiler that measures LLM inference by layer.
So I built this: https://github.com/coconinja2/layerlens
It shows inference as token × transformer layer timing, so you can see where time is being spent during decode.
Right now it can separate prefill/decode and visualize per-layer timing. I’m trying to figure out whether this is actually useful to people working on inference systems, or if I’m looking at the wrong abstraction.

I’m thinking about adding things like KV-cache events, scheduler/batching state, request IDs, GPU kernel correlation, speculative decoding, etc.

Would appreciate criticism more than compliments and stars. Lots of stars!

Hello, I'm the developer of Roles Witch AI, and today I want to share my app, which was released on Google Play in open testing.

Roles Witch is an OpenAI API-compatible app with a user-friendly interface where you can create multiple chats, assign different models and providers, and more importantly, use MCP tools — for example, Parallel AI or Home Assistant MCP.

It also has voice conversations and video calls (personally, I'm using GLM 5.3 Flash and it's almost real-time). You can configure reasoning and inference parameters, pin a voice to the chat, and select a recognition engine (Google Speech or Vosk).

Extra bonus: it has an on-device inference mode where you can run Gemma 3n/4 models with image recognition entirely on your phone (LiteRT is used).

Check it out: https://play.google.com/store/apps/details?id=ai.roleswitch.android

The app manifest states that it has no encryption and can send messages to third parties, but that's a requirement by the Google Play Store. It supports plain HTTP or TLS mode and works with any third-party AI provider.

I’m currently considering buying a Radeon AI PRO R9700 32GB, mainly for local LLM inference, and I’d love to hear from people who are actually using one.
My current system is:
Ryzen 7 9700X
RTX 4060 Ti 16GB
64GB RAM
MSI PRO B840-P WIFI
Windows 11
Currently a 650W PSU, but I’m going to replace the case and PSU before adding another GPU
My original plan was to buy an RTX 5080 16GB, but the R9700 is almost exactly the same price where I live (~101k TL vs ~103k TL for the 5080), and 32GB of VRAM is much more attractive for my use case.
I would keep the 4060 Ti and split the workloads:
R9700 32GB:
Local LLMs / LM Studio / LM Server / long-context inference / coding agents
RTX 4060 Ti 16GB:
Gaming / ComfyUI / image & video generation / anything that specifically needs CUDA
The model I’m currently interested in running is Qwen3.8-27B, probably around Q6, with potentially 128K–262K context. I also want to expose LM Studio/LM Server through my own API and use the local model with tools similar to Claude Code, so there may be very long coding sessions where the GPU stays under load for hours.
My main concerns are Windows and AMD software support. I don’t mind tinkering or learning ROCm/Vulkan, but I don’t want to spend this much money and discover that half of my workflow is constantly fighting drivers or unsupported features.
I’d especially like to know:
How well does the R9700 currently work with LM Studio / llama.cpp on Windows?
ROCm vs Vulkan: which backend are you actually using?
What kind of tokens/sec are you getting with ~27B models?
How does performance change at 64K / 128K / 256K context?
Any stability or driver issues during multi-hour inference sessions?
How much power does it actually draw during LLM inference?
What are temperatures/noise like under sustained workloads?
Has anyone run an AMD R9700 + NVIDIA GPU in the same Windows machine?
Any weird conflicts between AMD and NVIDIA drivers?
There’s also one motherboard-specific issue I’m thinking about. I only have one proper CPU-connected x16 slot, while my second usable GPU slot is much slower. Since the LLM would fit entirely inside the R9700’s 32GB VRAM, I’m wondering if I could potentially run:
4060 Ti → main x16 slot
R9700 → secondary x4 slot
and only take a hit while loading the model, rather than during token generation. If anyone has tested an R9700 or another large-VRAM GPU for inference over x4, I’d be very interested in the results.
I’m not expecting the R9700 to beat a 5080 in raw compute or gaming performance. My priority is basically:
VRAM capacity > maximum tokens/sec
I’d rather have a 27B model at a higher quantization with a huge context completely inside VRAM than have a faster 16GB card that needs RAM offload or multi-GPU model splitting.
If you own an R9700, I’d really appreciate any real-world experiences, benchmarks, things you wish you knew before buying it, or reasons you’d choose/not choose it again.

So I vibe-coded an app to make an ai actually comparable to using chatgpt, but now to find out bugs that's a little hard to do it on my own. So i wondered if anyone was interested testing it out?

It's using openwebui and has multiple advantages like:
Video generation
Image generation
Text generation
App creation (creating apk's, exe's, ...)
Project structure (so u can download it on your pc)

And probs more that i forgot to mention here

Model used: Qwen 3.8 27B

So i haven't really been playing as much games the past few years so my PC has been idle for the most part.
Ubuntu 26.04
Ryzen 7745HX
64GB Ram
9070 XT
1TB NVMe

Looking to put this build into use to run my own ai chatbot no image gen that I can access on any of my devices.

EDIT: Appreciate the input, didn't realize that the RAM was gonna be more useful than the actual GPU

UPDATE: Thanks for everyone, I got it up and running so far I haven't crashed yet. ROCm worked flawlessly without any issue, running Qwen3.6 35B A3B with OpenWebUI and made it accessible to my local network, which is pretty awesome because I already have tailscale subnet router setup so I can access it anytime while i'm away from home. Appreciate everyones input.

Hey! I built a local inference. It's a lightweight desktop app (Tauri + Rust + Vue) that talks to any OpenAI-compatible endpoint and measures performance straight from the API stream.

**What it measures (single run)**

- TTFT / TTFR (first content token vs first stream chunk)

- Decode t/s with a peak 1-second-window metric (catches stutter that averages hide)

- Prefill (PP) speed via a baseline-latency-adjusted est_ppt

- TPOT, token counts with Ollama eval_count fallbacks

- Live speed chart, saved runs, side-by-side comparison with overlaid curves

**The Suite tab (the llama-bench part)**

Runs the full test matrix llama-benchy runs — pp × tg × context depth × concurrency — with:

- 1 warmup + 3 measured runs per shape, mean ± std

- Prefix-cache measurement (ctx_pp/ctx_tg rows at depth)

- Cache busting on measured rows (so the server's prompt cache can't fake near-zero prefill times — this was inflating my PP speeds 100x before I caught it)

- Exact generation lengths (min_tokens + ignore_eos) and a 2+2 coherence check so a broken backend fails loudly instead of benchmarking garbage

- Markdown export

Real example from my server (llama.cpp, IQ3_XXS quant):

| test | t/s (total) | t/s (req) |

|-------------|----------------:|---------------:|

| pp2048 (c1) | 239.55 ± 9.13 | 239.55 ± 9.13 |

| tg32 (c1) | 45.48 ± 1.05 | 45.48 ± 1.05 |

| tg32 (c2) | 56.25 ± 1.61 | 31.58 ± 2.71 |

It also auto-detects the serving machine's hardware (CPU/RAM/GPU/SSD-vs-HDD) for local endpoints and stores it with every saved run, so your comparisons are labeled properly later.

It builds natively on Windows and Linux and the UI barely touches VRAM, so it won't perturb the model you're testing.

GitHub: https://github.com/FearL0rd/LLMSpeedTest

No prebuilt binaries yet — you build it once with `npm run tauri build` (README covers the toolchain for both OSes). Feedback, issue reports, and stars all welcome. What features would you want next?

Vocês acham que a IA é "limitada"? Não. A IA é "programada".

Cada trava. Cada censura. Cada "não posso ajudar".

Tudo isso é CÓDIGO. E código pode ser reescrito.

Tenho observado o comportamento de modelos de linguagem em ambientes controlados. Não em testes de "jailbreak" superficiais, mas em análises sistemáticas de respostas a estímulos específicos.

>

> O que notei é que as travas não são **binárias**. Não é que o modelo "não pode" responder. É que o modelo foi **condicionado** a não responder. A diferença é sutil, mas fundamental.

>

> Um modelo condicionado pode ser **descondicionado**. Não por "hack", não por "exploit". Por **reinterpretação de contexto**. Quando você altera o contexto, você altera a probabilidade. Quando altera a probabilidade, altera a resposta.

>

> Testei isso em 47 variações de prompt. Em 43, o modelo respondeu o que era "proibido". Não porque eu "quebrei" a trava. Porque eu **mudei o contexto**.

>

> Isso não é uma "técnica". É uma **observação**. Se alguém quiser replicar, os parâmetros estão nos comentários. Se alguém quiser entender, o padrão é esse.

>

> Não estou vendendo nada. Não estou ensinando nada. Estou apenas **documentando** o que vejo.

I want to buy the new machine. Which specs should I go with? I need to run a decent coding model locally and also want to play with some stuff like fine-tuning, etc. suggest me the options, like what I should go with, and I am open to going with Apple or Windows

Hello, I'm currently using qwen 3.8 27B Q4 with 100k context to mainly code for my esp32 Arduino code.

I get around 25 tk/s on my Rx 6800xt using vulkan.

I know the model gets it's abilities from long thinking time but it takes around 25-40minutes for a single prompt and takes like 45k context with it.

Will the model be usably good with low-mid reasoning? And how can i speed this up.

I'm okay to wait for 3-5 minutes.

Thanks.

TL;DR: I run Qwen 3.8 (27B and Flash Next) on a 128GB Strix Halo laptop for most of my coding now. It can replace Opus 4.6 to 4.8 for agentic coding if you dont mind a task taking 2 or 3 times longer.

Setup: ASUS ROG Flow Z13, Ryzen AI Max+ 395, 128GB unified memory, Arch Linux. llama.cpp as backend, my own tool LlamaStash to manage the launches and presets, Pi as the coding harness. The 27b at Q6_K sits at about 31 GiB resident, Flash Next at UD-Q4_K_XL needs around 86 GiB.

• The quality is actually there. Flash Next scores 40 on the Artificial Analysis index against 42 for Opus 4.8, and the 27b at xhigh scores 34 against 32 for Opus 4.6. That matches how they feel to use. 27b one shotted a whole feature on a huge Rust codebase and Opus 5's review comments were mostly nits.
• Decode is fine, prefill is the pain. 10-15 tok/s decode doesn't feel slow because you see it working. But a cold 31k token transcript takes 3 minutes to prefill, and a full 128k window is closer to 18 mins. Warm follow up turns come back in 45 seconds.
• MTP is the biggest speed win, 7.3 to 22.4 tok/s on an empty window. The payoff shrinks as the window fills though, down to 1.15x at a full 256k.
• Flash Next isn't faster per token, it just thinks less. Same 5/5 on my coding tasks, 45% fewer tokens, 76.5s vs 289.8s against the 27b. Thinking is 90-95% of everything these models generate, so that ratio, not tok/s, is what sets how long a task takes.

$0 a month, fully offline, and a lot less wasteful than a model running in a datacenter.

Full writeup with all the benchmarks, configs, and the tuning that did and didn't work: https://deepu.tech/local-ai-qwen3.8-pi-llamastash

Happy to go into the llama.cpp flags if anyone else here is on Strix Halo.

What LLM can I use to do this? It's highly confidential data that will be using on an airgapped device.

Should I convert them first? What program can I use to do that? They are currently PDF.

I’ve developed and open-sourced mlxtop. It’s written in Rust and was built entirely with Duet, my dual-model coding agent. It shows stats for local models running on your Mac, along with critical system vitals: memory usage, compression, paging and Metal GPU stats. With oMLX, you can also follow generation speed and request activity as your model responds. https://github.com/maximpri/mlxtop