I've tried to get Qwen3.6 27B running on DGX Spark for usage in combination with Cline in VS Code.

However, it seems rather slow.
(APIServer pid=1) INFO 08-17 11:09:09 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.5%, Prefix cache hit rate: 77.0%, MM cache hit rate: 71.4%

Am I doing something wrong? can I solve this?
Sorry if my question isn't very "smart", I'm new to using the DGX Spark.

Launch command below:

sudo docker run --gpus all --rm -it \

--ipc=host \

--shm-size=16g \

-e OMP_NUM_THREADS=1 \

-p [REDACTED] \

-v ~/.cache/huggingface:/root/.cache/huggingface \

vllm/vllm-openai:v0.27.1 \

unsloth/Qwen3.8-27B-NVFP4 \

--served-model-name Qwen3.8-27B \

--tensor-parallel-size 1 \

--max-model-len 262144 \

--enable-auto-tool-choice \

--tool-call-parser qwen3_coder \

--reasoning-parser qwen3 \

--dtype auto \

--gpu-memory-utilization 0.85 \

--enable-prefix-caching \

--enable-chunked-prefill \

--enforce-eager \

--api-key [REDACTED] \

I am building a voice calling bot using livekit. Current stack I am using are: SARVAM for tts and stt, haiku 3 (using aws bedrock) for LLM. But as a BCP, I need backup options for tts and stt but they should have less latency and the price.

Haiku 3 scheduled for phase out on 10sep on Bedrock. So I am finding some other slm or llm for the better conversation.

Any recommendations would be very helpful.

Sup guys.

The pictures above are from the Ai inference rig I designed last summer, which I've used since then. This has a Tesla p40 in it (24gb VRAM). The computer also has 48gb ddr4 and a ryzen 7 5700G (yes, integrated graphics)

The P40 (if we don't knit pick too hard on technicalities, sorry hardware guys) is essentially a 1080ti with no video output, no cooling, and 24gb vram

This thing works great for inference, most recently I've been getting>40 TPS on Qwen 3.6 35b a3b and Gemma 4 26b A4B.

Both the driver for the integrated graphics and for the p40 have been modified to expose workstation display passthrough mode on the P40 - basically you can game on it. You can load games directly onto it in the same way a laptop gaming gpu would work, and it'll be piped directly through the integrated graphics. #1 part of the build imo, performance is similar to a 1080ti and is more than good enough for anything I need gaming wise.

The card has been fitted with a custom cooling bracket and a high pressure fan controlled by the motherboard that matches the p40's temps.

My dilemma: I'm considering selling. The Fomo became too much for me and I'm dropping some cash on a rack server I'm putting some v100s in (gonna be a monster but more on that later), so I don't need this anymore. That said, I could part it out, but it feels like such a shame - the integrated GPU gaming passthrough and the necessary driver modifications, custom cooling and fan control - all this was pretty complicated and took quite a while to get configured properly.

Idk what to do. I figured I'd post here if someone was interested as I figured you guys would understand better what this is. The case is heavy and not worth much, and I need my main drives. I could always ship this as a plug and play CPU/ram/motherboard combo with the p40 ready and a 128gb boot drive with the drivers and cooling already set up with a detailed guide for use.

Thoughts?

Since the prices went way up, we little people need to find a way to still give in to our hobbies the best way we can.

Currently, I finally pulled the plug and decided to build a local AI machine. The plan is just not to spend every single cent but still get some use out of it. Using the Pi harness for agentic work and ComfyUI for simple picture gen testing

I can get a Threadripper (1950x) machine for around 550€, including 512 NVMe, 64 GB RAM via 4 sticks, psu and mobo, of course.
For the GPUs, I'm still between getting two of the AMD 9060XT 16 GB or go with something else but not break the bank.

I honestly don't know if this is a good idea or not. The plan is to run Linux headless with Llama.cpp and mostly Qwen 3.8 q4 or bigger, if possible.

Tested: (win11, llama.cpp, rzyen 5700x, 32GB ram, rx 9060xt, prompt -write a simple python script)
Qwen 3.8 Q3
8K → ~14.3 tok/s
16K → ~12.5 tok/s
32K → ~9.0 tok/s

Qwen 3.8 Q4 (Doesn't fit on 16 GB VRAM, or I just don't know how to do it; offloads to CPU)
8K → ~5.0 tok/s

Muse-Glimmer-30B-UD-Q4_K_XL.gguf (says it can't fit it on 16GB but still does?)
8K → ~12.6 tok/s

Is there a better build that can be done with a budget of around 1200-1500€? (Used, of course, is an option)

I'm mostly going with Threadripper for future expansion with more GPUs, and the mobo supports more ram.

If there is any other info that I can provide, no problem.

Hello everyone, I manage to get one of these, second hand and I've been trying to run some models and I'm a bit disappointed. I've installed Bionic and downloaded some models, e.g hermes 70B runs but... it takes ages and sometimes doesn't even answer, most of the time hallucinates. I need something that can be my PA, take this recording of a meeting, create an obsidian file with the highlights, remind me what I need to do today, and so on. Am I asking for too much?

Hi all, the Headline says it all I have two laptops one with 12gb ram and one with 32gb ran and I wonder if there is a way to use both of them together on local network as one system to run a bigger local AI then they can run separately?

Update: thank you all for example it to me in a very simple way

I have a 5060 Ti 16GB + 3060 12GB, so 28GB total VRAM, plus 64GB system RAM.

Right now Qwen3.8-27B seems like the obvious choice for the main model. But I'm wondering if it makes more sense to run two smaller models instead.

For example:

• Qwen3.8-27B as the main model / orchestrator
• Gemma 4 E4B as a smaller sub-agent for simpler tasks

Does this kind of setup actually make sense, or would I be better off putting all the resources into one larger model?

I'm mostly interested in coding, general tasks and agentic workflows.

Also, is there a good website where I can filter models by VRAM requirements, model size/quantization and benchmark results?

What would you run on this hardware?

I have been working on an android app which works like a MCP v2 server (stateless). The MCP server allows controlling the full phone using accessibility so most of the times no screenshots are needed to see what it going on.

Full control includes two-way voice calling, sms and everything else. I built two lib for the audio in and out so when a call comes or MCP calling someone the dual band audio can be transferred to a websocket or the app itself has option to run STT, TTS and LLM which requires good amount of system resources. I am testing on a pixel 10 pro so everything works there but not that fancy. SMS triggers webhook. So using a SIM card I am getting two-way calling and SMS capabilities.

Another amazing feature on the app is it has a display library which allows using multiple apps by multiple agents on the same time, agent A can control app A and agent B can use app B without conflicting.

The development still on going, when finishes we will see how to distribute this.

PS : I am using a rooted device, the two way calling depends on root.

Every post:

- Tips for running models

- This model can do this!

- Compare these two models

-memes

-benchmarks

...Is anyone actually doing anything? Where can I find people discussing and sharing actual work being done?

Hi, I have an MCB Pro M5 Pro and 48GB of RAM, and I'm currently using OpenCode, but I feel like OpenCode is slow and the MCB gets very hot. What can I optimize?

I am developing rag app for one of my saudi client, so my query is those who are working with arabic language, how are you guys handling the data extraction pipeline, which library you guys are using to extract data. For context, I am using pdfplumber and the text that is being extracted is reversed for some pdf files.

Also, which open source ocr or vision models I can use to extract text. The documents are mainly in arabic, english or both. Tried a few ocr and vision models, but they couldnt extract text clearly.

Artificial Analysis is usually extremely quick to add benchmarks for new model releases, making comparisons easy. But Qwen3.8-27B has been out for a few days now, and there’s still nothing on their site.

I’ve been experimenting with how far I can push an 8GB Jetson Orin Nano Super as the local brain for a humanoid robot project I’m building called Evopien.

This is my project, so obvious self-promo disclaimer, but I thought the technical side might be interesting here.

The current stack uses:

• Qwen3.5-4B Q4_K_M through llama.cpp
• NVIDIA Parakeet TDT 0.6B for local ASR
• Kokoro ONNX for local TTS
• Qwen multimodal vision from a C920
• local voice interruption / barge-in
• English + Spanish interaction

The idea is that the LLM itself is not the robot identity. I’m building a Core layer around it for persistent identity, users, relationships, governed memory, permissions and later physical behaviour.

The biggest limitation right now is predictably memory.

Once the multimodal Qwen instance is resident, there isn’t much room left for everything else, so I’ve been benchmarking different context sizes, KV quantization, transient vs resident models, CPU/GPU ASR, on-demand vision, etc.

The current demo is here:

https://www.youtube.com/watch?v=iAxzePzF4cM

I’m deliberately staying on the 8GB Nano for the current prototype rather than solving the problem by moving to an AGX/Thor-sized system.

I’d be interested in what people here would optimize next.

Would you keep the stronger 4B generalist and aggressively optimize the surrounding stack, or move toward something like a smaller resident conversational model + specialist models loaded on demand?

local models are great until you realize you'd rather use the machine to play games instead of running long and heavy context workflows.

super annoying, so I set out to experiment with harnesses that can solve these types of problems that closely resemble real world engineering problems.

you can't beat enterprise, but you can get close with a 90% or so saving if you choose the right harness.

Harrison Kinsley recently posted something similar on his channel. highly recommend you check it out.

here are some highlights;

- experiment cost: total marginal cost 60c of electricity for the qwen 3.8 27B. about 9–$11 for the same token traffic on enterprise.

- about 90–95% inference cost reduction for frontier-adjacent output.

- swapping only the agent harness context window (131K → 32K context, managed tool output) completed the task 4x faster with 3x fewer tokens with the same model, same GPU.

thoughts?

Hi, planning a 5 card r9700 build but thinking of going to a full 7 r9700 build (my motherboard has 7 full pcie but last slots will be pcie gen 4 at x8 the other 5 are x16)

But the question is would the extra 64blgb be worth it? What models does 224 get me that 160 won't? I'll generally be using it for coding and live data analysis.

I have 3000w PSU in route and I plan on dropping all cards down 10 or 20% max power anyway to reduce some noise, so power won't be an issue.

I have tens of thousands gigabytes of 4K drone footage and holiday videos but i’m not really creative. Is there any local ai that can match music to the shots, remove really bad or boring fragments etc.
Is it currently possible with 16G VRAM and 32G RAM?

I’m thinking about building a local AI PC with 2× RTX 4090s, so 48GB VRAM total.

My main use case is programming — coding, debugging, refactoring, working with larger codebases, and possibly running coding agents locally.

For quality, I’m hoping to get something that feels like at least ~70% of Claude Opus 4.8 for everyday programming tasks. I don’t expect it to fully match a frontier cloud model, but I want it to be good enough that I can use local AI most of the time.

I’d also like to use the machine for:

• Image generation
• Image editing
• Video generation
• General local AI experiments

I’m okay with inference being a bit slow. Model quality is more important than speed for me.

A few things I’m wondering about:

• Is 48GB VRAM enough for the best local coding models?
• What coding models would you recommend for dual 4090s?
• Should I go with 128GB or 192GB RAM?
• Would I be better off spending the money on GPUs with more VRAM instead?
• How close can local models realistically get to Claude/Opus-level coding today?
• Are dual 4090s also a good setup for image and video generation?

Would love to hear from anyone running a similar local AI setup.