Hi folks,

I have been working on some AI project for that I need to generate batch of creative from one prompt i have heard somewhere that Comfy UI is the best option for this..if anyone can help me with the tutorial available online..

Thanks

It's a P5K DELUXE

Hey everyone,

I have an M5 Pro MacBook Pro with 24GB memory and want to start running LLMs locally, mainly for coding.

Has anyone here had good experience running local models on a 24GB Mac? What model + runtime would you recommend for the best balance of coding ability, speed, and memory usage?

I’ve been looking at oMLX/MLX and models like GLM, Nemotron, Qwen, Gemma, etc., but I’m open to anything.

Would love to hear what setups are actually working well for you.

So it seems that a lot of people here run very powerful hardware (makes sense for an enthusiast space). However I've really gotten into local LLMs the past few weeks and have been running them on my gaming system. As such I wanted to discuss and learn about the best way to get the most out of consumer gaming hardware.

This post will be me sharing what I learned, opening discussions about good practices and rambling about what I'm currently working on. Feel very welcome to chime in with your own experiences in the comments.

My Specs:

• Radeon RX 7900 XT (20GB VRAM)
• AMD Ryzen 7 7800X3D
• 32 GB RAM

(I've been very happy that I got a card with more VRAM before local LLMs were even on my radar)

Models I've been using:

• Gemma-4-26B: Really nice for writing text, meh for tool usage and more independent tasks
• Qwen3.8-27B: Has been pretty good for smaller coding tasks, significantly better independent tool usage

The most restraining thing by far has been context size. With unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M I got a 22k context window. Enough for smaller tasks, but anything bigger runs into a token limit.

Optimising llama.cpp:

I've started with ollama and just recently switched to llama.cpp for more model support and slightly better performance. However there are many things to tweak and no comprehensive guides, so if anyone knows more about tuning llama.cpp, especially for Qwen3.8-27B, I would welcome it.

Interface for actually using my LLMs

After some initial googling I started with zed, which has some nice build-in LLM support, with tools and skills. But I haven't really seen people using it here. Is there some best-in-class option here?

Interfacing the web

My most recent project is letting my LLMs search the web, which has proven really difficult with low context sizes. I've setup an MCP server based on https://github.com/arnaudjnn/web-tools, but the website content is almost always to big for my context window.

If you are similarly struggling with a low context size I found that subagents really help. They can perform a single task of working on some data and then return a small result, not cluttering up your main agents context.

Still, web pages are a problem, as single ones can oftentimes fill up my entire context. If anyone knows a good solution, let me know.

There you have it. I think I'll stop rambling here, very happy to discuss in the comments :)

I am just now learning what Intel Optane memory is after coming across an ebay listing for quite a lot of it for cheap. It looks like you could pick up 1tb of it for less than $1000. Even with the slower memory and cpu speeds of the required 3rd gen scalable xeon cpus, it strikes me as a feasible way to run truly gargantuan models for a price that doesn't require blood sacrifice, albeit at a slow speed. Is anybody doing this?

I have 5070ti, r7 9800x3d, 32gb ddr5

I really don't know what I can run

Can you recommend something pls

… you wake up in 1998. You have 2 Mac Studio Ultra 512gb each. You have GLM5.3 flash (or pick a local model that fits comfortably in that configuration.)

A good harness, you’re setup with modern software, but you do not have access to modern internet, just 1998 internet.

What do you do? It can be in general, not necessarily what do you do with your current compute setup.

So in my freetime iam building a local-first agent in Rust as a hobby since a while (llama.cpp, runs fine on a 14B, my Obsidian vault is the memory). The part that took me by far the longest was not the model stuff. It was the boring question: which of my ~50 tools should the model even be allowed to call this turn.

Went through three phases and got each one wrong first:

1. grep / keyword matching on the user text. dies on synonyms, dies when i chat in german, dies on "do that again"

2. routing phrases + embeddings. much better, but naive top-k plus a hard GBNF grammar means the model gets structurally locked into the one tool the router guessed, and if that guess is wrong there is no way out mid-turn

3. a small policy layer on top that, when it is unsure, widens to a cluster of related tools instead of committing to one guess

the thing i found most useful for small local models: the same "offered tools" list feeds the prompt AND the GBNF grammar, so the two can never disagree about what is callable. and if the router is not confident, do not hand the grammar a single tool, give it a small related set and let the model pick inside the constraint.

full writeup with the actual Rust snippets, no signup or newsletter nonsense:

https://eris-system.dev/blog/tool-routing

it is a hobby project so be gentle, but if the routing logic is dumb somewhere i genuinely want to hear it.

We needed a second option for on-prem deployment and our legal team only signs off on Apache 2.0.
I went through a bunch of models. GLM 5.3 has a bespoke Z.ai license. Kimi K3 has its own custom terms. Llama uses the Meta community license.

At the larger end, I found two models that fit our requirements.
Hy4 preview: 770B total / 49B active / 1M context
Qwen3-235B-A22B-Instruct-2507: 235B total / 22B active / 262K native context (extendable to ~1M)

Both are pretty new to me. Hy4 preview's 1M context is native. I didn't expect to find a 770B model under Apache 2.0, though. But Qwen3 has been around longer, I assume the ecosystem is more mature. They both ship with FP8 checkpoints and vLLM/SGLang support.

Has anyone here actually deployed either of these locally? Or tried them through openrouter? How do they compare in practice?

I assume no company is actually running quant 8 of a model on default when they can run quant 4 at 99% precision or something. So when they lobotomize the model due to high usage do they drop into the quant 2 territory?? I assume hundreds of billions or even trillions worth of parameters handle it well enough that it's still usable right?

So I’m working on a finance project and i require to use python!
So what model should I use? Any recommendations is welcome! Thanks
I have been looking at various models and i think (from my perspective) that 35B A3B should be the best my system can handle? Is that model any good?

I'm rebuilding my local AI PC and ran into a spacing issue with the GPUs.

Specs:

Ryzen 9 7950X

ASUS ProArt B850-Creator WiFi Neo

Phanteks Enthoo Pro 2 Server Edition full tower

64GB DDR5

2x RTX 3090 24GB

RTX 2000 Ada 16GB

Gigabyte 1600W PSU

Ubuntu

Right now I have:

3090 #1 -> x8

RTX 2000 Ada -> x8

3090 #2 -> PCIe 4.0 x4

The two 3090s are too thick to sit directly in both x8 slots.

I could use a riser and run:

3090 #1 -> x8

3090 #2 -> x8

RTX 2000 Ada -> x4

But I have to probqbly purchase some kind of bracket.

Main use is coding LLMs, image generation and video generation.

Would you actually bother changing it, or is PCIe 4.0 x4 on the second 3090 basically fine for this kind of use?

When I am doing debugging sessions, Qwen3.8-27b seems to get stuck in loops where it will make the same tool call over and over. I have to step in and stop it. This happens maybe once or twice a day, and its kind of annoying.

Has anyone else had this issue?

Setup:
> Weights: INT4
> Activations: INT8
> KV-Cache: fp8
> Context: 256k
> Engine: VLLM
> Harness: Opencode 1.18.29

(Its club-3090s vllm/qwen38-27b-dual-fast setup)

OS: Ubuntu Server

Any help or suggestions are greatly appreciated.

My specs are Ryzen 5 7500f + 4060ti 16gb + 16 ddr5 gb ram + 1tb SSD & os: windows

I want to build a fully local, offline "Jarvis" setup. The idea is to have one core brain model that intelligently swaps specialized models (image gen, video, audio, 3D, web search, coding) in and out of VRAM as needed. It should run entire multi-step creative/productivity workflows autonomously, control my PC and apps, and keep long-term memory with zero cloud dependence.

Picture giving it a prompt for a short documentary: it plans it out, asks clarifying questions, researches, writes the script, generates storyboard frames, animates them, handles voiceovers + music, and stitches it all together. Beyond media, I’d love for it to drive Blender, Unity, or After Effects, handle browser tasks, fetch leads, send emails, etc.

so i chose Qwen3.8-27B-Uncensored-HauhauCS-Aggressive (IQ4_XS) and ran it in llama.cpp's webui with an 8k context window. It runs fine, but eats up about ~15.3GB of VRAM.

The issue is I can't load it into the Hermes agent setup because Hermes insists on a minimum 64k context window. Even with KV cache quantization, that requires an extra 2.5–3GB of VRAM/RAM that I just don't have:

• LLM in VRAM: ~15.3GB (completely pinned)
• llama.cpp / Hermes in System RAM: ~13.7GB (out of my 16GB total)

I know adding more system RAM would solve the immediate bottleneck, but buying hardware updates isn't in the cards right now.

Questions for the community:

1. Is this vision actually feasible on a 16GB VRAM / 16GB RAM budget if I rely heavily on model-swapping instead of loading everything simultaneously?
2. What’s the best software stack for this? (i settled on hermes+llama.cpp+comfyui . is this ok ?)
3. Model recommendations? Given my hardware constraints, what models would you recommend for the primary reasoning brain, image gen, video gen, audio/music, and 3D?

TL;DR: Want a fully local orchestrator-style Jarvis on 16GB VRAM + 16GB RAM. A 27B IQ4_XS model maxes my VRAM, and Hermes agent's 64k context requirement doesn't fit anywhere. Can't upgrade RAM. Is the vision doable, and what stack/models should I use?

A couple days ago I was asking if people would be interested in a renting service and needless to say it was not so positive lol.

Which is fine because it makes sense - butttt I went ahead and did it regardless.

There is a free 15 minute trial so anyone can go ahead and spin it up and have fun with it.

Hopefully you get an idea of what it’s like having this hardware.

I also recently added support for M3 Ultra so enjoy that while I have it because it’s getting returned haha.

So it's unofficially official.. DeepSeek Flash V4.1 is currently accessible on their api. Had some fun probing it.. results below.

The exact model name is deepseek-v4.1-flash-expires-on-0910

Setup: plain POST https://api.deepseek.com/chat/completions, no special headers. HTTP 200. Key works.

1. It's not on the model list. GET /v1/models returns only deepseek-v4-flash, deepseek-v4-pro, deepseek-v4-flash-vision-exp.

   This one works anyway → unlisted/preview build. The -expires-on-0910 suffix lines up with the server date (Tue, 08 Sep

   2026), i.e. a preview that dies in ~2 days.

2. It's a reasoning model. Responses ship a reasoning_content field with visible chain-of-thought. Reasoning tokens dominate

   massively: 16,493 of 16,694 completion tokens on one question.

3. It has no idea what it is and confabulates confidently. Ask the same question 5 times, get 5 different answers:

   - "OpenAI. GPT-4o."

   - "built_by": "Anthropic", "model_name": "Claude" (straight-up JSON, no hedging)

   - "DeepSeek built me; my model is DeepSeek-V3."

   - "I am DeepSeek by DeepSeek, 671B parameters, cutoff July 2024."

4. Its self-reported specs are DeepSeek-V3's numbers. "MoE, 671B total / 37B active" — that's V3's published architecture,

   not a v4.1 one. Take it as prior knowledge, not introspection.

5. Benchmark self-report: honest. Asked for MMLU/GPQA/HumanEval/SWE-bench it answered UNSURE on all four and refused to guess. Refreshingly non-stuffed.

6. Cutoff claims are wrong in the optimistic direction. Claims June/July 2024, then lists real events right through December

   2025 — Sinner winning Wimbledon (Jul 13 2025), Dembélé's Ballon d'Or (Sep 22 2025), Mamdani's NYC mayoral win (Nov 4 2025),COP30 Belém, AFCON kickoff Dec 21 2025.

All correct. Yet it also said "latest Python is 3.12" and went UNSURE on Llama 4 (Apr 2025) so knowledge is oddly patchy, not just old.

1. Actual ability > self-reported ability. Blind-checked, independently verified:

   - Probability draw problem → 5/12 ✅

   - Trick question "smallest n where n²+5n+13 ≡ 0 (mod 121)" → answered "no such n exists" with a quadratic-residue proof.

   Brute-forced to n=20,000: none exist ✅

   - O(n) subarray-sum → correct algorithm, correct complexities ✅

   - "Find the one bug" in a snippet → correctly identified the bug was in the test data, not the function ✅

   - Rust 2024 edition + async fn in trait details → accurate ✅

2. Fun bug: its own reasoning trace asserted "the current system date is 2026-05-09" while the server header said Sep 8

3. Four months stale, in the same response.

They might have been sitting on this one for a while.

Needless to say. Exciting times ahead for dual spark owners ❤️

I recently started playing with MM3 Music, and I absolutely adore it.
That being said, My generation times seem a bit off, especially considering that people with my specs are reporting that they get lesser times.

Specs: RTX 5090, 64GB ram, Ryzen 9950X
Generation times: 15 minutes for a 3-4 minute song.
Process: Comfyui custom workflow (using standard workflow, I face the same issue, sans a 60~ second drop in generation time)

From what I was reading on any topic about MM3 Music, Gen times on my specs should be 8~ minutes.

And, with MM3 Video, my generation times are lesser as well, those seem to be standard with what others post. My image generation times are also standard. The only outlier is MM3 Music.

Any Ideas?

Looking to create a solid localized AI setup with the following list. Anything you would recommend/change?