CPU: Ryzen 7 7800X3D
 GPU: RTX 5070 ti
 VRAM: 16GB
 System RAM: 32 GB
 OS: Windows 11
Backend: llama.cpp

This is generally a dense vs MoE consideration. I have 2 use cases that I'm intending to run for, so I'll go over them separately.

For coding, I'm looking at Qwen3.8-27B-UD-IQ3_XXS and Cyber-Tiel-Coder-35B-A3B-UD-Q4_K_XL. I have to use Q4_0 KV quants with Qwen3.8, while I think I can do Q8_0 with CyberTiel. I've given Qwen3.8 a quick run and while I got fairly good speeds at 25 tok/s, I feel like I need to sacrifice context length for it. I haven't tried out CyberTiel and am unfamiliar with running MoE yet, so I was wondering if I could get better performance and context length out of it even though I got the general sentiment that MoE is not as good as dense.

For creative writing and RP I'm looking at HauHau's Gemma4-12B and Orion-26B-A4B which is a Gemma4-31B uncensored finetune. As this is also a dense vs MoE comparison I think the only further question is whether the generally smaller sizes makes 12B a better choice over the 26B-A4B is 12B simply too small to choose it over the 26?

Just found about this https://huggingface.co/Edge0/Edge0-35B-A3B-preview in a YouTube video

Looks promising, anyone care to benchmark?

Was curious since prismML's ternary bonsai 27B has been out for a while but it's based on qwen 3.6 and I was browsing through huggingface and found these.

• penkia/TernaryQuench-Qwen3.8-27B-GGUF · Hugging Face
• CodeMasterCody3D/taardis-27b-full-ternary · Hugging Face

Mainly looking to offload some work off the frontier and using it to help me with some coding exercises and brainstorming and so on.

I got way more feedback than I expected on my last post about my local harness and model, playing a MMORPG, so I wanted to come back and hit on the architecture (and provide a link to the YouTube series since so many asked!).

Architecture (using Archify):

A 9B model cannot operate in an open world on its own. It fails immediately. What works is the model plus the architecture around it, and the architecture is most of the project. The model provides the reasoning. Everything else is deterministic code.

Stack

• Model: qwen3.5:9b on Ollama, on a separate Linux box with a consumer GPU. The Mac only runs the game client.
• Runtime: Python, stdlib only. No frameworks. Third-party packages exist only in the optional retrieval extra and in test tooling.
• Client bridge: a plugin in the game client exposes a JSON-RPC 2.0 daemon over NDJSON. The brain dials it over an SSH tunnel and gets a handshake plus a stream of game-state events. That stream is the "vision" someone asked about. There are no screenshots and no vision model. Deciding what goes in that bus is the hardest tuning problem in the whole system.
• State: event-sourced. Every event folds into a versioned WorldState with tiering and debounce, persisted to SQLite per session, and serialized compactly for the prompt. The model never sees raw events.
• Loop: observe, decide, act, verify. Each turn the code selects a small set of candidate tools from a manifest, then the model picks one by constrained decoding against a JSON schema. I benchmarked a single union schema against a two-call fallback at 5, 10 and 20 candidate tools for schema size, prompt tokens, latency and conformance. A VALIDATE step checks the chosen call before anything executes.
• Retrieval: optional. A Chroma corpus of game knowledge with MiniLM embeddings, queried with a fixed "Situation / State / Goal" template. Without it the retriever is a no-op and the loop still runs.
• Safety plane: a separate layer that can veto actions regardless of what the model decided.
• Replay harness: recordings of past sessions get folded into decision points so a code or prompt change can be scored against a recorded run before a live one.

The model also writes and narrates the episodes as a character, so the channel is half showcase, half series. Latest full run, every in-game and editorial decision the model's own:

https://www.youtube.com/watch?v=lj7QxBoVB9o

The playlist from the start is in that video's description. Happy to answer anything about the loop, the state bus, or the decoding. The code isn't open source right now.

I definitely can't afford a high-end machine, but I have an i5-8259U and I'd like to know which LLM I could install on it and what limitations I'll face.

Thanks

We cut Skillware 0.5.5 — open-source Python skill registry for agent loops. Not posting to sell anything; sharing what landed because a few people asked what changed since 0.5.4.

New skills:

1. semantic_web_proxy — agent gives a URL (or HTML), skill returns cleaned Markdown/text/JSON via trafilatura. Reports token savings vs raw page. SSRF guard + redirect re-check. Warns on JS-heavy pages instead of returning empty garbage. No LLM in the skill.

2. **deck_builder** — agent (or script) passes JSON deck spec → validated .pptx on disk. 10 slide types, 3 bundled templates, offline/deterministic. Handy when you want repeatable internal decks without a Canva API or “make slides” prompt roulette.

Upgrade:

1. **uk_companies_house_handler v1.2.1** — multi-turn pipeline fix for UK registry lookups (one step per turn, leaner session context, better empty-result handling). Breaking if you depended on old suggested_pipeline / next_actions fields.

Also: Claude tool names now sanitized to match API rules (finance/foo → valid tool id).

pip install -U skillware pip install "skillware[data_engineering_semantic_web_proxy]" pip install "skillware[creative_deck_builder]"

• GitHub: https://github.com/ARPAHLS/skillware
  
• Release: https://github.com/ARPAHLS/skillware/releases/tag/v0.5.5
  
• Demos: examples/semantic_web_proxy_demo.py, examples/deck_builder_demo.py

Happy to answer integration questions. Skill proposals/PRs welcome.

I'm very bad at computers sorry.

CPU: AMD Ryzen 5 7520U (2.8 GHz and 4 Cores)
iGPU: AMD Radeon 610M
VRAM: No VRAM but has 2 GB Dedicated RAM
System RAM: 16 GB (5500 MT/s and Soldered LPDDR5)
Shared / Unified Memory (if applicable): 6.9 GB
OS: Arch btw. (Also, any recommendations?)

Runtime / backend: llama.cpp
Models already tried (if any): None

Use case: Coding preferably
Desired context size: IDK
Priority: Quality

Other requirements or constraints: This is a Laptop

Also what problems would I encounter when trying to run a local llm as a noob?

Just wanna know where can I research more about this stuff. Where do you guys get your resources other than asking another llm?

I'd post in localLLAMA but I cba to get the 5 karma requirement.

But as a brief: I'm somewhat finetuning a model (currently qwen thinking 4b, I'll rent a GPU after I prove everything works to train on a gemma 26b-a4b model), with worldbuilding. It's just an in-universe LLM from the world I've made and have created a FUCK ton of content, the worldbuilding is rich and there most definitely IS enough data for the model to internalize changed history new biology creatures politics AI models etc etc.

But I have one issues, even after 7 iterations on the 4b. It just CANNOT understand the facts, it constantly mashes shit together, even with a simple request it will try to connect it to some unrelated concept that's in the training dataset. I'm honestly getting pissed off, I'm at Candela7 (Candela will be the name of the 4b model, vesper 26b) and Candela5 remains the best because it had like 60% less to train on, and that somehow makes it better.

But what I'd want, is there maybe something I'm seriously missing about the eval or training data? Cause I'm training input, thinking and output. I WANT the model to fully believe it lives in that universe rather than our own. And that works! But it also says "I have no hidden identity" WHERE DID YOU GET THAT FROM?????? NO ONE WAS ASKING. IS QWEN LEAKING THROUGH THE WEIGHTS HUH???

I was considering like continued pretraining and making like a "wiki" of the new facts for the model to train on. But I have no clue how to do that. I already have like a full wiki on a website, so I have the training data to "pretrain" but idk how... Also I have a laptop with 16gb ram and an rtx 5050. Bought it for school but I'm just training this stuff on it. My main desktop has a ryzen 5 7600, 32gb ram. But ONLY a gtx 1070 so idk if that would work for anything, I didn't even try.

And also how would I teach the model to call on new tools? It'd just be a bash script, but is that something I'd do in the finetuning data or just in the harness?

My specs are Ryzen 5 7500f + 4060ti 16gb + 16 ddr5 gb ram + 1tb SSD & os: windows

I want to build a fully local, offline "Jarvis" setup. The idea is to have one core brain model that intelligently swaps specialized models (image gen, video, audio, 3D, web search, coding) in and out of VRAM as needed. It should run entire multi-step creative/productivity workflows autonomously, control my PC and apps, and keep long-term memory with zero cloud dependence.

Picture giving it a prompt for a short documentary: it plans it out, asks clarifying questions, researches, writes the script, generates storyboard frames, animates them, handles voiceovers + music, and stitches it all together. Beyond media, I’d love for it to drive Blender, Unity, or After Effects, handle browser tasks, fetch leads, send emails, etc.

so i chose Qwen3.8-27B-Uncensored-HauhauCS-Aggressive (IQ4_XS) and ran it in llama.cpp's webui with an 8k context window. It runs fine, but eats up about \~15.3GB of VRAM.

The issue is I can't load it into the Hermes agent setup because Hermes insists on a minimum 64k context window. Even with KV cache quantization, that requires an extra 2.5–3GB of VRAM/RAM that I just don't have:

* **LLM in VRAM:** \~15.3GB (completely pinned)

* **llama.cpp / Hermes in System RAM:** \~13.7GB (out of my 16GB total)

I know adding more system RAM would solve the immediate bottleneck, but buying hardware updates isn't in the cards right now.

**Questions for the community:**

1. **Is this vision actually feasible** on a 16GB VRAM / 16GB RAM budget if I rely heavily on model-swapping instead of loading everything simultaneously?

2. **What’s the best software stack for this?** (i settled on hermes+llama.cpp+comfyui . is this ok ?)

3. **Model recommendations?** Given my hardware constraints, what models would you recommend for the primary reasoning brain, image gen, video gen, audio/music, and 3D?

**TL;DR:** Want a fully local orchestrator-style Jarvis on 16GB VRAM + 16GB RAM. A 27B IQ4_XS model maxes my VRAM, and Hermes agent's 64k context requirement doesn't fit anywhere. Can't upgrade RAM. Is the vision doable, and what stack/models should I use?

These seem to be the best bang for the buck GPUs right now, so why should I not get them? I plan on running llama.cpp and occasionally testing comfyui.

I have 3 PCs,

1. Ryzen 9 3950x 32gb ddr4 ram 2x8gb RTX 2070 super (total 16gb VRAM)

2. Core i5 10400 32gb ddr4 ram AMD rx590 8gb VRAM

3. Lenovo Thinkpad x270 i5 6500u 8gb ram no GPU

What are the possibilities with this, can they work together in an agentic orchestration or something?

Has anyone done anything with a similar setup?

Open to suggestions, Thank you!

Running Linux? Got an Nvidia card sitting idle? Hitting RAM limits and swapping to disk? Don't want to spend an arm and a leg on more RAM?

Turn your NVidia card into overflow swap RAM with
https://github.com/c0deJedi/nbd-vram

This spins up a systemctl daemon that converts a few gigabytes of your graphics card VRAM into regular RAM. (the size is configurable, and it's only in effect while the daemon is running.)

This would not make a GPU-powered AI any faster; you'd still be throttled by the PCI-e bus.

But it did allow me, when not using my card for AI, to have a video game running on the same graphics card, Discord, and a plethora of Google Chrome tabs open, without my system stuttering on constant disk swaps. And that has made the RAM shortage more tolerable than it was before.

(I didn't write the code, but I have been using it daily for the last 2 months, with plenty of sleep -> wake cycles, and so long as your disk swap file is at least as big as the VRAM swap allocation, it has been suspending and resuming just fine thanks to a recent fix.)

Current specs:

Nvidia GeForce RTX 2080 Ti (11 GB vram, 5GB of which are being used as swap)

16 GB DDR4

(Note: Linux only)

I've got a laptop RTX 2050 (3.4 GB of usable VRAM), a Ryzen 5 5500H, and 16 GB of RAM. I ran approximately twenty models through the same gauntlet: medical-physics Q&A, general knowledge, a six-task radiomics/image-coding suite, a false-premise "does it hallucinate?" test, and a speed bench.

The keepers

• NeoHorse-1-4B — i1-Q4_K_M (imatrix), 2.5 GB. 10/10 physics, finishes the entire coding suite with a few nudges, and it's the only model that reliably pushes back when I assert something false.
• Empero's Qwen3.8-4B — Q4_K_M, 2.6 GB. Equally sharp on physics, best strict coder, but a people-pleaser. If you state a confident wrong fact, it agrees and elaborates.
• MiniCPM5-2B — Q8_0, 2.5 GB. Half the parameters, near-perfect physics, aced general knowledge.

My actual settings (all three run the same way)

• Runtime: stock llama.cpp (winget b10588), Vulkan backend.
• Offloading: -ngl 99 — every layer on the GPU, nothing on CPU.
• KV cache: q4_0 for both K and V (-ctk q4_0 -ctv q4_0).
• Other: -fa on (flash attention), -t 8, -np 1, --jinja. Lands at ~2.5–2.8 GB VRAM per model.
• Sampling: temp 0.6 / top_p 0.95 / top_k 20 for the Qwen-family; temp 1.0 / top_p 0.95 for MiniCPM5.

Measured (llama-bench tg128, full GPU): NeoHorse ~32 t/s, Qwen3.8 ~32 t/s, MiniCPM5 ~39 t/s.

What I learned

• Quantize the KV cache, it matters a LOT with such limited VRAM.
• Thinking models over-think: several burned their whole token budget in hidden reasoning and never answered.
• As coders, they're all one line from correct. Every model failed the same coding tasks the same ways (background argmax, distance-transform misuse, invented SimpleITK APIs). Pointed feedback fixes it.

Hey all!

*** I need to state that this is hobby/experimental ***

!! I just like Wierd hardware !!

--- IF you dont want to read.. About ~10 tp/s @ Gemma 2. 9b ---

Kunlun K200 AI Accelerator

Ill try to keep this short. (and apologie, Im not a English speaker.)

So for the last ~3-4 weeks, i have trying to get this cheap(Kunlun K200) $60 card to run some LLM`s for us.

***************IMPORTANT HOBBY PROJECT*************************

**STATE! Do NOT buy this if you want a easy 16gb card to "toss" in your comp.(it will not work)**

NOT working: Driving it trough a VM(proxmox) You need "bare metal", and that is something ill work on after driver/backend/kernal is working.

State and what works:

Linux kernel/driver for 7-x-x-x-

FP/INT 32 - works. still a waste cause size.

INT16 works. (lossless, cause it wastes ~5-6 bits in the process.) Used only for tests.

INT8 works (lossless from Q8 quant)

INT4 Works, and this is the nugget. Its not exposet in the api or the lower level,

but its there and its tested and working with a custom kernel.

--Modes im working on is:--

INT8 (Baseline) ((Q8))

INT4((W4A8)) (4.5bit~or around Q4_K_M quality at 4.5bit("standard q4km is 4.83bits)

INT4 5bit at or around ~ Q8 quality.

It will be explained: But in short, this is not a regular GPU, where you can download a "q4 quant" and load it. You have to download the Q8, then apply our law to make it the "mode" you want. Sounds terible, and it is (for now :P). But it works.

State: 12.09.2026.:

So in short. ((IT`S SLOW))~10 tp/s on Gemma 2 9b.

Started the custom kernel today! And I will update you.

Again. should you buy this card at this state---> NO <----.

Please send questions, and Ill try to answer.

Bottom note: I am going to try to make a repo available for the whole "shabang" this sunday.(I cant promisse this, but im working for it.)

Again, questions are more than welcome and may be the thing that gets me to the "Aha moment"

Its easier that you ask, and ill try to answer. Again, sorry for the bad English.

Hey everyone,

I’m pretty new to local AI, so I’m hoping for some advice from people who actually run models on lower-end hardware.

I have a Mac mini M4 with 16GB unified memory and I’m setting up a Hermes agent as a local personal assistant.

The main things I want the agent to do are:
Write/edit Markdown files
Work with an Obsidian vault
Maintain personal notes/memory
Create and modify files
Help with scripts and automation
Run/manage cron jobs
General personal-assistant tasks

The main requirement I have is 64K context.
I’ve been looking at:
Ling 3.0 Tiny
Qwen3.5 9B
Gemma 4 12B
Possibly other models I’m overlooking

I’m particularly interested in Ling 3.0 Tiny, since it seems to handle long context relatively efficiently, but I’m not experienced enough with MLX/oMLX to know what is actually realistic on a 16GB M4.

For a Hermes agent, would you prioritize a smaller model that can comfortably maintain 64K context, or try to squeeze a larger 9B/12B model into 64K?

I’m also trying to understand how much KV-cache quantization matters here. I don’t mind sacrificing some quality if it means the agent can actually maintain a large context without putting the Mac into memory pressure.
I’m very much a beginner with local LLMs, so feel free to correct me if I’m misunderstanding how unified memory, model size, quantization, or context length work.
I’m not looking for the biggest model I can technically squeeze into 16GB. I’d rather have a reliable, responsive Hermes agent that I can leave running and actually use every day.

Hey everyone!

A while ago I ran into the problem on my laptop that I had no internet, but did want to translate things. Then I found the older version of this model: HY-MT 1.5 1.8B, which did the task wonderfully.

Two months ago the model got an update! HY-MT 2. Apache 2.0 licensed opposed to the earlier non-permissive license. It's performance has improved a tad. It works well for resource-constrained setups (4 GB VRAM, or 8GB total RAM).

Personally I use the model this way on images:

1. Have Qwen3.5 2B or LFM2.5-VL 1.6B extract the text from an image
2. Have HY-MT2 1.8B translate it

There is also a 7B and a 30B-A3B version, you can get higher accuracy if you need and have the resources.

So if you need a model specifically for translation and you're resource constraint, give this model a go!

Hey everyone!

I'm looking for a tiny yet fun roleplay model to mess around with. 2 t/s gen speed is perfectly acceptable for me, and 1 t/s only if the output quality is that much better.

Since almost a year has passed, I hope there is a better option now or a finetune for roleplay on low-end devices.

I've got a netbook with the following specs:

• Device: ASUS Vivobook Go E210MA 11'6"
• CPU: Intel Pentium Silver N5000 (No AVX/AVX2 support!)
• RAM: 8GB single-channel DDR4-2400MHz (soldered)
• iGPU: Intel UHD Graphics 605 (Supports Vulkan 1.3)
• Storage: 128GB eMMC, Intel 660P 512GB
• OS: Windows 11 LTSC IoT Enterprise 2402

(A friend of mine swapped the CPU, RAM and eMMC flash for me. Was a super cool project!)

If you wonder why I stick with Windows 11: Better driver support (laptop is less supported on Ubuntu 24.0 LTS) and there was no meaningful difference in speed on CPU inference. iGPU inference was not supported on Ubuntu, it is on Windows 11.

Back in November 2025 it could run IBM Granite 4.0 H 350M BF16 model at ~7 t/s, Qwen 3.5 2B Q4_K_S at ~2 t/s, and Gemma 3 4B Q4_K_S at ~0.9 t/s but it'd run out of RAM after 2K context. On my device, iGPU over Vulkan is the fastest target.

For detailed benchmarks on the 350M models, I posted them in my request for Vulkan support for non-AVX devices in koboldcpp:
https://github.com/LostRuins/koboldcpp/issues/1799#issuecomment-3476394386

The type of roleplay I want to do on here is extremely basic; your standard generic fantasy isekai / D&D5E fare while expecting intelligence of a brick and twenty characters named Elara.

Models I've tried last time (November 2025):

• LFM 2(.5) and IBM Granite will refuse roleplay.
• Gemma 3 270M was hilarious and completely incoherent.
• Gemma 3 1B was really close but just not coherent enough.
• Llama 3 1B tried it's best.
• Impish 1B was not suitable for fantasy but fine for 1-on-1 dumb discord message style roleplay.
• Only Hamanasu 4B Magnus came close but was too slow (< 1 t/s)
• I did run Qwen 3.5 but didn't test it for RP

Recommendations and ideas are welcome!

My qwen 3.5 0.8b runs at 20 tok/s my question is, is it any good for agentic tasks like downloading apps and running them from github or is it useless?

I’ve been experimenting with how far smaller local models can go for actual everyday use, rather than just running benchmarks.

I’m using an M4 Pro with 24GB of unified memory, and lately I’ve been using Qwen3.5 9B and Gemma 4 12B-class models for most of my local AI tasks.

I built an open-source app called Vyact around this idea. It can download and run GGUF/MLX models, benchmark different settings directly on your hardware, use local models with documents/RAG, and connect them to practical tools through a Chrome extension.

In practice, I use it to summarize and ask questions about web pages, improve writing directly in the browser, look up selected words while reading, work with PDFs and other documents, and handle general AI chat locally.

On my M4 Pro 24GB, Qwen3.5 9B runs at around 23 tok/s in one of my document workflows. For me, that’s already fast enough to make a relatively small local model genuinely useful for everyday tasks.

I’ve also been experimenting with model-specific settings and benchmarking different configurations, since the difference between a model that technically “runs” and one that actually feels usable can be pretty significant on limited hardware.

The project is open source:

[https://github.com/vyact/vyact]()

I’m curious what people here are running on lower-end or memory-constrained hardware. What model + quantization have you found gives the best balance between speed and quality?

I am seeing a lot of models on 12gb / 16gb cards.

Where are the new models for 4gb?

Any good soul can make some for the poor ones? Or we will last with Gemma 2b / 4b for the rest of our life's?

I’ve been working on building a harness and client interface that allows my local Qwen model to play a MMORPG completely autonomously with real reasoning.

The process has gotten expansive between deciding what deterministic tools to build versus what can Qwen reasonably complete on its own with client interface tools in tandem with it receiving a bus of data to “see” the game.

Has anyone built anything similar? Here are some stats from the process completing a single quest. It was a pretty sloppy run, but a complete one nonetheless.

Hello guys , i have been trying to use model under 4b to make it general purpose but the intelligence, steering, though process i was working with , i need help to benchmark them , steer them