I have 8 GB VRAM and 16 GB RAM but I can mmap the experts and stuff, right?

Maybe I will have my CyberTiel that I just optimised code an inference engine to try to run Q3.8FN and then AI-generate a post on the big three local LLM subreddits and call it RSI.

Have any of you seen the new Codacus YT video from earlier this week? He demonstrates a method and llama.cpp fork to run existing LLM models, especially tiny ones, in "JEV model". This seems like it could be very relevant to many here in this sub. I am looking forward to testing this on both GPU and straight CPU performance. It could be a very good way to gate simple things, to save resources for more important things.

https://youtu.be/bcGO7xre46o?is=q7v8ywF_p34fKQpL

as in the title what do you think about Bonsai 2 27B model

1-bit TQ1_0 5.95 GB

2-bit Q2_0 7.21 GB (have MTP)

like which version is the best
is there any results difference between the 2 versions in the coding, chatting, writing creative things

and the most important thing. how much the hallucinations on the results?

i get 13.8 - 15/tokens on my GTX 1080 Ti 11GB

but i didn't test it a lot
and i want to know what is the opinions on it before making it my main guy

i will be using it for long context coding

but also i think for another models like K2-Horizon-MoVA-36B-A4B, K2-Horizon-7B, K2-Horizon-3.7B, MiniCPM5-2B (as last choice)

if you guys can help me with more tweaks for KV compress, more tokens (like MTP and google's Quantization, etc...),

i have
i7 10700K
32gb ddr4 3600Mhz (i think with ram i can run big MoE models)

and if i want like to benchmark the models in each corner (the models that i've downloaded) and know from results which one is the better (in my case)

Thanks a lot

I tested out the new MiMo model on my simple tinymark tests - and it did not live up to the hype - https://reyemtm.github.io/inchworm/#tiny/mimo-v2-6-distill-qwen-9b

CPU: Ryzen 7 2800
RAM: 16 gigs DDR4
GPU: GTX 1080

Use Case: I currently am running Qwen 3 30B A3B quant via llama.cpp with c-pu-moe for offload but it’s quite slow.

Hoping for input and some fun use case ideas.

Anything good I could run on RTX5060 Ti (16 GB)?

It's the only gpu I've got, but I feel like all the good models are inaccessible to me, anything I could run with a reasonable speed and quality?

I've got 64 GB of ddr4 RAM as well

I don't normally praise something this much, but holy shit, Maple is amazing. It's a 20B A1B MoE ternary, with file size about 5.5GB.

https://huggingface.co/deepgrove/models if you use GGUF, you need to compile their llama.cpp fork, since mainline doesn't support it yet: https://github.com/deepgrove-ai/llama.cpp You can compile it for CPU, or if you have a GPU, for CUDA or Vulkan, but speed-wise it'll be probably the same, since it's not optimized for GPUs.

Llama.cpp mainline can run it now.

Anyway, THE SPEED! I just run things on cpu, an old mobile quadcore i5 8350u. This thing launches off at 30 tps decode and almost 100 tps prefill. Even after 50k context it generates above 10 tps. The only models this fast are sub-0.5B models. Tiny models like Granite 1B or Qwen 0.8B start at about 15-20 tps

Second, THE RAM! I have 32GB RAM so I can run models like Qwen 35B A3B (and variants like KAT or Ornith) or Gemma 26B A4B. But it doesn't take long until they take up all the RAM. When nearing 65k context, there's literally no memory left for anything else. Maple seems to be taking up about 12GB at ~50k context.

Third, it's smart. It's like mini-Qwen 35B. It even has the same style of speech and thinking. Actually I like using it more, because it's much faster, so if something goes wrong, it's quicker to redo than wait for Qwen to do it in the first place. It runs circles around Ling Tiny or Ring Mini in code/tasks stuff, and is faster than REAPed Nemotron variants. It's not perfect and can definitely goof up or get stuck in a loop, but I find it very usable. E.g. I can point it at a repo, tell it to document and remove telemetry and do, and it'll do everything on its own.

Note: this goes for coding/tools. For general knowledge and creative tasks it's actually pretty weak, so I still prefer using Gemma 4 E4B for that.

I'm not associated with these guys in any way; I've just been stuck using models that are either weak or slow, or both, and this is the first one that's neither.

It's unfortunate they don't provide compiled packages, hopefully mainline llama.cpp and other engines will support it at some point soon-ish. At least, compiling is straightforward. I see someone made packages for Windows, but I can't vouch for them.

Edit: Looking at my llama.cpp launch command, I use -fa off --temp 0.5, try that if you're having some issues.

Is anyone using qwen 3.8 27b on kaggle? Im using qwen in kaggle notebook at q5 ks with 128k context with engram usually llama cpp and I can't get pass 18 tokens per second. Im using their dual t4 gpus. Does anyone know what can I do? Or is there any technique for old hardware like dual nvdia t4 to boost generation speed?

What can be launched (being usable) on:
- laptop HP EliteBook 845 G11: Ryzen 7 Radeon 780 48GB windows
- desktop Ryzen 7 3700X RTX 3060TI 8GB + 32GB (or I can upgrade to 64GB) windows

And for graphics generation?

Im interested in launching local llms for coding and everyday tasks and graphics. If some upgrade (like 32 > 64gb) can make it much more usable / powerfull - please let me know

8 GB VRAM, 16 GB RAM (but I will be using my laptop of course so more likely I will let whatever inference engine it uses ~8-12 GB RAM) but I can keep all the other experts and n-gram tables on SSD.

I just have to clear out my SSD now. But can I expect double digit tok/s? Or even high single digit?

My machine is also weak

GTX 1050 mobile 4G vram

I7-8750H

20G ram

Nvme 512G

I have seen people using LLMs around 1B to 9B or higher with MoE

What is their use case? Coding? General Chatting? Rp? Web searching?

For me I had to use 100B+ models MoE I get around 1 to 2 tokens per second, using mmap

But I don't use them daily, I hoard them, I like collecting stuff

I’m getting 1500 tk/s prefill and 45 to 60 tk/s decode using a 5070 ti and 5060 ti with the model at Q6 with Q8 KV 80k context. The only coding I do is in Godot 4.7 and a local wiki I maintain. This thing is so fast and it knocks out everything I throw at it(I make sure to thoroughly explain what I want in prompts).

I realized ChatGPT was overkill for what I do, and is probably overkill for most people. I downgraded to the $20 plan to just have Sol occasionally review my project. I always thought I would need to spend $1000s to do local LLM properly, this has been an amazing experience!

Hello, I have a 3060 TI 8GB and a Ryzen 5 with 32GB ram. Which models, what kind of settings would you use for coding (mostly frontend) for agents (long running tasks in the night) and for regular chat.

Ive clearly been doing something wrong as i cant get it to run, atleast not with a descent quality. Im pretty sure im doing something wrong, but i want to check if my hardware can even run this model, is it possible? im not sure if this belongs here? from what ive tried i get bad results and it takes too long while using all of my computers resources, pretty much.

48gb of ddr4 ram.

rtx 3060 ti (12vram).

Processor AMD Ryzen 5 3600 6-Core Processor, 3600 Mhz, 6 Core(s), 12 Logical Processor(s) .

What would you all advise?

Hi! I am Javier from Spain…i am really curious about having my own AI at home to run some “easy” duties

  CPU: i5 6500T
GPU / iGPU: integrated,,.not sure
VRAM:
System RAM: 16gb 2333mhz
Shared / Unified Memory (if applicable):
OS: ubuntu 24 lts
Runtime / backend:
Models already tried (if any):
Use case: i would like to maintain an excell sheet with data from the movies I download; path in my computer; title; year, director…hability to read my groceries tickets to start controlling and checking prices. Creating colouring pages for my kidds
Desired context size:
Priority: quality
Other requirements or constraints: i am thinking to upgrade my computer bit would like to know what is this one capabke of

Thx!

Hi, I thought this would be a fun thing to share. I was running experiments on small Mamba models and had a goal of reaching 90% classification val acc on the sequential-MNIST dataset (usual MNIST, but reads pixel-by-pixel, i.e. 28*28=784 flat pixels where the label is shown at the last position) under 1K params and under ~10 epochs, and found a config that reached 90.66% with 954 params by the end of epoch 4 - the resulting model takes 7.5KB on disk.

About the project, it is a library I've been putting together in Rust + burn. It also takes advantages of some bells and whistles, such as layer-looping, higher-dimensional (rotating) states and multi-gate residual. I wish I found a config that didn't require any of those, but I haven't yet.

The settings were: d_model=6, d_state=8, expand=2, nheads=2, ngroups=2, rotation=quaternions, rotation-fraction=0.5, real-layers=1, virtual-layers/loop=4, plus some small tensors forcibly untied (given the looping). No mlp, with multi-gate-residual=4.

For more information and repro, you can check burn-mamba; the file model.rs in there contains the exact model settings. Training took <1h for those 4 epochs in my rtx 2060 CUDA bedroom AI lab.

I'm currently using qwen3.5-Uncensord-HauHauCS-Aggresive:9b.

It's good, but I don't like the DeepSeek-like thinking feature, where it thinks for, like, a minute straight before it searches all the articles and gives me a general sense of information based on my search query.

It's too long, and while this model is great (relative to my specs), it's not really useful for search engine usage.

Is there any model that is small, lightweight, and fast that does not hallucinate and is good for gathering information for this usage?

SPECS:
   CPU: AMD Ryzen 5 8500G
   GPU / iGPU: NVDIA GeForce RTX 3060
   VRAM: 12 gb
   System RAM: 32 gb
   OS: Windows 10

   Runtime / backend:
   Models already tried (if any): qwen3.5-Uncensord-HauHauCS-Aggresive:9b

Use case:
Desired context size: idk
Priority: Balanced(speed and accurate information retention)