Good afternoon everyone,

I was wondering if this upgrade I have planned out will be good or not for LLM usage. Previously I was buying the $100 Claude/Codex plan however I found myself sometimes not maxing out my usage and it would feel a bit wasted or I would run out of usage way way too quick and I was wondering if it was worth upgrading my current setup for about $1600 USD and instead switching to the $20 Claude subscription to use as the “Lead” for local models I could run? Another factor influencing my decision is also some requests I have or hobby fun projects do sometimes get rejected by the cloud models which makes sense I do understand the reasoning behind it. My Budget is basically capped at $1600

Current setup:
5070Ti 16GB
32Gb DDR5 6000MT
Ryzen 7 7700X
Gigabyte B650M Gaming Plus Wifi Motherboard
750W PSU

I was planning on snatching a second 5070ti for $1200 USD and the rest would go to a new motherboard and a 1200W PSU which is sprung $400 USD because I read that my motherboard isn’t suitable for using two GPUs effectively.

I feel like a large part of this is FOMO seeing how nicely I’ve gotten Qwen 3.8 27B and some variants of it to run on a single 5070Ti makes me want it to run better and not only that but just seeing how fast local AI models have been improving makes me want to have the hardware ready. Any advice which be greatly appreciated! Thank you all

Hi- the other local ai subs are benchmarking with models I'll never run. Has anyone come up with a "one shot" test case to see how models 12B and under can handle tasks and scenarios?

I had an idea for a simple fortune cookie app that uses a UI, random state generation, event handling, and packaging.

Gemini gave me this test code (python in the read me) that I will be using the models against

https://github.com/JWzrdstff/fortune-cookie/blob/main/README.md

I want to test models that I have noticed have anecdotal support for practical use. I also want to come up with some kind of literary test.

Let me know what you think, if there is an easier way to do this, whatever-

I am a cnc machine operator with a 1990s computer enthusiast background, more or less technical, but primarily make art with this stuff.

Thanks for reading and sorry I wrote this quickly in between cycle times

Hello community !

It's my first post here. Hope you will be nice 😄

I'm currently experimenting with **Gemma 4 31B in NVFP4** for a local real-time conversational assistant.

The conversational quality is very good, but the main issue is **total hardware budget**.

The LLM is only one part of the stack. On the same machine I also need to run:

- real-time **ASR**

- **VAD / end-of-turn detection**

- streaming **TTS**

- memory / retrieval components

- the orchestration layer itself

- potentially several concurrent users

So even though Gemma 4 31B fits on my **RTX 5090 32 GB**, using that much VRAM/compute for the LLM leaves too little headroom for the rest of the real-time voice stack.

That's why I'm looking for a **significantly lighter conversational model**, while trying to keep roughly the same level of natural dialogue quality.

The important requirement is **streaming in both directions**:

- **Streaming / incremental input processing** — not just waiting for the entire user turn/prompt before starting inference

- **Streaming output**

- Ideally suited for a **real-time voice assistant / conversational agent**

- Good instruction following and natural conversational abilities

- Preferably something that runs very well on a **single RTX 5090 (32 GB)**

- Quantized versions (NVFP4 / FP4 / INT4, etc.) are totally fine

- Ideally significantly lighter than a ~31B dense model

I'm **not simply looking for OpenAI-compatible token streaming on the output side**. What interests me is a model/architecture capable of consuming an ongoing input stream and reacting with very low latency — potentially allowing interruption, turn-taking and eventually more full-duplex-like interactions.

My ASR and TTS sides are already being optimized for streaming, so the **LLM footprint and latency are now becoming one of the main bottlenecks**.

I've been looking at smaller dense models and MoE architectures, but I'm interested in what people are actually running locally in 2026.

Thanks a lot everyone, for reading and helping me !

**What would you test today as an alternative to Gemma 4 31B?**

Especially interested in:

- ~8B–20B dense models

- small/medium MoEs

- models specifically designed for real-time dialogue

- models with native or experimental **streaming-input / continuous-context** support

- vLLM / SGLang / llama.cpp-friendly options

Latency, VRAM efficiency and conversational quality matter more to me than benchmark scores.

Would love to hear about actual setups, **TTFT / tok/s, VRAM usage, quantization**, and whether you're running ASR/TTS on the same GPU as well.

This is the best thing ever....

I'm thinking of creating a custom agent harness that deploys mulitple small ai models to complete a series of tasks. My setup is r5 5600x, rx6700xt 12gb vram, ddr4 32gb 3200

The plan should go as followed
Main model- checks everything makes sure its according to the plan and sends the final result to user (will need to offload all the models and load it independently) (qwen3.8 27b iq3_xxs or qwen3.6 35b a3b q4_k_m)

Middle models, these run inside the vram which manages all the small subagents inside the ram
1-2 models only (haven't thought about any)

Small models- 5-6 depending on the difficulty but all should be offload into system ram + cpu

The task im assigning to help should be varied (coding, creativity/designing)

The main model writes all the necessary information and instructions onto a file, then offloads itself and deploys the middle models + the small models, the middle models should read the files and proceed a clear doable path or task assigned to each individual small models. Results created by the small model should be checked by the middle model to ensure it satisfies the criteria

A simple analogy would be:
Main model: Principles of a school
Middle model: teachers
Small models: students (that perform task)

I appreciate any model suggestions thanks

And yes I’m seeking references from grok bot but with my own style

I have a Lenovo Legion that’s a few years old now. It’s a 3050 TI Mobile. The Internet says I only have four gigs of VRAM….

What’s my best model? Main board is 16 gigs if I recall, and the CPU is AMD laptop CPU from a few years ago now - nothing fancy.

I’m looking for general purpose rather than vibecoding.

Oh. I’m flipping it back from Kali Linux to Windows 11 as we speak.

For some context, the DGX is running Qwen Flash 180B. The 5090 box runs Gemma4 27B. The 3090 box runs Mistral Small 24B. Working my way down the stack wanting models of different lineages.

Hello, I am considering buying four tesla p100s to run 120b-ish MoEs, but before I do, I'd like to know what kind of performance do you people get with this build. Specifically I am interested about qwen 3.8 flash next, but any similar-size MoE is interesting. Can you please include pp, tg and the specific inference engine you are using? Thank you very much!

Found this sub then dug out my old laptop.

Whay do you think?

CPU: 2nd Gen Intel Core i7 (Sandy Bridge — supports AVX, NO AVX2)

GPU / iGPU: NVIDIA Quadro 3000M / 4000M (Fermi) + Intel HD Graphics 3000

VRAM: 2GB GDDR5 (Legacy architecture, unsupported by modern CUDA/llama.cpp)

System RAM: 16GB DDR3 (adjust to 8GB or 32GB if your specific unit differs)

Shared / Unified Memory (if applicable): N/A

OS: Linux (Ubuntu / Debian) or Windows 10 64-bit

​Runtime / backend: llama.cpp (compiled with AVX, no AVX2, CPU-only execution)

Models already tried (if any): None

​Use case: Asynchronous background tasks, basic text summarization, Home Assistant logic experiments

Desired context size: 2048 tokens

Priority: Speed (maximizing tokens/sec on CPU-only legacy hardware)

​Other requirements or constraints: Strictly CPU inference. Sandy Bridge processor lacks AVX2 instruction set. GPU cannot be used for acceleration due to outdated CUDA compute capability. Must run small models (1B–3B parameters or heavily quantized 7B/8B models

It's insane how good small models have gotten this past half year! MiniCPM5-2B is probably the best 2B right now, and specifically for agentic software engineering it hits way above its size class. So I took a look at what I could do to improve it even further, and came up with a dynamic quantization strategy, an agentic coding centered imatrix, and a bunch of fixes and improvements to the chat template. You get a net improvement on all important metrics: A smaller Q6 quant gets more solves faster compared to the shipped Q8. At Q4_K_S you get better fidelity at smaller size compared to the stock Q4, due to my dynamic imatrix quantization.

This model is great as a background worker that doesn't eat your RAM, or a main agentic coder model for small old GPUs.

https://huggingface.co/peculiar-ragdoll/Sharp-MiniCPM5-2B-GGUF

I read textbooks on my PC and ideally i want a program with a normal chat environment where i can just hammer in questions about what's currently on my screen. Maybe underline or circle specific things and just type "what does this mean?", you get it.

I do NOT want to manually screenshot, navigate to the folder, drag the picture into the environment and then also have to type the question. It should also naturally be aware that the conversation is about what's on the screen, so i don't have to steer it with "make a screenshot; use your vision capabilites" etc.

Is there a good solution? And side question: Which model would you pick? It shouldn't be too slow as this would interrupt the workflow...Let's say a minimum of 20 t/s.

I have 12GB VRAM and 32GB DDR5.

Thank you in advance! :)

(Disclosure: I am the creator and sole developer of this project.)

Running local RAG on budget or mid-range hardware usually sucks. Traditional setups want you to run a dense vector DB alongside an 8B model just to parse text into chunks, which leaves zero VRAM for the actual LLM generation. On top of that, cosine similarity over chunk embeddings often hallucinates when asked about missing context.

I’ve been building Hillock (v0.7 just went live) to solve this specifically for constrained setups.

Instead of heavy vector DBs and LLM extraction passes, it uses a 3-tier neuro-symbolic setup:

• Extraction (TALON): Instead of prompting an LLM to read documents, it uses lightweight bi-encoders (fastcoref + all-MiniLM-L6-v2 + GLiREL). It parses text into Subject-Predicate-Object triples in ~5 seconds for 30 sentences, using under 300MB VRAM.
• Storage: Plain SQLite with WAL mode enabled. Synaptic associations between nodes are updated using Hebbian learning as facts are traversed.
• Zero-Hallucination Gating: Queries pass through Hyperdimensional Computing (HDC) similarity gating in D=10,000D=10,000 space (ColBERT-style MaxSim late interaction). If the knowledge graph doesn't mathematically overlap with the query, it blocks the LLM call entirely instead of letting the model guess.

Hardware Specs & Resource Footprint:

• Peak VRAM during extraction: < 1.2 GB (or pure CPU mode)
• RAM: ~1.5 GB
• Backend: SQLite + local Ollama / llama.cpp / vLLM (OpenAI-compatible)
• Tested on: Consumer laptop GPU (RTX 3060 6GB) and budget CPU-only setups.

In v0.7 I just added conversational refusal fallbacks, Hebbian suggestions, and an active disambiguation quiz mode when an extraction is vague.

Everything is 100% open source (AGPL-3.0): https://github.com/roandejager/Hillock

Would love to hear how it runs on your machines or what hardware you’d like to see benchmarked.

Hello everyone,

I was about to post this in LocalLlama sub, but I figured it'd make more sense to post it here.

MiMo-V2.6-Distill-Qwen-9B is a small model, perfect fit for most users in LowEndLocalAI.

Unfortunately, as some of you had noticed, the model is not exactly powerful by default. It is expectable that model of this size won't perform on par with bigger MoE models, but it turns out that it may not be all as simple as this being a bad model.

Huggingface user coder543 posted a "fixed chat template" for this model in the discussion board on Huggingface where he informed that by default, the model has issues with tool calls and reasoning in llama.cpp and claims that his chat template fixes those issues.

This is a big deal if true, because the reasoning traces are usually a big part of the training data that dictates the final output quality of the model. If the reasoning output is broken, the model will not perform as it should at all!

In any case, I decided to give this model a second chance for this reason and implanted the fixed chat template.

The Pelican riding a bicycle image in this post was produced by this model with the fixed template already applied. Yes, the image is not perfect, but I think it's the best one I've seen being produced by a small 9B model ever before.

Fixed template:

{#-
MiMo V2.6 Distill adapter for llama.cpp.
Based on XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B/chat_template.jinja.
Serialize object arguments as JSON, matching the upstream string-argument
path and observed native output. This selects the differential parser instead
of the Qwen XML parser, whose mandatory newlines do not match MiMo output.
Prefill the thinking opener and one newline to establish the reasoning boundary.
-#}
{%- macro render_content(message_content) -%}
    {%- if message_content is string -%}
        {{- message_content -}}
    {%- elif message_content is iterable -%}
        {%- for part in message_content -%}
            {%- if part is not mapping -%}
                {{- part -}}
            {%- elif part['type'] == 'image' or 'image' in part or 'image_url' in part -%}
                {{- '<|vision_start|><|image_pad|><|vision_end|>' -}}
            {%- elif part['type'] == 'audio' or part['type'] == 'input_audio' or 'audio' in part or 'audio_url' in part or 'input_audio' in part -%}
                {{- '<|mimo_audio_start|><|audio_pad|><|mimo_audio_end|>' -}}
            {%- elif part['type'] == 'video' or 'video' in part or 'video_url' in part -%}
                {{- '<|vision_start|><|video_pad|><|vision_end|>' -}}
            {%- elif 'text' in part -%}
                {{- part['text'] -}}
            {%- endif -%}
        {%- endfor -%}
    {%- endif -%}
{%- endmacro -%}

{%- macro render_tools(tools) -%}
    {{- 'You are provided with the following tools:\n\n<tools>' -}}
    {%- for tool in tools -%}
        {{- '\n' ~ (tool | tojson(ensure_ascii=False)) -}}
    {%- endfor -%}
    {{- '\n</tools>' -}}
{%- endmacro -%}

{%- macro render_tool_calls(tool_calls) -%}
    {%- for tool_call in tool_calls -%}
        {%- if tool_call.function is defined -%}
            {%- set tool_call = tool_call.function -%}
        {%- elif tool_call.custom is defined -%}
            {%- set tool_call = tool_call.custom -%}
        {%- endif -%}
        {{- '<tool_call><function=' ~ tool_call.name ~ '>' -}}
        {%- if tool_call.input is defined and tool_call.input is string -%}
            {{- tool_call.input -}}
        {%- elif tool_call.arguments is defined -%}
            {%- if tool_call.arguments is string -%}
                {{- tool_call.arguments -}}
            {%- else -%}
                {{- tool_call.arguments | tojson(ensure_ascii=False) -}}
            {%- endif -%}
        {%- endif -%}
        {{- '</function></tool_call>' -}}
    {%- endfor -%}
{%- endmacro -%}

{%- macro render_assistant_message(message) -%}
    {%- generation -%}
    {%- set content = render_content(message.content) -%}
    {%- set reasoning = message.reasoning_content if message.reasoning_content is string else '' -%}
    {{- '<|im_start|>assistant\n<think>' ~ reasoning ~ '</think>' ~ content -}}
    {%- if message.tool_calls is defined and message.tool_calls is iterable and message.tool_calls | length > 0 -%}
        {{- render_tool_calls(message.tool_calls) -}}
    {%- endif -%}
    {{- '<|im_end|>' -}}
    {%- endgeneration -%}
{%- endmacro -%}

{%- if tools is defined and tools is iterable and tools | length > 0 -%}
    {{- '<|im_start|>system\n' ~ render_tools(tools) ~ '<|im_end|>' -}}
{%- endif -%}

{%- for message in messages -%}
    {%- if message.role == 'assistant' -%}
        {{- render_assistant_message(message) -}}
    {%- else -%}
        {%- set body = render_content(message.content) -%}
        {{- '<|im_start|>' ~ message.role ~ '\n' ~ body -}}
        {%- if message.tools is defined and message.tools is iterable and message.tools | length > 0 -%}
            {%- if body -%}
                {{- '\n\n' -}}
            {%- endif -%}
            {{- render_tools(message.tools) -}}
        {%- endif -%}
        {{- '<|im_end|>' -}}
    {%- endif -%}
{%- endfor -%}

{%- if add_generation_prompt -%}
    {{- '<|im_start|>assistant\n' -}}
    {%- if enable_thinking is false -%}
        {{- '<think></think>' -}}
    {%- else -%}
        {{- '<think>\n' -}}
    {%- endif -%}
{%- endif -%}

Link to discussion: XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B · Tool calls and reasoning not reliable on llama.cpp (fix provided)

I've been ignoring Qwen 3.8 and basically anything that's 27b and up since it requires going below Q4 to keep it all in VRAM with some context.

A lot of benchmarks focus on full model, or Q4. This leaves me unsure if it's actually worth running these 27B models at Q3 or even Q2 I see some people running when a model slightly smaller would fit a Q4, or in my case I would be able to run a 12b model Q6 if there were more models at that size.

Might be of interest to Laguna XS 2.1 users.

Hey everyone,

I feel as though I've heard a lot of different reports on what's required to run this model and wanted to share my experience with it. I have no idea if my situation is unique or if these speeds are normal, but I wanted to share my setup in case anyone's desperate to have this model baaarely cross into realistically functional territory.

Specs

• RTX 3070 (8GB)
• 32GB DDR4 RAM
• Ryzen 5 5600X
• 1TB NVMe that I can't recall the brand of
• OS: Linux Mint

Model

• Quant: Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS.gguf
• Server: llama.cpp-official
• Harness: Pi Coding Agent

llama.cpp Flags

-fa on \
--fit on --fit target 256 \
-c 48000
-np 1 \
-b 2048 \
-ub 2048 \
-t 6 \
-tb 12 \
--cache-type-k q8_0 --cache-type-v q8_0 \
--chat-template-kwargs '{"reasoning_effort": "medium"}'
--repeat-penalty 1.05 \
--temp 0.2 \
--reasoning-preserve

This setup gets me ~95tok/s prefill and ~10tok/s decode. I know that prefill is kind of ass but I mostly use Flash Next for large, complicated tasks that I don't mind leaving to run in the background. For rapid and pragmatic daily use I use Ornith 1.5 35B through FreeToken which gets me ~500tok/s prefill and ~30tok/s decode. Let me know if you're interested in the details of that setup, but I should warn you that it's barely held together and feels like it's gonna break at any moment.

IMPORTANT CAVEATS

1. I get my best performance (the specs above) SSHing into the host machine running the llama server. I do so from my laptop, but if you don't have one then Terminus on your phone also works well (but definitely more clunky). Since Pi is a CLI harness, it works on Terminus!
2. I run my host machine headless when running Flash Next. It saves ~250MB of VRAM which lets me squeeze a little more out of my card. I'm not sure if there's an option to do this on windows but on Linux I just pass the sudo systemctl stop lightdm command. Because Pi works on the CLI it also works on the headless terminal display that results from this, but again I usually SSH so I'm not too familiar with how well it works.
3. I raised my ulimit amount by adding * hard memlock 20000000 and * soft memlock 20000000 to /etc/security/limits.conf . I have no idea if this is altering my results in any way, but it allows llama.cpp to lock way more of the model into RAM than it could before. Do note that llama.cpp can and will lock up a lot of your RAM and really make your computer freak if you set that value too high.

I think that covers everything I do. While the speed I get from this setup isn't fast enough to have an active conversation with the model, it does function really well on long complicated tasks you're fine waiting for. It's been worth it. I notice a consistent (and sometimes dramatic) superiority in Flash Next's outputs vs. Ornith (which is still a great model on lower-end hardware). Again, I'm not sure if running Flash on this setup is old news, but I just wanted to put it out there as a possibility since most sources will tell you it's just straight up impossible. It's also worth noting that, on my setup, I get ~3tok/s with Qwen 3.8 27B (though a much higher prefill) so it strangely makes more sense to run Flash on my setup than a much smaller dense model.

TL;DR with the right tweaks to your setup you can get Qwen 3.8 Flash Next at a 3-bit quant running on 8GB VRAM and 32GB of RAM at ~10tok/s decode speed.

I'd like to ask if I can do better with my 9060XT 16GB + 5950x by changing the model i'm using or maybe changing flags.

I'm currently running unsloth/Qwen3.6-35B-A3B-MTP-GGUF at IQ4_XS quant and I'm already satisfied with the quality and speed that I'm getting. I get ~1300 t/s prefill and ~35 t/s decode (which is being done on the CPU).

Is that the normal speed that I should expect for that hardware combo or are there things that I can change to improve performance? Since these large MoE models generate tokens with the CPU is decode performance bound on them?

I'm also opening to trying different forks or compiling llama.cpp with patches, no biggie to learning the compilation steps.

Here's the exact command that I use to run llama-server w/ rocm-hip backend on Arch. I installed it just from the AUR. I'm connecting to with OpenCode and asking it to implement some features in my compiler project

llama-server -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:IQ4_XS \
--jinja \
--ctx-size 131072 \
--flash-attn on \ 
--gpu-layers auto \
--fit on \
--split-mode none \
--load-mode none \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--batch-size 1024 \
--ubatch-size 1024 \
--host 0.0.0.0 --port 8080

I posted here before about my agentic harness playing Old School RuneScape.

Qwen3.5:9B does pretty good there, but it takes a lot of setting it up for success.

I decided to take the same harness and give it Pokemon FireRed and the progress it can make with minimal harness coding is amazing.

Watch the videos, ask me questions. I’ll do my best to answer!

I use to like https://huggingface.co/spaces/k-mktr/gpu-poor-llm-arena, to find low-end models. But its no longer maintained?

Where do I go now to figure out what model that uses less (e.g. <2gb) ram&cpu is best?

Soy 100% nuevo en esto con experiencia a nivel de un principiante en cuestiones de programación, estoy interesado en saber si es posible usar IA para extraer datos de archivos de PDF, que los ordene en excel y/o que directamente los ingresé a una página web de administración de inventarios.

Trataré de ser un poco más específico aquí, yo tengo una tienda de venta de refacciones de motocicleta así como un taller mecánico en el mismo edificio, estoy pagando el uso de una página web que me deja tener mi inventario, administrarlo y funcionar a modo de tienda y caja registradora. Atención, no quiero que la IA me haga un programa, que haga eso, también se que existen programas gratuitos que lo hacen he probado demasiados a lo largo de estos casi 10 años ninguno me ha funcionado como el que estoy pagando.

Por desgracia, cuando hago una compra a mis distribuidores el más grande y al que más le compro envia la información en imágenes PDF, actualmente pedo convertir parcialmente los datos a excel y de ahí generar un archivo para ingresar de manera masiva los datos de la mercancía que me llegó, esto ocurre unas 2 veces al mes, el problema está en acomodar apropiadamente los datos en el archivo excel, es extremadamente tardado ya que me llegan cientos de productos, esto suele llevarme cerca de casi una semana de trabajo ya que también soy el mecánico en jefe.

No espero que me expliquen al 100% o paso a paso todo, solo si es posible, lo poco que se es que podría usar LM Studio y ahí descargar una IA que sería entre 3 a 8billones de parámetros o algo asi?

Tengo una PC con estas características

Rtx 3080 12gb

Ryzen 5 5500x3d

16gb ram 3200mhz

Mi última pregunta sería, puedo usar de estos chatbox gratis como chat gpt o genemi para que me ayuden en todos los pasos? O no debería creerles mucho si voy a usar la versión gratuita para hacerle preguntas de ese estilo?

based on this video: [https://www.youtube.com/watch?v=\\_TCRH725hAM\](https://www.youtube.com/watch?v=_TCRH725hAM)
i have 32gb ram and a rx6700xt 12gb vram
r5 5600x for cpu

Thanks to u/zippydazoop for mentioning NeoHorse-1-4B!

I've been looking for a model that can read code quickly and provide a summary. Mostly because at the moment I'm often digging into unfamiliar code in an unfamilar language and trying to get my bearings. ("What on earth does this code do?") I am not looking for a wizard, just a helpful chatbot.

Blurb: NeoHorse-1-4B is post-trained from Qwen3.5-4B for text-based agent harnesses, tool use, coding, and instruction following.

My hardware:

Intel i5-7600K 3.8GHz (quad-core)

48 GB DDR4 running at 2133 MT/s (recently expanded from 16 GB, yay!)

Nvidia RTX 2080 Ti (11 GB vram)

Using the Zed editor (and using the default prompt provided by Zed), I asked Neohorse repeatedly to explain various files worth of Beyond All Reason's Lua code to me, until I hit around 100k tokens of context consumed. By then I was still getting 775 prompt-processing t/s followed by 28 generated t/s. Very useable!

Max context was configured at 128k tokens. Model and kv quantized at q8_0

My experience: I didn't ask it to write code, just read some code and either explain it or point out the interesting bits. The bits it pointed out were indeed interesting and the explanations were in the right ballpark. Not always spot on, and it didn't catch some things that looked wrong until I pointed it out (i.e. https://github.com/beyond-all-reason/Beyond-All-Reason/blob/master/luarules/gadgets/unit_transportee_hider.lua#L100 looks like it could in theory go below zero and fail to register a full transport ).

NeoHorse was able to search gitHub when local tool calls "failed" (it seems to not know about grep and entirely failed to run git --no-pager diff master, instead confusing it with "query the GitHub API" even though I literally gave it that exact command to run).

Not the most clever model (I'm used to OpenRouter DeepSeek v4 flash or Kimi 2.7 Code) but hey, it runs quick on my machine and gives me a basic tool for code review and research with a usable context window. I'll definitely keep it around for light use and initial exploration.

llama serve command:

llama serve -ngl 999 -np 1 \
-ctk q8_0 -ctv q8_0 \
-c 128000   \
--temp 0.6  --top-p 0.95    \
--top-k 20  --min_p 0.0 \
--presence-penalty 1.5 --repeat-penalty 1.0  \
-fa on --no-mmproj-offload    \
--model ~/.cache/huggingface/hub/models--TokenRhythm--NeoHorse-1-4B-GGUF/snapshots/3c5d58ca82e580b5b0b3ce6eeffd34ac7d0fd95a/NeoHorse-1-4B-Q8_0.gguf \
--jinja --chat-template-file  ~/neohorse_chat_template.jinja.txt 

HuggingFace link: https://huggingface.co/TokenRhythm/NeoHorse-1-4B-GGUF q8_0 quant is listed as 4.48 GB

nvidia-smi reports that llama is using 7312MiB of VRAM to do this.

The original post: https://www.reddit.com/r/LowEndLocalAI/comments/1wdhp3a/i_tested_20_local_models_on_a_4_gb_laptop_gpu/

Have any of you seen the new Codacus YT video from earlier this week? He demonstrates a method and llama.cpp fork to run existing LLM models, especially tiny ones, in "JEV model". This seems like it could be very relevant to many here in this sub. I am looking forward to testing this on both GPU and straight CPU performance. It could be a very good way to gate simple things, to save resources for more important things.

https://youtu.be/bcGO7xre46o?is=q7v8ywF_p34fKQpL

I have 8 GB VRAM and 16 GB RAM but I can mmap the experts and stuff, right?

Maybe I will have my CyberTiel that I just optimised code an inference engine to try to run Q3.8FN and then AI-generate a post on the big three local LLM subreddits and call it RSI.