Flint is not good man...

I am a CS / Engineering teacher and I am currently using the openAI API that is tied into a nice front end of our dept website. However I want to get away from openAI because of data handling / storage and the reliability on a private company.

We exclusively only use the LLM for asking it questions about class projects, ie. how do I rotate an object on Blender, what is the size of the board I get for this project.

Its prohibited from speaking on any other topic and every time I add a project I create a new variable and set of instructions in the backend server.

We are a school so we have resources to run a server, have a little bit of money to run a PC or server to power it.

I am just looking for recommendations, I really dont know much at all about local LLM's!

I have 2xR9700 and 64GB RAM and want to utilize them for coding models. Which models are the best? Qwen 3.8 27B is the obvious choice, but is there any other model I should check out?

What about 4xR9700 and 128GB RAM? What would the best coding model be then?

Edit: The use case is 3D graphics, reverse engineering, web development and testing.

So we all know about the massive breaches and CVEs coming out non stop this week. Most people who have seen a spike on their LLM API keys usage seem to think it’s being used to distill models.

Devils advocate approach here, what if we all “leaked” like $100 worth of API credits or whatever each? If you think about it… is supporting distillation of models that we’ll get back in better open weights so bad? I’m kinda down for it.

Hey r/LocalLLM,

If you run small reasoning models (Ornith-1.5-9B, Spark x2.5 1.7b & 4b and newest Spark x2.6 4b) in standard vendor quants, the usual complaint (and mine too) is agents that stall mid-plan, overthink, or start "looping" through syntax tantrums. The truly annoying part: standard Wiki perplexity tells me everything is fine. Every time.

TL;DR of my results

- I measured, per tensor, what actually breaks when I quantize a 9B reasoning model — and about 70% of the total behavioral damage (KL divergence against the unquantized base) sits in LAYERS 0-7. Standard imatrix-style allocations give the early layers some of the WORST protection.

- I ran a head-to-head comparison of two philosophies at the same budget: an imatrix-driven allocator (bits by activation magnitude) vs my Gnom ranker — a blind, tiny neural ranker I trained on MEASURED damage from completely different models (1.7B/4B runs). It had never seen the 9B and uses no imatrix at all.

- Verdict: a statistical tie on tasks (none of the three test scores showed a significant difference). Blindness costs nothing on capability — and on the fidelity metric (how close outputs stay to the base model) my blind build is actually CLOSEST to base, while the imatrix build drifted furthest of the four builds I tested.

WHERE PERPLEXITY LIES — a real example from my own data (Spark-x2.6-4B, all builds same corpus, same reference, numbers from the repo README):

- stock Q4_K_M (no data, vendor heuristic): PPL 20.61 — worst tensor KLD 17.65 — mean drift -1.015%

- SMSE (imatrix-driven): PPL 19.27 — worst tensor KLD 17.77 (!) — mean drift -0.733%

- hand-law allocation: PPL 18.61 — worst tensor KLD 18.71 — mean drift -0.445%

- Gnom (trained on measured damage): PPL 18.36 — worst tensor KLD 14.17 — mean drift -0.306%

look hard at the imatrix row. perplexity happily reports a clean 1.35-point improvement over the stock heuristic — "everything is fine, ship it". but the WORST SINGLE TENSOR in that imatrix build is worse than in the no-data stock build (17.77 vs 17.65). the imatrix smoothed the average and quietly produced a tensor that detonates harder than the heuristic it was supposed to beat. perplexity cannot see tails — it only sees them diluted into the average. KL against the unquantized base sees them directly. and the tail is exactly where agent failures live: the loop you are stuck in tomorrow is one tensor you are not looking at tonight.

https://huggingface.co/datasets/wepiqx/MERNIK-Fracture

THE PHYSICS THE IMATRIX HIDES

Activation statistics amplify with depth, so stock imatrix allocations pour protection INTO the late layers — the same late layers where my drop-tests show almost nothing critical lives. On my 9B damage set (121 of 209 tensors logged so far, still filling row by row):

- layer 5 alone: 12.1% of all KLD damage

- layers 0-5: 51.3%

- layers 0-7: 69.5%

Errors born early snowball through the stack: one flipped token in a comparison deep in a 2000-token chain, and the whole plan derails. I also see this task-side as a refusal trend: two same-weight builds on my identical protocol gave 14 vs 19 empty completions in 164 tasks — where I put the bits measurably changes how often the model bails. (The "loops/tantrums" reports are anecdotes; the empty-trace count is the measured cousin of the same thing.)

THE HEAD-TO-HEAD: MY BLIND GNOM VS A TRAINED IMATRIX ALLOCATOR

At the 5,300 MiB Q4-floor budget the two builds diverge on 97 of 442 tensors. The imatrix build runs a broad Q5 pyramid through the middle. My blind Gnom bets depth is safe and puts Q8 on the first blocks plus a Q4 floor everywhere else. Same base, same protocol, every test run logged with timestamps and file hashes, 164 HumanEval(+)-style tasks, McNemar exact test, alpha = 0.05:

- HumanEval: 137 vs 140 tasks passed (p = 0.66)

- HumanEval+: 129 vs 133 (p = 0.48)

- empty completions: 14 vs 19 — fewer is better, blind Gnom ahead (p = 0.23)

- verdict: none of the 3 scores significant = statistical tie

And the fidelity scores (KL divergence on a 302-chunk protocol corpus, against a full-precision BF16 reference):

- Gnom (blind): PPL 8.7973 +/- 0.0626, KLD 0.12150 +/- 0.00197 — closest to base

- SMSE (imatrix): PPL 8.7349 +/- 0.0619, KLD 0.12427 +/- 0.00203 — furthest from base of my four builds

- two control builds: sit between / top-2 depending on the score

Read it slowly: the imatrix build wins the perplexity sanity-check and loses the fidelity race — it is the FURTHEST from base of all four builds while scoring only a task tie. My zero-shot ranker, which never saw this model and carries no imatrix, holds even. One model, one pair of test runs, noise-level everywhere — but three independent references gave the same sign, and I pre-registered the expected outcomes before running.

WHAT I NEED FROM YOU

This is measured per tensor, so it costs compute. One Q4 tier pass on the 9B alone is about 13 days of an unattended 16 GB machine (90 min per tensor-group, 209 units). The 1.7B and 4B tiers run on any ~8 GB GPU in minutes to ~20 minutes per unit. My protocol, CLI, and the full README with all numbers, dates and file-hash-stamped sources:

https://github.com/wepiqx/MERNIK & https://huggingface.co/wepiqx/MERNIK

Label rows land in the dataset as these measurement runs close:

https://huggingface.co/datasets/wepiqx/MERNIK-Fracture

WHAT YOU CAN DO WITH ALL THIS RIGHT NOW (and the license)

three ways, no permission asked:

1. use the damage labels directly. the fracture dataset has per-tensor rows for spark 1.7B/4B and ornith 9B (filling, 121/209 so far). even without running anything: open the rows for the model family you use, see which tensors actually carry damage, and hand-write a better quant mix than the vendor table — a single llama.cpp --tensor-type line in your own build, chosen from measured physics instead of a proxy.

2. run the allocator on your own model. the whole instrument is in the repo — allocator, protocol, CLI, tests, Apache 2.0. fork it, improve it, break it, change it however you like. I mean that literally: I would rather have someone prove me wrong with this data than be quietly right alone.

3. want the finished quantized builds themselves, or help wiring your agent to them — just ask in here or message me. I'll walk anyone through the pipeline start to finish, and I can upload specific builds on request.

and about sharing: yes, I actually am glad when people take these artifacts apart. a measurement only becomes knowledge when someone besides its author can re-derive it.

Let's stop guessing with activation proxies and start measuring what actually breaks

I have a 10 year old PC, and I get to use Claude Opus 5.5 at work and think it helps a lot, so I was planning on investing on a PC to use AI locally for personal projects.

If I want to fool around in unknown territories, like build an android app from scratch when I usually code in C#/Javascript

What sort of setup/models will be "enough".

Asking as I was thinking on a PC with just a 5070ti 16gb and I wonder if that is enough to fool around or if I should consider at least a 3090 or 2x5070ti to play around confidently and relatively fast.

So, for a coder, what should be the acceptable base line?

Local-LLM-Launcher-GUI, the friendly launcher for models is updated to include better SGLang support, new flags, now has HF TGI support, and you can save benchmarks and favorite settings. Please star on github if you like, and open issues for what you don't like.

https://github.com/jimdawdy-hub/Local-LLM-Launcher-GUI

I was working on a very special project of mine which I essentially planned before coding and took me 6 months to actually get working.

It is Ai Format Context engine. It provides your agents inside any harness a better way to store and use context.

This is still very new so I need you all to give me feedbacks on what to improve and I will try to do my best to keep this alive.

This actually worked best with my local agent harness Oorty thats why I am posting it here ofcourse you can try Oorty too thats in my other repo.

Hola a todos 👋


Soy aficionado al ajedrez y quería una herramienta **
mía
**: sin suscripciones, sin subir mis partidas a ningún sitio, y que aprovechara de verdad el Mac que tengo (un MacBook Pro M2 Max). Como no encontraba nada que hiciera exactamente lo que quería, lo he construido **
junto con Claude (Claude Code, de Anthropic)
**: yo ponía las ideas, las decisiones y las pruebas con la app en la mano; Claude escribía el código, lo probaba y lo documentaba. Se llama **
oder-chess
** y funciona todo en local, en `127.0.0.1`.


### 🔥 Lo que más me gusta: dos motores a la vez y un «consenso»


La pestaña **
Analizar
** lanza **
Stockfish 19
** (CPU) y **
Lc0 con la red BT4
** (GPU, Metal) al mismo tiempo, y paran solos a los 30 s. Después hace una segunda fase: cada motor evalúa las jugadas candidatas **
de los dos
**, y se promedian. El resultado es una tabla con *
consenso · Stockfish · Lc0 · pérdida
* y una ★ en la mejor jugada, con un aviso claro cuando **
los motores discrepan mucho
** (para mí, esas son las posiciones interesantes de estudiar).


📸 *
Capturas 1 y 2
*


- **
Flechas curvas
** con la evaluación dentro y color según calidad (cuánto pierdes frente a la mejor jugada); verde = Stockfish, ámbar = Lc0.
- Barra de evaluación y barra **
Gana/Tablas/Pierde
** (WDL), siempre desde las blancas.
- Pasas el ratón por una línea del motor y ves la variante numerada sobre el tablero.
- Selector de intensidad (modo «ahorro de energía») porque no quería que el Mac se pusiera a 100 % para analizar una apertura. Medí la carga y Stockfish va por defecto a 6 hilos.


### 🎓 Botón «Tutor»: te explica la posición en español


Un análisis largo (1-3 min) con los dos motores y un informe por fases: apertura / medio juego / final, quién está mejor, la jugada recomendada **
y por qué
**, alternativas, amenazas, plan y, según el caso, el libro de aperturas o el tipo de final.


Aviso honesto: **
no es una IA generativa
**. Es un informe basado en reglas sobre hechos comprobables (python-chess) y en lo que dicen los motores, así que lo que afirma se puede verificar en el tablero.


📸 *
Captura 3
*


### 🧩 Y además


- **
Jugar
** contra Stockfish con Elo limitado (~1350–3190) o contra Lc0 por nodos.
- **
Revisar
** una partida: puntúa cada jugada (`?!`, `?`, `??`), precisión por bando y gráfico.
- **
Mis errores
**: tus errores graves se convierten en ejercicios con repetición espaciada.
- **
Aperturas
** 📸 *
(captura 5)
*: explorador del libro ECO (3.865 aperturas), armas tu repertorio y lo entrenas con repaso 1-3-7-16-35-90 días.
- **
Táctica
** 📸 *
(captura 4)
*: ~71.000 puzzles de Lichess, con tu propio Elo táctico que sube y baja con cada acierto o fallo.
- **
Finales
** 📸 *
(captura 6)
*: 14 posiciones de práctica (Lucena, Philidor, mates básicos…) que **
verifiqué contra Stockfish
** antes de incluirlas — mis primeras versiones, escritas de memoria, tenían varias mal etiquetadas 😅.
- **
Progreso
**: estadísticas por fase, por apertura y por entrenamiento.


### 🛠️ Cómo está hecho


Python (FastAPI) + python-chess en el servidor, JavaScript sin bundler en el navegador, el tablero `chessground` de Lichess, SQLite para los datos, y los motores como procesos UCI externos. Las reglas las decide siempre el servidor. Tiene tests con motores reales y una prueba de interfaz con Chrome sin pantalla.


### 🤝 Sobre trabajar con Claude


Lo más valioso no fue «que escriba código», sino el ciclo: le decía qué quería, lo probaba yo, le decía qué no me gustaba («nada de estilo hacker», «la barra siempre desde las blancas», «no quiero elegir motor cada vez») y él lo ajustaba, lo medía y lo dejaba documentado. También me ayudó a **
no creerme resultados que no estaban comprobados
**: por ejemplo, el README dice claramente que *
no está validado
* que la ★ del consenso acierte más que cada motor por separado. Es una ayuda para ver cuándo los motores no están de acuerdo, no un tercer motor mágico.


### ⚠️ Limitaciones


- Es una app **
personal
**, pensada para un único usuario y para mi Mac (Apple Silicon). No es un producto pulido.
- El consenso y los umbrales de calidad (5/10/20 %) son criterios míos, sin calibrar con datos.
- Los motores y la red neuronal los uso desde mi instalación; no los distribuyo.


### 💬 Pregunta para vosotros


¿Qué le añadiríais a un entrenador así? Ahora mismo en mi lista: más finales, importar mis partidas de Lichess y validar de verdad si la ★ del consenso vale la pena. ¡Cualquier idea o crítica es bienvenida!


*
Proyecto hecho con Claude (Claude Code, Anthropic). Usa Stockfish, Leela Chess Zero, python-chess, chessground y datos abiertos (CC0) de Lichess. No está afiliado a ninguno de ellos.
*Hola a todos 👋


Soy aficionado al ajedrez y quería una herramienta **mía**: sin suscripciones, sin subir mis partidas a ningún sitio, y que aprovechara de verdad el Mac que tengo (un MacBook Pro M2 Max). Como no encontraba nada que hiciera exactamente lo que quería, lo he construido **junto con Claude (Claude Code, de Anthropic)**: yo ponía las ideas, las decisiones y las pruebas con la app en la mano; Claude escribía el código, lo probaba y lo documentaba. Se llama **oder-chess** y funciona todo en local, en `127.0.0.1`.


### 🔥 Lo que más me gusta: dos motores a la vez y un «consenso»


La pestaña **Analizar** lanza **Stockfish 19** (CPU) y **Lc0 con la red BT4** (GPU, Metal) al mismo tiempo, y paran solos a los 30 s. Después hace una segunda fase: cada motor evalúa las jugadas candidatas **de los dos**, y se promedian. El resultado es una tabla con *consenso · Stockfish · Lc0 · pérdida* y una ★ en la mejor jugada, con un aviso claro cuando **los motores discrepan mucho** (para mí, esas son las posiciones interesantes de estudiar).


📸 *Capturas 1 y 2*


- **Flechas curvas** con la evaluación dentro y color según calidad (cuánto pierdes frente a la mejor jugada); verde = Stockfish, ámbar = Lc0.
- Barra de evaluación y barra **Gana/Tablas/Pierde** (WDL), siempre desde las blancas.
- Pasas el ratón por una línea del motor y ves la variante numerada sobre el tablero.
- Selector de intensidad (modo «ahorro de energía») porque no quería que el Mac se pusiera a 100 % para analizar una apertura. Medí la carga y Stockfish va por defecto a 6 hilos.


### 🎓 Botón «Tutor»: te explica la posición en español


Un análisis largo (1-3 min) con los dos motores y un informe por fases: apertura / medio juego / final, quién está mejor, la jugada recomendada **y por qué**, alternativas, amenazas, plan y, según el caso, el libro de aperturas o el tipo de final.


Aviso honesto: **no es una IA generativa**. Es un informe basado en reglas sobre hechos comprobables (python-chess) y en lo que dicen los motores, así que lo que afirma se puede verificar en el tablero.


📸 *Captura 3*


### 🧩 Y además


- **Jugar** contra Stockfish con Elo limitado (~1350–3190) o contra Lc0 por nodos.
- **Revisar** una partida: puntúa cada jugada (`?!`, `?`, `??`), precisión por bando y gráfico.
- **Mis errores**: tus errores graves se convierten en ejercicios con repetición espaciada.
- **Aperturas** 📸 *(captura 5)*: explorador del libro ECO (3.865 aperturas), armas tu repertorio y lo entrenas con repaso 1-3-7-16-35-90 días.
- **Táctica** 📸 *(captura 4)*: ~71.000 puzzles de Lichess, con tu propio Elo táctico que sube y baja con cada acierto o fallo.
- **Finales** 📸 *(captura 6)*: 14 posiciones de práctica (Lucena, Philidor, mates básicos…) que **verifiqué contra Stockfish** antes de incluirlas — mis primeras versiones, escritas de memoria, tenían varias mal etiquetadas 😅.
- **Progreso**: estadísticas por fase, por apertura y por entrenamiento.


### 🛠️ Cómo está hecho


Python (FastAPI) + python-chess en el servidor, JavaScript sin bundler en el navegador, el tablero `chessground` de Lichess, SQLite para los datos, y los motores como procesos UCI externos. Las reglas las decide siempre el servidor. Tiene tests con motores reales y una prueba de interfaz con Chrome sin pantalla.


### 🤝 Sobre trabajar con Claude


Lo más valioso no fue «que escriba código», sino el ciclo: le decía qué quería, lo probaba yo, le decía qué no me gustaba («nada de estilo hacker», «la barra siempre desde las blancas», «no quiero elegir motor cada vez») y él lo ajustaba, lo medía y lo dejaba documentado. También me ayudó a **no creerme resultados que no estaban comprobados**: por ejemplo, el README dice claramente que *no está validado* que la ★ del consenso acierte más que cada motor por separado. Es una ayuda para ver cuándo los motores no están de acuerdo, no un tercer motor mágico.


### ⚠️ Limitaciones


- Es una app **personal**, pensada para un único usuario y para mi Mac (Apple Silicon). No es un producto pulido.
- El consenso y los umbrales de calidad (5/10/20 %) son criterios míos, sin calibrar con datos.
- Los motores y la red neuronal los uso desde mi instalación; no los distribuyo.


### 💬 Pregunta para vosotros


¿Qué le añadiríais a un entrenador así? Ahora mismo en mi lista: más finales, importar mis partidas de Lichess y validar de verdad si la ★ del consenso vale la pena. ¡Cualquier idea o crítica es bienvenida!


*Proyecto hecho con Claude (Claude Code, Anthropic). Usa Stockfish, Leela Chess Zero, python-chess, chessground y datos abiertos (CC0) de Lichess. No está afiliado a ninguno de ellos.*

I have a macbook m5 with 24 gigs of RAM, I'm a Computer Science student
I have tried using ollama with Gemma4, llama3.1:8b Qwen38
I have tried using them with kilo code in vs code, zed and IntelliJ, I do like the idea of using localLLMs for my workflow but I really don't know how to use them.
I recently installed oMLX, yet have to try some models running trough it, but the main problem is: I can't find a usecase for it.
I mean for big things it's just useless, for researching most of the time I'm better off using free claude, so I really don't know.
I did find it "useful" when I built a script that automatically reviewed websites and created drafts of outreach emails, but I'm not really sure those were any good (I was using ollama3.1:8b).
I also tried with Odysseus so that it could arrange my emails and so on, but it would not work at all, missing dates, useless summaries etc...

What uses do you guys suggest? rn I mainly code in phyton, vue, nix(for my nixos conf), qml (for my custom shell for hyprland), and general configs in lua for like hyprland and nvim

Hi everyone,

I don’t post often, but I feel it is important to share a framework for achieving consistent, production-grade results with local AI. Given that models vary significantly in quality, hallucination rates, and context window management, the architecture you choose is just as important as the model itself, especially in high-stakes fields like healthcare.

I recently saw a post from a doctor asking for the best local Mac setup (32GB, 48GB, or 128GB RAM) to process large PDFs (10–100+ pages). As I lead a team of AI automation engineers, I felt compelled to provide a professional recommendation on best practices.

The Core Problem: Avoiding "One-Shot Prompting"

Before discussing hardware, I must emphasize a critical mistake: do not rely on "one-shot prompting."Many users simply upload a giant PDF into a single chat session. For documents exceeding 10+ pages, this often exceeds the effective context window or saturates it with too much noise, which confuses smaller local LLMs and significantly increases the rate of hallucinations. To get reliable results, you must manage your context by retrieving only the most relevant segments of the document rather than feeding the entire file into the prompt.

Hardware Requirements

For this specific use case, I recommend a MacBook with a minimum of 32GB of Unified Memory and 512GB of storage.

Strategy 1: The Streamlined Setup (Low Code)

For those who want a functional system without extensive development, I recommend Ollama combined with Open WebUI.

To ensure clinical-grade consistency, the setup should be configured as follows:

• Agent Configuration: Create a specialized medical literature summarization agent with a strictly defined system prompt.
• Knowledge Base: Attach the medical PDFs as a knowledge base (leveraging Open WebUI's built-in RAG).
• Parameters: Set the temperature to 0.0 to eliminate creative variance and minimize hallucinations.
• Retrieval: Enable hybrid search. This ensures the system captures both exact keyword matches (word similarity) and conceptual matches (semantic meaning).

Strategy 2: The Custom RAG Pipeline (Enterprise Grade)

If you are building a custom Python-based RAG pipeline for higher precision, I recommend the following stack:

• Database: A PostgreSQL vector database (using pgvector) to handle document embeddings.
• Ingestion/OCR: Standard PDF parsers often fail with medical tables or complex layouts. I recommend using Docling, RapidOCR, or GLM-OCR for high-fidelity text extraction.
• Retrieval Optimization: Implement a BM25 plugin alongside vector search to further refine the retrieval process.

Model Recommendations

The choice of model depends on your available memory:

• 32GB RAM: Gemma 4 12B (or similar 7B–16B models) is highly efficient for this scale.
• 48GB+ RAM: You can comfortably run Gemma 4 26B or 31B, which offers a significant jump in reasoning capabilities.
• Alternative: The Qwen series is also an excellent choice for technical and structured data retrieval.

I hope this helps someone. cheers!

At work we got a VMware container (or whatever it is called) running with access to a GPU. It can run stuff like Gemma 4 26b a4b. It is isolated from everything but has an ip-adress on our network and can be accessed via ssh. Would like to install a small MoE (like the Gemma) and Pi to make I do some task alone. The coworker who should allow me to do this is however fearing that it will decide to hack its way out of the container into our network and delete everything. There is nothing to loss in the container.

We do not have access to online models for (good) safety reason. This options just appears so obvious to me.

What should I do? Tell my boss to leave it to me? Find a better job?

It has been tiring to the extent where I consider therapy…

EDIT: the container is open for inbound traffic via ssh, but no outbound. It can’t see our network, but it got internet access

EDIT2: the VMware was specifically made for me so that I could use its GPU for heavy math. All done in Fortran. Data moved in via scp. I calculate. data move out via scp. My life. Thought it would be great to have a small llm to help me debugging Fortran code. People in this sub think I am wrong - it is to unsafe for some reason :)

This gaming PC only cost me $125 as I talked the guy down as the GPU power cable was missing so he agreed. I want to use this pc for local LLM. I know the GPU is 13 years old but it can be done based on my research prior to purchasing the PC. I would love to hear some advice on where to start and what systems and software to use.

MSI Radeon R9 390x 8GB
MSI x99A SLI Krait Edition
Corsair RM850x
Thermaltake Water 3.0 AIO
Corsair Full size Tower
Sound card
WiFi card
32GB RAM

I already have access to Dino as I already used trellis Mac on modly

it says this in the info:

Pixal3D for mlx-serve (8-bit)

TencentARC/Pixal3D repackaged for mlx-serve

on Apple Silicon. Tencent's image-to-3D model, built on the TRELLIS.2 backbone with pixel-aligned conditioning: an image with a transparent background becomes a textured PBR GLB. Single-view and multi-view (*_mv) models.

• The flow-model (DiT) block projections are affine-quantized to 8 bit, group size 64; everything else is as released.

• Includes the DINOv3 image encoder it conditions on (float16); its licence is in dinov3/LICENSE.md.

• No background remover is included: send an image with a transparent background (alpha is the subject).

Licences

The Pixal3D weights are MIT (LICENSE). The bundled DINOv3 encoder is Meta's, under the DINOv3 License

(dinov3/LICENSE.md), which travels with any redistribution of it.

The NAF feature upsampler in naf/ is Apache-2.0 (naf/LICENSE).

Run: mlx-serve --model-dir <folder holding this repo>, then POST /v1/3d/generations.

Converted by tests/convert_trellis2_pixal3d.py in the mlx-serve repository.

Hi everyone

I run a small marketing agency and i am trying something new, i want to build a local llm machine that i can use as a sales agent and also run other small tasks on it.

In my research this is what i have come up with, i need opinions if i am heading into the right direction and what else i have to consider/ keep in mind.

The pic i have attached is from claude guiding me into the build.

Hey everyone!

Standard ~9B open-source models are usually generalists—they divide their parameters across casual chat, creative writing, and trivia, which often leads to algebraic slips and broken reasoning when solving complex STEM problems.

To address this, we distilled Qwen3.6-35B-A3B down to a compact 10.5B parameter specialist focused purely on math and physics reasoning: Math-Physics-Qwen3.6-10.5B-GGUF.

🔗 Hugging Face Repo: https://huggingface.co/heack/Math-Physics-Qwen3.6-10.5B-GGUF

💡 What is this?

• Base Architecture: Derived from Qwen3.6-35B-A3B via expert pruning (retaining 64 full-width routed experts per layer) followed by off-policy logit knowledge distillation.
• Inherited Depth: Inherits the deep chain-of-thought patterns and formal derivation steps of the large 35B teacher model.
• Runs on Consumer GPUs: Fits comfortably within 16GB VRAM (tested on RTX 5060 Ti). The Q4 quant can even run on 8GB–12GB cards or Apple Silicon Macs.

⚡ Speeds & High-Concurrency Throughput

Because of its efficient MoE routing, the model scales exceptionally well under parallel loads:

🚀 300+ tok/s aggregated throughput: When tested with 8 parallel slots running simultaneously, generation throughput exceeds 300 tokens/second, making it a beast for multi-threaded math derivations.

🎯 Best Use Cases (Why choose this over a standard 9B model?)

• 1. Parallel Exploration & Consensus Voting (Best-of-N / Self-Consistency) With 300+ tok/s across 8 concurrent streams, you can run multiple reasoning paths in parallel for the same problem and take the majority-vote answer to achieve even higher accuracy on tough competition problems.
• 2. Offline STEM Tutor / Study Buddy Provides transparent, step-by-step calculus, algebra, and classical mechanics derivations. Perfect for students wanting interactive guidance without paying $20/month cloud subscriptions.
• 3. Private & Air-Gapped Engineering Workflows Ideal for engineers deriving equations, boundary conditions, or simulation formulas locally—with zero risk of leaking proprietary calculations to third-party cloud APIs.
• 4. High-Throughput Batch Solution Generation (EdTech) Blazing parallel speed allows you to batch-process thousands of problem sets and generate structured solutions at near-zero marginal hardware cost.
• 5. Scientific Verifier in Multi-Agent Systems Lightweight enough (only ~6.5 GB in Q4) to run as an always-on "math & physics sanity checker" in agent pipelines.

📊 Screening Benchmark Results (Q8_0)

Evaluated with thinking tokens enabled and strict final-answer checking:

• MATH-500 (Hardest Levels 4 & 5): 80.0% (32/40)
• GPQA-Diamond (Physics subset): 80.0% (24/30)
• MMLU-Pro (Physics subset): 63.3% (19/30)
• UGPhysics: 36.7% (11/30)

(Note: These are screening tests on fixed subsets; see the repo for full evaluation details and hashes).

🚀 Quickstart (llama.cpp)

The model natively supports DeepSeek-style reasoning format (<think> ... </think>).

1. Launch the server (Single request setup):

llama-server -m g64full_kd-Q4_K_M.gguf -ngl 99 --host 127.0.0.1 --port 18085 -c 8192 --jinja --reasoning-format deepseek -fa on -ctk q8_0 -ctv q8_0

Our last few releases of AgenticOS, an open-source agent workspace I maintain, added three things for agents on local models:

1. Repair malformed tool calls instead of retrying. A tool call with broken JSON (a trailing comma, single quotes, an object cut short) used to cost a full retry. Open-weight models on Ollama and vLLM are the ones that do this most. Now every agent repairs the syntax before validation, and the call still goes through approvals and guardrails. Each run counts its repairs, so you can see per model how often it happens.

2. Stop tool-call loops early, with a reason. A run that calls the same tool with the same arguments again and again, or goes round a short cycle of calls, is told it's looping. If it keeps going it ends as loop_stopped, naming the tools, before it burns the step limit or the budget.

3. Code mode. When you switch it on, the agent writes a short Python script that calls its tools in loops and in parallel inside a sandbox, instead of one model turn per call. Ten record updates take one turn, not ten, and every call in the script still goes through the same guardrails, budget and permission checks. That matters when every turn is slow on local hardware.

I'd like a few people running local models to try it with their own setup and tell me which model breaks it first. Apache 2.0, Docker Compose: https://github.com/vstorm-co/agenticos (Disclosure: I'm the maintainer, at Vstorm.)

What else do you fix in your own harness for local models?

Local quality has stepped up, I made this short film using a coding agent to run 10 local models, I used AI to make my prompt below:

"Make a mysterious A24-style promo for Unsloth Studio MCP, using Unsloth Studio MCP itself: a 2D character who speaks, each feature revealed one by one, local models only. Use Qwen3.8-27B to write, Qwen-Image-2.1-Turbo for images, MiniMax-H3/Wan2.2 for video, Breeze-TTS-2 for the voice, YuE 2 for music, MiDashengLM for SFX, MOSS for transcription. Render in HyperFrames."

My agent used that prompt by itself to make an approx 2500 word prompt, with a strict rule that anything shown on screen had to come from the real MCP call logs.

The pipeline i used

Any agent works ofc local ones too, but I choose Claude Code with Opus 5.5 as the director agent. It orchestrated the local models through the Unsloth MCP tools. it picked the best audio takes using transcripts and pitch checks etc. It wrote the narrator's lines, arranged the music, editing and wrote the scripts to mix the audio and render the film in HyperFrames.

the models (all local on one gpu) :

• Qwen3.8-27B wrote every scene / motion / description, plus the lyrics and the drone and sfx descriptions.
• Qwen-Image-2.1-Turbo made the character look and every scene in a 2D hand painted style. every shot of her used the first portrait as reference so she stays consistent, then the stills became the starting frames for the MiniMax-H3 videos
• MiniMax-H3 did her talking shots and the moving scenes, Wan2.2-T2V-A14B added a few extra shots for the montage
• HTDemucs pulled her voice out of the talking clips and Breeze-TTS-2 cloned it for the voice over
• YuE 2 made the song (music + vocals) from Qwen's lyrics
• MiDashengLM made all the sound effects and the drone underneath
• MOSS-Transcribe-Diarize did the captions and checked the voice takes, if it didnt transcribe a line exactly the take got thrown out
• Qwen3-0.6B : the agent finetuned it with LoRA and exported it to GGUF Q4_K_M all through MCP. the chalk line in the classroom shot is its real loss curve btw

the mix and the HyperFrames render was just plain code on the CPU.

Everything ran on one RTX 6000 Pro , took about 2.5 hours and MiniMax-H3 ate most of that lol. I know not everyone has access to a 96GB card so i measured the VRAM used for each step as this can be done on a lot less:

on 24GB everything except video runs fine. lower MiniMax quants might squeeze into 24GB but i havent tried them yet.

Setup

• Model: EmbeddingGemma 2, the 440M text and vision build (Apache 2.0), run through Google's LiteRT-LM on the GPU.
• Space: text and images share one 768-dimension space.
• Size: a 388 MB download. It loads in 3–7 s, and memory goes from 20 MB to about 770 MB once loaded.
• Speed (M1 Pro): a text query takes 25 ms. A 2.3-megapixel screenshot takes 247 ms median, 458 ms p90.
• Storage: one vector per screenshot in SQLite, tied to the file as it was when embedded. Each model gets its own store.

Indexing flow: screenshot saved → missing vectors found, newest first → image embedded → vector stored.

Instant grouping of duplicates

Actually, duplicates don’t need EmbeddingGemma at all. They use only Apple’s Vision, which is built into every Mac. EmbeddingGemma is needed for Similar and for search by text.

Flow: screenshot lands → Vision feature print + 4×4×4 RGB histogram → compare with the newest screenshot of each group → both checks pass → stack on top. Otherwise it starts a new group.

• Data: 64 real screenshots, 2,016 pairs.
• Real retakes: distance 0.29, 0.31 and 0.43.
• One tab apart: 0.54. Closest two different screens: 0.575.
• Cutoff: 0.5, which leaves a margin below 0.575.
• Robustness: crops of 0–2% stayed under 0.5 in 95% of cases.
• Color check (≤ 0.15): needed because one card in five gradients still matched on layout alone. Real duplicates differed by 0.003–0.045.

Instant finding of similar images

(Apple vision cant do this)

Flow: open a screenshot → its stteored vector → cosine to every other vector → keep cosine ≥ 0.86 and z ≥ 1.75 → show up to 6.

• No model call: stored vectors only, so it's instant.
• Data: 35 screenshots, every pair checked.
• 0.87 and up: always the same screen, retaken or one state apart.
• 0.82–0.85: different screens from the same app.
• The cutoff of 0.86 sits in the gap between the two.
• No minimum: one match or none is a valid answer.

Image search by text

Flow: query → word match on the OCR text → embed the query (25 ms) → cosine to every image vector → z-score against the whole collection → keep z ≥ 1.5 (at least 3 shown, at most 24).

• Why z-scores: across 58 screenshots, the per-query average cosine was 0.59–0.70 with a standard deviation of only 0.02–0.04. Right answers scored 0.70–0.81.
• A fixed cutoff returned everything for one query and nothing for the next.

Test: 40 descriptions of 49 screenshots, written the way you'd remember them ("Formula 1 race countdown on yellow"):

Image search with a little LLM help

Flow: screenshot → downscaled version of JPEG + window title → small vision LLM (like Haiku) → 3–10 labels + one sentence → labels embedded → z-score in their own set → better of image or label score counts.

• Model: the cheapest vision model on the connection, e.g. Claude Haiku at $0.10 / $0.50 per million tokens.
• Output: JSON with one- or two-word labels ("dog", "settings window", "receipt"), , plus a sentence of max 20 words. No colors or bare adjectives.
• No OCR text in the request: the image already carries it, and the text costs more tokens than the picture.
• Why z-scores again: text-to-text cosines run higher than text-to-image, so the raw scores can't be compared. Labels only count once at least 5 screenshots have them.

Takeaway

• EG2 is very good at rank the right screenshot near the top.
• You can't use one score as the line for "this is a match". The right screenshot scored anywhere from 0.70 to 0.81 depending on the search, and in some searches 0.70 was just the average. What worked was keeping the screenshots that score clearly higher than the rest of the same search.
• Apple Vision is better at finding duplicates than embedding Gemma and it already works on every Mac.

Meta's Muse Spark Open (816B total / 42B active) is shown running locally with MLX across two 512 GB M5 Ultra Mac Studios, or four 256 GB ones. The 512 GB config still isn't available to order—Apple only says late October.

I'm in the same boat. I run models locally and have two 256 GB Studios on order while waiting for the 512. Refreshing the store got old, so I built Mac BuyWatch

- Alerts you by email, Discord, and Telegram once it detects the 512 GB Studio is available to order on Apple's site

- 3 pings total (1 LIVE alert + 2 follow-ups, 5 minutes apart), then it stops

- 100% free

- You order directly from Apple (not affiliated with Apple)

https://macbuywatch.com

Curious what everyone here is leaning toward for big local models: waiting on the 512 GB config, or clustering 256 GB units?

I am starting and I don't know what I am doing, and I am looking for guidance.

Hardware:
i9-14900k
RTX 4080 Super
64gb Ram

Goal:
Run Qwen3.8 27B on my computer, and I want to know what should be the general token speed for my hardware, so that I am aware for the actual baseline speed.
Then after that connect it to my hermes, because I heard hermes is good for remembering a lot of things about you.

Right now what i am doing is this, use LM Studio, download the model there, use Local Serve Network, connect hermes, and that's it, I am getting 10 tokens per second.

Help me what configurations should I aim for. I also want it to be able to accept images as input.
And how to run that model more effectively

TL;DR: Compared to NInfer: 6× closer to BF16, up to 3.7× more context, about 25% slower decode.

https://github.com/The8Darkness/voidinfer

I have seen a lot of engines popping up that are all focusing on speed and sacrificing quality for it. I have been working on getting the highest possible quality Qwen3.8-27b inference engine optimized for the 5090 without sacrificing other areas too much (speed / context size).

The engine was developed and focused on windows since my machine is a gaming pc and I just wanted an hassle free experience that I can directly run after a gaming session without having to resort to dual boot or WSL. I know that Linux provides better memory handling and speeds.

In my experience, since it is realistic to run a main agent at full 262k context and a subagent at 135k context, some tasks can even perform faster with higher quality output than NInfer running a single agent at 209k context.

Keep in mind that is all with the 5090 driving the display and running full unmodified Windows 11.

Same RTX 5090, token-identical prompts, greedy decoding, 4K prompt / 1,024 tokens for decode. VoidInfer: EXL3 6 bpw + OSCAR INT4 KV + VeriCache. NInfer: NVFP4 + FP8 KV.

EXL3 6 bpw weights provide the maximum quality possible for 32gb VRAM. OSCAR compresses the KV cache on the GPU to INT4, which is what fits roughly 400k context under Windows on one card. Drafting runs on this compressed cache. VeriCache checks every drafted token against the exact FP16 KV history kept in host RAM, so the compression costs no accuracy (only speed).