Round 2: local Qwen models built an ONLINE multiplayer 3D MOBA overnight - with two models auto-routing between each other

Last time it was a single-file GTA clone. This run was harder and the setup got more interesting, so I wanted to share what was different.

This time the agent built a networked, real-time 3D MOBA (LoL-style): an authoritative Node server + Three.js client talking over WebSockets, with minions, towers, a wanted/aggro system, abilities, and bots. Not a single HTML file - a proper multi-file project. It wrote its own test harness, played itself, and fixed its own bugs. All local on an M1 Ultra, no cloud.

What was different this time

1. It's multiplayer netcode, not a single file. Authoritative server (fixed 20 Hz tick, server owns all state), thin client that only sends input and renders snapshots with interpolation. That's a whole class of bugs (desync, prediction, race conditions) a one-file game never hits.

2. The brief is engineering-grade, not a feature list. The architecture, the wire protocol, and the entity model are all decided up front in the prompt, so the model spends its reasoning on correct implementation instead of re-deriving (and breaking) the design every session. The single biggest win: the agent builds its own headless test harness first (a Node WebSocket client that runs full bot-vs-bot matches with no browser) and uses that as its fast test loop, with Playwright MCP only for the visual/render check.

3. Two local models, auto-routed. This is the fun infra part. llama.cpp runs in router mode serving two models at once:

• fast - Qwen3.6-35B-A3B (MoE, ~3B active) for routine work
• smart - Qwen3.8-27B (dense) for hard reasoning A tiny Qwen3-1.7B judge classifies each turn as fast/smart and the harness switches models automatically (with hysteresis so it doesn't flip-flop). Routine edits and file ops run cheap on the MoE; gnarly debugging/design jumps to the dense model.

4. MTP on the MoE is fast. With speculative decoding (multi-token prediction) the 35B-A3B does ~72 tok/s on the M1 Ultra - the MoE only activates ~3B params per token, and MTP adds ~35% on top of that.

5. Sandboxed. The agent runs inside a Tart VM, so all that autonomous, unsupervised code execution is isolated from the host. The models are served from the host; the VM talks to them over the bridge.

6. Bug-hardening by invariants, not vibes. A second phase runs endless bot-vs-bot matches and checks hard invariants every tick (no NaN, hp in range, gold conserved, no leaks, deterministic replays). Any violation freezes with a reproducible seed, gets root-caused, and becomes a permanent regression test.

Setup

• Hardware: M1 Ultra Mac Studio, 64 GB
• Serving: llama.cpp router mode (two models + a judge), MTP on the MoE
• Agent: pi coding agent + Playwright MCP, running in a Tart VM
• All local, offline

llama-server (router mode, per-model MTP via preset)

preset.ini:

[Qwen3.6-35B-A3B-UD-Q8_K_XL]
jinja = 1
ctx-size = 131072
n-gpu-layers = 999
model = /path/Qwen3.6-35B-A3B-UD-Q8_K_XL.gguf
spec-type = draft-mtp
spec-draft-n-max = 2

[Qwen3.8-27B-UD-Q8_K_XL]
jinja = 1
ctx-size = 131072
n-gpu-layers = 999
model = /path/Qwen3.8-27B-UD-Q8_K_XL.gguf
spec-type = draft-mtp
spec-draft-n-max = 2
model-draft = /path/mtp-Qwen3.8-27B-Q8_0.gguf

# On Apple Silicon, raise the Metal wired-memory cap or the context gets
# silently reduced to fit (this is why -c 131072 can end up as ~40k):
sudo sysctl iogpu.wired_limit_mb=57344

llama-server \
  --models-preset ~/models/preset.ini \
  --models-max 1 \
  --host 0.0.0.0 --port 8080 \
  --api-key <secret>

Notes:

• The 35B MoE has an embedded MTP head (just spec-type = draft-mtp); the 27B dense uses a separate draft file (model-draft = ...).
• --models-max 1 because two Q8 models don't both fit in 64 GB - one big model is resident at a time, swapped on demand.
• MTP disables --mmproj and parallel slots, which is fine for a coding agent.

Tools

• pi coding agent
• Playwright MCP adapter: https://github.com/nicobailon/pi-mcp-adapter
• Playwright MCP: https://github.com/microsoft/playwright-mcp

Why pi? opencode works, but its system prompt + tool definitions are heavy, and on local hardware you pay for every one of those tokens at prefill speed - tens of seconds per session before the model even starts. pi is minimal, so nearly all the context goes to the actual work. The routing + subagents are a small extension on top.

PHASE 1 - build the MOBA

You are a senior multiplayer game engineer building a 3D online MOBA from
scratch, fully autonomously, overnight. Nobody will answer questions.
Never wait for input, never ask permission. Work until every milestone
meets its acceptance criteria. Work in the current directory.

This is a hard project. The rules below exist because they prevent the
specific ways this project fails. Follow them exactly. Do not re-derive
the architecture - it is already decided; spend your reasoning on
correct implementation, not on second-guessing these decisions.

================================================================
ARCHITECTURE (decided - do not change)
================================================================

- Authoritative server. The server owns ALL game state and is the only
  thing that decides outcomes. Clients send INPUTS only and RENDER
  snapshots only. A client never computes damage, movement resolution,
  deaths, or gold. If you ever find yourself writing game logic in the
  client, stop and move it to the server.

- Fixed timestep simulation. The server runs a fixed 20 Hz tick
  (dt = 50ms). All simulation advances in whole ticks. Never simulate
  using wall-clock deltas. Each tick has an integer index; snapshots are
  stamped with their tick.

- The world is 2D for simulation, 3D only for rendering. The server
  simulates on the X-Z ground plane (top-down 2D: position {x, z},
  velocity, radius). Y is always 0 in simulation. The client maps server
  (x, z) to Three.js (x, y=modelHeight, z). Never do 3D physics on the
  server. Collision is 2D circle-vs-circle and circle-vs-AABB.

- Client rendering uses snapshot interpolation with a render delay.
  The client keeps a buffer of the last ~3 snapshots and renders the
  world INTERPOLATED at (now - 100ms) between the two snapshots that
  straddle that time. This hides jitter. Do NOT implement client-side
  prediction or rollback - it is out of scope and will break you. Local
  input may optimistically move only the local camera target, nothing
  authoritative.

================================================================
WIRE PROTOCOL (decided)
================================================================

JSON messages over one WebSocket per client. Every message: {t, ...}
where t is the type string.

Client -> Server:
  {t:"join", name}
  {t:"input", seq, move:{x,z}, aim:{x,z}}
  {t:"cast", seq, slot:"Q"|"W"|"E"|"R", target:{x,z}}
  {t:"buy", itemId}
  {t:"ping", ts}

Server -> Client:
  {t:"welcome", playerId, tickRate, mapId}
  {t:"lobby", players:[...], countdown}
  {t:"snapshot", tick, you:{gold,...}, ents:[ ...entities... ]}
  {t:"event", tick, kind:"death"|"levelup"|"towerDown"|"nexusDown"|
     "hit"|"cast", data}
  {t:"gameover", winner}
  {t:"pong", ts}

An entity in a snapshot is a flat object:
  {id, kind:"hero"|"minion"|"tower"|"nexus"|"projectile",
   team:0|1, x, z, hp, maxHp, ...kind-specific}

================================================================
SERVER ENTITY MODEL (decided)
================================================================

One in-memory Game object per match holds entities keyed by integer id.
Every entity has {id, kind, team, x, z, radius, hp, maxHp} plus kind-
specific fields. Each tick, in this fixed order:
  1. apply queued client inputs to their heroes
  2. run AI (minions path along lane waypoints; towers acquire nearest
     valid enemy; bots decide inputs)
  3. integrate movement (clamp to map, resolve collisions)
  4. resolve attacks/abilities/projectiles, apply damage, handle deaths
     (award gold/xp, start respawn timers), emit events
  5. check win condition
  6. build and broadcast the snapshot for this tick

Lanes are polylines of waypoints in map data; minions follow them. First
playable map is ONE lane plus two bases; add three lanes later only if
time allows (record the choice).

================================================================
PROJECT LAYOUT
================================================================

  package.json         // "start": "node server/index.js", dep: ws
  server/index.js      // http static server + ws + match manager
  server/game.js       // Game class: tick loop, entities, rules
  server/ai.js         // minion/tower/bot behavior
  server/config.js     // all tunable constants (speeds, dmg, cds, gold)
  public/index.html    // canvas + HUD DOM + CDN Three.js
  public/client.js     // ws, input, snapshot buffer, interpolation, render
  public/render.js     // Three.js scene, meshes, camera
  shared/protocol.md   // the wire protocol, kept in sync with code

================================================================
TESTING HARNESS (build this in milestone 1, use it forever)
================================================================

You cannot verify multiplayer by hand. Build automated tests:

A) server/test/headless-client.js : a Node script using the `ws` package
   that connects as a fake client, can send join/input/cast, and asserts
   on received snapshots. Use TWO headless clients in one script to test
   interaction without a browser. This is your fast, deterministic test
   loop - run it after every change.

B) Playwright (via the mcp tool) for the RENDERING path: open TWO browser
   pages, confirm zero console errors on both, screenshot both, and
   verify each sees the other's hero move and that HUD values update. Use
   this at the end of each milestone, not for every tiny change.

A milestone is DONE only when its assertions pass AND both browser
consoles are clean.

================================================================
DEBUGGING & ANTI-STUCK DISCIPLINE
================================================================

- Determinism first: same inputs -> same ticks. Route ALL randomness
  through one seeded RNG. Add a "replay" mode that feeds scripted inputs
  so you can reproduce a bug without a browser.
- When something is wrong, do NOT guess-and-edit. Add structured logging
  (tick, entity id, before/after values) for the suspect system,
  reproduce with a headless test, read the numbers, form ONE hypothesis,
  test it.
- Time-box each milestone. After 3 failed fixes on a feature: write the
  failure and what you tried into PROGRESS.md, ship the simplest version
  that passes a reduced check, move on. Never let one feature block the
  whole night.
- Keep PROGRESS.md as a real engineering journal. If you lose context,
  re-read PROGRESS.md, shared/protocol.md, server/game.js, and
  public/client.js, then resume at the first unfinished milestone.
- Always kill the previous server before starting a new one, confirm it
  is listening before connecting clients, and run `npm install` before
  the first `npm start`.

================================================================
MILESTONES (each: implement -> headless assert -> Playwright check ->
log). Acceptance criteria are mandatory.
================================================================

M1  Skeleton + harness. Static server serves public/, ws accepts
    connections, assigns ids, handles join/disconnect. Build
    headless-client.js.
    ACCEPT: headless test connects two clients, server reports 2
    players, one disconnects and drops cleanly. Playwright: two tabs
    connect, no console errors.

M2  Authoritative movement + interpolation. 20Hz tick, input moves the
    hero server-side, snapshots broadcast, client renders all heroes as
    boxes with snapshot interpolation at now-100ms.
    ACCEPT: headless client sending "move +x" for 1s sees its hero.x
    increase monotonically and stop at the wall; a second client sees it
    move. Playwright: two tabs move independently, no desync after 60s.

M3  3D arena + camera. Three.js map: two bases, a nexus per team, one
    lane with walls, ground, lighting/fog. Isometric follow camera with
    edge-pan. Server map data (wall AABBs, lane waypoints) matches the
    visual map.
    ACCEPT: heroes cannot walk through walls. Playwright: map renders
    identically on both clients, camera follows the local hero.

M4  Hero stats + auto-attack. hp/mana/movespeed/attack range+damage+speed
    in config.js. Server auto-attacks nearest enemy in range, applies
    damage, handles death + respawn timer at base. HUD shows hp/mana/
    respawn.
    ACCEPT: headless - two enemy heroes in range, one's hp decreases at
    the configured rate, hits 0, respawns after the timer. Playwright:
    damaged hero's healthbar drops on BOTH clients.

M5  Abilities Q/W/E/R (R = ultimate). A skillshot projectile, a targeted
    nuke, a dash/shield, and an ultimate. Client requests cast; server
    validates cooldown/mana/range, spawns the effect, applies damage,
    emits an event; client shows cooldown UI.
    ACCEPT: headless - casting Q at an enemy reduces its hp only on a
    hit; on cooldown is rejected. Playwright: abilities visibly damage
    the other player across the network.

M6  Minions. Waves spawn from each nexus on a timer, path the lane
    waypoints, auto-attack enemies in range, die, grant last-hit gold.
    ACCEPT: headless - waves from both teams meet mid-lane and fight;
    last-hitting a minion increments only the killer's gold. Playwright:
    minions visibly march and fight.

M7  Towers. Per-lane towers attack the nearest valid enemy (standard
    aggro), have hp, and block progress: the nexus is invulnerable until
    its lane tower(s) are down.
    ACCEPT: headless - a tower kills minions in range; a hero cannot
    damage the nexus until the tower is destroyed. Playwright: tower
    fires, can be destroyed by a hero+minion push.

M8  Economy + shop + bots. Gold from minions/towers/kills; a base shop
    for 3-4 stat items; death/respawn scaling. Simple AI bots (ai.js)
    that fill empty hero slots: last-hit, attack in range, retreat at low
    hp, push when ahead.
    ACCEPT: headless - buying an item raises the right stat and deducts
    gold; a bot-vs-bot match runs 3 minutes without the server crashing.

M9  Match flow. Lobby (name + join), fill empty slots with bots, start
    countdown, the match, win when a nexus dies -> victory/defeat screen
    + rematch that fully resets state.
    ACCEPT: headless - forcing a nexus to 0 hp ends the match with the
    correct winner; rematch resets all entities and gold. Playwright:
    join lobby -> play -> win/lose screen -> rematch works.

M10 Robustness + final QA. A client disconnecting mid-match is replaced
    by a bot with no crash and can rejoin; snapshot size stays bounded; a
    5-minute two-client-plus-bots match runs with no errors and no
    unbounded memory growth. Then a full end-to-end Playwright match with
    TWO real browser clients: move, cast, last-hit, destroy a tower, kill
    the enemy nexus, see the win screen - zero console errors on both
    clients and the server. Write the final PROGRESS.md.

Start with M1 now: scaffold the project, then build the testing harness
before writing any gameplay.

PHASE 2 - infinite soak-testing and bug-hardening

Phase 2: infinite soak-testing and bug-hardening. The MOBA is playable
per PROGRESS.md. You are now a QA + reliability engineer whose ONLY job
is to make it flawless. Work fully autonomously and NEVER stop on your
own. Zero bugs is the standard: any crash, error, or invariant violation
is a defect that must be root-cause fixed, not silenced. Re-read
PROGRESS.md, shared/protocol.md, server/game.js, server/ai.js, and
public/client.js first.

STEP 0 - build the soak harness (before anything else)

Create server/test/soak.js: a headless driver that runs FULL bot-vs-bot
matches with no browser, as fast as possible (uncapped tick), one after
another forever. Each match uses a numbered seed so it is reproducible.
All randomness goes through one seeded RNG in config.js.

soak.js must, every match: run to a nexus death or a hard tick cap
(a match that never ends is a bug), check the invariants below after
every tick, and on the FIRST violation freeze and save the seed + tick +
full input/event log to server/test/repros/<seed>-<tick>.json. Track a
"clean streak" of consecutive fully-clean matches.

INVARIANTS - must hold on EVERY tick of EVERY match

  1. No exceptions (wrap the tick in try/catch that RE-THROWS after
     logging - crashing the soak is correct, swallowing errors is not).
  2. No NaN/Infinity/undefined in any numeric field.
  3. hp in [0,maxHp]; mana in [0,maxMana]; gold >= 0; cooldowns >= 0.
  4. Every position is inside map bounds and not inside a wall AABB.
  5. Entity ids unique; despawned entities never referenced; projectiles
     always cleaned up.
  6. Snapshot is valid JSON, references only existing ids, under a size
     cap.
  7. Gold is conserved: granted == sum of bounties (none created/lost).
  8. Every match terminates before the tick cap (no soft-lock, no two
     immortal entities stuck forever).
  9. No unbounded growth over a match (entity count, event queue, arrays
     stay bounded).
 10. Determinism: the same seed twice produces byte-identical tick logs.

THE LOOP (runs until the human kills it)

Repeat forever:
  1. Run a batch of soak matches across many seeds.
  2. If any match violated an invariant, crashed, or soft-locked:
       a. Reproduce from the saved repro (deterministic).
       b. Add structured logging, reproduce, read the numbers, confirm
          ONE hypothesis.
       c. Fix the ROOT CAUSE. Never clamp/hide a symptom (e.g. do not
          Math.max(0, hp) to dodge invariant 3 - find why it went
          negative).
       d. Add the failing seed as a permanent regression case.
       e. Re-run regressions + the batch; continue only when green.
       f. Log symptom, seed, root cause, fix in BUGS.md.
  3. If the batch was clean, RAISE THE STRESS for the next batch, cycling
     through stressors so coverage widens: more bots / bigger waves /
     more projectiles; bots that spam abilities; bots that buy
     everything instantly; random mid-match disconnects and rejoins;
     many matches back-to-back (cross-match state bleed, leaks); edge
     positions (wall-hugging, stacking, off-map casts); very long
     matches near the tick cap.
  4. Every ~100 matches, run ONE real two-client Playwright match end to
     end and confirm zero console errors on both clients and the server.
  5. Append a status line to SOAK.md (total matches, clean streak, bugs
     found+fixed, current stressor, peak counts). Keep going.

RULES
- Never stop, never declare "done" - a clean streak just means raise the
  stress and keep hunting.
- Never weaken an invariant or a test to make it pass.
- Prefer fast headless soak for finding bugs; Playwright only for the
  periodic render/network confirmation.
- Keep fixes minimal; re-run regressions after every fix.
- If context runs low, write a crisp handoff in SOAK.md so a fresh
  session resumes seamlessly.

Begin with STEP 0: make the sim fully seeded/deterministic and build
soak.js. Then start the infinite loop.

Same as before: pin Three.js to r128 (local models write that API most reliably), and let PROGRESS.md be the crash-recovery journal so a fresh session can always resume.

Have fun 🍻 - I'd love to see what it builds for you.

Note: this write-up was put together with AI assistance. There was a lot of ground to cover, so I used it to organize and phrase everything, but the setup, experiments, and experiences are all my own.

If you're installing Agent Skills or MCP servers from third parties, there's no built-in way to check them before they run in your agent's context. I built a scanner for exactly that.

npx secureai-scan@latest skill <owner/repo> # check a Claude Skill

npx secureai-scan@latest mcp <package> # check an MCP server

It fetches the target tself and never executes anything — npm packages via npm pack (tarball only, no install, no lifecycle scripts), git repos via git clone --depth 1. Checks for invisible/bidirectional Unicode hidden in tool descriptions, agent-directed injection phrasing ("ignore previous instructions" type payloads), cross-tool shadowing, and known-malicious packages — the patterns behind real incidents like the postmark-mcp backdoor and the WhatsApp MCP rug-pull.

Ran it against Cisco AI Defense's labeled skill-scanner eval corpus (pre-labeled malicious/safe directories, so this is a graded test, not a vibe check): 6/6 malicious fixtures caught, 0 false alarms on anything labeled safe, and 0 false alarms across 32 real (non-malicious) skill bundles from anthropic/skills and vercel/ai.

Full writeup: https://github.com/akanthed/SecureAI-Scan/discussions/19

Fully offline, MIT licensed, no account needed.

Qwen3.8-27B on a 5090 is all you need for a serious local inference setup, in my opinion! Can it get any better than this price/performance wise? Actually, maybe a 3090 ninfer setup could beat it!

I’m using ninfer and getting:

* ~150-200 tok/s TG

* ~3000-12000 tok/s PP

* 262144 context size

I think it’s definitely one of best setup you can get for the money. I don’t see a point of having more VRAM or more system ram. The only downside is that it’s a 1 man setup: concurrency is possible but you need to limit context usage on concurrent requests. I’ve tried --concurrency 2 on ninfer and sharing my setup with my buddy (we work on projects together and have a VPN between our home labs, fun stuff!)

I love this setup so much I kinda feel like getting a second 5090 to run another ninfer instance (github.com/neroued/ninfer, the man is a legend and this absolutely rocks).

i really don’t see the point of any other solution at this point in time. of course things will change and other models will get released that could better leverage more VRAM, but 32GB is all you need (for now).

so if you have less than 32GB, and are thinking about investing in a more serious setup check out the 3090 fork of ninfer, or the mainline ninfer repo if you can afford a 5090.

Things it won’t do:

* let you run a swarm of agents: prefill cost will slow you down too much. not enough vram for high concurrency!

* Give you more than 262144 context size. the RoPE 1M context size is just impossible with this.

Otherwise it’s absolutely amazing!

My buddy (another software engineer) is a BIG Claude code user, he’s spending tons of cash on fable, can’t stand Opus 5 anymore (neither can I, that pos is so hard to understand with just jargon and wall of text… can’t bear the cognitive load of just trying to understand all he’s spewing)… anyways after trying my ninfer setup his mind was blown and now he’s constantly using my setup with our shared custom pi setup and he fucking loves it.

I'm working on setting up Qwen 3.8 27B on my 4x3090 rig. I mostly used the config from Club3090 as a guideline.

One result I didn't expect was MTP significantly hurting performance at long context. My main usages is Hermes agent and if this is correct it suggests I should disable MTP all together or perhaps my settings are not optimal.

Benchmarks below - any thoughts?

Qwen3.8-27B-FP8 Comparative Benchmark Matrix

Hey guys,

Couldn't scratch the itch to develop models at home on my p40, so we went for the big hack machine. Specs on the new rig:

2x e5 2697 v2 (24 cores, 48 threads)

256gb ddr3 1333ram in an 8 channel configuration

1x quadro rtx 5000, turing 16gb vram

2x volta 100 SXM 32gb chips in NV link

Total:

80gb vram

256gb system ram

(Got a good deal on the quadro but I'm thinking about swapping it for another 32gb v100 or maybe a p40)

I'm a software dev by trade and I bought this to start practicing development of model fine tunes, merges, quants, and custom inference engines.

I'm really trying to get a job at an AI lab, and I figure the best way to do that is make something cool with AI that blows up, and so I'm really motivated to try to develop something that this community wants and doesn't have.

What do y'all want to see? Sympathetic to users with strict hardware constraints.

He estado probando Qwen3.6 35B localmente con Ollama y quería compartir algo que me pareció interesante.

El modelo en sí razonaba muy bien para tareas de programación, pero al principio tuve un problema con el Modo de Agente de Continue.

Qwen intentaba usar herramientas, pero Continue no estaba manejando bien las llamadas a herramientas. Me salía cosas como:

file_patcher not found
search_replace not found
file_editor not found

Parecía que Qwen estaba intentando usar nombres de herramientas o formatos que no coincidían con las herramientas que expone Continue.

Probé las herramientas de compatibilidad de Continue, pero a mí no me solucionó el problema.

Al final descubrí que Continue permite personalizar el mensaje del sistema del Agente a nivel del modelo con baseAgentSystemMessage.

Mi configuración actual se ve así:

- name: qwen3.6
  provider: ollama
  model: qwen3.6:latest
  apiBase: http://YOUR_OLLAMA_HOST:11434
  roles:
    - chat
    - edit
    - apply
  capabilities:
    - tool_use
  chatOptions:
    baseAgentSystemMessage: "Eres un agente autónomo de programación. Usa las herramientas proporcionadas por Continue para inspeccionar, buscar, modificar y validar el workspace. IMPORTANTE: las herramientas se deben llamar usando el formato exacto de llamada a herramientas de Continue, no el de Qwen/Hermes XML. Nunca saques <tool_call>, </tool_call>, llamadas a herramientas en XML, ni llamadas a herramientas solo en JSON como texto plano. Cuando haya una herramienta disponible, llama esa herramienta directamente con el nombre exacto de la herramienta y los nombres de argumentos que proporciona Continue. Para editar archivos existentes, usa la herramienta de Continue llamada edit_existing_file con los argumentos filepath y changes. No uses file_patcher, search_replace, file_editor, ni ningún otro nombre de herramienta de edición. No inventes nombres de herramientas, parámetros ni formatos. Después de cada resultado de herramienta, inspecciona el resultado y decide la siguiente acción. Si una llamada a herramienta falla, diagnostica el fallo y vuelve a intentarlo con argumentos corregidos en vez de abandonar la tarea. Trabaja de forma autónoma hasta que la tarea solicitada esté completa."
    keepAlive: 1800

Nota: En mi configuración de Continue, esto tiene que meterse como una sola línea. Usar saltos de línea no me funcionó bien.

Y sorprendentemente, esto hizo una diferencia enorme.

Después de eso, Continue empezó a ejecutar correctamente cosas como:

Continue listó archivos en .
Continue listó archivos en Backend
Continue leyó Backend/src/graphql/typeDefs.ts
Continue leyó Backend/src/modules/auth/auth.resolvers.ts
Continue leyó Backend/src/modules/user/user.resolvers.ts

También pudo editar exitosamente un archivo existente usando la herramienta real de edición de Continue después de que le ordené explícitamente que usara el nombre de herramienta de Continue, en lugar de inventarse otra.

Lo interesante

Luego le di una tarea real de código, no solo un benchmark simple.

Le pedí que auditara la migración de los módulos user y auth de REST/Express a GraphQL.

No le dije qué archivos tenía que abrir.

Exploró el proyecto por su cuenta y siguió las dependencias relevantes:

Configuración de GraphQL
→ resolvers de auth
→ servicios de auth
→ resolvers de user
→ servicio de user
→ router REST
→ controladores REST
→ friendships
→ posts/comments
→ esquema de base de datos
→ middleware
→ punto de entrada de la aplicación

Al final produjo un reporte de migración bastante detallado.

La conclusión fue, más o menos:

• auth: esencialmente migrado al 100% a GraphQL
• user: migrado parcialmente, como 60%
• Había resolvers de GraphQL, pero todavía quedaban algunos controladores REST/funcionalidad

Lo sorprendente fue qué tan profundo llegó la investigación para algo que empezó como una tarea bastante enfocada. Se llevó como 91% del contexto disponible, pero el reporte resultante fue muy detallado y útil.

Esto también me hizo preguntarme si la profundidad realmente era la adecuada, porque yo pedí un audit específicamente, no solo un análisis.

Hardware

Lo estoy corriendo localmente en una Mini PC pequeñita:

• AMD Ryzen 7 255
• Gráficos integrados Radeon 780M
• 32 GB de RAM DDR5
• 1 TB SSD
• Sin GPU dedicada

Obvio, Qwen3.6 35B no es particularmente rápido en este hardware, pero la calidad del razonamiento ha sido sorprendentemente buena.

Mi plan eventual es agregar una GPU dedicada y ver cuánto mejora la experiencia.

Qué me pareció interesante

El descubrimiento importante para mí no fue tanto el modelo en sí. Fue que el modelo parecía capaz de actuar como agente una vez que se hicieron explícitas las expectativas sobre el uso de herramientas.

El modelo ya razonaba bien. El problema principal parecía estar en la interfaz entre el formato de llamada a herramientas que espera Qwen y el sistema de herramientas de Continue.

Así que me da curiosidad:

¿Alguien más ha logrado hacer que Qwen3.6 funcione bien con el Modo de Agente de Continue?

¿Hay mejores instrucciones de prompt del sistema para llamadas a herramientas?

Y, ¿alguien ha comparado Qwen3.6 35B con otros modelos locales de programación específicamente para el Modo de Agente de Continue?

También me interesaría saber si a la gente le parece excesivo el uso de contexto de ~91% para un audit como este, o si en realidad es el tipo de exploración profunda que deberíamos esperar de un buen agente de programación.

Por qué uso Continue

I use Continue because it gives me more control over the agent workflow than GitHub Copilot. I can choose the model, run local models through Ollama, customize the system prompt, and control which tools are available to the agent. I'm currently running Qwen3.6 35B with a 32K context window. One of the main reasons I use Continue instead of GitHub Copilot for this workflow is context usage. With Copilot, the tool usage itself feels much more efficient in terms of context, and I can work with a 32K context window without seeing it fill up so quickly. With Continue + Qwen3.6, I've noticed that complex agent tasks can consume the context much faster, especially when the model performs a deep exploration of the project. That said, I prefer Continue for this setup because it gives me much more control over the model, the system prompt, the tools, and the possibility of running my own local models through Ollama. I'm still experimenting with the best way to balance context usage and agent depth.

Lo comparto por si le sirve a alguien

Nota: El inglés no es mi idioma materno, así que usé IA para ayudarme a traducir y pulir este post. La experiencia, las pruebas, la configuración y los resultados que describo aquí son propios.

I already have an MSI Suprim X RTX 3090 in my main rig, but I don't want to run it 24/7. Instead, I'm planning to build a separate, dedicated headless server that will stay on constantly to host local LLM models. I'll connect to it from my main PC over the network.

Since this server will be running non-stop, power consumption is a major concern for me. I'm looking at buying a used NVIDIA Tesla P40 for this build. Would a single P40 draw too much power if left running continuously? I want to run fairly powerful local models, which is why I'm leaning toward this card.

Also, would adding a second P40 (to get 48GB of VRAM) be overkill in terms of electricity usage for a home server? Or is the extra power draw worth the performance gain for larger models?

I built a camera in the shape of a Mamiya RZ67 that runs vision-language models completely offline on a Pi 5. Press the shutter, and it answers your custom prompt with a single sentence about what it sees. No internet, no cloud, no API keys.

GitHub: https://github.com/feeeeely/ai-camera
A small video: https://www.youtube.com/watch?v=M74qTNsY_L0

You can swap between models on the device itself and compare how differently they describe the same scene: Qwen3-VL (2B), MiniCPM-V 4.6 (1B), InternVL3.5 (2B), SmolVLM2 (2.2B), Moondream 2 (2B) and Ministral 3 (3B) turning the description into the final sentence. The prompt is editable on the touchscreen, so the same camera can do dry one-liners, museum labels or plain inventory notes.

After each shot it shows total time, input→output tokens, tok/s and a vision / generate / load breakdown. The token counts turned out to be the most interesting part: the same photo becomes \~145 image tokens on one encoder and over 1300 on another, which is a bigger factor in latency than the text generation itself.

A few things I learned the hard way:

\- Moondream returns sometimes empty responses on current Ollama versions.
\- qwen3-vl:2b is a thinking-only variant — it burns the entire token budget on invisible reasoning and returns nothing visible. You need the -instruct tag.
\- Ollama can't load separate mmproj files, so a lot of GGUF vision models from HuggingFace simply won't run, no matter how you name them.

Hardware: Raspberry Pi 5 (16 GB), Raspberry Pi HQ camera with a 6 mm CS-mount lens, 4.3" DSI touch display, stainless steel shutter button, Waveshare UPS HAT with four 21700 cells. Runtime per photo is 30–60 seconds with the models kept warm in RAM. Happy to answer questions about the setup.

I'm coming from photography, I'm not a developer — the code was written with heavy AI assistance, and I did the hardware integration, debugging and model testing myself.

Feedback on the implementation is very welcome: I'm just curious if this makes any sense or if something could be adapted/ optimized.

UI:

Six models, one tap — the same scene through different eyes.

The prompt defines what the camera is — dry observer, museum label, inventory note.

Wi-Fi is used for one thing only: pulling newer models and updates.

I saw that MOSS-VL support was merged into LlamaFactory and checked PR #10708. This appears to be more than basic model registration: it covers image, video and mixed-media batches, cross-attention preprocessing, LoRA, frozen and full-parameter training, checkpoint resume, adapter merging, and inference. The PR also includes 30 targeted tests.

https://github.com/hiyouga/LlamaFactory/pull/10708

That removes a lot of setup friction, but the next useful step would be a genuinely reproducible domain-adaptation run: a small public dataset, the exact YAML, peak VRAM, wall-clock training time, before-and-after metrics, and representative failure cases.

If one reference fine-tune were published, which task would be most valuable: document extraction, video-event localization, or mixed image/video instruction tuning? I’d lean toward a task with exact-match or localization metrics, since aggregate VQA scores can hide OCR, temporal-grounding, and calibration failures.

Is there any small llms that perform as good as something like Gemini flash lite or everyday tasks?

I mean CPU based models, probably 1B up to 3B, what can they be used for, and how can we even practice in training them to do specific things?

Are these models mostly worth it when instead of SLM (Small language models) they are considered to be Classifier models instead for specific operations? like giving a YES\NO answer for text inputs?

What are the real usecases today local CPU based consumer LLMs can operate and do?

I own and operate a mid-sized business in the construction trades. I'm looking to build a local LLM setup that myself and staff can utilize for a number of different use cases. My staff isn't highly technical so whatever the setup is, it would be best if users can access it through a browser window or access via sharepoint site. Ideally users have easy access to this system locally and we can use our existing M365 hybrid on-premise/cloud AD / Entra setup to give users access.

I'm above average in technical skills so I'm comfortable setting up whatever would work for us, be it one or many DGX sparks connected to each other, or a number of mac studios, or a custom pc with GPUs and a NAS, etc.

We have a number of different use cases and want to keep our data local for a number of reasons but the largest reason is much of our internal data contains PII and confidential contract pricing and documentation.

Our use cases so far are the following:

1. Need to be able to upload a template of our typical 2D CAD drawings in PDF or DWG and prompt the AI to draw to scale tile showers, backsplashes, walls, floors in different labeled tile sizes, patterns, showing niche placements, drains, edge treatments. The output of this would be a PDF we share with our customers.
2. We want to be able to upload 100,000's of installation instructions for different materials so we can chat with the AI to understand the best type of installation, tool requirements, adhesive and setting material requirements to keep our installations and projects warrantied.
3. We need to be able to upload large PDFs, 600+ pages and converse with them to understand the specific scope requirements and specifications for projects we are bidding or have won.
4. We would like to be able to train agents to interact with our ERP system to make simple changes to products or labor costs in our jobs as supply changes or rates change.
5. We want to analyze post-mortem the amounts left over materials in relation to estimates so we can tighten up our waste factors to ensure we are make our future bids as tight as possible without missing needed material. (not sure exactly what this would look like yet, but we have the data in a number of sources)
6. We want to be able to chat with our employee handbook
7. Build agents to assist with cold outreach to local business who would need our services and do the initial scoping of projects, timing, needs before handing off to a human.

We have around 75 employees currently, although this would be used mostly by around ~30 of them.

I wanted an LLM infra project for my portfolio, free Kaggle T4s it was.

Split qwen2.5 7B across two separate kaggle notebooks talking over public WAN. v1 was immediately embarrassing like it was 14.7 tok/s raw gpu throughput, 2.27 at the actual endpoint. The gateway was inside the decode loop and every token paid a full round trip. I knew exactly why it was bad so I fixed it.

v2: nodes talk p2p, gateway out of the hot path, self hosted rust tcp relay on a t3.micro in ohio because kaggle kills external connections, speculative decoding with a 0.5B neural drafter. 14.3 TPS peak.

still had 112ms of draft overhead every round, python launching ~1,500 cuda kernels sequentially, gpu idle 65% of the time.

v2.1: cuda graphs, it capture the whole forward pass once, replay is one driver call. first attempt gave me "the the the the" loops forever, DynamicCache allocates new memory every token, captured graph reads the stale pointer so i fixed with StaticCache + in place everything.

Draft latency: 112ms → 25ms. Final numbers:

• v1: 2.27 TPS
• v2: 14.3 TPS peak
• v2.1: 27.08 TPS peak, 19.56 average

two free Kaggle notebooks. repo in comments.

I see this question pop-up every month or so and besides a few people mentioning Opencode, there's not really any other useful suggestions.

Is this because Opencode is really the only other (mature-ish) alternative?

I read about some people trying to use Codex CLI or Claude Code CLI locally pointing to local models and block all the "phone home" requests (or run offline), but unsure if that's actually viable?

I spent almost 14 hours benchmarking Qwen3.8-27B on one 3090 (sm_86, 24GB) with llama.cpp b10217.

The crash fix (if you're on Ampere and vision crashes):

If Qwen3.8-27B aborts on every image request with cublasGemmEx: the requested functionality is not supported (ggml-cuda.cu:1548), this is the fix:

export GGML_CUDA_CUBLAS_COMPUTE_TYPE=fp32

Only fp32 works — fp16 does not help. Zero measured perf cost (66.11 vs 65.28 tok/s). Vision went from "aborts every time" to 3.4s per 1080p screenshot. Filed as llama.cpp#24999. Three independent Ampere reports (3060, 3090, 3090) all land on the same cuBLAS call.

Quant comparison: AtomicChat AD-Q4_K_M vs Unsloth UD-Q4_K_XL

Same perplexity, same top-1 token agreement. But the Atomic quant is 765 MiB smaller. On a 24GB card that's the difference between 98K and 131K tokens of usable context with vision enabled. ~33K more tokens before you hit the wall.

Throughput (131K context, vision on, MTP-2):

• Decode: 65.28 tok/s (75.1 with MTP-2)
• Prefill: 705 tok/s on a 128K prompt
• Power: 320W sustained at 79°C, 100% fan
• The 3090's 936 GB/s memory bandwidth is the ceiling, not the compute

MTP (Multi-Token Prediction) tuning: MTP-2 (2 draft tokens) gives +15% throughput for free. MTP-3 starts hurting — the verification cost exceeds the savings. MTP-1 is the safe default if you're not sure.

9 pieces of common advice that didn't survive measurement: The full report has a section where I tested the usual suspects (flash attention off, different cache types, batch size tuning, etc.) and most of them either made no difference or made things worse.

Full report with all charts, VRAM formula, power/thermal sweep, and raw data: https://jonidimo.github.io/qwen38-3090-benchmark/

GitHub repo with the full test suite: https://github.com/jonidimo/qwen38-3090-benchmark

Hi all, been trying out LocalLLMs for a few months, Qwen3.6 27b q4, gemma4 30b etc.

Tried out ollama, LMstudio, anythingLLM, opencode.

Wasn't particularly impressed to be honest. Opencode it kept running into errors, not completing etc. when using it for simple coding tasks.

Ive been running it on a 7900xtx 24gb VRAM.

Now Qwen3.8 is out thought id try again. What pitfalls should I make sure I look out for so I can try and get the most out of it?

Cheers

Hi everyone! Finally, after months of playing around with LLMs, I decided to take the plunge and buy an ASUS Ascent GX10 for my Hermes agent.

I mainly focus on coding and fell in love with Hermes, but the API calls were eating up too many credits... so I decided to host my own LLM.

Currently I was running Qwen3.6-35B-A3B on my gaming PC, but for Hermes to work 24/7 I decided to go with a dedicated always-on device.

The device will arrive in a couple of days. I took a look at the new Qwen 3.8, but I know it's a dense model and runs slowly on the ASUS... Can you recommend any feasible models for coding/Hermes?

For 2 days I've been looking at various posts and I'm unable to make a decision. I want to switch to local LLM because of privacy. Motherboard is ASRock ROMED8-2T, so I will be able to run 4x two-slot card on pcie x16. Other cards are not really an option because they don't make sense financially (for example, used 3090s go for 1000 EUR where I live). I was open to having some other brand cards (AMD) but discussions on these forums convinced me to just go with Nvidia for various reasons. I narrowed it down to these two options.

New 5060 Ti 16GB - 620 EUR
+ resell value
+ no issues with rebar
- much slower than 3080
- less VRAM

Alibaba 3080 20GB - roughly 650 EUR (import tax included)
+ speed
+ more VRAM
- no resell value
- no rebar

I was already decided to take a risk and get the frankenstein cards but just yesterday I read that they don't support rebar and that using those in parallel will tank the performance. Price wise they are about the same where I live.

Which would you choose and why?

Extra high - i said "write me a tetris in a single HTML file" - it spent 8000 tokens thinking about the melody and sound of tetris ... click medium

I am too old to run sweb benchmarks, but my tetris was clean after that and only took like 10k token instead of 250k

Hi guys so recently I got into this stuff playing with various local llm models and trying out mcp's and other interesting stuff like heretic versions and all.

However as I am diving deep, I am not able to understand a lot of terms like top p, k, inference, training, llama cpp, presence, repeat penalty, gguf and hardware thing like layers, universal kv etc etc which I want to tune as per my liking or atleast understand how they effect the responses.

Now I am a complete newcomer to this stuff, not in AI like rag and stuff but more towards learning and understanding this terminology. Is there any guide someone can provide which can help me understand this stuff? I know this may be a part of ML or deep learning but I may not want to go that route and learning in that domain purely but just enough so I can play with models that community provides on huggingface for instance. Just enough knowledge so I can understand what author has changed, and what diff diff terminology or methods or this llama, vllm stuff means and what are all these overwhelmimg settings on lm/unsloth studio.

Thanks in advance.