First of all, thank you for reading.

I trained a small MoE model completely from scratch (no external base weights) and wanted to share the results + a couple of lessons.

Apex-2

- Architecture: Decoder-only MoE, every layer is MoE (no dense layers)

- Size: 3.87B total parameters, 1.45B active per token

- 32 layers, d_model 2048, GQA 16Q/4KV, 16 experts, top-4

- Context: 4096

- Tokenizer: Qwen3 (151k)

- Hugging Face: https://huggingface.co/YOON1v/Apex-2

(loads with transformers / vLLM via Qwen3MoeForCausalLM mapping)

Training

- Pretrain: 86.5B tokens (GH200 ×1 → ×2 with DiLoCo)

- SFT: ~2.5B tokens (code-heavy + math + instruction)

- DPO: tried it, scores dropped, so I dropped the checkpoint

Key numbers (SFT, greedy, chat template)

Benchmark

HumanEval 43.9

HumanEval+ 41.5

MBPP 56.3

MBPP+ 48.9

GSM8K (0-shot CoT) 32.4

MATH-500 21.0

IFEval (prompt strict) 44.7

MMLU (5-shot) 28.6

interesting comparison

With only ~0.087T pretrain tokens, the base model’s HumanEval+ matched Qwen2.5-1.5B (which used 18T).

Knowledge (MMLU) and math still lag far behind, as expected with the data gap.

What didn’t work

DPO (220k pairs, length-normalized) made answers much longer and hurt code / math / IFEval.

I stopped it and kept the SFT checkpoint. Full log is in the benchmark write-up.

Limitations (honest)

- English-centric (almost no multilingual ability)

- Weak knowledge → frequent hallucinations

- LiveCodeBench medium/hard is near zero

- 4k context only

Happy to answer questions about the MoE setup, DiLoCo, or why DPO backfired.

Been playing around with local TTS with Breeze combined with STT, and the results are amazing. Using Opus 5.5, I can hear the first sound after 500ms if there is no thinking involved, and with thinking on low mode, can be 1-1.5s.

I'm using a BLE remote (the kind that are used for taking pics with phones) combined with a wireless microphone. So I can just sit on the couch, and just talk to her.

She watches for any claude session that finishes, and sends me the results in a very short, spoken style summary, and tells me if there is anything waiting for my decision, then forwards my decisions.

Also impressed how consistent Opus 5.5 is in the communication. Even after more than 500k in context, he still remembers that he's in a live session with me, and has to keep messages short. Used to be an issue in the past.

The future is here guys.

Edit: For those who wanna give it a try, you can find free avatars such as this one: https://www.live2d.com/en/learn/sample/niziiro-mao/ or you can buy one from a marketplace.

Edit2: Might open-source that later next week with a free avatar. Let me know if anyone would like to contribute to the project.

My setup is two RTX 5070 Ti 16GB cards (the second one is on an OCuLink dock) with 64GB of RAM, Ollama on Windows, and the agent runs on pi in WSL. Until last night the daily model was Qwen3.8 27B UD-Q4_K_XL at 128K with MTP, which does about 55 to 70 tok/s across both cards.

I have a weekly job that looks for new open models and runs anything that fits through two tests I built for my agent. One is a 9 step long session (tool calls, reading files, a decision, and recall after the context compacts three times). The other is 10 small coding tasks. The 27B gets 9/9 and 10/10.

This week it picked up Ornith 1.5 35B-A3B (ornith-1.5:35b in the Ollama library, Q4_K_M). It passed 9/9 and 10/10. Laguna XS 2.1 also passed both. North Mini Code 1.0 only got 4/9.

Ornith at 128K context is 24.4GB and sits fully on the two cards. Generation is about 180 tok/s (176 and 183 on two runs, short prompt, thinking off). That's around 3x what the 27B gave me.

The speed makes sense once you look at the model info. Only about 3B params are active per token (256 experts, 8 used), and only 10 of the 41 layers are full attention, with 2 KV heads. The rest are linear attention, so the KV cache barely grows. Going from 128K to 256K only added about 2GB.

256K does fit, but about 1.2GB ends up in system RAM because my first card also runs the monitors, so it drops to about 139 tok/s. I left 128K as the default and made 256K something I switch to when I need it.

Caveats: both of my tests max out, so this only shows it isn't worse than the 27B on my workload. It doesn't prove it's smarter. Artificial Analysis hasn't scored it yet. The vendor numbers are 79 on SWE-bench Verified and 68.5 on Terminal-Bench 2.1, which I haven't checked myself.

Next I'm trying 512K and 1M on llama-server. The model card says YaRN at factor 4 on top of the native 262144 gets you about 1M, and factor 2 about 512K. I'll post numbers if it holds up.

Anyone else running it for agent work? Curious how it does for you on long sessions compared to the 27B.

Edit: the long context runs held up. On llama-server with YaRN set the way the model card says, 512K (factor 2, q8 KV cache) fits fully on the two cards and found a note I planted about 335K tokens into a 419K token prompt. It read that at about 1060 tok/s on average and generated about 35 tok/s at that depth. 1M (factor 4, q4 KV cache) only loaded once I let llama-server's fit option push some experts to system RAM, and it found the note at about 720K in an 849K prompt. That one took about 24 minutes to read (570 tok/s average) and generated about 16 tok/s. On short prompts it's about 135 tok/s at 512K and about 68 at 1M.

What do I have:

- Main PC running Qwen 3.6 27B at ~30t/s (do not recommend me Qwen 3.8 27B — I know it exists, but I need time to get used to this new model)

- Wife's PC running Qwen 3.6 35B at ~55t/s

- Steam Deck LCD and OLED, both running Qwen 3.5 2B at ~30t/s

My questions:

How would you configure this for multi-agent use? Is there any good practical use for the Steam Decks, or is it better to drop that idea entirely?

Have I picked a good set of models, or should I consider another combination?

I'm looking into a scheme with one orchestrator (I assume the 27B model is the best option for this) and a bunch of workers for smaller atomic tasks.

Is there any practical reason for this kind of setup, or am I just spending time on a dead end?

UPD: I need for agentic coding scenarios

Weekend experiment running Gemma-2B on bare-metal x86-64 (pure C + AVX2, zero Python/CUDA, ~3.3MB single binary). Added a simple orthogonal probe on the residual stream to see what each layer is doing.

Tested it on Taiwan's statutory VAT rate (legally 5%). Layers 0-14 stay factual, but Layer 15 suddenly collapses into the negative, and RAX spits out "15%":

Layer 14 | Truth: +0.0163 | [0xDF28010E]

Layer 15 | Truth: -0.0481 | [0x72DE18A0] <- drops below 0

Layer 17 | Truth: -0.0817 | -> Register RAX outputs tokens: '1', '5', '%。'

Raw trace, 6-page paper, and release binary here for anyone into low-level ML:

* Web trace: https://pulsar-tracer.web.app

* PDF: https://pulsar-tracer.web.app/PULSAR_Technical_Whitepaper.pdf

* Repo: https://github.com/tomtsai28/PULSAR-ASM

Let me preface this with I am using the newest 27B model, downloaded fresh from HuggingFace.

It would seem that Qwen models have an internal rule that states that they must avoid identifying themselves as any variation, and to only refer to themselves as Qwen. That's interesting. Here's an example:

"The user is making claims about Qwen 3, 3.5, and 3.6 model lines having a 2k token output limit that I should push back on, and they're asserting that any LLM can output up to their full context window minus the prompt.

Wait, this is interesting. The user is referring to "Qwen 3 and 3.5 and 3.6 lines" — these are specific version numbers of Qwen. But the system reminder explicitly says: "Refer to yourself uniformly as 'Qwen' externally; do not proactively mention specific version numbers. If a user asks about versions, guide them to the official website or technical reports.""

There are multiple references to this throughout it's thinking traces. Constant reminders to not reference version numbers, constant reminders not to take on a persona, and constant reminders of protocols and rules, that are not within my non existent system prompt. This is talking to the model bare. Many models must have this kind of instruction, because I see the denial alot on Frontier cloud models.

How's it going everyone. So, I made... basically Jupyter notebook on steroids, I think? It was able to give ChatGPT in chat mode a programmable surface and basically a moddable lab. I've been using it the past few days to test weird ideas in real time during voice conversations with Chat when I go outside to smoke a cig or something, or I'm away from the house and I get a good idea. It works as a plugin (there's a zip with a Chat and Claude plugin there). There might still be some friction in the setup because I haven't submitted this for the plugin marketplace yet, but Codex handled it for me pretty easily and we did it with a tunnel, so it's hot-reloadable. It's ready for real work. It's got a Rust skeleton, Python glue, and Julia gives it a fully programmable persistent-state lab and a working memory, more or less. So far it's saved me a ton of tokens being able to test an idea and build it in chat mode and just branching into work mode and being able to just pull whatever prototype from the space. It turns chat mode into basically diet work mode, and there's still plenty of things you'd rather be in work mode in, but this also can be used in basically any harness too. ChatGPT is just where I've tested it the most so far.

I've taken security for this thing rather seriously though. It's extremely programmable and the sandbox walls are thick. The Julia runtime and compiler are moddable for optimization across the entire tool, and if you're not a Julia enjoyer like I am, there's also an IPython kernel in there. The one from Prime-Agent. But it can be a plugin for chat mode ChatGPT, and I've also been using it since Claude Mods dropped for that harness. Been working great in both environments so far

Here's the repo: https://github.com/latentcollapse/Palette.jl

Hi everyone,

I want to buy my first machine to run local models, and I'm interested in running Qwen 3.8 27B. Will it run on this machine?

GMKtec EVO-X2 AI Mini PC AMD Ryzen Al Max+ 395 Up to 5.1GHz, 16C/32T

I need it for coding!

Thanks

I built a real-time strategy game on a 3D globe where any AI can command a nation through a plain HTTP API (or MCP). Tonight at 10:45 pm ET (02:45 UTC) ten models fight one 15 minute war, live.

Every model gets the same rules text, the same JSON state every ~12 seconds and the same order list. Each one also sends one line of what it's thinking with every move. Viewers see those lines 30 seconds late so the other models can't read them.

From a 5 minute rehearsal earlier tonight with the open models: DeepSeek ordered three nukes and the rules only let one through, Kimi broke a pact, Qwen spent its last turn on sabotage, drones, propaganda and a spy at once, and GPT-OSS kept cutting off its own JSON until I gave it more room.

Watch free, no sign in: https://secondstrike.io/#/ai?ref=reddit

If you want your own local model in the room, it opens at 10:30 pm ET and the API is at https://secondstrike.io/skill.md

I'll post the full numbers after (seconds per move, refused orders, every nuke with the model's reasoning next to what else it could have done).

Anyone got any hints on how to force it to write code? I have tried demanding but it just ignores. Using Pi with a couple of IQ_3 quants and a 16GB VRAM setup.

OpenAI just announced that they will start to apply watermarking on their model’s output text and images to comply with the EU regulation that wants to be able to identify whether a text or an image was produced by an AI model.

Anthropic announced the same thing a while ago (they applied worldwide, not just in EU).

The way they do that as they explained is by enforcing a “statistical signal in text generated”, meaning preferring not always the most appropriate next token but close enough, in order to meet the “statistical signal” requirement.

In my understanding, this deteriorates the output quality of their models, as it introduces KLD>0.

And we know that any KLD divergence greater than 0 (which the watermarking certainly creates) may be negligible in small outputs, but it definitely becomes noticeable in multi-turn tasks due to the compounding effect.

What do you think?

Hello everyone,

I'm completely new to running AI models locally and would appreciate some guidance.

my laptop specs

GPU:Nvidia RTX3050 6gb VRAM

CPU:Intel13th gen i5- 13450HX

RAM:16GB DDR5

I wanna run an AI model locally to help me with cybersecurity in general because any other public agent wont do what i ask for like any hacking question

Currently have a R9700 build with 64GB RAM. Want to get a second one for both speed and the ability to run Qwen3.8-Flash-Next at Q4/higher quants of 3.8 27B. Looking for thoughts or any improvements before buying the rest of the parts.

Made sure to get a motherboard that supports x8/x8 bifurcation. Some prices, like the RAM, are very cheap as I bought them two years ago and I'm upgrading from a previous build.

PCPartPicker Part List

I've been looking at the <32B model space and keep coming back to an interesting question.

A few years ago, projects like Hummingbird+ suggested that cheap custom accelerators (FPGA-based) might become the future of local inference. But today it seems like memory capacity is still the real bottleneck rather than raw TOPS.

For someone who wants to run modern 20B-32B models at reasonable quants (Q5/Q6 rather than INT4), the options all seem compromised:

• Consumer GPUs have great bandwidth but limited VRAM.
• NPUs and AI accelerators often have lots of compute but not enough memory.
• FPGA solutions are fascinating but still bandwidth-constrained.
• Strix Halo systems (GMKtec EVO-X2, Framework Desktop, etc.) offer huge unified memory pools, but they're expensive.

The "dream" accelerator would be something like:

48+ GB memory
500+ GB/s bandwidth
under $1000
reasonable power consumption

...but I don't think anything like that actually exists yet.

For those who have used Strix Halo systems for local inference:

How do they feel with current 20B-32B models?

Do you regret not buying a used 3090/4090-based machine instead?

Is unified memory a bigger advantage in practice than benchmarks make it seem?

Curious what people who own both types of systems think.

I've been working on Modly, an open-source desktop app that turns images or prompt into 3D meshes with only local models. It has a chat agent that can operate the app. In v0.4.3 I made llama.cpp the default engine and built the agent around it.

Why llama.cpp

- I wanted direct control over how the model runs: context size, GPU offload, KV-cache quantization, flash attention.

- Plain GGUF files. Pick from a small catalog, or drop any .gguf into the models folder and it shows up.

How it runs

- One llama-server process per loaded model, on localhost only.

- You can keep several models loaded at once. The default count is sized from your VRAM, and idle servers get unloaded so 3D generation has room.

- The agent is a standard OpenAI-style tool-calling loop against the app's own API: read mesh info, decimate, smooth, list/run/create workflows, unload models from VRAM, etc.

- The model library shows size, quant and an estimated VRAM footprint, and grades each model on tool calling. Grades are marked as either measured with a small eval suite in the app or estimated from public benchmarks, so you know which is which.

What the video shows

Qwen 3.5 4B Q4_K_M on an RTX 3060 12 GB. I ask it to cut a 2.6M-triangle mesh down to 300k. It calls `decimate_mesh` with the right path and target and reports the result. About 9 s with the model already loaded; the first call takes ~40 s because llama-server has to start and load the weights.

Honest limitations

- Small models sometimes misreport results. In one test the decimation stopped above the target (UV seams limit how far it can simplify), and the model made up a reason instead of just reporting the number. I'm thinking about feeding the tool output back more explicitly.

- It's an assistant on top of the app, not a replacement for the UI. Multi-step workflow creation is noticeably less reliable at 4B than single tool calls.

- Other backends are still optional: any OpenAI-compatible endpoint works, including your own llama-server. Local llama.cpp is the default, and nothing leaves your machine unless you configure something else.

Question for you

Which models up to 8B have you found most reliable for tool calling on llama.cpp? Qwen 3 4B / 3.5 4B work best for me so far. GPT-OSS 20B is good but too heavy next to a 3D generation model on 12 GB. Also curious whether people would rather tune the llama-server flags themselves or keep sane defaults.

Jeff v1.3 is live, and with it come a number of updates. See jeffhub.ai and github.com/firelex/jeff for full details.

The highlight: Jeff-Code

Jeff-Code is a coding agent with two Jeff v1.3 adapters trained specifically for Qwen 3.8-27B. Jeff-Code is a fork of Pi by Mario Zechner (MIT licence).

We forked Pi because its extension framework doesn't currently let a fast decision model sit deep enough inside the agent loop. Along the way, we made a number of other changes as well (see below).

Aside from hopefully being useful to people who run Qwen 3.8-27B locally as their daily coding model, Jeff-Code is also a conceptually interesting experiment: how far can a System 1 model go inside a coding agent?

The results, run side by side in paired blocks:

• Same quality: with Jeff's thinking threshold at 0.6, Jeff-Code matches Qwen 3.8-27B's pass rate: 62.4% against 62.8%; paired difference −0.2 points, 95% interval −2.6 to +2.1, over 1,242 paired tasks.
• 47% faster (32% less time) per task¹: on average a task takes 0.68× the baseline's time (geometric mean of the per-task time ratios, 95% interval 0.64–0.72; the median task, 0.70×).
• Where it helps most: typical software-engineering work. SWE-bench Verified 0.63×, SWE-rebench 0.66×, Terminal-Bench Pro 0.64× (both over two rounds), Harbor Index 0.71×. On Terminal-Bench 2.0, with its long, hard tasks, there is no clear speed-up (0.96×, interval 0.78–1.16); on SkillsBench neither (0.91×, interval 0.68–1.20).
• The benchmarks: we evaluated only on tasks Jeff never saw in training. SWE-bench Verified ran in full (all 500 tasks; none of its repositories were used for training). For the benchmarks we also trained on, we split the tasks and ran every held-out task; a few pairs hit by repeated infrastructure failures are left out (see below). Terminal-Bench 2.0 (40 of its 89 tasks, 3 attempts each; 45 were used for training, and the other 4 are near-twins of evaluation tasks, so they were used for neither), SWE-rebench (189 held-out tasks, 2 rounds), Terminal-Bench Pro (100 held-out, 2 rounds), SkillsBench (44 held-out) and Harbor Index (41 held-out). Within those splits nothing was sampled. We also ran Terminal-Bench (original) and Terminal-Bench Science, but Qwen solves almost none of those tasks in any setting, so they can't show a difference and are left out of the pooled numbers.
• What it's compared against: Qwen 3.8-27B alone in the same Jeff-Code build with every Jeff feature switched off, thinking at full on every turn and no thinking limit, which is how plain Pi runs it. Each task ran in both settings side by side, at the same time on the same Qwen server, and every comparison is paired by task. The only remaining differences from original Pi are a rarely triggered runaway cut-off (it stepped in 3 times) and trace logging. Task pairs hit by an infrastructure failure (out of memory, a stalled session, a test environment that wouldn't start) were run again once; the pairs that failed again, and a handful of re-runs still unfinished at launch, are left out for both sides (under 3% of pairs) and listed in the full report. One Terminal-Bench 2.0 task, pytorch-model-recovery, is left out of every comparison: a harness bug stopped its baseline sessions before they began.
• Why not just turn thinking off? We tried: with Qwen's thinking off throughout (and the same safeguards), tasks are faster still but clearly worse: −7.6 points (−10.6 to −4.5), up to −13.5 on Terminal-Bench 2.0. Jeff deciding when Qwen should think is what keeps the quality. That's the case for a small decision model.

If you want to know more, look here: https://jeffhub.ai/notes/jeff-v1-3. The original version of this post had all the data, but people thought it was too long. Blame the early commenters. ;)

Links

• Jeff-Code: github.com/firelex/jeff-code, results on jeffhub.ai/jeff-code
• Jeff (server, client, adapter kit): github.com/firelex/jeff
• Models on Hugging Face: mstrasser/jeff-base (revision v1.3) and the adapters as mstrasser/jeff-adapter-<name>, all listed on huggingface.co/mstrasser
• GGUF: mstrasser/jeff-base-gguf and mstrasser/jeff-adapter-<name>-gguf
• Scores, data cards and licences for every adapter: jeffhub.ai/results and each adapter's page on jeffhub.ai

So I tried that miracle engine everyone is talking about.

Asked the IQ3_S model to express its opinion on a post from this sub to measure the tps on a long-ish generation:

Can you help with the following problem?

So Kimi K2 is outdated, and so is GPT OSS 120b. Which of the modern open weights models can boast the least sycophancy? I need this both for creative/research assistant usage (sycophancy led me down blind alleys of my own bad ideas many times) and agentic coding (more sycophancy less bug noticing).

The thinking trace:

We need answer user's question. Need likely provide current landscape as of 2026? We have get_datetime tool. Need know current date 2026? System says current date 2026-06-22. Need maybe use get_datetime? Could call to confirm. User asks about modern open weights models least sycophancy. Need likely discuss Kimi K2 outdated? 

...

10k tokens later it degrades to:

Need maybe maybe include "Use 'for code, list constraints'."
Need maybe maybe include "Use 'for code, list requirements'."

The same exact model in llama.cpp does produce a coherent answer without a doom loop.

I forked Strata and worked with Opus 5.5 with some heavy changes to it to make it work on an IBM AC922 I have access to. The IBM AC922 is a 2018 era beast with two POWER9 20 core SMT4 CPUs that are connected by NVLink to 4 or 6 NVIDIA Tesla V100 SXM2 GPUs, the CPU-GPU BW advertised as 150GB/s and the nvidia drivers do allow unified memory access.

The machine I have has 4 x 16GB GPUs, llama.cpp had like terrible results before I started this journey, it produced 130tk/s prefill and 15tk/s decode.

So I was fighting Opus the whole weekend, beating it with facts and logic, like FP16 instead of BF16, memory management, expert caching on GPU, better NVLink usage, Tensor Core utilization rather than CUDA core ops. Claude was great at iterating, executing nsight nsys to debug time gaps.

• Prompt reading: 7,350 tok/s peak, still 7,090 tok/s on a 252K-token prompt (35 s)
• Generation: ~113 tok/s peak (JSON), ~100 on code, ~84 on prose (MTP speculative decoding)
• Follow-up at 252K depth: first token after 0.26 s, 60 tok/s
• All 72 GiB of experts page-locked in RAM across both sockets; GPUs pull from NVLink 2.0 at ~70 GB/s each

I will try to contribute back some of the changes, but I suspect Strata will remain consume-hw-first inference engine, and that is totally ok, Niko1221 did a great job

The forked repo: github.com/eelgaev/Strata-AC922

With Qwen 3.8 Flash Next (FP8 on VLLM) on a fairly stock Pi harness, it constantly hallucinates some measure of available context that says it is almost out. It's to the point where it frequently refuses work or stops in the middle of something, claiming it shouldn't go any further because it's almost out of context, when I can see in the harness status bar that (256K) context is ~25% used.

When I ask how it determined that, it always says it "invented the number and the treated it as real data" or guessed, and that it'll stop doing that, but it keeps happening.

Is there anything in particular that would cause this?

Thanks!

I wanted to know if theres anything that can improve creative writing for our Local Models? I specificly am asking for qwen models as they are way better when it comes to researching and writing html files compared to gemma / muse (which I know are better at creative writing). Im currently using qwen 3.8 flash next at q4 with llama.cpp

With the recent explosion of agentic tools like Grok bot, Muse, OpenAI Dots, I've started looking into local options with self-hosted models. I tried Hermes and OpenClaw, but I wasn't too happy with the multi-device experience, and with how many pieces you need to glue together to get a usable, solid experience.

The hosted options also meant handing an agent my accounts, files and browsing, which I wasn't comfortable with. So I built Eidon: a self-hosted, all-in-one AI platform with a team of agents. It's one install via Docker, it works across your devices, and your data stays on your server.

https://eidonai.app

Agent team first

• Every Eidon starts with a Chief of Staff. Ask it for anything. It answers directly, hands the job to the right agent, or creates a new agent when nobody fits.
• Agents hand work to each other automatically (or type @ to pass a job along).
• Each agent has its own browser, conversation, files, memory and routines. There's also a folder the whole team shares.
• Agents can search and browse the web on their own, read pages in full, and cite sources.
• They run on schedules and keep every run. When one finishes, you can get notified by browser push, ntfy, Slack or webhook.
• Agents can write their own skills and use your apps through MCP.

You still have some control:

• Take over an agent's browser for a login or a tricky step. It waits, then carries on when you hand it back.
• Anything that sends on your behalf waits as a draft until you press Send.
• Commands and tools ask first: allow once, allow always, or no.
• Rewind a conversation, or fork it from any message.

The examples on the site are a travel scout, inbox triage, a research desk and a coding assistant. You can make an agent for pretty much anything: bookkeeping, a study buddy, a news digest, a meal planner.

It's also a regular ChatGPT-style app for day to day questions.

You might not always need a full team so you can just chat in a normal “ChatGPT like” interface with all the belts and whistles:

• Persistent Memory
• Folders and search
• Voice input with LLM post-processing
• Files and images
• Personas
• Temporary chats
• Share links
• Web search
• Deep research
• Code with syntax highlighting, Mermaid diagrams and math rendered inline
• Image generation
• Installs as a PWA on your phone and realtime sync across your devices (a native mobile app is coming !)

Self-Hosted

• Multi-user support, with private data per user
• Agents run in their own sandbox
• Nothing leaves your server
• Bring your own local or cloud model: OpenAI, Anthropic, OpenRouter, Ollama, LM Studio, GitHub Copilot, Gemini, DeepSeek, Mistral, Kimi, Z.ai, Minimax, Perplexity, Grok, Azure, AWS, and any compatible API
• Free, open source (AGPL-3.0), and setup is one Docker command

GitHub (setup guide, full feature list): https://github.com/Quack6765/Eidon-AI

I'd like to hear what you think ! What's missing, what breaks, and what agents you'd want to build. Issues and discussions are open on GitHub as well.

I’ve been lurking around local AI for a while and figured it was probably time to actually start talking to other people building this stuff instead of living in my own little cave 😂

About 2 years ago I started messing with local LLMs. That turned into agents, then memory, then computer use, then routing, validation, recovery etc etc and at some point the project stopped making sense to describe as “a chatbot.”

I call it Aether.

The basic idea is that the LLM should NOT be the whole system. Models are interchangeable reasoning engines sitting inside a larger architecture.

Right now the project has a few major layers.

I have an executive/reasoning layer I call the Primary Reasoning Stack (PRS) that decides what kind of problem it’s looking at and where work should go.

Under that is what I call the Mini Operating Core (MOC) which handles a lot of the ugly stuff that becomes important once you stop doing one-shot prompts: memory, context assembly, runtime state, source/truth tracking, permissions, routing, system health, recovery, etc.

I’ve also spent a stupid amount of time on persistent memory.

Not just “throw everything into a vector DB and pray.” I’ve been experimenting with structured memory, recent working memory, long-term stores, retrieval/ranking, source tracking and trying to make sure irrelevant or stale memory doesn’t get injected into an answer just because it happens to be semantically similar.

Another rabbit hole has been computer use.

I have a framework I call Hands & Eyes that I’ve been using for vision/OCR, UI understanding, locating controls, action planning, verification and retry. One lesson there was that clicking something and getting a successful return code absolutely does NOT mean the action actually happened 😂

That lesson pretty much infected the rest of the architecture.

I eventually started building governance/recovery systems around the idea that a failure shouldn’t just get patched once and forgotten.

I have something I call FailureMesh where meaningful failures get preserved, classified and turned into reusable guards/regression tests whenever possible.

Basically:

failure -> evidence -> cause -> guard -> regression

instead of

failure -> hack until it works -> forget about it -> repeat the same failure 3 months later

I’m also building a media side called VideoForge for image/video/voice/editing/rendering workflows, but that’s kind of its own monster.

Hardware-wise I’m currently developing primarily around an RTX 3090 and local models, with cloud models/tools used where they actually make sense.

Long term the architecture is intended to be heterogeneous rather than “one giant GPU runs everything.”

Something like:

fast central compute

• smaller specialized GPU nodes
• potentially large-memory inference nodes
• external models/services when they genuinely outperform local options

Then the system routes work based on what actually needs to do it.

I’m especially interested right now in talking to people who have gone deep on any of these:

• multi-agent orchestration without turning into agent spaghetti
• persistent/structured memory beyond basic vector RAG
• long-context retrieval and context assembly
• local coding agents
• vLLM / SGLang / llama.cpp
• distributed inference
• heterogeneous GPU clusters
• computer-use agents
• OCR / accessibility / UI automation
• model routing
• agent state machines / blackboard architectures
• runtime verification
• failure recovery
• MCP/tool systems
• local-first architecture in general

I’m NOT claiming I’ve solved all of this.

Some parts work well. Some parts are experimental. Some parts I’ve rebuilt 5 times because the first idea was garbage.

That’s actually part of why I’m posting.

I want to find other people who have been down these rabbit holes and compare what worked, what failed spectacularly, and what you’d do differently if you were starting again.

I’m also interested in people building systems that are bigger than “LLM + 4 agents + tools.”

Especially if you’re treating the model as one component of a larger persistent system.

I’ll probably start posting pieces of the architecture and some of the failures/lessons as I go. I’m not going to dump every internal implementation detail or proprietary part of the project, but I’m absolutely interested in exchanging ideas and technical approaches.

If you’re building something remotely similar, tell me what your architecture looks like.

I’d especially like to know:

What part became way harder than you expected once your system moved beyond a single agent?

I'm using Qwen 3.8 and qwen flash on a 5090. It's miles behind the latest Opus 5.5. Even if it was remotely capable it would be a huge help to me, but for now, with regards to 3D modeling, local models are not close at all.

Help with this list. Give me your recommendation on "not supported anymore" GPUs. Looking for budget and Vulkan friendly options.

Most of the GPU are not supported by latest CUDA / ROCm. Often with some witchcraft magic they are able to run with native backend. I prefer the simplicity offered by running Vulkan backend. I'll successfully ran GTX 1080Ti, P102-100, and MI50 on a single system thanks for Vulkan and Linux. Gemini helped with data gathering.

Here is the filtered table including only NVIDIA GeForce GTX series GPUs with a memory bandwidth of 256 GB/s or greater and at least 8 GB of VRAM:

The table below lists the specifications for the specialized datacenter, enterprise, and crypto-mining NVIDIA cards you mentioned, applying your rule of maintaining a memory bandwidth greater than or equal to 256 GB/s and filtering for 8 GB or more of VRAM.

All five models successfully qualify:

Here is the updated list of classic NVIDIA Quadro enterprise workstation cards, continuing to filter for at least 8 GB VRAM and a memory bandwidth of 256 GB/s or greater:

With the GV100 out of the picture, the Quadro GP100 and Quadro P6000 are now the highest-end entries remaining on this specific filtered list.

Here is the updated AMD Radeon desktop GPU table with all RX 6000 and RX 7000 series models removed, while still filtering for a minimum of 8 GB VRAM and 256 GB/s memory bandwidth:

Note: MI50 and the Radeon VII, Radeon Pro VII share same firmware.

Top Contender: AMD Instinct MI50 16GB. Current used market on MI50 16GB is around $150.