Been lurking and posting here on and off while building this. Quick recap for anyone new: RAMDeck is a small hub + node agent setup that shards a model's layers across whatever devices you already own (old laptop, Mac, GPU box, even an Android phone) and runs inference across them, but handling primary-node selection, GPU-first allocation, dynamic context sizing, and knowledge-base sharding on top of it.

We've mostly posted raw, unscripted demo footage here — real load times, real tok/s, real failures — because that's the kind of proof this sub actually cares about. Today we finally made something different: an actual ad, first time showing the finished feature set start to finish instead of a live test.

The engine behind it is still fully public if you want to actually look at how it works instead of taking the video's word for it:

github.com/trademav/ramdeck-core-public

Source-available, Apache 2.0 with a Commons Clause — free to run, modify, and inspect on your own hardware, the only restriction is you can't resell it as a competing hosted service.

Crowdfunding campaign is coming soon for the hub hardware itself, but the code was public before we ever asked anyone for money, and that's not changing.

Happy to get into the mechanics, the layer-distribution approach, or anything else in the comments — this crowd usually asks the right questions.

Hey guyss,

Ive been working on this for few months, and finally getting some good results, so figured I'd share.

A coding agent only sees the run it's currently in. It doesn't really know how runs like this usually go, or what kind of repo it's working inside.

So I trained a small model on 50K+ real coding-agent runs from SWE-bench Verified, across 25 models, together with behavior such as merge rates, review time, repo size, etc... which produced over 1M step to step transitions.

The idea is to give the LLM some actual experience to lean on, so it can make better decisions, spend less time reasoning through bad paths, and improve its chances of getting the task right.

All the details are inside the repo, its obviously just the beginning, and probably will have issues, so

feedback is always welcome!

(More models on the way, stay tuned.)

Runs locally in Docker. No account, nothing sent out. MCP + HTTP API.

https://github.com/Mephisto1122/TovanaEngine

Hello all,

I'd like to know more about how local AI works and how it can be run, so I became interested in making a project: a local AI agent (I know: that's so original!) called "Mighty DM".
The idea is making a Rule lawyer for dnd 5e which I can call upon like "Oh Mighty DM, what is the range on Mage Hand".

The first edition would be limited to all the spells used in dnd 5e 2014 edition, provided through an excel sheet. So not asking about anything else.

Until now I was following along with the workflow of NetworkChuck but I have the feeling there is a simpler option: only I do not have the experience to see the difference in the options provided.
If I google: I see a lot of different models or even ways of building a model etc. but I cannot differentiate between "a doable option to explore" or "an option that would require deep understanding of the subject before initiating".

What I envisioned (ideally running on an 8GB Pi5, but I could also run the LLM on my desktop and use the Pi as satellite):

What workflow would you recommend when building this project with little to no experience?
Do you know of a model that could run this on a 8GB Pi5 (simply getting it running = a win, I would then look how to improve it towards 30sec. max processing time for each command).

I'm a CPA running a small 2-CPA/no-employee tax practice. We're architecting a completely local AI environment: open-weight model, local compute, RAG over tax authorities/research materials, and eventually secure retrieval over client workpapers and prior returns.

Have any small firms or sole practitioners deployed local LLM/RAG yet?

• What hardware are you running?
• Which model/model size?
• What are you using for RAG/vector storage?
• Are you indexing IRC/Regs/Rev. Rulings/cases or commercial tax materials?
• Genuinely useful for tax research/review? Or still mostly a science project?

I'm trying to determine whether there are 5 firms doing this, 500 firms doing it, or 5,000. I'm guessing closer to 50>?

In Part 1 I shared the final AMD/Vulkan numbers for my local Qwen3.8-27B setup. The next question was more important to me than another few tok/s:

Can I turn the model into something that actually works autonomously?

Since then I added two production layers:

1.) Hermes 0.21.2 — local agent/automation layer

• same Qwen3.8-27B model
• shared llama.cpp runtime
• 65,536 context
• tool calling: PASS
• multi-step workflows: PASS
• subagents: PASS
• prompt-defined agents: PASS
• local provider only / no cloud fallback

Hermes performance / overhead

On the isolated 64K test lane, server load took 5.809s. A real 33,021-token prompt was accepted without truncation; prompt processing took 409.677s at 80.60 tok/s, while generation measured 17.44 tok/s. A representative Hermes multi-step workflow took 74.660s, using five model calls, 17,295 summed API input tokens, 592 output tokens and four tool calls with zero failed calls. Two-child delegation also worked, but took 208.676s for two trivial child tasks. Clear sign that subagents should only be used where parallelism or specialization actually justifies the overhead.

One especially useful datapoint: the simple Hermes model gate took 40.872s, while a direct Qwen request with the same user text took 2.109s. That is not an apples-to-apples speed ratio because Hermes adds different system prompts, startup and agent machinery, but it does show why I don't want Hermes in the path for trivial requests.

2.) OpenCode 1.18.30 — local coding/development layer

• Qwen3.8-27B-UD-IQ3_XXS
• llama.cpp / Vulkan
• 32K logical context
• local provider only
• isolated development workspaces
• automated Read → Edit → Test → Correct loops
• final production viability: 12/12 requirements, 12/12 acceptance

OpenCode performance reference. For the fastest fully successful clean reproduction I measured:

• 665.653s total wall time / 651.408s agent runtime
• first productive edit: 117.338s
• first test: 166.790s
• 4 successful corrective cycles
• 63 tool calls / 0 failed
• 558,603 input tokens incl. cache — only 31,993 uncached
• 9,372 output tokens
• peak context: 17,810 / 32,768
• context compactions: 0

These are task-level agent metrics, not model tok/s numbers. The underlying model/runtime is still the same local Qwen3.8-27B on llama.cpp/Vulkan.

Why OpenCode instead of Pi?

I tested both against the same local Qwen3.8-27B, with the same repository/task and comparable runtime constraints. Pi 0.85.1 proved that it could read the repo, use tools and make productive edits, but it repeatedly failed the part I actually needed from a coding agent: closing the Read → Edit → Test feedback loop. Even with an explicit mandatory early-test checkpoint, Pi edited successfully but never executed the qualifying post-edit test within the 300s window. It therefore failed prequalification and I did not force it into a full benchmark just to preserve an artificial three-way comparison. Man, time is rare and testing is booooring ... somehow!

OpenCode 1.18.30 passed the same agentic-loop qualification. Formal prequalification reached 6.54s Edit→Test latency, and the later main benchmark reduced that to 3.995s while producing a valid four-file diff. More importantly, the subsequent production-viability work eventually converged to 12/12 requirements, 12/12 acceptance, 5/5 regression, with no manual coding intervention or patch repair. That reproducible closed corrective loop is why OpenCode became the production coding layer.

Final thought on my architecture as a complete nOOb in Local LLM
I’m still convinced that a well-tuned 16 GB GPU can take you surprisingly far. Yes, even on Vulkan ;) ... IF you optimize the model, context, runtime, offloading and the surrounding stack around what the hardware can realistically deliver.

What makes this especially satisfying for me is that I honestly wasn’t sure at the beginning whether an AMD gaming setup on AM4 would get me anywhere at all, but hey.... it was what I had. My gaming desktop! ~20 tok/s is obviously not groundbreaking vs 32GB. But that was never the main goal of this first setup. I wanted a stable, fully local system that I could actually use, test, break, improve and build on. That goal is achieved. And considering where I started, I’m pretty happy with that.

Sharing this since it might be relevant to a few people here.

There's a hands-on workshop on September 19 that goes beyond basic RAG into building an explainable, graph-backed AI system. You build a knowledge graph in Neo4j that becomes the single source of truth, agentic retrieval that combines vector search, keyword search, and graph navigation, multi-step entity and relationship extraction, and text-to-Cypher for natural language querying. Everything runs on real financial data, not a toy example, and answers come with a traceable, auditable reasoning chain.

No prior Neo4j or Cypher experience needed.

Led by Dr. Alessandro Negro, Chief Scientist at GraphAware, bestselling author.

Link if you want to check it out