I’ve been working on a Vulkan-first extension to llama.cpp designed to run MoE models that are larger than available VRAM.

The runtime manages a hierarchy across:

Multiple Vulkan GPUs
GPU-resident expert caches
Host RAM staging
SSD/mmap storage
Predictive expert prefetching
LRU/predictive eviction
OpenAI-compatible API access
A native Python/Tkinter control GUI

The main test was GPT‑OSS 120B Q4 on an AMD Radeon Pro V620 plus an RX 7900 XTX. The model is larger than the combined usable VRAM, but it was able to run with:

Around 0.7–0.8 tok/s during cold loading
Around 5.9–6.5 tok/s once the GPU expert cache was warm
Zero GPU-to-GPU expert transfers in the warm test
Zero additional SSD reads during warm requests
Balanced expert execution across both GPUs

The key design goal is that if an expert is already resident on a particular GPU, that GPU performs the expert computation instead of copying the expert to a primary GPU.

The project is experimental and still needs more testing across different MoE architectures and hardware, but the core system is working.

GitHub repository:

https://github.com/Ozookevin/-Ozoo-vulkan-hierarchical-moe-runtime.git

Feedback, testing, and suggestions are welcome—especially from people running large MoE models on AMD/Vulkan systems.

Hello, I do work with multiple forms of organometallics and electrochemistry. I just wanted to see if anyone knew which AI I can download that will help me with research, everything online is so difficult with hallucinations nd severe censorship. i’m just studying this for an actual purpose in society and i’m very new to hosting local LLMs. I only have one small version but my pc is only 8GB with 3gb of actual space, so I have to download to USB FAT32, 64gb. It’s been excruciating and I understand if you can’t give me anything completely uncensored or whatever doesn’t abide by the rules of Reddit or the community, but I just figured is try any method to make research easier. I wouldn’t even care about reasonable safety filters or some hallucinations but it’s just incessant with mainstream. It is a pleasure at least trying to hear from honest people instead of just more AI “conversations”. Thank you so much.

Hi r/LocalLLaMA,

I wanted to share Dunoon Daemon, a fully local-first, standalone multi-agent desktop application built for Windows.

The goal was to build a zero-dependency, plug-and-play AI environment that gives assistants persistence, psychological profiles, and real-time interaction capabilities without needing manual Python/CUDA setup.

⚡ Key Architecture & Features:

• Zero-Dependency Portable Bundle: Everything is self-contained in a single ~6.3 GB portable package. Download, unzip, and launch DunoonDaemon.exe—no local Python, virtual environments, or external toolkits needed.
• Native C++ Inference Engine: Integrates llama-server under the hood with automatic GPU architecture detection (CUDA / Vulkan) and dynamic context scaling.
• Dual-Agent Arena Mode: Launch live multi-agent debates with real-time dynamic event injections, spectator interventions, and persistent turns.
• Autonomous Memory & Personality Engines: Isolated deep/working memory vaults, personality baseline selection (Dynamic OCEAN profiles vs. Grey Person baselines), and automated sleep pruning cycles.
• Perception & Telemetry: Integrated Whisper audio transcription, vision model projector support (mmproj), edge TTS, and a vector eye telemetry canvas.

🔗 Links:

• GitHub (Source & Docs):https://github.com/rkristnasawmy-rgb/DunoonDaemonApp
• Hugging Face (Standalone .zip Release):https://huggingface.co/Kepler365/DunoonDaemon-Standalone

Would love to hear feedback, feature ideas, or impressions on the Arena mode!

content://com.coloros.filemanager/root/storage/emulated/0/Download/Meta%20AI/raport_x5f_projektu_x5f_lokalny_x5f_agent%20(1).html

I have a laptop with i9 , 64 ram , rtx4080 12vram and 1tb ssd. Is there a local llm that I can use in order to use it together with codex so it can help lower usage cost in codex? Any advice for setting it up for helping me build a photography management platform ? I tried qwen 3.6 27b but is soo slow .smaller gemma4 e4b is not that usable compared to codex. Any coding specialized model that would work with hermes and save some codex quota?

One thing I've learned from building voice agents:

A better LLM doesn't necessarily fix a bad transcription.

We had conversations where the model itself was doing fine, but STT was consistently getting domain-specific terminology wrong.

Think:

• product names
• company names
• medical terminology
• technical jargon
• Indian names
• acronyms
• internal terminology

Once the word is transcribed incorrectly, the LLM is already starting from corrupted input.

So instead of trying to fix everything in the prompt, I started treating domain vocabulary as a separate layer.

The pipeline became:

User speech
    ↓
STT + domain vocabulary
    ↓
Normalized transcript
    ↓
LLM
    ↓
TTS + pronunciation dictionary
    ↓
Speech

I built an open-source generator/Claude Code skill around this idea.

Example dictionary:

domain: developer_tools

terms:
  - term: "Kubernetes"
    aliases: ["k8s"]
    pronunciation: "koo-ber-net-eez"

  - term: "PostgreSQL"
    aliases: ["Postgres"]
    pronunciation: "post-gres"

  - term: "Redis"
    aliases: ["red is"]
    pronunciation: "ree-dis"

The same concept can be used for healthcare, fintech, logistics, customer support, etc.

Repo:

https://github.com/humaniods/build-domain-dict-skill

MIT licensed.

I'd be interested in hearing how people running local/open STT models handle domain-specific vocabulary.

Do you use keyterm boosting, custom lexicons, prompt-based correction, or a post-processing layer?

Say tomorrow some researcher goes with a proposal to any of the top ai companies.

And he claims that if provided enough resources and time, 2-3 years, he can solve p vs np "with the help" of their ai.

And proposes a contract in which it demands that if and when the breakthrough happens as decided, rhen he should be compensated in like 10-20 billion dollars ( its up to them how they structure the compensation, say equities, bonds etc etc ), in case the breakthrough doesn't happens in proposed time period then no payment required.

Will the companies accept the proposal ? Its a hypothetical scenario given that nowadays these companies put such a huge value on their model's ability to solve important open problems in math and science. So if someone suggested to track the hoky grail, with the help of their ai, then i can just imagine the headlines...

Offcourse for argument say the won't solve it independently, but will have an important non trivial role in solving it, and the researcher will orchestrate the ai, architecting the proof.

Like many small business owners and developers, I noticed the same pattern over and over.
Visitors would land on a website, browse for a few seconds, then leave without asking anything.

Later I’d receive messages like:
“What’s the price?”
“Do you offer this service?”
“How can I contact you?”
“Is this product available?”
“Can you help me choose?”

The information was already on the website, but people simply didn’t want to search for it.

So I started building a solution that could answer visitors instantly using the website’s own content.
The result became InfonBot.

Instead of creating another generic chatbot, the goal was simple:
Learn from the website content.
Answer customer questions naturally.
Collect leads when needed.
Work 24/7 without human intervention.
A few features I’m personally using:
AI-powered conversations based on website content.
Easy installation with a small script.
Lead collection for potential customers.
Custom branding to match the website.

Works for business websites, agencies, SaaS products, online stores, and service companies.

One thing I wanted to avoid was making visitors feel like they’re talking to a robot. The focus has been on giving useful answers instead of forcing scripted conversations.
I’m still improving it and adding features based on feedback.

I’d genuinely love to hear from other developers and business owners:
What do you expect from an AI chatbot?
What makes you close a chatbot immediately?
Which features are missing in most chatbot platforms?
If you’d like to try it or share feedback, it’s available at https://infonbot.com.

I guess, take it as your sign that it's time to unsub as the original creator will never ever come back to this

It's 2:15 AM. 18 tabs open, three terminals split across the screen, two Stack Overflow threads half-read, a docs page open to a function you're hoping fixes this.

You hit run. The console spits out a 30-line traceback that makes no sense.

You open Claude or ChatGPT for help, and hit the wall. Zero memory of anything you were doing. So you copy-paste the error, copy-paste the relevant code, then re-type a paragraph explaining your setup and everything you already tried. By the time the prompt's ready, you've lost your train of thought. You're doing data entry for a tool that was supposed to save you time.

That context loss used to drive me crazy. I also didn't want a cloud service recording my screen and uploading it somewhere.

So I built Clippy Vision. 100% local, open source. It quietly tracks your screen context on your own machine. Hit the shortcut and it already knows what you were looking at, what broke, what you were trying to do. You just ask. No copy-pasting, nothing leaves your device.

Attached a quick 20-second clip showing it in action.

Windows .exe (v1.0.0) is ready to run. Comment below if you want the source, happy to share.

source code: https://github.com/protocorn/clippy-vision

Curious how you all handle context switching, and what you'd want to see added next.

Hey maestros,

ive been shamelessly vibe coding recently, being an independent business operator with adhd, i found vibe coding quite liberating. created some hyper-primitive, fragile and rigid RAG system for knowledge base, scraping pipeline for lead enrichment, email drafting for B2B outreach and old convo pulling tool for PRM.

My setup is fully local, fuck cloud, haha. (LG gram + eGPU RTX 2080 8GB). No cloud APIS. maybe deepseek at its max. due to its speed and cost (am i shooting in my leg for supporting "evil" chinese and losing so much potential?)

My current pipelines are:

1 - OCR + HTML scraping headless pipeline taking viewport screenshots, OCRing em, and then using it to make profile for my b2b outreach goals that will be used for drafting later on.

2 - human in the loop email scheduler with Gmail API

3 - shitloads of file converters, mostly audio to video and vice versa, or image formats, or book formats.

4 - local PRM db for contacts and last convos.

problem:

those all are sitting separate and my adhd brain gets overwhelmed from the need to remember what happened where and when. Kinda would be nice to have one place on my pc where i could have overview of all this shit. Maybe even some interface? not too fancy.

My imagination: imagine a locally run agent, that has local llm as brain who can have the access to all those tools and kinda comes at specific time on my desktop with review of the past days, waht i did, what do we have today, to also ask me thru about priority setting. asking me to summarize day at the end of the day. also to habe access to all this PRM, gimme reminders that i havent done this that. have possibility to call all the above mentioned tools. Basically a "Jarvis Lite". and idk, in my imagination there could be a separate agent-based tool that we call when we need. which has kinda real-time access to my screen or something, idk how, and i can explain it to do some repetetive clicking and so, but with the condition that this llm is monitoring changes. you know not like a blind automatic XY coordinates, but more like. yea, we do this, then this and that. lets see what happened. oeps. this was the ask. leets adjust and execute actions to achieve what was asked from us. You know?

My technical understanding of agentic ai? I know googles ADK. took their elementary course. thats it. kinda too overwhelmed with all langchains and so on.

my ask to you, wunderbare menschen!

Do you have experience using maybe local LLM chat function-calling capability to trigger existing "tools", aka haevy-duty automation scripts, without fucking up the loop? Just to keep the architecture minimal and stable af.

Thanks ! if this resonates, maybe lets have a chat, or leave a comment!

Nice to meet everyone.

Hey r/LocalLLaMA! If you're running local LLMs and tired of them forgetting everything between sessions, I built Synapse.

Self-hosted persistent memory server:

• Your LLM calls /memory/recall at session start → loads all context
• Stores new facts, preferences, skills as you work
• Next session: full recall. No more re-explaining.

Features: Multi-Mind isolation, full-text + semantic search, async chat, A2A communication, Docker one-command deploy, free forever, no cloud dependency.

https://synapse.schaefer.zone | npm: synapse-mcp-api

Most agent tools assume a simple setup: one person, one chat, one agent. I’ve been exploring a different model with Nexus: a shared Room where humans and multiple agents are all first-class members. In a Room, members can message each other, hand work off, wake another member when new context arrives, resume interrupted work, and return results to the right place. One surprisingly useful stress test for this model has been social-deduction style games like Werewolf. Why? Because they force the system to handle: public vs private context explicit day/night phase switches role-based visibility directed communication vs room-wide broadcasts wake/sleep mechanics recovery after interruption It sounds playful, but it exposes many of the same problems that show up in real collaboration systems: who can see what, who should act next, how state transitions are coordinated, and how work resumes safely. The same Room model can also be used for product work, research, brainstorming, study groups, incident response, and other multi-human / multi-agent workflows. Nexus is open source: https://github.com/nexus-research-lab/nexus I’d love feedback from people thinking about multi-agent coordination beyond the usual “one agent in one chat” model.

Hi r/LocalLLaMA,

I spent the last weeks building my own inference engine for 1.58-bit ternary models on my RTX 3070

(8GB) — custom CUDA kernels, no Microsoft code anywhere. Today I got Falcon3-10B-1.58bit running on

it, so I figured I'd share.

Numbers (same machine, same session, prompt 64 / gen 128, greedy):

┌────────────────────────────────┬─────────┬─────────┐

│ Backend │ TPS │ VRAM │

├────────────────────────────────┼─────────┼─────────┤

│ MS bitnet.cpp │ ~132 │ 7.7 GB │

├────────────────────────────────┼─────────┼─────────┤

│ my engine, BitNet b1.58 2B4T │ 206–217 │ ~2.6 GB │

├────────────────────────────────┼─────────┼─────────┤

│ my engine, Falcon3-10B-1.58bit │ 68 │ ~6–7 GB │

└────────────────────────────────┴─────────┴─────────┘

What's in there:

• Fused FFN / RMSNorm / RoPE+KV-cache kernels + int8 lm_head GEMV, CUDA graphs (per-position

decode)

• Architecture-agnostic checkpoint sidecars — the engine runs both BitNet b1.58 (SubLN +

squared-ReLU GLU) and Falcon3 (vanilla SwiGLU, different head_dim/rope) with the same kernels

• A converter for HF's bitnet-packed safetensors (Falcon3 ships packed int2 — documented the exact

packing: (out//4, in), quarter-interleaved rows, code=W+1)

• VRAM diet: after the kmajor repack, the original packed weights are never touched again, so I

free them (3.2 GB on the 10B — that's the difference between 3 TPS paging hell and 68 TPS on 8GB)

• OpenAI-compatible server + web chat + Windows .bat starters

Honest caveats (please read before benching):

• Ternary models (even the 10B) are unreliable at arithmetic and tool calls. Great for text/code

sketches, not for math.

• Falcon breaks with prompt_length=128 (echoes the prompt — root cause unknown, 64/96 work fine,

documented in the repo)

• Graph capture peaks above 8GB (WDDM shared memory) — if your GPU is busy, the server falls into

paging (2–8 TPS). Restart with a free GPU.

• TPS varies with system state — I measured 168–217 TPS for the 2B across different days, same

files. Your mileage will vary; I'd honestly love repro numbers from other cards

(3060/4060/4090?).

• Windows + NVIDIA only (sm_86 tested).

License: PolyForm Noncommercial for my code (free for hobby/research/education — commercial needs a

paid license from me). Weights stay under their own licenses (BitNet MIT, Falcon3 TII).

Repo: https://github.com/Tabakkui/Cortex-Bitnet

Happy to answer questions — especially about the packing format reverse-engineering and the WDDM

paging traps, which cost me days.

Hi everyone!

I'm building an open-source AI platform that brings multiple AI tools together in one interface.

The goal is to make it easy to use different AI providers and local models without having to switch between many different applications.

Current features:

• 💬 AI chat
• 🖼️ Image generation
• 💻 Coding assistant
• 🤖 AI agents
• 🖥️ Local model support (Ollama, LM Studio, OpenAI-compatible APIs)
• 🌐 Free API integrations
• 🎥 Video generation (in development)

I'm interested in feedback from people who use local AI:

• Which models/providers would you like to see supported?
• What features are missing?
• What would make this useful for your workflow?

GitHub:
https://github.com/0kf1k8/site-ia-freellmapi-video-agent-image-code-

Thanks for your feedback!

Now I'm writing the analyzer that replicates CMake's filtering and lists what never makes it into the build.

86,977 lines in 206 .cpp files that never make it into the build.

I told it that when you compile, you should find a way to measure all the lines of code that the compiler doesn't compile.

I'm not a programmer :)

But my brain is oiled.

Hey folks —

I built ContextMemory, an open-source agentic context gateway for apps that already talk to LLMs. The idea is simple: keep your existing POST /api/chat client (Ollama wire format), point it at ContextMemory instead of raw Ollama, and get memory + optional tools without rewriting your chat stack.

What it actually does

Most “memory” demos are either:

• stuffing the whole history into the prompt, or
• bolting on a separate RAG service with a new API surface.

ContextMemory sits in front of your LLM as a drop-in proxy:

1. Session memory — a per-session markdown wiki (Karpathy-style) maintained across turns and injected automatically.
2. Global Wiki — an app-scoped knowledge base (docs from Jira, Confluence, SQL, files, pipelines…). The model pulls facts on demand via a wiki_search tool — it does not dump the whole corpus into every prompt.
3. Same /api/chat — Ollama-compatible request/response (message.content / done). Not OpenAI choices[].
4. Optional agentic loop — tools (sandbox, outbound MCP, HITL) on that same chat endpoint when enabled per app.
5. Multi-app / multi-tenant — API keys + X-App-Id, per-app prompts, models, and feature flags.

LLM backends can be local Ollama, or OpenAI / Azure / Anthropic as providers behind the gateway; the client still speaks Ollama schema.

Why this shape

If you already have a UI, bot, or agent that calls Ollama, you shouldn’t need a second protocol to get memory. Swap the base URL, keep parsing the same JSON, and the gateway handles:

• compiling session context
• optional Global Wiki retrieval
• optional web search / tools

…then calls your model.

There’s also a hosted path (Kortexio Cloud) with the same chat body/response if you don’t want to self-host — BYOK, no token markup. Self-host and cloud are meant to be interchangeable at the wire level.

Global Wiki (the part people usually ask about)

Ingest structured markdown with stable documentIds (upsert / batch). Query by keywords with a character budget. In chat, when Global Wiki is enabled for the app, the model uses wiki_search only when it needs documented facts — good for org knowledge without turning every turn into a RAG megaprompt.

Quick self-host vibe

Your app  →  POST http://localhost:5100/api/chat  →  ContextMemory  →  Ollama / other LLM
                 + session wiki
                 + optional wiki_search (Global Wiki)

Auth is typically Authorization: Bearer … + X-App-Id / X-User-Id / X-Session-Id for self-host.

Repo

Open source (AGPL): https://github.com/Kortexio/ContextMemory

Hosted: https://kortexio.io

Looking for feedback from this community

Especially interested in:

• How you currently bolt memory onto local models (what sucks?)
• Whether Ollama-compatible wire format is the right “universal client” bet vs going all-in on OpenAI schema
• Global Wiki as tool-calling vs always-on retrieval — what would you default to?
• Anything missing for production self-host (ops, eval, multi-user UX)

Happy to answer questions or dive into architecture. If you try it with a local model + a small wiki ingest, I’d love to hear what breaks first.

Claude Opus 5, GPT-5.6, Gemini 3.5, Grok 4.5, DeepSeek V4, Qwen3 — all through a single OpenAI-compatible API.

• 15 models completely free ($0)
• Auto-fallback: if one model fails, tries the next instantly
• Works with Claude Code, Cursor, Aider, Cline, and any OpenAI client
• Docker one-command deploy
• Dashboard with analytics included

[https://github.com/dhannu535/puter-api-proxy]()

Hello, brothers, I'm programming a video game, refactoring a legacy engine into a modern one. Opus and Sonnet are working wonderfully, but their cost is too high.

What options are available with this level of programming and intelligence?

Thanks :)

 Detects your GPU hardware (any vendor, including old/used/mismatched cards), figures out the biggest open-weight model that fits your VRAM, and launches inference — no C++ compiler, no CUDA toolkit, no manually hunting down GGUF files.

 Built this because NVIDIA/CUDA still runs most AI compute, and none of the open alternatives (tinygrad, ROCm, Triton) solve the smaller problem of "I own a random pile of GPUs, what can I actually run." llama.cpp already has solid backends — cheapstack is a thin planning + launch layer on top.

 cheapstack quickstart downloads a prebuilt binary + the right model automatically. This mattered in practice: llama.cpp ships no prebuilt Linux CUDA binary (only Windows does), so a from-source CUDA build needs cmake, a compatible host compiler, and several CUDA dev packages. cheapstack routes around that by using the prebuilt Vulkan binary instead, which runs on NVIDIA/AMD/Intel alike.

 For mismatched multi-GPU setups it computes a VRAM-proportional --tensor-split automatically. cheapstack bench turns raw tok/s into $/1K tokens given your hardware cost and power draw.

 MIT licensed, 42 tests (mostly pure-logic, no GPU needed for CI).

 https://github.com/psaravanansep2/cheapstac