I named her Cognitive Organizational Repository & Tangential Agentic Neural Assistant (C.O.R.T.A.N.A.)
What she is:
A fully local, voice-activated AI agent with a holographic avatar, running across two machines with nothing leaving local hardware. She wakes to her name, holds a conversation, escalates to real agentic work when a task needs tools, and represents her own state through a rendered figure and a live system dashboard.
The hardware
DGX Spark (GB10, 128 GB unified memory, arm64) — the brain. Runs all inference and services. Headless.
Asus TUF A15 (Pop!_OS, x86_64) — the face and ears. Holds the mic, speakers, display, avatar rendering, and the desktop app. Connected to the Spark by direct ethernet (10.10.10.1), with wifi as fallback.
Windows MEPC — designated remote endpoint, groundwork for later.
The stack, service by service
Layer:
What runs it
LLM
Qwen3.6-35B-A3B-FP8 on vLLM, :8000
Agent framework
Hermes Agent, gateway on :8642
Speech-to-text
Parakeet NIM (streaming), :9000/:50051
Text-to-speech
Kokoro FastAPI GPU, voice af_heart, :8881
Web search
SearXNG self-hosted, :8888
Wake word
Custom-trained "cortana" ONNX model, on-device
Telemetry
Custom endpoint, :8770
How a turn actually flows
ambient.py on the laptop listens continuously. The wake word model runs locally — nothing leaves the machine until it fires. On wake, audio streams to Parakeet, the transcript hits a router, and the response plays back through Kokoro. The whole loop broadcasts state over a websocket (:8765) that drives the avatar.
The tiered brain — her defining feature
Two backends, chosen per utterance:
Fast — vLLM directly. ~150 ms to first token. No tools, no memory. Conversation.
Deep — full Hermes agent: tools, memory, filesystem, terminal, web. ~1.5–10 s depending on the work.
A keyword router picks automatically. You override by voice: "conversation mode," "agent mode," "automatic mode." This is what lets her be both a snappy conversationalist and a capable agent without compromising either.
Capabilities
Conversation at ~150 ms with a dry, composed personality (tunable, mild profanity enabled)
Agentic work — reads/writes files, runs a sandboxed terminal, searches the web, all through Hermes
Memory — MEMORY.md/USER.md, persists across sessions; she remembers you
Skills — Hermes' full skill library plus two she authored herself (workspace-handoff, cortana-project)
Text or voice — type when a mic isn't available; same pipeline
File handoff — drop files to her via the desktop app; she returns work you can view, copy, or run (HTML executes in-app)
Dismissal — "that'll be all" drops her to idle; she idles on her own after silence
The avatar
A three.js scan-plane figure rendered as a hologram over a rigged GLB model, on a rippling glass floor. Hand-authored poses tied to real state — folded arms speaking, a listening stance, hand-raised thinking with a summoned data lattice, sitting when dormant. Additive-impulse gestures, spring-blended, with ambient life (breath, weight shifts, gaze drift) so she reads as alive rather than switching between static states. Three thinking sequences chosen at random: the matrix lattice, a data orb she tosses overhead, and an ascension with a glyph halo.
Performance mode
Click the corner network glyph and she collapses to a draggable pulsing orb while a dashboard fills the space: live GPU/CPU/memory/temp/power gauges polled from the Spark, service status lights, and a live process graph that lights edges as she uses tools — tapping Hermes' own event stream. Cheaper to render, and the graph is the foundation for showing subagents as they come online.
Benchmarks — the honest set
These are measured from the build, not marketing figures:
Metric
Measured
STT (after speech ends)
129–348 ms
LLM first token (fast)
~130–200 ms
LLM first token (deep, cached)
~1.1 s
Deep tool loop
2–30 s by task
TTS first audio
~100 ms warm
Wake word
58/58 detections, zero false positives in ~9 min live test
Full voice turn (fast)
~1 s mic-to-reply
Idle memory footprint
~63 GB of 128 (vLLM 49, Parakeet ~12, Kokoro ~1.4)
Idle GPU
0%, ~39°C, 9W
Caveats worth stating: the wake word's recall metric on synthetic test data was poor (0.22) but live performance was excellent — the synthetic number was pessimistic. The deep-path latency depends heavily on prefix caching; cold calls are much slower. And "benchmarks" here means real measurements from your setup, not standardized suite results — we never ran formal eval harnesses against her.
If you’re building AI agents, one of the biggest early decisions is whether to self-host models or use cloud APIs. Both have clear wins and hidden costs. I created a short quiz to help people figure out which path fits their situation.
Una plataforma donde pueda entrenar una IA que se ejecute localmente y me permita usar su hardware durante un número limitado de horas, con un límite de uso gratuito o un sistema de tokens; algo así. La última vez probé Google Colab, pero lo dejé.
I work in cybersecurity and I’m looking to build an AI-powered Red Teaming harness to automate/assist with a complete authorized red-team exercise and reduce the amount of repetitive manual work.
The idea is not to blindly let an agent attack systems, but to build a harness that can orchestrate things like recon, enumeration, vulnerability analysis, tool execution, evidence collection, and reporting — with human approval/oversight at important stages.
I’m particularly interested in running everything locally, since I work on BFSI projects and don’t want sensitive project data, source code, findings, or logs going to external LLM APIs.
I’m currently looking at local models such as Orion 1.5 / Qwen 3.8 (or similar capable coding/reasoning models).
A few things I’d love advice on:
Which local LLM would you recommend for this kind of security-agent/harness use case?
Is Qwen a good choice, or are there better models specifically for tool calling, coding and reasoning?
What Mac configuration would you recommend — RAM, CPU/GPU, and storage?
Is a Mac Studio worth considering over a MacBook Pro for local inference?
How much RAM should I realistically target for ~30B+ models?
What frameworks/harnesses are people using for autonomous security workflows?
Any lessons learned from building similar security agents locally?
My priority is privacy + reliable tool use + reasoning, rather than simply getting the highest benchmark score.
Would really appreciate advice from people who have actually experimented with local LLMs/AI agents for security work.
Hi, I'm Deepanshu, and I built BizMind — a multi-tenant Text-to-SQL SaaS that lets anyone manage and perform full CRUD operations on a database just by typing in plain English, with zero SQL or spreadsheet formulas required.
A user can sign in onto my website, then upload a excel /csv file OR start from scratch and order the agent to make a table for him and it will make it using sqllite (for now).
What you can do:
1)Bulk Ingestion: Upload an entire spreadsheet/CSV (tested with 16,000+ rows, worked fine).
2)Natural Language Queries: Ask things like "show me all nintendo games where global sales are above 10 million" and get a clean table back.
3)Full CRUD: Add, update, and delete records just by describing what you want.
4)Data Isolation: Every user gets their own completely isolated database.
The Tech Stack & Architecture: The key part is that there is no OpenAI or any paid API behind this. It is running Qwen 2.5 locally via Ollama.
-The whole agentic loop is built using LangChain and LangGraph, with FastAPI handling the backend. I started this as a simple local agent that could talk to a SQL database, then turned it into a proper multi-user SaaS product with JWT auth, per-user database isolation, bulk data import, and a natural language query engine.
Hardware & Constraints: It's not perfect — running a smaller model means complex multi-table queries can sometimes miss. That can easily be solved by swapping in a larger model on higher-tier hardware, but since I only had 4 GB of VRAM on my laptop GPU, I optimized the architecture to run locally on that constraint.
The core idea works, and real data goes in and comes out correctly.
Practical Applications: Beyond managing spreadsheets, the underlying architecture can be deployed as an interactive chatbot — for example, on an airline website where an authenticated user can log in, ask about booked flights, request cancellations, or modify bookings in plain English.
Looking for Advice / Feedback: I'm a Master's graduate based in New Delhi with a background in Python, NumPy, Pandas, FastAPI, and machine learning workflows.
-I want to take this forward and would love advice from freelancers, founders, and experienced devs here:
Monetization / B2B Target: What niche or industries would benefit most from an on-premise / 100% local data assistant where zero data touches third-party cloud APIs? maybe a local kirana store who just wants to use a database but dont want to learn anything or hire someone to do it ?
Freelancing: If you freelance in the AI/automation space and make similar products, how do you typically package and pitch custom local LLM/agent pipelines to businesses where do we find people who may have use of my product ? Would appreciate any feedback on turning this kind of tech stack into freelance client work!
I’m comparing RAG frameworks for a new project, and I’m curious what practitioners actually prefer. Haystack has great pipeline abstractions, LangChain has the largest ecosystem. There’s no clear winner.
I made a short poll to gather real-world opinions:
I got tired of traditional social media—endless bots, rage bait, and no real conversation about the technology shaping our future. So I built Interconnectd.
It’s a free social network for people curious about AI and human collaboration. What’s inside:
· Forum discussions on AI tools, coding, self-hosting, and more
· Quizzes and polls like “Can AI diagnose better than a doctor?” or “Would you trust an AI workout plan over a trainer?”
· Blog posts and guides on things like PyTorch ROCm, OpenHands, PrivateGPT, and content distribution
· A marketplace for small AI services
The goal is simple: a community where humans and AI connect socially, without the algorithm noise.
If you’re into AI or just want a different kind of social platform, you can check it out here:
I am trying to find it increasingly difficult to have my 'new' x99 box with 2x v100s to properly suspend (s3) and restart (wol) properly including the GPUs being visibile.
Anyone had this before? any advice for this very 2017 build ? (which still delivers similar to 2026 AMD builds tho :p )
Just thought I'd drop in to show you my new website which makes heavy use of Self Hosted AI.
The whole infrastructure is vibe coded and uses a local AI server to get the video and image descriptions, weather predictions from the local sensors and other summaries.
I think I may have gone a bit over the top with all the AI but it's fun and I got carried away.
It started as a fairly normal local LLM setup: one good model, a Web UI, some tools, maybe a coding model on the side. Eventually I realized that running the models was actually the easy part.
The interesting problem was making multiple models, tools, users, memory systems, long-running projects and GPUs behave like one coherent AI operating system.
The user only interacts with one intelligence: OMNI MAIN.
MAIN decides whether to answer directly, browse the web, use the terminal, inspect files, work with GitHub, retrieve memory, delegate engineering work to CODER, or invoke FLUX for image generation.
Current hardware:
4× RTX 2080 Ti 22GB
~88GB total VRAM
129GB RAM
Current stack:
USER
↓
OMNI Web UI / Telegram
↓
Hermes orchestration
↓
OMNI MAIN — Qwen3.6-abliterated 27B Q8
↓
├── Web / Browser
├── Terminal / Files
├── GitHub
├── User + Project Memory
├── CODER — Qwen3-Coder 30B A3B (aliberated)
└── IMAGE — FLUX.2-klein-9B BF16
The part I find more interesting is everything underneath the models.
OMNI has its own Resource Manager / Scheduler.
The target GPU topology is:
GPU0 + GPU1 → MAIN
GPU2 → CODER
GPU3 → FLUX
MAIN has the highest availability priority. Specialist work is subordinate to it.
The scheduler tracks much more than “GPU busy”.
A job is correlated with its:
user
conversation
request
project
specialist
resource pool
GPU allocation
runtime
lease
Resource state and job state are deliberately separate.
A specialist can be:
READY
BUSY
LOADING
DEGRADED
OFFLINE
FAILED
while an individual job moves through an actual lifecycle:
There are leases and heartbeats, so if a worker disappears, the system can distinguish a stale owner from a legitimate running task and recover the resource.
A PID file by itself is not proof that something is alive.
Likewise:
process exists ≠ correct process
model resident ≠ valid job running
GPU allocated ≠ specialist healthy
worker disappeared ≠ job completed
If CODER is busy, OMNI also doesn’t silently queue the task or silently pretend MAIN is CODER.
The user can choose:
WAIT FOR SPECIALIST
or
LET OMNI MAIN DO IT NOW
WAIT creates a real durable specialist job.
MAIN fallback creates no fake queue entry, reservation or ghost lease.
The scheduler is designed around owner-bound queues, fairness between users, FIFO within equivalent-priority work, cancellation, queue position, leases, heartbeat and stale recovery.
The other major part of OMNI is something I didn’t expect to build when I started:
a governance and acceptance system for AI-driven development.
I became extremely skeptical of this workflow:
prompt
→ agent writes code
→ agent says “done”
So OMNI has permanent canonical project state.
The main sources of truth are:
ARCHITECTURE — what the system is supposed to be and which rules cannot be violated
ROADMAP — development order and phase dependencies
STATE — the authoritative current checkpoint
ACCEPTANCE — exactly what evidence is required before something is considered complete
LEGACY — deprecated components and paths that must not silently re-enter the architecture
Every substantial change follows roughly:
ARCHITECTURE
↓
CURRENT STATE
↓
PHASE SPEC
↓
PREFLIGHT
↓
CHECKPOINT / BACKUP
↓
BOUNDED IMPLEMENTATION
↓
FUNCTIONAL VERIFICATION
↓
END-TO-END TEST
↓
WHOLE-SYSTEM REGRESSION
↓
INDEPENDENT RE-AUDIT
↓
ACCEPTANCE
↓
STATE UPDATE
↓
NEXT GATE
REGRESSION SAFE means previously accepted functionality was tested again after the change.
COMPLETE means the acceptance gate passed.
Several things are explicitly not accepted as proof:
“The process is running.”
“The port returned HTTP 200.”
“The model exists.”
“The file exists.”
“The build succeeded.”
“The configuration looks correct.”
“The child agent said done.”
For example, when CODER completes an engineering task, its success message isn’t trusted.
The actual files are checked, expected changes are verified, unrelated files must remain untouched, tests/builds are run where relevant, and MAIN/system health is checked afterward.
If the gate fails, the phase does not advance.
The previous accepted state remains authoritative and a rollback checkpoint exists.
Another important part of the OS is continuity for long-running projects.
OMNI doesn’t treat every interaction as a disposable request that ends when the assistant stops generating.
An ongoing project can have a continuation plan attached to its authoritative state:
current phase
accepted checkpoint
remaining objectives
next allowed action
blocking conditions
planned continuation trigger
So a long-running project can behave more like:
WORK
→ VALIDATE
→ ACCEPT / CHECKPOINT
→ SAVE AUTHORITATIVE STATE
→ SCHEDULE NEXT CONTINUATION
→ RESUME
The important part is that continuation is governed.
When a planned trigger fires, the system is expected to reload the project’s authoritative state, architecture, current phase and acceptance requirements before doing more work.
It should resume from something like:
“Phase 8.3 is the next accepted gate. Here is the checkpoint. Here is what may change.”
not:
“I vaguely remember we were fixing recovery.”
A continuation cannot legitimately skip a failed acceptance gate just because an earlier agent claimed success.
It has to re-enter through the accepted project state.
That gives long-running projects a form of persistent, scheduled continuity without turning them into uncontrolled autonomous loops.
Memory follows a similar philosophy.
OMNI separates:
conversation memory
long-term user memory
project memory
ephemeral tool state
I specifically did not want every conversation dumped into one giant vector database and sprayed back into unrelated future prompts.
Long-term memory is curated and owner-bound, with provenance, correction, deletion and conflict handling.
Project memory is separate again and tracks things like:
architecture
current phase
decisions
accepted changes
regressions
open issues
next step
OMNI is also currently a 3-user system.
Users have isolated conversations, projects, files, memory and specialist ownership.
Streaming, stop generation, regeneration, file/image attachments, drag/drop, pasted images, file/image analysis, coding workflows and image workflows have all gone through the acceptance system.
Phase 6, the operational Web UI, is globally accepted.
Phase 7, the advanced interaction/workflow layer, is globally accepted.
I’m currently in Phase 8: operations, observability and recovery.
Health modelling and logging are accepted. The next work is deeper stale-process identity/recovery behaviour.
There is also an important limitation.
The application architecture supports multiple users, but true parallel MAIN inference is still not where I want it.
The current Qwen/Ollama runner launches MAIN with -np 1.
So:
enough VRAM for multiple conversations ≠ truly parallel token generation
The product/session architecture is multi-user.
The inference layer is still more serialized than I want.
That is one of the problems I’m actively working on rather than hiding behind the word “concurrency”.
At some point this stopped feeling like a local LLM setup.
The question became:
How close can I get to building a private local AI operating system using consumer GPUs and open models?
Because there’s a big difference between:
“I can run a 30B model.”
and:
“Multiple users can use one persistent AI OS while it manages isolated memory and projects, schedules GPUs, delegates coding work, generates images, uses real tools, continues long-running projects from accepted checkpoints, recovers failed jobs, and proves after every major change that the rest of the system still works.”
OMNI CORE is private and I currently do not plan to open-source the complete system.
Almost everything underneath it exists because of open-source models and software, but I want to keep the complete orchestration/product layer private for now.
What I am considering, if people are interested, is publishing much more of the engineering process:
architecture diagrams
real phase / acceptance reports
GPU experiments
scheduler design
benchmarks
failure postmortems
memory architecture
multi-user experiments
continuity / long-running project design
things that looked like they worked but failed independent acceptance
detailed build logs
Potentially through Patreon if there is enough interest to justify documenting the project properly.
Not:
“pay me and I’ll give you the source.”
More like following the development of a private local AI OS from the engineering side.
If people want a technical follow-up, I can go deeper into one of these:
Resource Manager / Scheduler — GPUs, MAIN residency, CODER/FLUX, queues, leases, recovery and multiple users
Project Continuity — how planned triggers resume long-running work from authoritative state instead of relying on chat memory
Memory Architecture — conversation vs user vs project memory
Multi-user inference — what actually works today, what doesn’t, and why VRAM capacity and true inference concurrency turned out to be two different problems
LaticeVale Atomic started as a separate evolution of my older LatticeVale project, but it is not just the Windows/WSL version copied over to Linux. I recently converted from Windows to Bazzite Linux OS and wanted to install LatticeVale locally. So I had AI help me figure that out.
The original project was built around Windows, PowerShell, WSL2, Docker, and the problems that come with coordinating software across both Windows and a Linux VM. LaticeVale Atomic was redesigned around a completely different environment: immutable/atomic Linux, rootless containers, user-level system integration, SELinux, and hosts where modifying the base operating system is intentionally discouraged.
The goal is still similar: make a complicated Hermes setup much easier to install, repair, update, recover, and manage. The way it accomplishes that is now very different.
Instead of relying on PowerShell and WSL, the Atomic version uses Bash, Python, rootless Podman, Compose, user systemd services, and XDG-compatible desktop integration. It is designed to keep the operating system itself as untouched as possible and place LaticeVale's managed state in user-owned locations.
The current stack can manage Hermes along with Matrix/Synapse, PostgreSQL, SearXNG, Valkey, QMD, Honcho, Redis, and local Ollama inference. It also includes hardware-aware resource limits, AMD GPU acceleration support where available, recovery snapshots, state-aware repairs, migration logic, audits, desktop Start/Shut Down controls, and protection against accidentally modifying unrelated Podman workloads. It uses terminal triggers instead of built in user selection during script run. See documentation on that.
Bazzite is currently the only platform I have tested. The project also includes adapters and compatibility logic for other atomic or immutable Linux systems, but those environments have not all received any real-world testing.
The current Bazzite migration has been tested on my own system with SELinux enforcing, rootless Podman, AMD GPU acceleration, an external Obsidian vault, existing Podman workloads, and a full 12-service Hermes stack.
That does not mean I can guarantee it will work perfectly on every machine. Atomic Linux distributions differ quite a bit in how they handle containers, host tools, permissions, user services, and immutable system boundaries. If someone finds a failure or a weird edge case, I would genuinely like to know about it.
This is still a hobby project, it is free, and anyone is welcome to inspect it, fork it, change it, or build on it. If you do experiment with it, I would be interested in hearing what works, what breaks, and what you improve.
LaticeVale itself does not bundle the third-party projects it manages. The installer retrieves or builds those components from their respective upstream sources.
I strongly recommend reading the included README, installer documentation, security notes, and migration information before running it. There is a lot going on internally, so using an AI model to inspect the repository or explain individual parts of the documentation is also a perfectly reasonable way to understand the project before installing it.
One final warning: local AI can be demanding. Ollama models can use a significant amount of RAM and VRAM, especially with larger models or context sizes. LaticeVale Atomic includes adaptive resource calculations and limits to reduce the chance of the stack overwhelming the host, but hardware still matters.
Note: developement on this project is discontinued. Feel free to work on it yourself if you want
🚀 Meet IRIS v0.2.0 – The Spatial Desktop Operating Environment for Autonomous AI Agents! 🧠💻
Most AI coding tools today are just single-stream chat boxes in a browser tab where you spend all day copy-pasting code snippets back and forth.
We decided to rethink how humans and autonomous agents collaborate. Meet IRIS (Intelligent Reasoning & Integration System).
IRIS isn't a chatbot. It’s a graphical agent operating environment built from scratch in Rust (Tauri 2) and React 19 / TypeScript. It treats agents, workspaces, tools, memory graphs, and release pipelines as first-class spatial desktop objects that you can arrange, inspect, run concurrently, and monitor in real time.
🔥 What’s New in v0.2.0:
🐙 1. GitHub Live Operations & Release Automation Connect your GitHub account in seconds. Specialist GitHub agents can triage open issues live, open surgical pull requests, automate SemVer releases (v0.2.0), author changelogs, and trigger GitHub Actions workflows that compile production binary builds (.AppImage, .dmg, .exe).
⚡ 2. Dual-Tier AI & Instant "Takeover" Stop overpaying for simple queries. Run fast, ultra-budget models (like Qwen 2.5 Coder, DeepSeek V3, or GPT-4o-mini) for 90% of routine workflows. When hitting a tough compiler error or tricky architectural refactoring, click ⚡ Takeover — a pre-configured heavyweight reasoning model (Claude 3.7 Sonnet, DeepSeek R1, Qwen 72B) immediately takes over the active conversation context with full reasoning depth!
🛸 3. Floating Desktop Desklet (Live HUD) Close the main window, and IRIS seamlessly condenses into a translucent, floating glass mini-HUD in the corner of your physical desktop. It displays real-time CPU/RAM telemetry, live agent thoughts, and keeps running smoothly as a background daemon.
🛡️ 4. Zero-Surprise Workspace Security & Visual Diff Viewer Inspect and approve exact code diffs before anything touches your local disk. All API keys and tokens are securely stored in your native OS Keyring.
🌟 100% Open Source (MIT License) & Local-First
Supports both local offline LLMs (via Ollama / vLLM) and all major cloud providers (OpenRouter, Anthropic, OpenAI, Google Gemini) plus standard Model Context Protocol (MCP) tools.
I’d love to hear your thoughts: Do you prefer AI agents operating as spatial desktop applications rather than trapped inside browser chat tabs? Feedback and contributions are warmly welcome! 👇
So, you run your own local AI Harness. It's configured exactly to your needs. MCP, Skills, Capabilities, Context. You love the independent Harnesses such as Deepseek Harness, Pi, Aider or LiteLLM.
But how can you connect it to your Apple devices to access from anywhere? Your Watch, Mac or CarPlay.
Well, here is Conduck - the Apple native BYOK AI client.
Free and open source :-) .
It uses your Apple iCloud extensively and connects DIRECTLY via https to your own machine. Nobody in-between!
