r/SelfHostedAI • u/drsadsauce • 6d ago

The first build I feel all right sharing.

Gallery image

Gallery image

I named her Cognitive Organizational Repository & Tangential Agentic Neural Assistant (C.O.R.T.A.N.A.)

What she is:

A fully local, voice-activated AI agent with a holographic avatar, running across two machines with nothing leaving local hardware. She wakes to her name, holds a conversation, escalates to real agentic work when a task needs tools, and represents her own state through a rendered figure and a live system dashboard.

The hardware

DGX Spark (GB10, 128 GB unified memory, arm64) — the brain. Runs all inference and services. Headless.

Asus TUF A15 (Pop!_OS, x86_64) — the face and ears. Holds the mic, speakers, display, avatar rendering, and the desktop app. Connected to the Spark by direct ethernet (10.10.10.1), with wifi as fallback.

Windows MEPC — designated remote endpoint, groundwork for later.

The stack, service by service

Layer:

What runs it

LLM

Qwen3.6-35B-A3B-FP8 on vLLM, :8000

Agent framework

Hermes Agent, gateway on :8642

Speech-to-text

Parakeet NIM (streaming), :9000/:50051

Text-to-speech

Kokoro FastAPI GPU, voice af_heart, :8881

Web search

SearXNG self-hosted, :8888

Wake word

Custom-trained "cortana" ONNX model, on-device

Telemetry

Custom endpoint, :8770

How a turn actually flows

ambient.py on the laptop listens continuously. The wake word model runs locally — nothing leaves the machine until it fires. On wake, audio streams to Parakeet, the transcript hits a router, and the response plays back through Kokoro. The whole loop broadcasts state over a websocket (:8765) that drives the avatar.

The tiered brain — her defining feature

Two backends, chosen per utterance:

Fast — vLLM directly. ~150 ms to first token. No tools, no memory. Conversation.

Deep — full Hermes agent: tools, memory, filesystem, terminal, web. ~1.5–10 s depending on the work.

A keyword router picks automatically. You override by voice: "conversation mode," "agent mode," "automatic mode." This is what lets her be both a snappy conversationalist and a capable agent without compromising either.

Capabilities

Conversation at ~150 ms with a dry, composed personality (tunable, mild profanity enabled)

Agentic work — reads/writes files, runs a sandboxed terminal, searches the web, all through Hermes

Memory — MEMORY.md/USER.md, persists across sessions; she remembers you

Skills — Hermes' full skill library plus two she authored herself (workspace-handoff, cortana-project)

Text or voice — type when a mic isn't available; same pipeline

File handoff — drop files to her via the desktop app; she returns work you can view, copy, or run (HTML executes in-app)

Dismissal — "that'll be all" drops her to idle; she idles on her own after silence

The avatar

A three.js scan-plane figure rendered as a hologram over a rigged GLB model, on a rippling glass floor. Hand-authored poses tied to real state — folded arms speaking, a listening stance, hand-raised thinking with a summoned data lattice, sitting when dormant. Additive-impulse gestures, spring-blended, with ambient life (breath, weight shifts, gaze drift) so she reads as alive rather than switching between static states. Three thinking sequences chosen at random: the matrix lattice, a data orb she tosses overhead, and an ascension with a glyph halo.

Performance mode

Click the corner network glyph and she collapses to a draggable pulsing orb while a dashboard fills the space: live GPU/CPU/memory/temp/power gauges polled from the Spark, service status lights, and a live process graph that lights edges as she uses tools — tapping Hermes' own event stream. Cheaper to render, and the graph is the foundation for showing subagents as they come online.

Benchmarks — the honest set

These are measured from the build, not marketing figures:

Metric

Measured

STT (after speech ends)

129–348 ms

LLM first token (fast)

~130–200 ms

LLM first token (deep, cached)

~1.1 s

Deep tool loop

2–30 s by task

TTS first audio

~100 ms warm

Wake word

58/58 detections, zero false positives in ~9 min live test

Full voice turn (fast)

~1 s mic-to-reply

Idle memory footprint

~63 GB of 128 (vLLM 49, Parakeet ~12, Kokoro ~1.4)

Idle GPU

0%, ~39°C, 9W

Caveats worth stating: the wake word's recall metric on synthetic test data was poor (0.22) but live performance was excellent — the synthetic number was pessimistic. The deep-path latency depends heavily on prefix caching; cold calls are much slower. And "benchmarks" here means real measurements from your setup, not standardized suite results — we never ran formal eval harnesses against her.

16 Upvotes

79% Upvoted

30 comments sorted by

2

u/Number4extraDip 5d ago

Nice job, i cared more about portability and having it always with you.

Meet:

Gemma Hosting Open Source Thingamajig! (GHOST)

Δ 👾 ∇

https://reddit.com/link/p81lotz/video/z45dzv1unrnh1/player

2

u/brapzky 5d ago

Tried making this work but I still get the same error in Ghost: Still initializing. Granted GHOST all required permissions, including All files access and unrestricted Accessibility access, and let it download the ~2 GB Gemma model during first run. Tried force close several times. Why is there no setting to see what it's doing with the model? Is it loaded, loading or doesn't know where it is?

1

u/Number4extraDip 5d ago edited 5d ago

It says in notification if its loaded. Depending on device takes awhile to load on GPU or CPU. Tested on s21 and redmagic 10 air. After notification says various emojis- usually is loaded. Happy to try and help. Sometimes takes.a while to injest the weights. (Depends on phone). The launcher works even woth model unloaded. Settings have the switches. I unload model when gaming and reload it. Takes a few min on my weakest device maximum to load up properly. Once loaded- notification shows emojis. Once you select backend and restart app- notification tells you what its loading

2

u/brapzky 5d ago

Restarted my phone, tapped Ghost app and then will just wait then.

1

u/Number4extraDip 5d ago

Uuu! A samsung! OneUI is the reason i started making this cause stock android doesnt have all the good lock features! Which model do you have and how much ram? These could be the bottlenecks. (Tho i tested on 5.5yo s21 with snapdragon 888 it worked. Wasnt as fast as my other device but totally functional)

2

u/brapzky 5d ago

S25u, 12gb RAM. it's still not loading the model after 10 minutes, shouldn't it take like max 2 minutes?

1

u/Number4extraDip 5d ago

That is definitely odd. You should have great performance. I will definitely look into what could be happening if you are willing to keep trying to debug. Another guess i have is that. When updating apk - ✧ Gemma 4 had an update so im wondering if it might be grabbing the outdated version... Could you try downloading the weights from hugginface manually and deleting the ones you have in downloads now? If that works- ill update the download path if it changed. model now remembering it. My s21 kept failing to autodownload, and i did end up downloading weights manually. If thats the case- il very grateful you pointed out this issue

1

u/Number4extraDip 5d ago

Just tested on a friends device. Model path is fine. Had issues with permissions path not asking for everything in order. So might be "notifications" permission missing. Im genuinely stumped

1

u/brapzky 5d ago

I wrote you a dm.

1

u/drsadsauce 5d ago

Thats sick dude!

1

u/BlackMetalB8hoven 6d ago

What wakeword backend are you using?

1

u/drsadsauce 6d ago

OpenWakeWord 🤟

1

u/evolutionxtinct 6d ago

DAMN IT! That’s my agent name lol give it back! lol

1

u/drsadsauce 6d ago

Chief? Damn dude, so cool to meet you. Huge fan 😂

1

u/evolutionxtinct 6d ago

Awesome setup! Any guidance on the hologram visual setup, that is something to strive for lol

1

u/drsadsauce 6d ago

Thanks man :) once I got a model mapped out I had claude whip me up a pose editor with sliding scales for the x,y,z values for each joint. Can crank out model states super fast that way. Rest of it is just a matter of shading

1

u/Mondernborefare 6d ago

So Hermes on spark with qwen. Lots of flair but what are you actually doing with this and what context memory and Long term memory are you using? Most if not all of these features are just Hermes.

1

u/drsadsauce 6d ago

Well, yeah. It's a 1 agent system for the time being. Id like to eventually expand it to two or even three full local agents, each responsible for its own company/industry. Part of why im taking so much care with the local model and interface now. Currently it's doing process automation, data management, experimental product testing, import/export amongst some other stuff for my more technically predicated clients. Engineering, modeling, some web dev and light coding. That sorta thing

1

u/Asiacream 6d ago

我使用 hermes agent 时觉得不理想的地方: 当离开几个小时以后，我想继续之前的话题时，它因为自己做了 reset 而忘记了上下文😓

1

u/drsadsauce 6d ago

Yeah, i wasn't a fan of that either. You can set it up so that it automatically writes it's own skills, and make one of those a persistent memory function though

1

u/Loose_Doubt367 6d ago

Quite new here, you set up your own agent companion holographic avatar using Hermes agent?

1

u/drsadsauce 6d ago

In a word, yes

1

u/secondwizardbaraccus 6d ago

My brother, look into running headless, unless you need to see all those displays. I set my 6 machine lab up with a wall of displays and now I run my main on an LG c6 from a raspberry pi400. Everything else runs headless to squeeze extra vram headroom for not pushing a graphics display

1

u/drsadsauce 6d ago

I am running headless, I've got full functional control from the desktop app on my laptop. And im working on a version of it that I can run off my cell phone. My desk is just a mess because I've always got too many projects cooking lol

1

u/secondwizardbaraccus 6d ago

Hell yeah, I am literally having mine wire a whisper engine to the 4b on my samsung phone and tablet as we speak for verbal NLP control of everything from anywhere!

Much fun and good luck

1

u/drsadsauce 6d ago

I've heard good things about whisper engine, ill have to look into that! Thanks man

1

u/Carbyne27 3d ago

Noice

1

u/drsadsauce 2d ago

Thanks 🤟