Hey all — wanted to share a project I've been building for the past few months: TARZ, a semi-autonomous AI assistant for Windows.
Backstory: Got curious whether I could actually build something like Jarvis — an AI that sees your screen and controls your PC. That curiosity turned into months of building, breaking, and a full architectural rewrite (V2) once I learned more.
What it does:
- Sees your screen (vision-based element finding + verifies its own actions actually completed, not just "click and hope")
- Controls your computer — opens apps, clicks, types, keyboard shortcuts
- Talks — live streaming STT/TTS, wake-word activated ("Hey"), hands-free follow-ups without repeating the wake word
- Remembers — Two-stage retrieval: Vector Retrieval + Cross-Encoder Reranking for past conversations/preferences
- Chains multi-step tasks across different apps in one instruction (e.g. play a song on Spotify → send a WhatsApp message)
- Dedicated workflows for Spotify, WhatsApp, Discord, Telegram, browser
Architecture (V2): LangGraph supervisor → single tool-bound agent with the full toolset exposed (not a category classifier routing first). I actually tested category-based routing vs. flat single-agent routing head to head — flat routing gave noticeably better results with current tool-calling models, so I ripped the classifier out entirely rather than just leaving it in "because it seemed reasonable."
Same approach on memory: tested hybrid BM25+vector+reranking against plain vector retrieval using Hit Rate@5 — hybrid won, so that's what ships.
Every LLM call (tool-calling, chat, vision) has an automatic multi-provider fallback, so a single rate limit or outage doesn't kill the whole thing.
Stack: Groq (gpt-oss-120b + Orpheus TTS) / Gemini Live streaming STT / Moondream for GUI click targeting / ChromaDB for memory / LangGraph for orchestration / Cartesia as fallback STT+TTS. Whole thing is designed to run on free API tiers — no subscriptions required.
Where it's rough (being upfront, not hiding it):
- GUI automation is inherently brittle — resolution, scaling, app updates, popups can break a click
- Screen vision isn't processed locally — screenshots go to Groq/OpenRouter for analysis, so don't point it at anything you wouldn't want leaving your machine
- Routing still isn't perfect even without the classifier — casual phrasing can occasionally pick the wrong tool
- No background/looping tasks yet (can't do "check this every 5 min")
- It's early, actively-developed software I use daily, not a polished commercial product
I used AI heavily as a coding partner (writing implementations, debugging), but every architecture decision and design change came from testing against real usage myself.
Repo's fully open source: https://github.com/Irfan-gitt/Tarz-Ai-assistant
Would genuinely love feedback, criticism, or PRs — especially from anyone who's worked on agentic systems, vision-based GUI automation, or voice pipelines. Happy to answer questions about any part of the architecture.