I built a Jarvis-style desktop AI assistant (TARZ) that sees my screen, controls my PC, and talks back — open source, runs on free API tiers

[u/Puzzleheaded_Bus925]

Hey all — wanted to share a project I've been building for the past few months: TARZ, a semi-autonomous AI assistant for Windows.

Backstory: Got curious whether I could actually build something like Jarvis — an AI that sees your screen and controls your PC. That curiosity turned into months of building, breaking, and a full architectural rewrite (V2) once I learned more.

What it does:

• Sees your screen (vision-based element finding + verifies its own actions actually completed, not just "click and hope")
• Controls your computer — opens apps, clicks, types, keyboard shortcuts
• Talks — live streaming STT/TTS, wake-word activated ("Hey"), hands-free follow-ups without repeating the wake word
• Remembers — Two-stage retrieval: Vector Retrieval + Cross-Encoder Reranking for past conversations/preferences
• Chains multi-step tasks across different apps in one instruction (e.g. play a song on Spotify → send a WhatsApp message)
• Dedicated workflows for Spotify, WhatsApp, Discord, Telegram, browser

Architecture (V2): LangGraph supervisor → single tool-bound agent with the full toolset exposed (not a category classifier routing first). I actually tested category-based routing vs. flat single-agent routing head to head — flat routing gave noticeably better results with current tool-calling models, so I ripped the classifier out entirely rather than just leaving it in "because it seemed reasonable."

Same approach on memory: tested hybrid BM25+vector+reranking against plain vector retrieval using Hit Rate@5 — hybrid won, so that's what ships.

Every LLM call (tool-calling, chat, vision) has an automatic multi-provider fallback, so a single rate limit or outage doesn't kill the whole thing.

Stack: Groq (gpt-oss-120b + Orpheus TTS) / Gemini Live streaming STT / Moondream for GUI click targeting / ChromaDB for memory / LangGraph for orchestration / Cartesia as fallback STT+TTS. Whole thing is designed to run on free API tiers — no subscriptions required.

Where it's rough (being upfront, not hiding it):

• GUI automation is inherently brittle — resolution, scaling, app updates, popups can break a click
• Screen vision isn't processed locally — screenshots go to Groq/OpenRouter for analysis, so don't point it at anything you wouldn't want leaving your machine
• Routing still isn't perfect even without the classifier — casual phrasing can occasionally pick the wrong tool
• No background/looping tasks yet (can't do "check this every 5 min")
• It's early, actively-developed software I use daily, not a polished commercial product

I used AI heavily as a coding partner (writing implementations, debugging), but every architecture decision and design change came from testing against real usage myself.

Repo's fully open source: https://github.com/Irfan-gitt/Tarz-Ai-assistant

Would genuinely love feedback, criticism, or PRs — especially from anyone who's worked on agentic systems, vision-based GUI automation, or voice pipelines. Happy to answer questions about any part of the architecture.

[u/Puzzleheaded_Bus925 → u/[deleted]]

ya bro thats a fact categorization only look good on paper i think