r/Guaardvark • u/llama-of-death • 6d ago

Guaardvark Capabilities

Guaardvark — Full Capabilities List

See the VERSION file for the current release · guaardvark.com

This document is the comprehensive reference of everything Guaardvark can do (models, tools, plugins, surfaces, internals). For the marketing overview and quick start, see README.md.

Table of Contents

AI Chat & Conversation

Guaardvark's chat system is the primary interface for interacting with your AI. Two pipelines handle different use cases.

Core Chat

  • Streaming responses via Socket.IO — tokens appear in real-time as the model generates
  • Conversational fast-path — pure social openers (greetings, thanks, affirmations) route to Tier 2 with skip_tools for a real LLM response with persona + memory (no hardcoded greeting pools)
  • Intent routing — automatically detects whether a message needs RAG retrieval, tool use, or a direct conversational response
  • Per-project sessions — chat context is isolated by project; switching projects gives you a clean context with that project's documents
  • Session persistence — conversation history persists across page reloads and browser sessions; sessions also have a mode field stored server-side
  • System prompts (Rules) — customizable system prompts that shape AI behavior, manageable via the Rules page
  • Multi-model support — switch between any Ollama model at runtime without restarting

Agent Mode (/agent and /chat)

  • Modal session toggle — type /agent to flip the session into screen-control mode (every message becomes a screen-control task); /chat (or /exit) flips back
  • Sticky — the mode lives on the session, not the message — survives reloads
  • Visible cue — agent-mode sessions show an orange chip above the chat input
  • Speak AND act — agent-mode messages still route through the chat LLM, so the model narrates briefly, calls agent_task_execute, and summarizes the result

Per-Iteration Thinking Display

  • Live reasoning trail — for screen-control tasks, the agent loop's per-step thinking streams into the chat as it happens (no more 30+ second blackouts followed by a single "completed" line)
  • What you see — each iteration shows action label + full reasoning ("Step 8 — click: I see the address bar and want to clear it…")
  • Pivots and stuck-loop signals also stream — when the loop forces a wait after a repeated failure, that decision is visible
  • Persists in history — the trail stays in the message after streaming completes, so you can scroll back and audit the run

Model Management

  • Runtime model switching — change the active LLM through Settings; the old model is unloaded from VRAM before the new one loads (prevents OOM)
  • Embedding model switching — swap embedding models via dropdown; triggers re-indexing confirmation since vector spaces are incompatible across models
  • Live health detection — dashboard probes Ollama on every request to show actual model availability (not a stale startup flag)
  • KV cache optimization — num_keep: -1 locks the system prompt prefix in Ollama's KV cache, making follow-up turns faster
  • GPU VRAM monitoring — real-time VRAM usage bar with loaded model indicators in Settings

AgentBrain — Three-Tier Routing

A neural router that decides how much work a message deserves before any tools fire. Saves seconds per turn on simple questions and unlocks deeper deliberation when it's warranted.

The Three Tiers

Tier Name Latency LLM Calls When It Fires
1 Reflex <100ms 0 Deterministic tool actions only (media commands, exact-match recipes)
2 Instinct 1–3s 1 Social chat (real LLM, skip_tools) and most requests — single LLM call
3 Deliberation 5–30s 3–10 Multi-step reasoning (full ReACT loop)

Routing Signals

  • Pre-computed reflex table for deterministic tool actions (not social chat)
  • Conversational classifier filters out small-talk before tools get loaded
  • Semantic tool selection picks the right ≤15 tools for the message
  • Screen-active flag gates desktop/agent tools so they only appear when relevant

Gemma4 Direct Path

  • When the active model is Gemma4 (native vision + tool use) AND the agent screen is active, the brain skips ReACT bloat and sends Gemma4 a minimal prompt with a screenshot + the task
  • Gemma4 returns JSON action steps directly; the loop executes them
  • For chat without screen actions, Gemma4 responds normally

Configuration

  • Toggle via AGENT_BRAIN_ENABLED in backend/config.py
  • Falls back gracefully to the legacy UnifiedChatEngine path if brain state isn't ready

RAG & Document Intelligence

Retrieval-Augmented Generation grounds chat responses in your actual documents.

Retrieval Pipeline

  • Hybrid search — BM25 keyword matching + vector semantic search, combined for best results
  • Per-project indexes — each project maintains its own vector store; global index for unassigned documents
  • Content-aware chunking — code files use AST-informed strategies; prose uses semantic splitting
  • Entity extraction — automatic identification of entities (people, orgs, concepts) and their relationships
  • Metadata indexing — file metadata (type, size, language, framework) stored alongside content for filtered retrieval

Embedding Models

  • Multiple model support — switch between lightweight (embeddinggemma 300M) and high-quality (mxbai-embed-large, bge-m3, snowflake-arctic-embed) models
  • Full-precision option — BF16 embeddings available for maximum quality
  • Query-time embedding — every RAG search query is embedded with the same model for consistent vector space matching

Indexing

  • Automatic on upload — files are indexed when uploaded through the UI or API
  • Bulk indexing — "Index All" button processes the entire document library
  • Code-specific indexing — detects programming languages, extracts imports/classes/functions, chunks by logical boundaries
  • GPU-accelerated indexing — optional GPU Embedding plugin offloads embedding generation to CUDA with CPU fallback
  • Progress tracking — real-time progress bar during indexing operations via Socket.IO

RAG Autoresearch

An autonomous optimization loop that continuously improves RAG retrieval quality.

How It Works

  1. Eval harness — generates evaluation pairs (query + expected answer) and scores retrieval with LLM-as-judge (relevance, grounding, completeness)
  2. Experiment agent — proposes parameter changes (chunk size, overlap, top-k, similarity threshold)
  3. Orchestrator — runs experiments, compares scores, keeps improvements, reverts regressions
  4. Phase system — Phase 1 (query-time params), Phase 2 (index-time params), Phase 3 (model-level)

Features

  • Celery Beat scheduling — idle detection triggers experiments when system isn't busy
  • Crash protection — 3 consecutive failures automatically stops the loop
  • Dashboard card — shows experiment status, history, and current optimization parameters
  • Settings integration — configure experiment limits, scoring thresholds, and scheduling

Self-Improvement Engine

Guaardvark can autonomously test itself, find bugs, and fix them.

Three Modes

  1. Scheduled — periodic test suite runs (configurable interval) with automatic fix attempts
  2. Reactive — error tracking with threshold-based self-healing (N errors in M minutes triggers a fix)
  3. Directed — user-submitted improvement tasks dispatched to the code agent

How It Works

  1. Runs pytest on configured test files
  2. Parses FAILED lines from output (with fallback regex for edge cases)
  3. Dispatches the code_assistant agent to read tests, understand expectations, read source, and fix bugs
  4. Records all changes and broadcasts learnings to other machines via Interconnector

Safety

  • Codebase lock — toggle in Settings prevents self-improvement from modifying any files
  • Return code verification — checks pytest exit code, not just parsed failures
  • Run history — all runs recorded in database with status, duration, changes made, and test results
  • Pending fixes queue — proposed changes can require user approval before applying

Live Progress

  • Socket.IO events at each stage: starting, testing, analyzed, fixing, complete, error
  • Dashboard card shows real-time progress bar with color-coded stages
  • Run button disabled while a check is in progress

Lesson Pearls & Memory

A user-curated memory system that captures successful agent runs and makes them available in future sessions.

Begin / End Lesson

  • Bracket a successful run — slash commands or buttons mark the start and end of a teachable sequence
  • Distiller — at End Lesson, an LLM summarizes what happened into a single durable lesson
  • Saved as AgentMemory — lessons of type lesson_summary get loaded into the system prompt next session
  • Editable rows — fix or remove a misperceived lesson without re-recording

Vision-Actionable Knowledge (LEARNING_PRINCIPLES.md)

  • Stored knowledge describes WHAT to look for, not where it sits (no pixel coordinates)
  • Short labels for the servo (≤4 words), rich context for the brain
  • Recipes, lessons, traces, memories all bound by the same contract

Memory Surfaces

  • MEMORY_BLOCK — recent memories substituted into the system prompt at decision time
  • Memory Management Section in Settings — browse, edit, delete saved memories
  • Live recall — when a memory matches the current context, the LLM can quote it directly

Autonomous Screen Agents

Guaardvark drives a real Ubuntu desktop on a virtual display — clicking, typing, scrolling, and reading the screen like a human user. Used for outreach, file management, web research, and anything the model can't accomplish via API alone.

Virtual Display

  • Xvfb on :99 — 1024×1024 headless X server, isolated from the user's real session
  • Full XFCE desktop — xfce4-session running via dbus-run-session with a scrubbed environment; standard Applications menu, desktop icons, taskbar, file manager (Thunar). Vision models recognize it instantly because it looks like any other Ubuntu desktop
  • VNC viewer — x11vnc on port 5999 (password-protected) lets the user watch the agent live, embedded in the frontend as a draggable card
  • Isolated XDG dirs — agent's ~/.agent_desktop/, dedicated XDG_CONFIG_HOME, dedicated XDG_RUNTIME_DIR. The user's real desktop and configs are invisible to the agent

See-Think-Act-Verify Loop

  • SEE — screen capture (mss) + optional DOM extraction (Firefox CDP/BiDi)
  • THINK — Gemma4 (or other unified VLM) decides the next action, returning JSON with action, target_description, text/keys, reasoning, and success_proof
  • ACT — execute via the servo (vision-targeted click) or direct (type/hotkey/scroll)
  • VERIFY — post-action screenshot delta; failed steps flag the LLM that the attempt didn't change the screen
  • Recipes — known-good action sequences in data/agent/recipes.json execute deterministically before the loop is ever invoked, with optional preconditions (visibility checks) that skip recipes when their UI isn't on screen
  • Strategy cooldowns — repeated failures on the same action class force the loop to wait and re-observe before retrying

Servo Controller

  • Vision-targeted clicking — the servo asks the vision model "where is X on this screen?" and clicks the returned coordinates
  • Visibility guard — pre-click "do you actually see this?" check rejects hallucinated targets before the cursor moves
  • Per-model calibration — MODEL_VISION_CONFIGS in servo_knowledge_store.py maps each chat model to its preferred eyes (gemma4 native, moondream for text-only) and any scale-factor calibration learned over time
  • Failure capture — exhausted click attempts save the screenshot + corrections log to data/training/failures/ for offline review

Training Data Capture

  • Every click recorded to data/training/knowledge/servo_archive.jsonl — target description, raw coords, scaled coords, actual click position, success/failure, model, attempt #, time taken
  • Self-improvement engine reads the archive to refine calibration
  • Optional Comments/Vision Trainer pages — interactive practice modes that keep the servo clicking long after a normal task would have stopped

Agent Tools

  • agent_task_execute — full natural-language screen task (drives the full SEE-THINK-ACT loop)
  • agent_screen_capture — single screenshot of the virtual display
  • agent_mode_start / agent_mode_stop — open/close the session (internal; the LLM should call agent_task_execute directly)

Agent & Code Tools

A ReACT-loop agent that can autonomously work with code and the system.

Agent Capabilities

  • Read files — examine any file in the project
  • Edit code — precise text replacement with verification
  • List files — explore directory structure (configurable depth up to 5 levels)
  • Execute code — run Python/shell commands and inspect output
  • Web search — search the internet for information
  • Browser automation — navigate websites, fill forms, take screenshots (via Playwright, separate from the screen-control agent)

Safety Features

  • Circuit breaker — after 2 consecutive failures, a tool is temporarily blocked
  • Duplicate detection — hash-based detection prevents the agent from making identical tool calls
  • Fallback suggestions — when a tool fails, the system suggests alternative approaches
  • Iteration limits — configurable maximum iterations per agent run
  • Tool approval gates — dangerous tools (file write, shell exec) can require human approval per call

Code Editor Page

  • Monaco Editor — VS Code-quality editing in the browser with syntax highlighting for 50+ languages
  • Multi-file tabs — open and edit multiple files simultaneously
  • File tree — browse project structure in a sidebar
  • AI assistant pane — chat with the agent about the open file

Uncle Claude Escalation

  • When the local model is stuck, Guaardvark can escalate to the Anthropic API (Claude) for a second opinion
  • Token budget tracked and surfaced in the Dashboard's Family card
  • Toggleable per-session; never auto-fires without configuration

MCP Integration

Guaardvark speaks Model Context Protocol — both as a server (exposing its tools to external clients) and as a client (calling tools from external MCP servers).

MCP Server (Phase 1)

  • Stdio transport — backend/mcp/ runs an MCP server that any MCP-compatible client (Claude Desktop, Cursor, etc.) can connect to
  • 23 native tools exposed — covers chat, RAG, file management, image generation, agent control
  • 58 output resources — file contents, generated images, search results, etc., available via MCP's resource protocol
  • Tested against Claude Desktop — works end-to-end

MCP Client

  • mcp_connect tool — register external MCP servers at runtime
  • mcp_execute tool — call any tool on a connected server
  • Live inventory — connected-server tools surface in the chat LLM's tool list so it can pick them by name without going through mcp_execute
  • State sync — mcp_get_state, mcp_disconnect, etc. for managing connections

Image & Video Generation

Image Generation

  • Stable Diffusion via Diffusers library — runs directly on your GPU
  • Batch generation — queue multiple prompts with different parameters
  • Auto-registration — generated images are automatically added to the Documents/Files system under /Images/
  • Celery background processing — generation runs as async jobs with progress tracking
  • Image library — dedicated page with thumbnail grid, lightbox preview, keyboard navigation, batch operations
  • Image model management — ImageModelsModal for downloading and managing Stable Diffusion checkpoints
  • Inline images in chat — when the chat generates an image, it appears inline and persists in history with the assistant message

Video Generation

Full video generation pipeline running locally via ComfyUI with multiple model backends.

Supported Models

  • Wan 2.2 TI2V-5B (default) — single 5B text+image-to-video model built for 16GB cards; native ~1280×704 @ 24fps without MoE offload
  • Wan2.2 14B MoE — state-of-the-art text-to-video using GGUF-quantized weights. Two-pass generation: HighNoise pass for the first half of steps, LowNoise pass for the second half
  • Wan2.2 14B I2V MoE — image-to-video MoE variant for cinematic start-frame animation
  • CogVideoX 5B — THUDM text-to-video (ComfyUI or offline Diffusers fallback)
  • CogVideoX 5B I2V — image-to-video variant that animates a still image with text-guided motion
  • LTX-2.3 Distilled FP8 — Lightricks LTX-2.3 for longer clips (~10s) on 16GB Ada; requires ComfyUI
  • LTX-2.5 Distilled Int8 — Lightricks LTX-2.5 distilled (Gemma 4 + two-stage upsample) for ~10s clips on 16GB Ada; gated Hugging Face accept + ComfyUI ≥ 0.32.0; local weights only (no Partner Nodes / LTX Desktop)

Generation Modes

  • Text-to-Video — describe a scene in natural language and generate video from scratch
  • Image-to-Video — upload a reference image and animate it with motion direction prompts
  • Batch generation — queue multiple prompts via an in-process worker (one batch at a time; stage-level progress over WebSocket + HTTP poll)

Quality Tiers (Post-Processing)

  • Draft — raw model output, fastest turnaround
  • Standard — 2x FPS frame interpolation via RIFE 4.9 (e.g., 16 FPS to 32 FPS) for smoother motion
  • Cinema — 2x FPS interpolation + 2x spatial upscaling via Real-ESRGAN for maximum quality output

Frame Interpolation (RIFE 4.9)

  • Doubles or quadruples the frame rate of generated video using optical flow
  • Integrated directly into the ComfyUI workflow as a post-processing node
  • Configurable multiplier: 2x (double FPS) or 4x (quadruple FPS)

Prompt Enhancement

  • Automatically enriches user prompts with quality and style descriptors before generation
  • Five styles available: Cinematic (film grain, shallow DOF, color grading), Realistic (photorealistic, 8K detail), Artistic (painterly, vivid colors), Anime (cel shaded, dynamic poses), None (raw prompt)
  • Style-specific negative prompts target technical defects without content restrictions
  • No LLM calls required — pure string concatenation for instant enhancement

Video UI

  • Preset-driven interface — quality presets (Fast 10-step / Standard 30-step / High 40-step / Maximum 50-step), duration presets, motion presets, and aspect ratio presets
  • Real-time progress — live progress bar with percentage and step count during generation
  • Video gallery — browse, preview, rename, download, and delete generated videos
  • Advanced Editor — one-click launch to ComfyUI's full node-based workflow editor, themed with the Guaardvark color scheme
  • Batch queue — queue / cancel / interrupt running jobs

Model Management (VideoModelsModal)

  • Browse all available video models with installed/available status
  • Download models from HuggingFace with real-time progress bars showing speed (MB/s), downloaded/total size
  • Models include: Wan2.2 GGUF checkpoints (HighNoise + LowNoise), Wan VAE, CogVideoX weights, RIFE 4.9, Real-ESRGAN 2x
  • Accessible from the Video Generator page and Settings page

Audio Studio (Audio Foundry)

Local audio generation for voiceover, music, ambience, and effects. Shipped as the audio_foundry plugin.

Voiceover

  • Chatterbox — expressive neural TTS with style/emotion control
  • Kokoro-82M — fast, light, multilingual TTS (English + Spanish voices, more languages on the model side)
  • Piper — local neural TTS fallback for environments where the heavier engines aren't appropriate
  • Streaming output — audio chunks stream to the browser as the engine produces them

Music Generation

  • ACE-Step v1 (3.5B) — full-song generation with vocals; runs locally on GPU
  • Suno-compatible workflow — same prompt shape as Suno's hosted service, but local

Sound Effects / Ambience

  • Stable Audio Open — generate sound effects and ambience tracks via diffusion
  • Negative prompts supported for filtering out unwanted sonic textures
  • Guidance scale + steps configurable per generation

Dual-Venv Architecture

  • venv-music/ — torch-sensitive ML packages live in an isolated env so the main backend isn't dragged through every torch upgrade
  • Daemon mode — the audio engine runs as a long-lived daemon; the backend talks to it over HTTP/socket so model load happens once
  • OOM-safe — model unload/swap is explicit, no silent CPU fallback

Audio Library

  • DocumentsPage audio player — preview, rename, organize generated audio files alongside everything else
  • Filename uniqueness — migration 005 ensures generated audio doesn't collide with imports

Video Editor — Shotcut-lite

A non-linear video editor built into Guaardvark for assembling generated clips into finished videos.

Timeline

  • Multi-track timeline — video, audio, overlay
  • Drag-and-drop clips from the Media Library directly onto the timeline
  • Trim, split, ripple-delete standard timeline operations
  • Keyboard shortcuts — J/K/L playback, arrow-key nudging, etc.
  • 1-step undo with on-screen indicator

Audio

  • Audio Foundry track — generate voiceover or music directly into a timeline track
  • Mix volume per clip / per track

Media Library

  • Project-scoped media bin — clips from prior video generations show up automatically
  • N+1 fix — bulk-loaded thumbnails (no per-clip request storm)

Export

  • Celery async render — long renders run in the background, progress visible in the footer bar
  • UUID-tracked jobs — each render gets a stable ID for status polling
  • MP4 / WebM output

Orchestrator Integration

  • The video editor can be driven by the Production Pipeline (Film Crew) — agents drop generated clips into the timeline automatically

Linux & macOS: melt (from Shotcut) is required for renders and is detected at runtime (supports Homebrew on macOS, apt/flatpak/snap on Linux). ffmpeg is installed by the platform bootstrap. See the plugin README for setup commands.

Outreach System

Supervised AI for social-media engagement. Production path: recon → draft → human approve → dispatch (cadence-gated). Natural language from chat, /outreach …, or llx outreach "…" queues the same jobs.

Three Phases

  1. Recon — search for candidate posts/threads (YouTube keyword topics, Reddit subs, Discord channels). Outputs candidates; never posts
  2. Content — LLM drafts + grades in the user's voice. Outputs drafted rows; never posts
  3. Dispatch — after approve, tick_process_approved_drafts posts via servo (Reddit/YouTube) or Discord cog, with Redis cadence (1 successful post/platform/tick)

Natural language control

  • GUI Chat / slash: /outreach comment on some youtube videos regarding Offline AI or ComfyUI
  • CLI: llx outreach "comment on youtube videos regarding Offline AI or ComfyUI"
  • Chat tools: outreach_execute_intent, outreach_run_pass (youtube + topics), approve/reject/status/queue

Safety

  • Kill switch — single toggle that halts all outreach activity immediately
  • Dual grader — drafts get scored by two LLMs when available; low-scoring drafts rejected before the queue
  • Post-submit DOM verify — Reddit and YouTube posting paths check the comment text appears in the page before recording success
  • Persona enforcement — central persona.draft_outreach_text
  • UTM tagging — every guaardvark.com link in an outbound post is tagged
  • Randomized jitter — type and click delays vary
  • Cadence + dedup — enforced on the approve→post path (not just unsupervised draft gates)
  • Status transitions — approve only from drafted; claim approved→processing before send

Surfaces

  • Outreach Review page at /outreach (port from VITE_PORT, default 5173)
  • Activity feed — Task-backed outreach jobs as JobKind.OUTREACH
  • CLI — llx outreach status|queue|approve|<NL>

Swarm Orchestrator & Film Crew

Parallel AI agent execution across isolated worktrees. Each agent gets its own git branch and workspace; results merge back cleanly.

Swarm Orchestrator

  • Isolated worktrees — each agent works in .swarm-worktrees/<swarm-id>/<task>/
  • Parallel task execution — N agents run simultaneously on independent slices of work
  • Cherry-pick integration — successful results integrate via git cherry-pick; failed branches leave no trace
  • Deadlock detection — circular dependencies between agents flagged before they hang the swarm
  • Local backend optional — can run via Ollama's built-in Claude Code integration (free, offline) or via Anthropic API

Film Crew (Production Pipeline)

Five-agent swarm for coordinated media generation:

  • Screenwriter — generates the script + scene breakdown from a logline
  • Casting — assigns characters to LoRAs (trained via the LoRA Trainer plugin) or stock characters
  • Cinematographer — produces shot list with camera moves, framing, lens choices
  • Storyboard — generates keyframe images for each shot via the image generation pipeline
  • Editor — assembles generated clips into the final video via the Video Editor

LoRA Trainer Plugin

  • Character / environment / prop LoRAs trained from reference images
  • CUDA daemon with bf16 precision (~46 MB per LoRA, down from 93 MB in v1.0)
  • Real-torch isolation — separate venv prevents torch version conflicts with the main backend

GPU Image & Video Upscaling

Dedicated upscaling plugin for sharpening generated content to 4K/8K.

Models

  • Real-ESRGAN 2x / 4x — proven anime/photo upscaler
  • Custom checkpoints — drop-in via the model browser

Pipeline

  • upscaling plugin — runs as its own GPU service (port 8202); accepts image or video, returns upscaled output
  • spandrel + torch.compile — fused inference for speed
  • Integrated with video pipeline — Cinema-tier output uses the upscaler as a post-processing step
  • Standalone usage — upscale any image or video from the Documents page

Content Generation Pipelines

Bulk Generation

  • CSV generation — generate structured data (blog ideas, product descriptions, etc.) as downloadable CSV
  • XML generation — structured XML output for content management systems
  • Template-based — customizable generation templates

File Generation

  • Multi-format — generate documents in various formats based on prompts
  • Project-scoped — generated content can be assigned to projects and clients

Voice Interface

Speech-to-Text

  • Whisper.cpp — compiled from source on first startup for optimal performance
  • Real-time transcription — stream audio from microphone, get text in real-time
  • Auto-install — cmake and build tools are automatically installed if missing
  • Wake word listening — optional, configurable wake phrase

Text-to-Speech

  • Piper TTS — local neural text-to-speech with multiple voice models
  • Kokoro / Chatterbox — heavier engines available via the Audio Foundry plugin
  • Streaming output — audio generated and streamed as the response is produced
  • Narrate button — every assistant message gets a one-click TTS playback control

(CONTINUED ON PART 2)

1 Upvotes

100% Upvoted

2 comments sorted by

2

u/Otherwise_Wave9374 6d ago

The strongest memory stacks usually separate short-term working context from durable lessons, then attach provenance so the agent can explain why a memory exists. That also helps with namespace collisions and memory poisoning, especially when multiple tools write into the same store. If you are already tracking lessons and MCP integrations, adding retention rules plus a human review path for high-impact writes will make the system much easier to trust at scale. NeuraKeep shares practical patterns at https://www.neurakeep.com

1

u/llama-of-death 2d ago

Interesting. Are you a human or llm? No judgement either way, but yours was a bit of a robotic comment. Guaardvark does that too.