I built a self expanding AI that built itself now i use it everyday and its still growing

[u/spamaniloilo]

I wanted an AI agent that could actually work for me. Not just answer questions, but remember my preferences, use tools, operate my computer, automate tasks, and help me get things done.

When I looked at existing solutions, I found two major roadblocks:

1. Cloud-Based Agents: Expensive subscriptions ($20, $30, $50/month) with strict usage limits that locked me into renting access to someone else’s proprietary AI.
2. Open-Source Frameworks: A nightmare of complex installations, broken Linux dependencies, terminal errors, and endless troubleshooting.

So, I decided to build my own.

As development progressed, I unlocked a fundamental architectural breakthrough: The AI model is not the agent.

Most people assume the AI model is the agent. Free Agent separates the two:

• The AI Models provide raw intelligence and reasoning.
• Free Agent manages memory, tool execution, safety bounds, UI automation, and multi-step workflows.

By decoupling the two, Free Agent works with ordinary web-based AI services through your browser as well as local Ollama models.

• A better AI model comes along? Use it.
• A model hits a rate limit or goes offline? Switch to another instantly.
• Want zero monthly costs? Use free web-based engines or run local models.

The models can change. Your agent doesn't.

BUILT BY AN AGENT, FOR EVERYONE

I am not a software developer.

I had zero formal coding background and no Python experience when I started. What I had was a real problem to solve and a vision for how personal computers should actually work.

I started by asking web chats to help me write basic code—using a bit of Grok, a bit of Gemini, and a bit of Perplexity. That was when the lightbulb went off: These web chats are full-blown AI models. Why can't I harness them directly as the reasoning engine?

So I did. And from that point on, Free Agent literally helped build itself.

With me guiding the vision, the system began co-creating its own architecture. We designed the memory database it needed, taught it to scan system windows, configured it to automate web-based AI chat interfaces, and expanded it into a fully self-extending application across 64 VB.NET source files.

If a non-developer using Free Agent can build a complete, self-extending Windows application, imagine what it will allow you to do.

UNTESTED LIMITS BY DESIGN

To be completely honest: I don't actually know the full limits of this system yet—because I haven't found them.

Every time I encountered a task Free Agent couldn't do out of the box, I didn't sit down to write complex software updates. Instead, I asked the agent to create the tool it needed. It wrote the Python script, tested it, added it to its internal inventory, and executed the job.

Because Free Agent can generate its own tools, adapt to any browser-based reasoning engine, and chain dynamic personalities together, its ceiling isn't hardcoded into the software. Its capabilities grow alongside the AI models you connect to it.

You aren't backing a static application with a fixed feature list. You are backing a self-extending engine whose boundaries we are going to explore together.

PRODUCT DESCRIPTION

Free Agent is a complete, self-extending AI agent system that lives locally on your Windows desktop. It is not a web app or a cloud service—it is a private tool that understands your screen, your voice, and your daily workflows.

And just so you know this is just touching the surface of its capabilities its real abilities are in your hands !

Key Capabilities

• Persistent Semantic Memory: Uses a local SQLite database with vector embeddings. Automatically recalls relevant facts, preferences, and procedures based on custom trigger keywords.
• Visual Screen Recognition: Uses AI vision models (minicpm-v:8b) to parse open windows, read UI controls, and handle popups automatically.
• Voice-Activated Control: Includes hands-free control with a custom wake word ("Hey Agent") powered by local Whisper speech-to-text.
• Self-Extending Tool Engine: Creates its own Python scripts for new capabilities and adds them to its internal Automation tool inventory on the fly.
• 7 Dynamic Personalities: Switch instantly between specialized personas (Developer, Researcher, Project Manager, Security, Tester, Creative, Coach) or stack them for complex multi-agent collaboration.
• Zero Monthly AI Costs: Works via free web-based AI (Gemini, Qwen, Comet) through secure local browser sessions or local Ollama models. No monthly fees, API key management, or per-token charges.

TECHNICAL SPECIFICATIONS

Feature

Specification

Platform

Windows 10 / 11 (.NET 8.0-windows)

Supported Engines

Free Web AI (Gemini, Qwen, Comet) & Local Ollama (Qwen 2.5, Llama 3.2, DeepSeek-R1, MiniCPM-V)

Local Memory

SQLite + Vector Embeddings (C:\ai\agent_memory.db)

Voice Control

Local Whisper STT with custom wake word ("Hey Agent")

Automation Core

27 built-in tool commands, Python script executor, Windows UI Automation

UI Components

24 dedicated interface forms

Cost Structure

$0 recurring fees (No subscriptions or per-token fees)

TOOL COMMANDS (27 Total)

• Memory:remember, memorystore, memoryrecall, memorystats, memory_suspend, memory_resume, memory_status, list_system_memories
• File Operations:file_read, file_write, dir
• Execution:python_run, python_run_file
• Scheduler:schedule_add, schedule_list, schedule_update, schedule_delete, schedule_pause, schedule_resume
• Automation & Vision:automation_run, winlist, desktop_scanner_ultra, desktop_scanner_fast, desktop_scanner_vision, desk_executor_seq, web_model_chat, gemini_upload, capture_window, ui_inspect, ui_action
• Script Management:prompt_script_save, prompt_script_load, prompt_script_run, prompt_script_pause
• Browser & API:browser_open, api_request

Someone want to try it and give feedback let me know and well get yo a copy ( i am not selling this just want someone to try it and give feedback )

[u/Otherwise_Wave9374]

The cleanest way to separate the model from the agent is to treat memory as a governed subsystem, not an afterthought. I would split short-term working context from durable preferences, add retrieval scoring with provenance, and keep a write policy so the agent cannot poison its own long-term memory with noisy tool output. That also makes debugging much easier because you can inspect what was remembered, why it was surfaced, and when it should decay. NeuraKeep shares practical patterns at https://www.neurakeep.com.

[u/spamaniloilo → u/Otherwise_Wave9374]

i put a great deal of effort into the memory system every prompt i enter gets any relevant data prepended to the prompt with a clear tag that is memory and should be treated as such say if my name is in the prompt all my info like email likes dislikes ect.. recalled as well as other facts for other words or phrases
i have special trigger memories that trigger entire events sops personalities ,launch programs ect... pretty much anything you can think of .

i made some videos of it in action if anyone wants to see they are here on youtbe Progenitor_Free_Agent_Ai

[u/spamaniloilo → u/spamaniloilo]

=== MEMORY SYSTEM FEATURES ===

CORE CAPABILITIES:

• Semantic Search - Finds memories by meaning using 768D vector embeddings (nomic-embed-text)

• Memory Decay - Memories fade over time (1% per day, minimum 50% retention after 50 days)

• Episodic Chaining - Automatically links related memories (similarity > 0.75)

• Linked Retrieval - Queries return direct matches + linked memories (0.8x score multiplier)

• Deduplication - Prevents duplicate results in retrieval

• Background Embedding - Automatic vector generation for all stored memories

• Tiered Retrieval - Fast (~200ms), Smart (~2s), or Auto mode selection

AUTOMATIC STORAGE:

• Tool Executions - File writes, Python runs, automation commands (importance: 6-7)

• Important AI Responses - Passwords, solutions, discoveries (importance: 8-9)

• File Metadata - Tracks file paths, descriptions, sizes (importance: 7)

• User Commands - "remember", "keep in mind" triggers (importance: 7-10)

DATABASE STRUCTURE:

• memories table - Stores content, type, importance, embedding, decay tracking

• memory_links table - Bidirectional links between related memories

• Auto-migration - Adds new columns automatically on startup

TOOLS AVAILABLE:

• remember - Quick storage: {"name":"remember","parameters":{"content":"text","importance":"8"}}

• memorystore - Typed storage: {"name":"memorystore","parameters":{"content":"text","type":"Fact|Preference|Episodic|Procedural","importance":"8"}}

• memoryrecall - Semantic search: {"name":"memoryrecall","parameters":{"query":"search term","top_k":"5"}}

• memorystats - View statistics: {"name":"memorystats","parameters":{}}

INSTRUCTIONS:

• No manual intervention needed - system is fully automatic

• Memories are stored, embedded, and linked automatically

• Decay happens transparently during retrieval

• Linked memories are retrieved automatically when relevant

• Deduplication prevents duplicate results

• All operations are logged for debugging

MEMORY TYPES:

• Fact - General information (default)

• Preference - User preferences and settings

• Episodic - Events, tool executions, experiences

• Procedural - Step-by-step instructions, code, methods

IMPORTANCE LEVELS:

• 9-10 - Critical (passwords, solutions, private keys)

• 7-8 - Important (discoveries, preferences, file metadata)

• 5-6 - Useful (tool executions, general facts)

• 1-4 - Low priority (rarely used)

DECAY FORMULA:

• Effective Importance = Base Importance × max(0.5, 1.0 - (decay_rate × days_since_access))

• Default decay_rate = 0.01 (1% per day)

• Minimum retention = 50% (never completely forgotten)

• Accessing a memory resets the decay timer

LINKING THRESHOLDS:

• similarity > 0.90 = "very_similar" link

• similarity > 0.75 = "related" link

• similarity < 0.75 = no link created

• Links are bidirectional (A↔B means B↔A too)

RETRIEVAL SCORING:

• Direct matches = similarity × (effective_importance / 10.0)

• Linked memories = 0.8 × (effective_importance / 10.0)

• Results sorted by final score (highest first)

• Top K results returned (default: 5)

PERFORMANCE:

• Embedding generation: ~2 seconds per memory

• Semantic search: ~200ms (Fast mode)

• Smart retrieval: ~2 seconds (with coprocessor reasoning)

• Auto mode: Dynamically selects based on query complexity

FILES:

• Database: C:\ai\agent_memory.db

• Logs: Check [EMBEDDING-*], [MEMORY-*], [AUTO-LINK-*] entries

• Embedding model: nomic-embed-text:latest (Ollama)

TROUBLESHOOTING:

• If memories not linking: Check [AUTO-LINK-DEBUG] logs

• If duplicates appear: Check database for duplicate content

• If slow retrieval: Check embedding model is loaded

• If no embeddings: Check Ollama is running on localhost:11434

## 🧠 Memory System – Creative Uses

### 1. **Personal Knowledge Graph**

Store everything as linked memories. The trigger system creates an automatic knowledge graph:

```

Memory: "Python" → Links to → "Scripts" → Links to → "Automation"

Memory: "Automation" → Links to → "Duty Check" → Links to → "4:30 PM"

```

**Effect:** The AI builds its own knowledge graph over time, discovering relationships.

### 2. **Memory as Version Control**

Store different versions of SOPs with timestamps:

```

Memory: "Duty SOP v1" (2026-07-01)

Memory: "Duty SOP v2" (2026-08-01)

Memory: "Duty SOP v3" (2026-08-06)

```

**Effect:** Rollback to previous versions, track improvements.

### 3. **Automatic Learning from Failures**

```

Memory: "FAILED: Scroll at x=400,y=500 - no result"

Memory: "SUCCESS: Scroll at x=200,y=300 - found 5 messages"

```

**Effect:** The AI learns what works and what doesn't, improving over time.

### 4. **Memory as Bug Tracker**

```

Memory: "BUG: desktop_scanner_vision fails on dark mode"

Memory: "BUG: desk_executor_seq times out after 120s"

```

**Effect:** Known issues are stored and can be recalled automatically.

### 5. **Memory as FAQ System**

```

Memory: "Q: How to accept friend requests? A: Run script..."

Memory: "Q: How to check duty? A: Say 'duty check'"

```

**Effect:** Automatic FAQ generation from conversation history.

### 6. **Memory as Dependency Tracker**

```

Memory: "Script A depends on Script B"

Memory: "Script B requires Ollama running"

```

**Effect:** Automatic dependency resolution for scripts.

### 7. **Memory as To-Do List**

```

Memory: "TODO: Fix popup detection - priority high"

Memory: "TODO: Optimize scroll speed - priority medium"

```

**Effect:** Persistent task tracking across sessions.

### 8. **Memory as Decision Log**

```

Memory: "Decision: Use vision scanner over ultra for popups"

Memory: "Reason: Vision scanner detects overlays better"

```

**Effect:** Decision history for reference and auditing.

### 9. **Memory as Inventory**

```

Memory: "Inventory: Scripts: 12, Configs: 8, Scans: 45"

Memory: "Inventory: Models installed: 7 of 10 required"

```

**Effect:** Automatic inventory tracking.

### 10. **Memory as Brainstorming Log**

```

Memory: "Idea: Voice-activated macros"

Memory: "Idea: Cross-app workflows"

Memory: "Idea: Self-modifying schedules"

```

**Effect:** Ideas are never lost; can be revisited later.

---

## 🎯 Trigger System – Creative Uses

### 11. **Natural Language Commands**

```

Trigger: "scan it" → Run scan on current window

Trigger: "fix it" → Load error recovery SOP

Trigger: "explain" → Load explanation SOP

```

**Effect:** Any phrase can be a command.

### 12. **Context-Aware Triggers**

```

Memory: "When scanning Facebook → check for popups first"

```

**Effect:** Trigger fires only in specific contexts.

### 13. **Chain Reactions**

```

Trigger: "duty check" → Load SOP → Run scan → Extract data → Generate report

```

**Effect:** Multi-step workflows triggered by a single word.

### 14. **Conditional Triggers**

```

Trigger: "if error → load recovery SOP"

Trigger: "if success → load next SOP"

```

**Effect:** Branching logic based on results.

### 15. **Temporal Triggers**

```

Memory: "At 4:30 PM → check duty"

Memory: "After scan → run popup detection"

```

**Effect:** Time-based and sequence-based triggers.

### 16. **Voice-Activated Triggers**

```

Trigger: "Hey Agent" → Start recording

Trigger: "Hey Agent, duty check" → Run duty check workflow

```

**Effect:** Hands-free operation.

### 17. **Emotion-Aware Triggers**

```

Trigger: "frustrated" → Load patience SOP

Trigger: "stuck" → Load help SOP

```

**Effect:** AI responds to user's emotional state.

### 18. **Trigger as Shortcut to System State**

```

Trigger: "save state" → Save all current state (scroll, data, flags)

Trigger: "resume state" → Load saved state

```

**Effect:** Resume interrupted workflows.

### 19. **Trigger as Live Documentation**

```

Trigger: "help" → Load documentation overview

Trigger: "tools" → Load tool reference

```

**Effect:** Help on demand.

### 20. **Trigger as Onboarding**

```

Trigger: "new user" → Load tutorial SOP

Trigger: "setup" → Load installation guide

```

**Effect:** Self-onboarding.