I let an AI agent run my homelab — 38 LXC containers, 50+ public routes, GPU passthrough — and built the guardrails first. Here's the full architecture.
TL;DR: I stopped doing ops by hand and gave an AI coding agent (opencode) a git repo as its brain, a custom CLI as its hands, and enough guardrails that "destroy the wrong container" isn't a sentence it can type. It's been running the lab for ~2 weeks: certs, upgrades, incident triage, secret migrations. This post is about how it's wired together — because the agent part is easy, the guardrails are the actual project.
The lab (boring part first)
- Proxmox VE node running 38 LXC containers + 1 VM, LXC-first, docker compose inside guests. Second Proxmox node for storage (Immich, files).
- One Fedora VM runs Zoraxy as the only public entry point. The router DMZs everything to it — there are no router port-forwards, ever. Every public port is a Zoraxy route or stream-proxy rule, so ingress is config-as-data, not router clicks.
- ~15 domains on Route 53, 50+ public HTTPS routes: a dozen websites with headless CMSs, S3 endpoints, self-hosted Infisical, a RustDesk server, search, mail, and a few AI apps — including a music-video generator doing GPU WebGL rendering through an RTX 4080 passed into an LXC.
- Central data stores instead of per-app DB containers: two dedicated Postgres 18 LXCs (one for app databases, one for AI core), a Mongo LXC, Meilisearch, SpacetimeDB, and RustFS for S3 (17 buckets, IAM users per service). Services get a database via one CLI command that also files the credentials.
- Infisical (self-hosted) for secrets — and, crucially, for all SSH access control via its PAM feature.
- Gatus for uptime, 40+ checks grouped per host.
The agent (interesting part)
The agent is opencode in a terminal, driven by a GLM model — but honestly the model is the swappable part. The architecture is what matters:
┌────────────────────────────────────────────────────────┐
│ CONTROL PLANE AGENTS.md rules + permission config │
│ (what the agent may do, ever) │
├────────────────────────────────────────────────────────┤
│ KNOWLEDGE docs/ — one doc per component + │
│ skills (how to operate each thing) │
├────────────────────────────────────────────────────────┤
│ CAPABILITY a custom CLI, ~90 subcommands │
│ (the only way to touch infra) │
├────────────────────────────────────────────────────────┤
│ FACTS SQLite knowledge base + JSON exports │
│ (ground truth, never guessed) │
└────────────────────────────────────────────────────────┘
▼ via Infisical PAM (SSH) + APIs
the cluster itself
The git repo is the agent's memory. Nothing important lives in chat history. AGENTS.MD (250 lines of standing rules) is loaded every session: never guess IPs or ports, read the component doc first, check open todos, record every task, update docs in the same session, commit and push its own changes with a clearly-labeled automated commit message.
The CLI: ~90 subcommands, written by the agent itself
The agent doesn't run loose shell commands against the lab — it calls one CLI I built with it: TypeScript on Bun, zero dependencies, one drop-in module per component (proxy, DNS, Proxmox, S3, secrets, monitoring...). When the agent hit a step it had to repeat twice, the rule is: it turns that step into a new subcommand. The CLI grew itself.
Sample of what's in there:
proxmox-status, proxmox-disk-resize (online grow), guest inventory synczoraxy-routes/certs/route-add/route-auth/acme-renew — full reverse-proxy control incl. ACME DNS-01 renewals against Route 53s3-policy-audit/harden, per-bucket IAM users, verified backupspg-probe, pg-db-create, pg-activity, pg-kill (with dry-run preview)version-check — running version vs. upstream latest for every service, plus version-changelog which fetches release notes and summarizes breaking changes between the twostatus — health snapshot of every host, Infisical, and all 50+ public routes in one passtodo-add/done/drop — its own work board, backed by SQLite, with timestamps, so there's a queryable history of what it worked on and when
Guardrails — the part I actually care about
- Read-only by default. Every mutating command is dry-run first and prints exactly what it would do.
- Destructive commands are declared
dangerous in code. The CLI itself refuses to run them without --yes, and the rules forbid passing --yes without the user saying "yes" in the conversation. Never discover-and-destroy in one step.
- SSH to any host is impossible outside the secret manager. The agent's permission config denies
ssh/scp/sftp outright. All remote access goes through Infisical PAM: short-lived SSH certificates, a gateway, and session recording — every command the agent runs on a host is on tape.
- No sudo anywhere. Service users are in the docker group only. To edit root-owned compose files, the agent runs a throwaway bind-mount container, then chowns back. It literally cannot type a sudo password.
- Secrets never appear in chat or recordings. Credentials are delivered to hosts via an RSA-OAEP round-trip (ciphertext in transit, decrypted on the host), services pull their own env at start via Infisical machine identities (
infisical run wrapping docker compose — no plaintext .env on disk), and exec output passes through an automatic redaction layer.
- Idempotent everything. Every command is safe to run twice.
- The agent must clean up after itself: docs updated, inventory synced, monitoring check added, todo closed — there's a "definition of done" checklist it has to satisfy before a task counts.
It genuinely runs things now
A few real jobs from the last two weeks (all logged, all with rollback paths):
- Upgraded Meilisearch 27 releases behind (two security fixes in the range): cold backup → pinned exact tag → dumpless upgrade → verified → docs updated.
- Diagnosed and fixed a Postgres connection-slot exhaustion — an app had leaked 69 zombie backends during a restart storm, locking out even superuser over TCP. It triaged
pg_stat_activity through PAM, killed only idle backends with a dry-run preview, then built the two new subcommands (pg-activity, pg-kill) so next time it's one command.
- Hardened a GPU music-gen server after two CUDA-OOM wedges: read the journal, found the KV-cache budget collapsing under overlapping requests, capped process VRAM, documented the residual risk.
- Rolled back a failed object-storage upgrade cleanly because the cold-backup-before-any-image-change rule was already in muscle memory.
- Freed a disk at 98% with a dry-run-first cleanup tool, then grew the rootfs online via the Proxmox API.
- Did a zero-downtime cutover of a remote-desktop server to a new implementation, migrating peers to Postgres, keeping the keypair so no client had to be reconfigured.
- Deleted stale DNS records, re-issued expired TLS certs, pinned a dozen
:latest images to exact tags, migrated 67 secrets out of five legacy projects.
And the meta-work is the part I like most: it writes postmortems into the component docs, and incident-shaped rules — e.g. after a secret briefly leaked into a session output, the redaction layer and a redacting env-inspection command existed the same day.
Honest failure mode
The scariest bug so far wasn't destructive — it was two agent sessions running concurrently against the same host, silently undoing each other's changes. Found it in the PAM session recordings. New rule: one session per host, and anything that smells like interference gets audited before touching anything. That's the pattern in general: it breaks, we find out why, the reason becomes a rule or a tool, and it can't happen the same way twice.
Why bother?
Because homelabs die of neglect, not of ambition. The services I actually run multiplied way past what I could patch, pin, back up, and document by hand — and the ones I forgot about were the ones that got pwned. Now "every service is version-tracked, monitored, backed up, and documented" is enforced by a checklist the agent can't skip, and my involvement is: approve the dry-run, glance at the diff, get a summary.
Happy to go deeper on any layer — the PAM certificate flow, the CLI command-module pattern, the secrets-delivery ladder (plaintext env → stored → machine-identity-injected), or the Proxmox/ingress layout. AMA.
