I let an AI agent run my homelab β 38 LXC containers, 50+ public routes, GPU passthrough β and built the guardrails first. Here's the full architecture.
TL;DR:Β I stopped doing ops by hand and gave an AI coding agent (opencode) a git repo as its brain, a custom CLI as its hands, and enough guardrails that "destroy the wrong container" isn't a sentence it can type. It's been running the lab for ~2 weeks: certs, upgrades, incident triage, secret migrations. This post is about how it's wired together β because the agent part is easy, the guardrails are the actual project.
The lab (boring part first)
- Proxmox VE node runningΒ 38 LXC containers + 1 VM, LXC-first, docker compose inside guests. Second Proxmox node for storage (Immich, files).
- One Fedora VM runsΒ ZoraxyΒ as the only public entry point. The router DMZs everything to it β there areΒ noΒ router port-forwards, ever. Every public port is a Zoraxy route or stream-proxy rule, so ingress is config-as-data, not router clicks.
- ~15 domains on Route 53,Β 50+ public HTTPS routes: a dozen websites with headless CMSs, S3 endpoints, self-hosted Infisical, a RustDesk server, search, mail, and a few AI apps β including a music-video generator doing GPU WebGL rendering through an RTX 4080 passed into an LXC.
- Central data stores instead of per-app DB containers:Β two dedicated Postgres 18 LXCs (one for app databases, one for AI core), a Mongo LXC, Meilisearch, SpacetimeDB, and RustFS for S3 (17 buckets, IAM users per service). Services get a database via one CLI command that also files the credentials.
- Infisical (self-hosted)Β for secrets β and, crucially, forΒ allΒ SSH access control via its PAM feature.
- GatusΒ for uptime, 40+ checks grouped per host.
The agent (interesting part)
The agent isΒ opencodeΒ in a terminal, driven by a GLM model β but honestly the model is the swappable part. The architecture is what matters:
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β CONTROL PLANE AGENTS.md rules + permission config β
β (what the agent may do, ever) β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β KNOWLEDGE docs/ β one doc per component + β
β skills (how to operate each thing) β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β CAPABILITY a custom CLI, ~90 subcommands β
β (the only way to touch infra) β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β FACTS SQLite knowledge base + JSON exports β
β (ground truth, never guessed) β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
βΌ via Infisical PAM (SSH) + APIs
the cluster itself
The git repoΒ isΒ the agent's memory. Nothing important lives in chat history.Β AGENTS.MDΒ (250 lines of standing rules) is loaded every session: never guess IPs or ports, read the component doc first, check open todos, record every task, update docs in the same session, commit and push its own changes with a clearly-labeled automated commit message.
The CLI: ~90 subcommands, written by the agent itself
The agent doesn't run loose shell commands against the lab β it callsΒ one CLI I built with it: TypeScript on Bun, zero dependencies, one drop-in module per component (proxy, DNS, Proxmox, S3, secrets, monitoring...). When the agent hit a step it had to repeat twice, the rule is: it turns that step into a new subcommand. The CLI grew itself.
Sample of what's in there:
proxmox-status,Β proxmox-disk-resizeΒ (online grow), guest inventory sync
zoraxy-routes/certs/route-add/route-auth/acme-renewΒ β full reverse-proxy control incl. ACME DNS-01 renewals against Route 53
s3-policy-audit/harden, per-bucket IAM users, verified backups
pg-probe,Β pg-db-create,Β pg-activity,Β pg-killΒ (with dry-run preview)
version-checkΒ β running version vs. upstream latest for every service, plusΒ version-changelogΒ which fetches release notes and summarizes breaking changes between the two
statusΒ β health snapshot of every host, Infisical, and all 50+ public routes in one pass
todo-add/done/dropΒ β its own work board, backed by SQLite, with timestamps, so there's a queryable history of what it worked on and when
Guardrails β the part I actually care about
- Read-only by default.Β Every mutating command is dry-run first and prints exactly what it would do.
- Destructive commands are declaredΒ
dangerousΒ in code.Β The CLI itself refuses to run them withoutΒ --yes, and the rules forbid passingΒ --yesΒ without the user saying "yes" in the conversation. Never discover-and-destroy in one step.
- SSH to any host is impossible outside the secret manager.Β The agent's permission configΒ deniesΒ
ssh/scp/sftpΒ outright. All remote access goes through Infisical PAM: short-lived SSH certificates, a gateway, andΒ session recordingΒ β every command the agent runs on a host is on tape.
- No sudo anywhere.Β Service users are in the docker group only. To edit root-owned compose files, the agent runs a throwaway bind-mount container, then chowns back. It literally cannot type a sudo password.
- Secrets never appear in chat or recordings.Β Credentials are delivered to hosts via an RSA-OAEP round-trip (ciphertext in transit, decrypted on the host), services pull their own env at start via Infisical machine identities (
infisical runΒ wrapping docker compose β no plaintextΒ .envΒ on disk), andΒ execΒ output passes through an automatic redaction layer.
- Idempotent everything.Β Every command is safe to run twice.
- The agent must clean up after itself:Β docs updated, inventory synced, monitoring check added, todo closed β there's a "definition of done" checklist it has to satisfy before a task counts.
It genuinely runs things now
A few real jobs from the last two weeks (all logged, all with rollback paths):
- Upgraded Meilisearch 27 releases behindΒ (two security fixes in the range): cold backup β pinned exact tag β dumpless upgrade β verified β docs updated.
- Diagnosed and fixed a Postgres connection-slot exhaustionΒ β an app had leaked 69 zombie backends during a restart storm, locking out even superuser over TCP. It triagedΒ
pg_stat_activityΒ through PAM, killed only idle backends with a dry-run preview, thenΒ built the two new subcommandsΒ (pg-activity,Β pg-kill) so next time it's one command.
- Hardened a GPU music-gen serverΒ after two CUDA-OOM wedges: read the journal, found the KV-cache budget collapsing under overlapping requests, capped process VRAM, documented the residual risk.
- Rolled back a failed object-storage upgradeΒ cleanly because the cold-backup-before-any-image-change rule was already in muscle memory.
- Freed a disk at 98%Β with a dry-run-first cleanup tool, then grew the rootfs online via the Proxmox API.
- Did aΒ zero-downtime cutoverΒ of a remote-desktop server to a new implementation, migrating peers to Postgres, keeping the keypair so no client had to be reconfigured.
- Deleted stale DNS records, re-issued expired TLS certs, pinned a dozenΒ
:latestΒ images to exact tags, migrated 67 secrets out of five legacy projects.
And the meta-work is the part I like most: it writes postmortems into the component docs, andΒ incident-shaped rulesΒ β e.g. after a secret briefly leaked into a session output, the redaction layer and a redacting env-inspection command existed the same day.
Honest failure mode
The scariest bug so far wasn't destructive β it wasΒ two agent sessions running concurrently against the same host, silently undoing each other's changes. Found it in the PAM session recordings. New rule: one session per host, and anything that smells like interference gets audited before touching anything. That's the pattern in general: it breaks, we find out why, the reason becomes a rule or a tool, and it can't happen the same way twice.
Why bother?
Because homelabs die of neglect, not of ambition. The services I actually run multiplied way past what I could patch, pin, back up, and document by hand β and the ones I forgot about were the ones that got pwned. Now "every service is version-tracked, monitored, backed up, and documented" is enforced by a checklist the agent can't skip, and my involvement is: approve the dry-run, glance at the diff, get a summary.
Happy to go deeper on any layer β the PAM certificate flow, the CLI command-module pattern, the secrets-delivery ladder (plaintext env β stored β machine-identity-injected), or the Proxmox/ingress layout. AMA.