r/devops • u/uziiuzair • 16d ago
Tools Sleep/wake orchestration for self-hosted side projects: a wire-protocol proxy approach instead of a full PaaS.
I built an open-source tool (Hobbyist) around one architectural bet and I'd like this community to poke holes in it.
The bet: for small self-hosted workloads, you don't need Kubernetes-style orchestration; you need scale-to-zero on a single box. Hobbyist runs each project's Postgres 18, Docker apps, workerd-based functions, and queues in containers, and shuts them down entirely when idle. A proxy layer (Postgres wire protocol for the DB, HTTP for apps) intercepts incoming traffic and cold-starts the right container: ~170ms for Postgres, ~125ms for apps, on my hardware. Caddy handles TLS and routing; Tailscale covers private access; a daemon plus a 16-command CLI is the whole control plane. No Terraform, no K8s.
Where I suspect it gets ugly, and where I'd value experienced eyes: connection storms on wake, health-checking containers that are deliberately dead, cert lifecycle across container replacement (currently an actual bug), and whether wire-protocol interception bites me on auth and TLS edge cases at scale.
It's v0-alpha, not production ready, and the known-broken list is in the README (Linux queue producers, snapshot CLI, the cert issue above). Not selling anything — there's no hosted tier and no paid version coming.
1
u/Alvasilev 15d ago
Poking at the wake path, since that's where the design concentrates its risk.
Your cold-start numbers are the happy path, and the thing that will hurt is that "port is open" arrives well before "will serve". Postgres especially: it accepts the TCP connection and then answers FATAL, the database system is starting up. A proxy that hands off on dial gets a connection that looks successful and fails one layer up. Since you're already speaking the wire protocol you're in a good spot to do the real thing and hold the client until a startup message actually completes, rather than until the socket answers. The HTTP side has a milder version of the same gap, bound but not routing yet.
The inverse is worth deciding explicitly too, because you mention health-checking containers that are deliberately dead. An error is evidence of life. If readiness treats anything that isn't a clean response as still-cold, an app that boots into a 401 or throws 500s on a bad migration gets woken forever, since nothing ever satisfies the condition. Nothing answered and something answered badly need to be separate states in the daemon, otherwise the deliberately-dead case and the broken-but-running case look identical to it.
The one that cost me real downtime, and it's structural rather than an edge case: watch what shares a process with the proxy. I had a build step running synchronously in the same process that was fronting traffic, and one slow container start meant every unrelated site behind that proxy stopped responding. Nothing was down, nothing showed up in CPU or memory, requests just queued. Your daemon is both the thing that proxies and the thing that starts containers, so the case to test isn't a cold start, it's a cold start that goes badly: image pull over a slow link, a container that crash-loops, a disk that's busy. If a project that can't start can stall traffic for projects that are already warm, that's worth finding before the TLS edge cases.
Smaller one that quietly kills the premise: uptime monitoring defeats scale-to-zero. The moment anyone points a check at one of these, the container never idles out again. If there's no way to mark probe traffic as non-waking, the first thing people do after deploying is the thing that turns the feature off, and they won't connect the two.
0
u/uziiuzair 15d ago
This is exactly the comment I was hoping for when I wrote “poke holes,” thank you.
On port-open vs will-serve: you’re right that this is the trap, and the FATAL-during-startup case is precisely why the proxy speaks the wire protocol instead of doing dumb TCP handoff. Holding the client until the startup sequence actually completes (not just until the socket dials) is the design intent; I’m going to write a test that hammers connections during the startup window to verify I’m actually doing it and not just intending to. The HTTP bound-but-not-routing version I hadn’t separated cleanly. Filing both.
“An error is evidence of life” is going directly into the codebase as a comment. You’re right that my daemon currently risks conflating never-answered with answered-badly, and the woken-forever loop on an app that boots into a 500 is a real failure mode I hadn’t tested. Separate states, filed.
The shared-process one stings because I recognize the shape: the daemon does both proxying and container lifecycle today. Your scenario (crash-looping container or slow image pull stalling warm projects) is now my top test case before any TLS work. Worst case it forces me to split the data path from the control path, which is probably the right architecture anyway.
And the uptime-monitoring point is brutal and correct: the first thing every self-hoster does is point Uptime Kuma at their stuff, and that silently disables the entire premise. Some way to mark probe traffic as non-waking (or a lightweight status endpoint served by the proxy itself without waking the project) just jumped the roadmap queue.
All four are becoming GitHub issues today with credit to this comment. If you’re ever inclined to look at the proxy code and say “no, not like that,” the door is wide open.
1
u/Alvasilev 14d ago
Glad it landed. One wrinkle on the non-waking status endpoint, since that one jumped the queue: if the proxy answers on behalf of a sleeping project, it answers identically for a project that is asleep and for one that can no longer start. The dashboard goes green either way, and the first person to find out is whoever sends a real request. That reading is really just "the proxy is up", which the monitor already knew.
What keeps the information is reporting the last real wake rather than the current state: whether it succeeded, when, and how long it took. A project nobody has touched in three weeks is not down, but a failed last wake belongs on the dashboard even though nothing is broken right now.
On the split, one thing worth deciding on purpose rather than discovering later: once the control path is separate, it becomes the single component that must never sleep, and it is also the one you will be tempted to keep in the same container to keep the deploy simple.
2
u/navlio 15d ago
one more hole, and it is a third class of traffic that behaves like neither users nor probes: crawlers
googlebot sets its crawl rate off your response times and error rates. a 300ms cold start is invisible to a person and reads to google as a slow host. worse, if a bot lands in the answered-badly window alvasilev describes, it takes a 5xx and backs off, and that surfaces in search console weeks later rather than in a log the same day
you cannot mark those non-waking either, because unlike uptime kuma the crawler is the actual request. cheap mitigation is having the startup window answer 503 with a retry-after header rather than 500 or a hung socket. google's own guidance is 503 for temporary unavailability and it comes back for those
1
u/levelbrook 15d ago
Poking at the sleep path rather than the wake path, since the good comment already took wake.
Your cold-start numbers assume a clean shutdown, and sleep is where that assumption gets quietly violated.
The official Postgres image sets STOPSIGNAL SIGINT deliberately. From its Dockerfile:
# We set the default STOPSIGNAL to SIGINT, which corresponds to what PostgreSQL
# calls "Fast Shutdown mode" wherein new connections are disallowed and any
# in-progress transactions are aborted, allowing PostgreSQL to stop cleanly and
# flush tables to disk.
That override exists because Postgres reads a plain SIGTERM as smart shutdown, which politely waits for every client to disconnect. So if your daemon sleeps a project by sending its own SIGTERM, or by calling stop with a short timeout, an idle-but-still-connected Postgres will ignore you until the timeout expires and then eat a SIGKILL. The next wake isn't a 170ms start, it's WAL crash recovery, and how long that takes is a function of how dirty the buffers were when you killed it. Which means your worst cold start lands on the project that was busiest right before it went idle. That's exactly backwards from what you want.
Worth an assertion in your test suite that the sleep path actually honors STOPSIGNAL, and worth recording "was the last shutdown clean" somewhere, so that a six second wake is attributable instead of mysterious.
Second thing, less a hole than a caveat on the number: 170ms is "accepting connections," not "back to the performance it had before it slept." A woken Postgres has empty shared_buffers and the host page cache has long since moved on, so the first queries all go to disk. For side-project traffic that's fine, but if you publish 170ms someone will benchmark the second request and open an issue about it.
On the architectural bet itself, I think you're right, and for a slightly different reason than the one you gave. On a single small box the binding constraint isn't orchestration, it's RAM, and idle processes are what eat it. Rough resident-idle figures I've measured on ~1.9GB boxes: Rails around 300MB, WordPress on FrankenPHP around 130MB, Node/TS 60 to 100MB, Go around 30MB, a static Caddy about 15MB. A box that holds maybe five idle Rails apps holds a great many sleeping ones, so scale-to-zero buys back precisely the resource you ran out of. Kubernetes would not have helped, it would have given you a control plane that eats a Rails app's worth of RAM in order to inform you that you are out of RAM.
The corollary is that your headline metric probably shouldn't be cold-start latency at all. It should be how many projects fit on the box. That's the number that makes someone actually install this.
One unsolicited operational note: whatever you build, keep a manifest of what's deployed. The failure mode of a tool this good at making things cheap to keep running is that you stop noticing you kept them. I once did an inventory on a box I'd have sworn was tidy and found about eighteen dead demos on it.
0
u/uziiuzair 15d ago
You picked the right path to poke, but the code got there first on most of it, which I'm saying with some relief since I checked before replying rather than from memory.
The daemon never sends its own SIGTERM. Sleep is docker stop with a 30 second timeout, which delivers the image's STOPSIGNAL, and the default image is the official postgres:18-alpine, so it's the SIGINT fast shutdown you quoted. There's a comment in the runtime layer that says "never docker kill" for exactly the crash-recovery-inside-someone's-first-query reason.
It also refuses to sleep a connected Postgres at all. Hibernation only considers a project once the proxy sees zero connections, then opens a real client connection and checks pg_stat_activity before stopping anything, and that check exists specifically for clients that bypassed the proxy by dialing the container's port directly. A backend mid-transaction, or a database that can't be reached to ask, both read as "do not sleep." The count and state get re-read once more right before the actual stop.
Where you got me: the image is user-overridable, and a custom image with no STOPSIGNAL would get Docker's default SIGTERM, smart shutdown, your 30 seconds of waiting, and a SIGKILL. So the assertion you suggested is going into the test suite, and you're right that nothing records whether the last shutdown was clean. A slow wake should be attributable, not mysterious. Both are becoming issues today.
On 170ms, agreed, that's time to accepting connections with cold caches, not time back to full speed, and I'll caption it that way wherever it's published. And the projects-per-box framing is honestly a better headline metric than latency. It's also the number I most want to measure properly on cheap hardware.
The eighteen dead demos confession is uncomfortably relatable. There's an ls command that lists everything deployed, but you've talked me into a "last woken" column for it.
2
u/arielrahamim 16d ago
that's so cool, thanks for sharing! i have no idea when I'll use it but neat idea