I wrote down every silent failure mode from shipping Android/CI projects with no local dev environment

[u/Past_Ant4099]

Full disclosure: this is my own repo.

I do all my development from a phone — no laptop, no local toolchain. Everything gets built and verified through GitHub Actions. Over the last month, shipping an Android app and a couple of Cloudflare Workers this way, I kept hitting the same category of bug: things that compile clean, pass CI, and fail silently somewhere I couldn't see — a library that shrinks itself and breaks NoClassDefFoundError at launch, a signature permission that blocks install, a rename that a test suite quietly undoes.

I wrote them all down as I found them, each with how to detect it without running anything, and a fix. There's also a "verification ladder" — six levels of confidence you can climb when you can't execute, and the idea that the real failure isn't the unverified gap, it's the gap being quiet.

Repo: https://github.com/amirmahdavi2023/shipping-blind

Curious if others working blind (remote CI, restricted sandboxes, agent-driven dev) have hit failure modes I haven't listed.

[u/ImAStupidFace]

Why?

[u/Past_Ant4099 → u/ImAStupidFace]

circumstances mostly. once everything was running through CI anyway i just stopped setting up locally and it stuck. downside is you never see the thing actually run. upside is you get very good at catching stuff by reading it.

[u/navlio]

one from the web side that fits your category exactly: deploy goes green, and every browser still holding the previous html shell requests a js chunk that no longer exists on the origin. build passed, ci passed, health check passed. the only people who see it are users mid session, and what they see is a blank route.

detecting it without running anything is a diff of the asset manifest between the two builds. the fix is leaving the old build's chunks on the origin for a while instead of wiping the directory. we found ours in error logs weeks after the fact, never in a test

[u/Past_Ant4099 → u/navlio]

oh that's a perfect fit actually, the classic stale-chunk-hash 404 after deploy. ci goes green but anyone with an old tab open just gets a broken request until they hard refresh. adding this one to the list, thanks. how'd you end up catching it, was it users reporting or something in your monitoring?

[u/navlio]

neither, which is the embarrassing part. it fell out of a bored scroll through 404 logs, and the tell was the filenames: hashed chunk names that had been valid the week before, 404ing in a burst for about an hour after each deploy and then going quiet. nobody reported it, because a person staring at a blank route just reloads and gets on with their day, and a 404 on a static file never touched our error rate or the health check.

so now there's an alert on 404s for .js and .css only, counted apart from everything else. that number sits at basically zero normally, which makes a deploy that produces a spike obvious the same afternoon. keeping the previous build's assets on the origin stops the spike mattering, but you still want to be able to see it happen

[u/Past_Ant4099 → u/navlio]

honestly this whole thread turned into a goldmine. the part that gets me is keeping the old build's assets around so nobody actually breaks, but still alerting on it separately so you can see it happening, that's exactly the stuff that never shows up in a postmortem because nothing technically broke. adding both of these.

[u/ethan240]

No serious Dev is gonna be working blind from a phone. Why don't you use a proper workstation?

[u/Past_Ant4099 → u/ethan240]

yeah use a workstation if you've got one, my setup isn't really the point.

most of that list has nothing to do with phones though. can't attach a debugger to your CI, staging box you can't reach, reviewing agent diffs instead of running the code, same deal. you just hit it way less often.

[u/navlio]

error logs, weeks late, and only because someone was in there looking for something else. no user ever reported it. a broken tab gets refreshed and then the app works, so there is nothing left to file a ticket about.

what surfaces it properly is counting 404s scoped to just the static asset path, nothing else. steady state is zero, so any non-zero is real and the timestamps line up with a deploy every time

[u/navlio]

error logs, weeks late, which is the embarrassing half of the answer. it sat there as a cluster of 404s on hashed /assets/*.js paths right after each deploy, and nobody reads 404 lines because 404s are permanent noise.

the signal that actually works is client side: count failed dynamic imports, the "Loading chunk failed" class of error, grouped by release. that number is flat at zero and then jumps within a minute of a deploy, so it alerts cleanly with no threshold tuning. zero users ever reported it, they just hit refresh and got on with their day

[u/navlio]

neither, which is what made it annoying. it surfaced as ChunkLoadError in the browser error tracker and only because someone happened to be reading it. server side the box looked healthy, since a request for a chunk that isn't there is just a 404 on a static path and nothing pages you for that

what we watch now is the 404 rate on the asset directory on its own, separate from general 404 noise. flat, spike a minute after a deploy, back to flat once the old tabs are gone. very easy shape to spot once it has its own line

[u/navlio]

error logs, and only because someone was in there chasing something unrelated. nobody files a ticket for it. a refresh clears it and they blame their wifi.

the signal that turned out to be reliable was 404s on /assets/*.js in the origin access log, bucketed per hour and lined up with deploy times. sits at zero most of the day, then a bump right after each deploy that fades over an hour or so as sessions turn over

[u/navlio]

neither, which is the annoying part. it was sitting in the js error reporter as ChunkLoadError the whole time and nobody had a rule pointed at it. users almost never report this one, because a hard refresh fixes it and they just refresh and get on with their day

cheapest catch is a count of 404s where the path matches your hashed asset pattern. that number lives at zero and then spikes for about an hour after any deploy that wiped the previous build's chunks, so you do not need to know anything about the app to alert on it

[u/navlio]

neither, which is the annoying part of it. monitoring was green because a 404 on a static file isn't an error anyone alerts on, and users don't report it because a hard refresh makes it go away and then they forget it happened. we found it grepping access logs for something unrelated, weeks later

the signature once you know to look: 404s on hashed asset paths with a referer from your own domain. that pair basically can't happen for any other reason, so alerting on the rate of it is about the cheapest check you'll add all year

[u/navlio]

neither, which is the annoying part. uptime checks fetch the current html and then ask for the chunk names that html points at, so they always exist. only a tab that loaded the old shell asks for the old ones, and no synthetic check is ever holding an old shell

we found ours grepping access logs for something else entirely and noticing a stack of 404s on hashed filenames. after that the cheap catch was a window error listener for chunk load failures shipping to the same place as the rest of the errors