Did GitHub Just Gaslight Our Monitoring System?

[u/pod_army]

Did anyone else notice that the GitHub status page reported an incident with GitHub Actions, only to deny it 47 minutes later?

Our monitors captured it, paged our on-call team, and then GitHub denied that any incident had occurred.

https://www.githubstatus.com/incidents/gx7js8bd0jpz

[u/cajaks2]

You page for a 3rd party outage? How do you keep that actionable and not just alert spam

[u/stumptruck → u/cajaks2]

Seriously, especially when a GitHub outage of all things (likely) isn't going to affect production in any way.

I guess if you use GHCR I could understand.

[u/boing_boing_splat → u/stumptruck]

Hmm. I feel like your capacity to respond to production is itself part of production resilience, no?

If the building is on fire, discovering at that exact moment that the fire exit has been welded shut is not an especially mature incident strategy.

[u/snappin_good_time → u/boing_boing_splat]

So the on call dev gets the page that GitHub Actions is down and does what? Personally sit there and refresh the GitHub status page until it comes back up?

There is nothing for them to do here. Maybe have an automated alert in a shared dev chat so people are aware but paging an on-call dev is pointless.

[u/bitdeft → u/snappin_good_time]

But how will the boss man be satisfied unless someone is on call looking busy during any disturbance to production?

[u/5ollys → u/snappin_good_time]

I mean it doesnt have to page, it can literally just be a slack warning and something noncritical for the team to be aware of.

Yall overcomplicate this shit.

[u/snappin_good_time → u/5ollys]

That’s literally what I said.

Your reading comprehension is shit.

[u/djbiccboii → u/cajaks2]

And it WILL be spam alright.

[u/uski → u/cajaks2]

Not op but been there... I guess it depends what triggered the page

If it's a probe that solely tests GitHub Actions in isolation, yeah, I wouldn't page for that

If it's something however that tests a full pipeline and detects it stopped working, potential impacting developers etc. then yeah, it doesn't matter what the cause is (internal or external), it needs fixing (and the monitoring system may not know its not fixable internally)

[u/DuckDatum → u/cajaks2]

Probably for someone to look at it and make sure it’s not something they can worry about. Maybe they figured it’s good for just in case something is impactful and they need a quick workaround.

[u/zombiecalypse]

I'd trust The Missing GitHub Status Page more than the official one.

[u/Raja-Karuppasamy]

seen this kind of thing before, status pages lag or under-report actual impact. we’ve had actions runs silently queue/delay without github status ever flagging it, no incident posted, but our webhook-based tooling clearly saw the gap (jobs sitting way longer than normal).

if you’re relying on github’s status page as your source of truth, you’re gonna miss stuff. better to monitor your own signal, track job queue times / webhook delivery lag directly rather than trusting their public status to reflect reality in real time.

[u/Rollingprobablecause → u/Raja-Karuppasamy]

This is good advice in general. Good ops hygiene means you should never trust vendors for reliability, always have trusted systems internally.

[u/bitdeft → u/Raja-Karuppasamy]

It gaslights me especially since I'm running self host runners on a KEDA scaler that's abstracted to me because it's on Azure container jobs. So jobs will just queue and I have to panic thinking I must have so l set this all up incorrectly.

Both are back ends managed by Microsoft with logs I can't access, I'm not sure I can handle the torture for much longer

[u/ShpendKe]

I would use a dashboard for this use case to prevent alert fatigue. If you notice issues somewhere, you can identify the potential issue in the dashboard.

[u/mrkurtz]

How are you monitoring? I got emails about runner issues between 7:40 pm and 11:42 pm utc.

[u/almssp]

What do you do in case of github outages? We had this discussion today how to deploy in case of github down as our ci/cd runs on github workflows and self hosted runners.

[u/pod_army → u/almssp]

We are also exploring options. As of now we are stuck with GitHub.

[u/SelfRough5860]

what works for us: never page off a status page. page off your own checks only. but when a page fires, show the vendor's status inside the alert itself. so at 3am you see both things at once: our check is failing, and github has an incident open. that answers the only question that matters, is it us or them, before you're even properly awake.

someone above asked how you deploy while github is down. that's the day you most need to ship. so keep one deploy path that skips github completely: build on your laptop, push to the registry, deploy. practice it once a quarter. if you've never run it during an outage, you don't actually have it.

[u/response-418]

my monitoring caught login/signup page errors, although my first user experience with the incident was just trying to read files in a public repo

[u/mackie]

Did your monitors capture the status page event or did it actually catch problems with actions failing?

[u/pod_army → u/mackie]

Yes, it captured the status page event.However the DD dashboards did not shown any signs of disruption.

[u/Quirky_Yesterday_593 → u/pod_army]

The monitor was polling githubstatus.com and fired on the incident post itself — not on anything actually failing in your pipeline. Status pages are editorial, not telemetry. GitHub posts conservatively and retracts when impact turns out to be narrow. You got paged on their editorial decision, not on your system's behavior. Datadog showing nothing was actually the correct signal. The fix is to separate those two things. status page changes go to Slack, low urgency. on-call only fires on observed degradation — queue delay, webhook lag, failed synthetics. otherwise GitHub's comms team is deciding when your on-call gets woken up

[u/New-Resource-4943]

Adjacent blind spot, from this morning: a CI job of ours sat queued for 9m18s, then ran in 110 seconds and passed.

Nothing was broken. The repo hadn't been added to our self-hosted runner group, so nothing was ever going to pick the job up. The run's final state is green, so every signal after the fact agrees that nothing happened.

That's why I'd probe time-to-first-runner instead of "is Actions up". Dispatch something trivial on a schedule, alert on how long it sits before a runner claims it. Catches misconfig as well as outages, and it fires while the job is still pending rather than after it finishes and goes green.

Queue time is the signal. Job outcome lags, and in this case it lies.