While the pipeline is green, CI looks like a free add-on to the repository. October 5–7, GitHub proved it is production: four critical incidents in three days.
The best-documented one hit on October 5, 18:48–22:49 UTC. Over the four-hour incident, 14.3% of workflow runs on GitHub-hosted runners and 26.5% of individual jobs missed the five-minute start target; during the ~90-minute peak, 47.0% of runs failed outright and 71.9% of jobs did not start on time. The root cause was a partial networking failure between one of GitHub's own datacenters and a subset of cloud-hosted regional database and storage services; shifting traffic to healthy regions and adding capacity fixed it. Self-hosted runners were untouched. The streak then continued: a critical on October 6 (several services degraded) and two more on the 7th (Git operations, Issues, Actions, Pull Requests).
A detail for architects — calmly, with no gloating: the root cause was connectivity between GitHub's own datacenter and its cloud services. A textbook hybrid topology failed at the very platform that hosts CI/CD for thousands of teams. Nobody is perfectly insulated from this class of failure, so the question is not "will it happen" but "what do we do when it does."
What this means for a team lead. First, price an hour of pipeline downtime: developers × hourly rate × hours of waiting. That is an honest lower bound: missed release windows and the cost of cold task context are not even in it. One number turns CI resilience from an abstraction into a budget line. Second, hybrid hosted/self-hosted runners are no longer just a cost play. On October 5 they were the dividing line between a 47% failure rate at the peak and zero impact — keeping some capacity in-house is now a reliability argument, not an accounting one. Third, having no runbook for a GitHub outage is tech debt, not paranoia. It holds three boring items: what happens to the merge queue, how release windows shift, and where artifact and dependency mirrors live. You write that document once and reach for it on the worst Thursday of the quarter.
A platform's SLA is the vendor's promise, not your continuity plan. A plan is something your team holds before the incident; afterwards, all that is left is improvisation.
📎 Sources: the October 5 report; October 6–7 incidents: cjwszcf57bff, qpfv5p86dmrl, djlmxz2zd0j7