Degradation with Webhooks, Pull Requests and Actions
Severity: Major
Category: Scalability
Service: GitHub
This summary is created by Generative AI and may differ from the actual content.
Overview
Degradation with Webhooks, Pull Requests, Actions, and Issues, with delays peaking at 7m, between 17:33 UTC and 17:55 UTC, and recovery by 18:07 UTC
Impact
Delays to pull requests, repository pushes, Actions workflows, and Webhooks, with delays peaking at 7m
Trigger
Incoming traffic spike and underlying hypervisor issues
Detection
Investigation started after reports of degraded performance for Actions, Pull Requests, and Webhooks
Resolution
Replacing the problem node at 17:49, after which all services saw recovery by 18:07
Root Cause
Underlying hypervisor issues and an incoming traffic spike, causing service timeouts which led to a connection storm and continual rebalances