Incident with Webhooks
This summary is created by Generative AI and may differ from the actual content.
Overview
On January 9, 2025, between 01:26 UTC and 01:56 UTC, GitHub experienced a disruption due to a deployment introducing a query that saturated a primary database server, affecting Git Operations, Webhooks, API Requests, Issues, Pull Requests, Actions, Pages, Codespaces, and Copilot. Services started recovering around 02:09 UTC.
Impact
Experienced widespread disruption to many services, with an average error rate of 6% and a peak at 6.85% of update requests.
Trigger
A deployment which introduced a query that saturated a primary database server.
Detection
Observed 500 responses when users tried to access various functionalities.
Resolution
Identified the source of the problematic query and rolled back the deployment. Deployed a fix and the majority of the services have recovered.
Root Cause
A deployment introduced a query that saturated a primary database server.