Incident with Webhooks

Severity: MajorCategory: Change ProcessService: GitHub

This summary is created by Generative AI and may differ from the actual content.

Overview

On January 9, 2025, between 01:26 UTC and 01:56 UTC, GitHub experienced a disruption due to a deployment introducing a query that saturated a primary database server, affecting Git Operations, Webhooks, API Requests, Issues, Pull Requests, Actions, Pages, Codespaces, and Copilot. Services started recovering around 02:09 UTC.

Impact

Experienced widespread disruption to many services, with an average error rate of 6% and a peak at 6.85% of update requests.

Trigger

A deployment which introduced a query that saturated a primary database server.

Detection

Observed 500 responses when users tried to access various functionalities.

Resolution

Identified the source of the problematic query and rolled back the deployment. Deployed a fix and the majority of the services have recovered.

Root Cause

A deployment introduced a query that saturated a primary database server.

See the original post