Incident with Pull Requests and API Requests

Severity: MajorCategory: MisconfigurationService: GitHub
This summary is created by Generative AI and may differ from the actual content.
Overview
On December 3, 2024, between 19:35 UTC and 20:05 UTC, GitHub experienced degraded performance in API requests, Actions, Pull Requests, and Issues due to an erroneous database credential change. The incident was mitigated by reverting the credential change at 19:52 UTC, and full recovery was monitored until resolution at 20:05 UTC.
Impact
Web and API requests for Pull Requests experienced a 3.5% error rate, Issues had a 1.2% error rate, and Actions had a 3.3% error rate in jobs, affecting users creating and commenting on Pull Requests and Issues.
Trigger
The incident was triggered by an erroneous database credential change impacting write access to Issues and Pull Requests data.
Detection
The issue was detected through reports of degraded performance for API Requests, Actions, and Pull Requests.
Resolution
The incident was mitigated by reverting the credential change at 19:52 UTC, and an alternative method was used for rollback after the initial attempt was unsuccessful. Monitoring continued until full recovery was confirmed at 20:05 UTC.
Root Cause
The root cause was an erroneous database credential change that impacted write access to Issues and Pull Requests data.
;