Incident with Webhooks
This summary is created by Generative AI and may differ from the actual content.
Overview
Between 14:24 and 14:53 UTC on 13 August 2026, a routine background job to delete an organization overwhelmed a key shared database, causing multiple GitHub services to return elevated errors and slower responses. Most affected was the webhook management API, with smaller impact to Git operations, pull requests, issues, and packages. The impact cleared on its own at 14:53 UTC once the job finished, and the incident was fully resolved at 15:36 UTC.
Impact
Failures peaked at approximately 1% for several minutes around 14:37 UTC. Affected users experienced brief increases in errors and slower responses, primarily when creating, listing, or updating webhooks, with smaller impacts to pull requests, issues, packages, and Git operations.
Trigger
A routine background job to delete an organization overwhelmed a key shared database, causing cascading degradation across multiple services.
Detection
Monitoring systems detected degraded performance starting at 14:45 UTC for webhooks. Subsequent detections occurred at 14:46 UTC for pull requests, issues, and Git operations, and at 14:56 UTC for packages. The source of degradation was identified by 14:58 UTC.
Resolution
The background job was temporarily disabled at 15:33 UTC, which fully mitigated the impact. An update was shipped that turns on the safer deletion path for organizations, along with caps on deletion holds on databases. All bulk deletion and cleanup jobs that write to shared databases are being audited to prevent similar issues in the future.
Root Cause
A routine background job for organization deletion used an unsafe deletion path that overwhelmed a key shared database with excessive load, causing cascading failures across multiple dependent services. The issue was exacerbated by the lack of caps on deletion holds on the shared database.