Incident with Webhooks
This summary is created by Generative AI and may differ from the actual content.
Overview
On June 11 2026, between 19:28 UTC and 21:06 UTC, GitHub webhook deliveries experienced significant latency spikes, with average latency reaching ~3.4 minutes and the 99th‑percentile latency up to 62 minutes. No events were lost; delayed events were queued and later delivered once processing caught up. The degradation stemmed from a recent change that shifted a portion of webhook traffic to a different region to relieve load, but the target region’s higher latency prevented delivery workers from keeping pace, creating a backlog. The incident was mitigated by rebalancing traffic distribution, after which the backlog drained and normal service resumed.
Impact
Webhook consumers faced delayed event deliveries, with latency increasing to several minutes and up to an hour for the worst‑case 99th‑percentile. While no events were lost, the delay impacted downstream processing and user experience. The incident lasted roughly 1 hour 38 minutes, affecting all webhook traffic during that window.
Trigger
A configuration change that altered webhook traffic distribution across regions. To offload a heavily loaded region, processing was moved to another region that exhibited higher latency, which caused the delivery workers to fall behind incoming volume and generate a backlog.
Detection
Monitoring systems detected degraded webhook performance, flagging increased delivery latency and a growing backlog of queued events. Alerts triggered investigation into the slowdown.
Resolution
Traffic distribution was rebalanced to restore processing to the original region, allowing delivery workers to catch up. As load normalized, the backlog cleared and webhook latency returned to normal levels. Ongoing monitoring was kept in place to verify stability.
Root Cause
The root cause was the regional traffic redistribution that sent webhook processing to a higher‑latency region, overwhelming the delivery workers and creating a backlog. This configuration change unintentionally introduced latency that the system could not handle under the existing load.