Delays to Escalations and Responder notifications
This summary is created by Generative AI and may differ from the actual content.
Overview
Between 23:03 UTC on June 9 and 01:13 UTC on June 10 2026, a subset of PagerDuty customers in the US service region experienced processing delays of up to ten minutes for timed escalations, responder requests, and scheduled status‑update notifications. Core product functions (incident notifications, event ingestion, incident creation) and the EU service region remained fully operational. The delays were caused by an unexpectedly high volume of Events API requests that saturated the asynchronous background‑processing worker fleet, creating a backlog that was later cleared after manual scaling and rate‑limiting actions.
Impact
The impact was limited to a small number of US‑region customers who saw up to ten‑minute delays in escalation, responder invitation, and status‑update notifications. All other PagerDuty functionalities—including direct incident notifications via SMS, phone, and the mobile app—continued to operate normally, and the EU region experienced no degradation. No outage of core services occurred.
Trigger
A higher‑than‑anticipated surge of requests to the PagerDuty Events API generated a heavy processing load on the asynchronous background‑processing workers. Because CPU utilization remained within normal bounds, the autoscaling mechanism did not provision additional capacity, allowing the request queue to grow and latency to increase.
Detection
Automated production tests monitoring task latency flagged a rise in processing latency at 23:30 UTC, which triggered an internal alert to the engineering team. Earlier, latency metrics had begun to climb at 23:03 UTC, indicating the formation of a backlog before the alert was raised.
Resolution
Engineers manually increased the number of background‑processing workers and applied targeted rate limits to the originating Events API traffic. The backlog began clearing immediately, and by 01:13 UTC the system returned to baseline latency. The incident was declared resolved after continued monitoring confirmed stable operation.
Root Cause
The autoscaling policy relied solely on CPU metrics, which failed to detect the growing queue caused by the spike in Events API traffic. Consequently, the background‑processing fleet lacked sufficient capacity to handle the load, leading to a backlog and the observed notification delays.