Delayed Event Processing
This summary is created by Generative AI and may differ from the actual content.
Overview
Between 22:23 and 23:06 UTC on March 26, 2026, a subset of PagerDuty customers in the US service region experienced delays in event processing and notification delivery due to a performance regression in the event transformation service following a dependency upgrade.
Impact
Approximately 3% of US accounts experienced notification delays with a mean delay of about 13 minutes. Service integrations were affected, while global orchestrations and other PagerDuty functionalities remained operational. No events were lost.
Trigger
The promotion of a dependency upgrade for the event transformation service to the US production environment at 22:14 UTC.
Detection
Monitoring systems detected processing delays, elevated backlog, and throttling shortly after the impact began at 22:23 UTC.
Resolution
Engineering teams initiated a rollback of the affected components at 22:56 UTC, which completed at 23:06 UTC, restoring service health and processing all queued events.
Root Cause
A dependency upgrade introduced a performance regression that caused increased resource consumption under sustained load. This issue was not detected during the graduated rollout in pre-production and regional production environments but reached system limits when exposed to the higher traffic volumes of the US production environment.