Loading
This summary is created by Generative AI and may differ from the actual content.
On October 20, 2025, starting at approximately 07:00 UTC, PagerDuty experienced service degradations affecting Stakeholder Notifications, Incident Notifications, and the PagerDuty Scribe Agent. Customers in India experienced missed voice notifications from the US region between 07:00 UTC and 07:50 UTC. Users outside India experienced minor delays in voice and SMS notifications and missed stakeholder notifications from the US region between 07:00 UTC and 08:40 UTC. Some customers in the EU and US regions missed Slack notifications between 08:39 UTC and 09:23 UTC. The PagerDuty Scribe Agent was unable to join calls for transcription in EU and US regions from 07:00 UTC to 19:49 UTC. Mobile push, email notifications, and EU voice/SMS deliverability were not impacted. The incident was triggered by a significant internet outage that degraded several notification dispatch routes. Voice and SMS notification issues were resolved by adjusting notification routes, while the Scribe Agent functionality recovered once the underlying internet outage was resolved. PagerDuty plans to improve its route selection algorithm, enhance feedback/monitoring of route configuration, improve Scribe Agent error messaging, and explore redundant hosting for the Scribe Agent's transcription service.
Customers in India experienced missed voice notifications from the US service region between 07:00 UTC and 07:50 UTC. Users outside India experienced minor delays in voice and SMS notifications, as well as missed stakeholder notifications from the US service region between 07:00 UTC and 08:40 UTC. Some customers in the EU and US Service Regions missed Slack notifications between 08:39 UTC and 09:23 UTC. The PagerDuty Scribe Agent was unable to join new incident calls to perform transcription in the EU and US Service Regions from 07:00 UTC to 19:49 UTC. Mobile push, email notifications, and voice/SMS deliverability for the EU Service Region were not impacted.
The incident was triggered by a significant internet outage that occurred at approximately 07:00 UTC on October 20, 2025. This outage resulted in partial degradation of several routes PagerDuty uses to dispatch SMS and voice notifications from the US Service Region.
PagerDuty detected a potential issue with delayed notifications in the US regions around 07:23 UTC (4:23 PM GMT+9) and began investigating shortly after the internet outage commenced.
Voice and SMS notification issues were resolved by adjusting notification routes during an internal incident call, based on deliverability measurements across providers and geographies. The PagerDuty Scribe Agent functionality recovered once the underlying internet outage was resolved at approximately 19:49 UTC.
The primary root cause was a significant internet outage that occurred at approximately 07:00 UTC on October 20, 2025, which caused partial degradation of several routes used to dispatch SMS and voice notifications. Contributing factors included an insufficient route selection algorithm that did not effectively retry across different providers, leading to retry exhaustion for some notifications, and a lack of redundant options for the PagerDuty Scribe Agent's underlying transcription service.