Notification Outage
This summary is created by Generative AI and may differ from the actual content.
Overview
On April 12th, 2023 between 22:12 UTC to 23:20 UTC, PagerDuty experienced a service degradation that resulted in a small subset of users experiencing a delay in the delivery of SMS, phone, and push notifications. The email notification channel was unaffected during this time.
Impact
A small subset of users experienced delays in SMS, phone, and push notifications, with delays up to 20 minutes. The email notification channel was unaffected.
Trigger
An infrastructure (server) mis-configuration that prevented network communication between notification delivery services.
Detection
The teams noticed an increase in notification processing times.
Resolution
Service was restored by rolling back the configuration to a previous known state.
Root Cause
Infrastructure mis-configuration preventing network communication between notification delivery services.