Investigating a Potential Issue
This summary is created by Generative AI and may differ from the actual content.
Overview
On February 13th, 2023, from 14:03 UTC to 14:13 UTC, PagerDuty experienced a service degradation that resulted in a small subset of users experiencing delayed email notifications in the US service region. This affected email notifications only. Other notification channels such as Push, SMS, Voice and Slack were unaffected during this time.
Impact
A small subset of users experienced delayed email notifications in the US service region, with delays ranging from 3-10 minutes. Other notification channels were unaffected.
Trigger
An increase in notification processing times due to a new configuration being rolled out to the notification system.
Detection
The issue was noticed at 13:55 UTC when there was an increase in notification processing times, leading to a backup of notifications.
Resolution
Service was restored by rolling back the configuration to a previous known state at 14:15 UTC.
Root Cause
The root cause was a new configuration that increased notification processing times, leading to a backup of notifications.