This summary is created by Generative AI and may differ from the actual content.
Overview
From August 21, 2025, 11:00 UTC to August 22, 2025, 15:00 UTC, some PagerDuty customers in the US and EU service regions experienced errors when attempting to perform actions such as acknowledging, adding notes to, or resolving incidents using incident cards in Microsoft Teams channels. Customers also received disconnection messages. Interacting with incident cards in Microsoft Teams chats and meetings, receiving incident updates in channels, chats, and meetings, and all other incident response features across PagerDuty continued to work as expected.Impact
Some PagerDuty customers in the US and EU service regions were unable to perform critical incident management actions (acknowledge, add notes, resolve) through Microsoft Teams channels for approximately 28 hours (August 21, 2025, 11:00 UTC to August 22, 2025, 15:00 UTC). This led to disconnection messages and hindered efficient incident response via this specific integration.Trigger
On August 21, 2025, at approximately 11:00 UTC, PagerDuty deployed an update to the Microsoft Teams integration service. This update introduced a change in how the integration verifies the linking of a Microsoft Teams channel to the processing of that channel's user interactions.Detection
PagerDuty engineers became aware of the issue through customer reports, which led them to initiate the Major Incident process at 14:21 UTC on August 22, after confirming these reports.Resolution
At 14:46 UTC on August 22, PagerDuty engineers performed an emergency rollback of the problematic update. Functionality was confirmed to be fully restored at 15:00 UTC, and the incident was resolved at 15:04 UTC.Root Cause
The root cause was a change introduced in an update to the Microsoft Teams integration service. This update altered how the integration verifies the linking of a Microsoft Teams channel to user interactions, creating an issue stemming from a discrepancy in how different webhook versions handled this linking.
