This summary is created by Generative AI and may differ from the actual content.
Overview
On November 15th, 2022, from 06:55 UTC to 15:05 UTC, the Visibility Console service experienced frequent timeout errors due to a dependency issue, causing unresponsiveness for users in the US region.Impact
Customers were unable to load the Visibility Console or faced extremely high load times, affecting their ability to view content and dashboards.Trigger
The incident was triggered by high CPU and network bandwidth consumption by another service on a shared node, affecting the Visibility Console's indirect dependency.Detection
PagerDuty became aware of the issue at 13:10 UTC after receiving customer reports about the Visibility Console not loading.Resolution
The issue was resolved by restarting the indirect dependency at 14:50 UTC and the direct dependency at 15:04 UTC, restoring service by 15:05 UTC.Root Cause
The root cause was a combination of high resource consumption by another service on a shared node and increased load from a mobile app update and a library update, which delayed issue identification and remediation.
