Delays in Incident Workflow Actions that use JavaScript steps
This summary is created by Generative AI and may differ from the actual content.
Overview
Between December 13, 2025 at 21:40 UTC and December 15, 2025 at 20:02 UTC, a small number of PagerDuty customers in the US service region experienced issues with Incident Workflows containing the JavaScript: Run Code action. Impacted customers experienced workflows that either failed to complete or experienced delays in execution.
Impact
A small number of customers were impacted, but the exact number is not specified. All other incident response functionality and notifications were unaffected.
Trigger
A code change introduced a bug to the service that handles Incident Workflow Action invocations on December 3, 2025 at 15:15 UTC, which was triggered by a spike in invocations of the JavaScript: Run Code action type on December 13 at 21:40 UTC.
Detection
Our internal monitoring system did not alert on-call responders to the increase in pending action invocations until December 14 at 12:54 UTC. The issue was further identified after receiving two customer reports on December 15.
Resolution
We manually triggered our major incident process to coordinate our response on December 15 at 18:14 UTC. We removed queued tasks older than 2 hours and rolled back the deployment to the version before the bug introduction on December 3. All JavaScript: Run Code actions that were triggered between December 13 at 21:40 UTC and December 15 at 16:41 UTC were not processed and caused the workflow that triggered them to enter a failed state. JavaScript actions triggered between December 15 at 16:41 and 18:41 UTC were eventually processed after a delay.
Root Cause
The root cause of the incident was a bug introduced by a code change on December 3, 2025, which prevented additional jobs from being processed once an internal concurrency limit was reached.