2014-01-28 Outage Postmortem
Severity: Major
Category: Dependencies
Service: npm
This summary is created by Generative AI and may differ from the actual content.
Overview
While making a change to simplify the Varnish VCL config on Fastly, a bug was added that caused all requests to go to Manta, including those that should have gone to CouchDB.
Impact
all requests to go to Manta, including those that should have gone to CouchDB
Trigger
change to simplify the Varnish VCL config on Fastly
Detection
proliferation of 403 responses led to a thundering herd
Resolution
setting up a separate Varnish instance to stage changes
Root Cause
setting the req.backend in a vcl_fetch function, and then calling restart to re-parse the rules