npm private modules outage on December 12th
This summary is created by Generative AI and may differ from the actual content.
Overview
npm private modules outage. All read and write access to private packages was interrupted. customer data was not in perfect sync with the source of truth-Stripe, our payments provider-and some customers renewed their subscriptions when they did not need. The underlying cause of this outage was a software bug that deleted all records from our customer database. We have now fully reconciled our customer database with Stripe's data, and have refunded all renewals that were made during the incident. We apologize to all of npm's customers for this outage.
Impact
All read and write access to private packages was interrupted. customer data was not in perfect sync with the source of truth-Stripe, our payments provider-and some customers renewed their subscriptions when they did not need to.
Trigger
The bug was deployed to production, it was triggered the first time the tool was used in this way. A DELETE with no WHERE clause was executed and all records were deleted from the targeted table.
Detection
Our support team escalates a Twitter complaint & npm engineering is alerted.
Resolution
We roll back the deploy to the relevant microservice. We deploy a mitigation to our payments service that treats all customer accounts as being in good standing, without checking the database. We clear the access cache entirely and access to private packages is restored for all customers. We restore the database from a backup.
Root Cause
a bug in the query builder software we use to interface with postgres, ormnomnom. Due to a bug in ormnomnom's 'delete' method, it omitted the WHERE clause for the query built by the list of filters.