This summary is created by Generative AI and may differ from the actual content.
Overview
Multiple Cloudflare services, including R2 object storage, experienced an elevated rate of errors for 1 hour and 7 minutes on March 21, 2025 (starting at 21:38 UTC and ending 22:45 UTC). While rotating credentials used by the R2 Gateway service (R2's API frontend) to authenticate with our storage infrastructure, the R2 engineering team inadvertently deployed the new credentials to a development instance of the service instead of production. When the old credentials were deleted from our storage infrastructure (as part of the key rotation process), the production R2 Gateway service did not have access to the new credentials. There was no data loss or corruption that occurred as part of this incident: any in-flight uploads or mutations that returned successful HTTP status codes were persisted.Impact
During the incident window, 100% of write operations failed and approximately 35% of read operations to R2 failed globally. Although this incident started with R2, it impacted other Cloudflare services including Cache Reserve, Images, Log Delivery, Stream, and Vectorize. The impact on read availability was significantly mitigated by our intermediate cache that sits in front of storage and continued to serve requests.Trigger
Old credentials were removed from our storage infrastructure to complete the credential rotation process, which caused authentication errors, as the R2 Gateway production Worker still had the old credentials. This is ultimately what resulted in degraded availability.Detection
R2 global availability alerts are triggered (indicating 2% of error budget burn rate).The R2 engineering team began looking at operational dashboards and logs to understand impact.Resolution
Deployed credentials to correct production Worker. R2 availability recovered.Root Cause
due to human error, we inadvertently omitted the --env parameter and deployed the new storage credentials to the wrong Worker (default environment instead of production).
