This summary is created by Generative AI and may differ from the actual content.
Overview
On September 4, 2025, between 15:30 UTC and 20:00 UTC (4 hours 30 minutes), GitHub experienced degraded REST API success rates for specific endpoints: git/refs, git/refs/*, and git/matching-refs/*. This incident primarily affected repositories with reference counts exceeding 22,000. The degradation was caused by a newly deployed code change that introduced latency to reference evaluations. The issue was mitigated by reverting the problematic code change.Impact
The incident resulted in an average request error rate of 0.5% for the affected REST API endpoints. Overall REST API availability remained high at 99.9999%. Customers with repositories containing over 22,000 references experienced 504 responses and elevated errors for 4 hours and 30 minutes.Trigger
The incident was triggered by the introduction of a new code change that added latency to reference evaluations, particularly impacting repositories with a high number of branches, tags, or other references.Detection
The incident was detected when customers began experiencing 504 responses for some API requests concerning repository references and tags. This led to an investigation by the GitHub team.Resolution
The incident was resolved by identifying and reverting the deployed code change that was causing the increased latency and 5XX errors. The rollback deployment was completed, leading to recovery for impacted customers.Root Cause
The root cause was a newly deployed code change that introduced additional latency to reference evaluations. This change had an outsized negative effect on REST API endpoints for repositories with a large number of references (over 22,000), leading to elevated error rates.
