Degraded Git Operations over SSH
This summary is created by Generative AI and may differ from the actual content.
Overview
On August 21, 2026, between 14:00 and 14:07 UTC, dotcom Git operations over SSH were degraded. Successful Git operations over SSH fell by more than 95% during the peak impact window, making clone, fetch, or push over SSH effectively unavailable to most users for approximately four minutes. Git operations over HTTPS were not affected. The incident was caused by a software defect in the load-balancing infrastructure that was triggered by a configuration change. The defect only occurred when connections passed through multiple layers of load balancers running the new configuration, which was not detected during canary testing. The incident was mitigated by rolling back the configuration change.
Impact
Successful Git operations over SSH fell by more than 95% during the peak impact window, making clone, fetch, or push over SSH effectively unavailable to most users for approximately four minutes. Git operations over HTTPS were not affected.
Trigger
A configuration change in the load-balancing infrastructure that triggered a latent software defect. The defect only manifested when connections passed through multiple layers of load balancers running the new configuration.
Detection
The degradation was detected during the incident window, leading to mitigation actions. The specific detection mechanism is not explicitly stated in the postmortem.
Resolution
The incident was mitigated by rolling back the configuration change. Future improvements include adding regression coverage for multi-layer load-balancer configurations and improving monitoring and alerting for Git operations over SSH to reduce time to detection and mitigation of similar issues.
Root Cause
A software defect in the load-balancing infrastructure that was triggered by a configuration change. The defect only occurred when connections passed through multiple layers of load balancers running the new configuration, which meant it was not detected during canary testing.