Ingest Outage in US and EU
This summary is created by Generative AI and may differ from the actual content.
Overview
On June 24, 2026, Honeycomb experienced a roughly 30‑minute severe degradation of data ingestion in its US and EU instances (impact window 17:55–18:17 UTC). A deployment that altered how services retrieve dataset schemas into local caches introduced a serialization change that caused remote cache reads to fail, leading to a MySQL stampede, rapid memory growth, and crash‑looping of ingestion, anomaly detection, and service‑maps services. The outage resulted in 5xx errors from ingestion APIs, partial telemetry loss, and downstream disruptions to triggers, SLO calculations, and Service Maps. The incident was resolved by rolling back the deployment to the previous build, after which services recovered and normal operation resumed.
Impact
The degradation caused ingestion services to reject a significant percentage of inbound telemetry, creating up to a half‑hour gap in ingested data. Customers saw 5xx errors from the ingestion APIs, and downstream services (Anomaly Detection, Service Maps) entered crash loops due to out‑of‑memory exceptions. Consequently, triggers may have failed to fire, SLI values were skewed, and Service Maps for the outage window were incomplete, leading to partial data loss and reduced observability during the incident period.
Trigger
A recent deployment introduced a change to the serialization of dataset schema cache payloads and metadata stored in memcached. The change included a feature flag and backward‑compatibility safeguards, but the deserialization logic incorrectly treated backward‑compatible schema versions as hard errors. This caused every remote cache read to fall back to the database, generating a MySQL stampede and overwhelming memory, which cascaded into service crashes.
Detection
Monitoring systems generated alerts for the crash‑looping services shortly after the failure began. Engineers observed elevated 5xx error rates from the ingestion APIs and identified that ingestion services were repeatedly restarting due to out‑of‑memory conditions. The alerts prompted the on‑call team to investigate and confirm the degradation.
Resolution
The on‑call team rolled back the problematic deployment, reverting all affected services to the last known good build. This stopped the database stampede, reduced memory pressure, and allowed the crash‑looping services to recover. After stability was restored, a corrected version of the serialization change was redeployed with additional tests to ensure proper backward‑compatible deserialization.
Root Cause
The root cause was a bug in the schema deserialization code introduced by the deployment. Although a feature flag and compatibility checks were present, the code mistakenly treated backward‑compatible schema reads as fatal errors, causing every remote cache retrieval to bypass memcached and query MySQL. The resulting database stampede and memory exhaustion led to service crashes and the observed ingestion outage.