RCA: Bitmovin Observability query and export outage (27 July 2026)

On 27 July 2026, Bitmovin Observability was unable to serve queries or data exports for approximately 20 minutes, with reduced data completeness for a further 24 minutes. Data ingestion was never interrupted, incoming data was buffered throughout, and no customer data was lost. This post explains what happened, what we have done, and what we are changing.

Impact

Between 12:33 and 12:53 UTC, the Bitmovin Observability query API, dashboards and data export were unavailable. Queries and exports resumed at 12:53. Data collected during the outage was buffered and then backfilled, completing at 13:17 UTC. Until that point, queries covering the affected period could return incomplete results. Data ingestion and collection were not affected at any time, and there was no data loss.

What happened

An internal query executed against our observability database cluster triggered a defect in the database engine. The memory used by this query was not correctly accounted for, which meant the cluster's memory safeguards never engaged. Instead of the query being limited or rejected, memory was consumed until nodes became unresponsive, and the failure propagated across the cluster rather than staying contained to the node serving the query.

Automated alerting notified our on-call engineers within 30 seconds. The cluster was restarted at 12:43 and completed recovery of all shards, resuming full write acceptance, at 12:53. Buffered data was then backfilled and fully available at 13:17.

Timeline (UTC)

What we are changing

• The query that triggered the defect was an ad hoc analytical query run by an operator, not a query our application issues in normal operation. We have instructed our engineering team not to run this class of query against the production cluster until the underlying defect is resolved.

• We are reporting the defect upstream and validating whether a fix is available in a newer version of the database engine.

Our ingestion buffering worked exactly as designed and protected customer data. We apologise for the disruption and for the incomplete data during the backfill window.

As of 15:17 CEST, the backfill has completed and all buffered data is available.
All systems are operational again. No data was lost.

We will provide a root cause analysis as soon as possible.

The database cluster has recovered. Querying (dashboards and API) and data export have resumed, and incoming data is being written normally again.

We are now triggering the backfill of the buffered data. Until it completes, recent data may be incomplete in queries and exports.

We are monitoring the results and will confirm once the backfill has finished.

Query operations are back to normal and we continue to monitor the situation. We apologize for any inconvenience caused due to this incident.

Root Cause Analysis — Encoding Configuration & Start Outage — 2026-07-07

Summary

On 07 July 2026, a non-backwards-compatible database migration was applied to a relational database schema at 11:40 UTC. The migration renamed several columns on a table. The currently running microservice responsible for the configuration of encoding resources still referenced the old column names in its entity mappings, so every read referencing the affected table began failing with a "Unknown column" error.

As a result, customers could not configure or start encodings. Affected API calls returned HTTP 500 responses. The first alert fired internally at 11:41 UTC, customer reports arrived shortly after. 

The root cause which was the non-backwards compatible database migration which has been applied was identified at 12:15 UTC, a migration reverting the previous change was applied at 12:31 UTC, and the incident was marked Resolved at 12:40 UTC.

Incident Timeline

Customer Impact

Customers could not configure and/or start encoding jobs. Affected requests returned an HTTP 500 Internal Server Error with error code 1004 — "An undefined error occurred". 

Root Cause

A migration renaming columns within a relational database table was applied to the production database.

An in-place CHANGE COLUMN rename was included in the database migration. The version of the service accessing the affected table running in production did still expect the old column names in its database entity mappings.

This bubbled up as an unhandled exception and was returned to callers as an HTTP 500 error. Given that the affected database serves as the primary repository for the encoding API, the resulting disruption was widespread, impacting configuration workflows for our entire customer base.

The database schema change and the corresponding service code change were not coordinated as a backwards-compatible, multi-step rollout. A schema-and-code change of this kind requires the database to remain compatible with the currently deployed service version at every step.

Next Steps and Preventive Actions

Detection & response

• Faster escalation — we've shortened the on-call escalation window so incidents reach the full response team within minutes of detection.

• Real-time error-rate alerting — we've reduced time to alert on API error rates to 2 minutes.

Prevention

• Formalized migration procedure — all database schema changes now follow a documented, mandatory playbook requiring backwards-compatible, multi-step rollouts, keeping the database compatible with the running service version at every step. This standard is applied uniformly across all engineering teams.

• Automated safeguards in CI and code review — we've added automated checks and compatibility tests that detect schema/service mismatches before a change can reach production.

The issue affecting Bitmovin Encoding has been resolved, and full functionality has been restored.

Our systems are stable, and we are no longer observing any errors or delays.

A Root Cause Analysis (RCA) will follow once our internal review is completed. Thank you for your patience throughout this disruption.

We apologize for all customers that were affected by this outage, unfortunately failed encodings cannot be restarted on our side but need to be recreated in your workflow.

A fix has been implemented, and we are currently monitoring the system to ensure full recovery.

Early indicators are positive, but we are validating to confirm that services are fully restored.

We will provide a final resolution update shortly.

We have identified the root cause and our Engineering team is actively working on implementing a fix for the identified issue impacting encoding start calls.

We are making progress and will share an update as soon as validation is underway or the fix is rolled out.

Thank you for your continued understanding.

This incident is now resolved. As of 17:32 UTC, all buffered data has been successfully backfilled into our central Observability data store, and ingestion has been operating normally on the alternative transport protocol since the failover at 16:52 UTC.

There was no data loss. Records buffered during the connectivity issue are now fully available for querying.

The incident began at 16:26 UTC and was fully resolved at 17:32 UTC. We apologize for any inconvenience and will follow up with a post-incident review once we have performed a full root cause analysis.

We have identified the cause of the connectivity issue between our US datacenter and our central Observability data store and have failed over to an alternative transport protocol. Real-time ingestion of US analytics data has been fully restored.

We are now draining the backlog of buffered data into the database. Some recently buffered records may appear with a short delay until this process completes. No data loss is expected.

We will continue to monitor the system and will post a final update once all buffered data has been backfilled.