Data Movement in Netflix Studio via Data Mesh

data-meshchange-data-capturedata-engineeringstreaming-pipelinesnetflix

Abstraction: Netflix Studio CDC streaming pipelines via configuration-driven Data Mesh

Key points:

Netflix Studio uses a Data Mesh platform for Change Data Capture (CDC) from MySQL/Postgres, enriched via GraphQL queries to Studio Edge, then landed in Apache Iceberg tables
Earlier approaches (direct ETL then event-driven Delta) had issues with tight coupling, schema stale data, and high learning curves; Data Mesh solved these with self-serve UI and end-to-end schema evolution
Processors are reusable, configuration-driven components (source connectors, GraphQL enrichment, Iceberg sink); schema evolution uses opt-in/opt-out flags to propagate upstream changes automatically
Genesis, a Node.js CLI semantic data layer, generates SQL for 240+ studio trackers from YAML Data Source Definitions, replacing hand-curated SQL scripts managed by a Java service called Lego
Data quality is ensured via hourly end-to-end audits (symmetric difference checks on primary keys) and synthetic heartbeat events to verify pipeline health independent of traffic patterns

Connections: Netflix · Apache Iceberg · Data Mesh · Change Data Capture · Streaming Data Pipelines