Design a system for tracking data lineage in a complex data ecosystem

Instruction: Outline a system for tracking data lineage across multiple data sources, transformations, and storage systems in a complex data ecosystem.

Context: This question checks the candidate's ability to design systems for maintaining data lineage, crucial for data governance, quality, and auditing purposes.

Updated

Official answer

Define the lineage questions: impact of a changed column, reproduction of a report, or tracing a failed partition. Model stable dataset, job and run identities, plus version/partition references and transformation edges. Keep design-time declarations separate from evidence of an actual run.

Instrument supported engines/connectors and emit events to a durable collector. Validate event schemas, deduplicate deliveries, handle out-of-order start/completion events and retain provenance. OpenLineage provides an interoperable job/run/dataset model; custom transformations still need suitable instrumentation or explicitly labeled inference.

Store queryable relationships with time/version scope so a new run does not erase the history required to explain an older report. A graph view can support upstream/downstream traversal, but column-level logic or dynamic SQL may be unavailable. Show unknown coverage instead of drawing unjustified precise edges.

Protect metadata because names, schemas and query text can reveal sensitive information. Separate access to lineage from access to underlying records, sanitize secrets and define retention. A centralized service also needs backup, replay and failure monitoring.

Test against known pipelines, including retries, renames, failed writes and dropped events. Measure verified connector coverage, capture delay, missing edges and agreement with observed execution. Automation reduces manual work; it does not guarantee comprehensive or accurate lineage.

Related Questions