Describe the process and tools you use for data lineage tracking.

Official answer

Lineage records which inputs and transformations produced an output. Distinguish design lineage, such as a declared table dependency, from observed run lineage tied to an execution and the versions or partitions actually used.

Capture stable dataset and job identities, run ID/status, code version, input/output partitions or snapshots, and relevant schema changes. OpenLineage models jobs, runs and datasets; integrations can emit events into a compatible backend. A catalog displays only lineage that its connectors or instrumentation actually capture. Installing a discovery tool does not automatically trace every field through custom code.

For daily active users, trace raw activity to identity mapping, eligibility filters, timezone/day boundaries, deduplication, distinct-user aggregation and the dashboard definition. Table-level edges may be insufficient to explain how one metric field was computed; add column or transformation detail where required.

Validate coverage with a known sample pipeline, including failed/retried runs, dynamic SQL and renamed datasets. Mark inferred or missing edges explicitly. Use lineage for impact analysis and incident investigation, while protecting sensitive metadata and SQL parameters. It supports governance but does not prove data accuracy, complete capture or legal compliance.