OPENLINEAGE

Open-source data lineage capture standard.

Definition

OpenLineage spec defines Run, Job, Dataset, Lineage events (START, COMPLETE, FAIL, ABORT, OTHER) in JSON. Integrations: Spark, Airflow, Flink, dbt, Great Expectations. Main backends: Marquez (reference), DataHub, Atlan, Collibra.

Origin

Launched by WeWork (Astronomer) 2020 ; donated to LF AI 2021.

Example in context

An Airflow DAG automatically sends OpenLineage events to Marquez on every task run.

Related terms

• Marquez — reference backend.