Make AirflowRunFacet more reliable for OpenLineage consumers

[kacpermuda]

Problem Statement

The AirflowRunFacet (and AirflowDagRunFacet, which is built in the same way) in the OpenLineage provider provides valuable metadata to OpenLineage consumers. While a spec file exists here, most fields are not required by it, and the final format of some fields is not consistently enforced by the OpenLineage integration. Instead, in some cases, the facet simply reflects the structure and data of the Airflow core models, which can vary between Airflow versions but also can present some information differently depending on whether we are dealing with single or multiple objects (see example below). This approach makes the facet prone to unexpected changes and inconsistencies, as it directly mirrors Airflow’s internal representation at any given moment.

For example:

• If Airflow changes a field type (e.g., a string becomes a dictionary), OpenLineage can propagate that change without warning, even if it contradicts the spec.
• If an attribute in Airflow’s DAG model is renamed, it may lead to missing or altered information in OpenLineage events.

Consumers of OpenLineage data must account for not only different OpenLineage versions but also varying Airflow versions, making integration more fragile compared to other facets, which are explicitly defined in the spec and backward-compatible.

We have some tests to verify how some parts of this facet will look like, but I'm not sure if we check cross-Airflow version stability of this facet so well. Also, there is always room for more tests, so I'll look into that.

Example
This issue caught my attention while testing how a DAG’s schedule is represented in OpenLineage events. As seen in #47150, the representation can vary. The dag.timetable field:

1. For single dataset it can look like

{"dataset_condition": {"__type": "dataset", "uri": "a", "extra": None}}

2. For multiple datasets it can look like (now we have additional objects key with a list of datasets):

{ "dataset_condition": { "__type": "dataset_all", "objects": [ {"__type": "dataset", "extra": None, "uri": "a"}, {"__type": "dataset", "extra": None, "uri": "b"} ] } }

And in Airflow 3 the use of dataset will be renamed to asset, so:

{ "asset_condition": { "__type": "asset", "uri": "a", "name": "a", "group": "asset", "extra": {} } }

This is just an example (and it could be somehow standardized on the serialization level in core Airflow), but it made me think if we can / should try to provide a more consistent experience for the OL consumers regardless of core Airflow.

Proposed Solution

We could consider the following approaches:

1. Be more strict about defining and enforcing the AirflowRunFacet specification

• Verify if the spec for AirflowRunFacet is relevant and up to date.
• Ensure that OpenLineage extracts and processes the relevant metadata to some stable form before including it in events, rather than mirroring Airflow’s internal model.
• Introduce validation mechanisms to enforce the correct structure, reducing the risk of breaking changes.

2. Extract crucial Information into separate, well-defined facets

• Identify the most important data points from AirflowRunFacet and move them into separate OpenLineage facets with a stable structure.
• Ensure these new facets are independent of direct changes in Airflow’s internal representation, enhancing long-term reliability.

This approach would admittedly place more burden on OpenLineage to handle differences across Airflow, but it would shift complexity away from consumers and ensure that OpenLineage events remain stable and predictable - ultimately making the ecosystem more robust and easier to integrate with in the long run.

Would love to hear your thoughts on whether you think this is a real problem and, if so, if OpenLineage should solve it and what other solutions might be worth considering.

CC @mobuchowski @JDarDagran