DagRun start_date is being reset after clearing the task. Any alternative?

[samvel1024]

Hey 👋

I was trying to get the "physical" execution date of the dag (i.e the actual time when the dag was triggered, nothing to do with the data interval or the schedule). If I recall correctly in the previous versions dag_run.start_date was a reliable way to get that timestampt. I observed that it was the same for all tasks and attempts within the same dag run (doesn't matter if it's manual or scheduled it always represents the time when dag was physically triggered).

However, I just realized that in one of the updates (maybe related to this #18708)) this behavior is changed. When a task fails and I clear it the start_date of the dag is changing. Logically, this does not make sense to me because start_date is a property of dag_run and it's something that already happened.

• I want to understand what is the reason behind this change.
• More importantly, it would be great to know if there's a better alternative to get the dag start date that will never change (even after cleraing/restarting tasks).

[alefrancia]

@ephraimbuddy could you please share your thoughts?

[ZaxR]

Any update on this? I currently use a custom operator as a workaround to capture an unchanging start_date, but it would simplify my DAGs greatly if there was a way to get this information directly.

[aborecki]

please check if next_execution_date is not what you need, in case of manual triggered dag it is current time, in case of scheduled dags it will be also current_time. For airflow 2.2+ there is also data_interval_end .
It is really not the current time, but the time of the scheduled run, which should be somewhat the same as current time.

[anaslebrigui]

We are currently having issues as well with this behavior.
I agree with @samvel1024, it doesn't really make sense to have the dag run start_date changing when tasks are cleared/retried.
It seems like this was not the case in previous Airflow versions. It would be great if there was a way to get the DAG run's actual start_date directly.

[Taragolis]

Clear task generate new DagRun, task doesn't executed itself, it required to DagRun.
For deterministic result consider to use intervals (depend on Timetable) or logical date

[karanjeets]

I agree with @samvel1024 that there should be a deterministic way of retrieving DAG start date. The intervals may work for scheduled runs, but what about manually triggered runs?

[karanjeets]

@ephraimbuddy - could you please share the motivation behind the change and what's the best path forward for the above use case?

[ephraimbuddy]

Hi @karanjeets and others, you can create an issue for this. I think the motivation at the time was false start dates for cleared runs.