Airflow dag code fileloc absolute path

[vazmeee]

Hi. I was trying to play around and try out the celery worker. I'm working with airflow 2.0.0, mysql backend.
Here's my setup.

Celery worker running on aws ec2 Red Hat Enterprise Linux release 8.3 (Ootpa)
Scheduler and webserver are running locally on my laptop. VERSION="18.04.5 LTS (Bionic Beaver)

For worker this was my dag directory
dags_folder = /home/ec2-user/airflow/dags/airflow_dags

and on my laptop I had it as
dags_folder = /home/<my_user_name>/airflow/dags/airflow_dags

On trying this out I got an error,
[2021-06-13 14:38:02,415: ERROR/ForkPoolWorker-2] Failed to execute task dag_id could not be found: custom_dag_1_test. Either the dag did not exist or it failed to parse..

on checking the info log right above it

[2021-06-13 14:38:02,237: INFO/MainProcess] Received task: airflow.executors.celery_executor.execute_command[f3746582-18ee-4a84-9955-5e9b5ff31d93]
[2021-06-13 14:38:02,248: INFO/ForkPoolWorker-2] Executing command in Celery: ['airflow', 'tasks', 'run', 'custom_dag_1_test', 'print_something', '2021-06-13T14:37:53.254677+00:00', '--local', '--pool', 'default_pool', '--subdir', '/home/<my_user_name>/airflow/dags/airflow_dags/dag_custom.py']
[2021-06-13 14:38:02,320: INFO/ForkPoolWorker-2] Filling up the DagBag from /home/<my_user_name>/airflow/dags/airflow_dags/dag_custom.py

The dag location is not relative to AIRFLOW_HOME and is taking the absolute location which was published by the scheduler.
And when I checked the db I found the same, the fileloc was
/home/<my_user_name>/airflow/dags/airflow_dags/dag_custom.py

Is there a reason why dag locations arent relative to $AIRFLOW_HOME in the database as well?
or is this part of a new update.
And are there some suggested practices while working with celery workers
or do the worker and scheduler have to be exactly the same with respect to file locations?

[potiuk]

This is a "hard" requirement to have the same DAG absolute path for all machines. This was a deliberate choice that has been made a long time ago to make more "opinionated" approach and avoid some common mistakes. For example someone can write Dags that read configuration from files placed in the DAG folder using absolute path, which will work locally but start failing in case you change the location of the DAG folder on a different machine. As far as I know it's been always like that.

[potiuk]

As far as I know this is issue is already reported #13692

[vazmeee]

I'll watch the mentioned issue. Thanks @potiuk

[potiuk]

BTW. The "absolute" location will likely be fixed soon (so maybe even we will not have to wait for 3.0). after thinking about it it will be backwards compatible, so we do not have to waiit that long. See https://github.com/apache/airflow/pull/16110/files#r640688180

[potiuk]

And #8061

[vazmeee]

This is great. Thanks for the update. I'll look out for these both.