Web UI not displaying the log when task fails - Permission Denied at temporary error file when using run_as_user

[kevinbsilva]

Apache Airflow version: 2.0.1

Environment: 2 Worker nodes and 1 Master

• Cloud provider or hardware configuration: Oracle Cloud
• OS (e.g. from /etc/os-release): Oracle Linux 7.8
• Kernel: Linux 4.14.35-1902.302.2.el7uek.x86_64 Bring in more resolution to hivestats #2 SMP Fri Apr 24 14:24:11 PDT 2020 x86_64 x86_64 x86_64 GNU/Linux

What happened:

When a task fails, the Web UI doesn't display the log. The URL to get the log is presented without the hostname. When we navigate to the log path and open the .log file in the OS, it shows a permission error when opening the temporary file generated to dump the error.

I noticed when we create the temporary file using NamedTemporaryFile it creates a restricted file, open only for reading. It can be written only by the user airflow. If any other user tries to write in the file, the Permission Error is raised.

The message that is displayed at the UI is:

*** Log file does not exist: /path/to/log/1.log
*** Fetching from: http://:8793/log/path/to/log/1.log
*** Failed to fetch log file from worker. Invalid URL 'http://:8793/log/path/to/log/1.log': No host supplied

We can see the hostname is not obtained when building the URL since the execution fails when dumping the error into the temporary file.

When we access the log in the OS, the full log is there but it shows the Permission Denied:

PermissionError: [Errno 13] Permission denied: '/tmp/tmpmg2q49a8'

What you expected to happen:

The print from the Web UI when the task fails:

The print from the Log file, showing the Permission Denied error when accessing the tmp file:

Anything else we need to know:

The errors occurs every time a task fails and the run_as_user and owner is not airflow.

When the task does succeed, the log is normal at the Web Ui.

I've added a os.chmod to the self._error_file at base_task_runner, after the NamedTemporaryFile is create, using the umask 0o0777 and now the logs are appearing normally, even when the task fails.

I pretend to create a PR adding that line of code but it depends if the community believes that opening up the permissions for the temp file is ok. As far as i know, i didn't noticed any sensitive informations or possible vulnerabilities from this change.

It's important to say that the task fails not because of that problem. The problem is that the log is inaccessible through the Web UI, which can slow down troubleshootings and so on.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!