Replies: 7 comments 18 replies
-
|
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. |
Beta Was this translation helpful? Give feedback.
-
|
Converted it into discussion, because this is not really something actionable, and likely a deployment issue. I think you should take a look at the deployment logs of yours - specifically look in detail in the logs of EKS and whether ther are any correlated events with the SIGTERMS of yours. IMHO - this is a result of simply some of your machines in the cluster being restarted - but what is the reason for that, hard to say. I think in order for someone to help you - you should make a deep look in your deployment logs and rather than show the details of airflow configuration show what happens with your deployment there. It would also be useful (for anyone who would be looking at it - not necessary me, I might not have time to look at detail) it might also be useful to see the exact logs of the sigterm being received, your deployment/EKS logs when you look deeply there should also show at least what is sending those TERM events. You should also look at things like your health check logs and other components in K8S that might get restarted. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @potiuk thanks for replying. We've spent the last 5-6 months trying to track this down, we consider our team kubernetes experts and believe we have ruled out any of those variables (the nodes airflow jobs running on, aren't being restarted, for example). We've ruled out OOM kills (raised memory request to an extremely high value just to rule things out) The deployment logs, unfortunately, don't show anything related to SIGTERM's which is what has made this issue so frustrating to troubleshoot. We've literally gone through every SIGTERM Github issue in Airflow and also when Googling "Airflow Sigterm kubernetes" we get many many hits with various recommendations and believe we've exhausted them. We run quite a huge kubernetes cluster and don't have issues like this on anything besides Airflow. We've literally just run the given helm chart. Would this issue be more appropriate to open in the helm chart Issues? |
Beta Was this translation helpful? Give feedback.
-
I have to believe you, and I am sorry you have such an experience, but facing the reality, none here has any more information than you have about what's going on on your system and it's going to be even more difficult without having even minimal information from logs and circumstances to be able to help you. For sure it is not a "common" problem. yes people sometimes have similar issues but taking into account hudreds of thousands of Airlfow installations there, we would be flooded with similar issues if it was a "common" theme. If we see no logs or circumstances describing what happens, it's simply impossible to help you to diagnose the problem and often even come up with plausible hypothesis. Wild guess: What I can suggest you - one of the reasons why it could happen is that for whatever reason the heartbeating mechanisms got slow or misconfigured. Possibly one of those parameters:
Maybe a combinations of some slowness of hardware, load, etc, cause tha some of the thresholds are too low and heartbeats are exceeding the threasholds and airlfow tasks get killed because they are not heartbeating often enough. But, that would usually be visible in logs correlated with the events that are happening - for example if you see a TERM in a worker looking in other logs for events that could trigger it is likely a good idea - without logs correlated with such events, it's next to impossible to say what could be the reason other than wild guessing. So if you could attempt to provide more logs, that could be helpful. That's one of the ways that can come by knowledge. And only you can help here. And since you seem to have a fairly reproducible case, there is another way if it is hard or impossible to correlate events manually. By experimentation and bisecting. This is also quite viable approach in engineering when you have a complex system built on top of multiple layers of abstraction (cloud, vms,. networking, filesystem, db, k8s, helm, python, resources etc). Assuming this is one of the parameters above, make some intelligent guesses, make a hypothesis and change of them - to a much higher value usually. And see if it helps, if it solves the problem - perfect (and then we can try to figure out what could be the reason if you report it back), If it increases/decreases the number of problems. that might also give some hints and narrow down the problem. Unfortunately - again - this is only something you and your team can do. I am happy to try to help to interpret the results of such an experimentation - but without having such results or logs, I am sorry but I can't help much. Also please remember that this is an OSS project: free for use, people here try to help in their free time, trying to come up with wild guesses based on incomplete information, without having access to your systems and being able to inspect and analyse it ( and basically doing it in our free time that we could have spend on trying to improve the project). So there is an option also to hire someone who can get access to your system and make even more deep analysis and guess. Which is a third option. |
Beta Was this translation helpful? Give feedback.
-
|
And one more thing - debug log might give more insight and more correlation opportunities if this is of any help. |
Beta Was this translation helpful? Give feedback.
-
|
First off, really appreciate you taking the time to respond -- I'm sure you're very busy and if it sounded like I was complaining that wasn't my intent. I agree, it's very possible it's something in my environment -- just trying to figure out how to get to the bottom of the issue. Will be going through your recommendations and gathering pertinent logs. For the third option -- yes, we may be at that point quite soon. If you have any recommendations (or are even open for hire yourself!) that would be great. Thank you again |
Beta Was this translation helpful? Give feedback.
-
|
What executor are you using? In the config it is Kubernetes and in the helm, but also in the helm you have workers defined with replicas: 1, is worker running but not being used? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @potiuk, we are also experiencing this issue. I tried to debug and found that whenever SIGTERM is received, scheduler logs contains this info at the exact same time: I couldn't find any other instances of this info log but if this is indeed from where it's getting logged, I am not able to figure out how it gets into this condition where pod status is marked as Here are the detailed logsScheduler logs when SIGTERM received: Corresponding task logs: Deployment Info:
|
Beta Was this translation helpful? Give feedback.
-
|
Unfortunatley - I have no big experience in debugging K8S Executor issues. But surely it looks like some race condition when statuis of the pod gets calculated. We have been busy with 2.7 release but maybe @dstandish or @ephraimbuddy might take a look at this after that. |
Beta Was this translation helpful? Give feedback.
-
|
@mschueler, @msardana94 I am curious whether you were able to solve the issue. We are facing the same issue after migrating from spot.io to karpenter. After a day of troubleshooting I have strong feeling that it is happening because of Taint nodes with a NoSchedule for Consolidation before Validation begins #651 issue on Karpenter I don't want to mislead you but just an idea to check, yet I am still investigating on my end :) |
Beta Was this translation helpful? Give feedback.
-
|
@mschueler did you ever figure out what was causing this ? We are also facing this exact same issue after migrating to the airflow official helm chart from the airflow community chart and have no idea what's causing this. We've debugged and ruled out almost all potential kubernetes related issues. |
Beta Was this translation helpful? Give feedback.
-
|
@b32n , @mschueler do you have Not so long ago by default it was set to 'true', 8 months ago it was changed to false: here
|
Beta Was this translation helpful? Give feedback.
-
|
Hard to say - but it looks like a query via some worker hanged on a low-level C-CODE that does not handle other signals properly because apparently gunicorn has to kill a child process with KILL. Might be a low-level issue with database driver for example - seen this happening with mysql for example. |
Beta Was this translation helpful? Give feedback.
-
|
Worker timeout may happen when request (from the browser) is taking longer than: web_server_worker_timeout, if it reaches the timeout then probably SIGTERM is sent and if it is still unresponsive then SIGKILL from gunicorn master. I would try poking around webserver to recreate the issue, one of the endpoints in webserver may be slow for example due to db size, also you could try increasing web_server_worker_timeout, but 120seconds by default seems long enough, on the other hand maybe when you would increase the timeout we could see the real reason it was taking so long as sooner or later it should throw some exception |
Beta Was this translation helpful? Give feedback.
-
|
The issue is recreated only when accessing the web server in the browser after a long time for any endpoint, including /. Once the first page loads, then everything else works fine. But if I access the web server in a fresh tab after a while I face this issue. Also this has been happening in a brand new installation with no dags so I’m not sure if database size could be an issue. Or ? |
Beta Was this translation helpful? Give feedback.
-
|
But this is good information. I will try increasing the web server timeout and see the logs to check what happens during the request. |
Beta Was this translation helpful? Give feedback.
-
|
Can you share details about your setup? Where are you running Airflow, in cloud? locally? What is the Airflow Version? PS: Set the timeout to at least 2 hours and 10minutes, maybe that is something with tcp idle timeout |
Beta Was this translation helpful? Give feedback.
-
Apache Airflow version
Other Airflow 2 version (please specify below)
What happened
We are seeing intermittent SIGTERMs on DAGs. There seems to be no rhyme or reason to the SIGTERMs (e.g. seems to happen to all our DAGs at some time or another, no pattern to the timing, etc)
The deploy is thru Helm chart to an EKS cluster running on EKS. It's happening in our nonprod and prod clusters both. We've tried different things in our nonprod environment to fix it, basically following ideas we found from Google searches (increasing resources, upgrading airflow version, checking logs [we've found nothing useful in the logs but will post as much info as I can here], increasing timeouts, and trying some settings* we found mentioned in other GitHub issues.
` - name: AIRFLOW__CORE__KILLED_TASK_CLEANUP_TIME
value: "3600"
value: "false"`
Nonprod: EKS 1.26 / Airflow 2.5.1.
Prod: EKS 1.25 / Airflow 2.2.4
Focusing efforts on Nonprod but just wanted to mention we're seeing the issue on multiple versions. Also, believe the original version we started on was 2.0.x something but we've been struggling with this issue since January (when we first started to setup Airflow 2.0 on k8s). As a workaround we are doing a retry where possible.
This is the exact error:
airflow.exceptions.AirflowException: Task received SIGTERM signalWould truly appreciate any help or insight into what we're doing wrong. I've tried to put as much information below as possible but if I'm missing something, please let me know.
Helm values file:
resulting Airflow.cfg (configmap)
What you think should happen instead
No response
How to reproduce
Intermittent. Schedule a DAG run.
Operating System
Kubernetes -- DAGs running on image based on Debian Bullseye
Versions of Apache Airflow Providers
apache-airflow-providers-grpc 3.1.0 gRPC
apache-airflow-providers-hashicorp 3.3.0 Hashicorp including Hashicorp Vault
apache-airflow-providers-http 4.2.0 Hypertext Transfer Protocol (HTTP)
apache-airflow-providers-imap 3.1.1 Internet Message Access Protocol (IMAP)
apache-airflow-providers-microsoft-azure 5.2.1 Microsoft Azure
apache-airflow-providers-microsoft-mssql 2.1.3 Microsoft SQL Server (MSSQL)
apache-airflow-providers-mysql 2.2.3 MySQL
apache-airflow-providers-odbc 3.2.1 ODBC
apache-airflow-providers-oracle 2.2.3 Oracle
apache-airflow-providers-postgres 5.4.0 PostgreSQL
apache-airflow-providers-redis 3.1.0 Redis
apache-airflow-providers-sendgrid 3.1.0 Sendgrid
apache-airflow-providers-sftp 2.6.0 SSH File Transfer Protocol (SFTP)
apache-airflow-providers-slack 7.2.0 Slack
apache-airflow-providers-snowflake 2.1.1 Snowflake
apache-airflow-providers-sqlite 3.3.1 SQLite
apache-airflow-providers-ssh 3.5.0 Secure Shell (SSH)
apache-airflow-providers-tableau 2.1.8 Tableau
Deployment
Official Apache Airflow Helm Chart
Deployment details
EKS 1.25 running Karpenter (cluster autoscaler replacement)
Anything else
Intermittent -- 5 - 50x a day
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions