Amazon EMR on Amazon EKS #17178

wanderijames · 2021-07-22T20:57:24Z

Adding operators for interacting with EMR on EKS as per #17175

Add operator for starting job
Add operator for canceling a job
Add operator for checking job state

closes: #17175
related: #17175

- Add operators for starting job - Add operator for canceling a job - Add operator for checking job state

boring-cyborg · 2021-07-22T20:57:27Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

mik-laj · 2021-07-22T22:00:59Z

We currently have one operator that allows us to run Spark job on Kubernetes. It works with both EKS and GCP as well as any other Kubernetes platform. - SparkKubernetesOperator. Why would anyone use this operator instead of the generic operator for Kubernetes?

wanderijames · 2021-07-23T06:11:24Z

We currently have one operator that allows us to run Spark job on Kubernetes. It works with both EKS and GCP as well as any other Kubernetes platform. - SparkKubernetesOperator. Why would anyone use this operator instead of the generic operator for Kubernetes?

Hey @mik-laj, I am aware of apache spark and livy operator and also EMR operator. However, EMR on EKS works differently because EMR launches virtual cluster in your EKS. The pods (spark master and executors) launched are ephemeral, only existing when a start job is invoked. For more information, kindly visit https://aws.amazon.com/emr/features/eks/

In addition, SparkKubernetesOperator is only suitable if you have Spark cluster has been setup in Kubernetes. In this case, it is not.

dacort

Added a few comments/questions.

Realizing there is #16766 already submitted, the big difference between the two seems to be the addition of a couple different operators and the default behavior in #16766 being that the Operator waits for the job to complete.

I haven't done a thorough review of this, but I think (depending on if they're needed) we should try to merge 1 or more of the additional operators into my PR. I'm just not familiar enough with real-world DAGs to know to which degree the additional operators are necessary. If you have any feedback on that, would much appreciate it!

dacort · 2021-07-23T22:30:10Z

airflow/providers/amazon/aws/operators/emr_containers_get_job_state.py

+from airflow.providers.amazon.aws.hooks.emr_containers import EmrContainersHook
+
+
+class EmrContainersGetJobStateOperator(BaseOperator):


I'm curious what the use case is for an operator that just returns the job state?

Apologies if this is a silly question - still getting up to speed on typical Airflow patterns.

Now that you asked me that question, I don't think this has a straight forward use case as an operator.

airflow/providers/amazon/aws/operators/emr_containers_get_job_state.py

dacort · 2021-07-23T22:38:46Z

airflow/providers/amazon/aws/operators/emr_containers_start_job.py

+            configuration_overrides=configuration_overrides,
+            tags=tags,
+            name=name,
+            client_token=client_token,


I don't see you defining client_token anywhere. Have you tested this in a live environment to validate that it works?

It is being used in line 89;

response = emr_containers.start_job(**self.start_job_params)

Right, but it's not initialized anywhere. The default is None, which means if the client doesn't pass in a client_token, a new job won't get created after the first one. I ran into a similar issue and had to generate a UUID as the default:

client_token or str(uuid4())

Thanks @dacort for this.

…state.py Co-authored-by: Damon P. Cortesi <d.lifehacker@gmail.com>

dacort · 2021-08-20T17:12:43Z

Here's my current thinking on this - I like the addition of the Start and Cancel operators. I think we should try to get #16766 merged and then you can 1. rebase off that commit and 2. add documentation to the emr_eks.rst file on how to use the two additional operators.

Thoughts?

wanderijames · 2021-08-22T12:24:54Z

Here's my current thinking on this - I like the addition of the Start and Cancel operators. I think we should try to get #16766 merged and then you can 1. rebase off that commit and 2. add documentation to the emr_eks.rst file on how to use the two additional operators.

Thoughts?

I agree @dacort , let's do that.

github-actions · 2021-10-07T00:06:02Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

eladkal · 2021-11-06T08:17:45Z

@wanderijames #16766 was merged. Would you like to proceed?

feat: Amazon EMR on Amazon EKS

b5e2993

- Add operators for starting job - Add operator for canceling a job - Add operator for checking job state

boring-cyborg bot added area:providers provider:amazon AWS/Amazon - related issues labels Jul 22, 2021

wanderijames added 2 commits July 23, 2021 00:17

fix: ensure the black and yaml checks passes

dbc2cf3

fix: ensure the flake8 tests passes

f0e765b

dacort suggested changes Jul 23, 2021

View reviewed changes

Update airflow/providers/amazon/aws/operators/emr_containers_get_job_…

c978e0d

…state.py Co-authored-by: Damon P. Cortesi <d.lifehacker@gmail.com>

github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Oct 7, 2021

github-actions bot closed this Oct 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Amazon EMR on Amazon EKS #17178

Amazon EMR on Amazon EKS #17178

wanderijames commented Jul 22, 2021

boring-cyborg bot commented Jul 22, 2021

mik-laj commented Jul 22, 2021

wanderijames commented Jul 23, 2021 •

edited

Loading

dacort left a comment

dacort Jul 23, 2021

wanderijames Aug 4, 2021 •

edited

Loading

dacort Jul 23, 2021

wanderijames Aug 4, 2021

dacort Aug 20, 2021

wanderijames Aug 22, 2021

dacort commented Aug 20, 2021

wanderijames commented Aug 22, 2021

github-actions bot commented Oct 7, 2021

eladkal commented Nov 6, 2021

		from airflow.providers.amazon.aws.hooks.emr_containers import EmrContainersHook


		class EmrContainersGetJobStateOperator(BaseOperator):

Amazon EMR on Amazon EKS #17178

Amazon EMR on Amazon EKS #17178

Conversation

wanderijames commented Jul 22, 2021

boring-cyborg bot commented Jul 22, 2021

mik-laj commented Jul 22, 2021

wanderijames commented Jul 23, 2021 • edited Loading

dacort left a comment

Choose a reason for hiding this comment

dacort Jul 23, 2021

Choose a reason for hiding this comment

wanderijames Aug 4, 2021 • edited Loading

Choose a reason for hiding this comment

dacort Jul 23, 2021

Choose a reason for hiding this comment

wanderijames Aug 4, 2021

Choose a reason for hiding this comment

dacort Aug 20, 2021

Choose a reason for hiding this comment

wanderijames Aug 22, 2021

Choose a reason for hiding this comment

dacort commented Aug 20, 2021

wanderijames commented Aug 22, 2021

github-actions bot commented Oct 7, 2021

eladkal commented Nov 6, 2021

wanderijames commented Jul 23, 2021 •

edited

Loading

wanderijames Aug 4, 2021 •

edited

Loading