Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

20139 - organize aws emr operators #20160

Merged
merged 7 commits into from
Dec 16, 2021
Merged

20139 - organize aws emr operators #20160

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
191a13b
20139 - organize aws emr operators
bhavaniravi Dec 9, 2021
a3ee38a
fix example
bhavaniravi Dec 10, 2021
b4b31b2
fix transient dependency warning
bhavaniravi Dec 10, 2021
7e7d57c
20139 - emr - fix test case import
bhavaniravi Dec 13, 2021
d3797fb
fix consistent classname - EMR
bhavaniravi Dec 14, 2021
8cdedfd
fix - test failure, stacktrace level
bhavaniravi Dec 16, 2021
359d3ad
rename testcases
bhavaniravi Dec 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions airflow/contrib/operators/emr_create_job_flow_operator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

"""This module is deprecated. Please use :mod:`airflow.providers.amazon.aws.operators.emr_create_job_flow`."""

import warnings
Expand Down
4 changes: 2 additions & 2 deletions airflow/providers/amazon/aws/example_dags/example_emr_eks_job.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
from datetime import datetime, timedelta

from airflow import DAG
from airflow.providers.amazon.aws.operators.emr_containers import EMRContainerOperator
from airflow.providers.amazon.aws.operators.emr import EmrContainerOperator

# [START howto_operator_emr_eks_env_variables]
VIRTUAL_CLUSTER_ID = os.getenv("VIRTUAL_CLUSTER_ID", "test-cluster")
Expand Down Expand Up @@ -69,7 +69,7 @@
# JOB_ROLE_ARN = '{{ conn.emr_eks.extra_dejson["job_role_arn"] }}'

# [START howto_operator_emr_eks_jobrun]
job_starter = EMRContainerOperator(
job_starter = EmrContainerOperator(
task_id="start_job",
virtual_cluster_id=VIRTUAL_CLUSTER_ID,
execution_role_arn=JOB_ROLE_ARN,
Expand Down
4 changes: 2 additions & 2 deletions airflow/providers/amazon/aws/example_dags/example_emr_job_flow_automatic_steps.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@
from datetime import datetime, timedelta

from airflow import DAG
from airflow.providers.amazon.aws.operators.emr_create_job_flow import EmrCreateJobFlowOperator
from airflow.providers.amazon.aws.sensors.emr_job_flow import EmrJobFlowSensor
from airflow.providers.amazon.aws.operators.emr import EmrCreateJobFlowOperator
from airflow.providers.amazon.aws.sensors.emr import EmrJobFlowSensor

# [START howto_operator_emr_automatic_steps_config]
SPARK_STEPS = [
Expand Down
10 changes: 6 additions & 4 deletions airflow/providers/amazon/aws/example_dags/example_emr_job_flow_manual_steps.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,12 @@
from datetime import datetime, timedelta

from airflow import DAG
from airflow.providers.amazon.aws.operators.emr_add_steps import EmrAddStepsOperator
from airflow.providers.amazon.aws.operators.emr_create_job_flow import EmrCreateJobFlowOperator
from airflow.providers.amazon.aws.operators.emr_terminate_job_flow import EmrTerminateJobFlowOperator
from airflow.providers.amazon.aws.sensors.emr_step import EmrStepSensor
from airflow.providers.amazon.aws.operators.emr import (
EmrAddStepsOperator,
EmrCreateJobFlowOperator,
EmrTerminateJobFlowOperator,
)
from airflow.providers.amazon.aws.sensors.emr import EmrStepSensor

SPARK_STEPS = [
{
Expand Down
Loading