Replies: 1 comment
-
This parameter was added in Amazon Provider v8.6.0, see Provider Changelog So i guess you need to update Amazon Provider in your MWAA cluster, see Installing Python dependencies in MWAA User Guide |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I am using the glue operator to run existing glue scripts. I am writing a task to run the glue script. My script look like the following
.deleteraw_glue_operator = GlueJobOperator(
task_id = "deleteraw_glue_operator",
job_name=f"deleteraw_{type}",
s3_bucket = f"{bucket}",
region_name = f"{region_name}",
script_location = f"s3://{bucket}/deploy/{filename}.py",
iam_role_arn = f"{arn}",
iam_role_name = f"{role_name}",
script_args = {
"JobParameters":{"--bucket":f"{bucket}",
"--logsPath":f"s3://{bucket}/logs/",
"--File":"User",
"--path":f"gateway/inbound/source/landing_skid/{type}/"},
"Tags":{"owner":f'{support@abc}',
"application":f"application",
"costcenter":f"costcenter}",
"environment":f"{enviornment"}" },
create_job_kwargs = {"GlueVersion":"2.0",
"NumberOfWorkers":10,
"WorkerType":"G.2X",
"TempDir"::"f"s3://{bucket}/temp/" },
verbose = True)
And I get the following as an error.
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/models/baseoperator.py", line 394, in apply_defaults
result = func(self, **kwargs, default_args=default_args)
File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/models/baseoperator.py", line 742, in init
raise AirflowException(
airflow.exceptions.AirflowException: Invalid arguments were passed to GlueJobOperator (task_id: deleteraw_glue_operator). Invalid arguments were:
**kwargs: {'iam_role_arn': 'arn:aws:iam::{account_id}:role/'}
I am using MWAA 2.5.1
Beta Was this translation helpful? Give feedback.
All reactions