Skip to content

Latest commit

 

History

History
119 lines (83 loc) · 4.16 KB

redshift.rst

File metadata and controls

119 lines (83 loc) · 4.16 KB

RedshiftSQLOperator

Overview

Use the :class:`RedshiftSQLOperator <airflow.providers.amazon.aws.operators.redshift>` to execute statements against an Amazon Redshift cluster.

:class:`RedshiftSQLOperator <airflow.providers.amazon.aws.operators.redshift.RedshiftSQLOperator>` works together with :class:`RedshiftSQLHook <airflow.providers.amazon.aws.hooks.redshift.RedshiftSQLHook>` to establish connections with Amazon Redshift.

example_redshift.py

Purpose

This is a basic example dag for using :class:`RedshiftSQLOperator <airflow.providers.amazon.aws.operators.redshift>` to execute statements against an Amazon Redshift cluster.

Create a table

In the following code we are creating a table called "fruit".

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_redshift.py
    :language: python
    :start-after: [START howto_operator_redshift_create_table]
    :end-before: [END howto_operator_redshift_create_table]

Insert data into a table

In the following code we insert a few sample rows into the "fruit" table.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_redshift.py
    :language: python
    :start-after: [START howto_operator_redshift_populate_table]
    :end-before: [END howto_operator_redshift_populate_table]

Fetching records from a table

Creating a new table, "more_fruit" from the "fruit" table.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_redshift.py
    :language: python
    :start-after: [START howto_operator_redshift_get_all_rows]
    :end-before: [END howto_operator_redshift_get_all_rows]

Passing Parameters into RedshiftSQLOperator

RedshiftSQLOperator supports the parameters attribute which allows us to dynamically pass parameters into SQL statements.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_redshift.py
    :language: python
    :start-after: [START howto_operator_redshift_get_with_filter]
    :end-before: [END howto_operator_redshift_get_with_filter]

The complete RedshiftSQLOperator DAG

All together, here is our DAG:

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_redshift.py
    :language: python
    :start-after: [START redshift_operator_howto_guide]
    :end-before: [END redshift_operator_howto_guide]

Resume a Redshift Cluster

To resume a 'paused' AWS Redshift Cluster you can use :class:`~airflow.providers.amazon.aws.operators.redshift_resume_cluster.RedshiftResumeClusterOperator`.

This Operator leverages the AWS CLI resume-cluster API

Pause a Redshift Cluster

To pause an 'available' AWS Redshift Cluster you can use :class:`~airflow.providers.amazon.aws.operators.redshift_pause_cluster.RedshiftPauseClusterOperator`.

This Operator leverages the AWS CLI pause-cluster API