Skip to content

Commit 03bbcb6

Browse files
author
Daniel Huang
committed
Merge pull request apache#26 in TP/incubator-airflow from feature/DATA-3640 to develop
* commit '57bed9d9a917c47aab63faf793f9ef13e328fdfa': (488 commits) DATA-3640: remove dev identifier DATA-3640: remove rc and set to 1.9.0 [AIRFLOW-XXX] Bump version to 1.9.0 again. [AIRFLOW-XXX] Bump version to 1.9.0+incubating [AIRFLOW-1525] Fix minor LICENSE and NOTICE issues [AIRFLOW-XXX] Bump version to 1.9.0 [AIRFLOW-XXX] Bump version to 1.9.0rc6 [AIRFLOW-1897][AIRFLOW-1873] Task Logs for running instance not visible in WebUI [AIRFLOW-XXX] Make sure session is committed [AIRFLOW-1896] FIX bleach <> html5lib incompatibility [AIRFLOW-XXX] Fix log handler test [AIRFLOW-XXX] Bump version to 1.9.0rc5 [AIRFLOW-1873] Set TI.try_number to right value depending TI state [AIRFLOW-1554] Fix wrong DagFileProcessor termination method call [AIRFLOW-1872] Set context for all handlers including parents [AIRFLOW-XXX] Add dask lock files to excludes [AIRFLOW-XXX] Bump version to 1.9.0rc4 [AIRFLOW-1839] Fix more bugs in S3Hook boto -> boto3 migration [AIRFLOW-XXX] Bump version to 1.9.0rc3 [AIRFLOW-1795] Correctly call S3Hook after migration to boto3 ...
2 parents 3d00d47 + 57bed9d commit 03bbcb6

File tree

377 files changed

+24033
-5832
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

377 files changed

+24033
-5832
lines changed

‎.editorconfig

+45
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# -*- coding: utf-8 -*-
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
root = true
15+
16+
[*]
17+
end_of_line = lf
18+
indent_style = space
19+
insert_final_newline = true
20+
trim_trailing_whitespace = true
21+
charset = utf-8
22+
23+
[*.py]
24+
indent_size = 4
25+
26+
[*.sh]
27+
indent_size = 4
28+
29+
[*.sql]
30+
indent_size = 4
31+
32+
[*.js]
33+
indent_size = 2
34+
35+
[*.css]
36+
indent_size = 2
37+
38+
[*.{md,rst}]
39+
indent_size = 2
40+
41+
[*.{yml,yaml}]
42+
indent_size = 2
43+
44+
[*.{htm,html}]
45+
indent_size = 2

‎.github/PULL_REQUEST_TEMPLATE.md

+26-24
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,27 @@
1-
Please accept this PR that addresses the following issues:
2-
- *(MANDATORY - replace with a link to JIRA - e.g. https://issues.apache.org/jira/browse/AIRFLOW-XXX)*
3-
4-
Testing Done:
5-
- Unittests are required, if you do not include new unit tests please
6-
specify why you think this is not required. We like to improve our
7-
coverage so a non existing test is even a better reason to include one.
8-
9-
Reminders for contributors (REQUIRED!):
10-
* Your PR's title must reference an issue on
11-
[Airflow's JIRA](https://issues.apache.org/jira/browse/AIRFLOW/).
12-
For example, a PR called "[AIRFLOW-1] My Amazing PR" would close JIRA
13-
issue #1. Please open a new issue if required!
14-
15-
* For all PRs with UI changes, you must provide screenshots. If the UI changes are not obvious, either annotate the images or provide before/after screenshots.
16-
17-
* Please squash your commits when possible and follow the [How to write a good git commit message](http://chris.beams.io/posts/git-commit/).
18-
Summarized as follows:
19-
1. Separate subject from body with a blank line
20-
2. Limit the subject line to 50 characters
21-
3. Do not end the subject line with a period
22-
4. Use the imperative mood in the subject line (add, not adding)
23-
5. Wrap the body at 72 characters
24-
6. Use the body to explain what and why vs. how
1+
Dear Airflow maintainers,
2+
3+
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
4+
5+
6+
### JIRA
7+
- [ ] My PR addresses the following [Airflow JIRA](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
8+
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
9+
10+
11+
### Description
12+
- [ ] Here are some details about my PR, including screenshots of any UI changes:
13+
14+
15+
### Tests
16+
- [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason:
17+
18+
19+
### Commits
20+
- [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
21+
1. Subject is separated from body by a blank line
22+
2. Subject is limited to 50 characters
23+
3. Subject does not end with a period
24+
4. Subject uses the imperative mood ("add", not "adding")
25+
5. Body wraps at 72 characters
26+
6. Body explains "what" and "why", not "how"
2527

‎.gitignore

+119-24
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,126 @@
1-
.idea/*
2-
*.bkp
3-
*.egg-info
4-
*.pyc
5-
.DS_Store
6-
.ipynb*
7-
.coverage
8-
.python-version
9-
airflow/git_version
10-
airflow/www/static/coverage/
11-
airflow.db
1+
# Airflow configuration
122
airflow.cfg
3+
unittests.cfg
134
airflow_login.py
14-
build
15-
cover
165
dbinit.py
17-
docs/_*
18-
dist
19-
env
206
initdb.py
21-
logs
22-
MANIFEST
237
secrets.py
24-
sftp-config.json
25-
unittests.cfg
26-
error.log
8+
9+
# Airflow sqlite databases
10+
airflow.db
2711
unittests.db
12+
13+
# Airflow temporary artifacts
14+
airflow/git_version
15+
airflow/www/static/coverage/
16+
logs/
17+
18+
# Byte-compiled / optimized / DLL files
19+
__pycache__/
20+
*.py[cod]
21+
*$py.class
22+
23+
# C extensions
24+
*.so
25+
26+
# Distribution / packaging
27+
.Python
28+
env/
29+
build/
30+
develop-eggs/
31+
dist/
32+
downloads/
33+
eggs/
34+
.eggs/
35+
lib/
36+
lib64/
37+
parts/
38+
sdist/
39+
var/
40+
wheels/
41+
*.egg-info/
42+
.installed.cfg
43+
*.egg
44+
45+
# PyInstaller
46+
# Usually these files are written by a python script from a template
47+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
48+
*.manifest
49+
*.spec
50+
51+
# Installer logs
52+
pip-log.txt
53+
pip-delete-this-directory.txt
54+
55+
# Unit test / coverage reports
56+
htmlcov/
57+
.tox/
58+
.coverage
59+
.coverage.*
60+
.cache
61+
nosetests.xml
62+
coverage.xml
63+
*,cover
64+
.hypothesis/
65+
66+
# Translations
67+
*.mo
68+
*.pot
69+
70+
# Django stuff:
71+
# *.log
72+
local_settings.py
73+
74+
# Flask stuff:
75+
instance/
76+
.webassets-cache
77+
78+
# Scrapy stuff:
79+
.scrapy
80+
81+
# Sphinx documentation
82+
docs/_build/
83+
84+
# PyBuilder
85+
target/
86+
87+
# Jupyter Notebook
88+
.ipynb_checkpoints
89+
90+
# pyenv
91+
.python-version
92+
93+
# celery beat schedule file
94+
celerybeat-schedule
95+
96+
# SageMath parsed files
97+
*.sage.py
98+
99+
# dotenv
100+
.env
101+
102+
# virtualenv
103+
.venv*
104+
venv*
105+
ENV/
106+
107+
# Spyder project settings
108+
.spyderproject
109+
110+
# Rope project settings
111+
.ropeproject
112+
113+
# PyCharm
114+
.idea/
115+
116+
# vim
117+
*.swp
118+
119+
# OSX
120+
.DS_Store
121+
122+
# SQL Server backups
123+
*.bkp
124+
125+
# Spark
28126
rat-results.txt
29-
/.eggs/
30-
/.tox/
31-
venv

‎.rat-excludes

+1
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ metastore_db
2020
.*csv
2121
CHANGELOG.txt
2222
.*zip
23+
.*lock
2324
# Apache Rat does not detect BSD-2 clause properly
2425
# it is compatible according to http://www.apache.org/legal/resolved.html#category-a
2526
kerberos_auth.py

‎.readthedocs.yml

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
#
2+
# Licensed under the Apache License, Version 2.0 (the "License");
3+
# you may not use this file except in compliance with the License.
4+
# You may obtain a copy of the License at
5+
#
6+
# http://www.apache.org/licenses/LICENSE-2.0
7+
#
8+
# Unless required by applicable law or agreed to in writing, software
9+
# distributed under the License is distributed on an "AS IS" BASIS,
10+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
11+
# See the License for the specific language governing permissions and
12+
# limitations under the License.
13+
14+
python:
15+
pip_install: true
16+
extra_requirements:
17+
- doc
18+
- docker
19+
- gcp_api
20+
- emr

‎.travis.yml

+1
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ jdk:
1919
services:
2020
- mysql
2121
- postgresql
22+
- rabbitmq
2223
addons:
2324
apt:
2425
packages:

‎CHANGELOG.txt

+38-32
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
AIRFLOW 1.8.2rc1, 2017-06-05
1+
AIRFLOW 1.8.2, 2017-09-04
22
-------------------------
33

44
9a53e66 [AIRFLOW-809][AIRFLOW-1] Use __eq__ ColumnOperator When Testing Booleans
@@ -46,37 +46,43 @@ bc8e912 [AIRFLOW-XXX] Updating CHANGELOG, README, and UPDATING after 1.8.1 relea
4646
AIRFLOW 1.8.1, 2017-05-09
4747
-------------------------
4848

49-
[AIRFLOW-1160] Update Spark parameters for Mesos
50-
[AIRFLOW 1149] Allow for custom filters in Jinja2 templates
51-
[AIRFLOW-1119] Fix unload query so headers are on first row[]
52-
[AIRFLOW-1089] Add Spark application arguments
53-
[AIRFLOW-1078] Fix latest_runs endpoint for old flask versions
54-
[AIRFLOW-1074] Don't count queued tasks for concurrency limits
55-
[AIRFLOW-1064] Change default sort to job_id for TaskInstanceModelView
56-
[AIRFLOW-1038] Specify celery serialization options explicitly
57-
[AIRFLOW-1036] Randomize exponential backoff
58-
[AIRFLOW-993] Update date inference logic
59-
[AIRFLOW-1167] Support microseconds in FTPHook modification time
60-
[AIRFLOW-1179] Fix Pandas 0.2x breaking Google BigQuery change
61-
[AIRFLOW-1263] Dynamic height for charts
62-
[AIRFLOW-1266] Increase width of gantt y axis
63-
[AIRFLOW-1290] set docs author to 'Apache Airflow'
64-
[AIRFLOW-1166] Speed up _change_state_for_tis_without_dagrun
65-
[AIRFLOW-1192] Some enhancements to qubole_operator
66-
[AIRFLOW-1281] Sort variables by key field by default
67-
[AIRFLOW-1244] Forbid creation of a pool with empty name
68-
[AIRFLOW-1243] DAGs table has no default entries to show
69-
[AIRFLOW-1227] Remove empty column on the Logs view
70-
[AIRFLOW-1226] Remove empty column on the Jobs view
71-
[AIRFLOW-1199] Fix create modal
72-
[AIRFLOW-1200] Forbid creation of a variable with an empty key
73-
[AIRFLOW-1186] Sort dag.get_task_instances by execution_date
74-
[AIRFLOW-1145] Fix closest_date_partition function with before set to True If we're looking for the closest date before, we should take the latest date in the list of date before.
75-
[AIRFLOW-1180] Fix flask-wtf version for test_csrf_rejection
76-
[AIRFLOW-1170] DbApiHook insert_rows inserts parameters separately
77-
[AIRFLOW-1150] Fix scripts execution in sparksql hook[]
78-
[AIRFLOW-1168] Add closing() to all connections and cursors
79-
>>>>>>> Stashed changes
49+
[AIRFLOW-1142] SubDAG Tasks Not Executed Even Though All Dependencies Met
50+
[AIRFLOW-1138] Add licenses to files in scripts directory
51+
[AIRFLOW-1127] Move license notices to LICENSE instead of NOTICE
52+
[AIRFLOW-1124] Do not set all task instances to scheduled on backfill
53+
[AIRFLOW-1120] Update version view to include Apache prefix
54+
[AIRFLOW-1062] DagRun#find returns wrong result if external_trigger=False is specified
55+
[AIRFLOW-1054] Fix broken import on test_dag
56+
[AIRFLOW-1050] Retries ignored - regression
57+
[AIRFLOW-1033] TypeError: can't compare datetime.datetime to None
58+
[AIRFLOW-1017] get_task_instance should return None instead of throw an exception for non-existent TIs
59+
[AIRFLOW-1011] Fix bug in BackfillJob._execute() for SubDAGs
60+
[AIRFLOW-1004] `airflow webserver -D` runs in foreground
61+
[AIRFLOW-1001] Landing Time shows "unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'" on example_subdag_operator
62+
[AIRFLOW-1000] Rebrand to Apache Airflow instead of Airflow
63+
[AIRFLOW-989] Clear Task Regression
64+
[AIRFLOW-974] airflow.util.file mkdir has a race condition
65+
[AIRFLOW-906] Update Code icon from lightning bolt to file
66+
[AIRFLOW-858] Configurable database name for DB operators
67+
[AIRFLOW-853] ssh_execute_operator.py stdout decode default to ASCII
68+
[AIRFLOW-832] Fix debug server
69+
[AIRFLOW-817] Trigger dag fails when using CLI + API
70+
[AIRFLOW-816] Make sure to pull nvd3 from local resources
71+
[AIRFLOW-815] Add previous/next execution dates to available default variables.
72+
[AIRFLOW-813] Fix unterminated unit tests in tests.job (tests/job.py)
73+
[AIRFLOW-812] Scheduler job terminates when there is no dag file
74+
[AIRFLOW-806] UI should properly ignore DAG doc when it is None
75+
[AIRFLOW-794] Consistent access to DAGS_FOLDER and SQL_ALCHEMY_CONN
76+
[AIRFLOW-785] ImportError if cgroupspy is not installed
77+
[AIRFLOW-784] Cannot install with funcsigs > 1.0.0
78+
[AIRFLOW-780] The UI no longer shows broken DAGs
79+
[AIRFLOW-777] dag_is_running is initlialized to True instead of False
80+
[AIRFLOW-719] Skipped operations make DAG finish prematurely
81+
[AIRFLOW-694] Empty env vars do not overwrite non-empty config values
82+
[AIRFLOW-492] Insert into dag_stats table results into failed task while task itself succeeded
83+
[AIRFLOW-139] Executing VACUUM with PostgresOperator
84+
[AIRFLOW-111] DAG concurrency is not honored
85+
[AIRFLOW-88] Improve clarity Travis CI reports
8086

8187
AIRFLOW 1.8.0, 2017-03-12
8288
-------------------------

‎LICENSE

+4-1
Original file line numberDiff line numberDiff line change
@@ -224,7 +224,8 @@ The text of each license is also included at licenses/LICENSE-[project]-[version
224224
(MIT License) Clock Plugin (https://github.com/Lwangaman/jQuery-Clock-Plugin)
225225
(MIT License) WebGL-2D (https://github.com/gameclosure/webgl-2d)
226226
(MIT License) Underscore.js (http://underscorejs.org)
227-
227+
(MIT License) Bootstrap Toggle (http://www.bootstraptoggle.com)
228+
(MIT License) normalize.css (http://git.io/normalize)
228229

229230
========================================================================
230231
BSD 2-Clause licenses
@@ -236,3 +237,5 @@ The text of each license is also included at licenses/LICENSE-[project]-[version
236237
(BSD 2 License) flask-kerberos (https://github.com/mkomitee/flask-kerberos)
237238
(BSD 2 License) Ace (https://github.com/ajaxorg/ace)
238239
(BSD 2 License) d3js (https://d3js.org)
240+
(BSD 3 License) parallel.js (https://parallel.js.org/)
241+

‎MANIFEST.in

+1-1
Original file line numberDiff line numberDiff line change
@@ -21,4 +21,4 @@ graft airflow/www/static
2121
include airflow/alembic.ini
2222
graft scripts/systemd
2323
graft scripts/upstart
24-
24+
graft airflow/config_templates

0 commit comments

Comments
 (0)