Changes release image preparation to use PyPI packages #12990
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I am testing this still, but might be good that you start reviewing it @ashb @kaxil. Once I do and we merge it, we should cherry-pick that one to v2-0-test and we can remove the 'Updates providers versions to 1.0.0 (#12955)' as discussed in https://apache-airflow.slack.com/archives/CCPRP7943/p1607617162178600?thread_ts=1607596760.166600&cid=CCPRP7943 |
potiuk
force-pushed
the
change-the-dockerhub-build-process-to-use-pip-packages
branch
from
a721261 to
465f5a2
Compare
potiuk
force-pushed
the
change-the-dockerhub-build-process-to-use-pip-packages
branch
from
465f5a2 to
29b632f
Compare
|
I tested it all, added a few fixes and it's ready to go. I used it now to rebuild and re-push all the rc prod images and it looks good. |
potiuk
force-pushed
the
change-the-dockerhub-build-process-to-use-pip-packages
branch
from
29b632f to
177582c
Compare
|
And small update for test vs. stable for pre-installed packages. We will have less problems if v2-0-test diverges to0 much for a while from v2-0-stable (hopefully NOT too much). |
potiuk
force-pushed
the
change-the-dockerhub-build-process-to-use-pip-packages
branch
2 times, most recently
from
4116aab to
990cc85
Compare
|
The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*. |
potiuk
force-pushed
the
change-the-dockerhub-build-process-to-use-pip-packages
branch
from
990cc85 to
217a86c
Compare
|
@ashb @kaxil -> I updated this PR a little - i clarified some of the docs/parameters and added some more clarifying comments in the Dockerfile. I also changed the default values in the way than when you just run the This is the final step of making the "Production image" true production-ready and once 2.0 is out, it makes it suitable now to make it into the "Docker official image". Once we release 2.0 I will complete the work on that: |
|
No time to review before Monday most likely, if someone else can review sooner 👍 |
kaxil
reviewed
Dec 11, 2020
potiuk
force-pushed
the
change-the-dockerhub-build-process-to-use-pip-packages
branch
from
217a86c to
f3dff00
Compare
Since we released all teh provider packages to PyPI now in RC version, we can now change the mechanism to prepare the production to use released packages in case of tagged builds. The "branch" production images are still prepared using the CI images and .whl packages built from sources, but the release packages are built from officially released PyPI packages. Also some corrections and updates were made to the release process: * the constraint tags when RC candidate is sent should contain rcn suffix. * there was missing step about pushing the release tag once the release is out * pushing tag to GitHub should be done after the PyPI packages are uploaded, so that automated image building in DockerHub can use those packages. * added a note that in case we will release some provider packages that depend on the just released airflow version they shoudl be released after airflow is in PyPI but before the tag is pushed to GitHub (also to allow the image to be build automatically from the released packages) Fixes: apache#12970
potiuk
force-pushed
the
change-the-dockerhub-build-process-to-use-pip-packages
branch
from
f3dff00 to
faee201
Compare
ashb
reviewed
Dec 12, 2020
ashb
reviewed
Dec 12, 2020
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
ashb
approved these changes
Dec 12, 2020
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
github-actions
bot
added
the
full tests needed
potiuk
deleted the
change-the-dockerhub-build-process-to-use-pip-packages
branch
potiuk
added a commit
that referenced
this pull request
Dec 13, 2020
* Changes release image preparation to use PyPI packages Since we released all teh provider packages to PyPI now in RC version, we can now change the mechanism to prepare the production to use released packages in case of tagged builds. The "branch" production images are still prepared using the CI images and .whl packages built from sources, but the release packages are built from officially released PyPI packages. Also some corrections and updates were made to the release process: * the constraint tags when RC candidate is sent should contain rcn suffix. * there was missing step about pushing the release tag once the release is out * pushing tag to GitHub should be done after the PyPI packages are uploaded, so that automated image building in DockerHub can use those packages. * added a note that in case we will release some provider packages that depend on the just released airflow version they shoudl be released after airflow is in PyPI but before the tag is pushed to GitHub (also to allow the image to be build automatically from the released packages) Fixes: #12970 * Update dev/README_RELEASE_AIRFLOW.md Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * Update dev/README_RELEASE_AIRFLOW.md Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> (cherry picked from commit db02773)
kaxil
pushed a commit
that referenced
this pull request
Jan 21, 2021
* Changes release image preparation to use PyPI packages Since we released all teh provider packages to PyPI now in RC version, we can now change the mechanism to prepare the production to use released packages in case of tagged builds. The "branch" production images are still prepared using the CI images and .whl packages built from sources, but the release packages are built from officially released PyPI packages. Also some corrections and updates were made to the release process: * the constraint tags when RC candidate is sent should contain rcn suffix. * there was missing step about pushing the release tag once the release is out * pushing tag to GitHub should be done after the PyPI packages are uploaded, so that automated image building in DockerHub can use those packages. * added a note that in case we will release some provider packages that depend on the just released airflow version they shoudl be released after airflow is in PyPI but before the tag is pushed to GitHub (also to allow the image to be build automatically from the released packages) Fixes: #12970 * Update dev/README_RELEASE_AIRFLOW.md Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * Update dev/README_RELEASE_AIRFLOW.md Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> (cherry picked from commit db02773)
kaxil
pushed a commit
that referenced
this pull request
Jan 22, 2021
* Changes release image preparation to use PyPI packages Since we released all teh provider packages to PyPI now in RC version, we can now change the mechanism to prepare the production to use released packages in case of tagged builds. The "branch" production images are still prepared using the CI images and .whl packages built from sources, but the release packages are built from officially released PyPI packages. Also some corrections and updates were made to the release process: * the constraint tags when RC candidate is sent should contain rcn suffix. * there was missing step about pushing the release tag once the release is out * pushing tag to GitHub should be done after the PyPI packages are uploaded, so that automated image building in DockerHub can use those packages. * added a note that in case we will release some provider packages that depend on the just released airflow version they shoudl be released after airflow is in PyPI but before the tag is pushed to GitHub (also to allow the image to be build automatically from the released packages) Fixes: #12970 * Update dev/README_RELEASE_AIRFLOW.md Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * Update dev/README_RELEASE_AIRFLOW.md Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> (cherry picked from commit db02773)
kaxil
pushed a commit
that referenced
this pull request
Jan 22, 2021
* Changes release image preparation to use PyPI packages Since we released all teh provider packages to PyPI now in RC version, we can now change the mechanism to prepare the production to use released packages in case of tagged builds. The "branch" production images are still prepared using the CI images and .whl packages built from sources, but the release packages are built from officially released PyPI packages. Also some corrections and updates were made to the release process: * the constraint tags when RC candidate is sent should contain rcn suffix. * there was missing step about pushing the release tag once the release is out * pushing tag to GitHub should be done after the PyPI packages are uploaded, so that automated image building in DockerHub can use those packages. * added a note that in case we will release some provider packages that depend on the just released airflow version they shoudl be released after airflow is in PyPI but before the tag is pushed to GitHub (also to allow the image to be build automatically from the released packages) Fixes: #12970 * Update dev/README_RELEASE_AIRFLOW.md Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> * Update dev/README_RELEASE_AIRFLOW.md Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com> (cherry picked from commit db02773)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since we released all teh provider packages to PyPI now in
RC version, we can now change the mechanism to prepare the
production to use released packages in case of tagged builds.
The "branch" production images are still prepared using the
CI images and .whl packages built from sources, but the
release packages are built from officially released PyPI
packages.
Also some corrections and updates were made to the release process:
the constraint tags when RC candidate is sent should contain
rcn suffix.
there was missing step about pushing the release tag once the
release is out
pushing tag to GitHub should be done after the PyPI packages
are uploaded, so that automated image building in DockerHub
can use those packages.
added a note that in case we will release some provider
packages that depend on the just released airflow version
they shoudl be released after airflow is in PyPI but before
the tag is pushed to GitHub (also to allow the image to be
build automatically from the released packages)
Fixes: #12970
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.