Poetry cannot be installed on Github Actions runner using latest installer and Python 3.8.

[KelSolaar]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• [N/A] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows
• Poetry version: Latest
• Link of a Gist with the contents of your pyproject.toml file: N/A

Issue

After changing from the old installer process, i.e. curl -L https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py -o get-poetry.py to the new installer process, i.e. curl -sSL https://install.python-poetry.org/ | POETRY_HOME=$HOME/.poetry python3 -, my builds are failing on Github Actions and Python 3.8: Poetry cannot be installed:

022-09-17T01:37:56.7561313Z ##[group]Run curl -sSL https://install.python-poetry.org | python3 -
2022-09-17T01:37:56.7561741Z �[36;1mcurl -sSL https://install.python-poetry.org | python3 -�[0m
2022-09-17T01:37:56.7572615Z shell: C:\Program Files\Git\bin\bash.EXE --noprofile --norc -e -o pipefail {0}
2022-09-17T01:37:56.7572971Z env:
2022-09-17T01:37:56.7573137Z   CI_PYTHON_VERSION: 3.8
2022-09-17T01:37:56.7573406Z   pythonLocation: C:\hostedtoolcache\windows\Python\3.8.10\x64
2022-09-17T01:37:56.7573650Z ##[endgroup]
2022-09-17T01:37:59.4702925Z Retrieving Poetry metadata
2022-09-17T01:37:59.4703126Z 
2022-09-17T01:37:59.4703425Z # Welcome to Poetry!
2022-09-17T01:37:59.4703691Z 
2022-09-17T01:37:59.4703874Z This will download and install the latest version of Poetry,
2022-09-17T01:37:59.4704250Z a dependency and package manager for Python.
2022-09-17T01:37:59.4704413Z 
2022-09-17T01:37:59.4704801Z It will add the `poetry` command to Poetry's bin directory, located at:
2022-09-17T01:37:59.4705053Z 
2022-09-17T01:37:59.4705200Z C:\Users\runneradmin\AppData\Roaming\Python\Scripts
2022-09-17T01:37:59.4705368Z 
2022-09-17T01:37:59.4705644Z You can uninstall at any time by executing this script with the --uninstall option,
2022-09-17T01:37:59.4705956Z and these changes will be reverted.
2022-09-17T01:37:59.4706100Z 
2022-09-17T01:37:59.4706188Z Installing Poetry (1.2.1)
2022-09-17T01:37:59.4706429Z Installing Poetry (1.2.1): Creating environment
2022-09-17T01:37:59.4706740Z Installing Poetry (1.2.1): An error occurred. Removing partial environment.
2022-09-17T01:37:59.4963567Z Traceback (most recent call last):
2022-09-17T01:37:59.4970320Z   File "<stdin>", line 935, in <module>
2022-09-17T01:37:59.4970900Z   File "<stdin>", line 914, in main
2022-09-17T01:37:59.4971289Z   File "<stdin>", line 545, in run
2022-09-17T01:37:59.4971657Z   File "<stdin>", line 566, in install
2022-09-17T01:37:59.4972175Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\contextlib.py", line 113, in __enter__
2022-09-17T01:37:59.4972602Z     return next(self.gen)
2022-09-17T01:37:59.4972971Z   File "<stdin>", line 638, in make_env
2022-09-17T01:37:59.4973331Z   File "<stdin>", line 624, in make_env
2022-09-17T01:37:59.4973679Z   File "<stdin>", line 314, in make
2022-09-17T01:37:59.4974144Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\venv\__init__.py", line 68, in create
2022-09-17T01:37:59.4974571Z     self._setup_pip(context)
2022-09-17T01:37:59.4975025Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\venv\__init__.py", line 289, in _setup_pip
2022-09-17T01:37:59.4975508Z     subprocess.check_output(cmd, stderr=subprocess.STDOUT)
2022-09-17T01:37:59.4976022Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\subprocess.py", line 415, in check_output
2022-09-17T01:37:59.4976513Z     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
2022-09-17T01:37:59.4977012Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\subprocess.py", line 493, in run
2022-09-17T01:37:59.4977453Z     with Popen(*popenargs, **kwargs) as process:
2022-09-17T01:37:59.4977935Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\subprocess.py", line 858, in __init__
2022-09-17T01:37:59.4978405Z     self._execute_child(args, executable, preexec_fn, close_fds,
2022-09-17T01:37:59.4978916Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\subprocess.py", line 1311, in _execute_child
2022-09-17T01:37:59.4979397Z     hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
2022-09-17T01:37:59.4979834Z FileNotFoundError: [WinError 2] The system cannot find the file specified
2022-09-17T01:37:59.5142470Z ##[error]Process completed with exit code 1.
2022-09-17T01:37:59.5272597Z Cleaning up orphan processes

An exception is raised in the new installer make definition:

    @classmethod
    def make(cls, target: Path) -> "VirtualEnvironment":
        try:
            # on some linux distributions (eg: debian), the distribution provided python
            # installation might not include ensurepip, causing the venv module to
            # fail when attempting to create a virtual environment
            # we import ensurepip but do not use it explicitly here
            import ensurepip  # noqa: F401
            import venv

            builder = venv.EnvBuilder(clear=True, with_pip=True, symlinks=False)
            context = builder.ensure_directories(target)

            if (
                WINDOWS
                and hasattr(context, "env_exec_cmd")
                and context.env_exe != context.env_exec_cmd
            ):
                target = target.resolve()

            builder.create(target)
        except ImportError:
            # fallback to using virtualenv package if venv is not available, eg: ubuntu
            python_version = f"{sys.version_info.major}.{sys.version_info.minor}"
            virtualenv_bootstrap_url = (
                f"https://bootstrap.pypa.io/virtualenv/{python_version}/virtualenv.pyz"
            )

            with tempfile.TemporaryDirectory(prefix="poetry-installer") as temp_dir:
                virtualenv_pyz = Path(temp_dir) / "virtualenv.pyz"
                request = Request(
                    virtualenv_bootstrap_url, headers={"User-Agent": "Python Poetry"}
                )
                virtualenv_pyz.write_bytes(urlopen(request).read())
                cls.run(
                    sys.executable, virtualenv_pyz, "--clear", "--always-copy", target
                )

        # We add a special file so that Poetry can detect
        # its own virtual environment
        target.joinpath("poetry_env").touch()

        env = cls(target)

        # we do this here to ensure that outdated system default pip does not trigger older bugs
        env.pip("install", "--disable-pip-version-check", "--upgrade", "pip")

        return env

Specifically, the venv._setup_pip definition seems to not find the Python interpreter for the Poetry environment:

    def _setup_pip(self, context):
        """Installs or upgrades pip in a virtual environment"""
        # We run ensurepip in isolated mode to avoid side effects from
        # environment vars, the current directory and anything else
        # intended for the global Python environment
        cmd = [context.env_exec_cmd, '-Im', 'ensurepip', '--upgrade',
                                                         '--default-pip']
        subprocess.check_output(cmd, stderr=subprocess.STDOUT)

I have a repo with a small repro case here: https://github.com/KelSolaar/ActionsRamblings/actions/runs/3071692472/jobs/4962601087

[neersighted]

This looks like an interesting edge case with Python 3.8 and Git Bash (or maybe MSYS bash in general?) triggering a bug in venv. See our CI for 3.8 here not using a pipe. Powershell also works very well (and is what is officially documented):

https://github.com/python-poetry/install.python-poetry.org/actions/runs/3005610039/jobs/4826098405

I suppose we could probably work around this by using the virtualenv zipapp in this case, but I wonder if it's worth fixing as we do not officially suggest using anything but Powershell on Windows (CMD and Git Bash are not tested and best effort only as you can always use Powershell). We could just consider it a known issue and let it expire with support for 3.8 in 6 months.

[KelSolaar]

We could just consider it a known issue and let it expire with support for 3.8 in 6 months.

That sound sensible to me!

[Secrus]

We could just consider it a known issue and let it expire with support for 3.8 in 6 months.

I am pretty sure Python 3.8 will be supported for the next 2 years (source). It's Python 3.7 that is EOL in 9 months (not 6).

[KelSolaar]

From a pure selfish standpoint and because I tend to follow Numpy schedule, it is April 2023:

On Apr 14, 2023 drop support for Python 3.8 (initially released on Oct 14, 2019)

[neersighted]

Ah, looks like I got the Numpy schedule in my head. @KelSolaar, could you try reproducing with Python 3.7 just for giggles?

[KelSolaar]

I just did: https://github.com/KelSolaar/ActionsRamblings/actions/runs/3108121099/jobs/5036970221

[neersighted]

Good to know it's the same on 3.7 -- we'll probably just declare this a known issue with Python 3.7 & 3.8 + MSYS Bash.

[Secrus]

So, the issue is only present when using the bash shell on Windows runner?

[neersighted]

That is correct, yes. We haven't drilled down deep enough to figure out if it's truly MSYS bash in general, or just the version shipped by Git (bash on GHA actions is Git Bash), but I think that's mostly unnecessary.

[sondrelg]

Hi 👋 I maintain a Github action for setting up Poetry (this one), and we've had a few reports of broken 3.8 workflows after switching to this installer.

I'm fine with this not being fixed, but wanted to ask: would you recommend using the old script from the main Poetry repo for these targets? That's what I'm planning to do, unless you have a better idea? 🙂

[KelSolaar]

I would need to double-check but I think the main reason for me to open this issue is that the old installer was not working, I updated to the new one which then broke with Python 3.8.

[sondrelg]

The action has previously used an older version (https://raw.githubusercontent.com/python-poetry/poetry/48339106eb0d403a3c66519317488c8185844b32/install-poetry.py) which works fine 🙂

[Secrus]

Hi wave I maintain a Github action for setting up Poetry (this one), and we've had a few reports of broken 3.8 workflows after switching to this installer.

I'm fine with this not being fixed, but wanted to ask: would you recommend using the old script from the main Poetry repo for these targets? That's what I'm planning to do, unless you have a better idea? slightly_smiling_face

In theory, it will install Poetry just fine, however, that script is no longer maintained and any changes to the installer are made here, in this repository. The script in Poetry main repo is scheduled for removal python-poetry/poetry#6676

[neersighted]

We should work just fine with Powershell -- so please use that. If you are experiencing breakage with Python 3.8 and Powershell, that is a different issue from what is tracked here and worth reporting. I would not depend on the old script in the Poetry repo.

[neersighted]

Also, if you really have to use the installer in Git Bash, and the old script works, it's probably worth analyzing why. The original theory here was that it was mostly external to the installer -- but given the old script works, that's evidently incorrect and it's worth figuring out why.

[sondrelg]

From what I can gather, this is the failing call in the installation script, and it fails because Popen is called with 'C:\\Users\\runneradmin\\.local\\venv\\Scripts\\python3.exe' (in my test where we force .local as the install path) when the contents of that dir are ['python.exe', 'pythonw.exe'].

This is a bit odd since the python3.exe executable name is defined here using sys. _base_executable, before the error reported above is raised downstream from here.

I don't have a Windows machine available, so not sure I'm the best to create a fix for this, but the problem seems simple at least 🙂 Any thoughts on this?

[miigotu]

I imagine if python3.exe was added here like python3 is added in the != nt case, it would solve the issue?

[sondrelg]

The old script we were using just did python = env_path.joinpath("Scripts/python.exe") (ref)

[neersighted]

Looks like a bit of misdiagnosis -- the issues are in the internals of the venv module and not related to how we invoke pip. It appears the old script unconditionally uses virtualenv, and the venv API is just broken on some older Python versions under git bash. Our immediate choices (for a quick fix) are to either use venv via the CLI entrypoint/interface (and not the Python API), or to use virtualenv on some (or all) platforms. As we're trying to avoid the external dependency, I think we just need to special case and use virtualenv on these broken Python-windows combinations.

[systematicguy]

Arriving from python-poetry/poetry#3388

I have the same issue when trying to pipe into python3 (worked around for now piping into python.)
I am on windows 10, using pyenv-win, my global python is set to 3.9.6.

[stewartmiles]

FYI: I came across this while root causing this issue outside of poetry (related is actions/runner-images#2690). The problem is due to python/cpython#25216 . It's possible to workaround this by subclassing
venv.EnvBuilder, and then overloading venv.EnvBuilder.ensure_directories() and venv.EnvBuilder.setup_python() to patch context.env_exe to point at the copied executable filename (i.e one of {'python.exe', 'python_d.exe', 'pythonw.exe', 'pythonw_d.exe'}) instead of the name derived from the current interpreter (sys._base_executable).