Infinite recursion caused by _copy_source_tree

[ssbarnea]

Description

When you have a recursive symlink inside your source tree, _copy_source_tree will endup stuck forever. Such recursive symlinks are common in temp folders, like .tox but it seems that .tox already has a special treatment being excluded.

One question is why we do need symlinks=True when calling shutil.copytree?

Expected behavior

Be able to install packages regardless if current repository has or not recursive symlinks.

The fact that .tox is excluded does hide the fact that installing current package as editable within tox would likely create a similar kind of problem.

These symlinks are needed for numerous reasons and we cannot really avoid them. Instead we should assure that _copy_source_tree does never endup trying to perform impossible tasks.

pip/src/pip/_internal/operations/prepare.py

Line 130 in 7ec0fa5

def _copy_source_tree(source, target):

pip version

21.1.1

Python version

3.9.4

OS

MacOS

How to Reproduce

Inside a repository create a recursive symlink:

mkdir .cache
ln -s ../.. .cache/foo

The existence of this file should not affect packaging but it does, at least for some installation methods.

Output

I was able to identify where the endless loop was happening by running pip with python -mtrace --trace --module pip install . >big.log. The interesting bit captured can be seen at https://gist.github.com/ssbarnea/1af810ec6f87a7c5fde7f1f963798421

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

--use-feature=in-tree-build should avoid this issue (and it's likely to become the default behaviour at some point).

[ssbarnea]

@pfmoore Adding parameters to pip is impossible. This bug is about random repositories failing to be install when such links exist. We have zero control on how people are running pip. In fact I am worried that even if we fix it in pip, we would still have occasional lock-ups when someone is running with older versions of pip.

At this moment this bug affects any users of ansible-lint and molecule as both create make use of such symlinks.

I am afraid that the only practical workaround I could look at is to start reusing .tox folder for the cache, as it is excluded by pip already. Those folders are not git tracked but this seems to not matter in this particular case. Most repositories where these tools are used are not python packages but those few that are, are likely to be directly affected by this.

Maybe someone has some better ideas?

It should be noted that these a posix hidden folders and that the expectation is that tools would look inside them by default. Would it make sense to make pip ignore any folder starting with dot?

[ichard26]

Pip hasn't been performing any out of tree builds, which require copying the source tree, in a while.