pip install --editable does not work as expected for namespace packages

[jonasrauber]

Environment

• pip version: pip 19.3.1
• Python version: Python 3.6.8
• OS: linux

Description
This is not a duplicate of #3

I have a mainpackage that is a standard package with an __init__.py. It has a subpackage, mainpackage.ext, that is a namespace package according to PEP 420 (no __init__.py).
All of this is located in a main_package folder with a setup.py.

Now I have a separate folder some_plugin with another setup.py. This folder contains a plugin for mainpackage. It therefore has a package mainpackage.ext.someplugin with an __init__.py in mainpackage/ext/someplugin/ but not in mainpackage/ or mainpackage/ext/.

Using a normal pip install, this works as expected. But when using pip install -e to install both main_package and some_plugin, the plugin cannot actually be imported.

Expected behavior
Same behavior between pip install and pip install -e, i.e. the ability to import mainpackage.ext.someplugin without a ModuleNotFoundError.

How to Reproduce
I've created a repository with a minimal example: https://github.com/jonasrauber/python_namespace_packages_editable_bug

Works

sudo pip3 install main_package/
sudo pip3 install some_plugin/

python3 -c "import mainpackage; print(mainpackage.a)"
# -> hello

python3 -c "import mainpackage.ext.someplugin; print(mainpackage.ext.someplugin.a)"
# -> plugin

Fails

sudo pip3 install -e main_package/
sudo pip3 install -e some_plugin/

python3 -c "import mainpackage; print(mainpackage.a)"
# -> hello

python3 -c "import mainpackage.ext.someplugin; print(mainpackage.ext.someplugin.a)"
# -> ERROR

# Traceback (most recent call last):
#   File "<string>", line 1, in <module>
# ModuleNotFoundError: No module named 'mainpackage.ext.someplugin'

[pfmoore]

Does the same behaviour happen if you use python setup.py develop rather than pip install -e? The editable install mechanism is implemented by setuptools, so I suspect this may be down to how setuptools handles PEP 420 namespace packages. Checking setup.py develop will help confirm this.

[jonasrauber]

Using python3 setup.py develop also doesn't work! (I initially reported it would work, but that was wrong.)

[sbidoul]

I think the problem is due to the presence of a __init__.py in your mainpackage.

I recommend this excellent documentation page about namespace packages which says that native namespace packages cannot have __init__.py in any of the distributions.

[sbidoul]

I you absolutely need a __init__.py in your mainpackage, putting

__path__ = __import__('pkgutil').extend_path(__path__, __name__)

in it should work. Make sure you then have no __init__.py in any of your plugins.

Note this is probably on the fringe of supported use cases, so YMMV.

[jonasrauber]

Your two comments seem contradictory. I am going to ignore the first one.

I do obviously need an __init__.py in mainpackage, because mainpackage is a normal package and only mainpackage.ext is a namespace package. This is a totally valid use case that is also supported by pip (just not by pip install -e).

I do have an __init__.py in mainpackage, but not in the mainpackage directory the some_plugin package as it should be.

The error I am reporting only occurs for --editable installs. Standard pip works fine.

[chrahunt]

I don't think this use case is supported by PEP 420.

Note the state of the system in pip install vs pip install -e and how packages could be visible:

With pip install, the contents of both projects are installed to <site-packages-dir>/ by pip, so your site-packages directory looks like:

/tmp/user/1000/tmp.GaM3pzV3dK/env/lib/python3.8/site-packages
├── mainpackage
│   ├── ext
│   │   └── someplugin
│   │       ├── __init__.py
│   │       └── __pycache__
│   │           └── __init__.cpython-38.pyc
│   ├── __init__.py
│   └── __pycache__
│       └── __init__.cpython-38.pyc
├── mainpackage-0.2-py3.8.egg-info
│   ├── dependency_links.txt
│   ├── installed-files.txt
│   ├── not-zip-safe
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   └── top_level.txt
└── someplugin-0.1-py3.8.egg-info
    ├── dependency_links.txt
    ├── installed-files.txt
    ├── not-zip-safe
    ├── PKG-INFO
    ├── SOURCES.txt
    └── top_level.txt

Whether mainpackage/__init__.py came from mainpackage or someplugin depends on which one was installed last - pip just overwrites it.

When we execute python -c 'import mainpackage.ext.someplugin', Python first imports mainpackage according to the PEP 420 specification:

If <directory>/foo/__init__.py is found, a regular package is imported and returned.

Once finished, Python searches for mainpackage.ext in the context of the existing normal package mainpackage (which has a __path__ set to the normal value for normal packages). The import machinery will only search in that package's directory for sub-packages. That's OK because all files were installed to the same location. Python finds mainpackage/ext/ and creates an implicit namespace package, and so on until our plugin module is imported.

With pip install -e, pip invokes setup.py develop which adds the project directories to <site-packages-dir>/easy-install.pth. So our site-packages directory looks like this:

/tmp/user/1000/tmp.UgfRlyFRMH/env/lib/python3.8/site-packages
├── easy-install.pth
├── easy_install.py
├── mainpackage.egg-link
└── someplugin.egg-link

And our project directory looks like this

/tmp/user/1000/tmp.UgfRlyFRMH/repo
├── main_package
│   ├── mainpackage
│   │   ├── ext
│   │   └── __init__.py
│   ├── mainpackage.egg-info
│   │   ├── dependency_links.txt
│   │   ├── not-zip-safe
│   │   ├── PKG-INFO
│   │   ├── SOURCES.txt
│   │   └── top_level.txt
│   └── setup.py
├── README.md
└── some_plugin
    ├── mainpackage
    │   └── ext
    │       └── someplugin
    │           └── __init__.py
    ├── setup.py
    └── someplugin.egg-info
        ├── dependency_links.txt
        ├── not-zip-safe
        ├── PKG-INFO
        ├── SOURCES.txt
        └── top_level.txt

And our easy-install.pth has

/tmp/user/1000/tmp.UgfRlyFRMH/repo/main_package
/tmp/user/1000/tmp.UgfRlyFRMH/repo/some_plugin

When we execute python -c 'import mainpackage.ext.someplugin', Python first executes the site module which will read and add the entries from <site-packages-dir>/easy-install.pth to sys.path. When it tries to import mainpackage it will resolve to /tmp/user/1000/tmp.UgfRlyFRMH/repo/main_package/mainpackage since it was first in our easy-install.pth. When it tries to resolve mainpackage.ext it will only look in the context of the existing package that was found in /tmp/user/1000/tmp.UgfRlyFRMH/repo/main_package/mainpackage.

I would confirm first that namespace packages work this way (with a non-namespace parent) maybe on discuss.python.org/c/packaging and then if this is supposed to work I would reach out on the setuptools issue tracker to see if anyone there has advice.

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.