Support for importers plugins

[aperezhortal]

This PR adds support for adding additional importers to the pysteps interface without modifying the pysteps source code. Instead, the users can add custom importers to pysteps by installing external packages, called plugins, that are automatically "discovered" and integrated to pysteps.

How does it work?

Let's suppose that we have a plugin (python package), named importer_abc_xyz, that contains the following importers:
importer_abc_xyz.importers.import_abc_yyy
importer_abc_xyz.importers.import_abc_zzz

When the plugin is installed, it adds entry_points to the python installation advertising the components of this plugin to other packages (in our case, pysteps).

Next, when pysteps is imported, it discovers the available entry_points that belong to the pysteps.plugins.importers group (the group name is defined in the plugin). These discovered importers are added as attributes to the io.importers module, and they are registered to the io.get_method interface.

After the external importers are registered, they become part of the pysteps interface and can be used together with the other available importers.

Plugin project template

To make the creation of the plugin easier, a cookiecutter project template, available at https://github.com/pySTEPS/cookiecutter-pysteps-plugin, can be used to quickly create a plugin project by executing:
$> cookiecutter https://github.com/pySTEPS/cookiecutter-pysteps-plugin
and follow the interactive instructions.

Documentation

• Plugin template documentation: https://cookiecutter-pysteps-plugin.readthedocs.io/en/latest/
• Pysteps documentation (plugins section): https://pysteps.readthedocs.io/en/importer_plugins_support/developer_guide/importer_plugins.html

PR progress [Updated]

• Add plugin support
• Minimal working example
• Add developers guidelines to develop custom plugins (docs here)
• Move the plugin cookiecutter-template to the PySTEPS github
• Improve the plugin cookiecutter-template documentation
• Add more tests (multiple plugins, tests unexpected behaviors, etc.)

[dnerini]

Hi @aperezhortal
This looks like a great idea!
I still have to familiarize myself with the details and give it a try, but I like it already ;-)

So, if I understand correctly, to install a new custom importer, I would have to first run the cookiecutter, then implement my importer in the template, right? After this, how is the registration into pysteps done? Simply using pip install?

One more question, I assume new plugins won't be shared with the rest of the community, correct? These will essentially remain private methods.

Once again, great idea, thanks!

[aperezhortal]

So, if I understand correctly, to install a new custom importer, I would have to first run the cookiecutter, then implement my importer in the template, right?

The first step is creating the plugin (a python package). To facilitate this, cookiecutter (and the cookiecutter-pysteps-plugin template) is used to create the skeleton of the plugin project. Then, the new importers need to be added to the barebone project.

After this, how is the registration into pysteps done? Simply using pip install?

When you install the package using pip install or python setup.py install, it add the entry points defined in the setup.py script to the python distribution. The installation adds the entry points a .txt file located in, for example, <path_to_my_python_environment>/site-packages/<name_of_my_importer_plugin>-.dist-info/entry_points.txt.

Then, when pysteps is importer, it calls the pysteps.io.interface.discover_importers() function that looks for the entry points available in the python distribution and registers the ones that belong to the "pysteps.plugins.importers" group (the group name was defined in the setup.py).

The main advantage of this approach is that the plugins will keep working if the pysteps version is updated without any user intervention.

One more question, I assume new plugins won't be shared with the rest of the community, correct? These will essentially remain private methods.

This is the best part, in my opinion. The users who create the plugins have ownership and they are free to distribute or not the packages. For example, if an organization uses "in-house" tools to read files that cannot be openly distributed, they can internally distribute the plugin.

(I will include the information of these anwers in the plugins documentation)

[dnerini]

This sounds really great, I personally think we should go ahead. What would be the next steps? How can we help?

[aperezhortal]

Great!
The plugin support is already in place. The next step should be writing a tutorial to add a new plugin and write the template documentation. Help with this is more than welcome

I will push the first draft soon with the minimal installation instructions to use it as a starting point. We can then incrementally improve the tutorial to make it as clear and easy to follow as possible.

[codecov]

Codecov Report

Merging #174 (b0fe791) into master (7d89f15) will decrease coverage by 0.20%.
The diff coverage is 72.97%.

@@            Coverage Diff             @@
##           master     #174      +/-   ##
==========================================
- Coverage   75.92%   75.72%   -0.21%     
==========================================
  Files         125      126       +1     
  Lines        9487     9491       +4     
==========================================
- Hits         7203     7187      -16     
- Misses       2284     2304      +20     

Flag	Coverage Δ
unit_tests	75.72% <72.97%> (-0.21%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pysteps/decorators.py	97.50% <ø> (-0.12%)	⬇️
pysteps/feature/blob.py	63.63% <ø> (ø)
pysteps/feature/shitomasi.py	83.78% <ø> (ø)
pysteps/feature/tstorm.py	93.96% <ø> (ø)
pysteps/io/interface.py	65.57% <51.21%> (-29.43%)	⬇️
pysteps/__init__.py	72.00% <100.00%> (+0.28%)	⬆️
pysteps/io/__init__.py	100.00% <100.00%> (ø)
pysteps/tests/test_plugins_support.py	100.00% <100.00%> (ø)
pysteps/tests/test_datasets.py	70.73% <0.00%> (-1.37%)	⬇️
pysteps/tests/test_io_saf_crri.py	85.00% <0.00%> (-1.05%)	⬇️
... and 32 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7d89f15...b0fe791. Read the comment docs.

[dnerini]

Very nice, Andres, very cool contribution!

[RubenImhoff]

Great work @aperezhortal! I have nothing to add. :)