Core upgrade of `pyproject.toml` #1229

daquintero · 2023-11-01T13:15:59Z

More details re #1068 and #1148

It's importing, but I haven't been able to run proper tests as I think maybe would be best when I get the API key setup.

Ready to be integrated with docs packaging I think, but I think this needed to be upgraded before for the corresponding docs build scripts to be included.

Worthwhile documentation:
Poetry extras installation configuration
Managing envrionments with poetry

Reproducing the envrionment configuration:

# install pipx
pipx install poetry

Poetry manages the virtual envrionment and reproducible builds as described in its documentation:

Poetry makes project environment isolation one of its core features.
What this means is that it will always work isolated from your global Python installation. To achieve this, it will first check if it’s currently running inside a virtual environment. If it is, it will use it directly without creating a new one. But if it’s not, it will use one that it has already created or create a brand new one for you.

For example, if using an existing mamba environment then you can make poetry work within that envrrionment by:

poetry env use /home/daquintero/.anaconda/envs/tidy3d/bin/python

You can find out information about the poetry envrionment by:

(tidy3d) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry env info

Virtualenv
Python:         3.10.0
Implementation: CPython
Path:           /home/daquintero/.cache/pypoetry/virtualenvs/tidy3d-OEiglnmL-py3.10
Executable:     /home/daquintero/.cache/pypoetry/virtualenvs/tidy3d-OEiglnmL-py3.10/bin/python
Valid:          True

System
Platform:   linux
OS:         posix
Python:     3.10.0
Path:       /home/daquintero/.anaconda/envs/tidy3d
Executable: /home/daquintero/.anaconda/envs/tidy3d/bin/python3.10

It is possible to make it use the system python by:

poetry env use system

You can see how it has changed the environment configuration:

(tidy3d) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry env use system
Deactivating virtualenv: /home/daquintero/.cache/pypoetry/virtualenvs/tidy3d-OEiglnmL-py3.10
(tidy3d) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry env info

Virtualenv
Python:         3.10.0
Implementation: CPython
Path:           /home/daquintero/.anaconda/envs/tidy3d
Executable:     /home/daquintero/.anaconda/envs/tidy3d/bin/python
Valid:          True

System
Platform:   linux
OS:         posix
Python:     3.10.0
Path:       /home/daquintero/.anaconda/envs/tidy3d
Executable: /home/daquintero/.anaconda/envs/tidy3d/bin/python3.10

One thing that I am trying to sort out is having clarity how to use the new pip extra configuration in terms of equivalence and also a proper poetry configuration.

The way to install development extras is through the following commands. There are more details here

poetry install --extras "dev"
poetry install -E dev

tylerflex · 2023-11-01T13:19:27Z

Thanks @daquintero. Just FYI, the tests here (pytest tests/) shouldn't require API key at all, anything using the web is mocked.

daquinteroflex · 2023-11-02T15:57:10Z

Few more comments as porting into the new GitHub account, and won't edit the last one. I will consolidate all these notes to the docs once the standard building flow is sorted, but for now, I am writing them here. Again this is my current state live notes haha.

There is a poetry issue with installing extras the standard way. Ie this complicates say the pip install -e .[dev] installation flow. (And of course, there is the standard poetry installation flow which creates its own virtual environment like above. According to the issue, this could lead to issues on the sphinx requirements build. This is an important comment explaining this. Note that dev-dependencies is equivalent to the <group>.dependencies that they recommend in their more modern versions. In any case, we have a workaround using this approach.

So in summary, to install the project with extras in a particular environment the installation flow is and run tests:

poetry env info # Check information about the poetry environment, configure it as instructions above.
poetry install -E dev # this should install all the development dependencies (note it is much slower than mamba)
poetry run pytest

However, I believe I think maybe the Github action for the run_tests must be updated to install the virtual environment accordingly in CI. I'll configure that iab but worth mentioning.

However on running this:

Note: This error originates from the build backend, and is likely not a problem with poetry but with scipy (1.6.1) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "scipy (==1.6.1)"'.

So this is occuring because scipy is being installed from scatch and is numpy.distutils.system_info.NotFoundError: No BLAS/LAPACK libraries found. To build Scipy from sources, BLAS & LAPACK libraries need to be installed. The question is why is scipy being fixed to this version.

The solution of how this person fixed it does not satisfy me in a production build. Also worth mentioning that the poetry builds are pretty long in the sense of multiple minutes. I'm wondering if this might be a python version problem based on this issue

If I upgrade python to version 3.10 then it builds. I suspect that this might not be a satisfiable solution.

However, I am still running into:

E   ValueError: inferring DataArray dimensions from dictionary like ``coords`` is no longer supported. Use an explicit list of ``dims`` instead.
------------------------------------------------------------------------------------------- Captured stdout --------------------------------------------------------------------------------------------
[17:54:48] WARNING: Default value for the field monitor 'colocate' setting has
           changed to 'True' in Tidy3D 2.4.0. All field components will be
           colocated to the grid boundaries. Set to 'False' to get the raw
           fields on the Yee grid instead.
           WARNING: Default value for the field monitor 'colocate' setting has
           changed to 'True' in Tidy3D 2.4.0. All field components will be
           colocated to the grid boundaries. Set to 'False' to get the raw
           fields on the Yee grid instead.
======================================================================================= short test summary info ========================================================================================
ERROR tests/test_components/test_IO.py - ValueError: inferring DataArray dimensions from dictionary like ``coords`` is no longer supported. Use an explicit list of ``dims`` instead.
ERROR tests/test_components/test_boundaries.py - ValueError: inferring DataArray dimensions from dictionary like ``coords`` is no longer supported. Use an explicit list of ``dims`` instead.
ERROR tests/test_components/test_custom.py - ValueError: inferring DataArray dimensions from dictionary like ``coords`` is no longer supported. Use an explicit list of ``dims`` instead.
ERROR tests/test_components/test_medium.py - ValueError: inferring DataArray dimensions from dictionary like ``coords`` is no longer supported. Use an explicit list of ``dims`` instead.
ERROR tests/test_components/test_monitor.py - ValueError: inferring DataArray dimensions from dictionary like ``coords`` is no longer supported. Use an explicit list of ``dims`` instead.
ERROR tests/test_components/test_simulation.py - ValueError: inferring DataArray dimensions from dictionary like ``coords`` is no longer supported. Use an explicit list of ``dims`` instead.
ERROR tests/test_data/test_monitor_data.py - ValueError: inferring DataArray dimensions from dictionary like ``coords`` is no longer supported. Use an explicit list of ``dims`` instead.
ERROR tests/test_data/test_sim_data.py - ValueError: inferring DataArray dimensions from dictionary like ``coords`` is no longer supported. Use an explicit list of ``dims`` instead.
ERROR tests/test_package/test_parametric_variants.py - ValueError: inferring DataArray dimensions from dictionary like ``coords`` is no longer supported. Use an explicit list of ``dims`` instead.
ERROR tests/test_plugins/test_adjoint.py
ERROR tests/test_plugins/test_component_modeler.py - ValueError: inferring DataArray dimensions from dictionary like ``coords`` is no longer supported. Use an explicit list of ``dims`` instead.
ERROR tests/test_plugins/test_mode_solver.py - ValueError: inferring DataArray dimensions from dictionary like ``coords`` is no longer supported. Use an explicit list of ``dims`` instead.
ERROR tests/test_plugins/test_polyslab.py - ValueError: inferring DataArray dimensions from dictionary like ``coords`` is no longer supported. Use an explicit list of ``dims`` instead.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 13 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================================================================== 13 errors in 7.47s ==========================================================================================

I need to install the old version to verify it does not have to do with me, taking a bit of time since I'm fighting environment settings from my uni work as need to sort out cpu. The older setup.py passes without any tests issues as expected.

I believe the problems might have to do with a versioning issue on xarray. I wonder if the poetry caret specification might explain dependency mismatches from the older version vs poetry. Aha!

On the original setup.py version:

xarray                    2023.10.1                pypi_0    pypi
xz                        5.2.6                h166bdaf_0    conda-forge
zipp                      3.17.0                   pypi_0    pypi

On the poetry installed version:

xarray                    0.16.2                   pypi_0    pypi
xz                        5.2.6                h166bdaf_0    conda-forge
zipp                      3.17.0                   pypi_0    pypi

To see the installed requirements you can use:

poetry show

Just found out the notation of the dependency versions changes in poetry, which means that it's not just a copy paste version management from setup.py to pyproject.toml Turns out this is not a single xarray issue, I think the way that the dependencies versions need to be described needs to be checked on every one. I am on this now and will write a summary.

Now I'm running into some issues with jax installation in poetry.

…etry

daquinteroflex · 2023-11-03T09:26:58Z

Great news! I think it's working now and has equivalent requirements, and tests are passing. So far it's only working till Python 3.10 as I need to understand what's the environmental building problem with Python 3.7 (I wonder if it might be a scipy build inconsistency.

Someone with a mac needs to test this, as I can only test for linux and windows.

(tidy3d) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry env use python
Using virtualenv: /home/daquintero/.cache/pypoetry/virtualenvs/tidy3d-OEiglnmL-py3.10
(tidy3d) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry env info

Virtualenv
Python:         3.10.0
Implementation: CPython
Path:           /home/daquintero/.cache/pypoetry/virtualenvs/tidy3d-OEiglnmL-py3.10
Executable:     /home/daquintero/.cache/pypoetry/virtualenvs/tidy3d-OEiglnmL-py3.10/bin/python
Valid:          True

System
Platform:   linux
OS:         posix
Python:     3.10.0
Path:       /home/daquintero/.anaconda/envs/tidy3d
Executable: /home/daquintero/.anaconda/envs/tidy3d/bin/python3.10
(tidy3d) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry install -E dev
Updating dependencies
Resolving dependencies... (11.9s)

Package operations: 99 installs, 1 update, 0 removals

  • Installing six (1.16.0)
  • Installing jmespath (1.0.1)
  • Installing numpy (1.26.1)
  • Installing python-dateutil (2.8.2)
  • Installing urllib3 (1.26.18)
  • Installing asttokens (2.4.1)
  • Installing botocore (1.26.10)
  • Installing certifi (2023.7.22)
  • Installing charset-normalizer (3.3.2)
  • Installing distlib (0.3.7)
  • Installing exceptiongroup (1.1.3)
  • Installing executing (2.0.1)
  • Installing filelock (3.13.1)
  • Installing idna (3.4)
  • Installing iniconfig (2.0.0)
  • Installing locket (1.0.0)
  • Installing ml-dtypes (0.3.1)
  • Installing packaging (23.2)
  • Installing parso (0.8.3)
  • Installing platformdirs (3.11.0)
  • Installing pluggy (1.3.0)
  • Installing ptyprocess (0.7.0)
  • Installing pure-eval (0.2.2)
  • Installing pytz (2023.3.post1)
  • Installing scipy (1.11.3)
  • Updating setuptools (67.8.0 -> 68.2.2)
  • Installing tomli (2.0.1)
  • Installing toolz (0.12.0)
  • Installing traitlets (5.13.0)
  • Installing tzdata (2023.3)
  • Installing typing-extensions (4.8.0)
  • Installing wcwidth (0.2.9)
  • Installing zipp (3.17.0)
  • Installing annotated-types (0.6.0)
  • Installing cachetools (5.3.2)
  • Installing cfgv (3.4.0)
  • Installing chardet (5.2.0): Installing...
  • Installing click (8.0.3)
  • Installing cloudpickle (3.0.0)
  • Installing colorama (0.4.6)
  • Installing commonmark (0.9.1)
  • Installing click (8.0.3)
  • Installing cloudpickle (3.0.0)
  • Installing colorama (0.4.6)
  • Installing commonmark (0.9.1)
  • Installing chardet (5.2.0)
  • Installing click (8.0.3)
  • Installing cloudpickle (3.0.0)
  • Installing colorama (0.4.6)
  • Installing commonmark (0.9.1)
  • Installing contourpy (1.1.1): Installing...
  • Installing cycler (0.12.1)
  • Installing decorator (5.1.1)
  • Installing cycler (0.12.1)
  • Installing decorator (5.1.1)
  • Installing contourpy (1.1.1)
  • Installing cycler (0.12.1)
  • Installing decorator (5.1.1)
  • Installing fonttools (4.43.1): Pending...
  • Installing fonttools (4.43.1): Installing...
  • Installing fonttools (4.43.1)
  • Installing fsspec (2023.10.0)
  • Installing h5py (3.10.0)
  • Installing identify (2.5.31)
  • Installing importlib-metadata (6.8.0)
  • Installing jaxlib (0.4.14)
  • Installing jedi (0.19.1)
  • Installing kiwisolver (1.4.5)
  • Installing matplotlib-inline (0.1.6)
  • Installing mypy-extensions (1.0.0)
  • Installing nodeenv (1.8.0)
  • Installing opt-einsum (3.3.0)
  • Installing partd (1.4.1)
  • Installing pathspec (0.11.2)
  • Installing pandas (2.1.2)
  • Installing pexpect (4.8.0)
  • Installing pillow (10.1.0)
  • Installing prompt-toolkit (3.0.39)
  • Installing psutil (5.9.6)
  • Installing pydantic-core (2.10.1)
  • Installing pygments (2.16.1)
  • Installing pyparsing (3.1.1)
  • Installing pyproject-api (1.6.1)
  • Installing pytest (7.4.3)
  • Installing pyyaml (6.0.1)
  • Installing requests (2.31.0)
  • Installing s3transfer (0.5.2)
  • Installing stack-data (0.6.3)
  • Installing types-pyyaml (6.0.12.12)
  • Installing virtualenv (20.24.6)
  • Installing black (22.3.0)
  • Installing boto3 (1.23.1)
  • Installing dask (2023.10.1)
  • Installing dill (0.3.7)
  • Installing gdspy (1.6.13)
  • Installing gdstk (0.9.46)
  • Installing h5netcdf (1.0.2)
  • Installing ipython (8.17.2)
  • Installing jax (0.4.14)
  • Installing matplotlib (3.8.1)
  • Installing memory-profiler (0.61.0)
  • Installing networkx (3.0)
  • Installing pre-commit (3.5.0)
  • Installing pydantic (2.4.2)
  • Installing pyjwt (2.8.0)
  • Installing pyroots (0.5.0)
  • Installing pytest-timeout (2.2.0)
  • Installing responses (0.23.3)
  • Installing rich (12.5.1)
  • Installing rtree (1.0.1)
  • Installing ruff (0.1.3)
  • Installing shapely (2.0.2)
  • Installing toml (0.10.2)
  • Installing tox (4.11.3)
  • Installing trimesh (3.20.0)
  • Installing xarray (2023.10.1)

Writing lock file

Installing the current project: tidy3d (2.4.3)
(tidy3d) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ ls
CHANGELOG.md  MANIFEST.in  img               poetry.lock     requirements      simulation_data.hdf5  tidy3d
COPYRIGHT     README.md    make_script.py    pyproject.toml  requirements.txt  test_local.sh         tox.ini
LICENSE       __pycache__  mode_solver.hdf5  pytest.ini      schema.py         tests
(tidy3d) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry run pytest
================================================= test session starts ==================================================platform linux -- Python 3.10.0, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/daquintero/tidy3d
configfile: pytest.ini
plugins: timeout-2.2.0
collected 842 items

tests/test_components/test_IO.py .............................................................                   [  7%]
tests/test_components/test_apodization.py .....                                                                  [  7%]
tests/test_components/test_base.py ......                                                                        [  8%]
tests/test_components/test_boundaries.py ..............                                                          [ 10%]
tests/test_components/test_custom.py ..................................                                          [ 14%]
tests/test_components/test_field_projection.py ...                                                               [ 14%]
tests/test_components/test_geometry.py ......................................................................... [ 23%]
.................................                                                                                [ 27%]
tests/test_components/test_grid.py ..................                                                            [ 29%]
tests/test_components/test_grid_spec.py ......                                                                   [ 30%]
tests/test_components/test_medium.py ............................                                                [ 33%]
tests/test_components/test_meshgenerate.py ...................                                                   [ 35%]
tests/test_components/test_mode.py ....                                                                          [ 36%]
tests/test_components/test_monitor.py ............................                                               [ 39%]
tests/test_components/test_parameter_perturbation.py ...                                                         [ 39%]
tests/test_components/test_sidewall.py ......................................................................... [ 48%]
......................................................................................                           [ 58%]
tests/test_components/test_simulation.py ....................................................................... [ 67%]
.......                                                                                                          [ 67%]
tests/test_components/test_source.py .........                                                                   [ 69%]
tests/test_components/test_types.py ....                                                                         [ 69%]
tests/test_components/test_viz.py ...                                                                            [ 69%]
tests/test_data/test_data_arrays.py .............                                                                [ 71%]
tests/test_data/test_monitor_data.py ......................                                                      [ 73%]
tests/test_data/test_sim_data.py ........................................................                        [ 80%]
tests/test_package/test_config.py ..                                                                             [ 80%]
tests/test_package/test_convert.py ..                                                                            [ 81%]
tests/test_package/test_log.py .........                                                                         [ 82%]
tests/test_package/test_main.py ..                                                                               [ 82%]
tests/test_package/test_make_script.py .                                                                         [ 82%]
tests/test_package/test_material_library.py ....                                                                 [ 83%]
tests/test_package/test_parametric_variants.py ................                                                  [ 84%]
tests/test_plugins/test_adjoint.py .....................................                                         [ 89%]
tests/test_plugins/test_component_modeler.py ............                                                        [ 90%]
tests/test_plugins/test_dispersion_fitter.py .........                                                           [ 91%]
tests/test_plugins/test_mode_solver.py ..............                                                                           [ 93%]
tests/test_plugins/test_polyslab.py ....                                                                                        [ 93%]
tests/test_plugins/test_resonance_finder.py ..........                                                                          [ 95%]
tests/test_plugins/test_waveguide.py ....                                                                                       [ 95%]
tests/test_web/test_cli.py .                                                                                                    [ 95%]
tests/test_web/test_material_fitter.py .                                                                                        [ 95%]
tests/test_web/test_task.py .                                                                                                   [ 95%]
tests/test_web/test_tidy3d_folder.py ...                                                                                        [ 96%]
tests/test_web/test_tidy3d_material_library.py .                                                                                [ 96%]
tests/test_web/test_tidy3d_task.py .........                                                                                    [ 97%]
tests/test_web/test_webapi.py .....................                                                                             [100%]

=================================================== 842 passed in 466.68s (0:07:46) ===================================================
(tidy3d) daquintero@DESKTOP-1T2BQKV:~/tidy3d$

I've been trying to sort out compatibility with Python 3.7, Ok one thing I've noticed is that if you don't delete the poetry.lock there's bound to be problems in dependency compatibility.

With python3.7, python3.8 and python 3.9:

(tidy3dpy38) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry install -E dev
Updating dependencies
Resolving dependencies... (54.2s)

Because tidy3d depends on jaxlib (0.3.14) which doesn't match any versions, version solving failed.
(tidy3dpy38) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry show
Error: poetry.lock not found. Run `poetry lock` to create it.
(tidy3dpy38) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ mamba activate tidy3dpy37
(tidy3dpy37) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry install -E dev
Updating dependencies
Resolving dependencies... (50.3s)

Because tidy3d depends on jaxlib (0.3.14) which doesn't match any versions, version solving failed.
(tidy3dpy37) daquintero@DESKTOP-1T2BQKV:~/tidy3d$
(tidy3dpy37) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ mamba activate tidy3dpy39
(tidy3dpy39) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ rm poetry.lock
rm: cannot remove 'poetry.lock': No such file or directory
(tidy3dpy39) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry env use python
Creating virtualenv tidy3d-OEiglnmL-py3.9 in /home/daquintero/.cache/pypoetry/virtualenvs
Using virtualenv: /home/daquintero/.cache/pypoetry/virtualenvs/tidy3d-OEiglnmL-py3.9
(tidy3dpy39) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ python --version
Python 3.9.0
(tidy3dpy39) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry install -E dev
Updating dependencies
Resolving dependencies... (47.5s)

Because tidy3d depends on jaxlib (0.3.14) which doesn't match any versions, version solving failed.

I'm inclined we should just increase the python requirements of the package if testing is not possible in older versions. Let's chat about this. For now I'm going to fix things to 3.10 for jax.

It would be beneficial to be able to run this in the Github action, I think I need to ask for access. I think we should have a poetry-upgrade branch and merge this there, but I need to check if I have permissions to do this.

Ok latest commit 5a12062..d57902d should be ready to merge into some branch

(tidy3dpy39) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ mamba activate tidy3d
(tidy3d) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry env use python
Using virtualenv: /home/daquintero/.cache/pypoetry/virtualenvs/tidy3d-OEiglnmL-py3.10
(tidy3d) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry install -E dev
Updating dependencies
Resolving dependencies... (12.7s)

No dependencies to install or update

Writing lock file

Installing the current project: tidy3d (2.4.3)
(tidy3d) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry lock
Updating dependencies
Resolving dependencies... (14.3s)
(tidy3d) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry lock
Updating dependencies
Resolving dependencies... (15.0s)
(tidy3d) daquintero@DESKTOP-1T2BQKV:~/tidy3d$ poetry run pytest
========================================================= test session starts =========================================================
platform linux -- Python 3.10.0, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/daquintero/tidy3d
configfile: pytest.ini
plugins: timeout-2.2.0
collected 842 items

tests/test_components/test_IO.py .............................................................                                  [  7%]
tests/test_components/test_apodization.py .....                                                                                 [  7%]
tests/test_components/test_base.py ......                                                                                       [  8%]
tests/test_components/test_boundaries.py ..............                                                                         [ 10%]
tests/test_components/test_custom.py ..................................                                                         [ 14%]
tests/test_components/test_field_projection.py ...                                                                              [ 14%]
tests/test_components/test_geometry.py ........................................................................................ [ 25%]
..................                                                                                                              [ 27%]
tests/test_components/test_grid.py ..................                                                                           [ 29%]
tests/test_components/test_grid_spec.py ......                                                                                  [ 30%]
tests/test_components/test_medium.py ............................                                                               [ 33%]
tests/test_components/test_meshgenerate.py ...................                                                                  [ 35%]
tests/test_components/test_mode.py ....                                                                                         [ 36%]
tests/test_components/test_monitor.py ............................                                                              [ 39%]
tests/test_components/test_parameter_perturbation.py ...                                                                        [ 39%]
tests/test_components/test_sidewall.py ........................................................................................ [ 50%]
.......................................................................                                                         [ 58%]
tests/test_components/test_simulation.py ..............................................................................         [ 67%]
tests/test_components/test_source.py .........                                                                                  [ 69%]
tests/test_components/test_types.py ....                                                                                        [ 69%]
tests/test_components/test_viz.py ...                                                                                           [ 69%]
tests/test_data/test_data_arrays.py .............                                                                               [ 71%]
tests/test_data/test_monitor_data.py ......................                                                                     [ 73%]
tests/test_data/test_sim_data.py ........................................................                                       [ 80%]
tests/test_package/test_config.py ..                                                                                            [ 80%]
tests/test_package/test_convert.py ..                                                                                           [ 81%]
tests/test_package/test_log.py .........                                                                                        [ 82%]
tests/test_package/test_main.py ..                                                                                              [ 82%]
tests/test_package/test_make_script.py .                                                                                        [ 82%]
tests/test_package/test_material_library.py ....                                                                                [ 83%]
tests/test_package/test_parametric_variants.py ................                                                                 [ 84%]
tests/test_plugins/test_adjoint.py .....................................                                                        [ 89%]
tests/test_plugins/test_component_modeler.py ............                                                                       [ 90%]
tests/test_plugins/test_dispersion_fitter.py .........                                                                          [ 91%]
tests/test_plugins/test_mode_solver.py ..............                                                                           [ 93%]
tests/test_plugins/test_polyslab.py ....                                                                                        [ 93%]
tests/test_plugins/test_resonance_finder.py ..........                                                                          [ 95%]
tests/test_plugins/test_waveguide.py ....                                                                                       [ 95%]
tests/test_web/test_cli.py .                                                                                                    [ 95%]
tests/test_web/test_material_fitter.py .                                                                                        [ 95%]
tests/test_web/test_task.py .                                                                                                   [ 95%]
tests/test_web/test_tidy3d_folder.py ...                                                                                        [ 96%]
tests/test_web/test_tidy3d_material_library.py .                                                                                [ 96%]
tests/test_web/test_tidy3d_task.py .........                                                                                    [ 97%]
tests/test_web/test_webapi.py .....................                                                                             [100%]

=================================================== 842 passed in 489.65s (0:08:09) ===================================================
(tidy3d) daquintero@DESKTOP-1T2BQKV:~/tidy3d$

tylerflex · 2023-11-03T11:55:32Z

pyproject.toml

@@ -44,6 +44,7 @@ pre-commit = "*"
 black = "22.3.0"
 ruff = "*"
 tox = "*"
+pylint = "*"


we stopped using pylint by the way, since it stopped working properly with pydantic 2.*. Something maybe you could also look at if you have time.

tylerflex · 2023-11-03T11:56:31Z

Another comment, we will stop supporting python 3.7 in tidy3d 2.5.*

tylerflex · 2023-11-03T11:57:23Z

I have a Mac, can you tell me what commands I should run and I can try it out? Thanks

daquinteroflex · 2023-11-03T12:27:21Z

No rush, are we supporting >=3.10? Or still have to support ~3.8, 3.9? poetry is struggling to resolve the jax requirements for those python versions but I can get to the bottom of it if we're supporting those versions.

So I've just re-verified that tests pass with the documentation requirements now included for >=3.10. What I do to test is:

# So far the only version that passes due to poetry being unable to resolve jax requirements for python<3.9, so this only works when python>=3.10 as in the pyproject.toml
cd tidy3d/
poetry env use python # should install a venv for the python version described
poetry install -E dev # install development packages
poetry run pytest # Run tests

I get on ubuntu:

=================================================== 842 passed in 412.80s (0:06:52) ====================================================

This is for 94aa817

…s pass

daquinteroflex · 2023-11-03T15:29:10Z

I have upgraded the github run_tests action so that it installs poetry, the envrionment from poetry and runs tox accordingly. I have not yet removed the python3.7<3.10 because I'm curious how this goes in terms of reproducibility of the errors in my local environment. Will approve the workflow to test and see how goes.

Discovered some issues with the poetry install pre 3.8 so maybe we have no choice but not to support them. There are some more instructions to follow. Ok, we're not supporting below 3.8.

…e action setup instructions

…e action setup instructions yaml link

… not working otherwise

… issue activating venv

… environment properly configured

daquinteroflex · 2023-11-03T17:00:24Z

Very close on all tests passing but not close enough! A bit tricky to debug but will do next week

daquinteroflex · 2023-11-06T11:17:10Z

Well I reckon this is pretty nice news that all tests are passing. I've made a poetry_upgrade branch where I intend to merge all the documentation into, alongside the installation scripts that will contain the documentation builds.

What do you think about this plan in terms of priority/execution @tylerflex :

This solves higher priority Configure package in pyproject.toml #1068
We merge into a temporary branch such as poetry_upgrade until I write the instructions for the team to setup the new development environment in the new documentation.
We then merge documentation and the source code since currently the build requirements are already ported into pyproject.toml since it will sort out reproducible builds/deployment issues - following this Merge docs and source repositories #1231
Then we can solve these a lot more straightforwardly in order of priority: Docs on physical constants have incorrect descriptions #1213 , Exclude some fields from documentation without making them private autoflex#11 and improve docs #1148

Pending to decide:

Would you like to review PRs in smaller chunks as we merge say into this poetry_upgrade branch or afterwards when everything goes into develop?
Are we happy with only maintaining python>=3.10 or should I delve deep into >=3.8 support? I haven't got them working after first-pass overview attempts to solve.
Are we happy to have a single test script rather than tox driven things, which is doable, but involved poetry-tox configuration that currently I bypass by moving all the tox scripts into the standard GitHub Actions?

tylerflex · 2023-11-06T20:56:39Z

Thanks @daquintero I will hopefully take a look at this and your other PRs in more detail tomorrow. But I think we still want to support use of python 3.8 on the front end at least. (if installed by, say pip). Is that still supported? And if so, maybe we should modify the tests to use pip (instead of poetry) to install if python < 3.10?

tylerflex · 2023-11-06T20:57:38Z

Do you still need me to test on Mac or is that settled (since it seems to work in the automated tests?)

daquinteroflex · 2023-11-07T09:25:17Z

Sounds good, I'll try to sort out the support for >3.8 now. I think no need to test as the tests passing guarantees it should work on Mac.

tylerflex · 2023-11-07T16:21:15Z

What do you think about this plan in terms of priority/execution @tylerflex :

This solves higher priority Configure package in pyproject.toml #1068

We merge into a temporary branch such as poetry_upgrade until I write the instructions for the team to setup the new development environment in the new documentation.

We then merge documentation and the source code since currently the build requirements are already ported into pyproject.toml since it will sort out reproducible builds/deployment issues - following this Merge docs and source repositories #1231

Then we can solve these a lot more straightforwardly in order of priority: Docs on physical constants have incorrect descriptions #1213 , Exclude some fields from documentation without making them private #1214 and improve docs #1148

I think this plan sounds good. just to clarify one thing: for step 3, will we merge the docs/tidy3d PR into our temporary poetry_upgrade branch? or into develop?

Would you like to review PRs in smaller chunks as we merge say into this poetry_upgrade branch or afterwards when everything goes into develop?

Probably smaller chunks.

Are we happy with only maintaining python>=3.10 or should I delve deep into >=3.8 support? I haven't got them working after first-pass overview attempts to solve.

Would be nice to support >= 3.8. best to check with backend folks to see what python versions they will need to have supported. I think we just need to make sure of two things.

The new build can be run on our production servers.
A user running python 3.8 can still use tidy3d with pip install tidy3d. (If they are unable to install dependency groups like jax, that might be an issue, but we could ask them to upgrade their python maybe).

Are we happy to have a single test script rather than tox driven things, which is doable, but involved poetry-tox configuration that currently I bypass by moving all the tox scripts into the standard GitHub Actions?

Fine by me, I think. tox was introduced mainly as a way to loop over OS and python versions, but if we can do it ourselves, no big deal from my perspective.

daquinteroflex · 2023-11-07T17:19:35Z

Ok so I've done the changes for pip install -e .[dev] now. I still need to sort out >3.8 support. I'd be inclined in doing a one big merge when we've finished the restructure and tests and everything so for now maybe poetry_upgrade? Yeah let's see!

daquinteroflex · 2023-11-08T09:54:23Z

I think turns out I accidentally fixed the support for python>=3.8 when I did the package installation changes. I believe this might be ready to merge into our temporary branch

I'll merge now into our poetry_upgrade branch and then we can rebase into develop on that PR when it comes in.

weiliangjin2021 · 2023-11-08T20:35:25Z

Just find this PR that discusses poetry env. I have two general questions about poetry env that always puzzles me. Maybe @daquintero you have explained it somewhere already.

Say that I've installed tidy3d to a virtual env managed by poetry, poetry recommends invoking the env by running poetry run python [code.py] or poetry shell. But this seems to only work if the command runs inside the tidy3d folder. In my experience, I won't run a simulation inside the repo folder unless running unit tests. So I'm quite confused about this design.
Is there a way to install python packages to a particular path. In pip, one can do PYTHONUSERBASE=SOME_PATH pip install ... In Poetry, it always installs in an virtual env, either local or under the user's folder.

daquinteroflex · 2023-11-09T11:05:45Z

Hi Weiliang,

Thanks for the questions.

Say that I've installed tidy3d to a virtual env managed by poetry, poetry recommends invoking the env by running poetry run python [code.py] or poetry shell. But this seems to only work if the command runs inside the tidy3d folder. In my experience, I won't run a simulation inside the repo folder unless running unit tests. So I'm quite confused about this design.

There's a few interesting things to mention. So as of our current implementation poetry and pip are interchangeable. Which means that you'll be able to do pretty much everything you used to do before using pip, ie pip install tidy3d[dev] in your own venv or anything else you might already have done. You could in theory have a dual managed pip and poetry environment which you can activate anywhere.

The other thing is that you have full control over the poetry virtual environment, and it's the same as any other python virtual environment with just a few more different commands. There's a few more instructions and explanations in this poetry env docs. For reference, to show you can use it beyond the actual project directory, I used to to test this before:

mamba create -n tidy3denv python==3.10 # create venv with mamba
mamba activate tidy3denv # activate the venv
poetry env use python # using the mamba venv python now
poetry env info # verify the venvs used by poetry and mamba
cd anywhere 
# you can use the python activated venv anywhere.

I've only used poetry within the project directory, and know you can use standard pip or mamba outside it, but I haven't yet tested with poetry. If you specifically want to use poetry outside I can probably find the set of commands to do that, but I think it'll just have to do with managing the poetry env use to one that is well defined for the project configuration. What do you think?

Is there a way to install python packages to a particular path. In pip, one can do PYTHONUSERBASE=SOME_PATH pip install ... In Poetry, it always installs in an virtual env, either local or under the user's folder.

Yep all the pip installor pythonpath functionality is still valid! It just comes down to configuring and managing the python environment you want to use, but you don't have to use poetry as you can use what you already do, even if I think you could do it with poetry if you wanted. poetry is just used to manage dependencies and releases, but you should be able to do everything you wanted to do before too.

weiliangjin2021 · 2023-11-09T17:34:05Z

Hi Dario, thank you for the detailed answer!

As for the 2nd question, I guess in one case, one has to use poetry install but not pip install: install dependence based on poetry.lock. In that case, are you aware of any poetry equivalence to PYTHONUSERBASE=SOME_PATH pip install?

daquinteroflex · 2023-11-10T09:59:18Z

I am thinking about this currently and will get back to you when I've got a clearer idea soon. So the poetry.lock is used just to compare the pyproject.toml with the verified environment that generates the lockfile, but it's only used in poetry.

My hunch on your query is that it has do to with configuring the poetry environment in a particular installation configuration for local path packages, which I think should be possible, but will look in detail to make sure it's equivalent to what you aim to do.

daquintero added 7 commits October 30, 2023 11:51

FEAT: Easy badge updates

4ce6eaf

FEAT: Upgraded all labels

e93ce2f

FEAT: A few more fun labels

eee3b5a

FEAT: First draft at a pyproject.toml without poetry and dynamic config

476c363

FEAT: Backbone poetry pyproject.toml

c4de44b

FEAT: Pre verification tests

c26d58d

FEAT: How equivalent is this?

27e8759

daquintero added 4 commits November 1, 2023 13:29

FIX: Something is getting installed except the poetry.extras

b4a0607

FEAT: Working pyproject.toml base installation 🚀

4e8f3ad

FEAT: Integrated old pyproject.toml into new pyproject.toml

103384f

FEAT: Working core pyproject.toml installation mostly equivalent

388b6de

daquintero marked this pull request as draft November 1, 2023 16:09

FEAT: Upgraded to the new group dependencies vs devdependencies in po…

598b1f0

…etry

daquintero added 3 commits November 3, 2023 09:51

Removed double dev dependencies

af2ef93

FEAT: Tests passing in python 3.10

5a12062

FEAT: Ready to merge

d57902d

tylerflex reviewed Nov 3, 2023

View reviewed changes

FEAT: Included documentation dependencies

395a713

FIX: Reverified poetry build including now docs requirements and test…

94aa817

…s pass

daquinteroflex changed the base branch from develop to poetry_upgrade November 3, 2023 15:02

daquinteroflex marked this pull request as ready for review November 3, 2023 15:03

daquinteroflex self-assigned this Nov 3, 2023

FEAT: Upgrade tox.ini and github actions to test with poetry

344e515

daquintero added 8 commits November 3, 2023 17:00

FIX: Something going wrong with the poetry install

1503e4b

FIX: Something going wrong with the poetry install, integrated exampl…

052596e

…e action setup instructions

FIX: Something going wrong with the poetry install, integrated exampl…

9814957

…e action setup instructions yaml link

FIX: Something going wrong with the poetry remove support 3.7

6ef064f

FIX: Something going wrong with the poetry remove support 3.7 typo

b0b8bdf

FIX: Something going wrong with the poetry remove support below 3.10,…

9c675ee

… not working otherwise

FIX: Something going wrong with the poetry remove support below 3.10,…

564935e

… issue activating venv

FIX: Something going wrong with the poetry remove support below 3.10,…

e4fe95b

… environment properly configured

FIX: Sort out github actions tk envrionment from tox port

0a6a776

FIX: Fixed pip extra install

28509ea

BUG: First attempt fix python support >3.8

0a77a4b

daquinteroflex merged commit 8a9bf6b into flexcompute:poetry_upgrade Nov 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Core upgrade of `pyproject.toml` #1229

Core upgrade of `pyproject.toml` #1229

daquintero commented Nov 1, 2023 •

edited

Loading

tylerflex commented Nov 1, 2023

daquinteroflex commented Nov 2, 2023 •

edited

Loading

daquinteroflex commented Nov 3, 2023 •

edited

Loading

tylerflex Nov 3, 2023

tylerflex commented Nov 3, 2023

tylerflex commented Nov 3, 2023

daquinteroflex commented Nov 3, 2023 •

edited

Loading

daquinteroflex commented Nov 3, 2023 •

edited

Loading

daquinteroflex commented Nov 3, 2023

daquinteroflex commented Nov 6, 2023 •

edited

Loading

tylerflex commented Nov 6, 2023

tylerflex commented Nov 6, 2023

daquinteroflex commented Nov 7, 2023

tylerflex commented Nov 7, 2023 •

edited by daquinteroflex

Loading

daquinteroflex commented Nov 7, 2023

daquinteroflex commented Nov 8, 2023 •

edited

Loading

weiliangjin2021 commented Nov 8, 2023

daquinteroflex commented Nov 9, 2023

weiliangjin2021 commented Nov 9, 2023

daquinteroflex commented Nov 10, 2023 •

edited

Loading

Core upgrade of pyproject.toml #1229

Core upgrade of pyproject.toml #1229

Conversation

daquintero commented Nov 1, 2023 • edited Loading

tylerflex commented Nov 1, 2023

daquinteroflex commented Nov 2, 2023 • edited Loading

daquinteroflex commented Nov 3, 2023 • edited Loading

tylerflex Nov 3, 2023

Choose a reason for hiding this comment

tylerflex commented Nov 3, 2023

tylerflex commented Nov 3, 2023

daquinteroflex commented Nov 3, 2023 • edited Loading

daquinteroflex commented Nov 3, 2023 • edited Loading

daquinteroflex commented Nov 3, 2023

daquinteroflex commented Nov 6, 2023 • edited Loading

tylerflex commented Nov 6, 2023

tylerflex commented Nov 6, 2023

daquinteroflex commented Nov 7, 2023

tylerflex commented Nov 7, 2023 • edited by daquinteroflex Loading

daquinteroflex commented Nov 7, 2023

daquinteroflex commented Nov 8, 2023 • edited Loading

weiliangjin2021 commented Nov 8, 2023

daquinteroflex commented Nov 9, 2023

weiliangjin2021 commented Nov 9, 2023

daquinteroflex commented Nov 10, 2023 • edited Loading

Core upgrade of `pyproject.toml` #1229

Core upgrade of `pyproject.toml` #1229

daquintero commented Nov 1, 2023 •

edited

Loading

daquinteroflex commented Nov 2, 2023 •

edited

Loading

daquinteroflex commented Nov 3, 2023 •

edited

Loading

daquinteroflex commented Nov 3, 2023 •

edited

Loading

daquinteroflex commented Nov 3, 2023 •

edited

Loading

daquinteroflex commented Nov 6, 2023 •

edited

Loading

tylerflex commented Nov 7, 2023 •

edited by daquinteroflex

Loading

daquinteroflex commented Nov 8, 2023 •

edited

Loading

daquinteroflex commented Nov 10, 2023 •

edited

Loading