TPI: Dynamic Run Pytest

[eleanorjboyd]

Refs: #21148

• macOS @joaomoreno
• linux @rzhao271
• windows @bhavyaus

Complexity: 4

Author: @eleanorjboyd

Create Issue

---

Prerequisites

Use the following repo of tests or use your own repo of pytests https://github.com/eleanorjboyd/inc_dec_example_repo

Background Information

This rewrite is for the underlying infrastructure of how python handles testing therefore all behavior visible to the user should be pretty much the same (minus maybe some error logs going in a better place or test names shortened). Therefore throughout this testing, you can reference the current testing in Python to compare if you think something doesn't look right (or you can always post here too!).

Steps

Part 0: Setup

1. make sure you have the setting "python.experiments.optInto": ["All"]
2. Open the inc_dec_example_repo/inc_dec_example_test_suite testing repo folder in vscode
3. Use the create environment command from the command palette, select venv and whatever python version you want >=3.7
   in the terminal run python -m pip install pytest
4. From the command palette run python: configure tests select pytest and "." (root directory)

Part 1: UI Nesting & Run

1. Go to the testing explorer, and confirm all the tests are discovered

Part 2: Debugging

1. add some break points to the pytest tests and check to make sure these break points are hit
2. make sure you can do other normal debugging actions (stop, restart, step over, etc)

Part 3: Large Files

1. In test_parameterized.py, change @pytest.mark.parametrize("num", range(1, 100)) to be much larger like @pytest.mark.parametrize("num", range(1, 10000)) and make sure it both works for discovery and run tests (tests should be displayed on the test explorer after discovery and run).

Part 4: Seeing Dynamic Run

1. Expand the testing explorer UI to see all the parameterized tests under test_wait_parameterized.py so you can see the nodes 1 - 10.
2. Click run all and make sure that you are seeing the nodes on the test UI finish at different times (ie they don't all resolve together, like some are still spinning while others are complete with a check or an x)

[joaomoreno]

@eleanorjboyd, I've added some simple Markdown formatting to the test plan item. Please remember to make this easy for testers to follow.

make sure you have the setting "python.experiments.optInto": ["All"]

Maybe a safer option would be "python.experiments.optInto": ["pythonTestAdapter"], since All might affect testing in unexpected ways, no?

[joaomoreno]

Go to the testing explorer, and confirm all the tests are discovered

Unsure if these are all the tests:

A better step description would be "confirm that X tests are discovered" or "here's a screenshot of all the tests which should be discovered".

[joaomoreno]

4. From the command palette run python: configure tests select pytest and "." (root directory)

Is this going to be the final UX? Feels strange to me that the user would have to do these manuals steps just to achieve what should've been discovered automatically in the first place since:

• Configure tests should've have to be a user action, but automatic
• pytest can be infered by looking into the sources
• . can be assumed to be the default, while giving the use a non-prompting option to change

Please ignore this if that's already being considered.