Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Script to run all checks locally #8798

Merged
merged 29 commits into from
Oct 4, 2022

Conversation

Avasam
Copy link
Collaborator

@Avasam Avasam commented Sep 27, 2022

Closes #8686

There's still an issue that needs to be fixed with the mypy test (see #8797), and I would like to be able to seamlessly run the pytype test on windows, if that's even feasible.

Summary output:
image

Please let me know how to better word the "next steps" in create_baseline_stubs.

Notes of things to do in follow-up PR(s):

Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! This looks like a great start.

I have a few questions, a few suggestions, and a few nits ;)

@Avasam Avasam requested a review from AlexWaygood September 27, 2022 20:13
@Avasam Avasam requested a review from AlexWaygood September 29, 2022 00:03
@AlexWaygood
Copy link
Member

AlexWaygood commented Sep 29, 2022

Here's the output currently if I try running `python scripts/runtests.py redis`.
(.venv) C:\Users\alexw\coding\typeshed>python scripts/runtests.py stubs/redis
Looks good! ✨ 🍰 ✨
50 files left unchanged.
All done! ✨ 🍰 ✨
50 files left unchanged.
error finding npx; is Node.js installed?

*** Testing Python 3.11 on win32
Testing third-party packages...
Running mypy --python-version 3.11 --show-traceback --warn-incomplete-stub --show-error-codes --no-error-summary --platform win32 --no-site-packages --custom-typeshed-dir C:\Users\alexw\coding\typeshed --no-implicit-optional --disallow-untyped-decorators --disallow-any-generics --strict-equality --enable-error-code ignore-without-code --config-file /tmp/...
testing redis (50 files)... success

*** Testing Python 3.10 on win32
Testing third-party packages...
Running mypy --python-version 3.10 --show-traceback --warn-incomplete-stub --show-error-codes --no-error-summary --platform win32 --no-site-packages --custom-typeshed-dir C:\Users\alexw\coding\typeshed --no-implicit-optional --disallow-untyped-decorators --disallow-any-generics --strict-equality --enable-error-code ignore-without-code --config-file /tmp/...
testing redis (50 files)... success

*** Testing Python 3.9 on win32
Testing third-party packages...
Running mypy --python-version 3.9 --show-traceback --warn-incomplete-stub --show-error-codes --no-error-summary --platform win32 --no-site-packages --custom-typeshed-dir C:\Users\alexw\coding\typeshed --no-implicit-optional --disallow-untyped-decorators --disallow-any-generics --strict-equality --enable-error-code ignore-without-code --config-file /tmp/...
testing redis (50 files)... success

*** Testing Python 3.8 on win32
Testing third-party packages...
Running mypy --python-version 3.8 --show-traceback --warn-incomplete-stub --show-error-codes --no-error-summary --platform win32 --no-site-packages --custom-typeshed-dir C:\Users\alexw\coding\typeshed --no-implicit-optional --disallow-untyped-decorators --disallow-any-generics --strict-equality --enable-error-code ignore-without-code --config-file /tmp/...
testing redis (50 files)... success

*** Testing Python 3.7 on win32
Testing third-party packages...
Running mypy --python-version 3.7 --show-traceback --warn-incomplete-stub --show-error-codes --no-error-summary --platform win32 --no-site-packages --custom-typeshed-dir C:\Users\alexw\coding\typeshed --no-implicit-optional --disallow-untyped-decorators --disallow-any-generics --strict-equality --enable-error-code ignore-without-code --config-file /tmp/...
testing redis (50 files)... success

--- success, 250 files checked ---
redis... success
Skipping pytype on Windows. You can run the test with WSL.
usage: regr_test.py [-h] [--all] [--platform [{linux,darwin,win32} ...]] [-p [{3.11,3.10,3.9,3.8,3.7} ...]] [packages_to_test ...]
regr_test.py: error: argument packages_to_test: No test cases found for 'redis'!

One or more tests failed. See above for details.
flake8: Success
Check consistent: Success
Check new syntax: Success
pyright: Failed
mypy: Success
stubtest: Success
pytype: Skipped
Regression test: Success

Some notes:

  • It's not obvious currently which tests are running when. pycln and black have very similar "success!!1!1!" messages, for example, and isort doesn't have one at all. It might be nice to add some print() statements before each subprocess.run() call: "Running black...", etc.
  • It looks like pyright_test.py is still marked as "Failed" if npx is not available; it should be marked as "skipped".
  • regr_test.py is correctly marked as "Skipped", but the error message from argparse is still printed to the terminal, which makes it look like it failed. If we use capture_output=True in the subprocess.run() call, we can stop any output being printed to the screen. We can then manually print the captured output in stdout and stderr, if and only if test cases were found. (We should do the same thing with the subprocess.run() call for pyright_test.py, as well.)

@Avasam Avasam requested a review from AlexWaygood September 29, 2022 17:03
@AlexWaygood
Copy link
Member

AlexWaygood commented Sep 30, 2022

(Nit): we currently get a bit of an ugly traceback if the script is interrupted halfway through:

(.venv) C:\Users\alexw\coding\typeshed>python scripts/runtests.py stubs/redis

Running pycln...
Looks good! ✨ 🍰 ✨
50 files left unchanged.
Traceback (most recent call last):
  File "C:\Users\alexw\coding\typeshed\scripts\runtests.py", line 55, in <module>
    subprocess.run([sys.executable, "-m", "pycln", path, "--all"])
  File "C:\Users\alexw\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 503, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "C:\Users\alexw\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1144, in communicate
    self.wait()
  File "C:\Users\alexw\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1207, in wait
    return self._wait(timeout=timeout)
  File "C:\Users\alexw\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1488, in _wait
    result = _winapi.WaitForSingleObject(self._handle,
KeyboardInterrupt
^C

It might be nice to wrap the whole if __name__ == "__main__" block inside a main() function, and then put the whole main() function inside a try/except block to catch KeyboardInterrupts, similar to how we do it in some of our other test scripts:

typeshed/tests/regr_test.py

Lines 179 to 185 in 380022c

if __name__ == "__main__":
try:
code = main()
except KeyboardInterrupt:
print_error("Test aborted due to KeyboardInterrupt!")
code = 1
raise SystemExit(code)

@Avasam
Copy link
Collaborator Author

Avasam commented Oct 1, 2022

(Nit): we currently get a bit of an ugly traceback if the script is interrupted halfway through:
[...]

A traceback can still appear, but it's clearer why.

image

Unless maybe we start piping the output of all tests, checking for a KeyboardInterrupt and then printing the output. I don't think that's really worth it.
As a user, you did interrupt a script mid-execution after all.

@Avasam Avasam requested a review from AlexWaygood October 2, 2022 13:08
@AlexWaygood
Copy link
Member

Unless maybe we start piping the output of all tests, checking for a KeyboardInterrupt and then printing the output. I don't think that's really worth it.

Agreed, it looks good now!

Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some very minor suggestions about some of the terminal output

Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thank you! Just one last comment, but other than that -- I've run out of things to say!

I'll leave this open for a day or two in case any other maintainers want to take a look, but this looks great to me :)

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
@JelleZijlstra JelleZijlstra merged commit e5d52a3 into python:master Oct 4, 2022
@Avasam Avasam deleted the run-all-checks-single-script branch October 5, 2022 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Single script to run PR checks locally
3 participants