pytest test count is inaccurate #128
Comments
|
See #124 for additional discussion. |
|
I see this in my nose plugin as well |
|
Yeah, I'm not sure how to fix it. Probably the right way to fix it is to use pytest/nose correctly instead of hacking around with manipulating TestSuite and TestCase into being almost but not quite okay for use with pytest and nose, but I haven't had a chance to investigate. |
cdent
added a commit
that referenced
this issue
Jun 10, 2016
This is a backwards compatible improvement to the way gabbi works with pytest. Tests are loaded (and yielded in the same way) but they are filtered more correctly, and fixture start and stops are done more correctly. This allows the test count to be accurate and test selection (using -k) to work properly. Fixes #128 Fixes #123 The details of this change are more clear with an understanding of how pytest plugins and hooks work, but the gist is that a plugin has been created and then forcibly installed. The plugin traverse the collected tests to find and filter out the two tests associated with each suite that start and stop fixtures. These are attached as attributes to the first and last selected (if any) tests in the relevant suite. When a test is asked to run: * if it has a 'starter', the start will run * then the test will run itself * it has a prior that has not run, that will be run * as the tests run if a test with a 'stopper' is found, the stopper will be run
cdent
added a commit
that referenced
this issue
Jun 16, 2016
This is a backwards compatible improvement to the way gabbi works with pytest. Tests are loaded (and yielded in the same way) but they are filtered more correctly, and fixture start and stops are done more correctly. This allows the test count to be accurate and test selection (using -k) to work properly. Fixes #128 Fixes #123 The details of this change are more clear with an understanding of how pytest plugins and hooks work, but the gist is that a plugin has been created and then forcibly installed. The plugin traverse the collected tests to find and filter out the two tests associated with each suite that start and stop fixtures. These are attached as attributes to the first and last selected (if any) tests in the relevant suite. When a test is asked to run: * if it has a 'starter', the start will run * then the test will run itself * it has a prior that has not run, that will be run * as the tests run if a test with a 'stopper' is found, the stopper will be run
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The stop and start methods of the TestSuite (which makes sure fixtures are started) are currently counted as tests, so for each yaml file there are N+2 tests.
Fixing this probably requires becoming properly acquainted with the pytest test collection and fixture handling and not using the current form of yielding tests.
The text was updated successfully, but these errors were encountered: