Wrong dynamic context output for setup_class and teardown_class

[thaiminhpv]

Summary

Wrong dynamic context output for setup_class and teardown_class, compared to setup_method.

P/s: the same happened for setup_module compared to setup_function

Note: this is not frontend issue, I think it might related to coverage.sqldata::CoverageData::contexts_by_lineno function.

Reproducer

# test_class.py
class TestClass:
    def setup_class(self):
        pass
        
    def teardown_class(self):
        pass

    def setup_method(self, method):
        pass
        
    def teardown_method(self, method):
        pass
    
    def test_A(self):
        pass
    
    def test_B(self):
        pass
    
    def test_C(self):
        pass

> ls -a
.  ..  .coveragerc  test_class.py
> cat .coveragerc
[html]
show_contexts = True
> pytest --cov-context=test --cov-report=html --cov=. 
=================================== test session starts ===================================
platform linux -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0
rootdir: ***
plugins: xdist-3.6.1, cov-5.0.0, anyio-3.7.1
collected 3 items

test_class.py ...                                                                   [100%]

---------- coverage: platform linux, python 3.11.9-final-0 -----------
Coverage HTML written to dir htmlcov


==================================== 3 passed in 0.04s ====================================
> open htmlcov/index.html

Expected vs actual result

Notice that setup_class only has test_class.py::TestClass::test_A|setup, while I expect it to either:

1. have context output exactly like setup_method,
2. or only has test_class.py::TestClass::|setup.

The same problem exists for teardown_class.

Actual result

Expected result

by photoshoping, of course

Version

> pip list | grep
coverage                                 7.6.1
pytest                                   8.3.2
pytest-cov                               5.0.0
pytest-xdist                             3.6.1

Not related but I need this fixed to do nedbat/coveragepy#1856

[thaiminhpv]

The same things happens for setup_module compared with setup_function

Minimal reproducer:

def setup_module():
    pass
    
def teardown_module():
    pass

def setup_function(function):
    pass
    
def teardown_function(function):
    pass

def test_A():
    pass

def test_B():
    pass

def test_C():
    pass

[nedbat]

Isn't it the case that setup_class is only run once, the first time it's needed? That's the whole point of setup_class. So it's run before test_A. Similarly, teardown_class is only run once after all of the class's tests are done, so it happens after test_C. There's no way for coverage to know that test_B is involved with setup_class or teardown_class.

If you had only run test_B, then you would see setup_class and teardown_class run for it, but coverage can't know that.

[thaiminhpv]

Thank you for explanation.

def setup_D():
    pass

def B():
    pass

class TestClass:
    def setup_class(self):
        setup_D()
        
    def test_A(self):
        pass
    
    def test_B(self):
        B()

If setup_class call function setup_D(), then when we run pytest, B() does not know whether setup_D() contribute to setup its environment.

As far as I understand, from user point of view, the purpose/motivation of "dynamic context" is to see "which test run this line".

In this case, "dynamic context" has failed to capture the fact that setup_D (or test_class.py::TestClass::setup_class) is testing function B().

Does this align with "dynamic context" feature scope?

I wonder if this is technically impossible, or requires significant change of the algorithm to make it work, or simply doesn't match the intention.

[nedbat]

The dynamic context can only record what actually happens while running the code. That's what the word dynamic means there. I added print statements to your code:

def setup_D():
    print("setup_D")

def B():
    print("B")

class TestClass:
    def setup_class(self):
        print("setup_class")
        setup_D()
        
    def test_A(self):
        print("test_A")
    
    def test_B(self):
        print("test_B")
        B()

Running it with pytest:

% pytest -s -vv issue_658.py
...
collected 2 items

issue_658.py::TestClass::test_A setup_class
setup_D
test_A
PASSED
issue_658.py::TestClass::test_B test_B
B
PASSED

Yes, setup_D affects test_B. But it is not run during test_B. Coverage.py doesn't have a way to connect those two together.

[thaiminhpv]

Oh, so it is a feature, not a bug 🙂. It is just how dynamic context works.

The "dynamic context" feature provides the trace data as-is. So that's why context of test_A|setup is different from context of test_B|setup, although they should be the same, which can make someone confuse/misunderstand if not look carefully.

Processing them to connect theses traces together is the work for another plugin, and therefore beyond the scope of coverage.py.

Thank you, I understand now.