Optionally run tests within classes with read-write locks concurrently

[kriegaex]

Previously, there was discussion at #3526, which finally led to creation of this feature request.

Use case and benefit

As a test automation developer, I want to be able to make sure that a specific test class runs in isolation, because otherwise it would upset other tests running concurrently. However, within my test class I want methods to run concurrently to decrease overall execution time, because the tests are slow and the structure of the test allows them to run independently and concurrently.

If e.g. my integration test class has 10 test methods, each taking 30 seconds to run, total runtime is 5 minutes when the tests run serially, but would only be 30-40 seconds when run concurrently. However, if I annotate my class with @Isolated, currently (Platform 1.11.3, Jupiter 5.11.3) this enforces all tests within the class to run serially, i.e. one after another.

Deliverables

I am suggesting one or more of the following:

• The JUnit platform is enabled to run all or a subset of test methods within a class (or more generally, sub-elements of a certain tree node) concurrently, even if same tree node as such runs in isolation of other nodes.
• @Isolated gets an optional parameter executionMode defaulting to ExecutionMode.SAME_THREAD, which can be overridden with a value of ExecutionMode.CONCURRENT, resulting in the test class still running in isolation from other classes, but methods within the class running concurrently.
• Ideally, for more fine-granular control on method level, @Execution annotations can be used to override the default intra-class execution mode specified in @Isolated.

[marcphilipp]

Thanks for raising the issue!

As a workaround, if you're using Gradle, it would be relatively easy to define a second test task for all those classes that shouldn't run in parallel but have their children executed concurrently and configure it with the following configuration parameters:

junit.jupiter.execution.parallel.mode.default=concurrent
junit.jupiter.execution.parallel.mode.classes.default=same_thread

Instead of annotating them with @Isolated, you could use a custom @Tag("sequential") or similar and configure one task to exclude and the other to include this tag.

---

Having said that, @Isolated is a special case of a class-level read-write @ResourceLock. As #4367 indicates, being able to override the execution mode of children of classes with such an annotation seems to be desirable in more cases than just for @Isolated.

What currently happens if there's a class-level read-write lock is that the execution mode of all of its children is forced to be SAME_THREAD. That is done because the assumption is that they all need exclusive access to the resource and would conflict with each other. As your use case indicates, that might not always be the case and we should think of a mechanism to avoid this from being enforced.

On the Jupiter side, this could be addressed by extending the existing @Execution annotation (with the target attribute from #4368) like this:

@Isolated
@Execution(value = CONCURRENT, target = CHILDREN, force = true)
class SomeTests {
	// ...
}

The relevant APIs for this on the Platform level are Node.getExclusiveResources() and Node.getExecutionMode(). Somehow, each Node needs to be able to signal that its ExecutionMode should not be overridden. This could be achieved via a new Node.isForcedExecutionMode(): boolean method.

Thoughts?

[leonard84]

The locking system is already complicated, so I'm not sure introducing @Execution(value = CONCURRENT, target = CHILDREN, force = true) is a good idea.
You must consider how it interacts with all the features because some users will inadvertently use them, which I think introduces problems such as this:

@Isolated
@Execution(value = CONCURRENT, target = CHILDREN, force = true)
class SomeTests {
    @ResourceLock("out")
    void testA(){}

    @ResourceLock("out")
    void testB(){}

    void testC(){}

    void testD(){}
}

You would expect that SomeTests runs isolated form other tests, but that testA/testB don't run concurrently with each other while they to run concurrently with testC and testD. However, that is not what would happen. Due to lock coarsening, @ResourceLock("out") would be pulled up to the class and testA and testB would loose their exclusive access due to @Execution(value = CONCURRENT, target = CHILDREN, force = true).

[marcphilipp]

You're right, I had overlooked that use case. 😕