-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support parallel JUnit 5 tests #1495
Comments
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this. |
This issue is still relevant. |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this. |
This issue is still relevant. |
@cowwoc have you considered using the Singleton Container pattern? |
@bsideup This goes against what I am trying to do. I am trying to run multiple tests in parallel, each connecting to a separate docker container. Imagine each docker container running a database instance. I don't want one test influencing the data seen by another test. |
I agree we should look into this - the JUnit4 support allowed parallel containers from the beginning, but unfortunately this wasn’t as easy for JUnit5. We should look into this at some point, but to be honest I think our capacity means this will need to come from a community contribution. Sent with GitHawk |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this. |
This issue is still relevant |
Is there any activity on that topic? |
This issue is still relevant. |
Contributions are welcome! :) |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this is a mistake, please reply to this comment to keep it open. If there isn't one already, a PR to fix or at least reproduce the problem in a test case will always help us get back on track to tackle this. |
@Stale bot: I think this is still relevant. |
@bsideup: @marciovmartins submitted this PR: #3220 Do you think this would safely enable parallelism? I would also appreciate much the possibility to run our test containers tests in parallel to drastically reduce our CI time! |
@mfbieber note that you already can do that, e.g. by making the container singleton: |
@bsideup this works with parallel tests in different classes? I think I had problems with that. I'll test again in the next days. |
@marciovmartins this works with _any_class in the current JVM (as it is a "standard" JVM's singleton, nothing fancy), no matter which test framework you're using. |
Thanks, @bsideup for the quick reply! I run into issues using this, though 😢. My tests start to fail when using the suggested singleton pattern. TL;DR: By setting I am running several test classes in parallel by setting
Several Containers are started of course (two, for two classes using testcontainers):
(And to my knowledge, there is no possibility to run several test methods of the same class in parallel in JUnit5. Someone, please correct me, if I am wrong 😸) Now, with the singleton pattern, I have two separate test classes, that both extend the
Those JUnit5 test classes look typically something like this:
This also starts two containers and the tests in the corresponding classes mostly fail:
This only makes sense if the singleton class gets loaded twice. So this must be rather an issue with the Gradle forking mechanism and how that runs the test classes (but I am no expert with that). Since both test classes retrieve the container URL and port through the Also, I was wondering how the mock server knows what response to return if several tests try to access it at once (in the case the singleton pattern would work)? It is interesting though, that the tests pass while not using the singleton pattern. How would you suggest actually running test classes in parallel, so that the singleton pattern works? |
@bsideup My original request was about running JUnit 5 tests in parallel. What do singleton test containers have to do with it? I don't want to reuse instances across tests. I want to run completely independent instances in parallel. |
@cowwoc parallel JUnit 5 tests run in the same JVM, this is why I suggested the singleton approach. If you fork the JVMs (e.g. with your build script), there is not much we can do. |
@bsideup My goal is to run independent tests in parallel and ensure as much as possible that one test cannot impact others. Consequently, while I don't really care whether I use a single or separate JVMs the latter is preferable because it guarantees that tests cannot impact one another. Regardless of whether we have a single or multiple JVMs, I need each test to run against a separate TestContainer instance. |
@cowwoc what you described already works out of the box. Containers started by Testcontainers are isolated from each other (random ports, temporary folders, etc etc) |
@bsideup So you're saying we're able to run parallel JUnit 5 tests and this issue can be closed as fixed? Can you comment on why https://www.testcontainers.org/test_framework_integration/junit_5/ still reads:
Is this out of date or are we missing something? |
At least with my experiments, this seems to be working (but not completely smooth). Once we have a reliable setup, I would also be willing to contribute to at least updating/expanding the documentation. Here is what I did: Concurrent execution via the JUnit5 configuration parameters (see https://junit.org/junit5/docs/snapshot/user-guide/#writing-tests-parallel-execution):
And in the test classes:
Without the
With the
I guess too many containers are started at the same time? Restarting docker "fixed" it. An additional But from time to time, a test fails due to a connection error:, e.g. In general, this results for a smaller project in a speedup from 12m 45s to 4m 57s (295 total tests) on our CI server and on my machine from ca. 20m to 2.5m! |
@mfbieber have you considered making the container static first? It looks like you were starting a container per test. Even without the parallel mode, just making the container static would already give a nice test time reduction. Also, the errors you're getting seem to be related to container startup failures. They may happen essentially, as you're trying to start too many things that once, and they are getting OOM-ed, for example. I would really recommend optimizing your setup (being able to start a container per test in an option, not a must) first. |
Yes, one container per test is okay (and expected from what I described above). And yes, I tried the static and not concurrent approach and the speedup is not that drastic and all tests but the first test fail. I guess the container answers only for the first test with the desired answer. I still don't understand how the mock server will know what to answer if I just have one container functioning as the mock server. I experienced similar issues with the suggested singleton AbstractBaseTestContainer approach. (The execution of the tests is what slows down everything and is, in this case, a result of a retry mechanism on HTTP error codes I am testing - I expect each test to run for ca. 1 minute). And @cowwoc also wanted this parallelism, as I understand his comment:
|
@mfbieber as the mock server process will be reused between the tests, you would obviously need to ensure that one test does not affect another. This can be done by either clearing MockServer's expectations (see https://mock-server.com/mock_server/clearing_and_resetting.html ) or by using per-test URL prefixes, so that each test will work with a unique set of URLs. This is outside of Testcontainers' responsibilities, as we only provide a container definition for MockServer. Starting 10s of 100s of containers per test session is possible, but expensive (time and resources wise). MockServer is implemented in Java and (inside the container) runs a Java process that consumes a good amount of memory (400BM or so, last time I checked). Now, if you have 8 CPUs, you will be running 8 tests in parallel, each starting and stopping 400MB process, resulting in 3.2Gb of memory allocated. Some Docker installations may not have that much memory. tl;dr: Testcontainers does not include any magic and an attempt to start more containers than you have resources on your machine will result in an error (from the containers' side, not in Testcontainers). There is more: starting 4 or 8 (just an example based on a common number of virtual CPU cores available) heavyweight containers at once may result in timeouts because it would consume a lot of CPU. |
@bsideup thanks for your reply and explanations. I am aware that it is resource expensive to run that many containers at once. Since running containers in parallel is possible (and we could close this issue), how about updating the documentation and providing configuration options for doing that? The static or singleton approach really didn't result in a significant speedup. Having to provide per-test URL prefixes, or ensure otherwise that the mock responses don't affect each other, is also very laborious. Regarding the connection timeout, could a |
That's the entire point of this ticket. I don't want a mock server. I want the real thing run in parallel. @mfbieber Instead of a static instance, maybe it's worth trying a |
Maybe, but as you also said, I also want it to run in parallel. The speedup is not really noticeable with just one mock server. |
Junit creates a new instance of your test class for each class, so |
Environment:
Issue: I would also like to see the support for running multiple test containers in parallel, not a singleton container shared between multiple tests running in parallel. I tried @cowwoc's suggestion of com.github.dockerjava.api.exception.BadRequestException: Status 400: {"message":"Duplicate mount point: /dev/shm"}
at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:237)
at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:124)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:33)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13)
at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
at org.testcontainers.shaded.com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:595)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:407)
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:325)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:323)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:311)
at com.axiom.turbo.TestContextManager.beforeEach(TestContextManager.java:43)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeEachCallbacks$1(TestMethodTestDescriptor.java:159)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeMethodsOrCallbacksUntilExceptionOccurs$5(TestMethodTestDescriptor.java:195)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeMethodsOrCallbacksUntilExceptionOccurs(TestMethodTestDescriptor.java:195)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeEachCallbacks(TestMethodTestDescriptor.java:158)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:125)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:65)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:129)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:127)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:126)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:84)
at org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
at java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
at java.base/java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java) My JUnit platform properties:
Not sure if the above exception is a limitation with As an aside, if running multiple Hi @rnorth and @bsideup can you please share what's stopping #3220 from being merged? I see all checks have passed. Is the implementation incomplete? |
@testphreak Testcontainers already supports running containers in parallel, so I guess there is a naming confusion with JUnit's parallel mode. The issue you're getting looks indeed like an issue with Docker, but consider reporting it separately, for BrowserDriverContainer. Nothing is stopping #3220, we just need time to properly review & merge it :) Thanks for reminding about it 👍 |
Hi @bsideup, I think I missed it in the documentation.
How do you run containers in parallel? Is there an example available. I tried |
@testphreak I pushed an example which I use for my parallel test cases with Testcontainers. Test pollution is a major challenge when running test suits in parallel. I think this solution is a good compromise between running everything in parallel and just running classes in parallel. Sometimes you require a fresh container context on class level and sometimes you require a fresh container context on method level. Maybe this helps for your use-case as well? |
@StefanHufschmidt Thank you for this. Out of curiosity, what is the overhead of each approach? For example, is |
Any ETA on this one? |
Would it be possible to make testcontainers create one database (inside the running db instance, not a new full container) per @test method. That would make the progress lighter (not running several db docker instances) but still allow isolation between tests |
Let's say my tests need to start 2 containers (a database + a web server), I wan't to split all my tests in 3 parallel executions. Can't just ask junit5 to split the tests into 3 parallel run, each test running its own container stack - so that each each parallel execution is isolated from the others? (so at the end I'd have 6 containers) |
https://www.testcontainers.org/test_framework_integration/junit_5/ states:
To me, the entire point of using Test Containers is to speed up parallel builds. I could easily get sequential tests working without it.
Please add support for parallel JUnit builds.
The text was updated successfully, but these errors were encountered: