[Enhancement]: Support TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE to be read from ~/.testcontainers.properties

[AndreasEK]

Module

Core

Proposal

When I understood these correctly:

• https://java.testcontainers.org/features/configuration/#customizing-docker-host-detection
• https://java.testcontainers.org/supported_docker_environment/#colima
• How to configure Testcontainers to work with Colima as container runtime? #5034

… on mac with colima, the environment variable TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE is required.

This is a nuicance, when acceptance tests are run from the IDE, since then every run configuration has to include this env variable, or maybe you would have to create a template for it, etc. I tried to work around that situation by using ~/.testcontainers.properties for it, but only when I set the actual environment variable, the tests work fine.

# See https://java.testcontainers.org/features/configuration/
#     https://dotnet.testcontainers.org/custom_configuration/

docker.client.strategy=org.testcontainers.dockerclient.EnvironmentAndSystemPropertyClientProviderStrategy

# export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock
# TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE -> docker.socket.override
docker.socket.override=/var/run/docker.sock

# export TESTCONTAINERS_HOST_OVERRIDE=$(colima ls -j | jq -r '.address')
# TESTCONTAINERS_HOST_OVERRIDE -> host.override
host.override=192.168.106.2

# Equivalent to the DOCKER_HOST environment variable. Colons should be escaped.
# export DOCKER_HOST="unix://${HOME}/.colima/default/docker.sock"
docker.host=unix\:///Users/aek/.colima/default/docker.sock

Running Integration Tests only with properties

With only the ~/.testcontainers.properties in place, ryuk cannot be detected:

❯ ./mvnw -P runITs verify
…
2024-04-09T11:28:42.171+02:00  INFO 94314 --- [Worblehat Bookmanager] [           main] o.t.d.DockerClientProviderStrategy       : Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///Users/aek/.colima/default/docker.sock
2024-04-09T11:28:42.173+02:00  INFO 94314 --- [Worblehat Bookmanager] [           main] org.testcontainers.DockerClientFactory   : Docker host IP address is localhost
2024-04-09T11:28:42.201+02:00  INFO 94314 --- [Worblehat Bookmanager] [           main] org.testcontainers.DockerClientFactory   : Connected to docker: 
  Server Version: 24.0.7
  API Version: 1.43
  Operating System: Ubuntu 23.10
  Total Memory: 1952 MB
2024-04-09T11:28:42.248+02:00  INFO 94314 --- [Worblehat Bookmanager] [           main] tc.testcontainers/ryuk:0.6.0             : Creating container for image: testcontainers/ryuk:0.6.0
2024-04-09T11:28:42.420+02:00  INFO 94314 --- [Worblehat Bookmanager] [           main] tc.testcontainers/ryuk:0.6.0             : Container testcontainers/ryuk:0.6.0 is starting: 6e2df29e93ed5ed0468fcf1ef7874729141880a77c450881d9d2e0cfe6d89de0
2024-04-09T11:29:43.096+02:00 ERROR 94314 --- [Worblehat Bookmanager] [           main] tc.testcontainers/ryuk:0.6.0             : Could not start container

java.lang.IllegalStateException: Wait strategy failed. Container is removed
        at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:511)
        at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:354)
 …
Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '.*Started.*'
        at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:47)
        at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52)
        at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:909)
        at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:500)
        ... 117 common frames omitted

Running Integration Tests with TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE

After exporting TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE connection is refused:

❯ export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock
❯ ./mvnw -P runITs verify
…
2024-04-09T11:30:41.183+02:00  INFO 95453 --- [Worblehat Bookmanager] [           main] o.t.d.DockerClientProviderStrategy       : Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///Users/aek/.colima/default/docker.sock
2024-04-09T11:30:41.186+02:00  INFO 95453 --- [Worblehat Bookmanager] [           main] org.testcontainers.DockerClientFactory   : Docker host IP address is localhost
2024-04-09T11:30:41.214+02:00  INFO 95453 --- [Worblehat Bookmanager] [           main] org.testcontainers.DockerClientFactory   : Connected to docker: 
  Server Version: 24.0.7
  API Version: 1.43
  Operating System: Ubuntu 23.10
  Total Memory: 1952 MB
2024-04-09T11:30:41.256+02:00  INFO 95453 --- [Worblehat Bookmanager] [           main] tc.testcontainers/ryuk:0.6.0             : Creating container for image: testcontainers/ryuk:0.6.0
2024-04-09T11:30:41.422+02:00  INFO 95453 --- [Worblehat Bookmanager] [           main] tc.testcontainers/ryuk:0.6.0             : Container testcontainers/ryuk:0.6.0 is starting: 98be78654377709c7a20226dab1d74e5172f5a1ba93d882f4c73c1b59b53c96e
2024-04-09T11:30:42.011+02:00  INFO 95453 --- [Worblehat Bookmanager] [           main] tc.testcontainers/ryuk:0.6.0             : Container testcontainers/ryuk:0.6.0 started in PT0.755355S
2024-04-09T11:30:42.018+02:00  WARN 95453 --- [Worblehat Bookmanager] [containers-ryuk] o.t.utility.RyukResourceReaper           : Can not connect to Ryuk at localhost:32802

java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.Net.pollConnect(Native Method)
        at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
        at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542)
        at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592)
        at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
        at java.base/java.net.Socket.connect(Socket.java:751)
        at org.testcontainers.utility.RyukResourceReaper.lambda$null$1(RyukResourceReaper.java:105)
        at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
        at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$2(RyukResourceReaper.java:101)
        at java.base/java.lang.Thread.run(Thread.java:1583)

Running Integration Tests with TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE & TESTCONTAINERS_HOST_OVERRIDE

❯ export TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock
❯ export TESTCONTAINERS_HOST_OVERRIDE=$(colima ls -j | jq -r '.address')
❯ ./mvnw -P runITs verify
…
2024-04-09T11:33:06.887+02:00  INFO 96744 --- [Worblehat Bookmanager] [           main] o.t.d.DockerClientProviderStrategy       : Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=unix:///Users/aek/.colima/default/docker.sock
2024-04-09T11:33:06.888+02:00  INFO 96744 --- [Worblehat Bookmanager] [           main] org.testcontainers.DockerClientFactory   : Docker host IP address is 192.168.106.2
2024-04-09T11:33:06.919+02:00  INFO 96744 --- [Worblehat Bookmanager] [           main] org.testcontainers.DockerClientFactory   : Connected to docker: 
  Server Version: 24.0.7
  API Version: 1.43
  Operating System: Ubuntu 23.10
  Total Memory: 1952 MB
2024-04-09T11:33:06.979+02:00  INFO 96744 --- [Worblehat Bookmanager] [           main] tc.testcontainers/ryuk:0.6.0             : Creating container for image: testcontainers/ryuk:0.6.0
2024-04-09T11:33:07.147+02:00  INFO 96744 --- [Worblehat Bookmanager] [           main] tc.testcontainers/ryuk:0.6.0             : Container testcontainers/ryuk:0.6.0 is starting: 51fee1161c67056b960ed87faabc665665df43b389acbd3d08e7ea805abdf44b
2024-04-09T11:33:07.770+02:00  INFO 96744 --- [Worblehat Bookmanager] [           main] tc.testcontainers/ryuk:0.6.0             : Container testcontainers/ryuk:0.6.0 started in PT0.790874S
2024-04-09T11:33:07.782+02:00  INFO 96744 --- [Worblehat Bookmanager] [           main] o.t.utility.RyukResourceReaper           : Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
2024-04-09T11:33:07.783+02:00  INFO 96744 --- [Worblehat Bookmanager] [           main] org.testcontainers.DockerClientFactory   : Checking the system...
2024-04-09T11:33:07.784+02:00  INFO 96744 --- [Worblehat Bookmanager] [           main] org.testcontainers.DockerClientFactory   : ✔︎ Docker server version should be at least 1.6.0

[myhau]

I would also like to use this feature. I have exactly the same use-case as @AndreasEK. Having to provide the env variable every time is very inconvenient (especially in the IDE).

By the way, it's already implemented in testcontainers-dotnet. See docker.socket.override and host.override options in this table: Custom Configuration.

Happy to implement this one if maintainers agree that it is a good idea.

[UkonnRa]

+1, I think the problem is in this piece: https://github.com/testcontainers/testcontainers-java/blob/main/core/src/main/java/org/testcontainers/DockerClientFactory.java#L164-L168

In String dockerSocketOverride = System.getenv("TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE");, it only takes env vars as consideration. Hope it will check some fields in DockerClientProviderStrategy as well, currently EnvironmentAndSystemPropertyClientProviderStrategy just ignores docker.socket.override...

I think it just a bug...