Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Colima #531

Closed
wants to merge 36 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
3d251b3
Add host port wait strategy to static helper
cristianrgreco Apr 6, 2023
15a7271
Implement and use CompositeWaitStrategy
cristianrgreco Apr 6, 2023
c05ae8c
Add colima workflow, disable others
cristianrgreco Apr 11, 2023
ef16dd7
Enable other jobs
cristianrgreco Apr 11, 2023
c38135a
Update colima workflow
cristianrgreco Apr 11, 2023
ab1931d
Update workflow
cristianrgreco Apr 11, 2023
159fba4
Update workflow
cristianrgreco Apr 11, 2023
0447b1d
Update workflow
cristianrgreco Apr 11, 2023
2e78961
Update workflow
cristianrgreco Apr 11, 2023
4f5cbe1
Update workflow
cristianrgreco Apr 11, 2023
d4bb41e
Update workflow
cristianrgreco Apr 11, 2023
efc660d
Update workflow
cristianrgreco Apr 11, 2023
ed49fbc
Update workflow
cristianrgreco Apr 11, 2023
bf93050
Update workflow
cristianrgreco Apr 11, 2023
265b608
Fix some flakiness
cristianrgreco Apr 12, 2023
fd76837
CompositeWaitStrategy defaults to max startupTimeout
cristianrgreco Apr 12, 2023
524584e
CompositeWaitStrategy fixes
cristianrgreco Apr 12, 2023
d4dc170
Remove unused var from test
cristianrgreco Apr 12, 2023
1bf199e
Re-add delay
cristianrgreco Apr 12, 2023
a0c384c
Document composite wait strategy
cristianrgreco Apr 12, 2023
2ba80d1
Use retry mechanism instead of listening ports for Reaper
cristianrgreco Apr 13, 2023
1317d2f
Increase reaper retries to 10
cristianrgreco Apr 13, 2023
ef94d4e
Docker event stream test helper should not fail on invalid JSON
cristianrgreco Apr 13, 2023
c33921f
Fix flaky test
cristianrgreco Apr 13, 2023
7e8bb9a
Try to install Docker via GHA
cristianrgreco Apr 14, 2023
804bcf0
Revert install Docker action
cristianrgreco Apr 14, 2023
3ce60fb
Enforce IPv4
cristianrgreco Apr 14, 2023
edec96e
Update pull policy test
cristianrgreco Apr 14, 2023
f962918
Update CompositeWaitStrategy as per GH discussion
cristianrgreco Apr 14, 2023
1362c89
Increase kafka test timeout
cristianrgreco Apr 14, 2023
116aed9
Remove pull stream parser, as it can hide errors
cristianrgreco Apr 14, 2023
05ea5df
Merge branch 'main' into colima-support-rebased
cristianrgreco Apr 17, 2023
67c0afc
Increase colima daemon disk
cristianrgreco Apr 17, 2023
0e91aaf
Increase elasticsearch test timeout
cristianrgreco Apr 17, 2023
0b24813
Adjust startup timeout/test timeout of all modules
cristianrgreco Apr 17, 2023
7074f7e
Increase startup timeouts of Kafka/MySQL modules
cristianrgreco Apr 18, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions .github/workflows/test-main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ jobs:
os-version: [ ubuntu-latest ]
node-version: [ 14.x, 16.x, 18.x ]
steps:
- name: Code checkout
- name: Code checkout
uses: actions/checkout@v3
with:
Expand Down Expand Up @@ -100,6 +101,36 @@ jobs:
CI_ROOTLESS: true
CI_PODMAN: true

colima:
runs-on: ${{ matrix.os-version }}
strategy:
fail-fast: false
matrix:
os-version: [ macos-latest ]
node-version: [ 14.x, 16.x, 18.x ]
steps:
- name: Code checkout
uses: actions/checkout@v3
- name: Setup Colima
run: |
brew install docker docker-compose
colima start --cpu 3 --memory 14 --disk 14 --runtime docker
colima status
colima --version
- name: Set environment
run: |
echo "DOCKER_HOST=unix://${HOME}/.colima/default/docker.sock" >> $GITHUB_ENV
echo "TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock" >> $GITHUB_ENV
echo "NODE_OPTIONS=--dns-result-order=ipv4first" >> $GITHUB_ENV
- name: Install NodeJS ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}
- name: Install dependencies
run: npm ci --omit=optional
- name: Run tests
run: npm run test:ci

smoke-test:
runs-on: ${{ matrix.os-version }}
strategy:
Expand Down
30 changes: 30 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,36 @@ jobs:
CI_ROOTLESS: true
CI_PODMAN: true

colima:
runs-on: ${{ matrix.os-version }}
strategy:
fail-fast: false
matrix:
os-version: [ macos-latest ]
node-version: [ 14.x, 16.x, 18.x ]
steps:
- name: Code checkout
uses: actions/checkout@v3
- name: Setup Colima
run: |
brew install docker docker-compose
colima start --cpu 3 --memory 14 --disk 14 --runtime docker
colima status
colima --version
- name: Set environment
run: |
echo "DOCKER_HOST=unix://${HOME}/.colima/default/docker.sock" >> $GITHUB_ENV
echo "TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE=/var/run/docker.sock" >> $GITHUB_ENV
echo "NODE_OPTIONS=--dns-result-order=ipv4first" >> $GITHUB_ENV
- name: Install NodeJS ${{ matrix.node-version }}
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node-version }}
- name: Install dependencies
run: npm ci --omit=optional
- name: Run tests
run: npm run test:ci

smoke-test:
runs-on: ${{ matrix.os-version }}
strategy:
Expand Down
17 changes: 13 additions & 4 deletions src/docker-compose-environment/docker-compose-environment.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ describe("DockerComposeEnvironment", () => {

const startedEnv1 = await env.up();
const dockerEventStream = await getDockerEventStream();
const dockerPullEventPromise = waitForDockerEvent(dockerEventStream, "pull", 2);
const dockerPullEventPromise = waitForDockerEvent(dockerEventStream, "pull");
const startedEnv2 = await env.withPullPolicy(new AlwaysPullPolicy()).up();
await dockerPullEventPromise;

Expand Down Expand Up @@ -89,8 +89,14 @@ describe("DockerComposeEnvironment", () => {

it("should support log message wait strategy", async () => {
const startedEnvironment = await new DockerComposeEnvironment(fixtures, "docker-compose.yml")
.withWaitStrategy(await composeContainerName("container"), Wait.forLogMessage("Listening on port 8080"))
.withWaitStrategy(await composeContainerName("another_container"), Wait.forLogMessage("Listening on port 8080"))
.withWaitStrategy(
await composeContainerName("container"),
Wait.forAll([Wait.forListeningPorts(), Wait.forLogMessage("Listening on port 8080")])
)
.withWaitStrategy(
await composeContainerName("another_container"),
Wait.forAll([Wait.forListeningPorts(), Wait.forLogMessage("Listening on port 8080")])
)
.up();

await Promise.all(
Expand All @@ -115,7 +121,10 @@ describe("DockerComposeEnvironment", () => {

it("should support health check wait strategy", async () => {
const startedEnvironment = await new DockerComposeEnvironment(fixtures, "docker-compose-with-healthcheck.yml")
.withWaitStrategy(await composeContainerName("container"), Wait.forHealthCheck())
.withWaitStrategy(
await composeContainerName("container"),
Wait.forAll([Wait.forListeningPorts(), Wait.forHealthCheck()])
)
.up();

await checkEnvironmentContainerIsHealthy(startedEnvironment, await composeContainerName("container"));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ import { StartedDockerComposeEnvironment } from "./started-docker-compose-enviro
import { dockerComposeDown } from "../docker-compose/functions/docker-compose-down";
import { dockerComposeUp } from "../docker-compose/functions/docker-compose-up";
import { waitForContainer } from "../wait-for-container";
import { Wait } from "../wait-strategy/wait";
import { DefaultPullPolicy, PullPolicy } from "../pull-policy";
import { dockerComposePull } from "../docker-compose/functions/docker-compose-pull";
import { Wait } from "../wait-strategy/wait";

export class DockerComposeEnvironment {
private readonly composeFilePath: string;
Expand Down
2 changes: 1 addition & 1 deletion src/modules/arangodb/arangodb-container.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ export class ArangoDBContainer extends GenericContainer {

public override async start(): Promise<StartedArangoContainer> {
this.withExposedPorts(...(this.hasExposedPorts ? this.opts.exposedPorts : [ARANGODB_PORT]))
.withWaitStrategy(Wait.forLogMessage("Have fun!"))
.withWaitStrategy(Wait.forAll([Wait.forListeningPorts(), Wait.forLogMessage("Have fun!")]))
.withEnvironment({ ARANGO_ROOT_PASSWORD: this.password })
.withStartupTimeout(120_000);

Expand Down
2 changes: 1 addition & 1 deletion src/modules/elasticsearch/elasticsearch-container.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import { ElasticsearchContainer } from "./elasticsearch-container";
import { Client } from "@elastic/elasticsearch";

describe("ElasticsearchContainer", () => {
jest.setTimeout(180_000);
jest.setTimeout(240_000);

// createIndex {
it("should create an index", async () => {
Expand Down
2 changes: 1 addition & 1 deletion src/modules/elasticsearch/elasticsearch-container.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ export class ElasticsearchContainer extends GenericContainer {
target: "/usr/share/elasticsearch/config/jvm.options.d/elasticsearch-default-memory-vm.options",
},
])
.withStartupTimeout(120_000);
.withStartupTimeout(180_000);

return new StartedElasticsearchContainer(await super.start());
}
Expand Down
2 changes: 1 addition & 1 deletion src/modules/kafka/kafka-container.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import * as fs from "fs";
import * as path from "path";

describe("KafkaContainer", () => {
jest.setTimeout(240_000);
jest.setTimeout(360_000);

// connectBuiltInZK {
it("should connect using in-built zoo-keeper", async () => {
Expand Down
2 changes: 1 addition & 1 deletion src/modules/kafka/kafka-container.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ export class KafkaContainer extends GenericContainer {
constructor(image = KAFKA_IMAGE) {
super(image);

this.withExposedPorts(KAFKA_PORT).withStartupTimeout(180_000).withEnvironment({
this.withExposedPorts(KAFKA_PORT).withStartupTimeout(300_000).withEnvironment({
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "BROKER:PLAINTEXT,PLAINTEXT:PLAINTEXT",
KAFKA_INTER_BROKER_LISTENER_NAME: "BROKER",
KAFKA_BROKER_ID: "1",
Expand Down
2 changes: 1 addition & 1 deletion src/modules/mongodb/mongodb-container.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import { MongoDBContainer, StartedMongoDBContainer } from "./mongodb-container";
import mongoose from "mongoose";

describe("MongodbContainer", () => {
jest.setTimeout(240_000);
jest.setTimeout(180_000);

// connect4 {
it("should work using default version 4.0.1", async () => {
Expand Down
2 changes: 1 addition & 1 deletion src/modules/mongodb/mongodb-container.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ export class MongoDBContainer extends GenericContainer {
public override async start(): Promise<StartedMongoDBContainer> {
this.withExposedPorts(MONGODB_PORT)
.withCommand(["--replSet", "rs0"])
.withWaitStrategy(Wait.forLogMessage(/.*waiting for connections.*/i))
.withWaitStrategy(Wait.forAll([Wait.forListeningPorts(), Wait.forLogMessage(/.*waiting for connections.*/i)]))
.withStartupTimeout(120_000);
return new StartedMongoDBContainer(await super.start());
}
Expand Down
2 changes: 1 addition & 1 deletion src/modules/mysql/mysql-container.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ export class MySqlContainer extends GenericContainer {
MYSQL_USER: this.username,
MYSQL_PASSWORD: this.userPassword,
})
.withStartupTimeout(120_000);
.withStartupTimeout(180_000);

return new StartedMySqlContainer(
await super.start(),
Expand Down
34 changes: 14 additions & 20 deletions src/modules/nats/nats-container.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,6 @@ import { AbstractStartedContainer } from "../abstract-started-container";
import { Wait } from "../../wait-strategy/wait";

const CLIENT_PORT = 4222;
const ROUTING_PORT_FOR_CLUSTERING = 6222;
const HTTP_MANAGEMENT_PORT = 8222;

const USER_ARGUMENT_KEY = "--user";
const PASS_ARGUMENT_KEY = "--pass";
Expand Down Expand Up @@ -35,18 +33,6 @@ export class NatsContainer extends GenericContainer {
return this;
}

private static ensureDashInFrontOfArgumentName(name: string): string {
if (name.startsWith("--") || name.startsWith("-")) {
return name;
}

if (name.length == 1) {
return "-" + name;
} else {
return "--" + name;
}
}

public override async start(): Promise<StartedNatsContainer> {
function buildCmdsFromArgs(args: { [p: string]: string }): string[] {
const result: string[] = [];
Expand All @@ -60,17 +46,25 @@ export class NatsContainer extends GenericContainer {
}

this.withCommand(buildCmdsFromArgs(this.args))
.withExposedPorts(
...(this.hasExposedPorts
? this.opts.exposedPorts
: [CLIENT_PORT, ROUTING_PORT_FOR_CLUSTERING, HTTP_MANAGEMENT_PORT])
)
.withWaitStrategy(Wait.forLogMessage(/.*Server is ready.*/))
.withExposedPorts(...(this.hasExposedPorts ? this.opts.exposedPorts : [CLIENT_PORT]))
.withWaitStrategy(Wait.forAll([Wait.forListeningPorts(), Wait.forLogMessage(/.*Server is ready.*/)]))
.withStartupTimeout(120_000);

return new StartedNatsContainer(await super.start(), this.getUser(), this.getPass());
}

private static ensureDashInFrontOfArgumentName(name: string): string {
if (name.startsWith("--") || name.startsWith("-")) {
return name;
}

if (name.length == 1) {
return "-" + name;
} else {
return "--" + name;
}
}

private getUser(): string {
return this.args[USER_ARGUMENT_KEY];
}
Expand Down
4 changes: 2 additions & 2 deletions src/modules/neo4j/neo4j-container.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import neo4j from "neo4j-driver";
import { Neo4jContainer } from "./neo4j-container";

describe("Neo4jContainer", () => {
jest.setTimeout(180_000);
jest.setTimeout(240_000);

// createNode {
it("should create a person node", async () => {
Expand Down Expand Up @@ -48,7 +48,7 @@ describe("Neo4jContainer", () => {

// apoc {
it("should have APOC plugin installed", async () => {
const container = await new Neo4jContainer().withApoc().withStartupTimeout(120_000).start();
const container = await new Neo4jContainer().withApoc().start();
const driver = neo4j.driver(
container.getBoltUri(),
neo4j.auth.basic(container.getUsername(), container.getPassword())
Expand Down
4 changes: 2 additions & 2 deletions src/modules/neo4j/neo4j-container.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,9 @@ export class Neo4jContainer extends GenericContainer {

public override async start(): Promise<StartedNeo4jContainer> {
this.withExposedPorts(...(this.hasExposedPorts ? this.opts.exposedPorts : [BOLT_PORT, HTTP_PORT]))
.withWaitStrategy(Wait.forLogMessage("Started."))
.withWaitStrategy(Wait.forAll([Wait.forListeningPorts(), Wait.forLogMessage("Started.")]))
.withEnvironment({ NEO4J_AUTH: `${USERNAME}/${this.password}` })
.withStartupTimeout(120_000);
.withStartupTimeout(180_000);

if (this.apoc) {
this.withEnvironment({
Expand Down
7 changes: 6 additions & 1 deletion src/modules/postgresql/postgresql-container.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,12 @@ export class PostgreSqlContainer extends GenericContainer {
POSTGRES_USER: this.username,
POSTGRES_PASSWORD: this.password,
})
.withWaitStrategy(Wait.forLogMessage(/.*database system is ready to accept connections.*/, 2))
.withWaitStrategy(
Wait.forAll([
Wait.forListeningPorts(),
Wait.forLogMessage(/.*database system is ready to accept connections.*/, 2),
])
)
.withStartupTimeout(120_000);

return new StartedPostgreSqlContainer(await super.start(), this.database, this.username, this.password);
Expand Down
6 changes: 3 additions & 3 deletions src/wait-strategy/health-check-wait-strategy.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ describe("HealthCheckWaitStrategy", () => {
const customGenericContainer = await GenericContainer.fromDockerfile(context).build();
const container = await customGenericContainer
.withExposedPorts(8080)
.withWaitStrategy(Wait.forHealthCheck())
.withWaitStrategy(Wait.forAll([Wait.forListeningPorts(), Wait.forHealthCheck()]))
.start();

await checkContainerIsHealthy(container);
Expand All @@ -32,7 +32,7 @@ describe("HealthCheckWaitStrategy", () => {
retries: 5,
startPeriod: 1000,
})
.withWaitStrategy(Wait.forHealthCheck())
.withWaitStrategy(Wait.forAll([Wait.forListeningPorts(), Wait.forHealthCheck()]))
.start();

await checkContainerIsHealthy(container);
Expand Down Expand Up @@ -84,7 +84,7 @@ describe("HealthCheckWaitStrategy", () => {
retries: 5,
startPeriod: 1000,
})
.withWaitStrategy(Wait.forHealthCheck())
.withWaitStrategy(Wait.forAll([Wait.forListeningPorts(), Wait.forHealthCheck()]))
.start();

await checkContainerIsHealthy(container);
Expand Down
2 changes: 1 addition & 1 deletion src/wait-strategy/host-port-wait-strategy.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ describe("HostPortWaitStrategy", () => {
.withExposedPorts(8081)
.withStartupTimeout(0)
.start()
).rejects.toThrowError("Port 8081 not bound after 0ms");
).rejects.toThrowError(/Port \d+ not bound after 0ms/);

expect(await getRunningContainerNames()).not.toContain(containerName);
});
Expand Down
4 changes: 2 additions & 2 deletions src/wait-strategy/log-wait-strategy.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ describe("LogWaitStrategy", () => {
it("should wait for log", async () => {
const container = await new GenericContainer("cristianrgreco/testcontainer:1.1.14")
.withExposedPorts(8080)
.withWaitStrategy(Wait.forLogMessage("Listening on port 8080"))
.withWaitStrategy(Wait.forAll([Wait.forListeningPorts(), Wait.forLogMessage("Listening on port 8080")]))
.start();

await checkContainerIsHealthy(container);
Expand All @@ -20,7 +20,7 @@ describe("LogWaitStrategy", () => {
it("should wait for log with regex", async () => {
const container = await new GenericContainer("cristianrgreco/testcontainer:1.1.14")
.withExposedPorts(8080)
.withWaitStrategy(Wait.forLogMessage(/Listening on port \d+/))
.withWaitStrategy(Wait.forAll([Wait.forListeningPorts(), Wait.forLogMessage(/Listening on port \d+/)]))
.start();

await checkContainerIsHealthy(container);
Expand Down