Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: use testcontainers-go in the integration tests #1129

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

chore: use testcontainers-go in the integration tests #1129

wants to merge 2 commits into from
+519 −301

Conversation

mdelapenya
Copy link

@mdelapenya mdelapenya commented Aug 8, 2024 •
edited
Loading

Hello from the Testcontainers for Go community, core maintainer here 👋

This PR is a PoC so you can evaluate if using testcontainers-go is useful for the project. I checked that dhui/dktest is already used and it's maintained by one of the core maintainers here, so I'm fine if this PR is discarded because of that. In the end testcontainers-go is already using the docker package, adding some sugar such as wait strategies and Go modules for certain technologies. I'll be super happy to share more context in detail if needed.

Regarding the PR:

What does this PR do?

It uses testcontainers-go v0.32.0 (latest release) in the cassandra and clickhouse packages, in order to simplify the experience of setting up and customise the container images representing those database technologies.

I've replaced the parallel calls to dktest using the testing library (testcases + t.Run + t.Parallel), so one container will be started per test. This can be improved to start just one container per suite, although I'm not familiar enough with the code base to touch that part yet. I'm pretty sure you'll indentify potential improvements in that regard.

Why is it important?

Testcontainers comes with ~50 Go modules representing the most popular databases out there, that could be used to represent the integrations for this wonderful library to run the migrations against them. Please check the modules out here: https://testcontainers.com/modules/?language=go

How to test this

Fetch the PR and move to the cassandra and clickhouse packages. From there, run `go test -v -count=1 ./...".

Related issues

@mdelapenya
Copy link
Author

mdelapenya commented Aug 8, 2024 •
edited
Loading

I'm seeing Error: ../../../go/pkg/mod/github.com/dhui/dktest@v0.4.1/dktest.go:38:48: undefined: types.ImagePullOptions in the CI probably caused by the usage of a higher version of the docker package (testcontainers-go uses v27).

Only running the cassandra and clickhouse tests, this does not happen:

➜ clickhouse (testcontainers-go) ✗ go test -v ./... -count=1
=== RUN   TestCases
=== RUN   TestCases/Test_TinyLog
=== RUN   TestCases/Test_TinyLog/yandex/clickhouse-server:21.3
2024/08/08 17:40:37 github.com/testcontainers/testcontainers-go - Connected to docker: 
  Server Version: 82+testcontainerscloud (via Testcontainers Desktop 1.15.1)
  API Version: 1.46
  Operating System: Ubuntu 20.04 LTS
  Total Memory: 7407 MB
  Testcontainers for Go Version: v0.32.0
  Resolved Docker Host: tcp://127.0.0.1:53614
  Resolved Docker Socket Path: /var/run/docker.sock
  Test SessionID: f4d2909631f6acfd8de5168b5c9e04f353c6316e77c73fe8ecf56d0e2eb40c2e
  Test ProcessID: b4f6fd75-379b-432e-913b-56c2b6defb9c
2024/08/08 17:40:37 🐳 Creating container for image testcontainers/ryuk:0.7.0
2024/08/08 17:40:37 ✅ Container created: a585879e48d4
2024/08/08 17:40:37 🐳 Starting container: a585879e48d4
2024/08/08 17:40:37 ✅ Container started: a585879e48d4
2024/08/08 17:40:37 ⏳ Waiting for container id a585879e48d4 image: testcontainers/ryuk:0.7.0. Waiting for: &{Port:8080/tcp timeout:<nil> PollInterval:100ms}
2024/08/08 17:40:37 🔔 Container is ready: a585879e48d4
2024/08/08 17:40:37 🐳 Creating container for image yandex/clickhouse-server:21.3
2024/08/08 17:40:37 ✅ Container created: 208c3bbbfb2e
2024/08/08 17:40:37 🐳 Starting container: 208c3bbbfb2e
2024/08/08 17:40:38 ✅ Container started: 208c3bbbfb2e
2024/08/08 17:40:38 ⏳ Waiting for container id 208c3bbbfb2e image: yandex/clickhouse-server:21.3. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0x140000e0000]}
2024/08/08 17:40:40 🔔 Container is ready: 208c3bbbfb2e
=== RUN   TestCases/Test_TinyLog/yandex/clickhouse-server:21.3/set_1_dirty
=== RUN   TestCases/Test_TinyLog/yandex/clickhouse-server:21.3/re-set_1_dirty
=== RUN   TestCases/Test_TinyLog/yandex/clickhouse-server:21.3/set_2_clean
=== RUN   TestCases/Test_TinyLog/yandex/clickhouse-server:21.3/re-set_2_clean
=== RUN   TestCases/Test_TinyLog/yandex/clickhouse-server:21.3/last_migration_dirty
=== RUN   TestCases/Test_TinyLog/yandex/clickhouse-server:21.3/last_migration_clean
=== RUN   TestCases/Migrate_TinyLog
=== RUN   TestCases/Migrate_TinyLog/yandex/clickhouse-server:21.3
2024/08/08 17:40:41 🐳 Creating container for image yandex/clickhouse-server:21.3
2024/08/08 17:40:41 ✅ Container created: 0966cf9b3b8f
2024/08/08 17:40:41 🐳 Starting container: 0966cf9b3b8f
2024/08/08 17:40:41 ✅ Container started: 0966cf9b3b8f
2024/08/08 17:40:41 ⏳ Waiting for container id 0966cf9b3b8f image: yandex/clickhouse-server:21.3. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0x14000510090]}
2024/08/08 17:40:44 🔔 Container is ready: 0966cf9b3b8f
    migrate_testing.go:30: UP
=== RUN   TestCases/Version_TinyLog
=== RUN   TestCases/Version_TinyLog/yandex/clickhouse-server:21.3
2024/08/08 17:40:44 🐳 Creating container for image yandex/clickhouse-server:21.3
2024/08/08 17:40:44 ✅ Container created: e4aa2dcf522e
2024/08/08 17:40:44 🐳 Starting container: e4aa2dcf522e
2024/08/08 17:40:44 ✅ Container started: e4aa2dcf522e
2024/08/08 17:40:44 ⏳ Waiting for container id e4aa2dcf522e image: yandex/clickhouse-server:21.3. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0x14000510240]}
2024/08/08 17:40:47 🔔 Container is ready: e4aa2dcf522e
=== RUN   TestCases/Drop_TinyLog
=== RUN   TestCases/Drop_TinyLog/yandex/clickhouse-server:21.3
2024/08/08 17:40:47 🐳 Creating container for image yandex/clickhouse-server:21.3
2024/08/08 17:40:47 ✅ Container created: eda3e97cb1e7
2024/08/08 17:40:47 🐳 Starting container: eda3e97cb1e7
2024/08/08 17:40:47 ✅ Container started: eda3e97cb1e7
2024/08/08 17:40:47 ⏳ Waiting for container id eda3e97cb1e7 image: yandex/clickhouse-server:21.3. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0x1400019b170]}
2024/08/08 17:40:50 🔔 Container is ready: eda3e97cb1e7
=== RUN   TestCases/Test_MergeTree
=== RUN   TestCases/Test_MergeTree/yandex/clickhouse-server:21.3
2024/08/08 17:40:50 🐳 Creating container for image yandex/clickhouse-server:21.3
2024/08/08 17:40:50 ✅ Container created: 23b9d0c36898
2024/08/08 17:40:50 🐳 Starting container: 23b9d0c36898
2024/08/08 17:40:51 ✅ Container started: 23b9d0c36898
2024/08/08 17:40:51 ⏳ Waiting for container id 23b9d0c36898 image: yandex/clickhouse-server:21.3. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0x14000534240]}
2024/08/08 17:40:53 🔔 Container is ready: 23b9d0c36898
=== RUN   TestCases/Test_MergeTree/yandex/clickhouse-server:21.3/set_1_dirty
=== RUN   TestCases/Test_MergeTree/yandex/clickhouse-server:21.3/re-set_1_dirty
=== RUN   TestCases/Test_MergeTree/yandex/clickhouse-server:21.3/set_2_clean
=== RUN   TestCases/Test_MergeTree/yandex/clickhouse-server:21.3/re-set_2_clean
=== RUN   TestCases/Test_MergeTree/yandex/clickhouse-server:21.3/last_migration_dirty
=== RUN   TestCases/Test_MergeTree/yandex/clickhouse-server:21.3/last_migration_clean
=== RUN   TestCases/Migrate_MergeTree
=== RUN   TestCases/Migrate_MergeTree/yandex/clickhouse-server:21.3
2024/08/08 17:40:54 🐳 Creating container for image yandex/clickhouse-server:21.3
2024/08/08 17:40:54 ✅ Container created: e7f66941cabb
2024/08/08 17:40:54 🐳 Starting container: e7f66941cabb
2024/08/08 17:40:54 ✅ Container started: e7f66941cabb
2024/08/08 17:40:54 ⏳ Waiting for container id e7f66941cabb image: yandex/clickhouse-server:21.3. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0x14000510630]}
2024/08/08 17:40:57 🔔 Container is ready: e7f66941cabb
    migrate_testing.go:30: UP
=== RUN   TestCases/Version_MergeTree
=== RUN   TestCases/Version_MergeTree/yandex/clickhouse-server:21.3
2024/08/08 17:40:57 🐳 Creating container for image yandex/clickhouse-server:21.3
2024/08/08 17:40:57 ✅ Container created: 6268932d75b3
2024/08/08 17:40:57 🐳 Starting container: 6268932d75b3
2024/08/08 17:40:57 ✅ Container started: 6268932d75b3
2024/08/08 17:40:57 ⏳ Waiting for container id 6268932d75b3 image: yandex/clickhouse-server:21.3. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0x14000510120]}
2024/08/08 17:41:00 🔔 Container is ready: 6268932d75b3
=== RUN   TestCases/Drop_MergeTree
=== RUN   TestCases/Drop_MergeTree/yandex/clickhouse-server:21.3
2024/08/08 17:41:00 🐳 Creating container for image yandex/clickhouse-server:21.3
2024/08/08 17:41:00 ✅ Container created: b63a4d71a512
2024/08/08 17:41:00 🐳 Starting container: b63a4d71a512
2024/08/08 17:41:00 ✅ Container started: b63a4d71a512
2024/08/08 17:41:00 ⏳ Waiting for container id b63a4d71a512 image: yandex/clickhouse-server:21.3. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0x140000e0000]}
2024/08/08 17:41:03 🔔 Container is ready: b63a4d71a512
=== RUN   TestCases/WithInstanceDefaultConfigValues
=== RUN   TestCases/WithInstanceDefaultConfigValues/yandex/clickhouse-server:21.3
2024/08/08 17:41:03 🐳 Creating container for image yandex/clickhouse-server:21.3
2024/08/08 17:41:03 ✅ Container created: a15f2f669c25
2024/08/08 17:41:03 🐳 Starting container: a15f2f669c25
2024/08/08 17:41:04 ✅ Container started: a15f2f669c25
2024/08/08 17:41:04 ⏳ Waiting for container id a15f2f669c25 image: yandex/clickhouse-server:21.3. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0x140002ee120]}
2024/08/08 17:41:06 🔔 Container is ready: a15f2f669c25
=== RUN   TestCases/WithInstanceDefaultConfigValues/yandex/clickhouse-server:21.3/set_1_dirty
=== RUN   TestCases/WithInstanceDefaultConfigValues/yandex/clickhouse-server:21.3/re-set_1_dirty
=== RUN   TestCases/WithInstanceDefaultConfigValues/yandex/clickhouse-server:21.3/set_2_clean
=== RUN   TestCases/WithInstanceDefaultConfigValues/yandex/clickhouse-server:21.3/re-set_2_clean
=== RUN   TestCases/WithInstanceDefaultConfigValues/yandex/clickhouse-server:21.3/last_migration_dirty
=== RUN   TestCases/WithInstanceDefaultConfigValues/yandex/clickhouse-server:21.3/last_migration_clean
--- PASS: TestCases (29.75s)
    --- PASS: TestCases/Test_TinyLog (3.85s)
        --- PASS: TestCases/Test_TinyLog/yandex/clickhouse-server:21.3 (3.85s)
            --- PASS: TestCases/Test_TinyLog/yandex/clickhouse-server:21.3/set_1_dirty (0.02s)
            --- PASS: TestCases/Test_TinyLog/yandex/clickhouse-server:21.3/re-set_1_dirty (0.02s)
            --- PASS: TestCases/Test_TinyLog/yandex/clickhouse-server:21.3/set_2_clean (0.02s)
            --- PASS: TestCases/Test_TinyLog/yandex/clickhouse-server:21.3/re-set_2_clean (0.03s)
            --- PASS: TestCases/Test_TinyLog/yandex/clickhouse-server:21.3/last_migration_dirty (0.02s)
            --- PASS: TestCases/Test_TinyLog/yandex/clickhouse-server:21.3/last_migration_clean (0.02s)
    --- PASS: TestCases/Migrate_TinyLog (3.32s)
        --- PASS: TestCases/Migrate_TinyLog/yandex/clickhouse-server:21.3 (3.32s)
    --- PASS: TestCases/Version_TinyLog (3.11s)
        --- PASS: TestCases/Version_TinyLog/yandex/clickhouse-server:21.3 (3.11s)
    --- PASS: TestCases/Drop_TinyLog (3.16s)
        --- PASS: TestCases/Drop_TinyLog/yandex/clickhouse-server:21.3 (3.16s)
    --- PASS: TestCases/Test_MergeTree (3.30s)
        --- PASS: TestCases/Test_MergeTree/yandex/clickhouse-server:21.3 (3.30s)
            --- PASS: TestCases/Test_MergeTree/yandex/clickhouse-server:21.3/set_1_dirty (0.02s)
            --- PASS: TestCases/Test_MergeTree/yandex/clickhouse-server:21.3/re-set_1_dirty (0.02s)
            --- PASS: TestCases/Test_MergeTree/yandex/clickhouse-server:21.3/set_2_clean (0.02s)
            --- PASS: TestCases/Test_MergeTree/yandex/clickhouse-server:21.3/re-set_2_clean (0.02s)
            --- PASS: TestCases/Test_MergeTree/yandex/clickhouse-server:21.3/last_migration_dirty (0.02s)
            --- PASS: TestCases/Test_MergeTree/yandex/clickhouse-server:21.3/last_migration_clean (0.02s)
    --- PASS: TestCases/Migrate_MergeTree (3.37s)
        --- PASS: TestCases/Migrate_MergeTree/yandex/clickhouse-server:21.3 (3.37s)
    --- PASS: TestCases/Version_MergeTree (3.13s)
        --- PASS: TestCases/Version_MergeTree/yandex/clickhouse-server:21.3 (3.13s)
    --- PASS: TestCases/Drop_MergeTree (3.22s)
        --- PASS: TestCases/Drop_MergeTree/yandex/clickhouse-server:21.3 (3.22s)
    --- PASS: TestCases/WithInstanceDefaultConfigValues (3.29s)
        --- PASS: TestCases/WithInstanceDefaultConfigValues/yandex/clickhouse-server:21.3 (3.29s)
            --- PASS: TestCases/WithInstanceDefaultConfigValues/yandex/clickhouse-server:21.3/set_1_dirty (0.02s)
            --- PASS: TestCases/WithInstanceDefaultConfigValues/yandex/clickhouse-server:21.3/re-set_1_dirty (0.02s)
            --- PASS: TestCases/WithInstanceDefaultConfigValues/yandex/clickhouse-server:21.3/set_2_clean (0.02s)
            --- PASS: TestCases/WithInstanceDefaultConfigValues/yandex/clickhouse-server:21.3/re-set_2_clean (0.02s)
            --- PASS: TestCases/WithInstanceDefaultConfigValues/yandex/clickhouse-server:21.3/last_migration_dirty (0.02s)
            --- PASS: TestCases/WithInstanceDefaultConfigValues/yandex/clickhouse-server:21.3/last_migration_clean (0.02s)
PASS
ok      github.com/golang-migrate/migrate/v4/database/clickhouse        30.767s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@mdelapenya