Skip to content

fix(library): vendor the sunsingerus mergo fork so the operator imports as a Go module - #2160

Merged
alex-zaitsev merged 4 commits into
Altinity:0.28.0from
stackgenhq:vendor-mergo-for-library-consumption
Nov 19, 2026
Merged

alex-zaitsev merged 4 commits into
Altinity:0.28.0from
stackgenhq:vendor-mergo-for-library-consumption

Conversation

@sks

@sks sks commented Nov 16, 2026

Copy link
Copy Markdown

What

Upstream wires mergo.WithSliceDeepMerge — used by the exported API packages (type_chi.go, type_templates.go, type_configuration_chop.go, type_chk.go) — through a replace directive:

replace github.com/imdario/mergo => github.com/sunsingerus/mergo v0.0.0-20230507185449-fc6fffa94450

Go replace directives do not propagate to consumers importing this repository as a library module. Any downstream require of clickhouse-operator fails at compile with:

undefined: mergo.WithSliceDeepMerge

(Raised as #2150, closed without a fix. (Supersedes the just-closed #2158 — same commits, moved to the org fork.))

The fix

Vendor the sunsingerus fork's stdlib-only sources at pkg/util/mergo (BSD-3-Clause, LICENSE included, provenance pinned to fc6fffa94450 in provenance.go) and swap the four API packages' imports to it. The non-propagating replace and the imdario/mergo require are gone from go.mod.

Net effect

A clean module consumer builds the exported API packages with zero consumer-side replaces:

require github.com/altinity/clickhouse-operator v0.27.4

Verified with a scratch consumer module (require, import pkg/apis/clickhouse.altinity.com/v1, exercise the mergo.WithSliceDeepMerge path) before and after: fails on stock upstream, passes with this patch.

No behavior change — the vendored sources are byte-identical to the fork at the pinned commit.

Why we care

stackgenhq/chinit — the cloud-adaptive node bootstrapper for stateful ClickHouse fleets on VMs (EC2 ASGs, Azure VMSS) — consumes the operator's config generation, keeper/chk machinery, and API types as a plain Go library. Today that requires carrying a fork of this repository solely for this vendoring. Landing this patch would let us drop the fork and require upstream directly, and we believe it unblocks library consumption for anyone else embedding the operator's model/config layer outside Kubernetes.

Follow-up we would like to explore (separate PRs)

  • Splitting the pkg/chop config accessors (Config(), the Access/Username reads) from the k8s.io/client-go client wiring — today importing pkg/chop for config drags the whole k8s client surface into a non-k8s consumer's tree.

Happy to adjust (path, naming, docs) to fit your conventions.

…ts as a Go module

Upstream wires mergo.WithSliceDeepMerge (used by the exported API
packages - type_chi.go, type_templates.go, type_configuration_chop.go,
type_chk.go) through a replace directive:

    replace github.com/imdario/mergo => github.com/sunsingerus/mergo

Go replace directives do not propagate to consumers importing this
repository as a library module. Any downstream require of
clickhouse-operator fails at compile with:

    undefined: mergo.WithSliceDeepMerge

(Raised as Altinity#2150, closed without a fix.)

This vendors the sunsingerus fork's stdlib-only sources at
pkg/util/mergo (BSD-3-Clause, LICENSE included, provenance pinned to
fc6fffa94450) and swaps the four API packages' imports to it. The
non-propagating replace and the imdario require are gone from go.mod.

Net effect: a clean module consumer builds the exported API packages
with zero consumer-side replaces:

    require github.com/altinity/clickhouse-operator v0.27.4

No behavior change - the vendored sources are byte-identical to the
fork at the pinned commit (provenance.go documents the pin; do not
modify, re-vendor wholesale).

Context: chinit (the cloud-adaptive node bootstrapper for stateful
ClickHouse fleets on VMs) consumes the operator's config generation,
keeper machinery and API types as a library. The k8s-client wiring
stays upstream; the mergo vendoring is what makes that possible
without a fork.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@alex-zaitsev
alex-zaitsev self-requested a review November 18, 2026 09:08
@alex-zaitsev

Copy link
Copy Markdown
Member

@sks , it does not compile after rebase to 0.28.0. Please fix or apply a patch:

diff --git a/pkg/apis/clickhouse.altinity.com/v1/merge.go b/pkg/apis/clickhouse.altinity.com/v1/merge.go
--- a/pkg/apis/clickhouse.altinity.com/v1/merge.go
+++ b/pkg/apis/clickhouse.altinity.com/v1/merge.go
@@ -24,7 +24,7 @@ import (
 	"strconv"
 	"strings"
 
-	"github.com/imdario/mergo"
+	"github.com/altinity/clickhouse-operator/pkg/util/mergo"
 	core "k8s.io/api/core/v1"
 	utiljson "k8s.io/apimachinery/pkg/util/json"
 	"k8s.io/apimachinery/pkg/util/strategicpatch"
diff --git a/pkg/apis/clickhouse.altinity.com/v1/type_templates.go b/pkg/apis/clickhouse.altinity.com/v1/type_templates.go
--- a/pkg/apis/clickhouse.altinity.com/v1/type_templates.go
+++ b/pkg/apis/clickhouse.altinity.com/v1/type_templates.go
@@ -15,7 +15,6 @@
 package v1
 
 import (
-	"github.com/altinity/clickhouse-operator/pkg/util/mergo"
 	log "github.com/golang/glog"
 
 	core "k8s.io/api/core/v1"

sks added 2 commits November 18, 2026 06:43
The merge of 0.28.0 brought in pkg/apis/clickhouse.altinity.com/v1/merge.go
importing github.com/imdario/mergo directly, which broke the build once the
replace was dropped. Switch it to the vendored package (fixes the MergoOptions
type mismatch in type_chi.go) and drop the now-unused import in
type_templates.go.
sks added a commit to stackgenhq/chinit that referenced this pull request Nov 18, 2026
… (backupconfig, live version, SecretGetter) (#15)

## What

Adopts three `clickhouse-operator` library surfaces that chinit was
hand-rolling, on top of a dependency bump to the operator 0.28.0 fork
tag — plus a Docker Compose acceptance test wired into CI.

### Dependency bump: `v0.27.4` → `v0.28.0` (`v0.28.0-stackgen.1`)

The fork tag carries the mergo vendoring (PR
Altinity/clickhouse-operator#2160) that makes the operator importable as
a library. 0.28.0 itself brings the template-precedence overhaul, the
digest-pinned image version fix (#2151), URL-encoding of credentials
with reserved characters (#2123), and the no-default-password auth fix
(#2156) — the last two land in chinit's connection and user paths for
free.

### 1. `backupconfig.Build` for clickhouse-backup's config.yml — **fixes
a storage-path bug**

`configureBackup` hand-formatted the config and wrote `s3.prefix` — but
clickhouse-backup's config struct has **no `prefix` field**: it reads
`s3.path` (verified against [pkg/config/config.go at
v2.5.0](https://github.com/Altinity/clickhouse-backup/blob/v2.5.0/pkg/config/config.go),
the version pinned in the golden AMI). go-yaml silently drops unknown
keys, so **backups were landing at the bucket root instead of under
`<prefix>/<cluster>/`**.

The adoption builds the config through the operator's
`pkg/model/chi/backupconfig.Build` (new in 0.28.0, stdlib-only — the
same builder the operator uses for its sidecar config Secrets):
- `s3.path` now carries the prefix (the bug fix)
- `object_disk_path` defaults beside the backups
(`object_disks/<prefix>/<cluster>`)
- the inline admin password (the 2.5.0 `password_file` compatibility
pin) and every `general` setting are byte-identical in effect

### 2. Live version for `ClusterSchemer` — removes a hardcoded `24.3`

`SchemaClientFactory.NewClient` hardcoded
`swversion.NewSoftWareVersion("24.3")` against 26.8 fleets. The schemer
gates SQL shapes on that version (`sqlCreateDatabaseDistributed`
switches `engine_full` on `>= 22.12`); a stale value takes the wrong
branch silently. `detectSoftwareVersion` mirrors the operator's own
app-based detection (`getHostSoftwareVersion` → `HostClickHouseVersion`
→ `SELECT version()`): one query per inventory host until one answers,
bounded by 5s, falling back to the operator's `MinVersion()` when
nothing answers — exactly the operator's "unable to acquire" behavior.

### 3. `SecretGetter` seam for the normalizer — fixes a latent crash

The operator's normalizer resolves
`security.clickhouse.tls.rootCASecretRef` **at load time** through
`subst.SecretGetter`. chinit passed `nil` — a spec carrying the ref
**panics** the loader (nil func call in `FetchSecretFieldValue`), not a
clean rejection.

`LoadCHIRequest` now takes a `SecretSource`; `secretGetter` bridges it
onto the operator seam:
- JSON secrets map every field to a `Data` key (the operator's key
lookup finds `ca.crt` etc.)
- single-value secrets expose the value under the secret's own name
- the namespace is accepted and dropped (cloud secret stores are not
namespaced)
- nil source keeps the historical behavior — specs without refs are
unaffected

The boot and remote-servers loaders pass the provider's secret store
(the same source the users renderer resolves `passwordFrom` with).

### 4. E2E acceptance test (`acceptance_test/e2e.go`) — write on writer,
read on reader

A one-command gate for the whole stack, run by CI on every PR:

```bash
go run ./acceptance_test/e2e.go
```

The harness builds the `chinit-node` image from the **current
checkout**, boots the compose cluster on fresh volumes, and verifies:

1. Keeper answers `ruok → imok`
2. `chinit-boot` finished on writer-0 and reader-0 (marker + live SQL
ping)
3. The `compose-demo` cluster lists both replicas on both nodes
4. **A row written on writer-0 arrives on reader-0**
(ReplicatedMergeTree replication — the core promise), with a per-run
marker so a reused cluster can't pass on stale data
5. The `events` replica is writable with all replicas active

Flags: `-keep`, `-reuse`, `-timeout`, `-cluster`, `-repo`. A failing
phase prints `compose ps` and the affected nodes' journals so CI output
is self-diagnosing. New CI job: `e2e (docker compose)`.

**Verified locally on Docker Desktop** against this branch: three
consecutive green runs (full rebuild+boot+write→read, `-keep`, `-reuse`)
with distinct markers each.

## What is deliberately NOT adopted

- **User password resolution via the normalizer**: the operator's
`valueFrom.secretKeyRef` path wires **pod ENV vars**
(`AppendAdditionalEnvVar`) — meaningless off-cluster. chinit's
render-time `passwordFrom` resolution stays.
- **Keeper mntr parsing**: the operator has no 4lw client (its
raft-safety code marks mntr an extension point); chinit is ahead here.

## Behavior preserved

- Full unit suite green: 13/13 packages, `-count=1`, conformance
included.
- `configureBackup` config semantics: same admin/inline-password shape,
same retention/bucket/region/compression wiring, timer install
untouched.
- Clusterops' loader keeps the nil-source default (it has no provider).
- CHK loading is untouched (no secret seam in the keeper normalizer).

## Notes

- `k8s.io/api` moves to a direct require — the specloader builds a
`core.Secret` for the seam.
- Fork tag `v0.28.0-stackgen.1` points at the mergo-vendored merge of
upstream 0.28.0 head (`fb1294eb8`).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@alex-zaitsev
alex-zaitsev merged commit 00df671 into Altinity:0.28.0 Nov 19, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants