Releases · Altinity/clickhouse-operator

@miguel-signoz

Added

Add configurable MaxConcurrentReconciles for CHK controller by @miguel-signoz in #2033. Default is 1. Closes #2032

reconcile:
  runtime:
    reconcileCHKsThreadsNumber: 1

Auto-heal Completed CHIs with sustained-NotReady hosts by @dashashutosh80 in #1998. Closes #1994. Tuned off by default:

reconcile:
  recovery:
    onStatus:
      # Recovery for a CHI whose .status.status is Aborted (reconcile did not complete)
      # when one of its host pods transitions to Ready — auto-resumes the reconcile.
      aborted:
      completed:
        # Action when a Completed CHI's pod flips Ready=True -> Ready=False and
        # stays NotReady for at least onPodNotReadyThreshold:
        #   none (default) — do nothing
        #   retry          — re-enqueue the CHI so the stuck host is force-restarted
        # OFF by default: force-recreating a Completed CHI's pod is destructive — it can
        # interrupt a replica's in-progress recovery and means hard downtime for a
        # single-replica shard. Opt in with `retry` only where that trade-off is acceptable.
        onPodNotReady: none
        # Minimum duration a pod must stay Ready=False before recovery fires, once enabled
        # (Go duration string; default 5m). Raise it for slow-recovering replicas.
        onPodNotReadyThreshold: 5m

Configure operator rootCA with a secret. Closes #1925

 clickhouse:
    access:
      scheme: https
      port: 8443
      rootCASecretRef:
        name: root-ca-secret
        key: ca.crt

Helm: Add podSecurityContext and podAnnotations to crdHook Job by @qlevasseur-genetec in #2009

Changed

Split Keeper services to internal and external ones. Closes #1982
Apply filter earlier when scraping metrics in order reduce memory usage by @dentiny in #2028. Closes #2019
Backward-incompatible config rename in ClickHouseOperatorConfiguration: reconcile.recovery.from.{aborted,completed} → reconcile.recovery.onStatus.{aborted,completed} (the from grouping level is removed; per-status action keys are unchanged, and the obsolete from key is silently ignored on load). If you set reconcile.recovery.from.aborted.onPodReady: none on 0.27.0/0.27.1 to disable Aborted auto-recovery, re-apply it as reconcile.recovery.onStatus.aborted.onPodReady: none — otherwise the default (retry) silently re-enables it on upgrade.

Security

Operator metrics port no longer exposes /debug/pprof. The operator's metrics HTTP listener (default :9999) served Go's shared http.DefaultServeMux, onto which net/http/pprof is transitively registered through a controller-runtime dependency — unintentionally exposing the /debug/pprof/* endpoints (including the CPU profile and trace handlers) to anything able to reach the operator's metrics Service, enabling unauthenticated profiling/DoS and heap/goroutine dumps. The listener now uses a dedicated mux that serves only the metrics endpoint; /metrics is unchanged.
Bumped versions of dependent libraries to address CVEs

Fixed

Add replica restart fix for operator when remote_servers configmap is propagated by @oo007 in #2024. Closes #2013
Escape reserved characters in element text by @AruneshDwivedi in #2034
Helm: Wire watchNamespaces value into operator ConfigMap by @jtomaszon in #2007. Closes #1919
Fixed incorrect watched namespace override #2008

FIPS Harness

Align FIPS test plan, requirements, and e2e FIPS coverage, fixed failing operator tests by @Elmo33 in #2011
Wire security.kubernetes.tls.minVersion into the Kubernetes API client transport by @Elmo33 in #2020
Expand FIPS TLS e2e coverage with host-run cipher probes and broader listener rejection checks by @Elmo33 in #2031

New Contributors

@oo007 made their first contribution in #2024
@jtomaszon made their first contribution in #2007
@dentiny made their first contribution in #2028
@miguel-signoz made their first contribution in #2033
@AruneshDwivedi made their first contribution in #2034

Full Changelog: release-0.27.1...release-0.27.2

@pkieszcz

NOTE: This is a major update to 0.27.0. Starting from 0.27.1 clickhouse-operator is FIPS-enabled

Added

secure/insecure flags to CHK in order to control ports by a single switch
FIPS-140 security hardening controls in operator configuration: https://github.com/Altinity/clickhouse-operator/blob/0.27.1/docs/security_hardening_fips.md
FIPS-140 reference configuration: https://github.com/Altinity/clickhouse-operator/blob/0.27.1/docs/fips_setup.md

Fixed

Fix permanent ArgoCD diff on resourceFieldRef.divisor by @pkieszcz in #1989
Fix inconsistent secret rendering in autogenerated remote_servers by @realyota in #1992. Closes #1991
Fixed skipped Delete when scale-to-0 Update fails by @dashashutosh80 in #1993. Closes #1990
Fixed .status.usedTemplates that were not properly cleaned when templates were removed
Bump dependent libraries to address CVEs

New Contributors

@pkieszcz made their first contribution in #1989

Full Changelog: release-0.27.0...release-0.27.1

@janeklb

Added

Reference CHK from CHI by a name rather than a service:

spec:
  configuration:
    zookeeper:
      keeper:
        name: my-keeper

[EXPERIMENTAL] Pre/post hooks. Allows to inject different SQL commands on reconcile events.
- supported events: Any, HostCreate, HostUpdate, HostStart, HostStop, HostConfigRestart, HostRollout, HostShutdown, HostDelete
- target controls where to run SQL: FirstHost (default), AllHosts, AllShards
- failurePolicy specifies what to do if hook execution fails with an error: Fail (default) or Ignore.
  See CRD for detailed description. Example:

  reconcile:
    host:
      hooks:
        pre:
          - sql:
              queries:
                - "SYSTEM STOP REPLICATION QUEUES"
            events:
              - HostShutdown

Now operator automatically restarts aborted reconcile if failed pod goes online after abort
Restart operator on operator configuration change. #1960. Closes #1930
Allow to exclude certain metrics from exporting. Noisy OS/CPU metrics are now excluded by default. #1975. Closes #1876
Exclusion rules are applied by default and can be changed in operator configuration:

    excludeMetricsRegexp:
      - "^metric\\.(OS.*CPU[0-9]+|CPUFrequencyMHz_[0-9]+)$"

Helm: Allow readiness and liveness probes for operator containers by @janeklb in #1976
Helm: Add security context for crdHook to enable security policy compliance by @qlevasseur-genetec in #1950

Changed

Enabled async_replication and use_xid_64 in Keeper default configuration. Requires Keeper 25.3 or above.
Whitelist Keeper four letter commands by default
Switched Keeper probes to ruok
Kubernetes client library has been upgraded to 0.30.14 by @dcoppa in #1970

Fixed

Fix watch ClickHouseOperatorConfiguration in operator namespace. #1959.
Fix chk probe crashloop by @hananbs in #1961. Closes #1962
Fix secret-backed env upgrade rollout. #1966. Closes #1963
Helm: fix configmap too long by @madrisan in #1949. Closes #1911
build(deps): bump go.opentelemetry.io/otel/sdk from 1.42.0 to 1.43.0 by @dependabot[bot] in #1953
Fix: format string issues by @destinyoooo in #1955
Fix: Prevent CrashLoopBackOff during image upgrade with RollingUpdate (#1926) by @dashashutosh80 in #1956. Closes #1926
Fixed a bug when host has not been removed from status fields hostsWithReplicaCaughtUp and hostsWithTablesCreated when removing from a cluster.
Prevent nil pointer panic in poll when CR Get fails by @wucm667 in #1974. Closes #1972
Scope schema discovery to target host's cluster in multi-cluster CHI by @lukas-pfannschmidt-tr in #1965. Closes #1964
Fix: use plain errors for clickhouse model messages in #1980 by @immanuwell

New Contributors

@qlevasseur-genetec made their first contribution in #1950
@destinyoooo made their first contribution in #1955
@hananbs made their first contribution in #1961
@wucm667 made their first contribution in in #1974
@immanuwell made their first contribution in #1980

Full Changelog: release-0.26.3...release-0.27.0

@dashashutosh80

Fixed

Prevent CrashLoopBackOff during image upgrade with RollingUpdate #1956 by @dashashutosh80. Closes #1926
Address CVEs in dependent libraries

Full Changelog: release-0.26.2...release-0.26.3

Fixed

Fixed a race condition when updating ClickHouse configuration and version altogether with a configuration setting that did not exist in the old version. Closes #1926
Fixed a bug when some Keeper nodes could be left offline after configuration changes
Fixed a bug when operator did not respect watched namespaces for Keeper. Closes #1923
Fixed potential races in configuration hash calculation. May close #1907
Updated dependent libraries to address CVE-2026-24051

Changed

Deleting multi-node CHI and CHK is now much faster
Added asynchronous_metrics_keeper_metrics_only to default Keeper configuration

Full Changelog: release-0.26.1...release-0.26.2]

@Slach

Fixed

Fixed Keeper startup that was slow due to missing quorum. Closes #1931 and #1856
Fixed Keeper deletion logic that could previously leave PVCs undeleted
Fixed hostName generation in status that might result in excessive schema propagation cycles
Fixed FQDN normalization to prevent trailing-dot inconsistencies between internal hostname representations
Bump stdlib version to address CVE
Bump base image version by @Slach in #1941. Closes #1940
Document custom service template behaviour for CHK by @realyota in #1939

Full Changelog: release-0.26.0...release-0.26.1

@lesandie

IMPORTANT: Due to ClickHouse upstream regression ClickHouse/ClickHouse#89693 DDL queries may not work on newly created ClickHouse pods. It affects Kubernetes deployments only in some new ClickHouse versions (25.8.10+ and above). The workaround is to restart ClickHouse pods. The problem is fixed by ClickHouse/ClickHouse#92339, see backports for different release branches. The fix is backported to Altinity Stable 25.8.16.10001 as well.
Closes #1883 and #1913

Added

Added an option to abort reconcile if STS needs to be recreated. It can be configured in operator configuration or CHI.

# Reconcile StatefulSet scenario
reconcile:
  statefulSet:
    recreate:
      # What to do in case operator is in need to recreate StatefulSet?
      # Possible options:
      # 1. abort - abort the process, do nothing with the problematic StatefulSet, leave it as it is,
      #    do not try to fix or delete or update it, just abort reconcile cycle.
      #    Do not proceed to the next StatefulSet(s) and wait for an admin to assist.
      # 2. recreate - proceed and recreate StatefulSet.
      # Triggered when StatefulSet update fails or StatefulSet is not ready
      onUpdateFailure: recreate

Added an option to configure system tables for metrics scrapping. The default is system.metrics and system.custom_metrics tables, but those can be changed with a regular expression if needed:

    tablesRegexp: "^(metrics|custom_metrics)$"

Changed

The suspend flag now immediately aborts a running reconcile. Previously, it did not affect the one that was running
When suspend flag is set, any reconcile attempt automatically sets CHI/CHK status to aborted.
Add optional registry prefix for operator and metrics images in Helm chart by @lesandie in #1928
Improve ClickHouse Keeper Grafana Dashboard by @discostur in #1872
Add CRDHook annotations by @eyyu in #1914
Hotfix crdhook, add imagePullSecrets by @Slach in #1917
Fix installer to default template URL to OPERATOR_VERSION by @realyota in #1910
sort keys in Settings.Keys() method for consistent order (fix manifest reconcile issue) by @mastercactapus in #1900
Multiple documentation fixes

Fixed

Fixed Keeper rolling update logic. Closes #1796 #1915
Fixed a bug when replica was not added to monitoring until it catches up the replication lag
Fixed version parsing for FIPS compatible builds of ClickHouse. Closes #1850
Fixed stop and suspend attributes for CHK that were previously ignored
Fix distributed_ddl.replicas_path mismatch that could prevent sharing (Zoo)Keeper between multiple clusters @Elmo33 in #1922
Fixed a bug when defaults.storageManagement.reclaimPolicy was not respected
Fixed slow initial connectivity to newly created pods caused by DNS search list exhaustion (ndots:5). Added trailing dot to FQDN and increased connect timeout
Fixed a bug where reconcile settings specified at CHI level (e.g. spec.reconcile.statefulSet.recreate.onUpdateFailure) were not inherited by cluster-level reconcile configuration

Other

stdlib has been upgraded to 1.25.6 to address CVEs
Operator has been certified for 25.8.16.10001 Altinity.Stable.

New Contributors

@rajdudhare1 made their first contribution in #1890
@siggy made their first contribution in #1906
@eyyu made their first contribution in #1914
@Elmo33 made their first contribution in #1922
@mastercactapus made their first contribution in #1900
@lesandie made their first contribution in #1928

Full Changelog: release-0.25.6...release-0.26.0

Changed

Hosts are not excluded from remote_servers anymore if restart is not needed. Previously, replicas in replicated cluster might be removed for a short time even if restart was not needed.
configuration.zookeeper section changes do not require restart anymore
Last reconciliation error and list of errors are now stored in CHI status

Fixed

actionPlan is now optional in status. That fixes operator upgrade problems that might happen in some environments.
Fixed excessive reconciles triggered by endpoint slices. Closes #1873
Fixed crash in CHK that might happen sometimes. Closes #1863
Fixed an issue with handling fractional requests/limits that would result in excessive reconcile. Closes #1849 #1821
Fixed a bug with operator crash in Terminating namespace. Closes #1871
stdlib was upgraded in order to address CVEs in dependent libraries

NOTE: Due to regression in upstream ClickHouse ClickHouse/ClickHouse#89693 schema propagation and DDL statements do not work with ClickHouse versions 25.8.10+ and newer until it is resolved.

Full Changelog: release-0.25.5...release-0.25.6

@janeklb

Added

The latest applied ActionPlan is now stored in chi-storage ConfigMap
volumeClaimTemplate.volumeAttributeClass attribute. Closes #1818
Configuration for DROP REPLICA behavior:

reconcile:
   host:
    drop:
      replicas:
        # Whether the operator during reconcile procedure should drop replicas when replica is deleted
        onDelete: yes
        # Whether the operator during reconcile procedure should drop replicas when replica volume is lost
        onLostVolume: yes
        # Whether the operator during reconcile procedure should drop active replicas when replica is deleted or recreated
        active: no

Now active replicas are never dropped. That solves a potential bug when replica could be dropped on a multi-volume node if newly added volume is not yet available.

Changed

Enabled ReadinessProbe for Keeper. Closes #1846
Enabled query_log for all DDL statements performed by operator
minor logging improvements by @janeklb in #1829

Fixed

Removed excessive logging isUpdatedEndpointSlice():unknown
Changed metrics collection query that could be broken in 0.25.4 for old ClickHouse versions

Helm updates

fix: correctly delete existing helm release artifacts by appending asset ID to URL by @Slach in #1827
Updated Helm repo location to https://helm.altinity.com by @DougTidwell in #1835
Helm: add metrics.enabled conditional checks for serviceMonitor by @KJone1 in #1834
Hotfix helm schema by @Slach in #1842

New Contributors

@KJone1 made their first contribution in #1834
@fkywong made their first contribution in #1837

Full Changelog: release-0.25.4...release-0.25.5

@AdheipSingh

Added

Operator configuration 'reconcile' section is now fully supported at CHI level under both 'reconcile' and old 'reconciling' name. Previously, only selected settings were available at CHI level.
Allow to exclude namespaces that operator watches. by @AdheipSingh in #1770

spec:
  watch:
    namespaces:
      include: []
      exclude: [] # new

Option to choose which probe should operator wait for during reconcile. Previously, it always waited for pod to be ready. This can now be configured in 'reconcile' section of operator or CHI:

spec:
  reconcile:
    host:
      wait:
        probes:
          startup: "yes"
          readiness: "no"

ZooKeeper compression support by @wilkermichael in #1809
Support watching multiple namespaces in OLM by @saeedhosseini21 in #1825. Closes #1491

Changed

system.custom_metrics table is currently scrapped for monitoring in addition to metrics and asynchronous_metrics. That allows to inject custom monitoring data from ClickHouse side.
Deprecated Endpoints API has been replaced with EndpointSlice. Closes 1801

Fixed

Fixed a bug with long environment variables used for secrets being truncated. Closes #1804
Fixed a bug that operator did not respect watched namespaces for CHK

Helm updates

Define values.schema.json by @Slach in #1815. Closes #1814
Added clickhouse-operator deployment strategy parameters to Helm chart by @Slach in #1789
Publish operator helm chart to helm.altinity.com in addition to artifacthub.io

Full Changelog: release-0.25.3...release-0.25.4

Uh oh!

Releases: Altinity/clickhouse-operator

Release list

release-0.27.2

Added

Changed

Security

Fixed

FIPS Harness

New Contributors

Contributors

Uh oh!

release-0.27.1

Added

Fixed

New Contributors

Contributors

Uh oh!

release-0.27.0

Added

Changed

Fixed

New Contributors

Contributors

Uh oh!

release-0.26.3

Fixed

Contributors

Uh oh!

release-0.26.2

Fixed

Changed

Uh oh!

release-0.26.1

Fixed

Contributors

Uh oh!

release-0.26.0

Added

Changed

Fixed

Other

New Contributors

Contributors

Uh oh!

release-0.25.6

Changed

Fixed

Uh oh!

release-0.25.5

Added

Changed

Fixed

Helm updates

New Contributors

Contributors

Uh oh!

release-0.25.4

Added

Changed

Fixed

Helm updates

Contributors

Uh oh!