[etcd-druid]

⚠️ Breaking Changes

• [OPERATOR] Claim logic for PodDisruptionBudgets have been removed. Already existing PodDisruptionBudget objects cannot be adopted anymore (gardener/etcd-druid#430, @aaronfern)
• [OPERATOR] Logic for deleting any leftover CronJobs created by etcd-druid:v0.6.0 has been removed. Please deploy a lower version of etcd-druid before upgrading if you still have any leftover CronJobs or manually delete them (gardener/etcd-druid#430, @aaronfern)

✨ New Features

• [OPERATOR] Applying the annotation druid.gardener.cloud/ignore-reconciliation on the ETCD CR will stop etcd-druid from reconciling it. (gardener/etcd-druid#446, @abdasgupta)
  • This is helpful for operators to apply any manual fixes to the ETCD components, such as manually fixing permanent quorum loss as per the playbook.

🐛 Bug Fixes

• [OPERATOR] Updated condition used to decide when the gardener.cloud/scaled-to-multi-node annotation is added (gardener/etcd-druid#455, @aaronfern)
• [OPERATOR] A bug has been fixed that caused the wrong minAvailable configuration being calculated for multi-node etcd PodDisruptionBudget. (gardener/etcd-druid#441, @timuthy)

🏃 Others

• [OPERATOR] Enables etcd-druid to trigger restarts of the etcd pods when peer URL is TLS enabled. (gardener/etcd-druid#421, @unmarshall)
• [OPERATOR] Etcd druid will now use policy/v1 for PodDisruptionBudgets for kubernetes >= 1.21. However, for kubernetes < 1.21, PodDisruptionBudgets will still default to policy/v1beta1 (gardener/etcd-druid#430, @aaronfern)
• [OPERATOR] Golang version used upgraded to 1.18.6 (gardener/etcd-druid#431, @aaronfern)
• [OPERATOR] etcd-custom-image upgraded to v3.4.13-bootstrap-8 (gardener/etcd-druid#432, @aaronfern)
• [OPERATOR] Added new field to the etcd CRD etcd.Spec.Etcd.clientService (gardener/etcd-druid#438, @aaronfern)
  • Users can now add annotations to the client service by specifying them in the above field of the etcd resource
• [OPERATOR] The BackupReady condition is not considered anymore when the PodDisruptionBudget configuration is calculated. This earlier blocked rolling out fixes that potentially solved problems with backup procedures. (gardener/etcd-druid#441, @timuthy)
• [OPERATOR] Enhance BackupReady condition to take into account statefulset being scaled down and the backup section not being defined (gardener/etcd-druid#415, @aaronfern)

[etcd-backup-restore]

🐛 Bug Fixes

• [OPERATOR] To avoid potential race-condition between go-routines updated probeEtcd func() to use shorter timeout. (gardener/etcd-backup-restore#532, @ishan16696)

🏃 Others

• [OPERATOR] Always update member peer URL, changed the way scale-up of etcd cluster is identified. (gardener/etcd-backup-restore#534, @unmarshall)
• [OPERATOR] Decreases the likelihood of potential race condition between the go-routines while closing the snapshotter. (gardener/etcd-backup-restore#537, @ishan16696)
• [OPERATOR] [bug-fix] backup-restore does not return error when it fails to update PeerURL of member. (gardener/etcd-backup-restore#540, @aaronfern)

📰 Noteworthy

• [OPERATOR] Updated golang version used to build images to 1.18.6 (gardener/etcd-backup-restore#535, @aaronfern)
• [OPERATOR] Updated golang version to 1.18 (gardener/etcd-backup-restore#535, @aaronfern)

[etcd-custom-image]

🏃 Others

• [OPERATOR] Etcd-custom-image will now retry fetching etcd configuration in case of any error (gardener/etcd-custom-image#26, @aaronfern)

[etcd-backup-restore]

🐛 Bug Fixes

• [OPERATOR] A bug has been fixed that caused the wrong minAvailable configuration being calculated for multi-node etcd PodDisruptionBudget. (gardener/etcd-backup-restore#441, @timuthy)

🏃 Others

• [OPERATOR] The BackupReady condition is not considered anymore when the PodDisruptionBudget configuration is calculated. This earlier blocked rolling out fixes that potentially solved problems with backup procedures. (gardener/etcd-backup-restore#441, @timuthy)

[etcd-backup-restore]

🏃 Others

• [OPERATOR] Decreases the likelihood of potential race condition between the go-routines while closing the snapshotter. (gardener/etcd-backup-restore#537, @ishan16696)
• [OPERATOR] [bug-fix] backup-restore does not return error when it fails to update PeerURL of member. (gardener/etcd-backup-restore#540, @aaronfern)

[etcd-druid]

🏃 Others

• [OPERATOR] Enables etcd-druid to trigger restarts of the etcd pods when peer URL is TLS enabled. (gardener/etcd-druid#421, @unmarshall)
• [OPERATOR] Golang version used upgraded to 1.18.6 (gardener/etcd-druid#431, @aaronfern)
• [OPERATOR] etcd-custom-image upgraded to v3.4.13-bootstrap-8 (gardener/etcd-druid#432, @aaronfern)

[etcd-backup-restore]

🐛 Bug Fixes

• [OPERATOR] To avoid potential race-condition between go-routines updated probeEtcd func() to use shorter timeout. (gardener/etcd-backup-restore#532, @ishan16696)

🏃 Others

• [OPERATOR] Always update member peer URL, changed the way scale-up of etcd cluster is identified. (gardener/etcd-backup-restore#534, @unmarshall)

📰 Noteworthy

• [OPERATOR] Updated golang version used to build images to 1.18.6 (gardener/etcd-backup-restore#535, @aaronfern)
• [OPERATOR] Updated golang version to 1.18 (gardener/etcd-backup-restore#535, @aaronfern)

[etcd-custom-image]

🏃 Others

• [OPERATOR] Etcd-custom-image will now retry fetching etcd configuration in case of any error (gardener/etcd-custom-image#26, @aaronfern)

[etcd-druid]

✨ New Features

• [OPERATOR] A Helm chart for deploying Etcd-Druid is now available in charts/druid. (gardener/etcd-druid#296, @timuthy)
• [DEVELOPER] Developers can now run Druid e2e tests via make test-e2e. Please see docs/development/local-e2e-tests.md for detailed information. (gardener/etcd-druid#296, @timuthy)

🐛 Bug Fixes

• [USER] Fix statefulset volumeClaimTemplate StorageClassName value population if etcd storageClass is an empty string. (gardener/etcd-druid#400, @shreyas-s-rao)
• [OPERATOR] An issue has been fixed that caused Etcd-Druid to update immutable fields sts.spec.serviceName and sts.spec.podManagementPolicy for older etcd resources that had different values configured. These updates must only happen when a etcd cluster is scaled up for the first time (1 -> x) because (a) then these values are mandatory and (b) a disruption is accepted. (gardener/etcd-druid#408, @timuthy)
• [OPERATOR] An issue has been fixed that caused Etcd-Druid to not consider the hostPath configuration in the referenced backup secret etcd.spec.backup.store.secretRef. (gardener/etcd-druid#412, @timuthy)

🏃 Others

• [OPERATOR] Liveness and startup probes for etcd were removed. After activating them in the last release, we noticed that they cause more harm than good since the startup time for etcd clusters varies and isn't predicable. Killing the etcd container in such a case doesn't solve the situation and will rather end in an endless loop of restarts. This change will cause a restart of etcd clusters. (gardener/etcd-druid#423, @timuthy)
• [OPERATOR] The Golang version used to compile Etcd-Druid has been updated to go 1.18.5. (gardener/etcd-druid#410, @timuthy)
• [OPERATOR] Adds a document mentioning the metrics for multi-node etcd. (gardener/etcd-druid#414, @ishan16696)

[etcd-backup-restore]

🐛 Bug Fixes

• [OPERATOR] Fix the probeEtcd func() to probe the corresponding Etcd by getting its Endpoint Status rather than just Get a key. (gardener/etcd-backup-restore#523, @ishan16696)

🏃 Others

• [OPERATOR] Handles the bolt database panic in case of database found to be corrupt. (gardener/etcd-backup-restore#521, @ishan16696)
• [OPERATOR] Added new metrics for multi-node etcd: etcdbr_defragmentation_duration_seconds, etcdbr_restoration_duration_seconds , etcdbr_cluster_size , etcdbr_is_learner , etcdbr_is_learner_count_total , etcdbr_add_learner_duration_seconds , etcdbr_member_remove_duration_seconds , etcdbr_member_promote_duration_seconds . (gardener/etcd-backup-restore#522, @ishan16696)
• [OPERATOR] Adds an annotation to etcd lease which indicates if the peer url is TLS enabled. (gardener/etcd-backup-restore#530, @unmarshall)

📰 Noteworthy

• [OPERATOR] Base alpine image upgraded from 3.15.4 to 3.15.6 (gardener/etcd-backup-restore#520, @aaronfern)

[etcd-custom-image]

🏃 Others

• [OPERATOR] Base alpine image upgraded from 3.15.4 to 3.15.6. (gardener/etcd-custom-image#24, @aaronfern)

[etcd-druid]

🐛 Bug Fixes

• [OPERATOR] This PR fixes an issue which caused the sts.spec.podManagementPolicy not to be updated to Parallel if an existing etcd cluster is scaled-up from 1 -> x. This can cause an issue if the cluster is afterwards completely scaled-down (aka hibernation) and scaled-up again. (gardener/etcd-druid#406, @timuthy)

🏃 Others

• [OPERATOR] Liveness and startup probes for etcd were removed. After activating them, we noticed that they cause more harm than good since the startup time for etcd clusters varies and isn't predicable. Killing the etcd container in such a case doesn't solve the situation and will rather end in an endless loop of restarts. This change will cause a restart of etcd clusters. (gardener/etcd-druid#424, @aaronfern)

[etcd-druid]

🐛 Bug Fixes

• [USER] Temporarily fixes an issue where druid tries to set spec.ServiceName to PeerServiceName by default, although older single-node etcds would have this field set to ClientServiceName, and updation of statefulset spec.ServiceName field is forbidden. (gardener/etcd-druid#403, @shreyas-s-rao)

[etcd-druid]

🐛 Bug Fixes

• [USER] Temporarily fix issue where PodManagementPolicy was trying to be updated from OrderedReady to Parallel for older shoots (created using etcd-druid:v0.8.5 and before), but the statefulset forbids updates to this field. (gardener/etcd-druid#402, @shreyas-s-rao)

[etcd-druid]

🐛 Bug Fixes

• [USER] Fix statefulset volumeClaimTemplate StorageClassName value population if etcd storageClass is an empty string. (gardener/etcd-druid#401, @shreyas-s-rao)

[etcd-druid]

⚠️ Breaking Changes

• [OPERATOR] etcd Statefulsets are not claimed anymore based on labels. Instead, the statefulsets are fetched using Name and Namespace combination. Thus, etcd.spec.selector does not have an effect on statefulsets anymore. (gardener/etcd-druid#365, @abdasgupta)
• [OPERATOR] The entrypoint for etcd-druid in its container image has been modified. (gardener/etcd-druid#360, @dimityrmirchev)

✨ New Features

• [DEVELOPER] Add support for running envtest on M1 Macbooks. (gardener/etcd-druid#396, @timuthy)
• [DEVELOPER] Deploying the etcd StatefulSet through a Helm chart has been abandoned. A codified version (component concept) is now used for this purpose. (gardener/etcd-druid#365, @abdasgupta)

🐛 Bug Fixes

• [OPERATOR] A bug has been fixed that caused the etcd-backup-restore side-car to connect to the etcd cluster via the peer-service URL. The side-car is supposed to use the client-service instead since it a) exposes client port 2379 and b) redirects traffic only to members which are ready to service traffic. (gardener/etcd-druid#388, @timuthy)
• [OPERATOR] An issue has been fixed that caused the liveness and readiness probes of etcd to always succeed even though an error was reported. This prevented defective etcd pods from being restarted automatically and caused unready candidates being considered as ready to serve traffic via the etcd service. (gardener/etcd-druid#396, @timuthy)
• [OPERATOR] A startup probe has been added to etcd to allow 2 minutes of initialization time before checking for etcd liveness. (gardener/etcd-druid#396, @timuthy)

🏃 Others

• [OPERATOR] The definition of the etcd.status.ready field was defined more precisely due to changed semantics of multi-node etcd clusters. etcd.status.ready is true whenever all underlying etcd replicas are ready. Please note, that the implementation for this check was not changed. (gardener/etcd-druid#389, @timuthy)
• [OPERATOR] Fixed an issue in the release job needed to add the correct image version config/default/manager_image_patch.yaml. (gardener/etcd-druid#397, @aaronfern)
• [OPERATOR] Added pod permission in etcd_role that now enable etcd-backup-restore to get/list/watch pods (gardener/etcd-druid#372, @aaronfern)
• [OPERATOR] Etcd-Druid's Golang version has been update to 1.18.4.. (gardener/etcd-druid#375, @timuthy)
• [OPERATOR] The correct image version has been set in config/default/manager_image_patch.yaml to match the current release. (gardener/etcd-druid#377, @timuthy)

[etcd-backup-restore]

⚠️ Breaking Changes

• [OPERATOR] Dropping the feature of passing storage container credentials through ENV for the following storage provider: S3, Swift, OCS, ABS, OSS. Please switch to pass the storage container credentials through volume file mount. (gardener/etcd-backup-restore#493, @ishan16696)

🏃 Others

• [OPERATOR] A new flag --service-endpoints has been added to the etcdbrctl server command. These (Kubernetes) service URLs ensure that etcd-backup-restore only connects to etcd member which are ready to server traffic. Especially the MemberAdd and Init steps require this. (gardener/etcd-backup-restore#512, @timuthy)
• [OPERATOR] Fixed a bug where etcd calls related to multi node operation were used in single node operation (gardener/etcd-backup-restore#504, @aaronfern)
• [OPERATOR] Assigned the correct Peer address to the Etcd after it restores from backup-bucket. (gardener/etcd-backup-restore#505, @ishan16696)
• [OPERATOR] No attempt is made to update member Peer URL when trying to promote a member (gardener/etcd-backup-restore#506, @aaronfern)
• [OPERATOR] An issue has been fixed that caused the Backup-Restore component to connect to the wrong etcd cluster for initializing and member-add procedures. (gardener/etcd-backup-restore#510, @timuthy)
• [OPERATOR] A new flag --service-endpoints has been added to the etcdbrctl server command. These (Kubernetes) service URLs ensure that etcd-backup-restore only connects to etcd member which are ready to server traffic. Especially the MemberAdd and Init steps require this. (gardener/etcd-backup-restore#513, @timuthy)

📰 Noteworthy

• [USER] For multi-node etcd: Added a feature of single member etcd restoration in case of data/data-dir of etcd member found to be corrupted or invalid. (gardener/etcd-backup-restore#509, @ishan16696)