Skip to content

Storage provisioner: deploy as DaemonSet and support Node Affinity#22945

Open
medyagh wants to merge 5 commits into
kubernetes:masterfrom
medyagh:storage_prov2
Open

Storage provisioner: deploy as DaemonSet and support Node Affinity#22945
medyagh wants to merge 5 commits into
kubernetes:masterfrom
medyagh:storage_prov2

Conversation

@medyagh
Copy link
Copy Markdown
Member

@medyagh medyagh commented May 8, 2026

PR Description

Summary

This PR upgrades minikube’s built-in dynamic storage-provisioner (hostpath provisioner) to support multi-node clusters properly.

The Problem:

Previously, the storage-provisioner ran as a single control-plane Pod. In a multi-node cluster:

  1. If a PersistentVolume (PV) host directory was created on Node B, it was physically inaccessible to pods scheduled on Node A.
  2. Because the StorageClass immediately provisioned volumes and lacked topology knowledge, pods could be scheduled on Node A, attempt to mount Node B's directory, and crash due to missing data.

The Solution:

  1. DaemonSet Deployment: The storage-provisioner now runs as a leader-less DaemonSet on every node in the cluster.
  2. Delayed Volume Binding (WaitForFirstConsumer): The standard StorageClass is updated to use volumeBindingMode: WaitForFirstConsumer. This delays volume provisioning until the Kubernetes Scheduler has assigned the consuming Pod to a specific target node.
  3. Node Affinity Enforcement: The provisioner running on the target node detects the scheduling choice, provisions the folder locally on that node, and stamps the resulting PV with pv.Spec.NodeAffinity matching the node. This restricts the consuming pod (and all future pods using the PV) to only run on that specific node.
  4. Deterministic Identity: Upgraded to use deterministic identities k8s.io/minikube-hostpath-<nodeName> so DaemonSet pod updates or crashes do not create orphaned PVs.

Detailed Changes

Core Provisioner Updates

  • go.mod: Upgraded sigs.k8s.io/sig-storage-lib-external-provisioner/v6 to sigs.k8s.io/sig-storage-lib-external-provisioner/v13 (v13.0.0) to ensure compatibility with modern client-go v1.36 APIs.
  • pkg/storage/storage_provisioner.go:
    • Initialized nodeName using NODE_NAME environment variable.
    • Generated deterministic, node-specific identities.
    • Implemented intercept checks in Provision() to ignore PVCs targeted at other nodes.
    • Programmed VolumeNodeAffinity generation on PV metadata specs.
    • Disabled leader election on the provisioner controller.

Manifest & Build Configuration Updates

  • deploy/addons/storage-provisioner/storage-provisioner.yaml.tmpl: Converted the workload template from Pod to DaemonSet and added Coordination lease RBAC rules.
  • deploy/addons/storageclass/storageclass.yaml: Configured volumeBindingMode: WaitForFirstConsumer.
  • Makefile / deploy/storage-provisioner/Dockerfile: Bumped tag to v6.0.1 and migrated custom arch build parameter to standard Buildx TARGETARCH.

Test Suite Additions

  • test/integration/multinode_test.go: Added validateStorageProvisionerNodeAffinity integration test. It deploys a pod targeted at a worker node (multinode-m02), verifies that the local PV is successfully created, and asserts that the correct NodeAffinity metadata points to multinode-m02.
  • test/integration/testdata/: Added node-affinity-pvc.yaml and node-affinity-pod.yaml test manifests.

Verification Results

Tested successfully against a local 2-node Docker driver cluster with the integration test suite:

$ go test ./test/integration -tags="integration" -run="TestMultiNode/serial/FreshStart2Nodes|TestMultiNode/serial/StorageProvisionerNodeAffinity" -binary="../../out/minikube" -profile="multinode" -minikube-start-args="--driver=docker" -cleanup=false -v

Log Output:

=== RUN   TestMultiNode/serial/FreshStart2Nodes
    ...
    multinode_test.go:98: (dbg) Done: out/minikube start -p multinode ... (1m23.14s)
=== RUN   TestMultiNode/serial/StorageProvisionerNodeAffinity
    ...
    multinode_test.go:702: Found correct NodeAffinity: multinode-m02
    multinode_test.go:620: (dbg) Done: out/minikube kubectl -p multinode -- delete pod affinity-pod: (36.74s)
    ...
--- PASS: TestMultiNode (125.36s)
    --- PASS: TestMultiNode/serial (125.31s)
        --- PASS: TestMultiNode/serial/FreshStart2Nodes (84.73s)
        --- PASS: TestMultiNode/serial/StorageProvisionerNodeAffinity (40.58s)
PASS
ok      k8s.io/minikube/test/integration        125.819s

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 8, 2026
@k8s-ci-robot
Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot requested review from nirs and prezha May 8, 2026 18:23
@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 8, 2026
@medyagh medyagh changed the title Storage provisioner Storage provisioner: deploy as DaemonSet and support Node Affinity May 8, 2026
medyagh added 2 commits May 8, 2026 18:26
…meBindingMode: WaitForFirstConsumer), a PersistentVolume (PV) is only provisioned and the PVC bound after a Pod using the PVC is successfully scheduled.
@k8s-ci-robot
Copy link
Copy Markdown
Contributor

@medyagh: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-minikube-none-docker-linux-x86 e1b7d75 link true /test pull-minikube-none-docker-linux-x86
pull-minikube-docker-docker-linux-x86 e1b7d75 link true /test pull-minikube-docker-docker-linux-x86
pull-minikube-docker-crio-linux-x86 e1b7d75 link false /test pull-minikube-docker-crio-linux-x86
pull-minikube-docker-containerd-linux-x86 e1b7d75 link true /test pull-minikube-docker-containerd-linux-x86
pull-minikube-docker-docker-linux-arm e1b7d75 link true /test pull-minikube-docker-docker-linux-arm
pull-minikube-kvm-crio-linux-x86 e1b7d75 link false /test pull-minikube-kvm-crio-linux-x86
pull-minikube-docker-containerd-linux-arm e1b7d75 link false /test pull-minikube-docker-containerd-linux-arm
pull-minikube-kvm-docker-linux-x86 e1b7d75 link true /test pull-minikube-kvm-docker-linux-x86
pull-minikube-kvm-containerd-linux-x86 e1b7d75 link true /test pull-minikube-kvm-containerd-linux-x86

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 12, 2026
@k8s-ci-robot
Copy link
Copy Markdown
Contributor

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants