Branch:
pvmigration| Base: karmada-io/karmada (CNCF Incubating Project)
Developed at: DCN Lab, Soongsil University
Development period: Jan 2025 – Aug 2025
Karmada supports multi-cluster workload failover — but StatefulSets with PersistentVolumes (PV) are not handled.
When a StatefulSet is rescheduled to a new cluster:
Cluster A (failed) Cluster B (target)
StatefulSet ───────────────▶ StatefulSet
PVC ─── PV (NFS) PVC ─── ??? ← PV missing → pod stays Pending
The StatefulSet arrives in the target cluster but its PVs do not. Pods remain in Pending because PVCs cannot bind.
Root cause: Karmada's failover pipeline propagates resources tracked in ResourceBinding, but PersistentVolume objects are cluster-scoped and excluded from the propagation path. No existing mechanism migrates them.
This branch adds three new controllers to pkg/controllers/pvsync/ and registers them in cmd/controller-manager/, alongside a targeted modification to the existing applicationfailover controller.
Watches Work objects in execution namespaces. For each Work containing a StatefulSet:
- Extracts
volumeClaimTemplatesto build PVC name prefixes - Connects to the member cluster via
ClusterDynamicClientSetFunc - Lists all PVs; filters by namespace + VCT prefix match (only relevant PVs)
- Serializes each PV spec as YAML, stores in a
ConfigMap - Creates/updates a metadata
WorkwithSuspendDispatching: true— metadata only, no actual PV created
Labels on metadata Work:
pvsync.karmada.io/type: metadata
pvsync.karmada.io/source-sts: <namespace>.<name>
pvsync.karmada.io/source-work: <work-name>
pvsync.karmada.io/cluster: <cluster-name>
Watches ResourceBinding for StatefulSets with Unhealthy aggregated status.
- Builds current cluster set from
rb.Spec.Clusters - Compares against clusters in PV metadata Works — detects rescheduling
- For each new target cluster: reads PV specs from ConfigMap metadata Works, creates
PV Work(NFS-backed PersistentVolume) - Parallel readiness check — spawns one goroutine per PV Work, polls phase (
Available/Bound) - When all PVs ready: releases
suspension.dispatchingwithretry.RetryOnConflictto handle update races - Deletes stale metadata Works from old clusters
Work naming: SHA1-hashed short suffix prevents name collisions across multiple PVs.
Manages full lifecycle of pvsync resources using the Kubernetes finalizer pattern.
- Registers
pvsync.karmada.io/cleanupfinalizer on all StatefulSetResourceBindingobjects - On RB deletion: removes all associated pvsync Works and deletes orphan PV/PVC from member clusters via dynamic client (
deleteOrphanPVResources) - On cluster change: garbage-collects Works from clusters no longer in
rb.Spec.Clusters
In evictBinding(), added StatefulSet detection:
- If the evicted resource is a StatefulSet, sets
suspension.dispatching = truebefore failover proceeds - Prevents Karmada scheduler from re-dispatching the StatefulSet while PV migration is in progress
| File | Type | Description |
|---|---|---|
cmd/controller-manager/app/controllermanager.go |
Modified | Registered pvSync, pvMigration, pvCleanup controllers |
pkg/controllers/pvsync/pvsync_controller.go |
New | PVSyncController |
pkg/controllers/pvsync/pvmigration_controller.go |
New | PVMigrationController |
pkg/controllers/pvsync/pvcleanup_controller.go |
New | PVCleanupController |
pkg/controllers/applicationfailover/rb_application_failover_controller.go |
Modified | StatefulSet suspension on eviction |
Net addition: ~1,000+ lines of Go across 5 files
| Decision | Rationale |
|---|---|
Metadata Work with SuspendDispatching |
Stores PV specs inside Karmada's own Work CRD — no external storage needed; suspension prevents unintended PV creation |
| Finalizer on ResourceBinding | Guarantees cleanup even if operator deletes the RB before migration completes |
| Parallel goroutine readiness check | Multiple PVCs per StatefulSet need independent readiness polling; sequential polling adds unnecessary latency |
retry.RetryOnConflict on suspension update |
ResourceBinding is updated by multiple controllers; retry prevents losing updates under concurrent writes |
| SHA1 short hash for Work names | Kubernetes name length limit (63 chars) exceeded with rb.Name + pvName + clusterName |
| Item | Status |
|---|---|
| NFS-only PV source type | CSI driver support planned |
| No upstream PR yet | Pending test coverage + CSI support |
| Unit / e2e tests | Not yet written |
| Watch-based readiness | Currently polling; Kubernetes Watch would be cleaner |
- 3 new Kubernetes controllers implemented from scratch in Go
- 1 existing controller modified (applicationfailover)
- 1 controller-manager registration updated
- Iterative development over 5 months with multiple design revisions