User volume formatted onto an in-use raw partition after "error writing GPT: failed to resize partition 2: invalid argument" #14583

Bug Report

What happened? What did you expect?

On 2026-10-06 at 11:30:23 UTC, on node singel-alpha, Talos failed to create the partition for the user volume u-garage:

error creating partition: error writing GPT: failed to resize partition 2: invalid argument

About 400 ms later, Talos formatted /dev/sda2 as XFS for that volume and mounted it at /var/mnt/garage. At that moment, /dev/sda2 was still the raw volume r-openebs, which the Mayastor io-engine had open. The Mayastor pool on it was destroyed, and Garage then wrote about 65 GB to the mounted filesystem.

We expected the volume to stay failed, or to ask for a reboot. We did not expect Talos to format a device.

Observed:

• There is no partition created line for u-garage before the format.
• The format took 73 s. talosctl get volumestatus showed u-garage at about 980 GB, which is the size of the raw partition. The configured maximum is 605 GiB.
• After a reboot, the kernel's partitions were sda1 u-local, sda2 u-garage and sda3 r-openebs. Before this change, r-openebs was sda2.

What we suspect (not verified): this is the situation smira described in #11223. The new table puts u-garage in front of a partition that is in use, so the kernel can't apply it. The table had already been written to disk, so the next reconcile found u-garage as partition 2 and formatted the kernel's /dev/sda2.

Steps to reproduce

This happened once. We have not tried to reproduce it. The same steps on two other nodes of the same cluster did not destroy data:

• On one node, io-engine had not started yet when u-garage was created, about one minute after a reboot.
• On the other node, we stopped io-engine before step 3.

Setup: one SATA SSD per node, holding a user volume and a raw volume. The raw volume is kept open by Mayastor io-engine, which uses /dev/disk/by-partlabel/r-openebs.

Disk layout before:

sda1  u-garage   UserVolumeConfig "garage", xfs, ~950 GiB
sda2  r-openebs  RawVolumeConfig "openebs", ~975 GiB, held open by io-engine

1. Apply a config with a smaller garage and a new local volume:

   apiVersion: v1alpha1
   kind: UserVolumeConfig
   name: garage
   provisioning:
     diskSelector:
       match: disk.wwid == "naa.<redacted>"
     minSize: 590GiB
     maxSize: 605GiB
   ---
   apiVersion: v1alpha1
   kind: UserVolumeConfig
   name: local
   provisioning:
     diskSelector:
       match: disk.wwid == "naa.<redacted>"
     minSize: 320GiB
     maxSize: 340GiB
   filesystem:
     type: xfs
   ---
   apiVersion: v1alpha1
   kind: RawVolumeConfig
   name: openebs
   provisioning:
     diskSelector:
       match: disk.wwid == "naa.<redacted>"
     minSize: 900GiB
     maxSize: 975GiB

   u-local goes failed because there's no free space, and u-garage stays unchanged.
2. Remove the garage document:

   talosctl -n <node> patch mc --patch @rm-garage.yaml   # kind: UserVolumeConfig, name: garage, $patch: delete
   

3. talosctl -n <node> wipe disk sda1 --drop-partition. Talos creates u-local as partition 1.
4. Apply the config from step 1 again, which restores the garage document.

Logs and diagnostics

We have no support bundle. The node had to be rebuilt to recover, and the reboot cleared its logs. The lines below come from the node's kernel log, which our log pipeline shipped to Loki (UTC, controller fields shortened):

11:21:33.892 [talos] volume unmount {"volume": "u-garage", "source": "/dev/sda1", "target": "/var/mnt/garage", "filesystem": "xfs"}
11:22:37.061 [talos] wiping block device "sda1" with fast method
11:24:26.017 [talos] deleted partition 1 from block device "sda"
11:24:45.361 [talos] partition created {"volume": "u-local", "disk": "/dev/sda", "partition": 1, "label": "u-local", "size": "340 GiB"}
11:24:45.769 [talos] formatting filesystem {"volume": "u-local", "device": "/dev/sda1", "filesystem": "xfs"}
11:30:23.116 [talos] volume status {"volume": "u-garage", "phase": "waiting -> failed", "error": "error creating partition: error writing GPT: failed to resize partition 2: invalid argument"}
11:30:23.122 [talos] error probing disk {"volume": "u-garage", "disk": "/dev/sda", "error": "failed to acquire shared lock while probing blockdevice"}
11:30:23.122 [talos] volume status {"volume": "u-garage", "phase": "failed -> failed", "error": "no disks matched for volume (1 matched selector): 0 have not enough space, 0 have wrong format, 1 have other issues"}
11:30:23.129 [talos] error probing disk {"volume": "u-garage", "disk": "/dev/sda", "error": "failed to acquire shared lock while probing blockdevice"}
11:30:23.131 [talos] error probing disk {"volume": "u-garage", "disk": "/dev/sda", "error": "failed to acquire shared lock while probing blockdevice"}
11:30:23.527 [talos] formatting filesystem {"volume": "u-garage", "device": "/dev/sda2", "filesystem": "xfs"}
11:31:36.689 [talos] volume status {"volume": "u-garage", "phase": "failed -> ready", "location": "/dev/sda2", "parentLocation": "/dev/sda"}
11:31:36.722 [talos] volume mount {"volume": "u-garage", "source": "/dev/sda2", "target": "/var/mnt/garage", "filesystem": "xfs"}

Environment

• Talos version: server v1.13.9; client v1.14.1 (current; the client used at the time of the incident is not known)
• Kubernetes version: v1.35.0
• Platform: bare metal, mini PC (SMBIOS: AZW EQ), 3 control-plane nodes; user and raw volumes on one Crucial CT2000BX500SSD1 (2 TB SATA SSD) per node
• System extensions: i915, intel-ice-firmware, intel-ucode, mei, tailscale. OpenEBS Mayastor v2.9.2 holds the raw volume open.
• Recent changes: the volume config change described in the steps. This was the first time we changed partitions in front of an in-use raw volume.