Skip to content

issue-6139: [Disk Manager] pools: fold 'configs' into 'pools', fix race with image deletion - #7516

Open
SvartMetal wants to merge 1 commit into
mainfrom
users/svartmetal/issue-6139/2
Open

SvartMetal wants to merge 1 commit into
mainfrom
users/svartmetal/issue-6139/2

Conversation

@SvartMetal

Copy link
Copy Markdown
Collaborator

Notes

Problem. If an image is deleted shortly after its pools are configured, an entry for it can be left behind in pools/configs. pools.ScheduleBaseDisks then keeps generating base disks for an image whose snapshot no longer exists: every dataplane.TransferFromSnapshotToDisk fails, the base disk goes creation_failed, pool size drops, and the next tick schedules another one. This loops forever (allocating a real NBS disk on every iteration) until someone calls DeletePool via the private API.

Root cause. A pool's desired state (capacity, image_size) lived in configs, while its row in pools was materialized lazily - by the first scheduling tick or the first base-disk transition. pools.ImageDeleting only scanned pools where status = ready, and deletePool early-returned when there was no pools row. So any pool that had been configured but not yet scheduled was invisible to deletion. The window is up to ScheduleBaseDisksTaskScheduleInterval (1m by default) and is hit by:

  • cancelling a CreateImage right after the image became ready (the ConfigurePool child tasks are not cancelled with the parent and keep running);
  • DeleteImage within a minute of creation;
  • AcquireBaseDiskSlot creating an on-demand pool (createPoolIfNecessary also wrote to configs only) followed by image deletion.

A second, much narrower window existed inside configurePoolTask: the image check and the config write are separate transactions, so ImageDeleting could run between them.

Fix.

  1. configs is folded into pools. pools gets capacity and image_size; configurePool and createPoolIfNecessary write the row immediately, so ImageDeleting sees every configured pool and marks it deleted, which configurePool already refuses to resurrect. The invariant is pool is configured ⇔ status = ready.

  2. The remaining race is closed by pools.ConfigurePool itself. configurePoolTask.checkImage runs before and after the write (and on Cancel). For the post-write check to be a fence, images.deleteImage now marks the image as deleting before scheduling pools.ImageDeleting.

Migration (two releases; everything legacy is marked TODO: remove after migration is completed):

  • This PR: capacity/image_size columns are added; capacity IS NULL marks a row not yet written by the new code. Pool writers dual-write configs so old nodes keep working during the rollout; reads fall back to configs only for NULL-capacity rows. CreateYDBTables (schema-init tool) backfills pools from configs, and takeBaseDisksToScheduleForPool migrates a pool lazily on its first tick.
  • Follow-up: drop the configs reads/writes and the table, make capacity non-nullable, remove the pool.status = ready resurrection line in the scheduler.

Issue

#6139

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Note

This is an automated comment that will be appended during run.

Note

All workloads for linux-x86_64-relwithdebinfo have completed.

Tip

Planned checks for linux-x86_64-relwithdebinfo.

  • 🚫 tasks + storage + fastshard (cancelled or timed out before reporting completion)
  • 🚫 blockstore (cancelled or timed out before reporting completion)
  • 🚫 disk + manager (cancelled or timed out before reporting completion)
  • 🚫 filestore (cancelled or timed out before reporting completion)

@SvartMetal SvartMetal added large-tests Launch large tests for PR disk_manager Add this label to run only cloud/disk_manager build and tests on PR labels Oct 11, 2026
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Note

This is an automated comment that will be appended during run.

Note

All workloads for linux-x86_64-relwithdebinfo have completed.

Tip

Planned checks for linux-x86_64-relwithdebinfo.

🟢 linux-x86_64-relwithdebinfo target: cloud/disk_manager/ (test time: 2115s): all tests PASSED for commit df5eb91.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
2216 2216 0 0 0 0 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

disk_manager Add this label to run only cloud/disk_manager build and tests on PR large-tests Launch large tests for PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant