Repository navigation
issue-6139: [Disk Manager] pools: fold 'configs' into 'pools', fix race with image deletion - #7516
Open
SvartMetal wants to merge 1 commit into
Open
SvartMetal wants to merge 1 commit into
SvartMetal wants to merge 1 commit into
Conversation
…ce with image deletion
Contributor
|
Note This is an automated comment that will be appended during run. Note All workloads for linux-x86_64-relwithdebinfo have completed. Tip Planned checks for linux-x86_64-relwithdebinfo.
|
Contributor
|
Note This is an automated comment that will be appended during run. Note All workloads for linux-x86_64-relwithdebinfo have completed. Tip Planned checks for linux-x86_64-relwithdebinfo. 🟢 linux-x86_64-relwithdebinfo target: cloud/disk_manager/ (test time: 2115s): all tests PASSED for commit df5eb91.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Notes
Problem. If an image is deleted shortly after its pools are configured, an entry for it can be left behind in
pools/configs.pools.ScheduleBaseDisksthen keeps generating base disks for an image whose snapshot no longer exists: everydataplane.TransferFromSnapshotToDiskfails, the base disk goescreation_failed, pool size drops, and the next tick schedules another one. This loops forever (allocating a real NBS disk on every iteration) until someone callsDeletePoolvia the private API.Root cause. A pool's desired state (
capacity,image_size) lived inconfigs, while its row inpoolswas materialized lazily - by the first scheduling tick or the first base-disk transition.pools.ImageDeletingonly scannedpools where status = ready, anddeletePoolearly-returned when there was nopoolsrow. So any pool that had been configured but not yet scheduled was invisible to deletion. The window is up toScheduleBaseDisksTaskScheduleInterval(1m by default) and is hit by:CreateImageright after the image became ready (theConfigurePoolchild tasks are not cancelled with the parent and keep running);DeleteImagewithin a minute of creation;AcquireBaseDiskSlotcreating an on-demand pool (createPoolIfNecessaryalso wrote toconfigsonly) followed by image deletion.A second, much narrower window existed inside
configurePoolTask: the image check and the config write are separate transactions, soImageDeletingcould run between them.Fix.
configsis folded intopools.poolsgetscapacityandimage_size;configurePoolandcreatePoolIfNecessarywrite the row immediately, soImageDeletingsees every configured pool and marks itdeleted, whichconfigurePoolalready refuses to resurrect. The invariant is pool is configured ⇔status = ready.The remaining race is closed by
pools.ConfigurePoolitself.configurePoolTask.checkImageruns before and after the write (and onCancel). For the post-write check to be a fence,images.deleteImagenow marks the image as deleting before schedulingpools.ImageDeleting.Migration (two releases; everything legacy is marked
TODO: remove after migration is completed):capacity/image_sizecolumns are added;capacity IS NULLmarks a row not yet written by the new code. Pool writers dual-writeconfigsso old nodes keep working during the rollout; reads fall back toconfigsonly for NULL-capacity rows.CreateYDBTables(schema-init tool) backfillspoolsfromconfigs, andtakeBaseDisksToScheduleForPoolmigrates a pool lazily on its first tick.configsreads/writes and the table, makecapacitynon-nullable, remove thepool.status = readyresurrection line in the scheduler.Issue
#6139