Skip to content

[Disk Manager] queue new ready snapshots of configured folders for backup - #7501

Draft
BarkovBG wants to merge 4 commits into
mainfrom
users/barkovbg/backup-folder-backfill-v1
Draft

BarkovBG wants to merge 4 commits into
mainfrom
users/barkovbg/backup-folder-backfill-v1

Conversation

@BarkovBG

Copy link
Copy Markdown
Collaborator

A new ready snapshot of a folder listed in SnapshotStorageBackupConfig.FolderIds is queued for backup. Stacked on #7313: the first three commits are that PR.

  • The snapshot goes to backup_queue in the transaction that makes it ready. The create task ID is the backup ID: a snapshot becomes ready once.
  • Snapshots of other folders are not queued automatically. An existing snapshot is queued by the snapshots backup admin command.
  • A queued snapshot that is deleted before the copy starts leaves the queue, as in [Disk Manager] Backup snapshots: remove copies from the follower bucket #7313.

…the copy

Back up a snapshot queued with `disk-manager-admin snapshots backup --id <id>`. Snapshots are no longer queued for backup on creation.

- `dataplane.BackupSnapshotData` holds the snapshot through a `backup_task_id` column of its dataplane row for the whole copy, so deletion waits for it. On success and on `Cancel` the copy clears the snapshot's queued chunks and releases the hold.
- The copy enqueues every chunk of the map that is not in the new `follower_chunks` table; a worker records each copied chunk there. A chunk shared by several snapshots is copied once, even if its creator is gone. `backup_chunks` is dropped: the copy is complete when none of its chunks is left in `backup_chunk_queue`.
- `backup_queue` rows carry a backup ID, so a snapshot can be queued again after a cancelled or failed attempt. `snapshots get` shows `backup_completed`.
…image is deleted

A chunk leaves the follower when it leaves chunk_blobs. Chunk IDs are never
reused, so a late write or delete of its object cannot hit a live chunk.

- The backup copy writes a follower_chunks row before it queues a chunk; the
  worker marks it copied. The last unref of a chunk moves its row to
  backup_chunk_delete_queue in the same transaction, and the regular
  dataplane.DeleteBackupChunks deletes the objects.
- dataplane.DeleteBackupSnapshotData waits until the snapshot is gone and
  deletes its chunk map.
- Deleting a snapshot moves its queued backup attempt to backup_delete_queue.
  snapshots.DeleteBackupMeta cancels that attempt, waits for it to end, then
  deletes the chunk map and meta.json. images.DeleteBackupMeta deletes the
  chunk map and meta.json of a deleted image.
… backup on deletion

Deleting a snapshot no longer cancels its backup: the deletion waits for the
running copy, as the copy holds the snapshot. The copy writes meta.json last,
while it holds the snapshot, so meta.json is never written after the
deletion removed it.

- dataplane.BackupSnapshotData gets the meta key and contents in its request
  and writes them after the chunk map. snapshots.BackupSnapshot and
  images.BackupImage no longer write meta.json themselves.
- backup_delete_queue no longer carries a backup ID, and
  snapshots.DeleteBackupMeta does not cancel the backup.
…ckup

Queue new ready snapshots of folders listed in SnapshotStorageBackupConfig.FolderIds for backup in the transaction that makes them ready, with the create task as the backup ID. Existing snapshots are queued by the snapshots backup admin command.
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Note

This is an automated comment that will be appended during run.

Note

All workloads for linux-x86_64-relwithdebinfo have completed.

Tip

Planned checks for linux-x86_64-relwithdebinfo.

🟢 linux-x86_64-relwithdebinfo target: cloud/tasks/,cloud/storage/,cloud/fastshard/ (test time: 272s): all tests PASSED for commit 6fb9e73.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
1745 1745 0 0 0 0 0

🟢 linux-x86_64-relwithdebinfo target: cloud/blockstore/ (test time: 1652s): all tests PASSED for commit 6fb9e73.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
7643 7642 0 0 0 1 0

🟢 linux-x86_64-relwithdebinfo target: cloud/filestore/ (test time: 5074s): all tests PASSED for commit 6fb9e73.

TESTS PASSED ERRORS FAILED FAILED BUILD SKIPPED MUTED?
4973 4972 0 0 0 1 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant