Repository navigation
store: publish the manifest before pruning stale experts - #34
Closed
GroveMinting wants to merge 1 commit into
Closed
GroveMinting wants to merge 1 commit into
GroveMinting wants to merge 1 commit into
Conversation
Author
|
Moving this to the fork: upstream has not been merging community PRs. The prune-ordering fix and the #32 Windows retry changes are now carried on GroveMinting/mini-AGI main, where they can be tested together. Closing this upstream PR rather than leaving it waiting. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
store.savedeletes expert files that are no longer in the new manifest before it publishes that manifest. If the process dies between the deletion and the atomic manifest replace, the old manifest still names the deleted files and the checkpoint cannot be loaded.This can happen on a prune, a shape-changing save, a crash, a SIGKILL, or ENOSPC.
Fix
Count stale expert files before publishing, then delete them only after the new manifest is durable. The deletion is placed after the existing manifest publication block, so this change is deliberately independent of the publication mechanism.
Stale extra files are harmless in the meantime: the loader reads only the files named in the manifest, and the next successful save removes them.
Compatibility with #32
This composes with #32, which adds a retrying
_replacefor Windows reader locks. It touches neither_saveznor the manifestreplacecall, and I verified the combined tree in both merge orders:tests.test_soakandtests.test_windows_runtimepass together (13 tests).Files
minagi/store.py— defer stale-expert deletion until after manifest publication.tests/test_soak.py— new regression tests for interrupted manifest publication, ENOSPC, retry, and prune ordering.The manual fault-injection soak runner stays in my fork; it is not needed for this fix.
Validation
aa9d53fand to Fix Windows training: UTF-8 logs, direct plot spawns, safe checkpoint replace #32's branch;tests.test_soakpasses 3/3 on both.