Repository navigation
Tags: Lightning-AI/litData
Tags
chore: bump version 0.2.72 (#892) Release 0.2.72 with the S3 credential refresh reliability work (#891) and the numpy deserialize copy fix (#887). Also corrects the changelog: the multi-node FullShuffle fix and the max_cache_size change were left under [unreleased] by #886, but #886 is the commit v0.2.71 was tagged at, so they shipped in 0.2.71. Moved them into that section and added the missing entry for #887. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
feat(streaming): pin node shards when they fit, else reshuffle global… …ly (#886) * feat(streaming): pin node shards when they fit, else reshuffle globally Keep unique chunks on a node across epochs when they fit in cache, and default max_cache_size to a conservative slice of free disk so checkpoints still have room. Co-authored-by: Cursor <cursoragent@cursor.com> * docs: keep shuffle and cache-size notes under unreleased Co-authored-by: Cursor <cursoragent@cursor.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests: drop unused shuffle expected list Co-authored-by: Cursor <cursoragent@cursor.com> * fix(streaming): cache shuffle chunk count and drop duplicate resume line Co-authored-by: Cursor <cursoragent@cursor.com> * tests: pin emulate-S3 cache size so CI does not evict under mmap Co-authored-by: Cursor <cursoragent@cursor.com> * feat(streaming): accept max_cache_size as size or free-disk fraction None still uses 75% of free disk with a 50GB reserve. "100G" pins bytes; 0.90 uses that share of currently free space. Co-authored-by: Cursor <cursoragent@cursor.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * style: drop escaped quotes in max_cache_size docstring for ruff D301 Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
bump version 0.2.68 (#877) * bump version 0.2.68 Align skills/README with POSIX-fast: WindowShuffle only on parallel FS, worker/WILLNEED RAM caps, and TokensLoader copies off mmap. Co-authored-by: Cursor <cursoragent@cursor.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
PreviousNext