Skip to content

Isolate Hugging Face dataset cache - #13

Open
GroveMinting wants to merge 1 commit into
volotat:mainfrom
GroveMinting:fix/hf-cache-isolation
Open

GroveMinting wants to merge 1 commit into
volotat:mainfrom
GroveMinting:fix/hf-cache-isolation

Conversation

@GroveMinting

Copy link
Copy Markdown

Fixes #12.

What

  • Give each non---keep-cache fetch a process-owned temporary datasets cache.
  • Place it inside the resolved output directory so large Arrow intermediates use the corpus filesystem rather than a potentially small /tmp.
  • Pass that path explicitly to load_dataset(cache_dir=...).
  • Remove only the owned path on success and conversion failure.
  • Complete cleanup before the streaming os._exit(0) path.
  • Fail instead of exiting successfully when cleanup fails.
  • Report both conversion and cleanup errors when both occur.
  • Leave --keep-cache on the normal shared Hugging Face cache without overriding it.

Why

The previous default recursively deleted ~/.cache/huggingface/datasets, which may contain unrelated datasets belonging to other projects. Mini-AGI should only remove cache files it created and owns.

Verification

python -m unittest discover -s tests -v

Seven tests pass. They cover:

  • preserving an unrelated shared-cache sentinel;
  • owned cache placement, use, and cleanup;
  • --keep-cache using the library default;
  • cleanup after load failure;
  • cleanup before streaming immediate exit;
  • cleanup failure preventing successful exit;
  • combined conversion and cleanup error reporting;
  • resolved-path cleanup after output symlink retargeting.

Also verified with compileall and git diff --check. The patch is independent of open PRs #7, #9, and #11.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dataset fetch deletes unrelated Hugging Face caches

1 participant