Skip to content

Reject a missing details file when reloading an eval - #1420

Open
b423016 wants to merge 1 commit into
huggingface:mainfrom
b423016:fix/load-details-task-match
Open

b423016 wants to merge 1 commit into
huggingface:mainfrom
b423016:fix/load-details-task-match

Conversation

@b423016

@b423016 b423016 commented Oct 8, 2026

Copy link
Copy Markdown

Summary

  • Reloading an eval from saved details never noticed a missing task. The check was name.startswith(name) because the loop variable shadowed the requested task.
  • Match the same rule the loader already uses when it skips files: drop the trailing few-shot suffix and compare the task name exactly. A prefix such as gsm does not count as gsm8k.

Test plan

  • One loaded task, one missing task
  • A shorter prefix is not treated as a match
  • No error when every requested task has a details file

The check compared each loaded name with itself, so a task with no details file was reported as present.

@HarshShroff HarshShroff left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this at d6a72b3 (macOS, Python 3.12). tests/unit/logging/test_evaluation_tracker.py gives 11 passed, 1 skipped. The startswith shadowing is real: on main the check can only fire when no details were loaded at all.

One thing about the names, though. The new docstring says the task list passed to load_details_datasets has no few-shot suffix. The only caller in the repo is Pipeline._load_responses_from_details, which passes list(self.tasks_dict.keys()), and Registry.load_tasks keys that dict by config.full_name, which is f"{expanded_task}|{config.num_fewshots}". So the requested names look like gsm8k|0, the same as the stored keys.

With those names every file is skipped (gsm8k is not in ["gsm8k|0"]) and the reload raises even though the file is there. To check, I took the keys from Registry(tasks="gsm8k|0").load_tasks(), saved one details file with save_details, and called Pipeline._load_responses_from_details on a stub holding that dict and the tracker. (I stubbed HfApi.list_models, because importing the task modules calls it and fails on my huggingface_hub version.)

tasks_dict keys: ['gsm8k|0']
ValueError: Task gsm8k|0 not found in details datasets. Check the tasks to be evaluated or the date_id used to load the details (2026-10-11T00-00-00.000000).

main gives the same error, so this PR does not make it worse. But the helper now writes the suffix-free convention down, and the new tests only pass names like gsm8k, so they can't see it. Calling load_details_datasets directly with ["gsm8k"] does load the file on both.

Letting the stored key match as well was enough for me:

if task_name not in task_names and detail_task_name(task_name) not in task_names:
...
loaded = {detail_task_name(name) for name in stored_names} | set(stored_names)

With that the same probe reloads the response, and the tracker tests still pass (11 passed, 1 skipped). If the suffix-free form isn't used anywhere, comparing the full name only would be simpler. A test that goes through names shaped like tasks_dict keys would cover it either way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants