Repository navigation
Conversation
The check compared each loaded name with itself, so a task with no details file was reported as present.
HarshShroff
left a comment
There was a problem hiding this comment.
I ran this at d6a72b3 (macOS, Python 3.12). tests/unit/logging/test_evaluation_tracker.py gives 11 passed, 1 skipped. The startswith shadowing is real: on main the check can only fire when no details were loaded at all.
One thing about the names, though. The new docstring says the task list passed to load_details_datasets has no few-shot suffix. The only caller in the repo is Pipeline._load_responses_from_details, which passes list(self.tasks_dict.keys()), and Registry.load_tasks keys that dict by config.full_name, which is f"{expanded_task}|{config.num_fewshots}". So the requested names look like gsm8k|0, the same as the stored keys.
With those names every file is skipped (gsm8k is not in ["gsm8k|0"]) and the reload raises even though the file is there. To check, I took the keys from Registry(tasks="gsm8k|0").load_tasks(), saved one details file with save_details, and called Pipeline._load_responses_from_details on a stub holding that dict and the tracker. (I stubbed HfApi.list_models, because importing the task modules calls it and fails on my huggingface_hub version.)
tasks_dict keys: ['gsm8k|0']
ValueError: Task gsm8k|0 not found in details datasets. Check the tasks to be evaluated or the date_id used to load the details (2026-10-11T00-00-00.000000).
main gives the same error, so this PR does not make it worse. But the helper now writes the suffix-free convention down, and the new tests only pass names like gsm8k, so they can't see it. Calling load_details_datasets directly with ["gsm8k"] does load the file on both.
Letting the stored key match as well was enough for me:
if task_name not in task_names and detail_task_name(task_name) not in task_names:
...
loaded = {detail_task_name(name) for name in stored_names} | set(stored_names)With that the same probe reloads the response, and the tracker tests still pass (11 passed, 1 skipped). If the suffix-free form isn't used anywhere, comparing the full name only would be simpler. A test that goes through names shaped like tasks_dict keys would cover it either way.
Summary
name.startswith(name)because the loop variable shadowed the requested task.gsmdoes not count asgsm8k.Test plan