trying to run multiple using olmes docker - #44

found the version

ok for real this time

Remove hard dependency on lm_eval.utils.eval_logger so oe_eval works with newer lm_eval releases where eval_logger is no longer exported (e.g. 0.4.9.2, 0.4.10).

Add oe_eval/lm_eval_logging.py:get_eval_logger() with compatibility behavior:

- use lm_eval.utils.eval_logger when present

- else call lm_eval.utils.setup_logging() when available

- then fallback to logging.getLogger('lm_eval')

Update logger wiring in:

- oe_eval/models/eleuther_huggingface.py

- oe_eval/models/eleuther_olmo_core.py

- oe_eval/models/litellm.py

- oe_eval/dependencies/ifeval/utils.py

Validation:

- rg 'utils\.eval_logger|from lm_eval\.utils import eval_logger' oe_eval => no matches

- uv run python -c 'import oe_eval.models.eleuther_huggingface' => success

- uv run python -c 'import oe_eval.models.eleuther_huggingface, oe_eval.models.eleuther_olmo_core, oe_eval.models.litellm, oe_eval.dependencies.ifeval.utils' => success

The `add_bos_token` attribute from lm_eval's VLLM class defaults to
None, but the HuggingFace fast tokenizer's `add_special_tokens` parameter
requires a bool. This causes a TypeError with newer transformers versions:
"'NoneType' object cannot be converted to 'PyBool'"

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The lm_eval VLLM._model_generate() signature changed to accept a
SamplingParams object instead of individual kwargs like max_tokens
and stop. Construct vllm.SamplingParams and pass it explicitly.

Also filter out HuggingFace-style kwargs (e.g. do_sample) that are
not valid SamplingParams fields, translating do_sample=False to
temperature=0.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

When tasks are passed as JSON strings (e.g. from external launchers),
suite names like "mmlu:mc::olmes" were not being expanded because
resolve_task_suite only ran on raw CLI strings before JSON parsing.
Now also checks TASK_SUITE_CONFIGS after parsing JSON task_name and
expands suites with extra params (num_shots, etc.) applied to each
expanded task.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

AGIEval, HELMET, BFCL and other dependency data files (jsonl, json,
txt, yaml) were not being included in wheel builds because the data
directories lack __init__.py. Add explicit package-data globs for
oe_eval to ensure these files are distributed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>