Skip to content

Add data parallel support for vllm to speed up evals - #50

Open
BrownianNotion wants to merge 5 commits into
allenai:mainfrom
BrownianNotion:vllm-data-parallel
Open

Add data parallel support for vllm to speed up evals#50
BrownianNotion wants to merge 5 commits into
allenai:mainfrom
BrownianNotion:vllm-data-parallel

Conversation

@BrownianNotion

@BrownianNotion BrownianNotion commented Mar 9, 2026

Copy link
Copy Markdown

Given that evals take very long for large models, it would be nice to have data parallel working for vllm [see #31 #39].

My understanding is that the current codebase tries to address this with multi-processing, similarly to the vllm docs https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/data_parallel.py.
However, this is currently broken #31 and additionally, unnecessary, since lm_eval already supports data parallel natively.

This PR should address these issues by adding lm_eval's data parallel support to Olmes. The first four commits are from #49 .

Tested with command on 4xh100s.

dp=2
tp=2
  olmes \
      --model "olmo-2-7b-1124" \
      --model-args '{"chat_model": true, "max_length": 4096, "metadata": {}, "trust_remote_code": true, "gpu_memory_utilization": 0.85, "data_parallel_size": '"$dp"', "tensor_parallel_size": '"$tp"'}' \
      --model-type vllm \
      --task-args '{ "generation_kwargs": { "max_gen_toks": 4096, "truncate_context": false } }' \
      --task gsm8k::tulu ifeval::tulu \
      --output-dir "eval_results"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant