Add KMMLU (Korean-native MMLU, 45 subjects) - #1425

Adds KMMLU (HAERAE-HUB/KMMLU), a Korean multiple-choice benchmark built from original Korean exams: 35,030 questions in 45 subjects (arXiv 2402.11548). This follows the proposal in #1424.

What it adds

One file, src/lighteval/tasks/tasks/kmmlu.py: 45 tasks named kmmlu:<subject> (for example kmmlu:accounting), one per HF config, test split, loglikelihood_acc.

The prompt copies the default kmmlu task of lm-evaluation-harness (v0.4.13), so scores from the two tools can be compared: the question, the four options as A. to D., then 정답： (full-width colon), choices " A" to " D", gold answer - 1. The dataset revision is pinned (d61b3f1). In #1424 I asked whether you'd rather have the multilingual template style (get_mcq_prompt_function, like cmmlu). Happy to switch if you prefer it.

How I checked it

I ran this task as a --custom-tasks file with lighteval 0.13.0, ran lm-eval 0.4.13 on the same model, and compared each question. Setup: Qwen/Qwen2.5-0.5B-Instruct (revision 7ae5576), float32, CPU, batch size 1, 0-shot, chat template off.

All 700 questions got the same prompt, tokens and predicted choice. Code and a reproduce.py that prints this table: https://github.com/CY-HYUN/kmmlu-lighteval

On main (7fa19dc) I checked that the config fields and Metrics.loglikelihood_acc the file uses are unchanged, and ran ruff check and ruff format with the repo's config (ruff 0.11.10). I haven't run it from a source install of main.

Limits

• 5 of 45 subjects, one small model, CPU only. A full 45-subject run on GPU isn't done yet.
• Only the log-likelihood variant. lm-eval also has kmmlu_direct, kmmlu_hard and chain-of-thought variants.
• lighteval tokenizes prompt and choice separately and lm-eval tokenizes them together, so they agree only when no token crosses the boundary. That held for the Qwen2.5 tokenizer on these 700 questions.

I used an AI coding assistant (Claude Code) to write and check this.