CNMLEQA is a large-scale benchmark dataset for evaluating large language models (LLMs) on the Chinese National Medical Licensing Examination (CNMLE).

This dataset has been officially published in Scientific Data:

Zong H, Cha J, Wang J, Song Y, Zhao Y, Shi M, Shen B. A Dataset for Evaluating Large Language Models on Chinese National Medical Licensing Examinations. Sci Data. 2026 Apr 17. doi: 10.1038/s41597-026-07261-9. PMID: 41998016.

Large language models have shown strong performance in medical QA and clinical reasoning. However, standardized evaluation benchmarks in non-English medical contexts remain limited.

CNMLEQA addresses this gap by providing a clinically grounded, expert-annotated benchmark for evaluating LLM performance in Chinese medical examinations.

• Clinically grounded: Derived from real CNMLE-style questions
• Multidimensional annotations: Enables fine-grained evaluation
• Non-English benchmark: Focused on Chinese medical context
• LLM-evaluated: Tested on state-of-the-art models

• Multiple-choice questions (MCQs)
• 5 options (A–E)
• Single correct answer
• Language: Chinese

• Knowledge-based questions
• Case-based clinical questions

Each question is annotated with multi-level clinical metadata.

{
  "id": "unique-id",
  "question": "Question text",
  "opa": "Option A",
  "opb": "Option B",
  "opc": "Option C",
  "opd": "Option D",
  "ope": "Option E",
  "answer": "Correct option",
  "question_type": "Knowledge-based / Case-based",
  "year": "Exam year",
  "source": "Data source"
}

Each question is further annotated across five clinical dimensions:

• Disease / Diagnosis
• Surgery
• Medication
• Laboratory Examination
• Symptom / Sign

All annotations were conducted and verified by clinical experts.

We evaluated CNMLEQA using multiple state-of-the-art LLMs, including:

• Gemini
• DeepSeek
• GPT series
• Qwen series
• LLaMA

Key Results

• Qwen2.5-32B: 90.88% (CNMLEQA-10k)
• DeepSeek-R1: 91.59% (CNMLEQA-3k)

Fine-tuning experiments (on Qwen models) further demonstrated significant performance improvements.

The dataset is publicly available at:

• Zenodo: https://zenodo.org/records/18951465
• Github: https://github.com/zonghui0228/CNMLEQA

If you use CNMLEQA in your research, please cite:

1. Zong H, Cha J, Wang J, Song Y, Zhao Y, Shi M, Shen B. A Dataset for Evaluating Large Language Models on Chinese National Medical Licensing Examinations. Scientific Data. 2026 Apr 17. PMID: 41998016. DOI: 10.1038/s41597-026-07261-9.

2. Zong H, Wu R, Cha J, et al. Large Language Models in Worldwide Medical Exams: Platform Development and Comprehensive Analysis. Journal of Medical Internet Research. 2024;26:e66114. PMID: 39729356 DOI: 10.2196/66114

3. Zong H, Li J, Wu E, et al. Performance of ChatGPT on Chinese national medical licensing examinations: a five-year examination evaluation study for physicians, pharmacists and nurses. BMC Medical Education 24, 143 (2024). PMID: 38355517 DOI: 10.1186/s12909-024-05125-7