Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning

PM4Bench is a strictly parallel multilingual vision-language benchmark over 10 languages and four tasks:

• MDUR — multi-discipline understanding and reasoning (1,730 items/language)
• MIQA — multi-image open-ended question answering (218 items/language)
• MSOCR — multi-scale OCR (100 items/language)
• MGUI — multilingual GUI grounding (200 items/language)

The paper uses controlled comparisons between interleaved and vision-only input to identify OCR as a key source of cross-lingual performance gaps. QGO then targets this bottleneck with OCR-centric GRPO on fully synthesized data. This repository contains the clean v2 evaluation, validation, construction, and training code for that study.

The former TSV/base64 representation is retired. The v2 dataset stores each image once and references it from portable JSONL manifests.

git clone https://github.com/opendatalab/PM4Bench.git
cd PM4Bench
python -m pip install -e .

Install optional dependencies only when needed:

python -m pip install -e '.[judge]'   # MIQA LLM-as-judge
python -m pip install -e '.[synthesis]'  # all four vision tasks and QGO OCR synthesis
python -m pip install -e '.[dev]'     # tests and lint

PM4Bench/
├── data/{mdur,miqa,msocr,mgui}/{language}.jsonl
├── assets/
│   ├── mdur/{traditional,vision}/...
│   ├── miqa/{traditional,vision}/...
│   ├── msocr/...
│   └── mgui/...
└── metadata/
    ├── release_inventory.json
    └── mgui/{gt,templates}/...

See DATA_FORMAT.md for schemas and asset resolution.

pm4bench validate --dataset-root /path/to/PM4Bench-snapshot

Predictions are JSONL records containing the manifest id (or legacy index) and a response string.

{"id": "a01_search_en_1", "response": "(318, 312)"}

pm4bench evaluate \
  --task mgui \
  --manifest /path/to/data/mgui/en.jsonl \
  --predictions predictions/mgui_en.jsonl \
  --coordinate-space normalized-1000

MDUR, MSOCR, and MGUI use deterministic local metrics. MIQA follows the paper's six-dimension LLM-as-judge protocol; the judge endpoint and model are always supplied by the user and no provider-specific URL or credential is stored in this repository.

The deterministic evaluator also exposes the paper's OCR diagnostics and the MGUI text-identification sanity check:

pm4bench evaluate --task mdur-ocr --manifest data/mdur/en.jsonl --predictions predictions.jsonl
pm4bench evaluate --task miqa-ocr --manifest data/miqa/en.jsonl --predictions predictions.jsonl
pm4bench evaluate --task mgui-text --manifest data/mgui/en.jsonl --predictions predictions.jsonl

Run MIQA's resumable judge with an explicitly selected OpenAI-compatible model and credential environment variable:

pm4bench judge-miqa \
  --manifest data/miqa/en.jsonl \
  --predictions predictions.jsonl \
  --output judgments/miqa_en.jsonl \
  --judge-model YOUR_JUDGE_MODEL

All four tasks use one interface, pm4bench render-vision --task TASK. It reads the released manifests, checks their content against the v2.0.0 snapshot, and writes checked plans, generated images, and a provenance report to a new directory. The same language, record-selection, and audit options apply to MDUR, MIQA, MSOCR, and MGUI.

python -m pip install -e '.[synthesis]'
python -m playwright install chromium
python -m pm4bench.vision.fonts --output ./fonts
pm4bench render-vision \
  --task miqa \
  --dataset-root /path/to/PM4Bench-snapshot \
  --fonts-root ./fonts \
  --output-root ./rendered-miqa \
  --language en --limit 2

Use --task mdur, msocr, or mgui to select another task. MGUI uses the historical system font stack, so omit --fonts-root for that task. Add --audit-only to check inputs without rendering; no fonts or browser are needed in audit mode.

Vision synthesis is the shared guide for setup, the CLI and Python API, task-specific rendering options, and output layout. Browser-based MDUR/MGUI and raster-based MIQA/MSOCR retain their own layout logic behind this interface. Rendering does not overwrite the released data; use the published images for comparable evaluation.

QGO-8B is the global-step-200 BF16 GRPO checkpoint derived from Qwen/Qwen3-VL-8B-Thinking. The exact release data is provided as 19,500 training and 500 validation rows with embedded images. See recipes/qgo/README.md.

To generate multilingual OCR images and visible-word targets and pack training Parquet, follow OCR_TRAIN_SYNTHESIS.md. Original text/background inputs and the published split inventory accompany the code.

• MDUR is derived from MMMU-Pro (Apache-2.0).
• MIQA is derived from MMDU (CC BY-NC 4.0).
• MSOCR and MGUI are PM4Bench-generated components.
• QGO-8B inherits the Apache-2.0 license of its Qwen base model.

Component-level provenance and licenses are documented in the dataset cards. Project-owned code and artifacts are Apache-2.0.

python -m pip install -e '.[dev]'
ruff check .
pytest

For problems with data or evaluation, open a GitHub issue with the task, language, sample identifier, and toolkit version. Do not include API keys or private model outputs.

@misc{gao2026benchmarkingboostingmultilingualcapabilities,
      title={Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning},
      author={Junyuan Gao and Jiahe Song and Jiang Wu and Runchuan Zhu and Guanlin Shen and Shasha Wang and Xingjian Wei and Haote Yang and Weijia Li and Bin Wang and Lijun Wu and Conghui He},
      year={2026},
      eprint={2503.18484},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2503.18484v3},
}