This is an implementation of COGS, as described in the paper [Composition-Grounded Instruction Synthesis for Visual Reasoning].
COGS is a data-efficient framework compositionally generates reasoning data to equip MLLM with complex visual reasoning in Chart and WebGUI understanding tasks. The repository includes:
- 🧩 Seed Data Decomposition in
question_decomp.py. - 📊 Question Generation via Factor Recomposition in
synData.py. - 📚 Data format convertion for SFT and RL training in
convert_data.
Both seed data decomposition and question generation are based on vllm. Create a conda environment and install vllm to use vllm serve:
pip install vllm>=0.8.5.post1
Start a vllm server follow the official instructions.
Our current implementation support Qwen2.5VL, Pixtral, and granite. You can always add your desired model to run_inf.vllm_inference.
See example for decompose 33% of ChartQAPro data, and corresponding data synthesis and convertion.
see example for decompose 33% of WebGUI data, and corresponding data synthesis. The data convertion works the same as chart.