feat: add RoboSpatial task#1347
Merged
Merged
Conversation
RoboSpatial is a spatial-reasoning benchmark for robotic manipulation scenes (RoboSpatial-Home) covering three sub-categories: compatibility, configuration, and context. Dataset: chanhee-luke/RoboSpatial-Home on HuggingFace. Per-category splits: compatibility (105), configuration (123), context (122) (350 items total). This port exposes: - robo_spatial (group) - robo_spatial_all (union of all three splits via dataset_kwargs.data_files) - robo_spatial_compatibility / robo_spatial_configuration / robo_spatial_context Metric: robo_spatial_score — task-specific scoring implemented in utils.py (point/region/affordance correctness; see pre_process.py for parsing).
kcz358
reviewed
May 22, 2026
Collaborator
There was a problem hiding this comment.
This one seems to be a bit redundant if configure the all split as an individual yaml
Address reviewer feedback on EvolvingLMMs-Lab#1347: the `robo_spatial` group only listed `robo_spatial_all`, which is already a standalone task and can be invoked directly via --tasks robo_spatial_all.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds RoboSpatial, a spatial-reasoning benchmark for robotic manipulation scenes (RoboSpatial-Home) covering three sub-categories:
Total: 350 items.
This port exposes:
Metric: `robo_spatial_score` — task-specific scoring (point / region / affordance correctness; see `pre_process.py` for parsing).
Files
Parity vs. local fork
Qwen3-VL-2B-Instruct, full test split on 8x H100, greedy decoding.
Per-doc analysis on the 309 shared questions matched by doc_id: 91.9% identical scores.
Delta (+2.6pp overall) is consistent with the qwen3_vl model-class drift we have observed on other ports (e.g. metavqa, egoplan2).
Test plan