feat(evals): add four deterministic VQA families by moshiii-wei · Pull Request #4541 · dimensionalOS/dimos

moshiii-wei · 2026-10-12T06:11:47Z

What is this feature?

Adds four image- and LiDAR-backed VQA families on top of existing VQA-Gen (QuestionAuthor → answer_question → cases.jsonl):

vertical_direction — top / middle / bottom from a single detection bbox
presence_and — both object categories detected (reject if either is missing)
object_color — dominant HSV bucket inside one detection crop
camera_bearing — left / ahead / right from LiDAR points in an EdgeTAM mask

New primitives live under dimos/evals/vqa/primitives/{color,bearing}.py. The OpenAI author already receives family metadata via Available families: … JSON; no separate prompt change.

Why do we need this?

Improves coverage toward related-work question types (spatial layout, multi-object presence, color attributes, camera-relative bearing) while keeping deterministic GT. Supports the goal of more families backed by auditable primitives rather than LLM answers.

How to Test

cd dimos
uv run pytest dimos/evals/vqa -q

Optional smoke on a calibrated recording (needs OPENAI_API_KEY for the author):

dimos evals vqa generate /path/to/calibrated-recording.db --start 0 --stop 50 --stride 10

Inspect audit/run.json for new family names and sample cases.jsonl questions.

Which issue(s) does this PR close?

Fixes

Checklist

I have read and approved the CLA.

Made with Cursor

Extend VQA-Gen with vertical_direction, presence_and, object_color, and camera_bearing so more spatial and attribute questions can be proposed by the author and answered from detector, color, and LiDAR primitives.

github-actions

go2 replay realtime (arm64)

Details

Benchmark suite	Current: `e5d619b`	Previous: `1e7b491`	Ratio
`peak memory`	`1592.867` MB	`1569.52` MB	`1.01`
`peak threads`	`344` threads	`352` threads	`0.98`
`network (transport)`	`2730.059` MB	`2722.016` MB	`1.00`
`disk write`	`1.699` MB	`1.492` MB	`1.14`
`instructions`	`115.537` G	`115.634` G	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

codecov · 2026-10-12T06:21:45Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
6255	1	6254	92

View the top 1 failed test(s) by shortest run time

dimos.memory.test_replay_module::test_module_publishes_recorded_messages

Stack Traces | 5.51s run time

recording = '.../popen-gw1/test_module_publishes_recorded0/memory.db'

    def test_module_publishes_recorded_messages(recording: str) -> None:
        cls = replay_module(recording)
        module = cls(dataset=recording)
        got: dict[str, list[float]] = {"odom": [], "goal": []}
        module.outputs["odom"].subscribe(lambda m: got["odom"].append(m.ts))
        module.outputs["goal"].subscribe(lambda m: got["goal"].append(m.ts))
        module.start()
        deadline = time.time() + 5
        while time.time() < deadline and len(got["odom"]) < 3:
            time.sleep(0.01)
        module.stop()
>       assert [round(t, 3) for t in got["odom"]] == [1.0, 1.1, 1.2]
E       assert [1.1, 1.2] == [1.0, 1.1, 1.2]
E         
E         At index 0 diff: #x1B[0m#x1B[94m1.1#x1B[39;49;00m#x1B[90m#x1B[39;49;00m != #x1B[0m#x1B[94m1.0#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         Right contains one more item: #x1B[0m#x1B[94m1.2#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         
E         Full diff:
E         #x1B[0m#x1B[90m #x1B[39;49;00m [#x1B[90m#x1B[39;49;00m
E         #x1B[91m-     1.0,#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
E         #x1B[90m #x1B[39;49;00m     1.1,#x1B[90m#x1B[39;49;00m
E         #x1B[90m #x1B[39;49;00m     1.2,#x1B[90m#x1B[39;49;00m
E         #x1B[90m #x1B[39;49;00m ]#x1B[90m#x1B[39;49;00m

cls        = <class 'dimos.memory.test_replay_module.Replay'>
deadline   = 1791785907.096488
got        = {'goal': [1.05], 'odom': [1.1, 1.199999999]}
module     = <dimos.memory.test_replay_module.Replay object at 0x1748f72d0>
recording  = '.../popen-gw1/test_module_publishes_recorded0/memory.db'

dimos/memory/test_replay_module.py:62: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

greptile-apps · 2026-10-12T06:24:08Z

Confidence Score: 2/5

[Medium impact] Adds four new question families to the evaluation system.

Fix the three color ground-truth defects before merging; the bearing-family availability issue is non-blocking.

Findings

Neutral colors get discarded ▶
Brown and pink get mislabeled ▶
Tied colors get arbitrary answers ▶
Bearing questions stay hidden ▶

T-Rex evidence

Complete executed neutral-color validation script

The authored script exercises five synthetic images through the real primitive, family dispatch, and dataset persistence paths, allowing both observed runs to be reproduced.

Neutral-color results with the filter disabled in memory

The diagnostic control executes the same five images with only the filter disabled and reads back saved labels, showing all neutrals classified and the predominantly white crop labeled white.

Neutral-color results on unmodified PR head

The unmodified head executes the same five images and reads back saved labels, confirming three neutral rejections and an incorrect blue label for the predominantly white crop.

Full executed script and exact production source excerpts

A command captures the complete test source, numbered filter and label-persistence excerpts, head revision, and tracked diff, showing the precise tested logic and no production edits.

Complete executable color reproduction script

The uploaded script constructs real RGB and BGR Image objects, executes the primitive and family dispatcher, and checks the cited hue interval, making the reproduction rerunnable.

Full source of the executed reproduction script

Captured cat output contains the entire executed script with its command, working directory, and exit code, preserving the full test source.

Base revision run with the color API absent

The same script ran against base 1e7b491 and exited 3 because the color module does not exist, establishing the honest pre-PR baseline.

PR revision run showing incorrect brown and pink labels

The script ran against the PR head and exited 1 after six failed color expectations, while controls passed and all 20 cited hues returned brown, confirming candidate1.

Exact source lines and unchanged production files

Executed git and numbered-source commands captured the faulty bucket branches and answer propagation with no tracked diff, linking the observed failures to unchanged PR code.

Complete tied-color reproduction script

The uploaded full source executes the base contract and current primitive, dispatcher, and dataset writer, making the confirmed reproduction repeatable.

Before PR: color proposal rejected

Executed the exact base revision’s proposal contract with object_color and captured its validation rejection, showing the family did not exist before this PR.

After PR: tied colors saved as red

Ran equal red/blue pixels through the real primitive and dataset writer and read back the red label, confirming ambiguous evidence becomes saved ground truth.

Exact source for tie selection and label persistence

Captured numbered source excerpts and git diff --exit-code, locating the unchecked tie and label propagation while confirming no production edits.

Executable bearing service reproduction

The full script executed against both revisions captures offered families and exercises generation plus direct answering, demonstrating the unnecessary range-estimator gate.

Service output before PR4541

Ran the synthetic service reproduction against the PR base and captured source excerpts and responses, showing that camera bearing had not yet been introduced.

Service output after PR4541 with and without a range estimator

Ran the same reproduction against PR head and captured generated datasets and direct answers, confirming that bearing is hidden without an estimator it never uses.

Evidence from the check

The authored script exercises five synthetic images through the real primitive, family dispatch, and dataset persistence paths, allowing both observed runs to be reproduced.

Command output from the check

The diagnostic control executes the same five images with only the filter disabled and reads back saved labels, showing all neutrals classified and the predominantly white crop labeled white.

Command output from the check

The unmodified head executes the same five images and reads back saved labels, confirming three neutral rejections and an incorrect blue label for the predominantly white crop.

Command output from the check

A command captures the complete test source, numbered filter and label-persistence excerpts, head revision, and tracked diff, showing the precise tested logic and no production edits.

Evidence from the check

The uploaded script constructs real RGB and BGR Image objects, executes the primitive and family dispatcher, and checks the cited hue interval, making the reproduction rerunnable.

Command output from the check

Captured cat output contains the entire executed script with its command, working directory, and exit code, preserving the full test source.

Command output from the check

The same script ran against base 1e7b491 and exited 3 because the color module does not exist, establishing the honest pre-PR baseline.

Command output from the check

The script ran against the PR head and exited 1 after six failed color expectations, while controls passed and all 20 cited hues returned brown, confirming candidate1.

Command output from the check

Executed git and numbered-source commands captured the faulty bucket branches and answer propagation with no tracked diff, linking the observed failures to unchanged PR code.

Evidence from the check

The uploaded full source executes the base contract and current primitive, dispatcher, and dataset writer, making the confirmed reproduction repeatable.

Command output from the check

Executed the exact base revision’s proposal contract with object_color and captured its validation rejection, showing the family did not exist before this PR.

Command output from the check

Ran equal red/blue pixels through the real primitive and dataset writer and read back the red label, confirming ambiguous evidence becomes saved ground truth.

Command output from the check

Captured numbered source excerpts and git diff --exit-code, locating the unchecked tie and label propagation while confirming no production edits.

Evidence from the check

The full script executed against both revisions captures offered families and exercises generation plus direct answering, demonstrating the unnecessary range-estimator gate.

Command output from the check

Ran the synthetic service reproduction against the PR base and captured source excerpts and responses, showing that camera bearing had not yet been introduced.

Command output from the check

Ran the same reproduction against PR head and captured generated datasets and direct answers, confirming that bearing is hidden without an estimator it never uses.

Summary

Adds four deterministic VQA families:

Objects are placed in the top, middle, or bottom image region.
Presence questions can check for two object categories at once.
Object color questions return the dominant color in an image crop.
Camera bearing questions place an object left, ahead, or right.

The color answers need fixes before merging. Neutral pixels are discarded, brown and pink are mislabeled, and tied color counts produce arbitrary expected answers. Actual image crops reproduced all three failures through family answering; the neutral-patch and tied-color errors also appeared in saved dataset labels.

Bearing generation additionally requires an unused range estimator for programmatic callers. This is a non-blocking coverage issue; the normal CLI path already supplies the estimator.

The established checks for two-name validation in presence_and and camera-coordinate bearing direction remain intact. The PR is not safe to merge until the three color ground-truth defects are fixed.

_{Reviews (1) · Last reviewed commit: 7e19576 · Reviewed by Greptile}

greptile-apps · 2026-10-12T06:24:11Z

+    valid = (sat >= 40) & (val >= 40)
+    if int(valid.sum()) < _MIN_VALID_PIXELS:
+        raise InsufficientEvidenceError(
+            "object color crop lacks enough saturated, well-lit pixels"
+        )


Neutral colors get discarded

dominant_color_in_bbox removes every pixel that could become white, black, or gray before counting colors. Solid neutral-colored objects are rejected, while a mostly neutral object can receive the color of a small saturated patch. A crop containing 95% white and 5% blue was accepted and saved with blue as its expected answer, corrupting evaluation ground truth. This needs fixing before merging.

Count neutral pixels too, and measure the winning color against the full usable crop.

Artifacts

Complete executed neutral-color validation script

The authored script exercises five synthetic images through the real primitive, family dispatch, and dataset persistence paths, allowing both observed runs to be reproduced.

Neutral-color results with the filter disabled in memory

The diagnostic control executes the same five images with only the filter disabled and reads back saved labels, showing all neutrals classified and the predominantly white crop labeled white.

Neutral-color results on unmodified PR head

The unmodified head executes the same five images and reads back saved labels, confirming three neutral rejections and an incorrect blue label for the predominantly white crop.

Full executed script and exact production source excerpts

A command captures the complete test source, numbered filter and label-persistence excerpts, head revision, and tracked diff, showing the precise tested logic and no production edits.

_{Ran code and verified through T-Rex}

greptile-apps · 2026-10-12T06:24:11Z

+    if hue < 150:
+        return COLOR_CHOICES.index("purple")
+    return COLOR_CHOICES.index("brown")


Brown and pink get mislabeled

_hue_to_bucket treats hues 150–169 as brown, so vivid pink and magenta pixels get labeled brown. Ordinary brown pixels, such as RGB (128, 64, 0), fall in the orange hue range instead. Actual image crops reproduced both errors through answer_question, where these labels become incorrect expected answers. This needs fixing before merging.

Use brightness along with hue to distinguish brown from orange, map pink or magenta to a suitable color group or reject unsupported colors, and add regression tests for both cases.

Artifacts

Complete executable color reproduction script

The uploaded script constructs real RGB and BGR Image objects, executes the primitive and family dispatcher, and checks the cited hue interval, making the reproduction rerunnable.

Full source of the executed reproduction script

Captured cat output contains the entire executed script with its command, working directory, and exit code, preserving the full test source.

Base revision run with the color API absent

The same script ran against base 1e7b491 and exited 3 because the color module does not exist, establishing the honest pre-PR baseline.

PR revision run showing incorrect brown and pink labels

The script ran against the PR head and exited 1 after six failed color expectations, while controls passed and all 20 cited hues returned brown, confirming candidate1.

Exact source lines and unchanged production files

Executed git and numbered-source commands captured the faulty bucket branches and answer propagation with no tracked diff, linking the observed failures to unchanged PR code.

_{Ran code and verified through T-Rex}

greptile-apps · 2026-10-12T06:24:11Z

+    winner = int(counts.argmax())
+    fraction = float(counts[winner] / counts.sum())
+    if fraction < _MIN_BUCKET_FRACTION:


Tied colors get arbitrary answers

counts.argmax() accepts a tied winner because _MIN_BUCKET_FRACTION is only 0.4. A crop split equally between red and blue passes with a fraction of 0.5 and saves red solely because it comes first in COLOR_CHOICES. Dataset generation reproduced this arbitrary expected answer; a model choosing the equally common blue would be marked wrong. This needs fixing before merging.

Reject ties and require a clear lead over the runner-up before saving a single dominant color.

Artifacts

Complete tied-color reproduction script

The uploaded full source executes the base contract and current primitive, dispatcher, and dataset writer, making the confirmed reproduction repeatable.

Before PR: color proposal rejected

Executed the exact base revision’s proposal contract with object_color and captured its validation rejection, showing the family did not exist before this PR.

After PR: tied colors saved as red

Ran equal red/blue pixels through the real primitive and dataset writer and read back the red label, confirming ambiguous evidence becomes saved ground truth.

Exact source for tie selection and label persistence

Captured numbered source excerpts and git diff --exit-code, locating the unchecked tie and label propagation while confirming no production edits.

_{Ran code and verified through T-Rex}

greptile-apps · 2026-10-12T06:24:11Z

+    requires_pointcloud=True,
+    requires_masks=True,


Bearing questions stay hidden

CAMERA_BEARING_FAMILY sets requires_pointcloud=True, but the generator uses that flag to require a range_estimator too. A caller of generate_frames_dataset with a point cloud and mask_estimator, leaving the optional range_estimator=None, never gets camera_bearing offered even though its answer needs no range estimator. This is a non-blocking loss of coverage for programmatic callers; the CLI is unaffected because it already provides one.

Separate the point-cloud requirement from the range-estimator requirement and cover this combination in a generation test.

Artifacts

Executable bearing service reproduction

The full script executed against both revisions captures offered families and exercises generation plus direct answering, demonstrating the unnecessary range-estimator gate.

Service output before PR4541

Ran the synthetic service reproduction against the PR base and captured source excerpts and responses, showing that camera bearing had not yet been introduced.

Service output after PR4541 with and without a range estimator

Ran the same reproduction against PR head and captured generated datasets and direct answers, confirming that bearing is hidden without an estimator it never uses.

_{Ran code and verified through T-Rex}

moshiii-wei requested review from Dreamsorcerer, TomCC7, aclauer, leshy, mustafab0, paul-nechifor and spomichter as code owners October 12, 2026 06:11

feat(evals): add four deterministic VQA families

da23e93

Extend VQA-Gen with vertical_direction, presence_and, object_color, and camera_bearing so more spatial and attribute questions can be proposed by the author and answered from detector, color, and LiDAR primitives.

moshiii-wei force-pushed the moshi/feat/vqa-four-families branch from 7e19576 to a0d8dc5 Compare October 12, 2026 06:11

github-actions Bot added the first-time-contributor PR opened by an author who had not previously committed to this repository label Oct 12, 2026

moshiii-wei force-pushed the moshi/feat/vqa-four-families branch from a0d8dc5 to da23e93 Compare October 12, 2026 06:12

[autofix.ci] apply automated fixes

e5d619b

github-actions Bot reviewed Oct 12, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 12, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat(evals): add four deterministic VQA families - #4541

feat(evals): add four deterministic VQA families#4541
moshiii-wei wants to merge 2 commits into
mainfrom
moshi/feat/vqa-four-families

moshiii-wei commented Oct 12, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

codecov Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Oct 12, 2026

Uh oh!

greptile-apps Bot Oct 12, 2026

Uh oh!

greptile-apps Bot Oct 12, 2026

Uh oh!

greptile-apps Bot Oct 12, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

moshiii-wei commented Oct 12, 2026

What is this feature?

Why do we need this?

How to Test

Which issue(s) does this PR close?

Checklist

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

go2 replay realtime (arm64)

Uh oh!

codecov Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 1 Tests Failed:

Uh oh!

greptile-apps Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 2/5

Findings

T-Rex evidence

Summary

Uh oh!

greptile-apps Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codecov Bot commented Oct 12, 2026 •

edited

Loading

greptile-apps Bot commented Oct 12, 2026 •

edited

Loading