feat(models): add SigLIP2 embedding wrapper by bogwi · Pull Request #4539 · dimensionalOS/dimos

bogwi · 2026-10-12T01:57:11Z

What is this feature?

SigLIP2Model (dimos/models/embedding/siglip2.py), an embedding wrapper for the fixed-resolution SigLIP 2 checkpoints, default google/siglip2-base-patch16-224. It matches SigLIPModel except on the text side: it pads to 64 tokens explicitly (the Gemma tokenizer has no max length, so padding="max_length" alone pads nothing) and lowercases (SigLIP 2 was trained on lowercased text).

Why do we need this?

SigLIP 2 is the successor of the SigLIP v1 embedder used by the localize stack as well meant for general perception workflows.

After testing, it should retire CLIP from the main call sites where CLIP is used.

How to Test

uv run python -c "from dimos.models.embedding.siglip2 import SigLIP2Model; m = SigLIP2Model(); print(m.embed_text('paper box').vector.shape)"

Which issue(s) does this PR close?

Fixes

Checklist

I have read and approved the CLA.

github-actions

go2 replay realtime (arm64)

Details

Benchmark suite	Current: `17aa2ce`	Previous: `1e7b491`	Ratio
`peak memory`	`1585.949` MB	`1569.52` MB	`1.01`
`peak threads`	`349` threads	`352` threads	`0.99`
`network (transport)`	`2721.677` MB	`2722.016` MB	`1.00`
`disk write`	`1.461` MB	`1.492` MB	`0.98`
`instructions`	`115.533` G	`115.634` G	`1.00`

This comment was automatically generated by workflow using github-action-benchmark.

greptile-apps · 2026-10-12T02:05:39Z

Confidence Score: 4/5

[Medium impact] Adds a new embedding model wrapper for image and text.

Safe to merge with two non-blocking concerns: ignored precision configuration and missing preprocessing regression tests.

Findings

Requested precision is ignored ▶
Text preprocessing lacks tests ▶

T-Rex evidence

Real model dtype comparison script

Executed the head wrapper and a dtype-forwarding control against the cached checkpoint, checking parameter precision, storage, and text inference to isolate candidate 0.

Before and after execution commands

Ran both real-model conditions offline and captured commands, working directories, exit codes, and output for a reproducible comparison.

Output capture script

Executed environment recovery and the comparison runner while saving their actual output with command and exit-code headers.

Prepared Python environment recovery commands

Restored the missing Python 3.12 executable behind the prepared virtualenv without replacing its CPU dependencies, allowing real model validation.

Prepared environment recovery output

Executed the recovery commands and confirmed Python 3.12.15, Torch 2.7.1+cpu, and Transformers 4.53.3 were available for validation.

Verified head source and setup

Captured the repository head, numbered product source, setup instructions, and empty product diff to identify the exact source under test.

Head wrapper ignores requested float16

Loaded the real model through the unchanged wrapper and ran text inference, showing accepted float16 configuration but float32 parameters and twice the expected parameter storage.

Control honors requested float16

Loaded the same checkpoint through a validation-only dtype-forwarding subclass and ran text inference, showing float16 parameters and exactly half the parameter storage.

Complete comparison execution output

Captured the comparison runner's actual commands and outputs, showing the expected failing head check and passing control check.

Product source unchanged after validation

Ran git diff --exit-code against siglip2.py after both checks and received exit code 0, confirming validation did not edit product code.

Isolated coverage inventory and preprocessing harness

The executed harness inventories checked-in tests and compares real-tokenizer preprocessing with isolated negative controls, making the coverage-gap demonstration reproducible.

Executed commands and output capture runner

The executed runner launches inventory and comparison runs with explicit offline environment settings and records each command, working directory and exit code, preserving traceable proof.

Checked-in test inventory with no SigLIP2 references

The inventory enumerated 552 tracked Python test files and extracted existing embedding-test parametrizations and assertions, confirming that SigLIP2 preprocessing lacks checked-in coverage.

Preprocessing results with intentional negative controls

Three runs separately disabled lowercasing, padding or truncation in memory while retaining a mocked fixed-shape encoder, showing that shape-only checks pass despite incorrect preprocessing.

Preprocessing results with unchanged PR code

The actual checked-in embed_text method processed mixed-case, short and long inputs through the real cached tokenizer, passing all three preprocessing checks.

Initial harness failure from incorrect EOS expectation

The initial run captured correct 64-token output with terminal EOS but failed an overly strict harness prefix assertion, documenting a corrected harness error rather than a product defect.

Evidence from the check

Executed the head wrapper and a dtype-forwarding control against the cached checkpoint, checking parameter precision, storage, and text inference to isolate candidate 0.

Evidence from the check

Ran both real-model conditions offline and captured commands, working directories, exit codes, and output for a reproducible comparison.

Evidence from the check

Executed environment recovery and the comparison runner while saving their actual output with command and exit-code headers.

Evidence from the check

Restored the missing Python 3.12 executable behind the prepared virtualenv without replacing its CPU dependencies, allowing real model validation.

Command output from the check

Executed the recovery commands and confirmed Python 3.12.15, Torch 2.7.1+cpu, and Transformers 4.53.3 were available for validation.

Command output from the check

Captured the repository head, numbered product source, setup instructions, and empty product diff to identify the exact source under test.

Command output from the check

Loaded the real model through the unchanged wrapper and ran text inference, showing accepted float16 configuration but float32 parameters and twice the expected parameter storage.

Command output from the check

Loaded the same checkpoint through a validation-only dtype-forwarding subclass and ran text inference, showing float16 parameters and exactly half the parameter storage.

Command output from the check

Captured the comparison runner's actual commands and outputs, showing the expected failing head check and passing control check.

Command output from the check

Ran git diff --exit-code against siglip2.py after both checks and received exit code 0, confirming validation did not edit product code.

Evidence from the check

The executed harness inventories checked-in tests and compares real-tokenizer preprocessing with isolated negative controls, making the coverage-gap demonstration reproducible.

Evidence from the check

The executed runner launches inventory and comparison runs with explicit offline environment settings and records each command, working directory and exit code, preserving traceable proof.

Command output from the check

The inventory enumerated 552 tracked Python test files and extracted existing embedding-test parametrizations and assertions, confirming that SigLIP2 preprocessing lacks checked-in coverage.

Command output from the check

Three runs separately disabled lowercasing, padding or truncation in memory while retaining a mocked fixed-shape encoder, showing that shape-only checks pass despite incorrect preprocessing.

Command output from the check

The actual checked-in embed_text method processed mixed-case, short and long inputs through the real cached tokenizer, passing all three preprocessing checks.

Command output from the check

The initial run captured correct 64-token output with terminal EOS but failed an overly strict harness prefix assertion, documenting a corrected harness error rather than a product defect.

Summary

Adds SigLIP2Model for fixed-resolution SigLIP2 checkpoints.

Callers can create image and text embeddings with SigLIP 2.

No blocking issue was established. The change is safe to merge with these non-blocking concerns.

_{Reviews (1) · Last reviewed commit: "feat(models): add SigLIP2 embedding wrap..." · Reviewed by Greptile}

codecov · 2026-10-12T02:05:46Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
6909	1	6908	110

View the top 1 failed test(s) by shortest run time

dimos.hardware.sensors.lidar.livox.test_e2e::test_live_loopback_handshake_and_stream

Stack Traces | 10.4s run time

spawn = <function spawn.<locals>.factory at 0x3b25865c0>
synth_pcap = PosixPath('.../pytest-1412/livox_e2e0/synth_mid360.pcap')

    @pytest.mark.native_e2e
    def test_live_loopback_handshake_and_stream(spawn: Spawn, synth_pcap: Path) -> None:
        driver_binary = _require_binary("mid360_native")
        virtual_binary = _require_binary("virtual_mid360")
    
        virtual_config = VirtualMid360Config(
            pcap=str(synth_pcap),
            rate=0.5,
            delay=1.0,
            lidar_ip="127.0.0.1",
            host_ip="127.0.0.1",
            mcast_data=SDK_MULTICAST_GROUP,
        )
        virtual_blob = {
            "topics": {},
            "config": virtual_config.to_config_dict(),
            "session": {"id": "pointlio-e2e", "links": []},
        }
        config = Mid360Config(
            host_ip="127.0.0.1",
            lidar_ip="127.0.0.1",
            multicast_ip=None,
            point_format="full",
        )
        driver_blob = {
            "topics": {
                "lidar": "/e2e_live_lidar#sensor_msgs.PointCloud2",
                "imu": "/e2e_live_imu#sensor_msgs.Imu",
            },
            "config": config.to_config_dict(),
            "session": {"id": "pointlio-e2e", "links": []},
        }
    
        spawn(driver_binary, driver_blob)
        spawn(virtual_binary, virtual_blob)
        # Handshake arms the virtual within ~1 s, then 2 s of half-rate stream.
        received = _collect(
            {"/e2e_live_lidar": PointCloud2, "/e2e_live_imu": Imu},
            seconds=10.0,
            enough={"/e2e_live_lidar": 3, "/e2e_live_imu": 120},
        )
    
        clouds = [PointCloud2.lcm_decode(data) for data in received["/e2e_live_lidar"]]
        imus = [Imu.lcm_decode(data) for data in received["/e2e_live_imu"]]
        # Large clouds drop under the default receive buffer, so the IMU packets
        # carry the rate assertion.
>       assert len(clouds) >= 3, f"expected >=3 clouds, got {len(clouds)}"
E       AssertionError: expected >=3 clouds, got 0
E       assert 0 >= 3
E        +  where 0 = len([])

clouds     = []
config     = Mid360Config(rpc_transport=<class 'dimos.protocol.rpc.zenohrpc.ZenohRPC'>, default_rpc_timeout=120.0, rpc_timeouts={'b...a_port=56400, host_cmd_data_port=56101, host_push_msg_port=56201, host_point_data_port=56301, host_imu_data_port=56401)
driver_binary = PosixPath('.../target/release/mid360_native')
driver_blob = {'config': {'cmd_data_port': 56100, 'enable_imu': True, 'frame_id': 'lidar_link', 'frequency': 10.0, ...}, 'session': ..., 'links': []}, 'topics': {'imu': '/e2e_live_imu#sensor_msgs.Imu', 'lidar': '/e2e_live_lidar#sensor_msgs.PointCloud2'}}
imus       = []
received   = {'/e2e_live_imu': [], '/e2e_live_lidar': []}
spawn      = <function spawn.<locals>.factory at 0x3b25865c0>
synth_pcap = PosixPath('.../pytest-1412/livox_e2e0/synth_mid360.pcap')
virtual_binary = PosixPath('.../target/release/virtual_mid360')
virtual_blob = {'config': {'delay': 1.0, 'host_ip': '127.0.0.1', 'lidar_ip': '127.0.0.1', 'lidar_netns': '', ...}, 'session': {'id': 'pointlio-e2e', 'links': []}, 'topics': {}}
virtual_config = VirtualMid360Config(rpc_transport=<class 'dimos.protocol.rpc.zenohrpc.ZenohRPC'>, default_rpc_timeout=120.0, rpc_timeo...27.0.0.1', host_ip='127.0.0.1', lidar_netns='', mcast_data='224.1.1.5', setup_network=True, alias_iface='dimos-mid360')

.../lidar/livox/test_e2e.py:302: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

bogwi requested review from Dreamsorcerer, TomCC7, aclauer, leshy, mustafab0, paul-nechifor and spomichter as code owners October 12, 2026 01:57

github-actions Bot reviewed Oct 12, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 12, 2026

View reviewed changes

Comment thread dimos/models/embedding/siglip2.py Outdated

Comment thread dimos/models/embedding/siglip2.py

github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Oct 12, 2026

feat(models): add SigLIP2 embedding wrapper

17aa2ce

bogwi force-pushed the danvi/feat/add-siglip2 branch from 4a49935 to 17aa2ce Compare October 12, 2026 02:35

github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Oct 12, 2026

Navigation Menu

Repository navigation

feat(models): add SigLIP2 embedding wrapper - #4539

feat(models): add SigLIP2 embedding wrapper#4539
bogwi wants to merge 1 commit into
mainfrom
danvi/feat/add-siglip2

bogwi commented Oct 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot left a comment •

edited

Loading

Uh oh!

greptile-apps Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

bogwi commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this feature?

Why do we need this?

How to Test

Which issue(s) does this PR close?

Checklist

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

go2 replay realtime (arm64)

Uh oh!

greptile-apps Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 4/5

Findings

T-Rex evidence

Summary

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 1 Tests Failed:

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

bogwi commented Oct 12, 2026 •

edited

Loading

github-actions Bot left a comment •

edited

Loading

greptile-apps Bot commented Oct 12, 2026 •

edited

Loading

codecov Bot commented Oct 12, 2026 •

edited

Loading