Correct predictions
GT: radio | Pred: radio
We introduce , an any-to-any model that jointly processes a diverse set of 1D, 2D and 3D modalities. can predict any modality conditioned on any one (or multiple) other modalities, enabling cross-modality prediction across dimensionalities. The learned multimodal representations show strong performance on downstream tasks requiring both 2D and 3D understanding, including 3D pose prediction, grasp proposal, and object classification.
We present , an Omni-Dimensional Multimodal Foundation Model capable of any-to-any modelling across a diverse set of 1D, 2D and 3D modalities.
Our method enables any-to-any prediction across:
This any-to-any prediction is enabled by a suite of novel tokenizers we delevop for 3D modalities, which enable masked modeling of 3D modalities in the same space as existing 1D and 2D modalities.
Our model is capable of a vast range of tasks out of the box beyond just the standard tasks of 3D generation from images. can perform generation steered by coarse input modalities, decompose objects into parts, retrieve views and objects and much more without any additional training.
Our model can generate can generate any modality of any dimensionality from any other modality of any dimensionality, 1D, multi-view 2D or 3D.
Hover any prediction cell to inspect samples. Click a cell to pin the pop-up panel.
By unifying the representation of our data into a shared token space, we can make any-to-any prediction for any input and output tractable regardless of data dimensionality or modality. As tokenizers do not exist for many 3D modalities, we develop our own novel tokenizer architectures for 7 3D modalities.
NeRF weights and part features can be represented as triplanes of features, each consisting of 3 dense multi-channel 2D feature tensors. We tokenize these triplanes using a modality-specific ViT-based VQ-VAE tokenizer.
3DGSs can be decoded from a sparse 3D SLAT which encodes high-fidelity 3D details. We densify SLATs via zero padding to obtain a dense 3D grid. Voxels, UDFs and NeRF grids are cast as dense 3D grids representing occupancy, distance to surface and RGB/density respectively, tokenized with a 3D convolutional VQ-VAE.
2D modalities are obtained from renders and tokenized with pre-existing tokenizers, with finetuning when necessary. We extract camera plucker rays from the rendering pipeline to help with modelling.
is a 1 billion parameter transformer encoder-decoder. To perform joint learning on a range of 1D, 2D and 3D modalities, we optimize with the multimodal masked modeling objective introduced in 4M. The tokenized modalities are augmented with a positional embedding corresponding to their structure. We use 2D sin-cos embeddings for 3D modalities cast as dense 2D, 3D sin-cos embeddings for 3D modalities cast as dense 3D and encoded plucker rays are used for multiview 2D modalities.
Our model can accept 2D modality inputs such as RGB images, depth maps, or edges and peform novel view synthesis in the space of that same modality, or any other modality. At any point, these 2D views can be translated into a 3D representation of any kind.
can also accept 3D modalities as inputs, such as NeRFs, meshes or voxels. This enables tasks such as rendering any 2D modality from a 3D input, or texturing a 3D model when combined with a 1D or 2D modality input such as RGB.
By conditioning generation on a combination of modalities, can perform generation from a coarse modality such as voxels or masks steered by a conditioning modality such as text or metadata. The final generation can be any modality of any dimensionality.
PartField features can determine object parts at multiple levels of granularity. We generate these features and use them to split generated 3D models into their constituent parts.
RGB input
3D model
Part features
Exploded model
Hint: Drag the left slider to switch example and the middle slider to change the number of parts.
Our model can use any modality as a query to generate DINOv2 and ULIP-2 features, which can be used for retrieval across modalities and dimensions.
Input: RGB
RGB retrieval (Top-3)
Hint: Use the arrows on the input to move the view. Drag the slider to change the retrieved modality.
RGB input
Mesh retrieval (Top-3)
Hint: Drag the slider to switch the retrieved 3D modality.
DINOv2 features can be used to query for similar views across modalities; use the arrows to move the query view relative to the current orientation.
ULIP features can be used to query for similar objects across modalities, independent of view.
generates geometrically consistent 3D outputs from a single view or a set of multiple views, matching or outperforming existing specialized methods.
Input
SAM3D
TRELLIS
Ours
GT
transfers multimodal representations to downstream tasks; rotate through the panels to compare qualitative and quantitative results.
| Method | NOCS MAE | NOCS PSNR | Mask IoU (%) |
|---|