mlx-swift-ZipSplat

Apple Silicon port of ZipSplat (Veicht, Hong, Barát, Pollefeys — ETH Zürich / Microsoft), built on mlx-swift.

Feed-forward 3D Gaussian Splatting: a handful of unposed photos in, a 3DGS scene out, in one forward pass. No poses, no intrinsics, no per-scene optimisation.

📖 API documentation · 🤗 weights

Five photos of an office reconstructed into 51,840 Gaussians, at default settings. The compression slider re-derives the scene in ~20 ms, so it can be dragged live.

Three products:

• MLXZipSplat — the model, a ZipSplatSession driver, and 3DGS .ply export.
• zipsplat-tool — CLI: images or a video clip → .ply, plus a leak-watching benchmark.
• Examples/ZipSplatDemo — SwiftUI macOS app: live view/frame count, compression slider, spherical-harmonics and token-colouring toggles, .ply export. Renders through Satin-Spark.

The DA3 ViT-g backbone comes from mlx-swift-da3 rather than being re-ported: ZipSplat's backbone is a trimmed copy of the same network.

All dependencies resolve from GitHub — mlx-swift, swift-argument-parser, and mlx-swift-da3 for the package; Satin-Spark 0.1.7+ additionally for the example app. A fresh clone builds without any sibling checkouts.

• macOS 15+, Apple Silicon
• Xcode 16+
• ~3 GB of disk for the converted weights

Easiest: launch the viewer and press Download Weights — it fetches the converted checkpoint (2.90 GB) from mnmly/zipsplat-mlx, resuming if interrupted. Or fetch it directly:

mkdir -p ~/models/zipsplat && curl -L -o ~/models/zipsplat/zipsplat-da3g-252p-f16.safetensors \
  https://huggingface.co/mnmly/zipsplat-mlx/resolve/main/zipsplat-da3g-252p-f16.safetensors

Licence. The ZipSplat code is Apache-2.0 but the weights are CC BY-NC 4.0 — non-commercial use only, inherited from DA3-Giant and DL3DV-10K. This applies to the converted copy too.

# download (5.79 GB)
mkdir -p ~/.cache/torch/hub/zipsplat
curl -L -o ~/.cache/torch/hub/zipsplat/zipsplat-da3g-252p.tar \
  https://huggingface.co/veichta/zipsplat/resolve/main/zipsplat-da3g-252p.tar

# convert
uv venv && uv pip install torch numpy einops safetensors
.venv/bin/python Scripts/convert_weights.py \
  --input  ~/.cache/torch/hub/zipsplat/zipsplat-da3g-252p.tar \
  --output ~/models/zipsplat/zipsplat-da3g-252p-f16.safetensors

~/models/zipsplat/zipsplat-da3g-252p-f16.safetensors is the default lookup path; override it with ZIPSPLAT_WEIGHTS or --weights.

xcodebuild -scheme zipsplat-tool -configuration release \
  -destination 'platform=macOS' -derivedDataPath .xcdd build

Use xcodebuild, not swift build/swift test — the latter can't load mlx-swift's default.metallib at runtime.

The built binary lands at .xcdd/Build/Products/release/zipsplat-tool; shown here as zipsplat-tool for brevity.

# a folder of images, a single image, or a video clip
zipsplat-tool reconstruct path/to/images -o scene.ply
zipsplat-tool reconstruct clip.mp4 --num-frames 12 -o scene.ply
zipsplat-tool reconstruct path/to/images \
  --compression 0.3 --views 8 --color-by-token -o compact.ply

Video frames are sampled evenly across the clip rather than taken from the start, which gives the widest camera baseline. Rotation metadata is honoured, and seeking uses zero tolerance so several requested times cannot collapse onto the same keyframe and quietly destroy the baseline.

scene.ply is a standard 3DGS file — open it in SuperSplat, Satin-Spark, or any 3DGS viewer.

import MLXZipSplat

let session = try ZipSplatSession(weights: weightsURL)
let images = try urls.map { try ImagePreprocessing.loadImage($0) }

session.loadViews(images)                                  // ~0.2 s for 5 views
let gaussians = session.gaussians(compression: 1.0)[0]     // ~0.02 s
try gaussians.writePLY(to: outputURL)

The split matters: the backbone, prepare, and the colour patch embed are all independent of compression, so loadViews caches them and gaussians(compression:) re-runs only the cheap tail. The Python reference viewer re-runs the entire 40-layer ViT-g on every slider tick; this doesn't. SessionTests.testCachedTailMatchesFullForward asserts the cached path is numerically identical to a full forward pass, so it's an optimisation, not an approximation.

open Examples/ZipSplatDemo/ZipSplatDemo.xcodeproj

See Examples/ZipSplatDemo/README.md.

Two knobs, and they are not interchangeable:

Gaussians = floor(views × 324 × compression) × 32

• View count (or frame count, for a clip) sets the ceiling — each 252 px view contributes 324 scene tokens, so one more view is +10,368 Gaussians and a wider baseline (better coverage, not just density). Changing it invalidates the cached backbone features, so it costs a full re-run (~0.2 s). The app samples views evenly across the selection rather than taking the first N, which gives the widest baseline for a walk-through or orbit.
• Compression scales down from that ceiling and re-runs only the cheap tail (~0.02 s), which is why it can be dragged live.

There is no way to exceed views × 324 × 32: gaussiansPerToken is baked into the trained checkpoint. And there is no epoch/iteration control — ZipSplat is feed-forward, with no per-scene optimisation by design.

M-series, Release build, 5 views at 252 px, fp16 (zipsplat-tool bench):

activeMemory stays flat at 4.4–4.5 GB across iterations — no leak. The ~5.2 GB cacheMemory is MLX's reusable buffer pool, and peak footprint is 6.1 GB.

A speed comparison against the reference is not meaningful here: ZipSplat's published numbers are CUDA, and the reference cannot run its renderer on macOS at all.

Parity is established at three levels, bisecting upward from a trusted input.

1. Per-stage fixtures (ParityTests, EmbeddingParityTests). Scripts/dump_fixtures.py runs the reference stage-by-stage and records whole-tensor statistics plus a deterministic slice at every boundary: patch embed → interpolated pos-embed → post-embedding tokens → backbone layers 39/29/19 → prepare → fuse → colour → head. The final Gaussians are stored in full. Fixtures are generated without xformers, because the Swift port mirrors the pure-PyTorch SwiGLUFFN fallback.

2. End-to-end .ply (Scripts/compare_ply.py). The real oracle — every Gaussian parameter through the full model plus the export conventions:

(f_dc, f_rest and opacity are stored in transformed spaces — SH and logit — where values pass through zero, so relative error is inflated on near-zero entries. Correlation and the render check below are the meaningful readings.)

3. Render parity (Scripts/render_compare.py). Both scenes rendered from identical novel views through gsplat's reference CUDA rasteriser: mean PSNR 44.83 dB, worst 43.34 dB over 8 orbit views. For scale, the model's own eval PSNR against ground truth is 21.77 dB, so the port's deviation sits ~24 dB below the model's own error. Requires a CUDA box.

Against CUDA specifically. Fixtures are generated with PyTorch on CPU, so the reference was also run on an RTX 3080 to check that nothing hides in that gap:

The reference is effectively device-independent here (TF32 is off by default in the pinned torch), so comparing against CPU and against CUDA give the same answer.

Earlier revisions of this table quoted 46.09 dB mean / 38.21 dB worst. Those were measured with a hardcoded orbit radius of 2.0 against a scene whose p95 extent is 2.25 — i.e. with the camera inside the Gaussian cloud, where a splat centimetres from the lens covers the frame and near-plane culling flips on a ~0.001 position change. render_compare.py now derives the radius from the scene extent; the figures above are from outside the cloud and are the ones to trust. Note the port's deviation is about 1200x larger than that device noise — it is real, not measurement error, and comes from fp16 weights plus MLX/PyTorch op-ordering differences. The render check above is what bounds it perceptually.

Alongside those: SessionTests asserts the cached-tail path matches a full forward pass and that the .ply round-trips; PreprocessingTests guards the resize and colour-space handling.

xcodebuild -scheme MLXZipSplat-Package -destination 'platform=macOS' test   # 11 tests

Colour by token — each scene token's group of 32 Gaussians in a distinct colour, showing how ZipSplat decouples placement from the pixel grid.

Requires Satin-Spark 0.1.7 or later, which fixed a degree-1 SH bit-field decode bug that dropped all but the first coefficient of each basis function — only the red channel survived, rendering every SH-lit splat with heavy false iridescence. The View-dependent colour toggle switches the degree-1 term off if you want DC-only.

ZipSplat's SH coefficients reach about ±6 against a packed-format default range of ±1, so the bridge fits the range per scene with Satin-Spark's SplatSHRangeFitter. It uses .quantile(1.0) (fit to the maximum) rather than the .standard 99.5th percentile: .standard has the better mean error but still clips the top 0.5%, and clipping is per channel, so those are exactly the coefficients that shift hue. On the rendered frame the two differ by a mean of 0.073/255 but up to 78 levels on 0.845% of pixels, hue-dominated.

Run ZIPSPLAT_VERIFY_BRIDGE=1 to check the bridge against Satin-Spark's own SplatPLYLoader: centre/scale/colour/rotation exact, opacity within one 1/255 step, 0 of 103,680 SH words differing under matched encodings. It also verifies that the Y-up frame conversion rotates the SH basis correctly — flipped coefficients evaluated at the flipped view direction reproduce the unflipped colour to 0.000000 over 20,000 samples.

• Preprocessing. The reference resizes with PIL LANCZOS. ImagePreprocessing reimplements PIL's 8-bit pipeline, including precompute_coeffs tap rounding and the clip8 round-and-clamp between passes. Agreement is max 1.00 / mean 0.05 uint8 levels — one quantisation step, the floor for a uint8 pipeline. Two things were needed to get there: decoding in the image's own colour space (the example photos are Display P3; drawing into DeviceRGB colour-manages them and shifts saturated colours by up to 60/255), and the inter-pass clamp (without it, Lanczos ringing at speculars differs by >30/255).
• xformers. The released checkpoint was trained with xformers' fused SwiGLU kernel, which is CUDA-only. Both this port and the fixtures use the pure-PyTorch fallback, so they agree with each other; a small drift against the as-trained kernel is unavoidable either way.
• Clustering layer. The reference clusters on the deepest prepared layer as of commit e1b592d ("Fix layer selection for clustering"), which changed it from the middle layer. The checkpoint's own training config records clustering_layer: 1 (the middle layer). This port follows the current reference code and exposes ZipSplatConfiguration/clusteringLayer. It only affects compression < 1.0.
• k-means. Deterministic (linspace init, no RNG) and bit-identical run to run, but argmin tie-breaking against PyTorch is not guaranteed, and the reference runs it under bf16 autocast. compression = 1.0 bypasses k-means entirely and is the exactly-comparable path.
• Reference-view selection. MLXDA3's ViT reorders views when no camera token is supplied and S >= 3; ZipSplat's vendored ViT has no such step. The port passes refViewStrategy: .first, which ParityTests.testReferenceViewSelectionIsIdentity asserts is an identity permutation.

Published at https://mnmly.github.io/mlx-swift-ZipSplat/, rebuilt by .github/workflows/docs.yml on every push to main. To build locally:

BUILD_DOC=1 Scripts/build_docs.sh          # static site into docs/
BUILD_DOC=1 Scripts/build_docs.sh preview  # local preview server

This port follows the upstream licence of ZipSplat. The vendored backbone lineage is DINOv2 (© Meta Platforms, Apache-2.0) and Depth Anything 3 (© 2025 ByteDance, Apache-2.0).