Skip to content

Add AnyCalib single-image camera calibration - #4682

Open
ahojnnes wants to merge 23 commits into
mainfrom
user/jsch/anycalib-calibration
Open

ahojnnes wants to merge 23 commits into
mainfrom
user/jsch/anycalib-calibration

Conversation

@ahojnnes

@ahojnnes ahojnnes commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Summary

New src/colmap/calibration module for single-image camera calibration:

  • AnyCalib ONNX backend (anycalib.h/cc): runs the exported AnyCalib network (DINOv2 + DPT decoder) to regress dense per-pixel camera rays, then fits intrinsics for the requested perspective model.
  • Ray fitting (ray_fitting.h/cc): closed-form pinhole/radial initialization (ported from upstream AnyCalib) plus Ceres refinement of pixel residuals for any perspective COLMAP model.
  • Calibrator factory (calibrator.h/cc, resources.h): backend selection with downloadable model resources.
  • Controller + CLI: new camera_calibration controller wired into automatic reconstruction, option manager, and colmap CLI.
  • ONNX export script: scripts/anycalib/export_onnx.py exports stage 1 to ONNX (fixed 322x322 input).
  • Refactor: moves ONNX helpers from feature/onnx_utils.* to util/onnx.*; new code uses img_points/cam_rays naming consistent with the rest of the codebase.

Benchmark: ETH3D / DSLR, uncalibrated

Reconstruction benchmark A/B with and without learned calibration, for both the
incremental and global mappers (no GT priors, SIFT, high quality, seed 0,
single-threaded scenes, deterministic/paired; relative-pose AUC in %). Global
runs reuse the incremental runs' features + raw matches and only re-verify with
the automatic reconstructor's global two-view settings (max_error 1.0, min
inliers 30, min inlier ratio 0.25, no guided matching).

Undistorted (13 scenes, 454 images)

Run AUC@0.5° AUC@1° AUC@5° AUC@10° reg.
incr base 36.71 59.76 88.76 93.61 453/454
incr +AnyCalib 37.86 (+1.14) 60.98 (+1.22) 89.59 (+0.83) 94.27 (+0.66) 453/454
global base 34.62 56.38 84.43 89.00 454/454
global +AnyCalib 36.47 (+1.84) 59.46 (+3.09) 87.60 (+3.17) 92.33 (+3.33) 454/454

Per-scene average gains: incremental +2.42/+1.84/+1.52/+1.21; global
+3.02/+3.10/+1.98/+2.34. Global+AnyCalib nearly reaches incremental-baseline
accuracy.

Per-scene deltas (calibrated − baseline, AUC pp @1°). Incremental: pipes +9.2,
office +7.7 (also 2→1 components), terrace +7.2, relief +7.0, courtyard +4.0,
relief_2 +2.3, kicker +2.0, facade +0.6, delivery_area +0.3, terrains −0.2,
electro −1.0, playground −3.4, meadow −11.8. Global: relief +14.3, relief_2
+9.5, kicker +7.9, terrace +6.1, delivery_area +5.0, terrains +4.5, pipes +3.8,
facade +2.7, office +2.7, playground +2.2, courtyard −0.7, electro −5.3,
meadow −12.5.

Caveats: meadow regresses in both mappers (consistent treatment effect on that
15-image scene); single seed, but deterministic, so deltas are pure treatment
effects. Calibration cost is negligible vs. matching (~90s for 14 images,
single-threaded CPU).

Distorted, train scenes (13 scenes, 454 images)

Same 4-way setup on eth3d-distorted with fresh extraction/matching
(SIMPLE_RADIAL). Note the distorted JPEGs do carry focal EXIF (20mm →
f=3359.7 prior, 2% off GT), so the baseline starts from a decent focal prior
with k=0 and the A/B tests focal refinement + distortion estimation.

Run AUC@0.5° AUC@1° AUC@5° AUC@10° reg.
incr base 10.50 25.92 68.49 81.33 452/454
incr +AnyCalib 13.74 (+3.25) 30.80 (+4.88) 72.96 (+4.46) 84.52 (+3.18) 451/454
global base 10.91 27.03 72.64 84.46 454/454
global +AnyCalib 13.30 (+2.39) 30.19 (+3.16) 74.53 (+1.89) 85.27 (+0.80) 453/454

Per-scene average gains: incremental +1.38/+1.48/+0.53/−0.54; global
+2.09/+3.26/+2.39/+1.38. Calibration helps more on distorted than undistorted,
but absolute levels stay far below undistorted (SIMPLE_RADIAL underfits the
true distortion).

Per-scene deltas (calibrated − baseline, AUC pp @1°). Incremental: facade
+14.1, courtyard +7.2, electro +6.6, relief +4.7, terrace +3.4, kicker +3.4,
relief_2 +1.3, terrains +1.1, delivery_area +0.6, office −0.0 (loses one
image), meadow −1.0 (−20 @5°), pipes −3.6, playground −18.5. Global: terrace
+7.7, playground +7.5, relief +6.1, electro +5.8, relief_2 +4.5, office +3.5,
facade +3.3, pipes +2.1, courtyard +1.6, meadow +0.7, terrains +0.7, kicker
+0.5, delivery_area −1.6 (electro loses one image).

Takeaway: incremental gains are volatile (playground/meadow collapse), while
global gains are uniform (12/13 scenes improve). Global+AnyCalib ≈
incremental+AnyCalib overall (30.19 vs 30.80 @1°).

Distorted with THIN_PRISM_FISHEYE, train scenes (13 scenes, 454 images)

Same 4-way setup reusing the distorted run databases (no re-extraction or
re-matching): base arms convert the SIMPLE_RADIAL cameras to
THIN_PRISM_FISHEYE preserving EXIF focal/principal point/k1, calib arms fit
AnyCalib directly to THIN_PRISM_FISHEYE; two-view geometries re-verified
(incremental defaults / global settings), then sparse-only mapping. Note both
arms lack redone guided matching (requires re-matching), so absolute numbers
slightly understate from-scratch runs.

Run AUC@0.5° AUC@1° AUC@5° AUC@10° reg.
incr base 44.84 66.20 89.33 92.73 445/454
incr +AnyCalib 45.67 (+0.83) 66.97 (+0.77) 89.81 (+0.48) 93.13 (+0.40) 445/454
global base 41.24 62.72 87.99 92.32 449/454
global +AnyCalib 45.04 (+3.80) 67.04 (+4.32) 91.19 (+3.20) 94.77 (+2.45) 453/454

Per-scene average gains: incremental +1.77/+1.56/+1.00/+0.88; global
+4.84/+4.56/+2.63/+1.79.

Per-scene deltas (calibrated − baseline, AUC pp @1°). Incremental: pipes +4.5,
terrace +3.1, kicker +2.9, relief_2 +3.0, relief +2.2, courtyard +2.5, office
+2.6, electro +2.5, terrains +1.3, facade −0.1, meadow −0.4, delivery_area −0.6,
playground −3.2. Global: playground +30.2, terrains +9.6, pipes +5.4, terrace
+4.1, electro +3.7, relief_2 +2.8, relief +2.1, courtyard +2.1, office +1.4,
kicker +0.9, delivery_area +0.3, facade +0.2, meadow −3.6 (but +5 registered
images, split over 2 components).

Takeaway: the model dominates (TPF base triples the SIMPLE_RADIAL baseline);
with the right model, incremental self-calibration leaves little headroom
(+0.77 @1°), while global still gains +4.32 @1° with playground +30.2 as the
standout.

Test plan

  • ctest -R \"calibration/(ray_fitting_test|anycalib_test|calibrator_test)|controllers/camera_calibration_test|util/onnx_test\": 5/5 pass.
  • colmap_feature_{aliked,loma,onnx_matchers}_test targets build (ONNX include consumers).
  • clang-format --dry-run --Werror clean on all changed C++ files.

@ahojnnes
ahojnnes marked this pull request as ready for review September 8, 2026 11:15

@B1ueber2y B1ueber2y left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the exciting efforts! I tried my best to walk through this gigantic PR.

Comment thread doc/cli.rst Outdated
Comment thread src/colmap/exe/sfm.cc
Comment thread src/colmap/calibration/ray_fitting.h Outdated
@B1ueber2y

Copy link
Copy Markdown
Contributor

cc: @javrtg

ahojnnes and others added 22 commits September 19, 2026 08:29
New src/colmap/calibration module with an AnyCalib ONNX backend that
regresses dense per-pixel camera rays, closed-form pinhole/radial
initialization plus Ceres refinement fitting intrinsics for any
perspective model (ray_fitting), and a calibrator factory with
downloadable model resources.

Adds a camera_calibration controller with automatic reconstruction
integration, a CLI command, and an ONNX export script. Moves the ONNX
helpers from feature/ to util/ as onnx.h/cc and uses img_points/cam_rays
naming throughout the new code.
Add --calibration/--calibration_model_path to evaluate.py, run calibration
in the extraction step only, and record calibration settings in the report
metadata.
- Fix wrong attribute name in the benchmark tests, which referenced
  `args.calibration` instead of `args.camera_calibration` and therefore
  failed, including the pre-existing panorama test.
- Create the calibrator lazily when the controller runs, mirroring the
  feature extraction and matching controllers, so that the network is not
  held in device memory during preceding pipeline stages.
- Skip images whose dimensions do not match their database camera, which
  would otherwise store focal length and principal point at the wrong
  scale, since only the fitted parameters are written back.
- Skip learned calibration in the automatic reconstruction pipeline if
  explicit camera parameters were provided, which the image reader writes
  to the database and which must not be overwritten.
- Re-validate the aggregated calibration and fall back to the closest
  single-image calibration, because the coefficient-wise median of
  individually valid calibrations is not itself guaranteed to be
  projection-stable for high-order distortion models.
- Reject implausible per-image predictions via HasBogusParams, with the
  bounds exposed as backend-agnostic calibration options.
- Define backend-specific options via shared_ptr, as for feature
  extraction.
- Reuse Median from colmap/math and remove the dead
  ReverseScaleAndShiftParams.
- Share IsValidCalibration between the per-image and the aggregation path,
  which previously used two divergent copies, so that neither applied the
  full validation. NOTE: per-image predictions are now also checked for
  projection stability.
- Pass the target camera model to AggregateCameraCalibrations explicitly,
  instead of requiring the caller to set it beforehand, and validate on a
  candidate, so that the camera is only modified on success.
- Compute the closest calibration in a single pass over the deviations.
- Use Camera::CreateFromModelId in the tests and merge duplicated cases.
The calibrator is created lazily in the controller's worker thread, where
an escaping exception terminates the whole process instead of failing the
stage. Previously it was created in the constructor on the main thread,
where it propagated normally. A corrupt or missing model file therefore
aborted the automatic reconstruction pipeline with SIGABRT after feature
extraction had already run.
Ceres terminates without taking a step when the closed-form init is already at the optimum (noise-free pinhole data), reporting bit-identical initial/final costs and breaking the strict < assertion on some toolchains. This matches the product's own final <= initial success contract.
Harden validation (non-throwing Check, IsValidCalibration edge cases,
tighter round-trip tolerance), validate ONNX dtypes in AnyCalib/ALIKED/LoMa
via new element-type accessors, harden ray fitting (LDLT definiteness,
inverse-focal epsilon, FOV bounds, invalid-prior tolerance), make the
calibration controller non-fatal, verify export weights hash and record
provenance, and extend tests and docs.
Extend the linear solvers in ray_fitting.cc with a fisheye variant that
fits the equidistant (theta) projection instead of X/Z, and dispatch
radial-fisheye models to it for their leading k1[, k2] coefficients.
Undistorted fisheye models use the equidistant pinhole fit. The FOV gate
uses an incidence-angle check for fisheye rays.

HasRadialDistortionPrefix now returns optional<pair<int, bool>> with the
coefficient count and fisheye flag instead of pointer out-params.

Tests: wide-FOV (~170 deg) round-trip cases for all fisheye models, plus
an init-only test (zero refinement iterations) pinning the closed-form
solution.
InitializeCameraParams now uses naive initialization (focal length from
the image span, principal point at the data center, zero distortion)
instead of the AnyCalib-style closed-form linear solvers. Stress tests
over strong distortion (incl. near-pole division), noise, sparsity,
partial coverage, and outliers showed the Ceres refinement reaches the
same optimum from the naive start every time, so the linear init only
saved iterations at the cost of ~270 lines of solver machinery.

Removes the now-unused max_fov_deg option (incl. the
CameraCalibration.max_fov_deg CLI flag). Tests: drop the obsolete
init-only test; FULL_OPENCV round-trip skips distortion-value assertions
since naive-start refinement may land on a projection-equivalent point
in the weakly identifiable subspace (projection check still binds).
- Widen before multiplication in ray index computation
- Add default cases to rot90 switches and drop const blocking move
- Keep intentional integer crop-offset division out of float context
- NOLINT intentional out-of-range enum cast and getenv in tests
- Wire --image_list_path through the camera_calibrator CLI and pass the selected image names to the calibration controller, as documented.
- Default CameraCalibration.camera_model to empty to preserve each database camera's existing model; an explicit value still converts all cameras. Reject invalid models gracefully before inference, since the camera-model switches throw on them.
- Standardize the new calibration files on SPDX license headers.
@ahojnnes
ahojnnes force-pushed the user/jsch/anycalib-calibration branch from 3b44352 to a90ab99 Compare September 20, 2026 05:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants