Conversation
ahojnnes
marked this pull request as ready for review
September 8, 2026 11:15
B1ueber2y
reviewed
Sep 14, 2026
B1ueber2y
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the exciting efforts! I tried my best to walk through this gigantic PR.
Contributor
|
cc: @javrtg |
New src/colmap/calibration module with an AnyCalib ONNX backend that regresses dense per-pixel camera rays, closed-form pinhole/radial initialization plus Ceres refinement fitting intrinsics for any perspective model (ray_fitting), and a calibrator factory with downloadable model resources. Adds a camera_calibration controller with automatic reconstruction integration, a CLI command, and an ONNX export script. Moves the ONNX helpers from feature/ to util/ as onnx.h/cc and uses img_points/cam_rays naming throughout the new code.
Add --calibration/--calibration_model_path to evaluate.py, run calibration in the extraction step only, and record calibration settings in the report metadata.
- Fix wrong attribute name in the benchmark tests, which referenced `args.calibration` instead of `args.camera_calibration` and therefore failed, including the pre-existing panorama test. - Create the calibrator lazily when the controller runs, mirroring the feature extraction and matching controllers, so that the network is not held in device memory during preceding pipeline stages. - Skip images whose dimensions do not match their database camera, which would otherwise store focal length and principal point at the wrong scale, since only the fitted parameters are written back. - Skip learned calibration in the automatic reconstruction pipeline if explicit camera parameters were provided, which the image reader writes to the database and which must not be overwritten. - Re-validate the aggregated calibration and fall back to the closest single-image calibration, because the coefficient-wise median of individually valid calibrations is not itself guaranteed to be projection-stable for high-order distortion models. - Reject implausible per-image predictions via HasBogusParams, with the bounds exposed as backend-agnostic calibration options. - Define backend-specific options via shared_ptr, as for feature extraction. - Reuse Median from colmap/math and remove the dead ReverseScaleAndShiftParams.
- Share IsValidCalibration between the per-image and the aggregation path, which previously used two divergent copies, so that neither applied the full validation. NOTE: per-image predictions are now also checked for projection stability. - Pass the target camera model to AggregateCameraCalibrations explicitly, instead of requiring the caller to set it beforehand, and validate on a candidate, so that the camera is only modified on success. - Compute the closest calibration in a single pass over the deviations. - Use Camera::CreateFromModelId in the tests and merge duplicated cases.
The calibrator is created lazily in the controller's worker thread, where an escaping exception terminates the whole process instead of failing the stage. Previously it was created in the constructor on the main thread, where it propagated normally. A corrupt or missing model file therefore aborted the automatic reconstruction pipeline with SIGABRT after feature extraction had already run.
Ceres terminates without taking a step when the closed-form init is already at the optimum (noise-free pinhole data), reporting bit-identical initial/final costs and breaking the strict < assertion on some toolchains. This matches the product's own final <= initial success contract.
Harden validation (non-throwing Check, IsValidCalibration edge cases, tighter round-trip tolerance), validate ONNX dtypes in AnyCalib/ALIKED/LoMa via new element-type accessors, harden ray fitting (LDLT definiteness, inverse-focal epsilon, FOV bounds, invalid-prior tolerance), make the calibration controller non-fatal, verify export weights hash and record provenance, and extend tests and docs.
Extend the linear solvers in ray_fitting.cc with a fisheye variant that fits the equidistant (theta) projection instead of X/Z, and dispatch radial-fisheye models to it for their leading k1[, k2] coefficients. Undistorted fisheye models use the equidistant pinhole fit. The FOV gate uses an incidence-angle check for fisheye rays. HasRadialDistortionPrefix now returns optional<pair<int, bool>> with the coefficient count and fisheye flag instead of pointer out-params. Tests: wide-FOV (~170 deg) round-trip cases for all fisheye models, plus an init-only test (zero refinement iterations) pinning the closed-form solution.
InitializeCameraParams now uses naive initialization (focal length from the image span, principal point at the data center, zero distortion) instead of the AnyCalib-style closed-form linear solvers. Stress tests over strong distortion (incl. near-pole division), noise, sparsity, partial coverage, and outliers showed the Ceres refinement reaches the same optimum from the naive start every time, so the linear init only saved iterations at the cost of ~270 lines of solver machinery. Removes the now-unused max_fov_deg option (incl. the CameraCalibration.max_fov_deg CLI flag). Tests: drop the obsolete init-only test; FULL_OPENCV round-trip skips distortion-value assertions since naive-start refinement may land on a projection-equivalent point in the weakly identifiable subspace (projection check still binds).
- Widen before multiplication in ray index computation - Add default cases to rot90 switches and drop const blocking move - Keep intentional integer crop-offset division out of float context - NOLINT intentional out-of-range enum cast and getenv in tests
- Wire --image_list_path through the camera_calibrator CLI and pass the selected image names to the calibration controller, as documented. - Default CameraCalibration.camera_model to empty to preserve each database camera's existing model; an explicit value still converts all cameras. Reject invalid models gracefully before inference, since the camera-model switches throw on them. - Standardize the new calibration files on SPDX license headers.
ahojnnes
force-pushed
the
user/jsch/anycalib-calibration
branch
from
September 20, 2026 05:52
3b44352 to
a90ab99
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New
src/colmap/calibrationmodule for single-image camera calibration:anycalib.h/cc): runs the exported AnyCalib network (DINOv2 + DPT decoder) to regress dense per-pixel camera rays, then fits intrinsics for the requested perspective model.ray_fitting.h/cc): closed-form pinhole/radial initialization (ported from upstream AnyCalib) plus Ceres refinement of pixel residuals for any perspective COLMAP model.calibrator.h/cc,resources.h): backend selection with downloadable model resources.camera_calibrationcontroller wired into automatic reconstruction, option manager, andcolmapCLI.scripts/anycalib/export_onnx.pyexports stage 1 to ONNX (fixed 322x322 input).feature/onnx_utils.*toutil/onnx.*; new code usesimg_points/cam_raysnaming consistent with the rest of the codebase.Benchmark: ETH3D / DSLR, uncalibrated
Reconstruction benchmark A/B with and without learned calibration, for both the
incremental and global mappers (no GT priors, SIFT, high quality, seed 0,
single-threaded scenes, deterministic/paired; relative-pose AUC in %). Global
runs reuse the incremental runs' features + raw matches and only re-verify with
the automatic reconstructor's global two-view settings (max_error 1.0, min
inliers 30, min inlier ratio 0.25, no guided matching).
Undistorted (13 scenes, 454 images)
Per-scene average gains: incremental +2.42/+1.84/+1.52/+1.21; global
+3.02/+3.10/+1.98/+2.34. Global+AnyCalib nearly reaches incremental-baseline
accuracy.
Per-scene deltas (calibrated − baseline, AUC pp @1°). Incremental: pipes +9.2,
office +7.7 (also 2→1 components), terrace +7.2, relief +7.0, courtyard +4.0,
relief_2 +2.3, kicker +2.0, facade +0.6, delivery_area +0.3, terrains −0.2,
electro −1.0, playground −3.4, meadow −11.8. Global: relief +14.3, relief_2
+9.5, kicker +7.9, terrace +6.1, delivery_area +5.0, terrains +4.5, pipes +3.8,
facade +2.7, office +2.7, playground +2.2, courtyard −0.7, electro −5.3,
meadow −12.5.
Caveats: meadow regresses in both mappers (consistent treatment effect on that
15-image scene); single seed, but deterministic, so deltas are pure treatment
effects. Calibration cost is negligible vs. matching (~90s for 14 images,
single-threaded CPU).
Distorted, train scenes (13 scenes, 454 images)
Same 4-way setup on
eth3d-distortedwith fresh extraction/matching(SIMPLE_RADIAL). Note the distorted JPEGs do carry focal EXIF (20mm →
f=3359.7 prior, 2% off GT), so the baseline starts from a decent focal prior
with k=0 and the A/B tests focal refinement + distortion estimation.
Per-scene average gains: incremental +1.38/+1.48/+0.53/−0.54; global
+2.09/+3.26/+2.39/+1.38. Calibration helps more on distorted than undistorted,
but absolute levels stay far below undistorted (SIMPLE_RADIAL underfits the
true distortion).
Per-scene deltas (calibrated − baseline, AUC pp @1°). Incremental: facade
+14.1, courtyard +7.2, electro +6.6, relief +4.7, terrace +3.4, kicker +3.4,
relief_2 +1.3, terrains +1.1, delivery_area +0.6, office −0.0 (loses one
image), meadow −1.0 (−20 @5°), pipes −3.6, playground −18.5. Global: terrace
+7.7, playground +7.5, relief +6.1, electro +5.8, relief_2 +4.5, office +3.5,
facade +3.3, pipes +2.1, courtyard +1.6, meadow +0.7, terrains +0.7, kicker
+0.5, delivery_area −1.6 (electro loses one image).
Takeaway: incremental gains are volatile (playground/meadow collapse), while
global gains are uniform (12/13 scenes improve). Global+AnyCalib ≈
incremental+AnyCalib overall (30.19 vs 30.80 @1°).
Distorted with THIN_PRISM_FISHEYE, train scenes (13 scenes, 454 images)
Same 4-way setup reusing the distorted run databases (no re-extraction or
re-matching): base arms convert the
SIMPLE_RADIALcameras toTHIN_PRISM_FISHEYEpreserving EXIF focal/principal point/k1, calib arms fitAnyCalib directly to
THIN_PRISM_FISHEYE; two-view geometries re-verified(incremental defaults / global settings), then sparse-only mapping. Note both
arms lack redone guided matching (requires re-matching), so absolute numbers
slightly understate from-scratch runs.
Per-scene average gains: incremental +1.77/+1.56/+1.00/+0.88; global
+4.84/+4.56/+2.63/+1.79.
Per-scene deltas (calibrated − baseline, AUC pp @1°). Incremental: pipes +4.5,
terrace +3.1, kicker +2.9, relief_2 +3.0, relief +2.2, courtyard +2.5, office
+2.6, electro +2.5, terrains +1.3, facade −0.1, meadow −0.4, delivery_area −0.6,
playground −3.2. Global: playground +30.2, terrains +9.6, pipes +5.4, terrace
+4.1, electro +3.7, relief_2 +2.8, relief +2.1, courtyard +2.1, office +1.4,
kicker +0.9, delivery_area +0.3, facade +0.2, meadow −3.6 (but +5 registered
images, split over 2 components).
Takeaway: the model dominates (TPF base triples the SIMPLE_RADIAL baseline);
with the right model, incremental self-calibration leaves little headroom
(+0.77 @1°), while global still gains +4.32 @1° with playground +30.2 as the
standout.
Test plan
ctest -R \"calibration/(ray_fitting_test|anycalib_test|calibrator_test)|controllers/camera_calibration_test|util/onnx_test\": 5/5 pass.colmap_feature_{aliked,loma,onnx_matchers}_testtargets build (ONNX include consumers).clang-format --dry-run --Werrorclean on all changed C++ files.