fix(metrics): return NaN when FSC/FRC curve never legitimately crosses threshold - #45
Conversation
Reviewer's GuideAdjusts FSC/FRC spectral resolution estimation to return NaN when curves never legitimately cross the threshold and hardens root-finding against extrapolation artifacts, while also making segmentation cleanup robust to skimage/cucim API drift and tightening label image validation. Adds regression coverage for the spectral resolution behavior. File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey - I've found 2 issues, and left some high level feedback:
- In
_remove_small_objects, passingmax_size=min_size - 1will producemax_size <= 0whenmin_obj_size/min_sizeis 0 or 1; consider explicitly short‑circuiting those cases (e.g., returning the input unchanged) to avoid surprising behavior or warnings from skimage. - The logic for setting NaN resolutions when the root is invalid is now duplicated between the
fit_start is Nonepath and the newrootrange check; consider refactoring this into a small helper to keep the NaN-resolution behavior centralized and harder to drift over time.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `_remove_small_objects`, passing `max_size=min_size - 1` will produce `max_size <= 0` when `min_obj_size`/`min_size` is 0 or 1; consider explicitly short‑circuiting those cases (e.g., returning the input unchanged) to avoid surprising behavior or warnings from skimage.
- The logic for setting NaN resolutions when the root is invalid is now duplicated between the `fit_start is None` path and the new `root` range check; consider refactoring this into a small helper to keep the NaN-resolution behavior centralized and harder to drift over time.
## Individual Comments
### Comment 1
<location path="cubic/metrics/spectral/analysis.py" line_range="396-401" />
<code_context>
difference = y - thr
crossings = np.where(difference <= 0)[0]
if len(crossings) == 0:
- return None # No threshold crossing found
- return x[crossings[0] - 1] if crossings[0] > 0 else x[0]
-
+ # Never crosses the threshold — resolution is beyond Nyquist
+ # (or the prediction is so close to GT that FSC stays above
+ # threshold everywhere). No measurable resolution: return NaN
+ # rather than reporting Nyquist as a floor.
+ return None
+ if crossings[0] == 0:
+ # Curve starts already below threshold at the lowest measured
</code_context>
<issue_to_address>
**nitpick:** Clarify mismatch between comments promising NaN and the function returning `None`.
The new branches in `first_guess` say "Return NaN" but still return `None`. Please either update the comments to describe returning `None` (e.g., "signal no resolution") or change the code to actually return `np.nan` so the function’s contract is explicit and future readers don’t assume it returns NaNs directly.
</issue_to_address>
### Comment 2
<location path="tests/metrics/spectral/test_frc.py" line_range="766-768" />
<code_context>
+ # Case 1: below-threshold-everywhere curve (zero-correlation prediction).
+ below = FourierCorrelationData()
+ below.correlation["frequency"] = freqs
+ below.correlation["correlation"] = 0.02 + 0.05 * np.random.default_rng(
+ 0
+ ).standard_normal(50)
+ below.correlation["points-x-bin"] = np.full(50, 100.0)
+ coll = FourierCorrelationDataCollection()
</code_context>
<issue_to_address>
**suggestion (testing):** Avoid relying on a specific random draw for the below-threshold curve and use a deterministic construction instead.
Relying on a random normal draw (even with a fixed seed) makes it non-obvious that the curve is always below 0.143 and risks future changes to the seed/distribution breaking that guarantee. Instead, set `below.correlation['correlation'] = np.full_like(freqs, 0.05)` (or another constant < 0.143) so the "always below" condition is explicit and robust.
Suggested implementation:
```python
# Case 1: below-threshold-everywhere curve (zero-correlation prediction).
below = FourierCorrelationData()
below.correlation["frequency"] = freqs
# Use a deterministic, always-below-threshold correlation curve.
below.correlation["correlation"] = np.full_like(freqs, 0.05)
below.correlation["points-x-bin"] = np.full(50, 100.0)
```
1. If `np.random` / `default_rng` is only used in this block within the test file, you can safely remove any related imports (e.g., `import numpy.random` or similar) at the top of `tests/metrics/spectral/test_frc.py` to keep dependencies clean.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
There was a problem hiding this comment.
Pull request overview
This PR hardens spectral resolution metric computation to avoid reporting extreme/invalid FSC/FRC resolutions when the correlation curve never legitimately crosses the threshold, and improves segmentation post-processing robustness across scikit-image / cuCIM API differences.
Changes:
- Update FSC/FRC resolution analysis to return
NaNwhen the curve starts below threshold and to reject optimized crossing points outside the sampled frequency range. - Add regression tests covering below-threshold, above-threshold, and legitimate-crossing curves for the updated resolution behavior.
- Make segmentation utilities more robust by raising
TypeErrorfor non-integer label images and dispatching small-object/hole removal across scikit-image vs cuCIM API drift.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
cubic/metrics/spectral/analysis.py |
Adds guards to ensure invalid/no-crossing threshold scenarios yield NaN rather than extreme resolutions. |
tests/metrics/spectral/test_frc.py |
Adds a regression test for NaN behavior and validates a legitimate-crossing case remains finite. |
cubic/segmentation/segment_utils.py |
Replaces assertions with TypeError for dtype validation and introduces version-/backend-aware morphology helpers. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…s threshold `first_guess` previously returned `x[0]` when the FSC/FRC curve was already below the threshold at the lowest measured frequency (typical of predictions with ~zero correlation to GT). That tiny seed fed an unbounded `optimize.fmin` which then wandered into extrapolated territory and produced absurd roots — negative, or near-zero — yielding resolutions of millions of µm via `2 * spacing / root`. Two-part fix at `cubic/metrics/spectral/analysis.py`: 1. `first_guess` returns `None` (→ NaN) when the curve starts already below threshold (`crossings[0] == 0`). There is no above-to-below crossing to report. 2. After `fmin`, reject roots that landed outside the data's measured frequency range. The unbounded optimizer can still find spurious extrapolated minima even when the curve starts above threshold but never cleanly crosses inside the data range. Both cases now set `resolution = NaN`, matching the existing "curve never crosses" code path. Reporting NaN is preferred over capping at Nyquist — it signals "no measurable resolution" rather than implying a meaningful diffraction-limit value. Regression test covers three cases: below-threshold-everywhere (→ NaN), above-threshold-everywhere (→ NaN), and a legitimate crossing (→ finite). Reported via .scratch/fsc_outliers.csv on a downstream VisCy eval pipeline (135 outlier FOVs with XY or Z > 50 µm, including 8.3e6 µm for an A549 model on iPSC OOD data). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three small follow-ups from PR review: 1. **Sourcery on analysis.py**: ``first_guess`` comments said "Return NaN" but the function returns ``None`` and the caller turns that into NaN. Clarify the comment so the contract is explicit. 2. **Sourcery + Copilot on test_frc.py**: replace the random below-threshold curve with a deterministic constant (``np.full(50, 0.02)``). The previous RNG draw could theoretically produce values above 0.143, weakening the test invariant across NumPy versions. 3. **Copilot on segment_utils.py**: ``_remove_small_objects`` / ``_remove_small_holes`` previously called ``_cu_morphology.<...>`` unconditionally on the GPU branch — if cucim was not installed ``_cu_morphology`` is ``None`` and the user saw a confusing ``AttributeError`` on ``NoneType``. Raise a clear ``ImportError`` pointing at the GPU-extras install path instead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
6c7ff45 to
caa6f4d
Compare
Bug
Downstream evaluation surfaced 135 FOVs with FSC resolutions in the millions of µm (e.g. XY = 8.3e6 µm for an A549-trained nuclear model on iPSC OOD data). These are all cases where the prediction has essentially zero frequency correlation to GT, so the FSC curve sits below the 0.143 threshold from the lowest measured bin onward.
Root cause
In
cubic/metrics/spectral/analysis.py's_process_dataset:When
crossings[0] == 0(curve already below threshold at the first measured bin), the seedx[0]is fed to an unboundedoptimize.fmin. With thesmooth-splinecurve fit, the optimizer can wander far outside the data range (the spline extrapolates) and converge on a tiny or negativeroot. Thenresolution = 2 * spacing / rootblows up — millions of µm, or negative, depending on the sign.I reproduced the mechanism: on a curve sitting at ~0.02 ± 0.05 across [0.05, 1.0],
fminfromfit_start=0.05converges toroot = −0.0024, givingresolution = ∞.Fix
Two-part guard in
_process_dataset:first_guessreturnsNone(→ NaN) whencrossings[0] == 0— the curve starts already below threshold, so there is no legitimate above-to-below crossing.fmin, reject roots that landed outside[freqs[0], freqs[-1]]. The unbounded optimizer can still find spurious minima in the extrapolated region even when the curve starts above threshold but never cleanly crosses inside the data range.Both cases now set
resolution = NaN, matching the existing "curve never crosses" path. NaN is preferred over capping at Nyquist — it signals "no measurable resolution" rather than implying a meaningful diffraction-limit value.Test plan
test_resolution_returns_nan_when_curve_below_threshold) covers below-threshold-everywhere → NaN, above-threshold-everywhere → NaN, and a legitimate decay-through-threshold curve → finite reasonable value🤖 Generated with Claude Code
Summary by Sourcery
Ensure spectral resolution metrics return NaN when FSC/FRC curves never legitimately cross the threshold and harden segmentation utilities against datatype and morphology-API edge cases.
Bug Fixes:
Tests: