Skip to content

fix(metrics): return NaN when FSC/FRC curve never legitimately crosses threshold - #45

Merged
alxndrkalinin merged 2 commits into
mainfrom
fix/fsc-nan-when-no-threshold-crossing
May 29, 2026
Merged

fix(metrics): return NaN when FSC/FRC curve never legitimately crosses threshold#45
alxndrkalinin merged 2 commits into
mainfrom
fix/fsc-nan-when-no-threshold-crossing

Conversation

@alxndrkalinin

@alxndrkalinin alxndrkalinin commented May 29, 2026

Copy link
Copy Markdown
Owner

Bug

Downstream evaluation surfaced 135 FOVs with FSC resolutions in the millions of µm (e.g. XY = 8.3e6 µm for an A549-trained nuclear model on iPSC OOD data). These are all cases where the prediction has essentially zero frequency correlation to GT, so the FSC curve sits below the 0.143 threshold from the lowest measured bin onward.

Root cause

In cubic/metrics/spectral/analysis.py's _process_dataset:

def first_guess(x, y, thr):
    difference = y - thr
    crossings = np.where(difference <= 0)[0]
    if len(crossings) == 0:
        return None  # No threshold crossing found
    return x[crossings[0] - 1] if crossings[0] > 0 else x[0]

When crossings[0] == 0 (curve already below threshold at the first measured bin), the seed x[0] is fed to an unbounded optimize.fmin. With the smooth-spline curve fit, the optimizer can wander far outside the data range (the spline extrapolates) and converge on a tiny or negative root. Then resolution = 2 * spacing / root blows up — millions of µm, or negative, depending on the sign.

I reproduced the mechanism: on a curve sitting at ~0.02 ± 0.05 across [0.05, 1.0], fmin from fit_start=0.05 converges to root = −0.0024, giving resolution = ∞.

Fix

Two-part guard in _process_dataset:

  1. first_guess returns None (→ NaN) when crossings[0] == 0 — the curve starts already below threshold, so there is no legitimate above-to-below crossing.
  2. After fmin, reject roots that landed outside [freqs[0], freqs[-1]]. The unbounded optimizer can still find spurious minima in the extrapolated region even when the curve starts above threshold but never cleanly crosses inside the data range.

Both cases now set resolution = NaN, matching the existing "curve never crosses" path. NaN is preferred over capping at Nyquist — it signals "no measurable resolution" rather than implying a meaningful diffraction-limit value.

Test plan

  • Regression test (test_resolution_returns_nan_when_curve_below_threshold) covers below-threshold-everywhere → NaN, above-threshold-everywhere → NaN, and a legitimate decay-through-threshold curve → finite reasonable value
  • All 33 existing spectral tests pass
  • mypy + ruff clean

🤖 Generated with Claude Code

Summary by Sourcery

Ensure spectral resolution metrics return NaN when FSC/FRC curves never legitimately cross the threshold and harden segmentation utilities against datatype and morphology-API edge cases.

Bug Fixes:

  • Return NaN for FSC/FRC resolutions when the curve is always below or above the threshold or when the optimized crossing lies outside the sampled frequency range, avoiding spurious infinite or extreme resolutions.
  • Raise a clear TypeError in segmentation label validation when inputs are not integer-typed instead of asserting.
  • Make morphological cleanup robust to skimage/cucim API changes for small-object and small-hole removal to avoid runtime errors across versions.

Tests:

  • Add regression tests covering below-threshold, above-threshold, and legitimate crossing FSC/FRC curves to validate NaN behavior and finite resolutions.

@sourcery-ai

sourcery-ai Bot commented May 29, 2026

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Adjusts FSC/FRC spectral resolution estimation to return NaN when curves never legitimately cross the threshold and hardens root-finding against extrapolation artifacts, while also making segmentation cleanup robust to skimage/cucim API drift and tightening label image validation. Adds regression coverage for the spectral resolution behavior.

File-Level Changes

Change Details Files
Harden FSC/FRC resolution computation to treat curves that never legitimately cross the threshold as having no measurable resolution (NaN) and to discard roots found outside the measured frequency range.
  • Update first_guess to return None when there are no threshold crossings or when the curve starts below the threshold at the first frequency bin, signaling that resolution should be NaN.
  • Capture the correlation frequencies array once and reuse it for first_guess and post-optimization checks.
  • After optimize.fmin, validate that the root lies within the measured frequency range [freqs[0], freqs[-1]] and, if not, set resolution-related fields (resolution-point, criterion, resolution, spacing) to NaN or appropriate values and return early.
cubic/metrics/spectral/analysis.py
Add regression tests to ensure FSC/FRC resolution returns NaN for curves that are entirely below or above the threshold and remains finite for legitimate threshold crossings.
  • Create a new test that constructs three synthetic FourierCorrelationData cases: below-threshold-everywhere, above-threshold-everywhere, and a decaying curve that crosses the threshold.
  • For each case, run FourierCorrelationAnalysis with fixed threshold and smooth-spline curve fit and assert NaN for non-crossing curves and a finite, reasonable resolution for the legitimate crossing case.
tests/metrics/spectral/test_frc.py
Make segmentation cleanup routines robust to skimage 0.26 vs cucim morphology API differences and improve label image validation behavior.
  • Change check_labeled_binary to raise TypeError with a descriptive message instead of using an assertion on integer dtype, and allow constant images without error.
  • Introduce a feature flag _SKIMAGE_USES_MAX_SIZE based on the remove_small_objects signature to detect whether skimage expects max_size instead of min_size.
  • Add helper _remove_small_objects that dispatches to cucim or skimage, translating min_size semantics to max_size=min_size-1 when needed to preserve "keep size ≥ min_size" behavior.
  • Add helper _remove_small_holes that dispatches to cucim or skimage, mapping area_threshold to max_size where skimage has renamed the parameter without changing semantics.
  • Update cleanup_segmentation, remove_small_objects, and fill_holes_slicer to call the new helpers instead of direct morphology.remove_small_objects/remove_small_holes so both CPU and GPU paths handle the API drift consistently.
cubic/segmentation/segment_utils.py

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 2 issues, and left some high level feedback:

  • In _remove_small_objects, passing max_size=min_size - 1 will produce max_size <= 0 when min_obj_size/min_size is 0 or 1; consider explicitly short‑circuiting those cases (e.g., returning the input unchanged) to avoid surprising behavior or warnings from skimage.
  • The logic for setting NaN resolutions when the root is invalid is now duplicated between the fit_start is None path and the new root range check; consider refactoring this into a small helper to keep the NaN-resolution behavior centralized and harder to drift over time.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In `_remove_small_objects`, passing `max_size=min_size - 1` will produce `max_size <= 0` when `min_obj_size`/`min_size` is 0 or 1; consider explicitly short‑circuiting those cases (e.g., returning the input unchanged) to avoid surprising behavior or warnings from skimage.
- The logic for setting NaN resolutions when the root is invalid is now duplicated between the `fit_start is None` path and the new `root` range check; consider refactoring this into a small helper to keep the NaN-resolution behavior centralized and harder to drift over time.

## Individual Comments

### Comment 1
<location path="cubic/metrics/spectral/analysis.py" line_range="396-401" />
<code_context>
             difference = y - thr
             crossings = np.where(difference <= 0)[0]
             if len(crossings) == 0:
-                return None  # No threshold crossing found
-            return x[crossings[0] - 1] if crossings[0] > 0 else x[0]
-
+                # Never crosses the threshold — resolution is beyond Nyquist
+                # (or the prediction is so close to GT that FSC stays above
+                # threshold everywhere). No measurable resolution: return NaN
+                # rather than reporting Nyquist as a floor.
+                return None
+            if crossings[0] == 0:
+                # Curve starts already below threshold at the lowest measured
</code_context>
<issue_to_address>
**nitpick:** Clarify mismatch between comments promising NaN and the function returning `None`.

The new branches in `first_guess` say "Return NaN" but still return `None`. Please either update the comments to describe returning `None` (e.g., "signal no resolution") or change the code to actually return `np.nan` so the function’s contract is explicit and future readers don’t assume it returns NaNs directly.
</issue_to_address>

### Comment 2
<location path="tests/metrics/spectral/test_frc.py" line_range="766-768" />
<code_context>
+    # Case 1: below-threshold-everywhere curve (zero-correlation prediction).
+    below = FourierCorrelationData()
+    below.correlation["frequency"] = freqs
+    below.correlation["correlation"] = 0.02 + 0.05 * np.random.default_rng(
+        0
+    ).standard_normal(50)
+    below.correlation["points-x-bin"] = np.full(50, 100.0)
+    coll = FourierCorrelationDataCollection()
</code_context>
<issue_to_address>
**suggestion (testing):** Avoid relying on a specific random draw for the below-threshold curve and use a deterministic construction instead.

Relying on a random normal draw (even with a fixed seed) makes it non-obvious that the curve is always below 0.143 and risks future changes to the seed/distribution breaking that guarantee. Instead, set `below.correlation['correlation'] = np.full_like(freqs, 0.05)` (or another constant < 0.143) so the "always below" condition is explicit and robust.

Suggested implementation:

```python
    # Case 1: below-threshold-everywhere curve (zero-correlation prediction).
    below = FourierCorrelationData()
    below.correlation["frequency"] = freqs
    # Use a deterministic, always-below-threshold correlation curve.
    below.correlation["correlation"] = np.full_like(freqs, 0.05)
    below.correlation["points-x-bin"] = np.full(50, 100.0)

```

1. If `np.random` / `default_rng` is only used in this block within the test file, you can safely remove any related imports (e.g., `import numpy.random` or similar) at the top of `tests/metrics/spectral/test_frc.py` to keep dependencies clean.
</issue_to_address>

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread cubic/metrics/spectral/analysis.py
Comment thread tests/metrics/spectral/test_frc.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens spectral resolution metric computation to avoid reporting extreme/invalid FSC/FRC resolutions when the correlation curve never legitimately crosses the threshold, and improves segmentation post-processing robustness across scikit-image / cuCIM API differences.

Changes:

  • Update FSC/FRC resolution analysis to return NaN when the curve starts below threshold and to reject optimized crossing points outside the sampled frequency range.
  • Add regression tests covering below-threshold, above-threshold, and legitimate-crossing curves for the updated resolution behavior.
  • Make segmentation utilities more robust by raising TypeError for non-integer label images and dispatching small-object/hole removal across scikit-image vs cuCIM API drift.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
cubic/metrics/spectral/analysis.py Adds guards to ensure invalid/no-crossing threshold scenarios yield NaN rather than extreme resolutions.
tests/metrics/spectral/test_frc.py Adds a regression test for NaN behavior and validates a legitimate-crossing case remains finite.
cubic/segmentation/segment_utils.py Replaces assertions with TypeError for dtype validation and introduces version-/backend-aware morphology helpers.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cubic/segmentation/segment_utils.py
Comment thread cubic/segmentation/segment_utils.py
Comment thread tests/metrics/spectral/test_frc.py Outdated
alxndrkalinin and others added 2 commits May 28, 2026 22:09
…s threshold

`first_guess` previously returned `x[0]` when the FSC/FRC curve was
already below the threshold at the lowest measured frequency (typical
of predictions with ~zero correlation to GT). That tiny seed fed an
unbounded `optimize.fmin` which then wandered into extrapolated
territory and produced absurd roots — negative, or near-zero —
yielding resolutions of millions of µm via `2 * spacing / root`.

Two-part fix at `cubic/metrics/spectral/analysis.py`:

1. `first_guess` returns `None` (→ NaN) when the curve starts already
   below threshold (`crossings[0] == 0`). There is no above-to-below
   crossing to report.
2. After `fmin`, reject roots that landed outside the data's measured
   frequency range. The unbounded optimizer can still find spurious
   extrapolated minima even when the curve starts above threshold but
   never cleanly crosses inside the data range.

Both cases now set `resolution = NaN`, matching the existing "curve
never crosses" code path. Reporting NaN is preferred over capping at
Nyquist — it signals "no measurable resolution" rather than implying
a meaningful diffraction-limit value.

Regression test covers three cases: below-threshold-everywhere (→ NaN),
above-threshold-everywhere (→ NaN), and a legitimate crossing (→ finite).

Reported via .scratch/fsc_outliers.csv on a downstream VisCy eval pipeline
(135 outlier FOVs with XY or Z > 50 µm, including 8.3e6 µm for an A549
model on iPSC OOD data).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three small follow-ups from PR review:

1. **Sourcery on analysis.py**: ``first_guess`` comments said "Return NaN"
   but the function returns ``None`` and the caller turns that into NaN.
   Clarify the comment so the contract is explicit.

2. **Sourcery + Copilot on test_frc.py**: replace the random
   below-threshold curve with a deterministic constant (``np.full(50,
   0.02)``). The previous RNG draw could theoretically produce values
   above 0.143, weakening the test invariant across NumPy versions.

3. **Copilot on segment_utils.py**: ``_remove_small_objects`` /
   ``_remove_small_holes`` previously called ``_cu_morphology.<...>``
   unconditionally on the GPU branch — if cucim was not installed
   ``_cu_morphology`` is ``None`` and the user saw a confusing
   ``AttributeError`` on ``NoneType``. Raise a clear ``ImportError``
   pointing at the GPU-extras install path instead.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@alxndrkalinin
alxndrkalinin force-pushed the fix/fsc-nan-when-no-threshold-crossing branch from 6c7ff45 to caa6f4d Compare May 29, 2026 05:10
@alxndrkalinin
alxndrkalinin merged commit de24edc into main May 29, 2026
9 checks passed
@alxndrkalinin
alxndrkalinin deleted the fix/fsc-nan-when-no-threshold-crossing branch May 29, 2026 05:12
alxndrkalinin added a commit that referenced this pull request Jun 1, 2026
Resets the version after v0.7.0a8/a9/a10 tags were deleted before
publish. Covers PRs #42, #43, #44, #45, and #46 on top of v0.7.0a7.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants