Support all Cellpose v4 models (SAM + DINO) in GPU-resident segmentation - #51

…us-anchor + paths.py + release tooling (#479)

* feat(viscy-models): add MorphEm foundation wrapper

MorphEmModel wraps CaicedoLab/MorphEm (DINO-pretrained ViT-S/16 on
microscopy) as a frozen feature extractor, mirroring the existing
DINOv3 / OpenPhenom / CELL-DINO foundation wrappers.

MorphEm is single-channel: each microscopy channel is encoded
independently and the CLS tokens are mean-pooled to a fixed embedding.
preprocess_2d applies the published PerImageNormalize recipe (per-image
per-channel z-score) before the bilinear resize to 224.

Includes a transformers-5.x compatibility shim: MorphEm's 4.x-era
trust_remote_code VisionTransformer never populates all_tied_weights_keys,
which 5.x's meta-device loader requires, so from_pretrained raises
AttributeError on transformers 5.9.0. A lazy, idempotent class-level
all_tied_weights_keys = {} default on PreTrainedModel fixes the load; it
is shadowed by any model whose tie_weights() runs, so other loads are
unaffected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(dynacell/eval): wire MorphEm as a deep feature extractor

Adds "morphem" as the fifth FeatureKind. Because the cache layer,
_BACKBONE_KEYS, and the cross-condition (infection) probe's
_FEATURE_TYPES all derive from get_args(FeatureKind), the new backbone
auto-propagates to embeddings, per-FOV/dataset feature metrics, and the
infection-classification AUROC rows. MorphEm is model-name-keyed in the
artifact cache (the DINOv3 pattern) rather than weights-sha-keyed.

MorphEmFeatureExtractor wraps MorphEmModel and calls preprocess_2d
explicitly (unlike CellDinoModel, MorphEmModel.forward does not
normalize inline). load_eval_models / precompute-gt gain a morphem gate
that soft-skips on a null pretrained_model_name (parity with celldino's
null weights_path).

reporting/tables.py adds MorphEm cosine/FID columns and backfills the
previously-omitted CellDINO columns so the feature-similarity table is
complete. Test fixtures (parallel aggregator, precompute build block)
are updated to track the new fifth backbone.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(dynacell/eval): enable MorphEm by default

Adds feature_extractor/morphem to the eval.yaml defaults list (with an
inline pretrained_model_name: null base placeholder and gt/pred_morphem
force_recompute keys) and build.morphem to precompute.yaml, so MorphEm
loads on every eval and precompute-gt run by default — the same posture
as DINOv3 and CELL-DINO. The new feature_extractor/morphem/default.yaml
group resolves the CaicedoLab/MorphEm hub id from the shared
HF_HUB_CACHE; pretrained_model_name: null is the explicit-disable path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(dynacell/eval): partition cross-condition probe by model

run_for_group received every condition save dir of a grouped bucket,
which folds many models — so multiple dirs mapped to the same condition
token (e.g. two *_denv dirs) and it raised "duplicate condition",
silently skipping the whole bucket's infection-vs-mock probe. This is
why the Infection-AUROC table rows were empty.

Partition eval_dirs by model prefix (the dir name minus the trailing
_{mock,denv,zikv} token) and probe each (model, pool, organelle) group
independently; the duplicate-condition guard now applies within a group
only. A mock reference is still required per group, and in-distribution
iPSC dirs (no token) are still ignored. The per-condition CSV output and
filename are unchanged, so the reporting layer reads them as before.

The probe loops _FEATURE_TYPES, so morphem rows now appear automatically
once it runs. Test file brought under version control with a regression
test for the multi-model case.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(dynacell/eval): make feature cache dimension-aware to self-heal stale caches

A CP/deep feature group could hold arrays of mixed column counts after a
recipe change (e.g. a GLCM toggle, or a CP_FEATURE_VERSION whose output
dim changed without a version bump) or an interrupted partial rebuild.
The manifest identity check compares recorded params, not array dims, so
the stale entries survived and later crashed _cp_dropzero_zscore with a
cryptic boolean-index IndexError — needing a manual force_recompute to
clear (hit during the MorphEm re-eval on er_celldiff).

Make the cache dimension-aware:
- write_features_to_group records the artifact's feature dim in a
  `feature_dim` group attribute (from the latest non-empty write; the
  zero-cell (0,0) sentinel never sets it).
- read_features_from_group returns None (-> cache miss -> recompute) for
  any stored array whose column count disagrees with that attribute, so
  stale/partial-rebuild entries self-heal on the next full eval instead
  of poisoning the metric step. Groups written before the attribute
  existed have no recorded dim, so nothing is dropped (bootstrap-safe).
- _cp_dropzero_zscore now raises an actionable StaleCacheError naming the
  remedy on a pred-vs-GT dim mismatch, as defense-in-depth for the
  uniform-staleness case the per-group attribute can't catch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(dynacell): document A549 condition-pooled training data

Both joint and A549-only fits read a single condition-pooled
<TARGET>_all.zarr (mock + ZIKV + DENV), built by assemble.py with
condition=None — not mock-only. Record this plus the non-obvious bits:
the condition is dropped from the pooled-store FOV names, there is no
*joint* train-set fragment (joint leaves author the BatchedConcatDataModule
children inline), and the deliberate train-pools / eval-per-condition
asymmetry behind the mock/denv/zikv table split.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore(dynacell): add cpdino deps (cellpose>=4.2, cubic PR#51, dinov3)

Cellpose-DINO (cpdino) instance segmentation needs cellpose>=4.2 (adds the
DINO backbones + pretrained_model="cpdino"), its dinov3 backbone, and cubic
0.8.0a2 (git @eb04eef, PR alxndrkalinin/cubic#51) whose device-resident
segment_cellpose supports all Cellpose v4 models (SAM + DINO). PyPI cubic
0.7.0 predates DINO support, so pin to the git ref.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146hBqKUrvARvadnvCZAmyb

* feat(dynacell/eval): add cpdino instance-segmentation backend

Cellpose-DINO ViT-L, run GPU-resident on the raw in-focus slice via cubic's
segment_cellpose with normalize=True and NO CLAHE/downscale — faithful to the
cpv4 benchmark (verified: foreground IoU=1.0 vs the saved cpv4 masks). New
segmentation_cpdino.py provides segment_cpdino_instances (nucleus / direct
whole-cell) and segment_whole_cell_cpdino (whole-cell + nucleus carve, which
replaces the nuclei-seed EDT watershed). load_cellpose_model gains a
model_name arg (cpsam default; cpdino selects the DINO backbone, cubic tiles
384). prepare_segmentation_model routes backend=cpdino to the DINO model for
both nucleus and membrane.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146hBqKUrvARvadnvCZAmyb

* feat(dynacell/eval): dispatch + validate the cpdino instance backend

_validate_instance_ap_config accepts cpdino for both nucleus and membrane
(membrane needs nuclei_channel_name for the carve seeds; both need
compute_instance_ap=true). The per-FOV instance dispatch adds a cpdino branch:
nucleus -> fov_cpdino_nucleus_instances; membrane -> cpdino nucleus seeds
(inline from the GT nucleus channel, mirroring the watershed seed path) +
fov_cpdino_whole_cell_instances. One cpdino model serves both channels.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146hBqKUrvARvadnvCZAmyb

* feat(dynacell/eval): key the instance cache on cpdino params

_CacheContext gains cpdino_params (resolved from segmentation.cpdino).
_instance_identity keys cpdino on its own param block (raw image + normalize,
no CLAHE), not the cellpose robust-clip params, and records the GT nuclei
source for whole-cell (the carve depends on it). Adds cpdino_infer_kwargs plus
fov_cpdino_nucleus_instances / fov_cpdino_whole_cell_instances, mirroring the
cellpose/watershed helpers over the shared _fov_instances cache machinery
(stem instance_masks/<target>__cpdino.zarr, so cpdino never aliases old caches).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146hBqKUrvARvadnvCZAmyb

* feat(dynacell/eval): make cpdino the default instance backend

Adds the segmentation.cpdino config block (model_name, normalize=true,
flow/cellprob thresholds, min_size=15, subtract_nuclei=true) and documents
cpdino as the recommended instance backend (global default stays supermodel
for ER/mito binary DICE). Flips all 38 instance-AP leaves (19 cellpose nucleus
+ 19 cellpose_watershed membrane) to backend=cpdino. cpdino whole-cell +
nucleus carve lifts membrane mAP from ~0.002-0.03 (watershed rings) to
~0.6-0.7 (validated on A549 mock), and carving no longer degrades it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146hBqKUrvARvadnvCZAmyb

* feat(dynacell/eval): prewarm GT instance cache via precompute-gt build.instances

Adds build.instances to dynacell precompute-gt: computes GT cpdino instance
labels (nucleus direct; membrane whole-cell + GT-nucleus carve, reading the
nucleus channel from io.nuclei_gt_path or the GT store) into
instance_masks/<target>__cpdino.zarr with the same identity the eval reads.
Warming GT instances once per (test-set, target) lets the many parallel model
buckets hit the cache instead of racing to recompute the shared GT instances.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146hBqKUrvARvadnvCZAmyb

* test(dynacell/eval): cover the cpdino instance backend

CPU logic (whole-cell nucleus carve, dispatch/validation guards, cache
identity, infer-kwarg stripping) plus CUDA-gated end-to-end (dino_vitl
backbone, 2D slice shape/dtype, empty slice).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146hBqKUrvARvadnvCZAmyb

* feat(dynacell/eval): default the eval-config generators to cpdino

The grouped + instance-AP config generators (and the ablation instance-AP
patcher) emitted the old cellpose/cellpose_watershed instance backends. Point
them at cpdino for both nucleus and membrane so regenerating the leaves stays
consistent with the committed defaults. Drops the ablation no-carve watershed
override: cpdino segments the whole cell directly, so the nucleus carve no
longer collapses AP (validated: carve ~= no-carve, mAP ~0.6-0.7) and it
inherits the eval.yaml subtract_nuclei=true default like the grouped leaves.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146hBqKUrvARvadnvCZAmyb

* fix(dynacell/eval): load the separate GT-nuclei store for cpdino membrane

_separate_nuclei_path gated the cross-store GT-nuclei open on
backend=='cellpose_watershed', so cpdino whole-cell on A549 (membrane in
CAAX_*.ozx, nuclei in H2B_*.ozx) got pos_nuclei=None, fell back to the CAAX
plate, and crashed with "Channel Nuclei is not in the existing channels".
cpdino membrane carves the nucleus and needs the same separate store, so
include it in the gate. iPSC single-store (cell.zarr) still returns None and
reads Nuclei from the GT plate. Regression test added.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146hBqKUrvARvadnvCZAmyb

* feat(dynacell/eval): score the raw input as a no-model prediction

Add an `io.pred_is_source` mode to the dataset_ref hook: when set, the
prediction channel resolves to the manifest's declared source channel
(e.g. Phase3D) instead of `{target}_prediction`, and `io.pred_path`
defaults to the GT store (source + target share one store), so pred/GT
positions match by construction. This evaluates the raw phase input
directly against the fluorescence GT with no virtual-staining model —
a zero-model floor below the untrained randinit baseline.

The pred-channel collision check still fires when an explicit
pred_channel_name disagrees with the resolved value, so typo protection
is preserved in both modes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146hBqKUrvARvadnvCZAmyb

* feat(dynacell/eval): add phase-input baseline eval leaves

Eight single-leaf configs that score the raw Phase3D input against the
fluorescence GT via cpdino (the no-model floor), using io.pred_is_source:
iPSC {nucleus,membrane} and A549 {nucleus,membrane} x {mock,denv,zikv}.
Membrane leaves carry nuclei_channel_name + (A549) the per-condition H2B
nuclei store for the whole-cell carve seeds. Save dirs use the
eval_phase_{org}[_{cond}] convention so the paper tables resolve them as
a leading floor column alongside randinit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146hBqKUrvARvadnvCZAmyb

* feat(dynacell/eval): add nucleus-area focus anchor + slab MIP helpers

Add a robust in-focus plane estimator for 2D instance segmentation that
anchors on nuclear geometry instead of spectral sharpness. The per-z
nuclear foreground fraction (Otsu on the GT nucleus channel) is a smooth,
unimodal curve — zero at the stack caps, peaked at the nuclear equator —
so its argmax cannot be dragged to an out-of-focus cap by the high-frequency
artifacts (confocal phase speckle, basal-membrane web) that fool the
waveorder transverse-band estimator on iPSC data. A guard band drops the
outer 15% of planes and a constant/near-empty volume falls back to the
stack center.

- nucleus_area_plane / resolve_nucleus_area_planes: the anchor.
- slab_mip: max-project a +/-halfwidth slab per timepoint (halfwidth=0
  reduces to the single plane, byte-identical to a plain index).
- resolve_focus_instance_planes: shared plane resolver dispatching on
  segmentation.focus_anchor (nucleus_area | phase_midband) so the eval and
  the GT pre-warm cannot diverge.

Pure additions (unused until wired in the next commit) + unit tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146hBqKUrvARvadnvCZAmyb

* feat(dynacell/eval): default 2D instance focus to the nucleus-area anchor

Wire resolve_focus_instance_planes + slab_mip into the eval instance path
and the GT pre-warm, and make nucleus_area the eval.yaml default
(focus_anchor: nucleus_area, focus_slab_halfwidth: 1). The old phase-midband
estimator on iPSC confocal data slid the plane to the stack edge for a subset
of FOVs (e.g. 42/116598/10604 -> z=36 of 40 while the nuclei are sharp at
z~20), because the confocal Phase3D reconstruction turns to high-frequency
speckle toward the cap and the transverse-band estimator maximizes exactly
that. The nucleus-area anchor lands mid-stack (the nuclear equator, which is
also where the membrane draws clean whole-cell polygons), so one anchor
serves both nucleus and whole-cell; the +/-1 slab makes a one-plane error
harmless.

- pipeline._process_one_fov: resolve the plane via the shared helper (nucleus
  volume = the GT nucleus target for nucleus, the GT-nuclei source for
  whole-cell), then slab_mip the target/prediction/nuclei stacks.
- precompute_cli: mirror the same anchor + slab so pre-warmed GT masks match
  the eval identity (else the eval would false-hit phase-midband GT masks).
- pipeline_cache: record focus_anchor + focus_slab_halfwidth in the instance
  cache identity (phase estimator params only under phase_midband), so the
  new default auto-invalidates the old phase-anchor masks; frac/sharpest
  identities stay byte-stable.

Deep-feature focus_slab (feature_metrics.focus_slab) is unchanged and still
phase-based — a separate, out-of-scope concern.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0146hBqKUrvARvadnvCZAmyb

* feat(dynacell/eval): add canonical artifact-path module paths.py

Foundation (PR 1) of the artifact-path standardization: evolve
save_paths.py into a single source-of-truth grammar module.

- Vocab/alias maps (org, train_set incl. a549__deconv, model paper->code;
  no unext2 alias — ambiguous), paper display registry (+celldiff_r2,
  +unext2_timm_scratch; CELL-Diff display-collapse preserved),
  resolve_model (identity from ckpt_path + joint-celldiff=R2 rule).
- Grammar: checkpoint_dir, prediction_store, eval_leaf (track/gt_repr/
  component), gt_cache_dir (frozen), pred_cache_dir, metrics_repo_dir,
  iter_organelle_evals; GT-repr-aware ORGANELLE_EVAL_TARGET.
- normalize_legacy: deconv provenance (a549->a549__deconv,
  joint->joint__legacy_deconvgt), dual_nucl_memb->dual_nucleus_membrane,
  bare-a549=iPSC-trained, ablation legacy set; returns None for UNMAPPED.
- Tuple validity: __deconv er/mito-only; no ipsc__deconv / forward
  joint__deconv.

save_paths.py kept as a thin re-export shim (retains eval_save_dir/
PAPER_KEY) so not-yet-refactored consumers + the generator drift-guard
stay green; PR 3 deletes it when it refactors those consumers.
test_save_paths.py -> test_paths.py (271 pass); ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1Qzmc3NxeMTerNjzBJgZE

* docs(dynacell/eval): correct MorphEmExtractor.extract_features return shape

extract_features returns the model's batch embedding of shape (1, D)
(D=384 for ViT-S/16), but the docstring claimed a 1-D (384,) vector.
Match the docstring to the actual return and the (N, D) extractor
contract used elsewhere.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1Qzmc3NxeMTerNjzBJgZE

* fix(dynacell/eval): gate the separate GT-nuclei store to membrane targets

_separate_nuclei_path returned io.nuclei_gt_path for any cpdino /
cellpose_watershed instance-AP run, including target_name='nucleus'
where the cpdino path never reads a separate nuclei store — opening and
position-validating a store that is never used. Only the whole-cell
membrane paths carve a nucleus footprint, so gate the separate-store
logic on target_name == 'membrane'.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1Qzmc3NxeMTerNjzBJgZE

* fix(dynacell/eval): raise a clear error for missing membrane nuclei channel

_build_gt_instances for target_name='membrane' reads
segmentation.nuclei_channel_name and passes it to get_channel_index. The
precompute config comment suggests running with compute_instance_ap off,
which can leave nuclei_channel_name null — then get_channel_index(None)
fails with an opaque error. Raise an explicit ValueError naming the
missing key and the fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1Qzmc3NxeMTerNjzBJgZE

* fix(dynacell/eval): keep celldiff_r2 variants distinct in resolve_model

resolve_model matched "celldiff_r2" as a substring of the full ckpt path
string, so celldiff_r2_iterative / _sliding_window / _denoise all
collapsed onto bare celldiff_r2 — their predictions and evals would share
one on-disk path, contradicting normalize_legacy (which parses the
variants correctly). Match the most specific variant as a path SEGMENT
(or explicit model_name) instead, longest-first.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1Qzmc3NxeMTerNjzBJgZE

* fix(dynacell/eval): map A549 ablation + dynacell-FT eval dirs in normalize_legacy

Two eval-dir families were silently returning None (UNMAPPED):

- A549 ablation eval dirs (eval_vscyto3d_<abltoken>_<organelle>_<cond>):
  the ablation branch looked for the organelle at the end of the name
  without stripping the trailing A549 condition first, so `_denv` defeated
  _last_organelle_token and every A549 ablation dir failed to map — and
  the condition was hardcoded to None. Strip the condition before the
  organelle and carry it through.

- dynacell-FT ablation models (vscyto3d_cytolandft /
  vscyto3d_infectionft_dynacellft) save to evaluations_<variant>[_a549trained]/
  (no _with_embeddings suffix). Those parents were in no map, so the dirs
  were UNMAPPED even though the models are registered. Add the four parents
  (iPSC-FT base / A549-trained) to _EVAL_PARENT_TRAIN_SET.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1Qzmc3NxeMTerNjzBJgZE

* fix(dynacell/eval): correct metrics_repo_dir path and reject ablation/forward pairs

Three path-construction correctness fixes in paths.py (tests co-located):

- metrics_repo_dir default repo_root used parents[4] (= .../applications),
  so the base doubled to .../applications/applications/dynacell/results/...
  Use parents[5] (the repo root). Tests always passed repo_root=, so this
  never surfaced.
- metrics_repo_dir dropped the gt_repr axis that eval_leaf carries, so raw
  and deconv-GT ER/mito evals would overwrite each other in the git-tracked
  mirror. Add the gt_repr param + deconv_gt subdir, mirroring eval_leaf.
- _tuple_is_valid accepted an ablation model paired with a forward train_set
  (a spurious path). Require an ablation model to pair only with its closed
  ablation train_set — the mirror of the existing ablation-train_set rule.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1Qzmc3NxeMTerNjzBJgZE

* docs(dynacell/eval): fix eval_leaf subdir order in docstring

The eval_leaf docstring listed instance_ap above deconv_gt, but the code
nests deconv_gt above instance_ap. Align the docstring to the code so the
paper vendor-copy of the grammar stays in parity.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1Qzmc3NxeMTerNjzBJgZE

* feat(dynacell): cpdino whole-cell _seg_cleaned driver for A549 mantis

Add the segmentation step of the A549 mantis eval chain: build the
whole-cell instance mask (_seg_cleaned.zarr) from the VSCyto3D _vs
predictions, upgrading the backend from Cellpose-v3 to cpdino
(Cellpose-DINO), consistent with cpdino being the default instance-seg
backend for dynacell evals.

Runs segment_cpdino_instances on the membrane_prediction max-Z
projection, repeats the 2D labels over Z, and drops sub-threshold
instances (remove_small_instances_3d, min 1e5 vox) before writing a
single-channel uint16 <stem>_seg_cleaned.zarr alongside the _vs store.
The mask is organelle-independent (nucleus/membrane VS channels only),
so one run serves all four organelle targets. slurm array 0-8 covers the
9 test stores; UV_PROJECT_ENVIRONMENT points at the cpdino-eval venv.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(dynacell): repoint fcmae ER/mito A549 train leaves to raw mantis stores

Wave-1 of the A549 raw-flip retrain. The A549 ER/mito training data was
regenerated with raw GFP as the canonical `Structure` channel and moved
from `…/a549/mantis_v1/train/` to `…/a549/mantis/train/`. Repoint the
`data_path` in the 8 fcmae_vscyto3d_{scratch,pretrained} × {a549,joint} ×
{er,mito} train leaves accordingly (A549 child only in the joint leaves;
the iPSC dataset_v4 children are unchanged).

`target_channel: Structure` is unchanged — the new store keeps that name
for the raw channel (deconv preserved separately as Structure_deconvolved),
so repointing while holding the name is exactly the deconv→raw target flip.
Verified via --print-resolved-config: all 8 resolve to mantis/train raw
Structure, max_epochs=200, devices=4, no trainer-level resume ckpt_path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(dynacell): correct unetvit3d joint batch_size to match single-set

The joint unetvit3d leaves set batch_size=4 with num_samples=2. Because
BatchedConcatDataModule does NOT divide by num_samples (unlike HCSDataModule),
that yields 8 GPU samples/step — 2x the single-set unetvit3d (batch=4,
num_samples=2 -> 4 samples/step). Every other joint family (fcmae, celldiff,
pix2pix) already follows joint.batch_size = single/num_samples. Set the joint
unetvit3d batch_size to 2 so the effective per-step sample count matches the
single-set, keeping the a549-only vs joint comparison apples-to-apples (and
avoiding the larger effective batch that could OOM the single H200).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(dynacell): repoint fnet/celldiff/unetvit ER/mito train leaves to raw mantis

Waves 2-3 of the A549 raw-flip retrain. Repoint the A549 training data_path
in the fnet3d_paper, celldiff (celldiff_r2 run-dir), and unetvit3d ER/mito
a549+joint train leaves from the old deconv-era `…/a549/mantis_v1/train/` to
the regenerated raw `…/a549/mantis/train/{SEC61B,TOMM20}_all.zarr` (A549 child
only in the joint leaves; iPSC dataset_v4 children unchanged). target_channel
stays `Structure`, which now names the raw GFP channel — so this is the
deconv->raw target flip for these families.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(dynacell): add pix2pix3d Run-D ER/mito train leaves (gap-fill)

Gap-fill for the A549 raw-flip retrain: neither ER nor mito a549/joint
pix2pix3d had a modernized leaf — the existing train.yml compose the stale
LSGAN base recipe (lambda_l1=100, no LeCam, 20ep, single GPU). Author
train_4gpu_modernized.yml for all 4 tuples ({er,mito}x{a549,joint}) by
copying the finalized membrane "Run-D" leaves and swapping the target to
Structure/SEC61B (ER) and Structure/TOMM20 (mito):

- modernized overlay (nonsat + R1 + G-EMA, equal LR) + lambda_l1=10 +
  lecam_gamma=0.3 + 40 epochs + monitor loss/validate_ema
- 4-GPU H200 DDP (hardware_4gpu + sbatch.constraint=h200; the GAN OOMs on
  80GB h100)
- A549 child reads the regenerated raw mantis/train store; joint iPSC child
  reads dataset_v4/train/{SEC61B,TOMM20}.zarr; joint batch_size=2 (=single/
  num_samples). Verified via --print-resolved-config.

The stale LSGAN train.yml are left in place (superseded by the modernized
leaf, as in the membrane family); they are not launched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(dynacell): sync bundled A549 manifest registry to mantis .zarr stores

Hand-sync the 12 VisCy bundled a549-mantis manifests (the registry the
resolver actually reads) to the regenerated canonical manifests: stores.test
-> …/a549/mantis/test/<stem>_<cond>.zarr, stores.train -> pooled
…/mantis/train/<stem>_all.zarr, with h2b/caax routed to the combined
dual_nucl_memb store (target_channel Nuclei/Membrane unchanged). VisCy-only
additive fields updated: cell_segmentation ->
…/mantis/test/<stem>_<cond>_seg_cleaned.zarr; gt_cache_dir unchanged
(…/eval_cache/<gene>_<cond>). test_manifest_sync (canonical ⊆ VisCy) green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(dynacell): migrate A549 predict/eval to combined dual_nucl_memb store

D8 merged the A549 nucleus (H2B) + membrane (CAAX) test stores into one
dual_nucl_memb_<cond>.zarr. Repoint the hardcoded (manifest-bypassing) A549
references that still pointed at the old separate .ozx stores:

- 3 dual predict-set data_path: mantis_v1/test/CAAX_<cond>.ozx ->
  mantis/test/dual_nucl_memb_<cond>.zarr (predict input for the VSCyto3D dual
  model).
- 10 hand-authored membrane eval leaves' nuclei_gt_path:
  mantis_v1/test/H2B_<cond>.ozx -> mantis/test/dual_nucl_memb_<cond>.zarr.

Whole-cell membrane eval needs no code change: nuclei_gt_path now equals the
membrane gt_path (same dual store), so the cross-store nuclei branch
(_separate_nuclei_path) self-disables and nuclei read in-store from the Nuclei
channel. Update the 4 tests that encoded the old separate-store/.ozx layout
(nuclei-store resolution, eval gt/seg suffixes, predict/train data_path
expectations) to the dual_nucl_memb/.zarr reality.

The generated grouped/instance-AP eval leaves are NOT touched here — they are
regenerated in Phase 10 (walk-based) and pick up the repointed nuclei store
from the manifest automatically.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(dynacell): sync nucleus/membrane manifest XY spacing to 0.1494

Mirror the canonical spacing fix into the VisCy bundled registry: the 6
a549-mantis-{h2b,caax}-<cond> manifests declared XY 0.116 (v2 source pitch)
but the dual_nucl_memb stores are resampled to 0.1494. The resolver reads
spacing from this registry, so eval must see 0.1494. Keeps canonical ⊆ VisCy
(test_manifest_sync green).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(dynacell): raise 4-GPU DDP process-group timeout to 3h

Two Wave-1 fcmae retrains (34683418, 34683421) died at ~11h with a NCCL
watchdog collective timeout (30 min) on the tiny epoch-end metric ALLREDUCE:
on the contended shared FS, per-epoch checkpoint writes (386 MB/ckpt across
~12 concurrent 4-GPU DDP jobs) stalled one rank past the default 30-min
process-group timeout, aborting the run mid-training.

Replace the string strategy `ddp_find_unused_parameters_true` with an explicit
DDPStrategy(find_unused_parameters=True, timeout=timedelta(hours=3)) in both
4-GPU DDP configs — the fcmae overlay (fcmae_vscyto3d_fit.yml) and the GAN
topology (ddp_4gpu_gan.yml, used by pix2pix3d). A 3h timeout tolerates
transient epoch-boundary I/O stalls while still failing a genuinely hung rank.
Verified: DDPStrategy(timeout=3h) instantiates; both leaves --print-resolved-
config cleanly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(dynacell): add linked README navigation tree

Add a navigable README.md system across applications/dynacell: every 1-level
subfolder (configs/, src/, examples/, tests/, tools/) plus key deeper folders
(configs/benchmarks[/virtual_staining], src/dynacell[/data,/_manifests,
/evaluation]) has a README with a purpose blurb, an annotated Contents list,
and Up/Subdirectories navigation links (relative markdown) wiring the tree
together. Existing rich docs (virtual_staining, evaluation) and the two
CLAUDE.md files are linked, not duplicated. Docs-only; no code touched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(dynacell): emit canonical paths from eval-config generators

Radiant path standardization (Stage A). Replace the local legacy path logic
in generate_grouped_eval_configs.py + generate_instance_ap_eval_configs.py
with paths.py calls: save_dir_for -> eval_leaf(...), pred_cache_dir_for ->
pred_cache_dir(...) (instance-AP via track="instance_ap"), and _CODE_TO_PAPER
model naming -> paths.PAPER_KEY. Drop the now-dead _PARENT_DIR/_DIR_INFIX; add
_TRAIN_SET_TO_CANONICAL for the *_trained spellings. Walk-based tuple discovery
over the on-disk prediction zarrs is unchanged — only the emitted output-path
grammar is canonicalized; gt_cache_dir stays manifest-frozen. Test expectations
updated to the canonical paths. 77 passed, ruff clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(dynacell): DDP timeout as h:m:s string, not class_path dict

The prior commit expressed the DDPStrategy timeout as a datetime.timedelta
class_path/init_args dict. That renders + instantiates in Python, but the
LightningCLI/jsonargparse FIT parse rejects it ("Not of type timedelta:
Expected a string with form h:m:s ... got {'class_path': ...}") — the two
fcmae resubmits died at ~40s (exit 2) before training. jsonargparse parses
timedelta from an "h:m:s" string, so use timeout: "3:00:00". Validated via
`dynacell fit --config <resolved> --print_config` (exit 0), which exercises
the real CLI parse the earlier --print-resolved-config check did not.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(dynacell): eval submitters use paths.eval_leaf + train_set

Radiant path standardization (Stage A). submit_evaluation_job.py and
submit_evaluation_batch.py now import from dynacell.evaluation.paths and build
the eval save_dir via eval_leaf(organelle, model, train_set, test_set,
condition) instead of the legacy save_paths.eval_save_dir. Read the training
set from benchmark.train_set with a transitional fallback to trained_on (the
predict-leaf codemod renames that key in a later stage). run_root heuristic
fixed for the deeper canonical leaf (save_dir.parent, not .parent.parent).
ruff clean; dry-run renders confirm canonical eval_leaf paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(dynacell): cross_condition probe handles canonical component subdirs

The canonical eval grammar nests dual-model components as
<test>__<cond>/<component>/ (e.g. a549__mock/nucleus/). The probe's
_detect_condition / _model_group_key matched only eval_dir.name via endswith,
so a component subdir (ending in nucleus/membrane, no condition token) raised
ValueError / dropped from grouping. Add _condition_and_leaf(eval_dir) which
walks up to the <test>__<cond> leaf (skipping a trailing component and any
deconv_gt/instance_ap subtrack); group key becomes (leaf.parent, leaf-cond,
component_rel) so conditions collapse per component while keeping a dual
model's two components in separate groups. Single-target leaves unchanged.
5 probe tests pass; pipeline imports clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(dynacell): add --resume flag to benchmark job launcher

Re-training resubmits must never restart from scratch: the standing
policy is to resume from the latest checkpoint. Add --resume (from
<run_root>/checkpoints/last.ckpt) and --resume-from CKPT (explicit path)
to submit_benchmark_job.py; both append --ckpt_path to the fit command
via a new @@resume_arg template placeholder. Fit mode only (predict has
its own checkpoint handling); the checkpoint must exist or submission
fails fast with a pointer to the cleanup step.

Motivated by the A549 raw-flip retrain matrix, where three fcmae scratch
runs hit the NCCL epoch-boundary watchdog at ~12-13h and restarting fresh
wasted ~10h each. LightningCLI fit restores model+optimizer+loop state,
so resume is bit-clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1Qzmc3NxeMTerNjzBJgZE

* feat(dynacell): add --ckpt best/last override for predict jobs

Phase-9 re-predict must use the retrained model's new best checkpoint,
but predict leaves hardcode model.init_args.ckpt_path at a specific epoch
(e.g. the A549 ER/mito leaves point at pre-flip deconv epochs like
epoch=137, dated May). Add --ckpt {best,last,PATH} to the launcher
(predict mode only; fit uses --resume): 'best' resolves the best-by-monitor
checkpoint from last.ckpt's ModelCheckpoint.best_model_path state in the
leaf's checkpoint dir (the recipe uses monitor=loss/validate, save_top_k=5
with the loss absent from the default filename, so the checkpoint state is
the authoritative source), with a highest-epoch fallback.

So Phase 9 becomes `predict ... --ckpt best` with no per-campaign leaf
churn and no risk of silently predicting from stale/deconv weights.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1Qzmc3NxeMTerNjzBJgZE

* feat(dynacell/tools): add public checkpoint-zoo assembly tool + docs

assemble_release_checkpoints.py builds (and idempotently updates) the
models/ tree of the public S3 mirror (dynacell_v1) from the checkpoints
pinned in the canonical benchmark predict__*.yml configs -- so the release
always matches what the evals consumed, with no hand-transcribed table.

Layout: {ipsc,a549,joint}/{nucleus,membrane,er,mito}/{model_slug}/ with the
original epoch=NNN-step=MMMM.ckpt name preserved (last/best_ep pins resolved
back to their canonical epoch sibling) + config.yaml + a top-level
checkpoints.csv manifest. Dry-run by default; --execute copies.

Model scope defaults to the paper set (fnet3d, unext2, vscyto3d, unetvit3d,
celldiff); pix2pix3d is excluded unless requested via --models. Cells are
classified resolved / pending / not_trained: the ER/Mito A549+Joint models
are deconvolved-fluorescence in v1 and flagged provenance=deconv->raw, so a
re-run after the raw-fluorescence retrains re-pin their predict configs
overwrites exactly those leaves. See RELEASING_CHECKPOINTS.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PYBc3W7UKZsstkVQoEJ7ik

* feat(dynacell/eval): add canonical_model_name for run-dir canonicalization

Phase-7 migration consolidates on-disk checkpoint run dirs into the
canonical tree, but the run-dir names carry training-recipe suffixes that
are not part of the model vocabulary: fcmae_vscyto3d_pretrained_ws8500
(warmup-8500 recipe of fcmae_vscyto3d_pretrained) and
pix2pix3d_unetvit_modernized_lambdaL1_10_lecam_40ep (a recipe of
pix2pix3d_unetvit). checkpoint_dir() takes `model` verbatim, so the
migration codemod must map each run-dir name to its code key first.

canonical_model_name() resolves the longest PAPER_KEY code key K such that
name == K or name startswith K + "_" (longest-first so an exact ablation
key or celldiff_r2 wins over its prefix), and raises on no match rather
than guessing. Distinct from resolve_model, which recovers the code from a
config benchmark block + ckpt path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1Qzmc3NxeMTerNjzBJgZE

* fix(dynacell): re-base best_model_path in --ckpt best resolver

`_resolve_best_ckpt` read the ModelCheckpoint `best_model_path` verbatim.
That value is stored as an absolute path into the directory where training
ran, so after a checkpoint dir is moved or renamed (e.g. the canonical-path
migration) `Path(best).is_file()` fails and resolution silently falls
through to the highest-epoch checkpoint — a strictly more overfit model
(monitor is loss/validate, min mode). Roughly 15/24 campaign models would
resolve to the wrong ckpt post-move, with no error.

Re-base the stored basename onto the current ckpt_dir (the best ckpt file
travels with the dir), preferring it over the literal stored path, before
the highest-epoch fallback. Add a regression test for the moved-dir case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1Qzmc3NxeMTerNjzBJgZE

* refactor(dynacell): drop the deconv_gt eval track from paths.py

The deconv-GT eval track was dropped for the A549 raw-flip campaign, but
paths.py still modeled it: eval_leaf/metrics_repo_dir emitted a deconv_gt/
subdir, gt_cache_dir emitted an _deconv variant, ORGANELLE_EVAL_TARGET had
a for_repr(gt_repr) deconv branch, and normalize_legacy routed the 33
iPSC-trained ER/mito A549 legacy deconv-GT eval dirs into gt_repr="deconv".
Migrating as-is would have materialized 33 leaves into the abandoned track.

Remove the gt_repr concept end to end: the parameter from eval_leaf,
gt_cache_dir, metrics_repo_dir, iter_organelle_evals; the gt_repr field
from CanonicalKey; the deconv eval-target map + for_repr (ORGANELLE_EVAL_TARGET
collapses to a plain dict). normalize_legacy now returns None for the legacy
iPSC-trained ER/mito A549 deconv-GT eval dirs so they are NOT migrated (left
in place), instead of colliding with a future raw-GT eval of the same model.
The a549__deconv / joint__legacy_deconvgt train_set relabels are unchanged
(distinct concept: model trained on deconv target, not GT representation).

No external consumer passed gt_repr. Update tests accordingly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1Qzmc3NxeMTerNjzBJgZE

* docs(cytoland): add generated notebooks for vcp tutorials

Regenerate the paired .ipynb notebooks from the jupytext percent-format
tutorial scripts (quick_start, hek293t, neuromast) so notebook users can
open them directly without running jupytext first. Generated with the
command documented in the tutorials README.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F2zb4bV8uFvVkEDiAwCp4T

* fix(dynacell): map nonstandard nucleus_ zarr prefix in normalize_legacy

The unetvit3d A549-test nucleus predictions were written with a nonstandard
`nucleus_<model>_<cond>.zarr` prefix instead of the canonical `nucl_`, so
normalize_legacy skipped all three (mock/denv/zikv) even though they have
consuming eval dirs and no `nucl_` A549 alternate — they would have been
silently stranded by the path migration.

Add `nucleus` to _ZARR_ORG_PREFIX; _starts_with_longest already matches the
longest prefix, so `nucleus_` wins over `nucl_` with no ordering change and
no regression to `nucl_` zarrs. The migration planner now maps every A549
prediction (0 genuine gaps).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1Qzmc3NxeMTerNjzBJgZE

* fix(cytoland): install unpublished tutorial deps from GitHub

The vcp_tutorials install cells listed `cytoland` and `viscy-utils`, which
are not published to PyPI (only viscy-data/models/transforms are), so
`pip install cytoland ...` failed with "No matching distribution found for
cytoland". Install both unpublished packages from the GitHub monorepo via
git+subdirectory URLs; their siblings still resolve from PyPI. Applies to
quick_start, hek293t, and neuromast, with regenerated notebooks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F2zb4bV8uFvVkEDiAwCp4T

* fix(dynacell/eval): raise on missing backbone for non-SAM cellpose models

load_cellpose_model unconditionally set model.backbone="sam_vitl" when the attr
was missing, contradicting the comment and silently mislabelling DINO models
(cpdino/cpdino-vitb) as SAM — cubic would then pick the wrong tile size. Apply the
4.1.x fallback only for the SAM models (cpsam/cpsam_v2); raise a clear
upgrade-cellpose error otherwise. Addresses Copilot review on #479.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAaYiNdzcCjoJWYopUqteD

* fix(dynacell): skip nonconforming epoch=*.ckpt in --ckpt best fallback

The highest-epoch fallback in _resolve_best_ckpt sorted every epoch=*.ckpt by
int(re.match(r"epoch=(\d+)", name).group(1)); a nonconforming file (e.g.
epoch=final.ckpt) makes re.match return None and crashes mid-submit with
AttributeError. Filter to files matching epoch=<digits> and take the max; skip the
rest. +regression test. Addresses Copilot review on #479.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAaYiNdzcCjoJWYopUqteD

* feat(dynacell): add ViT sliding-window inference for large test volumes

The UNetViT3D generator is fixed at its trained input_spatial_size (512x512),
so it cannot run a forward pass on the new 640x960 A549 test set — it raises
`x spatial size ... does not match expected ...`. Fully-convolutional models
(FCMAE/FNet) are unaffected.

Add a module-level `_sliding_window_inference` helper that tiles an input into
input_spatial_size windows with overlap-averaging (the deterministic analog of
CellDiff's flow-matching conditional inpainting, which is inapplicable to a
single-pass generator). Refactor `DynacellUNet.predict_sliding_window` to use
it and add a `predict_method='sliding_window'` branch to
`DynacellGAN.predict_step` (pix2pix3d), which previously supported only
`full_image`. A 512x512 input degenerates to a single tile, so iPSC-test
predictions are unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAaYiNdzcCjoJWYopUqteD

* feat(dynacell): add --resume-predict for wall-time-killed predict jobs

Predict jobs were not resumable: a plain resubmit crashes because the
HCSPredictionWriter raises FileExistsError on the already-written
prediction channel (overwrite=False), and --overwrite alone re-runs
every FOV from scratch. For long CELL-Diff predicts that cannot fit all
FOVs in one 4-day wall window, --overwrite would restart at FOV 0 each
time and never finish.

--resume-predict skips FOVs already fully written (via the DataModule's
exclude_fov_names hook) and sets overwrite=True so the kept partial FOV
and any re-done FOVs can be rewritten. Completeness is per-FOV: a FOV is
done when its written T dimension equals the input FOV's T. Predict
configs set z_window_size to the full stack, so each timepoint is one
write_sample call and T grows monotonically — a crash mid-FOV leaves
T < input T. Detection is metadata-only (reads array shapes, not voxels).

Reuses the leaf's checkpoint; rejects --ckpt combos (would mix
predictions from two models) and fit mode.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HAaYiNdzcCjoJWYopUqteD

---------

Co-authored-by: Alexandr Kalinin <alxndrkalinin@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>