Skip to content

Multi-sample runs: UPDATE_FROM_OME duplicates markers when samples differ in exposure time #165

Description

@adamjtaylor

Description of the bug

Multi-sample runs fail because the shared markers.csv is inflated with duplicate marker rows. The root cause is that UPDATE_FROM_OME dedupes markers on a map that still includes per-sample exposure metadata, so any channel whose exposure time differs across samples produces one row per distinct exposure. Single-sample runs are unaffected, which is why existing test data never caught it.

MCQUANT aborts with:

Exception: The number of channels in markers.csv doesn't match the image

The image is fine — instrumenting n_channels() in-container returns the correct count, and that count matches the rows of the input markers.csv. But the UPDATE_FROM_OME generated markers.csv has 64 rows instead of 44: the 44 correct unique marker_names, ragged-duplicated (DNA8 ×4, DNA3 ×3, GFAP ×3, several ×2).

The samples were imaged with different per-channel exposure times.

  • modules/local/omevalidation reads <Plane ExposureTime> into each marker's metadata (exposure_time, exposure_time_unit).
  • subworkflows/local/update_from_ome/main.nf builds one shared markersheet across all samples and dedupes it with .unique() on maps that still carry exposure_time.

So any (cycle, channel) whose exposure differs across samples is never collapsed, yielding one row per distinct exposure.

In the triggering run, of 44 (cycle, channel) pairs, 16 have more than one distinct exposure time across the 4 samples. Those extras sum to 20, giving 44 + 20 = 64. Per-marker multiplicity matches exactly (e.g. cycle 8 / ch 29 has 4 distinct exposures → DNA8 ×4).

The guard meant to catch this ("Inconsistent marker exposure data across samples") never fires: it .maps over the single toSortedList emission as if it were one marker (entry.cycle_number on a List), so the condition is never evaluated.

Scope / other triggers

The deeper flaw is that the shared markersheet merges per-sample OME metadata and then dedupes on the whole map, so any per-sample-varying field inflates it. Today the only such fields are exposure_time / exposure_time_unit (everything else comes from the single shared markers.csv), so exposure is the practical trigger — including the sub-case where some samples have a null exposure and others don't.

A related path produces the same symptom: if samples disagree on a cycle's channel_count, CHANNEL_DELTA_INDEX/agg gets duplicate cycle keys and combine(agg, by:0) fans the markers out. The per-sample count check in fix 3 below covers all of these at once.

Suggested fixes:

Generate the markersheet per sample. Run UPDATE_FROM_OME (and the markers.csv construction) per meta.id, so each MCQUANT receives a sheet built from and validated against its own image. This treats differing per-sample acquisition metadata (e.g. exposure times) as legitimate and removes the cross-sample .unique() that causes the inflation, covering the exposure, channel_count, and any-future-field triggers in one change.

  • Key the markers.csv collectFile/combine in workflows/mcmicro.nf:178 by meta.id (today it broadcasts one shared sheet to every sample).
  • The cross-sample consistency guard (update_from_ome L90-99) is then unnecessary; if enforcing identical panels across a cohort is still wanted, make it an explicit opt-in check (and fix it — it currently no-ops over the toSortedList emission).
  • Defense-in-depth: a PRELUDE assertion that each sample's marker count equals its image SizeC, failing with a clear message.
  • Independently, make MCQUANT's error report both counts + the marker file path.

Command used and terminal output

Relevant files

No response

System information

  • nf-core/mcmicro dev @ 54a9c89
  • Nextflow 25.10.6

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions