An end-to-end biomedical computer vision pipeline for nuclei segmentation, instance separation, quantitative evaluation, failure analysis, and morphological profiling in microscopy images.

The project develops a nuclei-analysis system progressively from classical image processing to a custom compact U-Net, validation-tuned watershed instance separation, and external benchmarking against pretrained StarDist and Cellpose-SAM models.

Evaluation is performed on a frozen 102-image test set containing 3,678 annotated nuclei.

Instance matching threshold: IoU ≥ 0.50

Cellpose-SAM achieved the strongest overall benchmark performance.

The custom pipeline, however, was developed independently from classical preprocessing through semantic segmentation, failure analysis, robustness experimentation, validation-based instance-separation tuning, and quantitative morphology.

StarDist and Cellpose-SAM were subsequently evaluated as pretrained external benchmarks against the same frozen test set.

Microscopy segmentation involves more than predicting whether individual pixels belong to nuclei.

A useful biomedical image-analysis system should be able to:

1. identify nuclear regions,
2. separate touching or overlapping nuclei,
3. detect individual objects,
4. quantify segmentation quality,
5. measure nuclear morphology,
6. identify model failure modes,
7. compare alternative segmentation approaches.

The project therefore follows the experimental progression:

Microscopy Images
        │
        ▼
Classical Computer Vision Baseline
        │
        ▼
Custom U-Net Semantic Segmentation
        │
        ▼
Failure Analysis
        │
        ▼
Robustness Experiment
        │
        ▼
Marker-Controlled Watershed
        │
        ▼
Validation-Based Watershed Tuning
        │
        ▼
Instance-Level Evaluation
        │
        ▼
Morphological Profiling
        │
        ▼
External Model Benchmarking
        ├── StarDist
        └── Cellpose-SAM

This progression allows pixel-level segmentation, object-level detection, counting, morphology, robustness, and external model performance to be evaluated separately.

The project uses microscopy nuclei images from the 2018 Data Science Bowl dataset.

Each sample contains one microscopy image and an independent binary mask for every annotated nucleus.

sample_id/
├── images/
│   └── sample_id.png
└── masks/
    ├── nucleus_1.png
    ├── nucleus_2.png
    └── ...

A deterministic split was generated once and stored in:

data/splits.json

The 102-image test set remained frozen during model development and watershed parameter tuning.

The test set contains:

3,678 annotated nuclei

Raw microscopy images are excluded from the Git repository.

An interpretable non-learning segmentation pipeline was implemented first.

RGB Image
    │
    ▼
Grayscale
    │
    ▼
CLAHE
    │
    ▼
Gaussian Denoising
    │
    ▼
Otsu Thresholding
    │
    ▼
Morphological Cleanup
    │
    ▼
Distance Transform
    │
    ▼
Foreground Markers
    │
    ▼
Watershed

The pipeline combines:

• CLAHE contrast enhancement
• Gaussian filtering
• Otsu thresholding
• morphological opening and closing
• distance transforms
• connected-component markers
• marker-controlled watershed segmentation

The classical pipeline remained competitive on images whose intensity distributions matched its thresholding assumptions but was less reliable across the heterogeneous microscopy domains present in the dataset.

A compact U-Net was implemented from scratch using PyTorch.

The network contains:

• convolutional encoder blocks,
• max-pooling downsampling,
• bottleneck features,
• transposed-convolution upsampling,
• encoder-decoder skip connections,
• single-channel segmentation logits.

Approximate model characteristics:

Trainable parameters: 483,441
Checkpoint size:      ~1.88 MB

The compact architecture was intentionally selected to make experimentation inexpensive while retaining the core U-Net encoder-decoder design.

Input resolution:   128 × 128
Optimizer:          Adam
Loss:               BCEWithLogitsLoss + Dice Loss
Batch size:         8
Training images:    468
Validation images:  100

Best validation Dice:

0.8582

Selected checkpoint:

models/best_unet.pth

Both approaches were evaluated using the same frozen 102-image test split.

Absolute improvement:

Dice: +0.0325
IoU:  +0.0482

The U-Net achieved higher Dice on:

76 / 102 images

while:

Classical CV better: 25 images
Equal:                 1 image

The learned model therefore improved average segmentation quality without uniformly dominating the handcrafted baseline.

Dataset-level averages can hide severe individual failures.

The dataset contains substantial visual heterogeneity, including:

• dark fluorescence images,
• bright-background microscopy,
• dense nuclear populations,
• elongated structures,
• unusual staining,
• substantial contrast and intensity variation.

A dedicated failure-analysis pipeline was implemented to compare classical CV and U-Net predictions at the image level.

Five severe U-Net failures were identified using:

Dice < 0.50

Strong U-Net cases:

Mean image intensity: 30.67
Mean contrast:        16.26

Severe failure cases:

Mean image intensity: 138.21
Mean contrast:         25.50

This suggests sensitivity to appearance and domain variation.

Classical Dice: 0.067
U-Net Dice:     0.718
Gain:          +0.651

Classical Dice: 0.839
U-Net Dice:     0.472
Difference:    -0.367

The experiment demonstrates why biomedical computer-vision systems should be examined at both aggregate and individual-image levels.

A second U-Net training experiment introduced stronger appearance augmentation to investigate whether the severe domain-shift failures could be reduced.

The experiment partially succeeded.

Mean Dice across the severe baseline failure cases increased from:

0.0952 → 0.1668

Several previously failed samples recovered partially.

However, overall test performance decreased:

Baseline U-Net Dice: 0.8353
Robust U-Net Dice:   0.8278

Baseline IoU:        0.7480
Robust IoU:          0.7305

Across the test set:

Improved images: 19
Degraded images: 82
Unchanged:        1

The augmented checkpoint was therefore not selected as the final model.

This negative result is intentionally retained because it demonstrates an important trade-off: improving selected domain-shift failures does not necessarily improve overall generalization.

Semantic segmentation answers:

Which pixels belong to nuclei?

It does not necessarily answer:

How many individual nuclei are present?

Touching nuclei can form one connected foreground region.

The U-Net semantic prediction was therefore converted into individual instances using marker-controlled watershed segmentation.

U-Net Probability Map
        │
        ▼
Binary Semantic Mask
        │
        ▼
Distance Transform
        │
        ▼
Local-Maxima Detection
        │
        ▼
Watershed Markers
        │
        ▼
Marker-Controlled Watershed
        │
        ▼
Individual Nuclear Instances

Watershed performance depends strongly on marker-generation parameters.

Three parameters were optimized:

min_distance
peak_rel_threshold
min_object_size

A grid search containing 45 configurations was performed exclusively on the 100-image validation split.

The test set was not used for parameter selection.

min_distance        = 9
peak_rel_threshold  = 0.20
min_object_size     = 15

Validation performance:

Micro Precision:  0.8448
Micro Recall:     0.7191
Micro F1:         0.7769

Macro F1:         0.8145
Matched IoU:      0.8040
Count MAE:        7.11

These parameters were frozen before final test evaluation.

Predicted nuclei were matched one-to-one against ground-truth nuclear masks.

A predicted object is considered correctly detected when:

IoU(prediction, ground truth) ≥ 0.50

Ground-truth nuclei:  3,678
Predicted instances:  3,013

True positives:       2,459
False positives:        554
False negatives:      1,219

Median per-image detection rate:

0.8377

Before tuning:

Instance Micro F1: 0.6308
Count MAE:         13.34

After validation-based tuning:

Instance Micro F1: 0.7350
Count MAE:          9.72

Improvement:

F1:        +0.1042
Count MAE: -3.62 nuclei/image

This experiment demonstrates that post-processing is not merely cosmetic.

Carefully optimized instance-separation parameters substantially changed the object-level performance of an otherwise unchanged semantic segmentation model.

After instance separation, each detected nucleus can be converted into quantitative measurements.

The pipeline extracts:

Area
Perimeter
Circularity
Eccentricity
Solidity
Mean intensity
Centroid coordinates

Morphological profiling was performed on a 30-image cohort.

Nuclei profiled:   1,016
Mean area:         512.08 px²
Median area:       284.00 px²
Mean circularity:  0.887
Mean eccentricity: 0.650
Mean solidity:     0.955
Mean intensity:    68.93

This converts segmentation output into structured quantitative measurements suitable for downstream biomedical image analysis.

The final pipeline visualization deliberately includes:

• a best-performing case,
• a representative case,
• a failure case.

For each case the figure displays:

Original microscopy image
Ground truth
U-Net probability map
Semantic prediction
Watershed instance segmentation
Morphological centroids

The failure case is intentionally retained rather than removed from the presentation.

This makes the limitations of the custom model visible instead of reporting only favorable examples.

The project was subsequently benchmarked against the pretrained:

StarDist 2D_versatile_fluo

model.

StarDist directly predicts star-convex nuclear instances and therefore provides an external instance-segmentation baseline fundamentally different from the custom semantic-U-Net-plus-watershed pipeline.

The same frozen 102-image test set was used.

Mean Dice:   0.7667
Median Dice: 0.9142

Mean IoU:    0.7015
Median IoU:  0.8419

Ground-truth nuclei: 3,678
Predicted instances: 3,279

True positives:      2,576
False positives:       703
False negatives:     1,102

StarDist produced slightly higher instance F1 than the custom U-Net + watershed system:

Custom pipeline: 0.7350
StarDist:        0.7405

while achieving substantially lower counting error:

Custom pipeline: 9.72
StarDist:        7.54

Its mean semantic Dice was lower because the pretrained model exhibited severe failures on several heterogeneous microscopy domains.

A second external benchmark was performed using:

Cellpose-SAM cpsam_v2

The model was evaluated on the same frozen 102-image test set.

Inference was performed using an NVIDIA GPU.

Mean Dice: 0.9030
Mean IoU:  0.8424

Micro Precision: 0.9111
Micro Recall:    0.8804
Micro F1:        0.8955

Mean matched IoU: 0.8517
Count MAE:        3.78

Cellpose-SAM achieved the strongest overall benchmark performance.

It substantially improved both instance detection and counting accuracy compared with the custom pipeline and StarDist.

Importantly, this benchmark is presented as an external pretrained-model comparison, not as a component trained as part of the custom pipeline.

The comparison illustrates three different approaches to nuclei segmentation:

Highly interpretable and computationally inexpensive, but dependent on image-processing assumptions.

A compact task-specific learned semantic model combined with classical instance separation.

A specialized pretrained nuclear instance-segmentation model based on star-convex object representations.

A large pretrained generalist segmentation model that achieved the strongest overall performance on this test set.

The objective of the project is therefore not to claim that the compact custom model surpasses large pretrained systems.

Instead, the project demonstrates the complete process of building, diagnosing, improving, evaluating, and benchmarking a biomedical computer-vision pipeline.

The project reports both pixel-level and object-level performance because semantic overlap alone does not determine whether individual nuclei are correctly detected.

microscopy-cell-analysis/
│
├── data/
│   └── splits.json
│
├── models/
│   ├── best_unet.pth
│   └── best_unet_robust.pth
│
├── outputs/
│   │
│   ├── features/
│   │   ├── classical_test_results_full.csv
│   │   ├── unet_test_results_full.csv
│   │   ├── robust_unet_test_results_full.csv
│   │   ├── baseline_vs_robust_unet.csv
│   │   ├── model_failure_analysis.csv
│   │   ├── instance_evaluation_results.csv
│   │   ├── watershed_tuning_results.csv
│   │   ├── stardist_test_results.csv
│   │   ├── cellpose_test_results.csv
│   │   └── ...
│   │
│   └── figures/
│       ├── classical_segmentation_pipeline.png
│       ├── analysis_summary.png
│       ├── failure_analysis_gallery.png
│       ├── final_pipeline_results.png
│       ├── final_results_dashboard.png
│       └── model_comparison.png
│
├── src/
│   ├── explore_dataset.py
│   ├── create_splits.py
│   ├── dataset.py
│   │
│   ├── classical_segmentation.py
│   ├── evaluate_classical.py
│   │
│   ├── unet.py
│   ├── train.py
│   ├── evaluate_unet.py
│   │
│   ├── train_robust.py
│   ├── evaluate_robust_unet.py
│   ├── analyze_failures.py
│   │
│   ├── evaluate_instances.py
│   ├── tune_watershed.py
│   │
│   ├── extract_morphology.py
│   ├── morphology_analysis.py
│   │
│   ├── evaluate_stardist.py
│   ├── evaluate_cellpose.py
│   │
│   ├── visualize_results.py
│   ├── create_final_figure.py
│   ├── create_final_dashboard.py
│   └── create_model_comparison.py
│
├── requirements.txt
├── .gitignore
└── README.md

git clone <repository-url>
cd microscopy-nuclei-segmentation

Windows:

python -m venv .venv
.\.venv\Scripts\Activate.ps1

Linux/macOS:

python -m venv .venv
source .venv/bin/activate

Install dependencies:

python -m pip install -r requirements.txt

Core libraries include:

PyTorch
OpenCV
NumPy
Pandas
Matplotlib
SciPy
scikit-image

Place the extracted Data Science Bowl training samples under:

data/raw/stage1_train/

Expected structure:

data/raw/stage1_train/<sample_id>/images/
data/raw/stage1_train/<sample_id>/masks/

Raw data is intentionally ignored by Git.

python src/create_splits.py

python src/train.py

python src/evaluate_unet.py

python src/evaluate_classical.py

python src/evaluate_instances.py

Watershed tuning uses the validation set only.

python src/tune_watershed.py

The selected parameters should then remain frozen during final test evaluation.

python src/analyze_failures.py

python src/extract_morphology.py

python src/create_final_figure.py
python src/create_final_dashboard.py
python src/create_model_comparison.py

StarDist and Cellpose have substantially different dependency stacks from the core PyTorch pipeline.

Separate environments are recommended to avoid dependency conflicts.

Example:

python -m venv .venv-stardist
.\.venv-stardist\Scripts\Activate.ps1

The benchmark used:

StarDist:   0.9.2
TensorFlow: 2.21.0
NumPy:      1.26.4
Model:      2D_versatile_fluo

Run:

python src/evaluate_stardist.py

Example:

python -m venv .venv-cellpose
.\.venv-cellpose\Scripts\Activate.ps1

The benchmark used:

Cellpose: 4.2.1.1
Model:    cpsam_v2

GPU acceleration is strongly recommended.

The benchmark environment detected:

CUDA available: True
GPU: NVIDIA GeForce RTX 4050 Laptop GPU

Run:

python src/evaluate_cellpose.py

The code should select GPU execution when CUDA is available.

The classical pipeline provides an interpretable baseline.

It also reveals cases where handcrafted assumptions remain effective and prevents treating deep learning as automatically superior.

The objective was to implement and understand a complete segmentation system rather than only invoke pretrained models.

The custom U-Net provides control over:

• data preparation,
• architecture,
• loss functions,
• training,
• probability prediction,
• thresholding,
• failure analysis,
• post-processing.

High semantic Dice does not guarantee correct nucleus counting.

Several touching nuclei can be represented by one large foreground region while still achieving substantial pixel overlap.

For that reason the project evaluates:

Dice
IoU

Precision
Recall
F1
Matched-instance IoU
Count MAE

Selecting post-processing parameters on the test set would leak information from the final evaluation set into system development.

Watershed parameters were therefore selected exclusively using the validation split and frozen before test evaluation.

The robustness experiment did not improve overall test performance.

Removing it would hide useful information.

Instead, the experiment demonstrates that targeted augmentation can improve selected failure domains while simultaneously degrading the dominant distribution.

Failure analysis and negative experiments are therefore treated as part of the model-development process.

A custom model should not be evaluated in isolation.

StarDist provides a specialized nuclei instance-segmentation baseline, while Cellpose-SAM represents a substantially larger pretrained generalist segmentation approach.

Their inclusion provides context for the strengths and limitations of the compact custom pipeline.

The custom U-Net exhibits severe failures on a small subset of images whose appearance differs substantially from the dominant training distribution.

Custom U-Net inference uses:

128 × 128

images.

This reduces computational cost but can remove fine boundary information and small nuclear structures.

The custom pipeline achieved:

Precision: 0.8161
Recall:    0.6686

The system therefore tends to miss or merge some nuclei rather than generating excessive false-positive instances.

Instance performance depends on handcrafted watershed parameters.

Although these were selected correctly using validation data, the optimal parameters may vary across microscopy domains.

Morphological features are extracted from predicted instances.

Segmentation errors therefore propagate directly into downstream measurements such as area, circularity, and eccentricity.

The project contains 670 microscopy images.

Performance on this dataset should not be interpreted as evidence of clinical or laboratory deployment readiness.

StarDist and Cellpose-SAM were evaluated using pretrained weights.

Their results therefore represent transfer performance from substantially larger external training regimes and are not directly equivalent to training the compact U-Net only on this project's training split.

Potential extensions include:

• higher-resolution U-Net training,
• pretrained encoder backbones,
• residual U-Net variants,
• boundary-aware losses,
• multi-scale inference,
• stronger domain normalization,
• test-time augmentation,
• uncertainty estimation,
• learned instance-separation approaches,
• GPU training of the custom architecture,
• cross-dataset generalization experiments,
• morphology validation against ground-truth instances,
• systematic inference-efficiency benchmarking under identical hardware.

• image preprocessing
• contrast enhancement
• thresholding
• morphology
• distance transforms
• connected components
• marker-controlled watershed

• PyTorch
• U-Net implementation
• segmentation losses
• training and validation
• model checkpointing
• GPU and CPU inference environments

• semantic nuclei segmentation
• nuclear instance separation
• object matching
• nuclei counting
• morphological profiling

• Dice
• IoU
• precision
• recall
• F1
• matched-instance IoU
• count MAE
• failure-case analysis

• deterministic train/validation/test splits
• frozen test evaluation
• validation-only hyperparameter tuning
• classical baseline comparison
• negative-result reporting
• domain-shift analysis
• pretrained external benchmarking

• classical image processing
• custom U-Net
• StarDist
• Cellpose-SAM

This project develops and evaluates an end-to-end microscopy nuclei-analysis pipeline rather than treating segmentation as an isolated model-training task.

The custom U-Net improved semantic segmentation over the classical baseline:

Classical Dice: 0.8028
U-Net Dice:     0.8353

Classical IoU:  0.6998
U-Net IoU:      0.7480

Validation-tuned watershed substantially improved instance separation:

Instance F1:
0.6308 → 0.7350

Count MAE:
13.34 → 9.72

Failure analysis exposed significant domain-shift behaviour, while a robustness experiment demonstrated the trade-off between recovering difficult cases and preserving overall performance.

The final system was then benchmarked against specialized pretrained segmentation models:

                         Dice     Instance F1    Count MAE

Custom U-Net + WS       0.8353       0.7350         9.72
StarDist                 0.7667       0.7405         7.54
Cellpose-SAM             0.9030       0.8955         3.78

Cellpose-SAM achieved the strongest overall benchmark performance.

The central contribution of the project is therefore the complete experimental workflow:

baseline
→ model development
→ failure analysis
→ robustness testing
→ instance separation
→ validation-based optimization
→ quantitative morphology
→ external benchmarking

The repository demonstrates practical computer-vision engineering across classical image processing, deep learning, biomedical segmentation, object-level evaluation, experimental design, and quantitative image analysis.