CARMEN

flowchart LR
  IN["9 modalities @ 100 Hz<br/>ECG · ABP · PPG · CVP · CO2<br/>AWP · ICP · RESP-Imp · RESP-Flow"]
  IN --> SC["Scaler<br/>per-variate loc / scale"]
  SC --> PT["Patchify<br/>200 samples = 2 s / token"]
  PT --> PE["Residual-MLP projection<br/>+ modality embedding"]
  PE --> TR["Transformer encoder<br/>GQA · GLU FFN · RoPE · LSCNorm"]
  SC -.->|"loc / scale as AdaLN conditioning"| TR
  TR --> O1["reconstruction head<br/>→ features · anomaly scoring"]
  TR --> O2["cross-modal heads<br/>→ cross-modal reconstruction"]
  TR --> O3["block next-patch head<br/>→ forecasting · roll-out"]

CARMEN is a foundation model for the continuous waveforms recorded at the bedside and in the operating room. A single Transformer encoder (~30M parameters) is pretrained across 9 signal modalities and transfers to three families of task without architectural surgery: feature extraction for downstream heads, cross-modal waveform reconstruction, and waveform forecasting.

Two design choices carry most of the weight. Every modality is tokenized the same way — raw patches, one shared encoder — so a single model covers all nine instead of one model per signal. And because per-patient normalization would otherwise throw away the absolute level of a pressure waveform, the (loc, scale) stripped out by the scaler is fed back into every layer as AdaLN modulation (LSCNorm), keeping clinically meaningful magnitudes available to the encoder.

The nine modalities, grouped by what drives the waveform:

🫀  Cardiovascular  ·  locked to the cardiac cycle
	0	ECG	electrocardiogram
	1	ABP	arterial blood pressure, invasive
	2	PPG	photoplethysmography — peripheral pulse
	3	CVP	central venous pressure
	6	ICP	intracranial pressure
🫁  Respiratory  ·  locked to the ventilation cycle
	4	CO2	capnography — expired CO₂
	5	AWP	airway pressure
	7	RESP_Impedance	chest-impedance respiration
	8	RESP_Flow	ventilator flow

git clone https://github.com/dlcjfgmlnasa/CARMEN.git && cd CARMEN
pip install -e .              # or: pip install -r requirements.txt

Requires Python ≥ 3.10, PyTorch ≥ 2.2, einops ≥ 0.7.

Pretrained weights will be published here upon acceptance of the paper. Until then this repository ships the model implementation and inference API only — the release asset referenced below is not yet available. You can still build the model from a config and run a forward pass without weights (see examples/00_smoke_test.py).

Weights are distributed as a GitHub Release asset and are not committed to git. Download a checkpoint into checkpoints/ — see checkpoints/README.md:

curl -L -o checkpoints/carmen.pt \
  https://github.com/dlcjfgmlnasa/CARMEN/releases/download/v1.0/carmen.pt

Each checkpoint embeds its own ModelConfig, so the architecture is reconstructed automatically — you never specify it by hand.

import torch
from carmen import DownstreamModelWrapper, make_batch

# 1. Load the pretrained encoder (frozen, eval mode)
wrapper = DownstreamModelWrapper("checkpoints/carmen.pt", device="cpu")

# 2. Pack raw 1-D signals (100 Hz) into a batch — one patient, multiple modalities
t = torch.linspace(0, 30, 3000)
batch = make_batch(
    [("ecg", torch.sin(2*torch.pi*1.2*t)),
     ("ppg", torch.sin(2*torch.pi*1.2*t - 0.6)),
     ("abp", 80 + 30*torch.sin(2*torch.pi*1.2*t - 0.3))],
    patch_size=wrapper.patch_size,
)

# 3. One feature vector per patient
features = wrapper.extract_features(batch)   # (B, d_model)

No checkpoint yet? python examples/00_smoke_test.py verifies the install against a randomly initialized model.

CARMEN.from_pretrained(path) returns the bare encoder; DownstreamModelWrapper adds loading, freezing, pooling and LoRA on top. batch below is what make_batch returns.

from carmen import CARMEN, DownstreamModelWrapper

model   = CARMEN.from_pretrained("checkpoints/carmen.pt")                  # bare encoder
wrapper = DownstreamModelWrapper("checkpoints/carmen.pt", device="cuda")   # + freeze / pool / LoRA

# ── Representations ──────────────────────────────────────────────────────
feats = wrapper.extract_features(batch)                 # (B, d_model)      pooled, frozen
feats = wrapper.extract_features(batch, pool="none")    # (B, N, d_model)   per patch
enc   = model.extract_features(batch)                   # dict of raw encoder outputs

# ── Generation ───────────────────────────────────────────────────────────
abp   = model.generate_cross_modal(batch, target_signal_type=1)["waveform"]
                                                        # (B, N, patch_size)
pred  = model.forecast(batch)                           # (B, N, K, patch_size)
roll  = model.generate(batch, n_steps=10)               # (n_steps, B, patch_size)

# ── Adaptation & scoring ─────────────────────────────────────────────────
wrapper.inject_lora(rank=8)                             # LoRA on q_proj / v_proj
score = wrapper.get_reconstruction_loss(batch, mask)    # scalar MSE, anomaly scoring

from carmen import BiosignalSample, PackCollate, CHANNEL_NAME_TO_SIGNAL_TYPE

sample = BiosignalSample(
    values=ecg,                 # 1-D tensor @ 100 Hz
    length=ecg.numel(),
    channel_idx=0, recording_idx=0, n_channels=1, win_start=0,
    sampling_rate=100.0,
    signal_type=CHANNEL_NAME_TO_SIGNAL_TYPE["ECG II"],   # -> 0
    session_id="patient-001",   # samples sharing a session are paired cross-modally
    start_sample=0,
)
batch = PackCollate(max_length=8192, patch_size=200)([sample])

📓  quickstart.ipynb — end to end: build → features → generate

• 00_smoke_test.py — build from config and run a forward, no weights needed
• 01_extract_features.py — load a checkpoint, extract pooled features
• 02_downstream_probe.py — linear probe and LoRA on frozen features
• 03_cross_modal_generation.py — ECG + PPG → ABP
• 04_forecasting.py — block forecast and autoregressive roll-out

python examples/00_smoke_test.py
python examples/01_extract_features.py checkpoints/carmen.pt

carmen/
├── model.py         CARMEN encoder + inference API
├── config.py        ModelConfig (embedded in every checkpoint)
├── checkpoint.py    checkpoint save / load
├── wrapper.py       DownstreamModelWrapper (load / freeze / LoRA), LinearProbe
├── batch.py         make_batch / to_device — raw signals -> PackedBatch
├── loss.py          MaskedPatchLoss (reconstruction scoring)
├── data/            PackCollate (bin-packing), BiosignalSample, signal-type maps
└── modules/         attention (GQA), GLU FFN, RMSNorm / LSCNorm, patch embedding,
                     packed scalers, RoPE / attention bias
examples/            runnable examples + quickstart notebook
checkpoints/         put downloaded weights here (gitignored)

• Cross-modal reliability. Not every source → target pair is physiologically supported. The pairs the model was trained to transfer across are listed in carmen.CROSS_PRED_ALLOWED_PAIRS (ECG↔ABP, ECG↔PPG, ABP↔PPG, AWP↔RESP_Flow).
• Denormalized output is approximate. generate_cross_modal(..., denormalize=True) rescales with the source signal's loc/scale, because the target's own level is unknown. Treat the absolute magnitude accordingly.
• any_variate batching is non-deterministic. PackCollate trims a patient's variates to one common length so they pair up; with unequal-length inputs that length is drawn at random. Seed random.seed() if you need reproducible batches, or use collate_mode="ci".

This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number : RS-2024-00439677 , NTIS number:2460003917)

A paper describing CARMEN is in preparation. Until it is out, please cite this repository:

@software{carmen2026,
  title  = {CARMEN: A Cardiorespiratory Foundation Model for Continuous Physiological Waveforms},
  author = {The CARMEN Authors},
  year   = {2026},
  url    = {https://github.com/dlcjfgmlnasa/CARMEN}
}

Released under the Apache License 2.0.