New York Smells

Unofficial implementation of COIP (Contrastive Olfaction-Image Pretraining) from New York Smells: A Large Multimodal Dataset for Olfaction — training, evaluation, and pretrained checkpoints.

Paper · Project page · Dataset · Models

Machines have no sense of smell, largely because there is no data. This dataset pairs 7,000 images with readings from a Cyranose 320 electronic nose — 32 conducting-polymer sensors — captured in the wild across New York City. COIP learns a joint embedding between the two modalities, so that a smell can retrieve the image of what produced it, and a linear probe on the olfaction embedding can name the scene, object, or material.

git clone https://github.com/cvondrick/new-york-smells && cd new-york-smells
python -m venv .venv && source .venv/bin/activate
pip install -e .

PyTorch must match your CUDA driver — for CUDA 12.6:

pip install --index-url https://download.pytorch.org/whl/cu126 torch torchvision

Embed smell and vision in a shared space, with no dataset download:

import torch
from nys.models import from_pretrained

model = from_pretrained("transformer")            # downloads ~96 MB, cached

image = torch.randn(4, 3, 224, 224)               # ImageNet-normalised
smell = torch.randn(4, 28, 32)                    # 28 timesteps x 32 sensors

image_emb, smell_emb = model(image, smell)        # (4, 128) each, L2-normalised
similarity = smell_emb @ image_emb.t()            # cosine similarity

Four olfaction encoders are released — transformer, cnn, mlp, smellprint.

python -m nys.data.download --root data/new-york-smells

That fetches the parquet configs -- 31 files, ~16 GB, a few minutes. Add --layout olfaction for the 44 MB sensors-only config, which is everything the probing tables need and nothing retrieval does.

The per-session tree carries the same samples spread across 34,970 files, and is the only source for depth, depth_raw, rs_rgb and clip_features. Use --layout files if you need those modalities:

python -m nys.data.download --root data/new-york-smells \
  --layout files --modalities depth,clip_features

data.source picks which layout the loader reads: auto (default) prefers parquet and falls back to the tree, while parquet and files pin one. Both yield identical samples -- tests/test_data.py asserts it tensor-for-tensor.

No training required — --pretrained fetches the weights from the Hub (~96 MB, cached) and evaluates them:

python -m nys.evaluation.retrieval --pretrained transformer \
  --set data.root=data/new-york-smells

python -m nys.evaluation.probe --pretrained transformer --task scene \
  --set data.root=data/new-york-smells --set data.balance_categories=true

Swap transformer for cnn, mlp, or smellprint. From Python it is from_pretrained("transformer"), as in the Quickstart above; both go through nys/models/pretrained.py.

python -m nys.train --config configs/coip_transformer.yaml \
  --set data.root=data/new-york-smells --set output_dir=runs/transformer

python -m nys.evaluation.retrieval --config configs/coip_transformer.yaml \
  --set data.root=data/new-york-smells --checkpoint runs/transformer/best.pth

python -m nys.evaluation.report runs --out runs/RESULTS.md

Or the whole paper, training all four encoders from scratch:

scripts/reproduce_table1.sh data/new-york-smells runs   # retrieval
scripts/reproduce_table2.sh data/new-york-smells runs   # recognition

Every field in nys/config.py is settable from the command line with --set section.field=value, and any run is fully described by its config:

python -m nys.train --config configs/coip_cnn.yaml \
  --set train.batch_size=256 --set train.loss=infonce --set data.split=session

Two towers into a shared 128-d L2-normalised space:

Image — torchvision ResNet-50, ImageNet-initialised and finetuned, with the classifier replaced by Linear(2048, 128).

Smell — one of four encoders, chosen with model.smell_encoder:

The raw input is concat(baseline_raw[:14], sample_raw[:14]) → (28, 32), z-scored per sensor using training-split statistics only. raw_length=14 is the global minimum timestep count across sessions; shorter recordings are edge-padded rather than dropped.

Loss (train.loss):

• infonce (default) — symmetric in-batch InfoNCE, τ = 0.07. Negatives are the other pairs in the batch.
• memory_bank_cmc — CMC-style memory-bank NCE with nce_k=1024. Negatives come from an EMA bank holding an embedding for every sample, so their count is independent of batch size. Not equivalent to the above; both are supported end to end.

Both split at a level above the individual recording, so an object's two recordings never straddle the boundary.

The id order in splits.json is load-bearing. The N = 933 retrieval gallery is a positional random_split(seed=42) over the test list, so re-sorting the ids selects a different gallery — worth about 5 mean rank. tests/test_data.py guards this.

nys/
  config.py             dataclass config; YAML + --set overrides
  train.py              contrastive pretraining
  losses.py             infonce | memory_bank_cmc
  data/
    download.py         Hub download, filtered by modality
    dataset.py          SmellDataset — the one dataset class
    splits.py           splits.json -> id lists
    labels.py           scene (8) / object (49) / material (51)
    stats.py            per-sensor normalisation, cached to disk
  models/
    image.py            ResNet-50 + projection
    smell.py            transformer | cnn | mlp | smellprint
    coip.py             joint model, checkpoint I/O
    pretrained.py       from_pretrained()
    probe.py            linear head
  evaluation/
    retrieval.py        cross-modal retrieval
    probe.py            linear probing
    published.py        chance rates and table labels
    report.py           results tables
configs/                one YAML per smell encoder
scripts/                reproduce_table{1,2}.sh, export_checkpoints.py, sweep_probe.py
tests/                  pytest tests -q

Tests needing the dataset skip cleanly without it; point NYS_DATA_ROOT at a copy to run them.

@article{ozguroglu2025smell,
  title={New York Smells: A Large Multimodal Dataset for Olfaction},
  author={Ozguroglu, Ege and Liang, Junbang and Liu, Ruoshi and Chiquier, Mia and DeTienne, Michael and Qian, Wesley Wei and Horowitz, Alexandra and Owens, Andrew and Vondrick, Carl},
  journal={arXiv preprint arXiv:2511.20544},
  year={2025}
}

Code MIT, dataset and checkpoints CC BY 4.0.