Official PyTorch implementation of "ECC: Encoder-Centric Corruption for Fine-Grained Vision in VLMs" (ECCV 2026).

[Paper] · [Project Page] · [arXiv]

TL;DR. Modern VLMs suffer from a granularity mismatch: global contrastive alignment biases them toward low-frequency semantics and away from fine detail. ECC is a corruption-to-reconstruction (C2R) pretraining framework that unifies masking and feature-level noising inside the encoder, and explicitly disentangles de-masking from de-noising via a disruption loss. The result is a vision backbone that captures a broader frequency spectrum, improving both fine-grained recognition and the visual foundation of VLMs.

• Encoder-Centricity. Corruption and restoration occur within the encoder—the component actually transferred for downstream tasks—so noising directly shapes transferable representations.
• Feature-Level Noising. Noise injected at intermediate encoder layers (block 2) captures high-frequency detail that pixel-space perturbation fails to model.
• Task Disentanglement. A disruption loss suppresses cross-corruption (masked↔noised) attention, preventing interference between the de-masking and de-noising objectives.
• Strong & scalable. Surpasses prior MIM and noise-based C2R baselines (up to +8.1% over SimMIM, +4.4% over MAE), and consistently improves CLIP, AIM-v2, and LLaVA when used as a vision-only pretraining strategy.

ECC builds on the C2R paradigm but departs from prior Decoder-style designs (e.g., DiffMAE, MaskDiT) that prevent noise-induced signals from being integrated into the encoder. ECC applies Conjunctive Corruption—masked tokens are retained while a separate subset of visible tokens is noised—and processes all tokens within a unified encoder. The total objective combines mask reconstruction, denoising, and the disruption loss:

L_total = L_MIM + L_Denoise + λ · L_disruption     (λ = 0.1)

The disruption loss raises the entropy of masked-token attention rows in the affinity matrix, recalibrating the weight distribution so that masked tokens minimally interfere with noisy visible tokens.

git clone https://github.com/<user>/ECC.git
cd ECC

conda create -n ecc python=3.10 -y
conda activate ecc

# Install PyTorch matching your CUDA version (see https://pytorch.org)
pip install torch torchvision

Environment used in the paper: ViT-B backbone, 4 × A100 GPUs.

Pretraining uses ImageNet-1K. Downstream evaluation covers fine-grained recognition, classification, segmentation, and detection:

Expected layout:

data/
├── imagenet/
│   ├── train/
│   └── val/
├── cub/
├── nabirds/
├── inat2017/
├── inat2018/
├── stanford_cars/
└── aircraft/

Pretrain a ViT-B encoder on ImageNet-1K. By default we use a 400-epoch schedule, a global batch size of 4096, feature-level noise injected at encoder block 2, and a disruption-loss weight of λ = 0.1. The framework also supports longer schedules (800 / 1600 epochs) and larger backbones (ViT-L).

Key configuration:

After pretraining, the encoder is transferred and fine-tuned on each downstream dataset. We use AdamW with a cosine-decay schedule, layer decay 0.65, drop path 0.1, 20 warmup epochs, and 100 training epochs, together with standard augmentations (RandAug, Mixup, CutMix, label smoothing, random erasing).

ECC is used as a vision-only pretraining strategy that strengthens the visual foundation of VLMs. To plug an ECC-pretrained encoder into a VLM framework (CLIP / AIM-v2 / LLaVA-1.5), replace the vision encoder with the ECC checkpoint and follow each framework's official training recipe.

This codebase builds upon MAE, SimMIM, and related masked image modeling works. We thank the authors for releasing their code.

For questions, please open an issue or contact <hyesong@ssu.ac.kr>.