ArtUnmasked

Artist Attribution for Artwork via Compositional Tag Signatures and Deep Embedding Matching

This repository accompanies a research paper currently under peer review. Code and data are provided for reproducibility purposes under the Apache License 2.0.

ArtUnmasked is a multi-stage artist attribution pipeline for fine-art images. Given an unknown artwork, the system identifies the most likely artist by combining:

1. AI/Real Detection (SPAI) — Screens out AI-generated images using spectral learning on a ViT-Tiny backbone before attribution begins.
2. TagMatch — Encodes each artist's style as a sparse composition signature over aspect–concept tag combinations. A rarity-weighted cosine score ranks candidate artists at scale.
3. DINO + CLIP Nearest-Neighbour Matching — Verifies the top candidates with high-dimensional visual embeddings (DINOv3 + CLIP), using patch-level comparison for fine-grained attribution.
4. Heatmap Visualisation — Produces interpretable patch-similarity heatmaps to show which regions of the query image most resemble the attributed artist's body of work.

The system is evaluated on a WikiArt-derived dataset spanning thousands of artists across imitation, real-overlap, and diffusion-generated image categories.

Query Image
     │
     ▼
┌─────────────────────────────────────────┐
│  Stage 1: AI / Real Screening (SPAI)    │
│  ViT-Tiny · Spectral Learning           │
│  → discard AI-generated images          │
└────────────────────┬────────────────────┘
                     │ real artwork only
                     ▼
┌─────────────────────────────────────────┐
│  Stage 2: TagMatch Attribution          │
│  CLIP text embeddings · concept vocab   │
│  Aspect–concept cosine similarity       │
│  Rarity-weighted composition signatures │
│  → Top-K artist candidates              │
└────────────────────┬────────────────────┘
                     │ candidate list
                     ▼
┌─────────────────────────────────────────┐
│  Stage 3: DINO + CLIP Verification      │
│  DINOv3 (vit_small_plus_patch16)        │
│  CLIP (ViT-H-14 / ViT-B-16 / ViT-L-14) │
│  Global CLS + patch-level cosine sim    │
│  → Ranked attribution result            │
└────────────────────┬────────────────────┘
                     │
                     ▼
┌─────────────────────────────────────────┐
│  Stage 4: Heatmap Visualisation         │
│  7×7 pooled patch embeddings            │
│  → Interpretable similarity overlay     │
└─────────────────────────────────────────┘

Full per-class results are reported in the paper. Ablation results across CLIP backbones are in ablation_outputs/.

Python: 3.10+

Core dependencies:

Install all dependencies into a virtual environment:

python -m venv .venv
source .venv/bin/activate
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118
pip install timm open_clip_torch safetensors pandas numpy scikit-learn \
            Pillow opencv-python albumentations yacs tqdm matplotlib

A GPU is strongly recommended. CPU fallback is supported but timings will be significantly higher.

ArtUnmasked/
├── ArtUnmasked/                          # Core Python package
│   ├── CONFIG.py                         # Centralised YACS config (all file paths & hyperparameters)
│   ├── main.py                           # End-to-end inference entry point
│   ├── utils.py                          # Image validation, repair utilities, CustomTimer
│   │
│   ├── ArtWork_Tagging/                  # TagMatch module
│   │   ├── tag_match.py                  # TagMatch inference engine → (df, top10_acc, top5_acc)
│   │   ├── calculate_artist_signatures.py # Build tensorized artist composition signatures
│   │   ├── compute_cosine_similarity.py  # Tag–image cosine scores (mode="train"/"test")
│   │   ├── compute_clip_text_embeddings.py # Embed concept vocabulary with CLIP
│   │   ├── compute_clip_image_embeddings.py # Embed images with CLIP
│   │   └── Artwork_Vocabulary - Sheet2.csv # Aspect–concept vocabulary (~100 pairs)
│   │
│   ├── dinov3_clip_inference/            # DINO + CLIP matching module
│   │   ├── embedders.py                  # DINOv3Embedder & CLIPEmbedder wrappers
│   │   ├── save_dinov3_clip_embeddings.py # Bulk embedding generation (safetensors)
│   │   ├── dino_v3_clip_inference.py     # Nearest-neighbour attribution
│   │   └── compute_heatmaps.py           # Patch-level similarity heatmaps
│   │
│   ├── ai_vs_real/                       # SPAI integration (AI-generated image detection)
│   │   └── spai/                         # Modified SPAI codebase (see Third-Party section)
│   │
│   ├── analysis/                         # Visualisation and distribution plots
│   └── data/                             # Embeddings, CSVs, and dataset splits
│
├── ablation_runner.py                    # CLIP backbone ablation study (3 variants)
├── run_eval.py                           # Full pipeline evaluation
├── create_eval_dataset.py                # Evaluation dataset construction
├── wikiart/                              # WikiArt data fetching utilities
│
├── ablation_outputs/                     # Ablation results (CSV + LaTeX tables)
├── eval_outputs/                         # Evaluation results
├── Journal_Submission___ArtUnmasked/     # Paper draft and supplementary materials
│
├── NOTICE                                # Third-party attribution (SPAI)
├── LICENSE                               # Apache License 2.0
└── README.md                             # This file

All file paths and hyperparameters are centralised in ArtUnmasked/CONFIG.py using YACS. Key config groups:

To override a setting without editing the file, monkey-patch the CfgNode before calling any module function (as ablation_runner.py does via patch_configs()).

Large binary files (safetensors embeddings, model checkpoints) are excluded from the repository. See the paper supplementary for download instructions.

cd /home/thesalad/PycharmProjects/ArtUnmasked
python -m ArtUnmasked.main

The pipeline stages are individually toggled inside main.py. By default the full flow runs:

1. Save test DINO+CLIP embeddings
2. Run DINO+CLIP nearest-neighbour matching
3. Compute heatmaps

SPAI screening and TagMatch are available as commented-out blocks and can be enabled as needed.

python ablation_runner.py

This sweeps three CLIP backbones (ViT-H-14, ViT-B-16, ViT-L-14) with fixed TagMatch hyperparameters:

• Train: top_k_tags=25, max_comb_size=3
• Test: max_tags_per_image=15, max_comp_size=3

Outputs (written to ablation_outputs/):

The LaTeX table is included in the paper template via:

\input{../ablation_outputs/ablation_tables}

Steps that hit the cache report 0.0 s in the timing columns. The LaTeX caption notes this explicitly.

python run_eval.py

Evaluates the full pipeline on the eval dataset and reports per-class accuracy metrics, printed to stdout and saved to eval_outputs/.

If WikiArt embeddings or composition signatures are not yet generated, run the relevant steps from ArtUnmasked/main.py or call directly:

from ArtUnmasked.ArtWork_Tagging.compute_clip_text_embeddings import embed_concepts
from ArtUnmasked.ArtWork_Tagging.compute_cosine_similarity import compute_cosine_similarity
from ArtUnmasked.ArtWork_Tagging.calculate_artist_signatures import build_artist_signatures

embed_concepts()                          # → concept_embeddings.npz
compute_cosine_similarity(mode="train")   # → artist_tags_cosine_similarities.csv
build_artist_signatures()                 # → global_compositions.safetensors + .json

TagMatch attributes artwork via composition signatures: compact, sparse representations of each artist's recurring stylistic combinations.

Training phase:

1. Embed the concept vocabulary (e.g., landscape__mountains) using the CLIP text encoder.
2. For each training image, compute cosine similarities between its CLIP embedding and all concept embeddings; retain concepts with z-score ≥ 1.75.
3. Form tag combinations of size 2–3 (max_comb_size=3); keep the top-k most distinctive per image.
4. Aggregate across an artist's training images into a sparse COO tensor (artists × compositions) weighted by per-composition rarity.

Inference phase:

1. Repeat steps 1–3 on the test image (max_tags_per_image=15).
2. Score each artist by rarity-weighted cosine similarity between the test composition vector and the artist's signature.
3. Return a ranked candidate list; report Top-5 and Top-10 accuracy.

The TagMatch function signature is:

df, top10_acc, top5_acc = run_tagmatch_inference(cfg, ...)

The ai_vs_real/ directory contains a modified version of SPAI (Any-Resolution AI-Generated Image Detection by Spectral Learning), originally developed by the Centre for Research and Technology Hellas (CERTH) and the University of Amsterdam.

Original repository: https://github.com/mever-team/spai Original licence: Apache License 2.0 Original copyright: © 2025 CERTH and University of Amsterdam

Modifications made for ArtUnmasked (in compliance with the Apache 2.0 licence):

The modified SPAI code is redistributed under the Apache License 2.0. All modifications are documented in:

• ArtUnmasked/ai_vs_real/spai/NOTICE — detailed change log attributing modifications to Akshad Vijay Chidrawar (2025)
• NOTICE (project root) — top-level attribution notice

The full Apache 2.0 licence text is available at ArtUnmasked/ai_vs_real/spai/LICENSE.

This project is licensed under the Apache License, Version 2.0.

Copyright 2025 Akshad Vijay Chidrawar

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

See the LICENSE file at the project root for the full licence text.

• GPU: A CUDA-capable GPU is required for meaningful wall-clock timings. CPU fallback is available but substantially slower.
• Memory: The WikiArt patch embeddings file (WIKIART_PATCH_EMBEDDINGS.safetensors) is ~871 MB. DINO+CLIP global embeddings are ~44 MB. Ensure sufficient disk and VRAM (≥ 8 GB recommended).
• Caching: Expensive steps (concept embedding, cosine similarity, artist signatures) are automatically cached. Re-running will skip cached steps and report 0.0 s in timing outputs.
• OOM prevention: The ablation runner calls torch.cuda.empty_cache() and deletes the CLIP model between variants.
• Torch compile: torch.compile() is applied to both DINOv3 and CLIP models; the first forward pass incurs a one-time compilation cost.

If you use ArtUnmasked or the TagMatch algorithm in your research, please cite our paper:

@article{artunmasked2025,
  title     = {ArtUnmasked: A Multimodal Classifier for Real, AI, and Imitated Artworks},
  author    = {Chidrawar, Akshad Vijay and others},
  journal   = {Under Review},
  year      = {2025}
}

Citation will be updated upon publication.

• SPAI — CERTH and University of Amsterdam for the original spectral AI-detection framework (mever-team/spai)
• WikiArt — Publicly available artwork metadata used for training and evaluation
• timm — Ross Wightman and the timm community for the DINOv3 model hub
• OpenCLIP — LAION for the open-source CLIP training and weights
• DiffusionDB — Wang et al. for the diffusion-generated image dataset used in SPAI evaluation