TIGeR: Unified Text-to-Image Generation and Retrieval

Leigang Qu, Haochuan Li, Tan Wang^*, Wenjie Wang^*, Yongqi Li, Liqiang Nie, and Tat-Seng Chua

^* Corresponding authors

TIGeR is a training-free framework that lets a multimodal large language model choose between two actions for a text prompt:

1. retrieve the best matching image from a gallery; or
2. generate a new image when the gallery is not suitable.

This repository provides organized SEED-LLaMA and LaVIT implementations for TIGeR-Bench.

• What TIGeR does
• Installation
• Model checkpoints
• Quick start
• Run individual stages
• Outputs
• Configuration
• Troubleshooting
• Citation

TIGeR represents gallery images as discrete visual tokens and uses the same MLLM throughout the pipeline:

1. Prepare the gallery: tokenize TIGeR-Bench images and compute unconditional gallery priors.
2. Retrieve: use constrained forward beam search to rank gallery images for each prompt.
3. Generate: produce a candidate image for each prompt.
4. Rerank: score retrieved candidates in the reverse image-to-text direction.
5. Route: compare retrieval and generation likelihoods and select the better action.
6. Evaluate: report retrieval, generation, and unified-task metrics.

TIGeR-Bench is downloaded directly from Hugging Face. No user-specific dataset paths are required.

The released environment targets Linux, Python 3.8, NVIDIA GPUs, and CUDA-compatible PyTorch.

Choose a backend and use its own environment: SEED and LaVIT require different dependency versions. The instructions below cover SEED; see the LaVIT guide for LaVIT setup and usage.

git clone https://github.com/LgQu/TIGeR.git
cd TIGeR/SEED

conda create -n tiger python=3.8 -y
conda activate tiger
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

The dependency versions are pinned for compatibility with the released SEED-LLaMA implementation, including transformers==4.29.2.

Download the official SEED-LLaMA tokenizer, language model, and diffusion decoder from the SEED repository. The expected layout is:

checkpoints/
└── seed-llama-8b-sft/
    └── ... model files ...

seed_llama_tokenizer_hf/
└── ... tokenizer files ...

diffusion_model/
└── ... diffusion model files ...

Point TIGeR to these directories:

export CKPT_ROOT=/path/to/checkpoints
export TOKENIZER_ROOT=/path/to/seed_llama_tokenizer_hf
export DIFFUSION_ROOT=/path/to/diffusion_model
export CUDA_VISIBLE_DEVICES=0

CKPT_ROOT is the parent directory of seed-llama-8b-sft, while TOKENIZER_ROOT and DIFFUSION_ROOT point directly to their respective model directories.

The benchmark is public, so authentication is normally unnecessary. If your environment requires it, set HF_TOKEN. You may also set HF_HOME or HF_DATASETS_CACHE to choose a cache location.

All commands below are run from the SEED directory.

First, inspect the complete workflow without loading a model or creating outputs:

DRY_RUN=1 bash prepare_benchmark.sh all
DRY_RUN=1 bash run_pipeline.sh all

Then prepare TIGeR-Bench. This tokenizes the images and computes the unconditional gallery priors used by retrieval:

bash prepare_benchmark.sh all

Finally, run retrieval, generation, reverse reranking, routing, and evaluation:

bash run_pipeline.sh all

These are large GPU jobs. For long-running experiments, running one stage at a time is easier to monitor and resume.

# Encode benchmark images and prompts.
bash prepare_benchmark.sh tokenize

# Compute unconditional gallery scores and logits.
bash prepare_benchmark.sh priors

The priors stage requires the files produced by tokenize.

bash run_pipeline.sh retrieve
bash run_pipeline.sh generate
bash run_pipeline.sh rerank
bash run_pipeline.sh route
bash run_pipeline.sh evaluate

Run these stages in the listed order. Their dependencies are:

Generated artifacts are kept inside SEED and ignored by Git:

SEED/
├── intermediate_results/tiger_bench/
│   ├── img_ids_*.pt
│   ├── t2i_mat_*.pt
│   ├── sumlogprobs_*.pt
│   └── uncond_logits_*.pt
├── generated/tiger_bench/
│   ├── ids_*.pt
│   └── images/
└── results/tiger_bench/
    ├── retrieval.pt
    ├── rerank_retrieval.pt
    ├── decision.pt
    └── ... evaluation reports ...

The most important files are:

• retrieval.pt: rankings produced by forward beam search.
• rerank_retrieval.pt: rankings after reverse reranking.
• decision.pt: likelihoods used to choose retrieval or generation.

The scripts expose commonly adjusted settings as environment variables:

For example:

NUM_BEAMS=200 RANK_BATCH_SIZE=32 bash run_pipeline.sh retrieve

To use the 14B checkpoint:

export MODEL_VERSION=14b
export CKPT_ROOT=/path/to/checkpoints
bash prepare_benchmark.sh priors
bash run_pipeline.sh all

In this case, CKPT_ROOT must contain seed-llama-14b-sft.

Make sure all three model locations are set:

echo "$CKPT_ROOT"
echo "$TOKENIZER_ROOT"
echo "$DIFFUSION_ROOT"

Reduce the batch size for the failing stage. For retrieval, also reduce NUM_BEAMS:

NUM_BEAMS=200 RANK_BATCH_SIZE=16 bash run_pipeline.sh retrieve

Reducing NUM_BEAMS changes the retrieval configuration and may affect the reported results.

Choose a writable cache directory and, if necessary, provide a token:

export HF_HOME=/path/to/huggingface_cache
export HF_TOKEN=your_huggingface_token

Never commit access tokens to the repository.

The custom SEED implementation requires the pinned version:

python -m pip install "transformers==4.29.2"

TIGeR/
├── assets/                  # paper figures
├── SEED/
│   ├── configs/             # model and tokenizer configuration
│   ├── models/              # SEED-LLaMA and image tokenizer
│   ├── transformers_patch/  # generation behavior required by SEED
│   ├── prepare_benchmark.sh # benchmark preprocessing entry point
│   ├── run_pipeline.sh      # TIGeR inference/evaluation entry point
│   └── README.md            # backend-specific reference
├── LaVIT/
│   ├── models/              # LaVIT model and visual tokenizer
│   ├── prepare_benchmark.sh # benchmark preprocessing entry point
│   ├── run_pipeline.sh      # TIGeR inference/evaluation entry point
│   └── README.md            # LaVIT setup and usage
├── LICENSE
└── README.md

This work builds on SEED-LLaMA and LaVIT. See the notices for SEED and LaVIT for attribution and license details.

If you find TIGeR useful, please cite:

@article{qu2024unified,
  title   = {Unified Text-to-Image Generation and Retrieval},
  author  = {Qu, Leigang and Li, Haochuan and Wang, Tan and Wang, Wenjie and Li, Yongqi and Nie, Liqiang and Chua, Tat-Seng},
  journal = {arXiv preprint arXiv:2406.05814},
  year    = {2024}
}

Original TIGeR code is released under the Apache License 2.0. The LaVIT-derived code under LaVIT/ is governed by the LaVIT Community License Agreement and use policy. Model checkpoints and other third-party components retain their upstream terms.