SV-DRR: High-Fidelity Novel View X-Ray Synthesis Using Diffusion Model

MICCAI 2025

Authors: Chun Xie, Yuichi Yoshii, Itaru Kitahara
University of Tsukuba | Tokyo Medical University Ibaraki Medical Center

• [2025-10-06] 🔥 Training instructions and code are now available!
• [2025-10-05] 🎉 Inference code and pretrained models are now available! You can now run SV-DRR on your own X-ray images.

We propose a novel view-conditioned diffusion model for synthesizing multi-view X-ray images up to 1024x1024 resolution from a single view.

The name SV-DRR, short for Single-View DRR, is inspired by Digitally Reconstructed Radiography (DRR).

Unlike DRR, which renders X-ray projections from a 3D CT volume, our method synthesizes novel views directly from a single 2D projection.

Download the preprocessed DRRs

We synthesized 3000 DRR images for each CT by uniformly sampling on a sphere, but only the first 1500 images (on the PA view side) were used for the experiment. The images are organized as follows:

image_complex_fb_256  
    |-LIDC-IDRI-0001  
        |-0000.png   <- 0000 is always the standard PA view  
        |-0001.png          
        |-...
        |-2999.png 
        |-camera.json
    |-LIDC-IDRI-0002
    |-...
    |-camera_views.json  <- Spherical coordinates of each view, in radians
    |-patients.json  

To ensure compatibility and reproducibility, follow these steps to set up the environment:

1. Clone the Repository:

   git clone https://github.com/xiechun-tsukuba/svdrr.git
   cd svdrr

2. Create a Python Virtual Environment:

   conda create -f environment.yaml

You can download the pretrained models by either:

Option 1: Automated Download (Recommended)

python scripts/download_models.py

This will download all models into the models/ directory. Shared components will be stored in the shared/ folder, and symbolic links will be created in each model folder accordingly.

Option 2: Manual Download from Hugging Face

• 256 resolution: https://huggingface.co/xiechun-tsukuba/svdrr-dit-fb-256
• 512 resolution: https://huggingface.co/xiechun-tsukuba/svdrr-dit-fb-512
• 1024 resolution: https://huggingface.co/xiechun-tsukuba/svdrr-dit-fb-1024

Important Note: The coordinate system of LIDC-IDRI-DRR is opposite to the intuitive one — the polar angle increases downward, and the azimuth angle increases when rotating to the left. To invert the pose coordinate system, use the --flip_pose option.

Default views (azimuth angles from -90° to 90° in 5° increments):

python test_svdrr_DiT.py --model_path models/DiT-fb-512 \
    --image_path demo/real_xray.jpg \
    --log_dir outputs/ \
    --image_size 512 \
    --simple_pose

User-specified views defined in camera_views.json:

python test_svdrr_DiT.py --model_path models/DiT-fb-512 \
    --image_path demo/real_xray.jpg \
    --log_dir outputs/ \
    --image_size 512 \
    --poses demo/camera_views.json

Perform inference on the LIDC-IDRI-DRR dataset:

python test_svdrr_DiT.py --model_path models/svdrr-DiT-fb-256 \
    --dataset {path/to/dataset/} \
    --log_dir outputs/ \
    --image_size 256 

First, download the base model pretrained by PixArt-Σ:

# For 256x256 resolution (default)
python scripts/download_base_model.py

# For 512x512 resolution
python scripts/download_base_model.py --size 512

# For 1024x1024 resolution
python scripts/download_base_model.py --size 1024

This will download the appropriate PixArt-Σ pretrained weights to models/base-model/ and prepare them for SV-DRR training.

To train from the base model at 256×256 resolution:

accelerate launch train_svdrr_DiT.py \
    --pretrained_model_name_or_path "models/base_model/256" \
    --resolution 256 \
    --train_batch_size 64 \
    --lr_warmup_steps 1000 \
    --learning_rate 5e-6 \
    --train_data_dir {path/to/256/dataset/} \
    --output_dir "checkpoints/svdrr-DiT-fb-256" \
    --tracker_project_name "svdrr-DiT-fb-256" \
    --dataloader_num_workers 16 \
    --checkpointing_steps 1000 \
    --validation_steps 1000 \
    --num_validation_batches 8 \
    --checkpoints_total_limit 20 \
    --max_train_steps 200000 \
    --ct_thickness "thin" \
    --xray_orientation "PA" \
    --device_specific_seed \
    --use_seedable_sampler 

After reaching the maximum training steps, the ready-to-use inference pipeline will be automatically saved to the final-pipeline/ folder under the specified output_dir.

Resume Training: To resume training from a checkpoint, use: --resume_from_checkpoint {checkpoint} Here, {checkpoint} can be the path to a specific checkpoint file or simply "latest".

Convert checnkpoint to pipeline: To convert an intermediate checkpoint into a ready-to-use pipeline, run:

python script/ckpt2model_DiT.py \
    --base_model {path/to/base_model} \
    --ckpt {path/to/checkpoint} \
    --size {256|512|1024} \
    --output {path/to/output}

Here, --base_model can be any base model downloaded in Step 1, or a previously trained SV-DRR pipeline.

To continue training at 512×512 resolution:

accelerate launch train_zero1to3_DiT.py \
    --pretrained_model_name_or_path "{path/to/svdrr_model/256}" \
    --resolution 512 \
    --train_batch_size 32 \
    --lr_warmup_steps 1000 \
    --learning_rate 3e-6 \
    --train_data_dir "{path/to/512/dataset/}" \
    --output_dir "checkpoints/svdrr-DiT-fb-256-512" \
    --tracker_project_name "svdrr-DiT-fb-256-512" \
    --dataloader_num_workers 16 \
    --checkpointing_steps 1000 \
    --validation_steps 1000 \
    --num_validation_batches 8 \
    --checkpoints_total_limit 20 \
    --max_train_steps 100000 \
    --ct_thickness "thin" \
    --xray_orientation "PA" \
    --device_specific_seed \
    --use_seedable_sampler \

accelerate launch train_svdrr_DiT.py \
    --pretrained_model_name_or_path "{path/to/svdrr_model/512}" \
    --resolution 1024 \
    --train_batch_size 8 \
    --lr_warmup_steps 1000 \
    --learning_rate 1e-6 \
    --train_data_dir "{path/to/1024/dataset/}" \
    --output_dir "checkpoints/svdrr-DiT-fb-256-512-1024" \
    --tracker_project_name "svdrr-DiT-fb-256-512-1024" \
    --dataloader_num_workers 16 \
    --checkpointing_steps 1000 \
    --validation_steps 1000 \
    --num_validation_batches 8 \
    --checkpoints_total_limit 20 \
    --max_train_steps 100000 \
    --ct_thickness "thin" \
    --xray_orientation "PA" \
    --device_specific_seed \
    --use_seedable_sampler \

This repository is based on the codebases below:

• Zero1to3-hf (HuggingFace Diffusers implementaiton of Zero1to3)
• PixArt-Σ
• HF Diffusers

If you find this work useful, a citation will be appreciated via:

@InProceedings{XieChu_SVDRR_MICCAI2025,
        author = { Xie, Chun AND Yoshii, Yuichi AND Kitahara, Itaru},
        title = { { SV-DRR: High-Fidelity Novel View X-Ray Synthesis Using Diffusion Model } },
        booktitle = {proceedings of Medical Image Computing and Computer Assisted Intervention -- MICCAI 2025},
        year = {2025},
        publisher = {Springer Nature Switzerland},
        volume = {LNCS 15963},
        month = {September},
        page = {572 -- 582},
        doi = {https://doi.org/10.1007/978-3-032-04965-0_54}
}

@misc{xie2025svdrr,
        title = {SV-DRR: High-Fidelity Novel View X-Ray Synthesis Using Diffusion Model}, 
        author = {Chun Xie and Yuichi Yoshii and Itaru Kitahara},
        year = {2025},
        eprint = {2507.05148},
        archivePrefix = {arXiv},
        doi = {https://doi.org/10.48550/arXiv.2507.05148}, 
}