The official implementation of DoRA can be found here and is now fully supported by HuggingFace PEFT.

The code and checkpoints for reproducing the results of the four experiments reported in the paper will be released soon. Please be aware that releasing the code and checkpoints to the public necessitates an internal review process within the company, which can be time-consuming. We appreciate your understanding in this matter.

Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Min-Hung Chen

Paper: https://arxiv.org/abs/2402.09353

DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning, specifically employing LoRA for directional updates to efficiently minimize the number of trainable parameters. By employing DoRA, we enhance both the learning capacity and training stability of LoRA while avoiding any additional inference overhead. DoRA consistently outperforms LoRA on fine-tuning LLaMA, LLaVA, and VL-BART on various downstream tasks, such as commonsense reasoning, visual instruction tuning, and image/video-text understanding

DoRA is now supported by the Huggingface PEFT package. You can install the PEFT package using

pip install git+https://github.com/huggingface/peft.git -q

After PEFT is installed, you can simply set the use_dora argument of LoRAConfig to True for applying DoRA. An example could be as follows:

from peft import LoraConfig, get_peft_model

# Initialize DoRA configuration
config = LoraConfig(
    r=8, 
    lora_alpha=16, 
    target_modules=[
      "q_proj",
      "k_proj",
      "v_proj",
      "o_proj",
      "gate_proj",
      "up_proj",
      "down_proj",
    ],
    lora_dropout=0.1,
    bias="none",
    task_type="CAUSAL_LM",
    use_dora=True
)

Please refer to the official documentation for more details.

You can also toy with DoRA on finetuning Diffusion Model. See (huggingface/diffusers)[https://github.com/huggingface/diffusers/tree/main/examples/advanced_diffusion_training#dora-training]. Another good tutorial would be this collab notebook from Linoy Tsaban.

In general DoRA finetuning on diffusion model is still experimental and is likely to require different hyperparameter values to perform best compared to LoRA.

Specifically, people have noticed 2 differences to take into account your training:

1. LoRA seem to converge faster than DoRA (so a set of parameters that may lead to overfitting when training a LoRA may be working well for a DoRA)
2. DoRA quality superior to LoRA especially in lower ranks the difference in quality of DoRA of rank 8 and LoRA of rank 8 appears to be more significant than when training ranks of 32 or 64 for example.

• Example From Linoy Tsaban(Images generated by DoRA are on the left and LoRA on the right):

• Example From merve:

[NOTE that] 💡 While fine-tuning with DoRA by utilizing the configuration of LoRA can already achieve better results most of the time, achieving optimal performance compared to LoRA still requires adjustments to the hyperparameters.

We suggest starting with a slightly lower learning rate than that of LoRA, and users may also experiment with varying lora dropout ratios.

User may also start with half of the rank of the LoRA configuration which oftentime can already results in comparable or even superior accuracy compared to that of LoRA.

The code will be released under https://github.com/NVlabs soon and will contain four directories:

./commonsense_reasoning contains the code to finetune LLaMA-7B/13B using DoRA on the commonsense reasoning tasks. This directory is modified based on LLM-Adapter.

./instruction_tuning contains the code to finetune LLaMA-7B and LLaMA2-7B using DoRA and DVoRA (DoRA+VeRA) with the cleaned Alpaca instruction tuning dataset. This directory is modified based on VeRA.

./image_video_text_understanding contains the code to finetune VL-BART using DoRA for the image/video-text understanding tasks. This directory is modified based on VL-Adapter.

./visual_instruction_tuning contains the code to finetune LLaVA-1.5-7B on the visual instruction tuning tasks with DoRA. This directory is modified based on LLaVA.

1. You can start with reproducing the results in table 1, try replacing the PEFT package in the LLM-Adapter code base with the latest version of PEFT which supports DoRA finetuning.
2. Please install the python packages with the provided requirements.txt, since there are some issue with the latest version of accelerate and transformer package.
3. Use the hyperparamter specified in the paper for finetuning LLaMA-7B/13B.

Shih-Yang Liu: shihyangl@nvidia.com or sliuau@connect.ust.hk

Copyright © 2024, NVIDIA Corporation. All rights reserved.

This work is made available under the NVIDIA Source Code License-NC. Click here to view a copy of this license.

If you find DoRA useful, please cite it by using the following BibTeX entry.

@article{liu2024dora,
  title={DoRA: Weight-Decomposed Low-Rank Adaptation},
  author={Liu, Shih-Yang and Wang, Chien-Yi and Yin, Hongxu and Molchanov, Pavlo and Wang, Yu-Chiang Frank and Cheng, Kwang-Ting and Chen, Min-Hung},
  journal={arXiv preprint arXiv:2402.09353},
  year={2024}
}