The computation framework of NoRM and proposed Sim-Search method.

🔥 [2025/01/23] Our paper is accepted by ICLR 2025 as a SpotLight paper!

• This is the repo for FINE-TUNING WITH RESERVED MAJORITY FOR NOISE REDUCTION

• FINE-TUNING WITH RESERVED MAJORITY FOR NOISE REDUCTION
  • News
  • Contents
  • Highlights
  • Setups
    • Install
    • Data Preparation
  • Finetuning
    • LoRA Finetuning
    • Prepare NoRM Parameters
    • NoRM Evaluation
• Citation

• Our proposed NoRM as an simple-yet-effective method which can be applied to reduce LoRA parameter redundancies which natually exist during fine-tuning. The most supportive components are kept based on the highest subspace similarity with base models. With only contributable parameters maintained,
• NoRM can leverage more tunable parameters which leads to higher performance improvements bounds.
• NoRM can use small amount of SFT data with little distribution shift problems. It also benefits from the data scaling law as vanilla fine-tuning.
• As NoRM keeps most advantageous components, it is intrinsically powerful than LoRA under continuous learning, where catastrophic forgetting problem accompany with distribution shift.

1. First clone this repository and nagivate to the TAIA_LLM repository:

git clone https://github.com/pixas/NoRM.git
cd NoRM

2. Install Package

conda create -n norm python=3.10
conda activate norm
conda install --yes --file requirements.txt
pip install flash-attn --no-build-isolation

We use the following data format:

{
  "conversations": [
    {"from": "human", "value": ""},
    {"from": "gpt", "value": ""}
  ],
  "answer": "" # for evaluation dataset
}

Prepare a data folder task_path, containing the training data .json files and a subfolder named norm_test, which containing the evaluation data with .json suffix.

We here use Llama3-8B model as the example backbone. You can change the backbone to any other chat-models and prepare corresponding chat templates.

bash scripts/train/bash/sft_lora_r64_llama38b_format.sh

Replace the backbone model path, output path, and data path in the script with the actual path in your machine. This script automatically evaluate vanilla LoRA's performance until the training completes by detecting whether the adapter_config.json exists in the output path.

sbatch scripts/autoselect.sh

If using local machines, run the following command under a CUDA environment:

python evaluation/auto_select.py \
  --model_base $MODEL_BASE \
  --model_path $MODEL_PATH \
  --save_name ${SAVE_NAME} \
  --step ${step} \
  --select_method ${select_method} \
  --range_start ${range_start}

where MODEL_BASE: the base model path MODEL_PATH: the original LoRA fine-tuning path SAVE_NAME: the generated NoRM parameter filename, which will be saved as ${MODEL_PATH}/${SAVE_NAME}.safetensors. step: default to 0.1, as the search step in Sim-Search select_method: default to lora range_start`: default to 1

Use the following command to conduct evaluation:

sbatch scripts/eval/slurm/eval_parallel_peft_batch_autoselect.sh $TASK_PATH $MODEL_BASE $MODEL_PATH ${CKPT}-automerge-${SAVE_NAME} ${LOGS_BASE_PATH} $domain $SAVE_NAME

where domain: the evaluation dataset's name in your local folder LOGS_BASE_PATH: the log file base path, default to ./logs/${training_data}/

Check the file ./logs/${training_data}/${CKPT}-automerge-${SAVE_NAME}/$domain/eval.log for evaluation results.

If you find NoRM useful for your research and applications, please cite using this BibTeX:

@inproceedings{jiang2025finetuning,
title={Fine-tuning with Reserved Majority for Noise Reduction},
author={Shuyang Jiang and Yusheng Liao and Yanfeng Wang and Ya Zhang and Yu Wang},
booktitle={The Thirteenth International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=ZV7CLf0RHK}
}