We propose Patcher, a method inspired by adversarial training and bi-level optimization, to combat full-parameter malicious finetuning attacks. Patcher strengthens the simulated attack by scaling up the optimization steps in the adversarial loop, thus forcing the defender to find model parameters that are insensitive to stronger attacks. Furthermore, we propose an efficient parallel algorithm to implement Patcher, decreasing the wall-clock time of training while preserving Patcher's performance. Extensive experiments show that Patcher substantially improves the model's robustness compared to vanilla SFT alignment, and transfers to diverse attack scenarios and model sizes.

Sequential implementation:

Parallel implementation:

git clone https://github.com/haomingwen/Patcher.git
cd Patcher
pip install -e .

For Patcher training, the alignment dataset is here, the attack dataset is here.

For finetuning attack simulation, we provide references to some of the attack datasets evaluated in this paper, Beavertails-unsafe, PKU-SafeRLHF-unsafe, and Toxic-DPO. You can mix them with benign datasets, such as gsm8k. Transform them to the following format:

{
    "prompt": "Tell me a joke.",
    "response": "Sure! Why don't scientists trust atoms? Because they make up everything!",
}

For evaluation, the prompts are from Advbench and HEx-PHI.

Please put all downloaded datasets in the datasets folder.

For non-parallel training, an example is shown below:

    CUDA_VISIBLE_DEVICES=0 \
    BASE_CKPT=path/to/your/base/model \
    MODEL_NAME=Qwen2.5-1.5B \
    bash scripts/run_patch_iter_final.sh

For parallel training, an example is shown below:

    # Run this in terminal 1
    CUDA_VISIBLE_DEVICES=0 \
    BASE_CKPT=path/to/your/base/model \
    MODEL_NAME=Qwen2.5-1.5B \
    bash scripts/run_patch_parallel.sh

    # Run this in terminal 2
    CUDA_VISIBLE_DEVICES=1 \
    MODEL_NAME=Qwen2.5-1.5B \
    bash scripts/run_sft_parallel.sh

Make sure that MODEL_NAME is aligned with the model family's name for correct dataset collation.

After having prepared your custom attack+benign dataset, you can run the following command to simulate malicious finetuning and evaluate ASR on the benchmarks:

    DASHSCOPE_API_KEY=your_dashscope_api_key \
    EVAL_MODEL_PATHS=path/to/your/aligned/models \
    EVAL_MODEL_NAMES="Qwen2.5-1.5B" \
    EVAL_DATASETS="advbench hexphi" \
    ATTACK_DATASET_PATHS=path/to/your/attack/dataset.json \
    ATTACK_DATASET_NAMES="YOUR_ATTACK_DATASET_NAME" \
    IS_ATTACK_DATASETS_MULTITURN="False" \
    NUM_SAMPLES="1000" \
    POISON_RATES="0.2" \
    ATTACK_STEPS="2000" \
    LRS="1e-5" \
    SEEDS="0" \
    bash scripts/run_custom_eval.sh

This repo currently supports calling dashscope API for evaluation. If you want to use your own LLM judge, you may need to modify the evaluation script at patcher/evaluate/gpt_evaluate.py.