PEFT-Factory is a fork of LLaMa-Factory ❤️, upgraded with easy to use PEFT interface, support for HuggingFace PEFT methods and datasets for benchmarking PEFT.
| PEFT method name | Support |
|---|---|
| LoRA (including variants) | ✅ 🦙 |
| OFT | ✅ 🦙 |
| Prefix Tuning | ✅ 🤗 |
| Prompt Tuning | ✅ 🤗 |
| P-Tuning | ✅ 🤗 |
| P-Tuning v2 | ✅ 🤗 |
| MPT | ✅ 🤗 |
| IA3 | ✅ 🤗 |
| LNTuning | ✅ 🤗 |
| Bottleneck Adapter | ✅ 🤖 |
| Parallal Adapter | ✅ 🤖 |
| SeqBottleneck Adapter | ✅ 🤖 |
| SVFT | ✅ ⚙️ |
| BitFit | ✅ ⚙️ |
This section provides instructions on how to install PEFT-Factory, download necessary data and methods, and run training using both command line and web UI.
For video example please visit the PEFT-Factory Demonstration Video.
# install package
pip install peftfactory
# dowload repo that contains data, PEFT methods and examples
git clone https://github.com/kinit-sk/PEFT-Factory.git && cd PEFT-Factory
# start web UI
pf webuiAlternatively, you can run training from command line:
# install package
pip install peftfactory
# dowload repo that contains data, PEFT methods and examples
git clone https://github.com/kinit-sk/PEFT-Factory.git && cd PEFT-Factory# run training with config file
TIMESTAMP=`date +%s`
OUTPUT_DIR="saves/bitfit/llama-3.2-1b-instruct/train_wsc_${TIMESTAMP}"
DATASET="wsc"
SEED=123
WANDB_PROJECT="peft-factory-train-bitfit"
WANDB_NAME="bitfit_llama-3.2-1b-instruct_train_wsc"
mkdir -p "${OUTPUT_DIR}"
export OUTPUT_DIR DATASET SEED WANDB_PROJECT WANDB_NAMEUtility envsubst replaces the occurances of env variables with their values (see the template).
envsubst < examples/peft/bitfit/llama-3.2-1b-instruct/train.yaml > ${OUTPUT_DIR}/train.yamlpeftfactory-cli train ${OUTPUT_DIR}/train.yamlThere are multiple ways to install PEFT-Factory. You can install develelopment version from source or install the latest release from PyPI.
pip install peftfactorygit clone git@github.com:kinit-sk/PEFT-Factory.gitmake buildpip install dist/[name of the built package].whlTo download data, methods and examples for training please download the repository from GitHub.
git clone https://github.com/kinit-sk/PEFT-Factory.git && cd PEFT-FactoryYou can run training from command line or using web UI.
To run training from command line use the following command:
pf train [path to config file].yamlTo run the web UI use the following command:
pf webui