English Version

New Features

1. Multimodal: Supported a large number of multimodal datasets and restructured the multimodal architecture. Some models now support grounding tasks.
2. Web-ui: Added support for RLHF, evaluation, and quantization.
3. Evaluation Functionality: Refactored the evaluation functionality, now using OpenCompass internally, supporting over 50 evaluation datasets.
4. Deployment Functionality: VLLM infer_backend now supports multimodal models.
5. Agent Training: Refactored the construction, training, and deployment of agent datasets, making the agent pipeline more complete.
6. Human Alignment: Added alignment algorithms such as KTO and CPO, and refactored the human alignment code.

New Models

1. openbuddy-llama3-70b
2. Deepseek-coder-v2
3. llava1.5, llava1.6, llava-next-video
4. gemma2
5. Florence
6. phi3-4k
7. internlm2.5, xcomposer2.5
8. internvl2
9. codegeex4
10. mistral-7b-instruct-v0.3

New Datasets

1. Over 30 foundational multimodal datasets, including GQA, RefCOCO, and Llava-pretrain.
2. Swift-mix general mixed dataset.
3. Video-chatgpt video dataset.

中文版本

新功能

1. 多模态：支持了非常多的多模态数据集，并重构了多模态架构，部分模型开始支持grounding任务
2. Web-ui：支持了RLHF、评测和量化
3. 评测功能：进行了重构，内部使用了OpenCompass，支持50+评测集
4. 部署功能：VLLM infer_backend支持多模态模型
5. Agent训练：重构了Agent数据集构造、训练、部署，Agent链路更加完整
6. 人类对齐：增加了KTO、CPO等对齐算法，并重构了人类对齐的代码

新模型

1. openbuddy-llama3-70b
2. Deepseek-coder-v2
3. llava1.5, llava1.6, llava-next-video
4. gemma2
5. Florence
6. phi3-4k
7. internlm2.5, xcomposer2.5
8. internvl2
9. codegeex4
10. mistral-7b-instruct-v0.3

新数据集

1. GQA、RefCOCO、Llava-pretrain等30+多模态基础数据集
2. swift-mix通用混合数据集
3. video-chatgpt视频数据集

What's Changed

• fix py38 by @Jintao-Huang in #1152
• refactor rlhf by @hjh0119 in #1090
• Fix/web UI 0617 by @tastelikefeet in #1158
• Add more datasets by @tastelikefeet in #1065
• support openbuddy-llama3-70b by @Jintao-Huang in #1162
• Add Deepseek coder v2 model by @tastelikefeet in #1164
• Update web-ui labels by @tastelikefeet in #1168
• fix hf download by @tastelikefeet in #1169
• fix kto bugs by @hjh0119 in #1173
• Fix preprocessor by @tastelikefeet in #1174
• refactor ui by @tastelikefeet in #1178
• fix DPO by @hjh0119 in #1181
• fix bugs by @Jintao-Huang in #1171
• 在argument.py添加hccl参数，用于NPU环境下训练使用 by @jiaozhentian in #1165
• Custom agent dataset training support by @hjh0119 in #1184
• RLHF UI by @tastelikefeet in #1182
• fix kto by @hjh0119 in #1187
• update agent doc by @hjh0119 in #1189
• Fix device map in limited mem by @tastelikefeet in #1190
• Fix sk will be uploaded by @tastelikefeet in #1192
• Fix dataset concatenation by @tastelikefeet in #1193
• Fix glm4v dataloader by @CjangCjengh in #1183
• fix glm4v images by @Jintao-Huang in #1194
• [TorchAcc] Cache the compiled results and remove some xla flags by @anw90 in #1160
• Fix vqa dataset loading by @tastelikefeet in #1195
• [TorchAcc] Add acc_steps arguments to speed up torchacc training by @anw90 in #1196
• fix minicpm-v data_collator by @Jintao-Huang in #1197
• Add a tuner to train a part of the model by @tastelikefeet in #1200
• Fix ConversationPreprocessor loading by @tastelikefeet in #1201
• Fix map cache on multi node by @tastelikefeet in #1203
• refactor multi-modal by @Jintao-Huang in #1212
• support llava 1.5 by @Jintao-Huang in #1217
• add in_browswer by @tastelikefeet in #1220
• [TorchAcc] Add USE_TORCH_XLA=0 flag for native swift scripts by @anw90 in #1221
• Fix glm4v batch_size>1 by @Jintao-Huang in #1223
• Add debug log support by @tastelikefeet in #1226
• Fix qlora deploy by @Jintao-Huang in #1224
• Add new dataset by @tastelikefeet in #1227
• fix refcoco by @tastelikefeet in #1232
• Fix bugs by @tastelikefeet in #1241
• fix bugs by @Jintao-Huang in #1242
• refactor inference by @Jintao-Huang in #1245
• Add more datasets by @tastelikefeet in #1246
• Support gemma2 by @Jintao-Huang in #1247
• Support gemma2 by @Jintao-Huang in #1248
• add push to hub tracker by @tastelikefeet in #1249
• support Florence series model by @hjh0119 in #1244
• update eval by @Jintao-Huang in #1251
• reset snapshot_download to modelscope by @tastelikefeet in #1254
• Feat/open compass by @tastelikefeet in #1213
• fix output empty will be filtered by @tastelikefeet in #1256
• add phi3 models by @tastelikefeet in #1259
• update dataset preprocess by @Jintao-Huang in #1257
• fix rlhf by @tastelikefeet in #1261
• fix rlhf by @Jintao-Huang in #1260
• Support llava1.6 mistral by @Jintao-Huang in #1268
• Add new dataset by @tastelikefeet in #1270
• support vllm & multimodal by @Jintao-Huang in #1229
• Fix bugs by @Jintao-Huang in #1275
• fix llamapro by @tastelikefeet in #1276
• fix xcomposer deploy by @Jintao-Huang in #1280
• Fix/0703 2 by @tastelikefeet in #1284
• Fix llamapro merge and quantized training by @tastelikefeet in #1289
• update lr_scheduler by @Jintao-Huang in #1288
• support internlm2.5 by @Jintao-Huang in #1291
• fix vllm by @Jintao-Huang in #1292
• fix florence & deploy by @Jintao-Huang in #1293
• fix mini-internvl-phi3 template by @Jintao-Huang in #1298
• fix florence by @hjh0119 in #1301
• Refactor UI and fix some bugs by @tastelikefeet in #1300
• fix dependency by @tastelikefeet in #1306
• support codegeex4 by @Jintao-Huang in #1305
• support internvl2 by @hjh0119 in #1304
• support llava video by @Jintao-Huang in #1307
• fix docs by @Jintao-Huang in #1309
• support lr_scheduler_kwargs by @Jintao-Huang in #1310
• Fix internvl2 template by @hjh0119 in #1308
• Fix bugs by @tastelikefeet in #1311
• support warmup_stable_decay by @Jintao-Huang in #1312
• Support xcomposer2.5 by @Jintao-Huang in #1287
• Fix bugs by @tastelikefeet in #1319

New Contributors

• @jiaozhentian made their first contribution in #1165
• @CjangCjengh made their first contribution in #1183
• @anw90 made their first contribution in #1160

Full Changelog: v2.1.1...v2.2.1

中文

新功能

• 支持了SimPO对齐算法，请查看文档
• 支持多模态部署能力
• web-ui支持量化和评测功能，并支持多模态界面推理和部署

新模型

• ChatGLM4和ChatGLMv
• Qwen2系列
• llava1.5/1.6系列模型
• mini-internvl系列模型
• paligemma系列模型
• Yuan2模型

Bug修复

请查看下方的详细提交记录

English

New Features

• Add SimPO alignment algorithm. Please refer to the documentation.
• Support for multimodal deployment capabilities.
• Web UI now supports quantization and evaluation command, as well as multimodal inference and deployment.

New Models

• ChatGLM4 and ChatGLMv
• Qwen2 series
• llava1.5/1.6 series models
• mini-internvl series models
• paligemma series models
• Yuan2 model

Bug fixing

Please check the update logs for details

What's Changed

• [TorchAcc][Experimental] Integrate more model in torchacc by @Zhikaiiii in #683
• update custom_val_dataset by @Jintao-Huang in #991
• fix val_dataset by @Jintao-Huang in #992
• fix args by @Jintao-Huang in #996
• fix pt deploy lora by @Jintao-Huang in #999
• Feat/phi3 paligemma by @tastelikefeet in #998
• Add copy to template encode by @tastelikefeet in #1003
• 更新了Command-line-parameters.md里面的一个链接 by @KPCOFGS in #1001
• fix cogvlm2 history by @Jintao-Huang in #1005
• fix minicpm-v by @Jintao-Huang in #1010
• Fix gradio conccurency limit by @tastelikefeet in #1016
• Update code by @Jintao-Huang in #962
• support paligemma by @tastelikefeet in #1004
• update docs table by @Jintao-Huang in #1021
• Fix docs table by @Jintao-Huang in #1024
• fix docs and a bug by @tastelikefeet in #1023
• fix arguments by @Jintao-Huang in #1026
• fix custom by @Jintao-Huang in #1028
• fix vllm==0.4.* slower than vllm==0.3.* by @Jintao-Huang in #1035
• fix app_ui host & port by @Jintao-Huang in #1036
• fix bugs by @Jintao-Huang in #1038
• support mini-internvl by @hjh0119 in #1032
• support multimodal deploy by @Jintao-Huang in #1029
• Support SimPO Algorithm by @hjh0119 in #1037
• fix phi3-vision bug by @Jintao-Huang in #1039
• update arguments by @Jintao-Huang in #1043
• update arguments by @Jintao-Huang in #1044
• fix deepseek-vl by @Jintao-Huang in #1046
• support sharegpt dataset format by @Jintao-Huang in #1052
• fix mini-internvl-4b by @hjh0119 in #1053
• Fix mini-internvl-4b by @hjh0119 in #1054
• fix vllm==0.4.3 by @Jintao-Huang in #1055
• fix deepseek-vl template by @Jintao-Huang in #1057
• Fix citest test_run.py by @Jintao-Huang in #1059
• ✏️ fix typo by @bertjiazheng in #1064
• fix argument by @hjh0119 in #1066
• Support glm4 by @Jintao-Huang in #1069
• update wechat by @Jintao-Huang in #1071

New Contributors

• @Zhikaiiii made their first contribution in #683
• @KPCOFGS made their first contribution in #1001
• @bertjiazheng made their first contribution in #1064

Full Changelog: v2.0.5...v2.1.0

New Features

1. Support for peft 0.10.x version, with the default value of the tuner_backend parameter changed to peft. The interface of peft has been dynamically patched to support parameters like lora_dtype.
2. Support for vllm+lora inference.
3. Refactored and updated the README file.
4. Added English versions of the documentation. Currently, all documents have both English and Chinese versions.
5. Support for training 70B models using FSDP+QLoRA on dual 24GB GPUs. Script available at: https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/llama2_70b_chat/qlora_fsdp/sft.sh
6. Support for training agents and using the ModelScopeAgent framework. Documentation available at: https://github.com/modelscope/swift/blob/main/docs/source/LLM/Agent%E5%BE%AE%E8%B0%83%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5.md
7. Support for model evaluation and benchmark. Documentation available at: https://github.com/modelscope/swift/blob/main/docs/source/LLM/LLM%E8%AF%84%E6%B5%8B%E6%96%87%E6%A1%A3.md
8. Support for multi-task experiment management. Documentation available at: https://github.com/modelscope/swift/blob/main/docs/source/LLM/LLM%E5%AE%9E%E9%AA%8C%E6%96%87%E6%A1%A3.md
9. Support for GaLore training.
10. Support for training and inference of AQLM and AWQ quantized models.

New Models

• MAMBA series models. Script available at: https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/mamba-1.4b/lora/sft.sh
• DeepSeek VL series models. Documentation available at: https://github.com/modelscope/swift/blob/main/docs/source_en/Multi-Modal/deepseek-vl-best-practice.md
• LLAVA series models. Documentation available at: https://github.com/modelscope/swift/blob/main/docs/source/Multi-Modal/llava%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5.md
• TeleChat models. Script available at: https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/telechat_12b/lora/sft.sh
• Grok-1 models. Documentation available at: https://github.com/modelscope/swift/blob/main/docs/source_en/LLM/Grok-1-best-practice.md
• Qwen 1.5 MoE series models for training and inference.
• dbrx models for training and inference. Script available at: https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/dbrx-instruct/lora_mp/sft.sh
• Mengzi3 models for training and inference. Script available at: https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/mengzi3_13b_base/lora_ddp_ds/sft.sh
• Xverse MoE models for training and inference. Script available at: https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/xverse_moe_a4_2b/lora/sft.sh
• c4ai-command-r series models for training and inference.
• MiniCPM series models for training and inference. Script available at: https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/minicpm_moe_8x2b/lora_ddp/sft.sh
• Mixtral-8x22B-v0.1 models for training and inference. Script available at: https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/mixtral_moe_8x22b_v1/lora_ddp_ds/sft.sh

New Datasets

• Support for the Ruozhiba dataset: https://github.com/modelscope/swift/blob/main/docs/source_en/LLM/Supported-models-datasets.md

What's Changed

• Fix RsLoRA by @tastelikefeet in #567
• Fix yi-vl merge lora by @Jintao-Huang in #568
• Add doc for tuner module by @tastelikefeet in #571
• update agent documentation by @tastelikefeet in #572
• Update agent doc to fix some conflicts by @tastelikefeet in #573
• support vllm lora by @Jintao-Huang in #565
• Support llava by @Jintao-Huang in #577
• fix app-ui max_length is None by @Jintao-Huang in #580
• support train_dataset_mix_ds using custom_local_path by @Jintao-Huang in #582
• Fix LRScheduler by @tastelikefeet in #586
• compat with transformers==4.39 by @Jintao-Huang in #584
• Fix weight saving by @tastelikefeet in #589
• fix mix_dataset_sample float by @Jintao-Huang in #594
• Refactor all docs by @tastelikefeet in #599
• fix tiny bugs in docs by @tastelikefeet in #600
• fix issue template and add a pr one by @tastelikefeet in #601
• Fix/security template by @tastelikefeet in #603
• update docs by @Jintao-Huang in #604
• support Mistral-7b-v0.2 by @hjh0119 in #605
• fix deploy safe_response by @Jintao-Huang in #614
• Fix Adalora with devicemap by @tastelikefeet in #619
• update ui by @tastelikefeet in #621
• support TeleChat-12b by @hjh0119 in #607
• fix save dir (additional_files) by @Jintao-Huang in #622
• fix Telechat model by @hjh0119 in #623
• Add Grok model by @tastelikefeet in #629
• add missing files by @tastelikefeet in #631
• support qwen1.5-moe model by @hjh0119 in #627
• support Telechat-7b model by @hjh0119 in #630
• support model Dbrx by @hjh0119 in #643
• fix ui by @tastelikefeet in #648
• fix typing hint by @Jintao-Huang in #649
• support Mengzi-13b-base model by @hjh0119 in #646
• support Qwen1.5-32b models by @hjh0119 in #655
• fix plot error by @tastelikefeet in #651
• Support FSDP + QLoRA by @tastelikefeet in #659
• move fsdp config path by @tastelikefeet in #662
• change the default value of ddp_backend by @tastelikefeet in #667
• fix ui log by @tastelikefeet in #669
• support Xverse-MoE model by @hjh0119 in #668
• Support longlora for transformers 4.38 by @tastelikefeet in #456
• add ruozhiba datasets by @tastelikefeet in #670
• compatible with old versions of modelscope by @tastelikefeet in #671
• Fix data_collator by @tastelikefeet in #674
• [TorchAcc][Experimental] Integrate TorchAcc. by @baoleai in #647
• update Agent best practice with Modelscope-Agent by @hjh0119 in #676
• support c4ai-command-r model by @hjh0119 in #684
• Support Eval by @tastelikefeet in #494
• fix anchor by @tastelikefeet in #687
• Fix/0412 by @tastelikefeet in #690
• support minicpm and mixtral-moe model by @hjh0119 in #692
• fix device_map 4 (qwen-vl) by @Jintao-Huang in #695
• fix multimodal model image_mode = 'CMYK' (fix issue#677) by @Jintao-Huang in #697
• feat(model): support minicpm-v-2(#699 ) by @YuzaChongyi in #699

New Contributors

• @hjh0119 made their first contribution in #6...

New Features:

1. Added support for swift export, enabling awq-int4 quantization and gpt-int2,3,4,8 quantization. Models can be pushed to the Modelscope Hub. You can view the documentation here.
2. Enabled fine-tuning of awq quantized models.
3. Enabled fine-tuning of aqlm quantized models.
4. Added support for deploying LLM with infer_backend='pt'.
5. Added web-ui with task management and visualization of training loss, eval loss, etc. Inference is accelerated using VLLM.

New Tuners:

1. Lora+.
2. LlamaPro.

New Models:

1. qwen1.5 awq series.
2. gemma series.
3. yi-9b.
4. deepseek-math series.
5. internlm2-1_8b series.
6. openbuddy-mixtral-moe-7b-chat.
7. llama2 aqlm series.

New Datasets:

1. ms-bench-mini.
2. hh-rlhf-cn series.
3. disc-law-sft-zh, disc-med-sft-zh.
4. pileval.

What's Changed

• Fix vllm==0.3 deploy bug by @Jintao-Huang in #412
• Support deepseek math by @Jintao-Huang in #413
• update support_vllm by @Jintao-Huang in #415
• fix zero3 & swift lora by @Jintao-Huang in #416
• Support peft0.8.0 by @tastelikefeet in #423
• update readme by @Jintao-Huang in #426
• fix pai open with 'a' by @Jintao-Huang in #430
• default load_best_model_at_end=False by @Jintao-Huang in #432
• support openbuddy mixtral by @Jintao-Huang in #437
• support gemma by @Jintao-Huang in #441
• Support ms bench mini by @Jintao-Huang in #442
• Add roadmap and contributing doc by @tastelikefeet in #431
• support peft format by @tastelikefeet in #438
• update contributing.md by @Jintao-Huang in #446
• fix link by @tastelikefeet in #447
• Fix rlhf dataset by @tastelikefeet in #451
• Add task management for webui by @tastelikefeet in #457
• Support swift export by @Jintao-Huang in #455
• Fix llm quantization docs by @Jintao-Huang in #458
• fix get_vllm_engine bug by @Jintao-Huang in #463
• use cpu export by @Jintao-Huang in #462
• Fix llama2 generation config by @Jintao-Huang in #468
• Support editing model_id_or_path by @tastelikefeet in #469
• Support pt deploy by @Jintao-Huang in #467
• Fix swift deploy bug by @Jintao-Huang in #470
• fix deploy dep by @Jintao-Huang in #471
• Support LLaMAPRO and LoRA+ by @tastelikefeet in #472
• Support internlm2 1.8b by @Jintao-Huang in #473
• fix deepseek moe device_map by @Jintao-Huang in #476
• fix peft compatible bug by @tastelikefeet in #482
• Fix deepspeed init bug by @Jintao-Huang in #481
• fix scripts docs by @Jintao-Huang in #483
• Update swift export and update docs by @Jintao-Huang in #484
• support gptq export by @Jintao-Huang in #485
• fix docs & readme by @Jintao-Huang in #486
• fix app-ui bug by @Jintao-Huang in #488
• Support peft0.9 by @tastelikefeet in #490
• support torchrun_args for dpo cli and support web_ui model deployment by @slin000111 in #496
• Support transformers 4.33.0 by @tastelikefeet in #498
• Update deepspeed config by @Jintao-Huang in #500
• move docs to classroom by @tastelikefeet in #503
• Support yi 9b by @Jintao-Huang in #504
• Update yi sh by @Jintao-Huang in #506

Full Changelog: v1.6.0...v1.7.0

New Models:

1. deepseek-math series

New Datasets:

1. sharegpt-gpt4-mini
2. disc-law-sft-zh
3. disc-med-sft-zh

Bug Fix

1. Fix vllm==0.3 & swift deploy bug.
2. Fix zero3 & swift lora bug.

Full Changelog: v1.6.0...v1.6.1

New Features:

1. Agent Training
2. AIGC support: controlnet, controlnet_sdxl, dreambooth, text_to_image, text_to_image_sdxl
3. Compatibility with vllm==0.3.*

New Models:

1. qwen1.5 series
2. openbmb series

What's Changed

• update openbmb sh by @Jintao-Huang in #361
• Fix openbmb model name by @tastelikefeet in #362
• support dpo cli and add examples controlnet and dreambooth by @slin000111 in #344
• support openbmb minicpm by @Jintao-Huang in #364
• Support agent training, etc. by @tastelikefeet in #352
• fix tuner by @tastelikefeet in #365
• Fix agent doc by @tastelikefeet in #366
• Fix data format in readme by @tastelikefeet in #367
• fix lazy_tokenize bug by @Jintao-Huang in #369
• Fix length penalty by @Jintao-Huang in #371
• fix loss by @tastelikefeet in #372
• update compute loss by @Jintao-Huang in #375
• fix system='' bug by @Jintao-Huang in #374
• fix system='' bug by @Jintao-Huang in #378
• Support PAI compat by @Jintao-Huang in #373
• fix doc by @tastelikefeet in #376
• Fix the conflict between agent and CT by @tastelikefeet in #379
• fix cogagent_18b_chat sh typo error by @Jintao-Huang in #381
• Fix loss scale by @tastelikefeet in #383
• Feat/qwen1.5 by @tastelikefeet in #385
• fix template name by @tastelikefeet in #389
• update readme by @Jintao-Huang in #386
• update readme by @Jintao-Huang in #390
• Support max model len by @Jintao-Huang in #392
• Support vllm max model len by @Jintao-Huang in #394
• fix arguments bug by @Jintao-Huang in #395
• support vllm 0.3 by @Jintao-Huang in #396
• fix deepspeed_config_path bug by @Jintao-Huang in #398
• fix file name by @slin000111 in #397
• Add qwen1.5 scripts by @tastelikefeet in #393
• fix many bugs by @Jintao-Huang in #399
• Fix baichuan2 int4 bug by @Jintao-Huang in #400
• Fix qwen1half deploy bug by @Jintao-Huang in #402
• fix readme and test_llm by @tastelikefeet in #404
• update readme by @Jintao-Huang in #405

Full Changelog: v1.5.4...v1.6.0