English Version

New Features:

1. Support for GPTQ & AWQ quantization of multimodal LLMs.
2. Support for dynamic addition of gradient checkpointing in the ViT section to reduce memory consumption.
3. Support for multimodal model pre-training.

New Models:

1. llama3.2, llama3.2-vision series
2. got-ocr2
3. llama3.1-omni
4. ovis1.6-gemma2
5. pixtral-12b
6. telechat2-115b
7. mistral-small-inst-2409

New Datasets:

1. egoschema

中文版

新特性：

1. 支持多模态LLM的gptq&awq量化.
2. 支持动态在vit部分增加gradient_checkpointing, 减少显存消耗.
3. 支持多模态模型预训练.

新模型：

1. llama3.2, llama3.2-vision系列
2. got-ocr2
3. llama3.1-omni
4. ovis1.6-gemma2
5. pixtral-12b
6. telechat2-115b
7. mistral-small-inst-2409

新数据集：

1. egoschema

What's Changed

• fix win32 quote by @tastelikefeet in https://github.com/modelscope/ms-swift/pull/2065
• Fix yi template by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2067
• fix rlhf zero3 by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2072
• Update qwen2-vl最佳实践.md by @Digital2Slave in https://github.com/modelscope/ms-swift/pull/2058
• fix RLHF & max_length by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2075
• Support Mistral-small-inst-2409 by @DaozeZhang in https://github.com/modelscope/ms-swift/pull/2077
• dynamic vit gradient_checkpointing by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2071
• fix qwen2.5 template by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2081
• fix multiprocess remove_columns by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2088
• Support for fine-tuning Pixtral-12B. by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2090
• fix vllm tokenizer by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2099
• Fix the issue with media_offset in owl3 when batch_size > 1. by @LukeForeverYoung in https://github.com/modelscope/ms-swift/pull/2100
• fix deploy openai compat by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2101
• fix dataset preprocess by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2102
• fix cpu infer device_map by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2103
• fix infer device_map by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2105
• Support for fine-tuning Llama 3.1 Omni. by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2106
• support vllm & qwen2-vl video by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2110
• Fix qwen2-vl zero2/3 by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2114
• fix qwen2-audio by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2116
• [TorchAcc] fix: fix find_labels and can_return_loss by @baoleai in https://github.com/modelscope/ms-swift/pull/2120
• support got-ocr2 by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2123
• Support for fine-tuning and deployment of the Llama 3.2 series models. by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2130
• Support fine-tuning MLLama. by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2132
• fix not impl bug by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2134
• Compat vllm & qwen2-vl by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2136
• fix requirements by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2137
• fix model_type by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2138
• fix deploy vllm by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2141
• fix docs by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2142
• Fix VLM lora by @tastelikefeet in https://github.com/modelscope/ms-swift/pull/2140
• support mllm pt by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2146
• [TorchAcc] fix: fix save config and additional file for swift and peft by @baoleai in https://github.com/modelscope/ms-swift/pull/2149
• update quant_device_map by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2154
• fix qwen2-audio by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2157
• fix template by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2160
• compat trl==0.11 by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2169
• Support for Egoschema, a new video dataset by @DaozeZhang in https://github.com/modelscope/ms-swift/pull/2173
• Update FAQ by @slin000111 in https://github.com/modelscope/ms-swift/pull/2165
• fix mplug-owl3 infer by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2175
• Support quant mllm by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2177
• update setup.py by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2205
• fix bugs by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2207
• support telechat2 by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2210
• Support ovis 1.6 by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/2211

New Contributors

• @Digital2Slave made their first contribution in https://github.com/modelscope/ms-swift/pull/2058
• @LukeForeverYoung made their first contribution in https://github.com/modelscope/ms-swift/pull/2100

Full Changelog: https://github.com/modelscope/ms-swift/compare/v2.4.2...v2.5.0