English Version

New Features:

1. ms-swift paper published: https://arxiv.org/abs/2408.05517
2. Web-UI supports audio and video.
3. Support for deploying audio and video models using the OpenAI API.
4. Utilizes a new multimodal training framework.
5. supports inference acceleration for video models (lmdeploy & internvl2 series).

New Models:

1. idefics3-8b-llama3
2. llava-hf 72b, 110b, llama3-llava
3. deepseek-coder-v2, deepseek-coder-lite-v2, deepseek-v2

中文版

新功能：

1. 发布了 ms-swift 论文：https://arxiv.org/abs/2408.05517
2. Web-UI 支持音频和视频。
3. 支持使用 OpenAI API 部署音频和视频模型。
4. 采用新的多模态训练框架。
5. 支持视频模型的推理加速（lmdeploy 和 internvl2 系列）。

新模型：

1. idefics3-8b-llama3
2. llava-hf 72b、110b、llama3-llava
3. deepseek-coder-v2、deepseek-coder-lite-v2、deepseek-v2

What's Changed

• support video deploy by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1632
• fix infer deploy by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1665
• Fix bugs by @tastelikefeet in https://github.com/modelscope/ms-swift/pull/1664
• Add generation config support for swift eval by @Yunnglin in https://github.com/modelscope/ms-swift/pull/1659
• Add a bunch of small features by @tastelikefeet in https://github.com/modelscope/ms-swift/pull/1669
• Support lmdeploy&vllm support video by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1668
• [TorchAcc] fix: add mark_step when GA is greater than 1 by @baoleai in https://github.com/modelscope/ms-swift/pull/1677
• Support audio and video in web-ui by @tastelikefeet in https://github.com/modelscope/ms-swift/pull/1681
• Fix lmdeploy deploy seed by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1678
• add paper link by @tastelikefeet in https://github.com/modelscope/ms-swift/pull/1684
• set default writer_batch_size argument in mllm rlhf by @hjh0119 in https://github.com/modelscope/ms-swift/pull/1689
• New multimodal framework by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1594
• fix video component by @tastelikefeet in https://github.com/modelscope/ms-swift/pull/1692
• refactor internvl2 by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1625
• fix model dtype in rlhf by @hjh0119 in https://github.com/modelscope/ms-swift/pull/1696
• Fix minicpmv 2.6 & video/plain-text bug by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1698
• Fix internvl2 device map by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1701
• Support deepseek-v2 base by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1708
• [TorchAcc] update data bucketing strategy by @baoleai in https://github.com/modelscope/ms-swift/pull/1706
• [TorchAcc] consolidate FSDP ckpt after saving instead of during loading by @baoleai in https://github.com/modelscope/ms-swift/pull/1709
• fix deploy stream by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1713
• update docs by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1714
• Fix mllm rlhf by @hjh0119 in https://github.com/modelscope/ms-swift/pull/1707
• support idefics3 by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1720
• Fix minicpm-v-v2.6 device_map='auto' & agent deploy by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1733
• Fix kto by @hjh0119 in https://github.com/modelscope/ms-swift/pull/1737
• support llava-hf 72b, 110b, llama3-llava-hf by @Jintao-Huang in https://github.com/modelscope/ms-swift/pull/1741
• Fix glm4v rlhf by @hjh0119 in https://github.com/modelscope/ms-swift/pull/1745

New Contributors

• @Yunnglin made their first contribution in https://github.com/modelscope/ms-swift/pull/1659

Full Changelog: https://github.com/modelscope/ms-swift/compare/v2.3.0...v2.3.1