MyGit

v2.3.0

modelscope/ms-swift

版本发布时间: 2024-08-09 23:43:07

modelscope/ms-swift最新发布版本:v2.5.0(2024-10-10 10:21:04)

English Version

New Features

  1. Support for readthedocs documentation site at: https://swift.readthedocs.io/en/latest
  2. Support Megatron architecture training for QianWen series models, and added new pt command for pretraining. See docs: https://swift.readthedocs.io/en/latest/LLM/Megatron-training.html
  3. Support LMDeploy for inference and deployment, improving inference acceleration for multi-modal models. See: https://swift.readthedocs.io/en/latest/Multi-Modal/LmDeploy-inference-acceleration.html
  4. Support passing lora target modules via regular expressions
  5. Support configuring max_memory usage for each GPU in device_map
  6. export command supports BitsAndBytes quantization
  7. export command supports Ollama export: https://swift.readthedocs.io/en/latest/LLM/OLLaMA-Export.html
  8. Support Q-GaLore algorithm
  9. Support RLHF training for multi-modal models: https://swift.readthedocs.io/en/latest/Multi-Modal/human-preference-alignment-training-documentation.html
  10. Support evaluation on 100+ datasets for multi-modal models: https://swift.readthedocs.io/en/latest/LLM/LLM-eval.html
  11. Support resizing input images when memory usage is too high for multi-modal models
  12. Modified default lora injection for multi-modal model training. Now takes effect on LLM and projector, results are better without significantly increasing training memory.
  13. Support PEFT 0.12, and added new tuner: fourierft
  14. Support rope-scaling for multi-modal models
  15. Support streaming processing of datasets to reduce memory usage, enable with --streaming
  16. Support vLLM multi-modal inference and deployment
  17. Support grounding task for popular multi-modal models.

New Models

  1. qwen2-audio series
  2. qwen2-math
  3. codegeex4
  4. internvl2 series
  5. llava video
  6. xcomposer2.5
  7. cogvlm2-video
  8. numina-math
  9. mistral-nemo
  10. llama3.1 series
  11. mistral-large
  12. gemma-2-2b
  13. internlm2.5 1.8b 20b
  14. minicpm-v-v2_6-chat

Check: https://swift.readthedocs.io/en/latest/LLM/Supported-models-datasets.html

New Datasets

  1. zhihu-kol and zhihu-kol-filtered
  2. SA1B series multi-modal zh datasets

Check: https://swift.readthedocs.io/en/latest/LLM/Supported-models-datasets.html

中文版本

新功能

  1. 支持readthedocs文档库， 地址：https://swift.readthedocs.io/zh-cn/latest
  2. 支持千问系列模型的Megatron结构训练，并支持了新的pt命令用于预训练，详见文档：https://swift.readthedocs.io/zh-cn/latest/LLM/Megatron%E8%AE%AD%E7%BB%83%E6%96%87%E6%A1%A3.html
  3. 支持LMDeploy的推理和部署，更好地支持了多模态模型的推理加速，详见：https://swift.readthedocs.io/zh-cn/latest/Multi-Modal/LmDeploy%E6%8E%A8%E7%90%86%E5%8A%A0%E9%80%9F%E6%96%87%E6%A1%A3.html
  4. 支持以正则表达式方式传入lora target模块
  5. 支持配置device_map各GPU用量的max_memory
  6. export命令支持BitsAndBytes量化
  7. export命令支持Ollama导出：https://swift.readthedocs.io/zh-cn/latest/LLM/OLLAMA%E5%AF%BC%E5%87%BA%E6%96%87%E6%A1%A3.html
  8. 支持Q-GaLore算法
  9. 支持多模态模型的RLHF训练：https://swift.readthedocs.io/zh-cn/latest/Multi-Modal/%E4%BA%BA%E7%B1%BB%E5%81%8F%E5%A5%BD%E5%AF%B9%E9%BD%90%E8%AE%AD%E7%BB%83%E6%96%87%E6%A1%A3.html
  10. 支持多模态模型100+数据集的评测能力：https://swift.readthedocs.io/zh-cn/latest/LLM/LLM%E8%AF%84%E6%B5%8B%E6%96%87%E6%A1%A3.html
  11. 支持多模态模型显存占用过高时对输入图片进行缩放
  12. 修改了多模态模型训练的默认lora注入，目前对LLM和projector生效，不显著提高训练显存情况下效果更好
  13. 支持PEFT0.12，并支持了新的tuner：fourierft
  14. 支持多模态模型的rope-scaling
  15. 支持数据集的流式处理，降低显存消耗，使用--streaming开启
  16. 支持了vLLM的多模态推理部署能力
  17. 对部分多模态模型支持了grounding任务

新模型

  1. qwen2-audio系列模型
  2. qwen2-math
  3. codegeex4
  4. internvl2系列模型
  5. llava video
  6. xcomposer2.5
  7. cogvlm2-video
  8. numina-math
  9. mistral-nemo
  10. llama3.1系列
  11. mistral-large
  12. gemma-2-2b
  13. internlm2.5 1.8b 20b
  14. minicpm-v-v2_6-chat

参考：https://swift.readthedocs.io/zh-cn/latest/LLM/%E6%94%AF%E6%8C%81%E7%9A%84%E6%A8%A1%E5%9E%8B%E5%92%8C%E6%95%B0%E6%8D%AE%E9%9B%86.html

新数据集

  1. zhihu-kol和zhihu-kol-filtered数据集
  2. SA1B系列中文多模态数据集

参考：https://swift.readthedocs.io/zh-cn/latest/LLM/%E6%94%AF%E6%8C%81%E7%9A%84%E6%A8%A1%E5%9E%8B%E5%92%8C%E6%95%B0%E6%8D%AE%E9%9B%86.html

What's Changed

New Contributors

Full Changelog: https://github.com/modelscope/ms-swift/compare/v2.2.0...v2.3.0

相关地址：原始地址 下载(tar) 下载(zip)

查看：2024-08-09发行的版本

微信小程序
MyGit:GitHub仓库更新&通知小工具