Major new features since v0.0.4

• Support conversation templates
• Support new optimization algorithms, e.g. LISA
• Update requirements to support latest models
• Fix bugs in qlora/lora scripts
• Fix tokenization parallelism bug
• Improve script interfaces

What's Changed

• README refactor by @shizhediao in https://github.com/OptimalScale/LMFlow/pull/607
• Improve interface of finetuning scripts by @research4pan in https://github.com/OptimalScale/LMFlow/pull/611
• resize banner by @shizhediao in https://github.com/OptimalScale/LMFlow/pull/612
• Doc Reformat by @shizhediao in https://github.com/OptimalScale/LMFlow/pull/614
• Dev update transformers by @yaoguany in https://github.com/OptimalScale/LMFlow/pull/616
• Added QLoRA support for Decoder transformers with tune_strategy "Normal" by @TensorBlast in https://github.com/OptimalScale/LMFlow/pull/613
• announce long context support by @shizhediao in https://github.com/OptimalScale/LMFlow/pull/621
• fix deepspeed zero3 config bugs by @yaoguany in https://github.com/OptimalScale/LMFlow/pull/622
• Update version.py by @hendrydong in https://github.com/OptimalScale/LMFlow/pull/624
• update qr code by @shizhediao in https://github.com/OptimalScale/LMFlow/pull/625
• FIX BUG: trust_remote_code flag didn't take effect by @conght in https://github.com/OptimalScale/LMFlow/pull/633
• Add explanations about supported CUDA versions by @research4pan in https://github.com/OptimalScale/LMFlow/pull/634
• Added citation for RAFT by @shizhediao in https://github.com/OptimalScale/LMFlow/pull/635
• Update qrcode by @shizhediao in https://github.com/OptimalScale/LMFlow/pull/636
• [Features] Support multi_modal training by @lianqing11 in https://github.com/OptimalScale/LMFlow/pull/628
• Add scripts to convert raw file to text-only json by @research4pan in https://github.com/OptimalScale/LMFlow/pull/638
• speculative decoding by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/630
• [Feature] Speculative Inference by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/640
• add readme for speculative decoding by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/641
• update news about speculative decoding by @shizhediao in https://github.com/OptimalScale/LMFlow/pull/642
• update llama flash attention by @yaoguany in https://github.com/OptimalScale/LMFlow/pull/646
• update qrcode by @shizhediao in https://github.com/OptimalScale/LMFlow/pull/647
• [FIX] Fix multi-modal training by @lianqing11 in https://github.com/OptimalScale/LMFlow/pull/648
• Fix: --disable_group_texts 1 keep short samples by @research4pan in https://github.com/OptimalScale/LMFlow/pull/649
• Support all types with --disable_group_texts 1 by @research4pan in https://github.com/OptimalScale/LMFlow/pull/650
• Fix model downloading for CPU-only servers by @research4pan in https://github.com/OptimalScale/LMFlow/pull/651
• add block size to fingerprint by @RolandMinrui in https://github.com/OptimalScale/LMFlow/pull/653
• update qrcode by @shizhediao in https://github.com/OptimalScale/LMFlow/pull/654
• Update QR code for wechat by @research4pan in https://github.com/OptimalScale/LMFlow/pull/656
• update qrcode by @shizhediao in https://github.com/OptimalScale/LMFlow/pull/657
• update qr code for wechat by @research4pan in https://github.com/OptimalScale/LMFlow/pull/667
• Update version of datasets dependency by @research4pan in https://github.com/OptimalScale/LMFlow/pull/668
• Add flash attention install for A6000 by @research4pan in https://github.com/OptimalScale/LMFlow/pull/669
• Update hf_decoder_model.py by @yaoguany in https://github.com/OptimalScale/LMFlow/pull/670
• fix bugs in llama flash attention by @yaoguany in https://github.com/OptimalScale/LMFlow/pull/681
• code exection class and test cases by @Bob17293729 in https://github.com/OptimalScale/LMFlow/pull/674
• Update README to reflect changes in v0.0.6 by @research4pan in https://github.com/OptimalScale/LMFlow/pull/696
• fix bugs in requirements.txt since previous one can cause errors by @xu1868 in https://github.com/OptimalScale/LMFlow/pull/697
• fix merge lora bug by @Dominic789654 in https://github.com/OptimalScale/LMFlow/pull/698
• Upgrade transformers deps to support mistral by @research4pan in https://github.com/OptimalScale/LMFlow/pull/700
• add lisa code and lisa args by @Dominic789654 in https://github.com/OptimalScale/LMFlow/pull/701
• add GPU memory check script by @Dominic789654 in https://github.com/OptimalScale/LMFlow/pull/702
• Support multi-gpu inference by @research4pan in https://github.com/OptimalScale/LMFlow/pull/699
• src/lmflow/args.py typo fix by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/703
• add more info when fail to import flash attn by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/704
• Add script to finetune llama-2 with lisa by @research4pan in https://github.com/OptimalScale/LMFlow/pull/705
• add LISA args introduction by @Dominic789654 in https://github.com/OptimalScale/LMFlow/pull/706
• add lisa bibtex by @Dominic789654 in https://github.com/OptimalScale/LMFlow/pull/708
• Rpan add news by @research4pan in https://github.com/OptimalScale/LMFlow/pull/709
• Fix typo in README.md by @research4pan in https://github.com/OptimalScale/LMFlow/pull/710
• Update lisa code by @Dominic789654 in https://github.com/OptimalScale/LMFlow/pull/711
• Fix recent bad commits, which lead to OOM in 7B by @research4pan in https://github.com/OptimalScale/LMFlow/pull/712
• Add more memory saving options for LISA script by @research4pan in https://github.com/OptimalScale/LMFlow/pull/716
• Add stable version 0.0.7 by @research4pan in https://github.com/OptimalScale/LMFlow/pull/717
• Export --lisa_interval_steps by @research4pan in https://github.com/OptimalScale/LMFlow/pull/718
• Update documentation.yaml to fix github check failure when submitting PRs by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/723
• support more model by @Dominic789654 in https://github.com/OptimalScale/LMFlow/pull/724
• Update README.md to temporary allow latest fix by @research4pan in https://github.com/OptimalScale/LMFlow/pull/725
• Rpan dev lisa by @research4pan in https://github.com/OptimalScale/LMFlow/pull/715
• ShareGPT format support by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/720
• Add latest support for flash-attention from hf by @research4pan in https://github.com/OptimalScale/LMFlow/pull/727
• Fix LISA init OOM caused by deepspeed by @research4pan in https://github.com/OptimalScale/LMFlow/pull/728
• Improve installation documentation in README.md by @research4pan in https://github.com/OptimalScale/LMFlow/pull/733
• Add temporary dataset document by @research4pan in https://github.com/OptimalScale/LMFlow/pull/734
• Improve README and requirement.txt by @research4pan in https://github.com/OptimalScale/LMFlow/pull/735
• documentation update and action fix by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/736
• documentation page style update and readme dataset update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/737
• Add explanation for LISA with model-parallelism by @research4pan in https://github.com/OptimalScale/LMFlow/pull/738
• typo fix and disable html logo by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/739
• Update dependency for transformers by @research4pan in https://github.com/OptimalScale/LMFlow/pull/740
• Rpan lisa more models by @research4pan in https://github.com/OptimalScale/LMFlow/pull/742
• Fix: typo in --lisa_layers_attribute by @research4pan in https://github.com/OptimalScale/LMFlow/pull/743
• Add inferencer by @research4pan in https://github.com/OptimalScale/LMFlow/pull/744
• Improve LISA layers_attribute mapping logics by @research4pan in https://github.com/OptimalScale/LMFlow/pull/746
• Change --disable_group_texts default to True by @research4pan in https://github.com/OptimalScale/LMFlow/pull/747
• readme (zh) update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/748
• readme (zh) typo fix by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/749
• Change version to 0.0.7 in version.py by @research4pan in https://github.com/OptimalScale/LMFlow/pull/750
• readme (ko) update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/751
• Change lmflow.org to a temporary ip by @research4pan in https://github.com/OptimalScale/LMFlow/pull/753
• Add more clarification for data server problem by @research4pan in https://github.com/OptimalScale/LMFlow/pull/755
• Remove flash-attn in requirements.txt by @research4pan in https://github.com/OptimalScale/LMFlow/pull/757
• ko and zh readme update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/758
• ko readme update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/761
• [New feature] Integrate DPO by @gzliyu in https://github.com/OptimalScale/LMFlow/pull/762
• readme jp update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/763
• Recover data server by @research4pan in https://github.com/OptimalScale/LMFlow/pull/764
• remove data server address notice by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/765
• readme update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/766
• hindi readme update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/769
• conversation dataset (sharegpt format) custom template dev by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/767
• custom template document update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/770
• conversation template doc typo fix by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/771
• template info update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/772
• add support for llama-3 template by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/773
• run llama 3 with lisa by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/776
• Remove redundant statements in setup.py by @research4pan in https://github.com/OptimalScale/LMFlow/pull/779
• Add contributor support by @research4pan in https://github.com/OptimalScale/LMFlow/pull/780
• Contrib README typo fix by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/782
• add chatml conversation template by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/781
• Custom conversation template improvement and document update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/783
• Update examples for Full-param SFT and LISA by @research4pan in https://github.com/OptimalScale/LMFlow/pull/784
• Fixes & updates on lora, qlora scripts and hf_decoder_model by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/787
• Add phi3 conversation template support by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/788
• Add trust_remote_code option to finetune shells by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/789
• Finetune shell typo fix by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/790
• Remove lora qlora aggregated shell by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/792
• README update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/793
• README update, adding conversation template examples by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/794
• Add DPO support by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/797
• [New feature] Integrate DPO by @gzliyu in https://github.com/OptimalScale/LMFlow/pull/759
• README update, remove lora save aggregate shell by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/800
• Merge LoRA and base model by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/802
• README zh update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/803
• README kr update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/804
• README es update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/807
• README jp update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/805
• README hindi update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/808
• Fix eval_dataset number log. by @uApiv in https://github.com/OptimalScale/LMFlow/pull/809
• DeepSeek conversation template support by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/810
• Add DeepSeek template and template register by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/814
• Change conversation template file structure by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/817
• Yizhenjia template update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/819
• add lisa-diffusion project by @shaoshitong in https://github.com/OptimalScale/LMFlow/pull/818
• Add chatglm3 template by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/820
• Add chatglm3 template by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/822
• Support yi and yi1.5 template by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/824
• Yizhenjia template update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/826
• Add zephyr template by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/828
• Customized conversation template guide update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/829
• Guide update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/830
• Add finetuning doc by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/832
• Doc dataset page update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/835
• Add paired conversation dataset description by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/838
• Add supported models table by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/839
• Readme update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/843
• Reward modeling support by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/836
• [Bug fix] Tokenization multiprocessing fix by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/845
• [Bug fix] Blocking function args missing fix by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/846
• [Bug fix] Fix tokenizer multiprocessing in reward model by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/847
• Long context summarize demo by @HALIS-sh in https://github.com/OptimalScale/LMFlow/pull/844
• [Model Support] Qwen2 update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/849
• [Usability] Add preset lora target modules by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/850
• [Model] hf model modification and inheritance change by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/852
• Add langchain chatbot by @YanxinLu in https://github.com/OptimalScale/LMFlow/pull/848
• Add multi node README by @research4pan in https://github.com/OptimalScale/LMFlow/pull/853
• Usability update by @wheresmyhair in https://github.com/OptimalScale/LMFlow/pull/857

New Contributors

• @TensorBlast made their first contribution in https://github.com/OptimalScale/LMFlow/pull/613
• @conght made their first contribution in https://github.com/OptimalScale/LMFlow/pull/633
• @wheresmyhair made their first contribution in https://github.com/OptimalScale/LMFlow/pull/630
• @RolandMinrui made their first contribution in https://github.com/OptimalScale/LMFlow/pull/653
• @Bob17293729 made their first contribution in https://github.com/OptimalScale/LMFlow/pull/674
• @xu1868 made their first contribution in https://github.com/OptimalScale/LMFlow/pull/697
• @gzliyu made their first contribution in https://github.com/OptimalScale/LMFlow/pull/762
• @uApiv made their first contribution in https://github.com/OptimalScale/LMFlow/pull/809
• @shaoshitong made their first contribution in https://github.com/OptimalScale/LMFlow/pull/818
• @HALIS-sh made their first contribution in https://github.com/OptimalScale/LMFlow/pull/844
• @YanxinLu made their first contribution in https://github.com/OptimalScale/LMFlow/pull/848

Full Changelog: https://github.com/OptimalScale/LMFlow/compare/v0.0.4...v0.0.8