Summary of release notes:

Along with many improvements to experiment tracking, rollout logging, and configuration flexibility, new highlight features include:

• Support for T5-based student models. Check out this example, where we show how to fine-tune a FLAN-T5 model on CNN/DailyMail for summarization.

• Support for parameter-efficient tuning methods. Some of our preliminary results have shown LoRA to be a promising technique in scaling RLHF under low-resource settings and hope users get the chance to explore its potential. We've seen a ~30% reduction in memory usage and ~20% reduction in wallclock time for the same performance (quick report here)

• Out-of-the-box support for 8-bit Adam(W) optimizers via TimDettmers/bitsandbytes, leading to a 15% decrease in memory allocation in one of our baseline examples (related report).

Other interesting examples are in the works, so stay tuned!

What's Changed

• ILQL indicies on wrong device by @cat-state in https://github.com/CarperAI/trlx/pull/105
• Fix ppo ratio inaccuracy by @reciprocated in https://github.com/CarperAI/trlx/pull/108
• Set RNG seeds across multiple dependencies by @jon-tow in https://github.com/CarperAI/trlx/pull/113
• Set seed after default config instantiation by @jon-tow in https://github.com/CarperAI/trlx/pull/114
• Move queries on the device by @reciprocated in https://github.com/CarperAI/trlx/pull/115
• Add ppo randomwalks example by @reciprocated in https://github.com/CarperAI/trlx/pull/119
• Add unit tests to ensure valid example configs by @jon-tow in https://github.com/CarperAI/trlx/pull/120
• updating gptj-config by @Dahoas in https://github.com/CarperAI/trlx/pull/109
• Fix get distributed config by @reciprocated in https://github.com/CarperAI/trlx/pull/122
• Add local rollout logging by @thomfoster in https://github.com/CarperAI/trlx/pull/124
• Add support for more CausalLMs by @jon-tow in https://github.com/CarperAI/trlx/pull/103
• Add hydra head support for GPTNeo by @jon-tow in https://github.com/CarperAI/trlx/pull/126
• Add BloomModel hydra support by @jon-tow in https://github.com/CarperAI/trlx/pull/129
• Simplifying logic to merge configs by @leshanbog in https://github.com/CarperAI/trlx/pull/134
• add: load function for AccelerateRLModel by @dongs0104 in https://github.com/CarperAI/trlx/pull/136
• Add OptimizerConfig and SchedulerConfig by @jon-tow in https://github.com/CarperAI/trlx/pull/135
• Remove incorrect default config settings by @jon-tow in https://github.com/CarperAI/trlx/pull/137
• Update TRL acknowledgement by @osanseviero in https://github.com/CarperAI/trlx/pull/138
• Fix context overflow by @reciprocated in https://github.com/CarperAI/trlx/pull/131
• Fix seeding per process by @reciprocated in https://github.com/CarperAI/trlx/pull/141
• Set device-specific seeding with global rank by @jon-tow in https://github.com/CarperAI/trlx/pull/143
• Freeze hydra model branches by @jon-tow in https://github.com/CarperAI/trlx/pull/140
• Refactor RL model wrapper into a trainer module by @jon-tow in https://github.com/CarperAI/trlx/pull/144
• Logging learning rate by @leshanbog in https://github.com/CarperAI/trlx/pull/147
• Fix instantiating base transformer from a custom config by @reciprocated in https://github.com/CarperAI/trlx/pull/149
• Linear LR scheduler by @leshanbog in https://github.com/CarperAI/trlx/pull/150
• Update pre-commit version and add isort by @jon-tow in https://github.com/CarperAI/trlx/pull/152
• fix: configure flake8, fix errors, add trackers config by @Mistobaan in https://github.com/CarperAI/trlx/pull/157
• Features/use-python-3.8-in-ci by @Mistobaan in https://github.com/CarperAI/trlx/pull/159
• Add bitsandbytes optimizer support by @aicrumb in https://github.com/CarperAI/trlx/pull/133
• initial commit for trlx LORA support by @ethankim00 in https://github.com/CarperAI/trlx/pull/110
• Fix default delta_kwargs handling by @jon-tow in https://github.com/CarperAI/trlx/pull/171
• Add T5 model by @PhungVanDuy in https://github.com/CarperAI/trlx/pull/145
• Fix wandb.errors.RequireError as reported in #162 by @ayulockin in https://github.com/CarperAI/trlx/pull/167
• Update README.md by @LouisCastricato in https://github.com/CarperAI/trlx/pull/180
• Update ILQL details by @reciprocated in https://github.com/CarperAI/trlx/pull/156
• Add OpenAI Summarize RLHF with trlX by @PhungVanDuy in https://github.com/CarperAI/trlx/pull/175
• Fix HuggingFace model.save_pretrained for DDP by @jon-tow in https://github.com/CarperAI/trlx/pull/181
• Update generation utilities by @reciprocated in https://github.com/CarperAI/trlx/pull/172

New Contributors

• @thomfoster made their first contribution in https://github.com/CarperAI/trlx/pull/124
• @leshanbog made their first contribution in https://github.com/CarperAI/trlx/pull/134
• @dongs0104 made their first contribution in https://github.com/CarperAI/trlx/pull/136
• @osanseviero made their first contribution in https://github.com/CarperAI/trlx/pull/138
• @Mistobaan made their first contribution in https://github.com/CarperAI/trlx/pull/157
• @aicrumb made their first contribution in https://github.com/CarperAI/trlx/pull/133
• @ethankim00 made their first contribution in https://github.com/CarperAI/trlx/pull/110
• @PhungVanDuy made their first contribution in https://github.com/CarperAI/trlx/pull/145

Full Changelog: https://github.com/CarperAI/trlx/compare/v0.3...v0.4