With GPT-NeoX 2.0, we now support upstream DeepSpeed. This enables the use of new DeepSpeed features such as Curriculum Learning, Communication Logging, and Autotuning.

For any changes in upstream DeepSpeed that are fundamentally incompatible with GPT-NeoX 2.0, we do the following:

• Attempt to create a PR to upstream DeepSpeed
• Stage the PR on DeeperSpeed 2.x, so that there's always a DeepSpeed version that's guaranteed to work with GPT-Neox 2.x.

Therefore, we recommend using DeeperSpeed 2.x unless your use-case relies on a specific upstream DeepSpeed feature that we haven't merged into DeeperSpeed 2.x yet.

What's Changed

• Mup Support in https://github.com/EleutherAI/gpt-neox/pull/704
• Bring deepspeed_main up-to-date in https://github.com/EleutherAI/gpt-neox/pull/746
• Latest DeepSpeed Support in https://github.com/EleutherAI/gpt-neox/pull/663
• Curriculum Learning Support in https://github.com/EleutherAI/gpt-neox/pull/695
• Autotuning Support in https://github.com/EleutherAI/gpt-neox/pull/739

Full Changelog: https://github.com/EleutherAI/gpt-neox/compare/v1.0...v2.0