Bug Fixes

1. Increased Robustness to Checkpoint Loading

We've patched several edge cases in loading sharded checkpoints, especially with DTensors, which should decrease memory usage when loading checkpoints. We've also hardened retry logic against object cloud failure, ensuring higher robustness to transient network issues.

What's Changed

• Raise daily test timeout by @mvpatel2000 in https://github.com/mosaicml/composer/pull/3172
• fix remote file naming by @cli99 in https://github.com/mosaicml/composer/pull/3173
• [fix] DTensor + SHARD_GRAD_OP + use_orig_params by @bigning in https://github.com/mosaicml/composer/pull/3175
• Bump db sdk by @dakinggg in https://github.com/mosaicml/composer/pull/3176
• Build latest pytorch nightly images by @dakinggg in https://github.com/mosaicml/composer/pull/3179
• Add FP8 TransformerEngine activation checkpointing by @cli99 in https://github.com/mosaicml/composer/pull/3156
• Enabling the computation of validation loss and other metrics when using sequence parallelism by @ShashankMosaicML in https://github.com/mosaicml/composer/pull/3183
• Update mosaic_fsdp_utils.py by @vchiley in https://github.com/mosaicml/composer/pull/3185
• Fix the FSDP.optim_state_dict_to_load OOM by @bigning in https://github.com/mosaicml/composer/pull/3184
• Revert "Update mosaic_fsdp_utils.py" by @vchiley in https://github.com/mosaicml/composer/pull/3187
• Bump databricks-sdk from 0.24.0 to 0.25.1 by @dependabot in https://github.com/mosaicml/composer/pull/3190
• Add version tag to local builds by @mvpatel2000 in https://github.com/mosaicml/composer/pull/3188
• Update NeptuneLogger by @AleksanderWWW in https://github.com/mosaicml/composer/pull/3165
• Filter neptune warning in doctests by @mvpatel2000 in https://github.com/mosaicml/composer/pull/3195
• Removal of metrics deepcopy before computing the metrics by @gregjauvion in https://github.com/mosaicml/composer/pull/3180
• Fix MLFlow Tag Name for Resumption by @KuuCi in https://github.com/mosaicml/composer/pull/3194
• Fix mistral gating by @dakinggg in https://github.com/mosaicml/composer/pull/3199
• Bump version to 0.21.3 by @mvpatel2000 in https://github.com/mosaicml/composer/pull/3198

New Contributors

• @gregjauvion made their first contribution in https://github.com/mosaicml/composer/pull/3180

Full Changelog: https://github.com/mosaicml/composer/compare/v0.21.2...v0.21.3