Features：

1. Support pipeline parallel, including interleaved and non-interleaved pipeline scheduler.
2. Support sequence parallel.
3. Support model evaluation.
4. Support tf32 with flash-attention.
5. Support tensorboard writer for recording training performance metrics.
6. Support customed uniscale logger.
7. Support calculating model's accuracy and perplexity metrics.
8. Support oss storage and checkpoint asynchronous uploading.
9. Support automatically loading the latest checkpoint.
10. Support checkpoint snapshot.
11. Support monitoring the status of training jobs, and alarm abnormal status.
12. Support torch profiler.
13. Support simple memory profiler.

Optimizations：

1. Overlapping optimizer parameters broadcast with model forward.
2. Overlapping optimizer last bucket gradients allreduce with compute norm.