MS-AMP 0.3.0 Release Notes

MS-AMP Improvements

• Integrate latest Transformer Engine into MS-AMP
• Integrate with latest Megatron-LM
• Add a website for MS-AMP and improve documents
• Add custom DistributedDataParallel which supports FP8 and computation/computation overlap
• Refactor code in dist_op module
• Support UT for distributed testing
• Integrate with MSCCL

MS-AMP-Examples Improvements

• Support pretrain GPT-3 with Megatron-LM and MS-AMP
• Provide a tool to print the traffic per second of NVLINK and InfiniBand
• Print tflops and throughput metrics in all the examples

Document Improvements

• Add performance number in Introduction page
• Enhance Usage page and Optimization Level page
• Add Container Images page
• Add Developer Guide section