core_v0.4.0
版本发布时间: 2023-12-15 07:18:34
NVIDIA/Megatron-LM最新发布版本:core_r0.8.0(2024-08-13 20:12:40)
Key Features and Enhancements
Models
- BERT
- RETRO
- T5
Parallelism
- Mixture of Experts support for GPT
- Model parallel efficient Distributed Data Parallel (DDP)
- Context Parallel (2D Tensor Parallel) support
Datasets
- GPT Dataset
- Blended Dataset