Highlights

LLM and MM

Models

• Megatron Core RETRO

  • Pre-training
  • Zero-shot Evaluation

• Pretraining, conversion, evaluation, SFT, and PEFT for:

  • Mixtral 8X22B
  • Llama 3
  • SpaceGemma

• Embedding Models Fine Tuning

  • Mistral
  • BERT

• BERT models

  • Context Parallel
  • Distributed checkpoint

• Video capabilities with NeVa

Performance

• Distributed Checkpointing

  • Torch native backend
  • Parallel read/write
  • Async write

• Multimodal LLM (LLAVA/NeVA)

  • Pipeline Parallelism support
  • Sequence packing support

Export

• Integration of Export & Deploy Modules into NeMo Framework container
  • Upgrade to TRT-LLM 0.9

Speech (ASR & TTS)

Models

• AED Multi Task Models (Canary) - Multi-Task Multi-Lingual Speech Recognition / Speech Translation model
• Multimodal Domain - Speech LLM supporting SALM Model
• Parakeet-tdt_ctc-1.1b Model - RTFx of > 1500 (can transcribe 1500 seconds of audio in 1 second)
• Audio Codec 16kHz Small - NeMo Neural Audio Codec for discretizing speech for use in LLMs
  • mel_codec_22khz_medium
  • mel_codec_44khz_medium

Perf Improvements

• Transcribe() upgrade - Enables one line transcribe with files, tensors, data loaders
• Frame looping algorithm for RNNT faster decoding - Improves Real Time Factor (RTF) by 2-3x
• Cuda Graphs + Label-Looping algorithm for RNN-T and TDT Decoding - Transducer Greedy decoding at over 1500x RTFx, on par with CTC Non-Autoregressive models
• Semi Sorted Batching support - External User contribution that speeds up training by 15-30%.

Customization

• Context biasing for CTC word stamping - Improve accuracy for custom vocabulary and pronunciation
  • Longform Inference
  • Longform inference support for AED models
• Transcription of multi-channel audio for AED models

Misc

• Upgraded webdataset - Speech and LLM / Multimodal unified container

Detailed Changelogs

ASR

TTS

LLM and MM

Export

General Improvements