Microsoft DeepSpeed* Optimization on Large Models

As models get larger, to ensure they fit into device memory, libraries and techniques must be enabled to help reduce the memory size.

This webinar provides:

• The basic steps to enable Microsoft DeepSpeed* on Intel Gaudi AI accelerators
• Information on how ZeRO1 and ZeRO2 memory optimizers and activation checkpointing can be used to reduce memory use on a large model.
• Instructions on how to use Intel Gaudi AI software APIs to detect the peak memory of any model and provide guidance on when to use these techniques.
• A Q&A session.

Additional Resources

• Documentation
• DeepSpeed User Guide
• System access:
  • AWS* DL1 Instance
  • Intel® Tiber™ AI Cloud
• Model Repository
  • Tutorial: Enable DeepSpeed on an Intel Gaudi AI Accelerator
  • A Simple Get Started Guide for DeepSpeed with PyTorch*​
  • Scripts for Training a Megatron DeepSpeed Model (Based on GPT*)
  • Examples of Hugging Face* Model Architectures, Tasks, and Device Distributions
• User Forum

Featured Speakers
 

Greg Serochi

Developer advocate and applications engineering