We’re excited to introduce the release of Intel® Gaudi® software version 1.16.0, bringing forth numerous enhancements and updates for an improved GenAI development experience. We added preview support for vLLM, a fast and easy-to-use library for LLM inference and serving. Performance optimizations will be added in subsequent releases. For more information see the Intel Gaudi software vLLM.
Added preview support for Mixture of Experts (MoE) BF16 pretraining using Megatron-DeepSpeed. This release includes a recipe for pretraining Mixtral 4x7B and Mixtral 8x7B. For more information see Megatron-DeepSpeed models.
We have improved the performance of various LLM models, including LLaMA 2 7B/13B/70B FP8 for pre-training, and LLaMA 2 7B/70B FP8 and Mistral 7B for inference. For more information, check out the Intel Gaudi's model performance page.
In this release, we’ve also upgraded the validated versions of several libraries, including PyTorch* 2.2.2, DeepSpeed 0.14.0, PyTorch Lightning 2.2.4, Megatron-DeepSpeed PR #372, and Ray 2.20. Lastly, a reminder that support for Debian* 10.10 will be deprecated following the 1.16 release. In subsequent releases, Lazy mode execution will be deprecated for Intel Gaudi software, and PyTorch Eager mode and torch.compile will be the default execution mode.
For more information on this release, see the release notes.