New Model Support 

• Support for Llama 3.2 (1B and 3B), Gemma 2 (2B and 9B), and YOLO* v11

• LLM support on NPUs: Llama 3 8B, Llama 2 7B, Mistral-v0.2-7B, Qwen2-7B-Instruct, and Phi-3 Mini Instruct

• Preview: Support for Flax, a high-performance Python* neural network library based on JAX*. Its modular design allows for easy customization and accelerated inference on GPUs. 

New Jupyter* Notebooks

Noteworthy notebooks added: Sam2, Llama3.2, Llama3.2 - Vision, Wav2Lip, Whisper, and Llava. 

Intel® GPUs GPU Optimizations  

• Optimizations for built-in GPUs on Intel® Core™ Ultra Processors (Series 1) and Intel® Arc™ Graphics include KV Cache compression for memory reduction along with improved usability and model load time optimizations to improve first token latency for LLMs.

• Dynamic quantization was enabled to improve first token latency for LLMs on built-in Intel® GPUs without impacting accuracy on Intel Core Ultra processors (Series 1). Second token latency will also improve for large batch inference.

NNCF Updates

The Neural Network Compression Framework (NNCF) implements a new method for generating synthetic text data. This allows LLMs to be compressed more accurately using data-aware methods without datasets.

Intel® Hardware Support

Support for Intel® Xeon® 6 processors with P-cores (formerly codenamed Granite Rapids) and Intel Core Ultra 200V processor family (formerly codenamed Arrow Lake-S) 

GenAI API Enhancements 

• Preview: GenAI API enables multimodal AI deployment with support for multimodal pipelines for improved contextual awareness, transcription pipelines for easy audio-to-text conversions, and image generation pipelines for streamlined text-to-visual conversions. 

• A speculative decoding feature was added to the GenAI API for improved performance and efficient text generation using a small draft model that is periodically corrected by the full-size model. 

• Preview: LoRA adapters are now supported in the GenAI API for developers to quickly and efficiently customize image and text generation models for specialized tasks. 

• The GenAI API now also supports LLMs on NPUs, allowing developers to specify NPU as the target device.