A person stands at the end of a row of servers looking at bright blue data points projected on a wall in the distance

What Is Intel® Advanced Matrix Extensions (Intel® AMX)?

Expand, simplify, and accelerate your AI capabilities to meet compute demands for deep learning workloads with this integrated accelerator on the latest generations of Intel® Xeon® Scalable processors.1

Key Takeaways

  • Intel® AMX is part of the Intel® AI Engines available on the latest generations of Intel® Xeon® Scalable processors.

  • Intel® AMX accelerates deep learning training and inferencing workloads and minimizes the need for additional specialized hardware.

  • Intel® developer tools and enablement resources help make it easier to take advantage of Intel® AMX.

author-image

By

Accelerate your demanding deep learning training and inferencing workloads by taking advantage of Intel® Advanced Matrix Extensions (Intel® AMX), an integrated accelerator on the latest generations of Intel® Xeon® Scalable processors, including the upcoming Intel® Xeon® 6 processor with P-cores. Learn how to activate Intel® AMX on your CPU with our step-by-step instruction guides so you can start optimizing your AI pipeline, improving efficiency, maximizing value, and reducing TCO at your organization.

Intel® Xeon® Scalable Processors and Intel® Advanced Matrix Extensions

Deep learning workloads, such as those that that rely on generative AI, large language models (LLMs), and computer vision, can be incredibly compute intensive, requiring high levels of performance and, often, additional specialized hardware to ensure successful AI deployment. The associated costs of these requirements can quickly escalate, and adding discrete hardware solutions can create unnecessary layers of complexity and compatibility issues.

To help make your deep learning workloads more efficient and cost-effective and easier to train and deploy, Intel® AMX on Intel® Xeon® Scalable processors delivers acceleration for inferencing and training while minimizing the need for specialized hardware.

Intel® AMX is one of two Intel® AI Engines integrated into 4th Gen Intel Xeon, 5th Gen Intel Xeon, and Intel® Xeon® 6 processors with P-cores, that can help you make the most of your CPU to power AI training and inferencing workloads at scale for benefits including improved efficiency, reduced inferencing, training, and deployment costs, and lower total cost of ownership (TCO). As a built-in accelerator that resides on each CPU core and placed near system memory, Intel® AMX is often less complex to use than discrete accelerators, leading to faster time to value.

While there are many ways organizations can support advanced AI workloads, a foundation based on Intel® Xeon® Scalable processors with powerful, integrated AI accelerators can help you achieve your training and inferencing performance objectives while reducing system complexity and deployment and operational costs for greater business return.

How Intel® AMX Works

Intel® AMX is a dedicated hardware block found on the Intel® Xeon® Scalable processor core that helps optimize and accelerate deep learning training and inferencing workloads that rely on matrix math.

Intel® AMX enables AI workloads to run on the CPU instead of offloading them to a discrete accelerator, providing a significant performance boost.2 Its architecture supports BF16 (training/inference) and int8 (inference) data types and includes two main components:

  • Tiles: These consist of eight two-dimensional registers, each 1 kilobyte in size, that store large chunks of data.
  • Tile Matrix Multiplication (TMUL): TMUL is an accelerator engine attached to the tiles that performs matrix-multiply computations for AI.

Together, these components enable Intel® AMX to store more data in each core and compute larger matrices in a single operation. Additionally, Intel® AMX is architected to be fully extensible and scalable.

Benefits of Intel® AMX for Better Business Outcomes

Intel® AMX enables Intel® Xeon® Scalable processors to boost the performance of deep learning training and inferencing workloads by balancing inference, the most prominent use case for a CPU in AI applications, with more capabilities for training.

Many Intel customers are taking advantage of Intel® AMX to enable better outcomes for their organizations. Focusing on GenAI workloads, Intel® Xeon® 6 processors with P-cores can deliver 2x higher GPT-J-6B (bf16) performance vs. 5th Gen Intel Xeon 3. On 5th Gen Intel® Xeon® processors, customers can experience up to 14x better training and inference vs. 3rd Gen Intel® Xeon® processors.4

Primary benefits of Intel® AMX include:

  • Improved performance
    CPU-based acceleration can improve power and resource utilization efficiencies, giving you better performance for the same price.
    For example, 5th Gen Intel® Xeon® Platinum 8592+ with Intel® AMX BF16 has shown up to 10.7x higher real-time speech recognition inference performance (RNN-T) and 7.9x higher performance/watt vs. 3rd Gen Intel® Xeon® processors with FP32.5
  • Reduced total cost of ownership (TCO)
    Intel® Xeon® Scalable processors with Intel® AMX enable a range of efficiency improvements that help with decreasing costs, lowering TCO, and advancing sustainability goals.
    As an integrated accelerator on Intel® Xeon® Scalable processors that you may already own, Intel® AMX enables you to maximize the investments you’ve already made and get more from your CPU, removing the cost and complexity typically associated with the addition of a discrete accelerator.
    Intel® Xeon® Scalable processors with Intel® AMX can also provide a more cost-efficient server architecture compared to other available options, delivering both power and emission reduction benefits.
    In a comparison with AMD Genoa 9654 servers, 5th Gen Intel® Xeon® Platinum processors with Intel® AMX delivered up to 2.69x higher batched Natural Language Processing inference (BERT-Large) performance and 2.96x higher performance per watt.6
  • Reduced development time
    To simplify the process of developing deep learning applications, we work closely with the open source community, including the TensorFlow and PyTorch projects, to optimize frameworks for Intel® hardware, upstreaming our newest optimizations and features so they’re immediately available to developers. This enables you to take advantage of the performance benefits of Intel® AMX with the addition of a few lines of code, reducing overall development time.
    We also provide access to free Intel® development tools, libraries, and resources.

Intel® AMX Deep learning Use Cases

Intel® AMX can be deployed in a wide range of deep learning use cases to provide a significant performance boost that results in greater end user and business value.

  • Recommender systems: Use Intel® AMX as a more cost-effective solution for AI recommender models that boost the responsiveness of product, content, and service recommendations for use cases, including e-commerce, social media, streaming entertainment, and personalized banking. For example, content providers often use Intel® AMX to accelerate delivery of targeted movie or book recommendations and ads or to deliver a deep learning‒based recommender system that accounts for real-time user behavior signals and context features such as time and location in near-real time. 5th Gen Intel® Xeon® processors are delivering up to 8.7x higher batch Recommendation System inference performance (DLRM) and 6.2x higher performance/watt vs. 3rd Gen Intel® Xeon® processors with FP32.7
  • Natural language processing (NLP): Accelerate text-based use cases to support and scale NLP applications, such as those used in healthcare and life sciences to extract insights from clinical notes or process large amounts of medical data to help with early detection of health issues and improve care delivery. In financial services, Intel® AMX can be used to improve online chatbot responsiveness to help connect customers with the information they need more quickly while freeing limited staff up to address more-complex requests.
    Similar to the cost savings benefits for recommender systems, Intel® AMX can be a more cost-effective solution for NLP. For example, when used to deploy the BERT-Large AI Natural Language model, Intel® AMX on 4th Gen Intel® Xeon® processors provided up to 79 percent savings when compared to AMD Genoa 9354.8
  • Generative AI: Leverage Intel® AMX to accelerate the performance of deep learning training and inference workloads for generative AI use cases such as content generation, including images, videos, and audio, language translation, data augmentation, and summarization. For example, a performance evaluation of Intel® Xeon® Platinum 8480+ processors with Intel® AMX for BF16 data types compared to Intel® Xeon® Platinum 8380 processors for FP32 data types reduced Stable Diffusion text to image generation time to less than five seconds and fine-tuning of Stable Diffusion models to less than five minutes.9
  • Computer vision: Reduce the time from video and image capture to insight and action to deliver exceptional customer experiences and help your business improve efficiency and reduce operational costs. For example, in retail stores, Intel® AMX can help minimize transaction time for customers using computer vision‒enabled frictionless checkout and support near-real-time monitoring of shelves to track inventory data and instantly notify staff when an item is out of stock. In manufacturing, accelerated analysis of video from computer vision cameras on robotic arms can help enable time and cost savings with automated defect detection capabilities.

To find additional examples of how Intel® customers are using Intel® AMX to drive better business outcomes, visit our customer spotlight library.

Get Started with Intel® AMX

We offer a wide variety of development resources to help you take advantage of the integrated Intel® AMX accelerator in your Intel® Xeon® Scalable processors.

To get started, review step-by-step instructions for boosting performance with Intel® AMX in the following guides:

For more in-depth technical information, tutorials, code examples, and testing modules, access:

You can access all of our tuning guides for Intel® Xeon® Scalable processors in our developer software tools catalog.

To help you streamline your AI development efforts, we offer our Intel® oneAPI Toolkits, components, and optimizations, including:

Experiment with Intel® AMX Today

In addition to consulting our reference materials, you can experiment with Intel® hardware, Intel® AMX, and other integrated acceleration features using Intel® Developer Cloud.

This free online platform for learning, prototyping, testing, and running workloads also includes support for a number of Intel® software development toolkits, tools, and libraries.

Expand and Enhance AI Capabilities on Your CPU with Intel® AMX

As your organization looks for solutions to meet growing compute demands to support deep learning training and inferencing workloads, Intel® AMX can help boost performance using the Intel® hardware you may already own, without the cost and complexity that comes with additional specialized hardware and in comparably less development time, using Intel® optimizations in popular open source frameworks and access to free Intel® development tools and resources.

Discover More

Intel® Accelerator Engines

Learn more about built-in acceleration capabilities in Intel® Xeon® Scalable processors.

Browse engines

Intel® Xeon® Scalable Processors

Find out why these powerful processors are the right choice for your demanding workloads.

Browse processors

Product and Performance Information

1

Availability of accelerators varies depending on SKU. Visit the Intel Product Specifications page for additional product details.

2

See [A16] and [A17] at intel.com/processorclaims: 4th Gen Intel® Xeon® Scalable processors. Results may vary.

3

See [9A10] at intel.com/processorclaims: Intel® Xeon® 6. Results may vary.

4

See [A15-A16] at intel.com/processorclaims: 5th Gen Intel Xeon Scalable processors. Results may vary.

5

See [A23] at intel.com/processorclaims: 5th Gen Intel Xeon processors. Results may vary.

6

See [A210] at intel.com/processorclaims: 5th Gen Intel Xeon processors. Results may vary.

7

See [A20] at intel.com/processorclaims: 5th Gen Intel Xeon processors. Results may vary.

8

Based on 4th Gen Intel® Xeon® processors that deliver up to 5.60x faster than the 4th Gen AMD EPYC while running a BERT-Large workload. This performance drives a fleet reduction from 50 to nine servers, which saves USD 424K kWh per year of electricity, reduces 719,546 kgCO2 over four years, and drives a cost savings of USD 1.38M. BERT-Large: 8462Y+: 1-node, 2x 4th Gen Intel® Xeon® Scalable processor 8462Y+(32c/2.8 GHz, 300W TDP) on Supermicro SYS-221H-TNR server with 1024 GB (16x64 GB/4800) DDR5 memory, microcode 0x2b000161, HT on, Turbo on, SNC off, Ubuntu 22.04.2 LTS, 5.15.0-58-generic, 1x 1.92T SAMSUNG MZQL21T9HCJR-00A07, Framework=Intel® TF 2.11.dev202242, Python 3.8, AI Model=BERT-Large, step size=30, warmup=10; Batched results: Best scores achieved using FP32(BS=64[16 cores/instance]), BFloat16(BS=64[4 cores/instance], int8-AMX(BS=64[1 cores/instance), Real Time(BS1) results while maintaining 130ms latency; SLA: Best scores achieved using FP32 (32 cores/instance), BFloat16 (four cores/instance), int8-AMX (four cores/instance), tested by Intel, April 2023. BERT-Large: 9354: 1-node, 2x AMD EPYC processor 9354 (32c/3.25 GHz, 280W TDP) on Supermicro H13DSH server with 1536 GB (24x64 GB /4800) DDR5 memory, microcode 0xa101111, SMT on, Boost on, NPS=1, Ubuntu 22.04.2 LTS, 5.15.0-58-generic , 1x 1.92T SAMSUNG MZQL21T9HCJR-00A07, Framework=Stock TF 2.10.1, ZenDNN=v4.0, Python 3.8 , AI Model=BERT-Large, step size=30, warmup=10; Batched results: Best scores achieved using FP32(BS=64[1 cores/instance]), tested by Intel, April 2023. Costs are based on Intel estimated and information from thinkmate.com:

  • For a 50-server fleet of AMD EPYC 9354, estimated as of March 2023:
    • CapEx costs: USD 1.01M
    • OpEx costs (four years, includes power and cooling utility costs, infrastructure, and hardware maintenance costs): USD 732.6K
    • Energy use in kWh (four years, per server): 43169, PUE 1.6
    • Other assumptions: Utility cost USD 0.1/kWh, kWh to kg CO2 factor 0.42394
  • For a nine-server fleet of 4th Gen Intel® Xeon® 8462Y as of March 2023:
    • CapEx costs: USD 222K
    • OpEx costs (four years, includes power and cooling utility costs, infrastructure, and hardware maintenance costs): USD 139K
    • Energy use in kWh (four years, per server): 51242, PUE 1.6
    • Other assumptions: Utility cost USD 0.1/kWh, kWh to kg CO2 factor 0.42394
9

4th Gen Intel® Xeon® Scalable processor fine-tuning: Test by Intel on 12/09/2022. One to four nodes, 2S, Intel® Xeon® Platinum 8480+ 56 cores on Dennard Pass platform and software with 512 GB memory (16x32 GB DDR5 4800 MT/s [4800 MT/s]), microcode 0x90000c0, HT on, Turbo on, Rocky Linux 8.7, 4.18.0-372.32.1.el8_6.crt2.x86_64, 931.5G SSD. Multiple nodes connected with 200 Gbps OmniPath. PyTorch 1.13, IPEX 1.13, Transformers 4.24.0, Accelerate 0.14, Diffusers 0.8.0, oneDNN 2.6.0, oneCCL 2021.7.1. 4th Gen Intel® Xeon® Scalable processor inference: Test by Intel on 12/09/2022. One-node, 2S, Intel® Xeon® Platinum 8480+ 56 cores on Archer City platform and software with 1024 GB (16x64 GB DDR5 4800 MT/s [4800 MT/s]), microcode 0x2b000111, HT on, Turbo on, Ubuntu 22.04.1 LTS, 5.15.0-56-generic, 1.5 TB SSD. Multiple nodes connected with 100 Gbps Ethernet Controller I225-LM. PyTorch (commit ID: 26d1dbc) + PR 81852, Transformers 4.25.1, Accelerate 0.14, Diffusers 0.8.0, oneDNN 2.6.0. 3rd Gen Intel® Xeon® Scalable processor inference: Test by Intel on 12/09/2022. One-node, 2S, Intel® Xeon® Platinum 8380 CPU @ 2.30 GHz 40 cores on WHITLEY platform and software with 512 GB (16x32 GB DDR4 3200 MT/s [3200 MT/s]), microcode 0xd000375, HT on, Turbo on, Ubuntu, 5.15.0-56-generic, 7.0 TB SSD. Multiple nodes connected with 10 Gbps Ethernet Controller X710 for 10GBASE-T. PyTorch (commit ID: 26d1dbc), Transformers 4.25.1, Accelerate 0.14, Diffusers 0.8.0, oneDNN 2.6.0. Performance varies by use, configuration, and other factors. Learn more at intel.com/PerformanceIndex. Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates. See backup for configuration details. No product or component can be absolutely secure.