The open source project, llama.cpp, is a light LLM framework that is gaining popularity. Its high performance and customizability have turned the project into a thriving and dynamic community of developers, researchers, and hobbyists. Approximately one year since launch, the GitHub* project has more than 600 contributors, 52,000 stars, 1,500 releases, and 7,400 forks. Thanks to recent code merges, llama.cpp now supports more hardware, including Intel® GPUs across server and consumer products. Intel’s GPUs join hardware support for CPUs (x86 and ARM) and GPUs from other vendors.
The original implementation was created by Georgi Gerganov. The project is mainly for educational purposes and serves as the main playground for developing new features for the ggml library, a tensor library for machine learning. With the recent updates, Intel is bringing AI everywhere to more users by enabling inference on far more devices. Llama.cpp is fast because it’s written in C and has several other attractive features:
- 16-bit float support
- Integer quantization support (4-bit, 5-bit, 8-bit, etc.)
- No third-party dependencies
- Zero memory allocations during runtime
SYCL Backend for Intel GPUs
There are several backends of ggml to support and optimize for different hardware. We chose SYCL* (a direct programming language) and oneMKL (a high-performance BLAS library) from oneAPI to develop the SYCL backend because it supports GPUs from different vendors. SYCL is a programming model to improve productivity on hardware accelerators. It is a single-source, embedded, domain-specific language based on pure C++17.
The SYCL backend supports all Intel GPUs. We have verified with:
- Intel® Data Center GPU Max and Flex Series GPU
- Intel® Arc™ Discrete GPU
- Built-in Intel Arc GPU in Intel® Core™ Ultra CPU
- iGPU in 11th, 12th, and 13th Gen Intel Core CPUs
With llama.cpp now supporting Intel GPUs, millions of consumer devices are capable of running inference on Llama. Compared to the OpenCL (CLBlast) backend, the SYCL backend has significant performance improvement on Intel GPUs. It also supports more devices, like CPU and other processors with AI accelerators in the future. Please refer to guide to learn how to use the SYCL backend: llama.cpp for SYCL.
Run LLM on Intel GPU Using the SYCL Backend
A detailed guide is available in llama.cpp for SYCL. It can run on all Intel GPUs supported by SYCL and oneAPI. Server and cloud users can run on Intel Data Center GPU Max and Flex Series GPUs. Client users can try it out on their Intel Arc GPU or iGPU on Intel Core CPUs. We have tested the iGPUs of 11th Gen Core and newer. The older iGPU works but with poor performance.
The only limitation is memory. The iGPU uses the host shared memory. The dGPU uses its own memory. We recommend using the iGPU with 80+ EUs (11th Gen Core and newer) and shared memory is more than 4.5 GB for llama2–7b-Q4 model (total host memory is 16 GB and more, and half memory could be allocated to iGPU).
Install the Intel GPU Driver
Both Linux* and Windows* (WLS2) are supported. For Linux, we recommend Ubuntu* 22.04, which was used for development and testing.
Linux:
sudo usermod -aG render username
sudo usermod -aG video username
sudo apt install clinfo
sudo clinfo -l
Output (example):
Platform #0: Intel(R) OpenCL Graphics -- Device #0: Intel(R) Arc(TM) A770 Graphics
or
Platform #0: Intel(R) OpenCL HD Graphics -- Device #0: Intel(R) Iris(R) Xe Graphics \[0x9a49\]
Windows: Install Intel GPU Drivers.
Enable the oneAPI Runtime
First, install the Intel® oneAPI Base Toolkit to get the SYCL compiler and oneMKL. Next, enable the oneAPI runtime:
- Linux: source /opt/intel/oneapi/setvars.sh
- Windows: "C:\Program Files (x86)\Intel\oneAPI\setvars.bat\" intel64
Run sycl-ls to confirm that there are one or more Level Zero devices. Please confirm that at least one GPU is present, like [ext_oneapi_level_zero:gpu:0].
Build by one-click:
- Linux: ./examples/sycl/build.sh
- Windows: examples\sycl\win-build-sycl.bat
Note, the scripts above include the command to enable the oneAPI runtime.
Run an Example by One-Click
Download llama-2–7b.Q4_0.gguf and save to the models folder:
- Linux: ./examples/sycl/run-llama2.sh
- Windows: examples\sycl\win-run-llama2.bat
Note that the scripts above include the command to enable the oneAPI runtime. If the ID of your Level Zero GPU is not 0, please change the device ID in the script. To list the device ID:
- Linux: ./build/bin/ls-sycl-device or ./build/bin/main
- Windows: build\bin\ls-sycl-device.exe or build\bin\main.exe
Summary
The SYCL backend in llama.cpp brings all Intel GPUs to LLM developers and users. Please check if your Intel laptop has an iGPU, your gaming PC has an Intel Arc GPU, or your cloud VM has Intel Data Center GPU Max and Flex Series GPUs. If yes, please enjoy the magical features of LLM by llama.cpp on Intel GPUs. We welcome developers to try and contribute to SYCL backend to add more features and optimization on Intel GPU. It’s a good project to learn oneAPI programming model for cross-platform development.