Intel® Query Processing Library (Intel® QPL)
| Enable the Intel® In-Memory Analytics Accelerator (Intel® IAA) for Data Analytics, Database as a Service (DBaaS), and Session Management | |
|
Overview
Intel IAA and How Intel QPL is Connected
Intel® IAA is one of the newest workload accelerators featured on 4th generation Intel® Xeon® processors. This accelerator provides compression and decompression of very high throughput combined with analytic primitive functions commonly used for data filtering during analytic query processing. Intel® QPL helps developers activate this accelerator's advanced data analytics capabilities. It also helps increase query throughput for in-memory database and analytics workloads, and decreases the memory footprint for analytics workloads, including commercial in-memory databases, open source in-memory databases, and columnar formats for big data analytics.
Intel IAA Resources |
Intel QPL Resources |
Explore Enabled Applications
View applications enabled with Intel IAA or Intel QPL:
Download and Help Intel QPL Evolve
Intel QPL is part of the open standards initiative. We welcome you to continue to contribute to its open source community. |
Features
Intel IAA Features
Intel QPL uses Intel IAA hardware that provides compression and decompression of very high throughput combined with analytic primitive functions. The primitive functions are commonly used for data filtering during analytic query processing.
Intel IAA targets:
- Big data applications and in-memory analytic databases
- Application-transparent uses such as memory page compression
- Data integrity operations, such as CRC-64
Intel IAA supports Huffman encoding and Deflate. For the Deflate format, Intel IAA supports indexing the compressed stream for efficient random access.
Intel QPL Functions
Operations
Intel QPL supports:
- Deflate compression and decompression with the history size limited to 4 KB
- Huffman-only compression and decompression
- Filter operations
Execution Paths
Intel QPL supports several execution paths that help to achieve the optimal system resources use:
- Hardware Path: All hardware-supported functions are executed by Intel IAA.
- Software Path: All supported functionalities are executed by the software library in the CPU.
- Auto Path: Intel QPL automatically dispatches execution of the requested operations either to Intel IAA or to the software library depending on internal heuristics (a load-balancing feature).
NUMA Support
Intel QPL is non-uniform memory access (NUMA)-aware and respects the NUMA node ID of the calling thread. If a user needs to use a device from a specific node, it can be done in multiple ways
Workload Performance
Experience significant performance gains through the combined power of Intel IAA and Intel QPL.
Up to 3x better performance and
66% latency reduction
for RocksDB on 4th generation Intel Xeon Platinum processors versus the prior generation.
Up to 1.59x better performance and a
25% reduction in memory bandwidth
per query for a ClickHouse* database on 4th generation Intel Xeon Platinum processors compared to the prior generation.
2.01x average performance per watt efficiency
for RocksDB when using Intel IAA and Intel QPL versus Zstd.
52% lower total cost of ownership (TCO)
for RocksDB on 4th generation Intel Xeon processors with Intel IAA compared to 3rd generation Intel Xeon processors at the same performance requirement
† Look for the corresponding row in the linked table.
Documentation & Code Samples
Code Samples
This section contains several simple low-level C API examples, illustrating how you can use Intel QPL with C or C++ code.
- Example for Deflate Compression with Canned Mode
- Example for Scan Operation
- Example for Deflate Compression with Dynamic Block
Specifications
Intel QPL only supports the Intel® 64 platform.
Software Path Requirements
Minimum requirements
x86-64 CPU with Intel® Advanced Vector Extensions 2 support (Intel® microarchitecture code name Broadwell).
Recommended requirements for better performance
x86-64 CPU with Intel® Advanced Vector Extensions 512 support (Intel® microarchitecture code name Skylake processor or higher).
Hardware Path Requirements
Execution using a hardware path is only available on Linux*.
Additionally, the operating system must meet the following requirements:
- Linux kernel version 5.18 or later. Public versions of Linux kernels can be found at The Linux Kernel Archives.
- Intel® Virtualization Technology (Intel® VT) for Directed I/O (Intel® VT-d) is enabled through the BIOS menu.
Processors: Intel Xeon Scalable processors
Accelerators: Intel IAA
Languages: C and C++
Operation Systems: Linux, Windows* (Windows is only for Software Path)
Compilers: GCC*, Clang
Get Help
Your success is our success. Access these resources when you need assistance.