Data compression is an important part of modern computing ecosphere including cloud services, data bases, network applications. It allows not only to significantly decrease the storage space requirements, but also to improve overall network (and other inter-CPU/nodes communications) throughput.
There are many data compression methods basically divided into lossless (Zlib, Bzip2, LZxx and others) and lossy (with controlled data losses, like floating-point compression, image/video compression etc.). From the lossless compression methods, the Zlib compression stands apart as one of the oldest and most adopted compression.
Intel software products have several implementations of Zlib, which sometimes is referred to as “deflate”, compression. In this article we will discuss Intel's two different solution and their benefits – Intel® Intelligent Storage Acceleration (Intel® ISA-L) library and Intel® Integrated Performance Primitives (Intel® IPP).
Intel® Intelligent Storage Acceleration Library
Intel® ISA-L provides a wide range of functionalities targeting storage applications:
- Erasure codes - Fast block Reed-Solomon type erasure codes for any encode/decode matrix in GF(2^8).
- CRC - Fast implementations of cyclic redundancy check. Six different polynomials supported.
- iscsi32, ieee32, t10dif, ecma64, iso64, jones64.
- Raid - calculate and operate on XOR and P+Q parity found in common RAID implementations.
- Compression - Fast deflate-compatible data compression.
- Decompression - Fast inflate-compatible data compression.
In this article, we’ll focus on last two functionalities: “deflate” compression and decompression.
Intel® ISA-L, as well as canonical Zlib, supports RFC 1951 deflate standard in terms of compressed data format.
Intel® ISA-L provides its own interface to compression, decompression and service functions. Basically, there are two directions of operation support: stream-based and stateless. The stream-based interface, like Zlib, allows to:
- Create stream objects for compression and decompression, they are of different types: isal_zstream for compression and inflate_state for decompression.
- Initialize the object, perform service functions on it (reset stream, set dictionary buffer for compression/decompression, set custom Huffman table for better compression, read/write Gzip/Zlib headers).
- Perform compression/decompression operations controlling buffer flushing.
The stateless compression and decompression are “one shot” functions without keeping the intermediate data.
The Intel® ISA-L provides 4 levels of compression, which affect operation performance and compression ratio.
Intel® Integrated Performance Primitives Library
One more Intel® software product supporting “deflate” format is Intel® IPP which is a component of the Intel® oneAPI Base Toolkit, in its data compression domain (or, library). Unlike Intel® ISA-L, Intel® IPP functions for Zlib don’t provide full functionality to do deflate-like compression/decompression, but rather optimized core functions which help canonical Zlib to operate faster. It is done by using of source code patch files, which being applied to various open-source versions of Zlib, convert them into Intel® IPP-optimized full functional Zlib libraries.
In terms of compatibility, Intel® IPP, being integrated to open source Zlib, is 100% compatible with traditional Zlib library even on ABI (application binary interface) level. This is because high-level function interfaces are not affected by Intel® IPP integration and only a few of extra high-level functions were added to the end of the list of exported functions. This compatibility allows to use Intel® IPP Zlib on binary level, by simple substituting of traditional library with new optimized by Intel® IPP. You need to build Intel® IPP-optimized library using standard build procedure for Zlib (Refer to the readme file provided with Intel® IPP product in “components/interfaces/ipp_zlib” directory).
Regarding compression levels and service functionality, everything is the same as in original Zlib: levels 0 through 9 (plus Intel® IPP provides one additional level “-2”), stream control functions, flushing control, etc.
Choosing Between Intel® ISA-L and Intel® IPP Zlib
The selection of specific Intel® provided Zlib library can be done on a base of requirements of particular application to data compression. The table below shows features and differences between two libraries.
| Feature | Intel® ISA-L | Intel® IPP Zlib |
|---|---|---|
| Compression/Decompression Speed | Very High | High |
| Compression Ratio(Quality of compression) | Moderate | From Moderate to High |
| Ease of Use | Easy with direct calls to Intel® ISA-L | Easy if standard Zlib interface is already used in application |
| Standard Zlib API source/binary compatibility | No | Yes |
| Open Source | Yes | No |
The performance/compression difference between Intel® ISA-L and Intel® IPP Zlib is presented on the charts below:
Configuration: Testing by Intel as of 12/1/2020. Intel® Xeon® Gold 5218 CPU freq 2.3 GHz, 256GB, Linux OS: Red Hat Enterprise Linux* 8.1. Intel® ISA-L 2.30.0 /Intel® IPP 2020 Update 3/ Zlib 1.2.11
The L<number> points are compression levels. The most of Intel® ISA-L points are at the upper-left side of picture, while Intel® IPP Zlib points are in the bottom-right. The performance measured is on single-thread. Three relevant standards followed RFC-1951 (deflate), RFC-1952 (gzip), and RFC-1950 (zlib).
Thus, the compression library usage strategy is:
- If you need more performance and adopt less compression ratio, you should choose Intel® ISA-L.
- If you need better compression ratio at the cost of lower performance, you should choose Intel® IPP Zlib.
Notices & Disclaimers
Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.
Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates. See backup for configuration details. No product or component can be absolutely secure.
Your costs and results may vary.
Intel technologies may require enabled hardware, software or service activation.
© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.