Visible to Intel only — GUID: GUID-02912C63-BBAB-4F88-9F8E-97197CCE4E17
basic_statistics_dense_batch.cpp
column_accessor_homogen.cpp
cor_dense_batch.cpp
cov_dense_batch.cpp
dbscan_brute_force_batch.cpp
df_cls_hist_batch.cpp
df_cls_hist_batch_random.cpp
df_cls_traverse_model.cpp
df_reg_hist_batch.cpp
df_reg_hist_batch_random.cpp
df_reg_traverse_model.cpp
kmeans_init_dense.cpp
kmeans_lloyd_dense_batch.cpp
knn_cls_brute_force_dense_batch.cpp
knn_reg_brute_force_dense_batch.cpp
knn_search_brute_force_dense_batch.cpp
linear_kernel_dense_batch.cpp
linear_regression_dense_batch.cpp
pca_cor_dense_batch.cpp
pca_precomputed_cor_dense_batch.cpp
pca_precomputed_cov_dense_batch.cpp
rbf_kernel_dense_batch.cpp
svm_two_class_thunder_dense_batch.cpp
basic_statistics_dense_batch.cpp
column_accessor_homogen.cpp
connected_components_batch.cpp
cor_dense_batch.cpp
cov_dense_batch.cpp
csr_accessor.cpp
dbscan_brute_force_batch.cpp
df_cls_dense_batch.cpp
df_reg_dense_batch.cpp
directed_graph.cpp
graph_service_functions.cpp
jaccard_batch.cpp
jaccard_batch_app.cpp
kmeans_init_dense.cpp
kmeans_lloyd_dense_batch.cpp
knn_cls_brute_force_dense_batch.cpp
knn_cls_kd_tree_dense_batch.cpp
knn_search_brute_force_dense_batch.cpp
linear_kernel_dense_batch.cpp
linear_regression_dense_batch.cpp
logloss_dense_batch.cpp
louvain_batch.cpp
pca_dense_batch.cpp
pca_precomputed_dense_batch.cpp
polynomial_kernel_dense_batch.cpp
rbf_kernel_dense_batch.cpp
shortest_paths_batch.cpp
sigmoid_kernel_dense_batch.cpp
subgraph_isomorphism_batch.cpp
svm_multi_class_thunder_csr_batch.cpp
svm_multi_class_thunder_dense_batch.cpp
svm_nu_cls_thunder_csr_batch.cpp
svm_nu_cls_thunder_dense_batch.cpp
svm_nu_reg_thunder_csr_batch.cpp
svm_nu_reg_thunder_dense_batch.cpp
svm_reg_thunder_csr_batch.cpp
svm_reg_thunder_dense_batch.cpp
svm_two_class_smo_csr_batch.cpp
svm_two_class_smo_dense_batch.cpp
svm_two_class_thunder_csr_batch.cpp
svm_two_class_thunder_dense_batch.cpp
triangle_counting_batch.cpp
Mathematical Notations
Mathematical Notations
Correlation and Variance-Covariance Matrices
Mathematical Notations
Principal Components Analysis Transform
Singular Value Decomposition
Association Rules
Mathematical Notations
Mathematical Notations
Cholesky Decomposition
QR Decomposition
Outlier Detection
Distance Matrix
Distributions
Engines
Moments of Low Order
Mathematical Notations
Quality Metrics
Sorting
Normalization
Mathematical Notations
Visible to Intel only — GUID: GUID-02912C63-BBAB-4F88-9F8E-97197CCE4E17
Data Compression
When large amounts of data are sent across devices or need to be stored in memory or in a persistent storage, data compression enables you to reduce network traffic, memory, and persistent storage footprint. oneDAL implements several most popular generic compression and decompression methods, which include ZLIB, LZO, RLE, and BZIP2.
General API for Data Compression and Decompression
The CompressionStream and DecompressionStream classes provide general methods for data compression and decompression. The following diagram illustrates the compression and decompression flow at a high level:
Data Compression and Decompression Flow
To define compression or decompression methods and related parameters, provide Compressor or Decompressor objects as arguments to CompressionStream or DecompressionStream constructors respectively. For more details on Compressor and Decompressor, refer to Compression and Decompression Interfaces.
Use operator << of CompressionStream or DecompressionStream to provide input data for compression or decompression stream. By default, all compression and decompression stream methods allocate the memory required to store results of compression and decompression. For details of controlling memory allocation, refer to Compression and Decompression Interfaces.
The following methods are available to retrieve compressed data stored in CompressionStream:
Copy compressed data blocks into a contiguous array using the copyCompressedArray() method.
You can define the data blocks to copy by specifying the number of bytes to copy. The method copies the data from the beginning of the stream and removes the copied data from CompressionStream, so next time you call the copyCompressedArray() method, it copies the next block of data. To copy all the data, before a call to copyCompressedArray(), call the getCompressedBlocksSize() method to get the total size of compressed data in the stream.
Call the getCompressedBlocksCollection() method.
Unlike the copyCompressedArray() method, getCompressedBlocksCollection() does not copy compressed blocks but provides a reference to the collection of compressed data blocks. The collection is available until you call the getCompressedBlocksCollection() method next time.
The following methods are available to retrieve decompressed data stored in DecompressionStream:
Copy decompressed data blocks into a contiguous array using the copyDecompressedArray() method.
You can define the data blocks to copy by specifying the number of bytes to copy. The method copies the data from the beginning of the stream and removes the copied data from DecompressionStream, so next time you call the copyDecompressedArray() method, it copies the next block of data. To copy all the data, before a call to copyDecompressedArray(), call the getDecompressedBlocksSize() method to get the total size of decompressed data in the stream.
Call the getDecompressedBlocksCollection() method.
Unlike the copyDecompressedArray() method, getDecompressedBlocksCollection() does not copy decompressed blocks but provides a reference to the collection of decompressed data blocks. The collection is available until you call the getDecompressedBlocksCollection() method next time.
Compression and Decompression Interfaces
CompressionStream and DecompressionStream classes cover most typical usage scenarios. Therefore, you need to work directly with Compressor and Decompressor objects only in the cases as follows:
CompressionStream and DecompressionStream classes do not cover your specific usage model.
You want to control memory allocation and deallocation for results of compression and decompression.
You need to modify compression and decompression default parameters.
The Compressor and Decompressor classes provide interfaces to supported compression and decompression methods (ZLIB, LZO, RLE, and BZIP2).
Compression and decompression objects are initialized with a set of default parameters. You can modify parameters of a specific compression method by accessing the parameter field of the Compressor or Decompressor object.
To perform compression or decompression using the Compressor or Decompressor classes, respectively, provide input data using the setInputDataBlock() method and call the run() method. This approach requires that you allocate and control the memory to store the results of compression or decompression. In general, it is impossible to accurately estimate the required size of the output data block, and the memory you provide may be insufficient to store results of compression or decompression. However, you can check whether you need to allocate additional memory to continue the run() operation. To do this, use the isOutputDataBlockFull() method. You can also use the getUsedOutputDataBlockSize() method to obtain the size of compressed or decompressed data actually written to the output data block.
You can use your own compression and decompression methods in CompressionStream and DecompressionStream. In this case, you need to override Compressor and Decompressor objects.
Examples
C++:
https://github.com/oneapi-src/oneDAL/tree/master/examples/daal/cpp/source/compression/compressor.cpp
- Java*:
-
NOTE:The Java interfaces are deprecated in the oneDAL library and may no longer be supported in future releases.