Intel® oneAPI Collective Communications Library Release Notes

ID 763686
Updated 6/23/2025
Version 2021.16
Public

By Hung-Ju Tsai, Orel Yehuda

Overview

The Intel® oneAPI Collective Communications Library (oneCCL) enables developers and researchers to more quickly train newer and deeper models. This is done by using optimized communication patterns to distribute model training across multiple nodes.

The library is designed for easy integration into deep learning (DL) frameworks, whether you are implementing them from scratch or customizing existing ones.

  • Built on top of lower-level communication middleware - MPI and OFI (libfabrics) which transparently support many interconnects, such as Cornelis Networks, InfiniBand, and Ethernet.
  • Optimized for high performance on Intel® CPUs and GPUs.
  • Allows the tradeoff of compute for communication performance to drive scalability of communication patterns.
  • Enables efficient implementations of collectives that are heavily used for neural network training, including allreduce, and allgather.

2021.16 Release

Major Features Supported

  •  
    Functionality Subitems CPU GPU
    Collective operations Allgather X X
      Allgatherv X X
      Allreduce X X
      Alltoall X X
      Alltoallv X X
      Barrier X X
      Broadcast X X
      Reduce X X
      ReduceScatter X X
    Point to Point Send X X
      Recv X X

    Group API

    All collectives

    X

    X

     

    Point to Point

    X

    X
    Data types [u]int[8, 16, 32, 64] X X
      fp[16, 32, 64], bf16 X X
    Scaling Scale-up X X
      Scale-out X X
    Programming model   1 rank / process per device 1 rank / process per device

     

  • Service functionality

    • Interoperability with SYCL*:
    • Construction of oneCCL communicator object based on SYCL context and SYCL device
    • Construction of oneCCL stream object based on SYCL queue
    • Construction of oneCCL event object based on SYCL event
    • Retrieving of SYCL event from oneCCL event associated with oneCCL collective operation
    • Passing SYCL buffer as source/destination parameter of oneCCL collective operation

What's New 2021.16

  • Added SYCL graph support for Record and Replay for Allgather, Allreduce, Alltoall, ReduceScatter and Broadcast
  • Added SYCL-based implementation of ring algorithm for Allgather
  • Added SYCL-based implementation for Broadcast
  • Added multithread support for Allgather and ReduceScatter scale up impementation
  • Added attribute in the communicator to specify blocking operations for CPU
  • Bug fixes

Removals

  • split_communicators is deprecated in 2021.15.0 and will be removed later.

Notice of Deprecation

Deprecation of C++ API

  • In oneCCL version 2021.17 included with the 2025.3 oneAPI release, oneCCL will add support for a new C API that closely follows the NVIDIA Collective Communications Libary (NCCL)* API standard. The existing C++ API will remain available and will remain the default API for the 2021.17 release. Details explaining how an application may link against and use the new API will be shared in this release.

See the oneCCL C API RFC document to view the proposed API and provide any feedback. 

  • In oneCCL version 2022.0 included with the 2026.0 oneAPI release, oneCCL will use the new NCCL* like C API by default. This is a breaking change. The legacy C++ API will remain available, and details explaining how an application may link against and use the legacy API will be included in the 2022.0 release.

Applications cannot use both the C and C++ APIs simultaneously.

Support for the legacy C++ API shall remain in the release until future notice. The schedule for legacy API removal will be announced here.

System Requirements

please see system requirements.  

 

Previous Releases

2021.15.0, 2021.15.1 and 2021.15.2

What's New 2021.15.2

  • Bug fix - Improvement of User Experience based on setting of Environment Variables.

What's New 2021.15.1

  • Bug fixes

Major Features Supported

  •  
    Functionality Subitems CPU GPU
    Collective operations Allgather X X
      Allgatherv X X
      Allreduce X X
      Alltoall X X
      Alltoallv X X
      Barrier X X
      Broadcast X X
      Reduce X X
      ReduceScatter X X
    Point to Point Send X X
      Recv X X

    Group API

    All collectives

    X

    X

     

    Point to Point

    X

    X
    Data types [u]int[8, 16, 32, 64] X X
      fp[16, 32, 64], bf16 X X
    Scaling Scale-up X X
      Scale-out X X
    Programming model   1 rank / process per device 1 rank / process per device

     

  • Service functionality

    • Interoperability with SYCL*:
    • Construction of oneCCL communicator object based on SYCL context and SYCL device
    • Construction of oneCCL stream object based on SYCL queue
    • Construction of oneCCL event object based on SYCL event
    • Retrieving of SYCL event from oneCCL event associated with oneCCL collective operation
    • Passing SYCL buffer as source/destination parameter of oneCCL collective operation

What's New 2021.15

  • Support for Average for Allreduce and Reduce-Scatter
  • Extend Group API to also support collective operations.
  • New split_communicator API with updated parameters.
  • Performance optimizations for scaleup for Alltoall

Removals

  • split_communicators is deprecated in 2021.15.0 and will be removed later.

 

2021.14

 

2021.14 Release

Major Features Supported

  •  
    Functionality Subitems CPU GPU
    Collective operations Allgather X X
      Allgatherv X X
      Allreduce X X
      Alltoall X X
      Alltoallv X X
      Barrier X X
      Broadcast X X
      Reduce X X
      ReduceScatter X X
    Point to Point Send X X
      Recv X X
    Data types [u]int[8, 16, 32, 64] X X
      fp[16, 32, 64], bf16 X X
    Scaling Scale-up X X
      Scale-out X X
    Programming model Rank = device 1 rank per process 1 rank per process

     

  • Service functionality

    • Interoperability with SYCL*:
    • Construction of oneCCL communicator object based on SYCL context and SYCL device
    • Construction of oneCCL stream object based on SYCL queue
    • Construction of oneCCL event object based on SYCL event
    • Retrieving of SYCL event from oneCCL event associated with oneCCL collective operation
    • Passing SYCL buffer as source/destination parameter of oneCCL collective operation

What's New 2021.14

  • Optimizations on key-value store support to scale up to 3000 nodes
  • New APIs for Allgather, Broadcast and group API calls
  • Performance Optimizations for scaleup for Allgather, Allreduce, and Reduce-scatter for scaleup and scaleout
  • Performance Optimizations for CPU single node
  • Optimizations to reuse Level Zero events.
  • Change of the default mechanism for IPC exchange to pidfd

2021.13.0 and 2021.13.1

2021.13.1

Bug fixes.

2021.13.0

Major Features Supported

Table1

Functionality Subitems CPU GPU
Collective operations Allgatherv X X
  Allreduce X X
    Alltoall X X
    Alltoallv X X
    Barrier X X
    Broadcast X X
    Reduce X X
    ReduceScatter X X
Point to Point Send X X
    Recv X X
Data types [u]int[8, 16, 32, 64] X X
   fp[16, 32, 64], bf16 X X
Scaling Scale-up X X
  Scale-out X X
Programming model Rank = device 1 rank per process 1 rank per process

Service functionality

  • Interoperability with SYCL*:
  • Construction of oneCCL communicator object based on SYCL context and SYCL device
  • Construction of oneCCL stream object based on SYCL queue
  • Construction of oneCCL event object based on SYCL event
  • Retrieving of SYCL event from oneCCL event associated with oneCCL collective operation
  • Passing SYCL buffer as source/destination parameter of oneCCL collective operation

What's New

  • Optimizations to limit the memory consumed by oneCCL
  • Optimizations to limit the number of file descriptors maintained opened by oneCCL.
  • Align the support for in-place for the Allgatherv and Reduce-scatter collectives to follow the same behavior as NCCL*.
    In particular, the Allgatherv collective is in place when:
    send_buff == recv_buff + rank_offset, where rank_offset = sum (recv_counts[i]), for all I<rank.
    Reduce-scatter is in-place when recv_buff == send_buff + rank *recv_count. 
  • When using the environment variable CCL_WORKER_AFFINITY, oneCCL enforces the requirement that the length of the list should be equal to the number of workers. 
  • Bug fixes.

Removals

  • oneCCL 2021.12 removes support for PMIx 4.2.2

 

2021.12

Major Features Supported

Table1

Functionality Subitems CPU GPU
Collective operations Allgatherv X X
  Allreduce X X
    Alltoall X X
    Alltoallv X X
    Barrier X X
    Broadcast X X
    Reduce X X
    ReduceScatter X X
Point to Point Send X X
    Recv X X
Data types [u]int[8, 16, 32, 64] X X
    fp[16, 32, 64], bf16 X X
Scaling Scale-up X X
  Scale-out X X
Programming model Rank = device 1 rank per process 1 rank per process

Service functionality

  • Interoperability with SYCL*:
  • Construction of oneCCL communicator object based on SYCL context and SYCL device
  • Construction of oneCCL stream object based on SYCL queue
  • Construction of oneCCL event object based on SYCL event
  • Retrieving of SYCL event from oneCCL event associated with oneCCL collective operation
  • Passing SYCL buffer as source/destination parameter of oneCCL collective operation

What's New

  • Performance improvements for scaleup for all message sizes for AllReduce, Allgather, and Reduce_Scatter.
    Optimizations also include small message sizes that appear in inference apps. 
  • Performance improvements for scaleout for Allreduce, Reduce, Allgather, and Reduce_Scatter.
  • Optimized memory usage of oneCCL.
  • Support for PMIx 4.2.6.
  • For the 2021.12 release, the Third Party Programs file has been included as a section in this product’s release notes rather than as a separate text file.
  • Bug fixes.

Removals

  • oneCCL 2021.12 removes support for PMIx 4.2.2

 

2021.11.0, 2021.11.1 and 2021.11.2

2021.11.2 Release

This update provides bug fixes to maintain driver compatibility for Intel® Data Center GPU Max Series.

2021.11.1 Release

This update addresses stability issues with distributed Training and Inference workloads on Intel® Data Center GPU Max Series.

2021.11.0 Release

Major Features Supported

Table1

Functionality Subitems CPU GPU
Collective operations Allgatherv X X
  Allreduce X X
    Alltoall X X
    Alltoallv X X
    Barrier X X
    Broadcast X X
    Reduce X X
    ReduceScatter X X
Point to Point Send X X
    Recv X X
Data types [u]int[8, 16, 32, 64] X X
    fp[16, 32, 64], bf16 X X
Scaling Scale-up X X
  Scale-out X X
Programming model Rank = device 1 rank per process 1 rank per process

Service functionality

  • Interoperability with SYCL*:
  • Construction of oneCCL communicator object based on SYCL context and SYCL device
  • Construction of oneCCL stream object based on SYCL queue
  • Construction of oneCCL event object based on SYCL event
  • Retrieving of SYCL event from oneCCL event associated with oneCCL collective operation
  • Passing SYCL buffer as source/destination parameter of oneCCL collective operation

What's New

  • Added point to point blocking communication operations for send and receive.
  • Performance optimizations for Reduce-Scatter.
  • Improved profiling with Intel® Instrumentation and Tracing Technology (ITT) profiling level.

Removals

  • Support of Intel® C++ Compiler Classic (icc) is removed

2021.10.0

 

Major Features Supported

Table1

Functionality Subitems CPU GPU
Collective operations Allgatherv X X
  Allreduce X X
    Alltoall X X
    Alltoallv X X
    Barrier X X
    Broadcast X X
    Reduce X X
    ReduceScatter X X
Data types [u]int[8, 16, 32, 64] X X
    fp[16, 32, 64], bf16 X X
Scaling Scale-up X X
  Scale-out X X
Programming model Rank = device 1 rank per process 1 rank per process

Service functionality

  • Interoperability with SYCL*:
  • Construction of oneCCL communicator object based on SYCL context and SYCL device
  • Construction of oneCCL stream object based on SYCL queue
  • Construction of oneCCL event object based on SYCL event
  • Retrieving of SYCL event from oneCCL event associated with oneCCL collective operation
  • Passing SYCL buffer as source/destination parameter of oneCCL collective operation

What's New

  • Improved scaling efficiency of the Scaleup algorithms for ReduceScatter
  • Optimize performance of oneCCL scaleup collectives by utilizing the new embedded Data Streaming Accelerator in Intel® 4th Generation Xeon Scalable Processors (formerly code-named Sapphire Rapids)

Removals

  • Sockets provider will be removed starting with 2021.10 release
  • Support of Intel® C++ Compiler Classic (icc) will be removed starting with 2021.11 release

2021.9

 

Major Features Supported

Table1

Functionality Subitems CPU GPU
Collective operations Allgatherv X X
  Allreduce X X
    Alltoall X X
    Alltoallv X X
    Barrier X X
    Broadcast X X
    Reduce X X
    ReduceScatter X X
Data types [u]int[8, 16, 32, 64] X X
    fp[16, 32, 64], bf16 X X
Scaling Scale-up X X
  Scale-out X X
Programming model Rank = device 1 rank per process 1 rank per process

Service functionality

  • Interoperability with SYCL*:
  • Construction of oneCCL communicator object based on SYCL context and SYCL device
  • Construction of oneCCL stream object based on SYCL queue
  • Construction of oneCCL event object based on SYCL event
  • Retrieving of SYCL event from oneCCL event associated with oneCCL collective operation
  • Passing SYCL buffer as source/destination parameter of oneCCL collective operation

What's New

  • Improved scaling efficiency of the Scaleup algorithms for Alltoall and Allgather
  • Add collective selection for scaleout algorithm for device (GPU) buffers

2021.8

 

Major Features Supported

Table1

Functionality Subitems CPU GPU
Collective operations Allgatherv X X
  Allreduce X X
    Alltoall X X
    Alltoallv X X
    Barrier X X
    Broadcast X X
    Reduce X X
    ReduceScatter X X
Data types [u]int[8, 16, 32, 64] X X
    fp[16, 32, 64], bf16 X X
Scaling Scale-up X X
  Scale-out X X
Programming model Rank = device 1 rank per process 1 rank per process

Service functionality

  • Interoperability with SYCL*:
  • Construction of oneCCL communicator object based on SYCL context and SYCL device
  • Construction of oneCCL stream object based on SYCL queue
  • Construction of oneCCL event object based on SYCL event
  • Retrieving of SYCL event from oneCCL event associated with oneCCL collective operation
  • Passing SYCL buffer as source/destination parameter of oneCCL collective operation

What's New

  • Provides optimized performance for Intel® Data Center GPU Max Series utilizing oneCCL.
  • Enables support for Allreduce, Allgather, Reduce, and Alltoall connectivity for GPUs on the same node

Known issues and limitations

  • The 'using namespace oneapi;' directive is not recommended, as it may result in compilation errors  when oneCCL is used with other oneAPI libraries. You can instead create a namespace alias for oneCCL, e.g. 

namespace oneccl = ::oneapi::ccl; oneccl::allreduce(…);

  • Limitations imposed by Intel® oneAPI DPC++ Compiler:
  • SYCL buffers cannot be used from different queues

2021.7 and 2021.7.1

2021.7.1 Release

Intel® oneAPI Collective Communications Library 2021.7.1 has been updated to include functional and security updates. Users should update to the latest version as it becomes available.

2021.7 Release

Major Features Supported

Table1

Functionality Subitems CPU GPU
Collective operations Allgatherv X X
  Allreduce X X
    Alltoall X X
    Alltoallv X X
    Barrier X X
    Broadcast X X
    Reduce X X
    ReduceScatter X X
Data types [u]int[8, 16, 32, 64] X X
    fp[16, 32, 64], bf16 X X
Scaling Scale-up X X
  Scale-out X X
Programming model Rank = device 1 rank per process 1 rank per process

Service functionality

  • Interoperability with SYCL*:
  • Construction of oneCCL communicator object based on SYCL context and SYCL device
  • Construction of oneCCL stream object based on SYCL queue
  • Construction of oneCCL event object based on SYCL event
  • Retrieving of SYCL event from oneCCL event associated with oneCCL collective operation
  • Passing SYCL buffer as source/destination parameter of oneCCL collective operation

What's New

  • no change from previous release.

Known issues and limitations

  • The 'using namespace oneapi;' directive is not recommended, as it may result in compilation errors  when oneCCL is used with other oneAPI libraries. You can instead create a namespace alias for oneCCL, e.g. 

namespace oneccl = ::oneapi::ccl; oneccl::allreduce(…);

  • Limitations imposed by Intel® oneAPI DPC++ Compiler:
  • SYCL buffers cannot be used from different queues

 

2021.6

 

Major Features Supported

Table1

Functionality Subitems CPU GPU
Collective operations Allgatherv X X
  Allreduce X X
    Alltoall X X
    Alltoallv X X
    Barrier X X
    Broadcast X X
    Reduce X X
    ReduceScatter X X
Data types [u]int[8, 16, 32, 64] X X
    fp[16, 32, 64], bf16 X X
Scaling Scale-up X X
  Scale-out X X
Programming model Rank = device 1 rank per process 1 rank per process

Service functionality

  • Interoperability with SYCL*:
  • Construction of oneCCL communicator object based on SYCL context and SYCL device
  • Construction of oneCCL stream object based on SYCL queue
  • Construction of oneCCL event object based on SYCL event
  • Retrieving of SYCL event from oneCCL event associated with oneCCL collective operation
  • Passing SYCL buffer as source/destination parameter of oneCCL collective operation

What's New

  • Intel® oneAPI Collective Communications Library now supports Intel® Instrumentation and Tracing Technology (ITT) profiling
  • Intel® oneAPI Collective Communications Library can be seamlessly integrated with Windows platforms with WSL2 (Windows Subsystem for Linux 2) support
  • Enhanced application stability with runtime dependency check for Level Zero, in Intel® oneAPI Collective Communications Library

Known issues and limitations

  • The 'using namespace oneapi;' directive is not recommended, as it may result in compilation errors  when oneCCL is used with other oneAPI libraries. You can instead create a namespace alias for oneCCL, e.g. 

namespace oneccl = ::oneapi::ccl; oneccl::allreduce(…);

  • Limitations imposed by Intel® oneAPI DPC++ Compiler:
  • SYCL buffers cannot be used from different queues

2021.5

 

What's New

  • Added support for output SYCL event to track status of CCL operation
  • Added OFI/verbs provider with dmabuf support into package
  • Bug fixes

Known issues and limitations

  • The 'using namespace oneapi;' directive is not recommended, as it may result in compilation errors  when oneCCL is used with other oneAPI libraries. You can instead create a namespace alias for oneCCL, e.g. 

namespace oneccl = ::oneapi::ccl; oneccl::allreduce(…);

  • Limitations imposed by Intel® oneAPI DPC++ Compiler:
  • SYCL buffers cannot be used from different queues
    oneccl::allreduce(…);

2021.4

 

What's New

  • Memory binding of worker threads is now supported
  • NIC filtering by name is now supported for OFI-based multi-NIC
  • IPv6 is now supported for key-value store (KVS)

Known issues and limitations

  • The 'using namespace oneapi;' directive is not recommended, as it may result in compilation errors  when oneCCL is used with other oneAPI libraries. You can instead create a namespace alias for oneCCL, e.g. 

namespace oneccl = ::oneapi::ccl; oneccl::allreduce(…);

  • Limitations imposed by Intel® oneAPI DPC++ Compiler:
  • SYCL buffers cannot be used from different queues

 

2021.3

 

What's New

  • Added OFI-based multi-NIC support
  • Added OFI/psm3 provider support
  • Bug fixes

Known issues and limitations

  • The 'using namespace oneapi;' directive is not recommended, as it may result in compilation errors  when oneCCL is used with other oneAPI libraries. You can instead create a namespace alias for oneCCL, e.g. 

namespace oneccl = ::oneapi::ccl; oneccl::allreduce(…);

  • Limitations imposed by Intel® oneAPI DPC++ Compiler:
  • SYCL buffers cannot be used from different queues

 

2021.2

 

What's New

  • Added float16 datatype support.
  • Added ip-port hint for customization of KVS creation.
  • Optimized communicator creation phase.
  • Optimized multi-GPU collectives for single-node case.
  • Bug fixes

Known issues and limitations

  • The 'using namespace oneapi;' directive is not recommended, as it may result in compilation errors  when oneCCL is used with other oneAPI libraries. You can instead create a namespace alias for oneCCL, e.g. 

namespace oneccl = ::oneapi::ccl; oneccl::allreduce(…);

  • Limitations imposed by Intel® oneAPI DPC++ Compiler:
  • SYCL buffers cannot be used from different queues

 

2021.1

What's New

  • Added [u]int16 support
  • Added initial support for external launch mechanism
  • Fixed bugs

Known issues and limitations

  • The 'using namespace oneapi;' directive is not recommended, as it may result in compilation errors  when oneCCL is used with other oneAPI libraries. You can instead create a namespace alias for oneCCL, e.g. 

namespace oneccl = ::oneapi::ccl; oneccl::allreduce(…);

  • Limitations imposed by Intel® oneAPI DPC++ Compiler:
  • SYCL buffers cannot be used from different queues

 

 

Notices and Disclaimers

Intel technologies may require enabled hardware, software or service activation.

No product or component can be absolutely secure.

Your costs and results may vary.

© Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries. Other names and brands may be claimed as the property of others.

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.

The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

​

Third Party Programs File

1

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.