Oracle has released a network plugin dedicated to distributed AI training.
Dubbed the Oracle Collective Communications Library (OCCL), the tool integrates with similar libraries from Nvidia and AMD like NCCL and RCCL, respectively, through their standard network-plugin interfaces. Supporting RoCEv2 and InfiniBand, it monitors queue-pair transfers and dynamically distributes traffic across network paths with less congestion.
OCCL also automatically reroutes traffic around supported link or network interface card (NIC) failures and restores connectivity when hardware recovers. The software supports single-planar and multi-planar remote direct memory access (RDMA) networks in a vendor-agnostic fashion. Oracle said this provides a software-defined way to better support collective-communication resilience and traffic distribution while maintaining flexibility in the underlying network hardware.
According to internal benchmarks using Nvidia GB300 systems with 16-128 nodes and four GPUs per node, OCCL increased average NCCL bus bandwidth by 2-10% for operations using AllReduce, a distributed computing collective used to aggregate and synchronize data across every worker GPU.
Oracle’s benchmarking tests also recorded just over 14% for AllToAll operations, where every worker node or GPU communicates and exchanges pieces with every other node in the cluster.
“OCI GPU clusters use RDMA networks that can be deployed with a multi-planar architecture, in which network capacity is provided through multiple independent planes,” explained Lilian Quan, senior principal solutions architect for Oracle.
”This design creates path diversity and helps isolate failures. To gain the full benefit, collective traffic must use capacity across the available planes and respond to congestion or failures without interrupting the training job.
“The OCCL network plugin places the needed networking intelligence for congestion control and link/NIC failure recovery in the collective-communication transport layer. AI frameworks continue to use NCCL or RCCL, and the collective-communication library uses its standard network-plugin interface for inter-node data transfers. The OCCL network plugin uses this interface to manage the network paths used for those transfers.”
OCCL is preinstalled in supported Oracle HPC images within Oracle Cloud Infrastructure (OCI). No separate control plane, agents, or application-side services are required.
With OCCL, Oracle bolsters the pitfalls of large-scale AI training, where distributed clusters are often prone to congested network paths or failed links and NICs.
As covered in the latest SDxCentral Magazine, tech giants like OpenAI have put their minds to enabling networks for the pressures of AI workloads. The ChatGPT maker's contribution comes in the form of the Ethernet-based Multipath Reliable Connection (MRC). The transport protocol, created with support from AMD, Broadcom, Intel, Microsoft, and Nvidia, is currently deployed across all of OpenAI’s largest supercomputers, including the Oracle-built Zettascale10.
Comments