Google DeepMind unveiled a way to train advanced AI models across distributed data centers.

Known as decoupled distributed low-communication (DiLoCo), the architecture isolates local disruptions such as hardware failures and network bottlenecks so that other parts of the system can keep learning efficiently. It accomplishes this by partitioning large-scale training workloads into decoupled compute “islands,” between which data is exchanged asynchronously

Detailed in a paper from the Google unit, decoupled DiLoCo decomposes a global cluster of processors into independent, asynchronous "learners." Usually, a global consensus is needed at every step of the process – the requirement that every single part of a computer cluster must be in agreement and perfectly synchronized before the system can move to the next step.

With DiLoCo, each learner group operates on its own data shard at its own speed and communicates parameter fragments to a central, lightweight synchronizer – the central coordinator of the distributed training system – that aggregates updates asynchronously.

The synchronizer utilizes a "minimum quorum" strategy, a specific threshold that defines the minimum number of independent learners required to complete their work before the system moves the training forward. This is supported by an adaptive grace window, a temporal buffer designed to maximize sample efficiency without sacrificing the system's overall speed; and token-weighted merging, a dynamic weighting system used to reconcile the divergent states of different learners. With the elements, faster learners or those processing more data are weighted appropriately, maintaining model stability without sacrificing speed.

According to Google, the approach allows for strictly zero global downtime. The system reportedly maintains a training goodput of nearly 90% under aggressive simulated hardware failures, referring to the time spent executing useful training steps. This is in contrast to traditional elastic methods, where goodput can fall to 40% during runtime.

“This infrastructure is also self-healing,” claimed Arthur Douillard, staff research scientist at Google DeepMind. “Decoupled DiLoCo continued the training process after the loss of entire learner units, and then seamlessly reintegrated them when they came back online.”

The system was tested using Google’s Gemma 4 AI models, with Douillard claiming the team trained a 12-billion parameter model across four separate U.S. regions using 2-5 Gb/s of wide-area networking.

“Notably, the system achieved this training result more than 20-times faster than conventional synchronization methods. This is because our system incorporates required communication into longer periods of computation, avoiding the ‘blocking’ bottlenecks where one part of the system must wait for another,” Douillard wrote. “By enabling training jobs at internet-scale bandwidth, it can tap any unused compute wherever it sits, turning stranded resources into useful capacity."

According to the researcher, the system allows the ability to mix different generations of its Tensor Processing Unit (TPU) in a single training run, extending the shelf life of existing hardware and increasing the total compute available for model training.

The DiLoCo reveal comes in a busy week for Google, with its Next event unveiling another distributed-based solution in the form of Virgo. The networking fabric, designed to connect accelerator racks, orchestrates workloads for its AI Hypercomputer platform. Capable of interconnecting 134,000 chips, the scale-out fabric can support upwards of 47 petabits per sector of non-blocking bi-sectional bandwidth in a single east-west fabric.