Traditionally, training state-of-the-art Large Language Models (LLMs) relies on a monolithic cluster of accelerators, where all devices are collocated and tightly coupled via high-bandwidth to constantly synchronize gradients. However, physical and cost constraints make scaling a single slice increasingly difficult.
To scale further, we can connect multiple compute slices across geographically distributed datacenters. However, when training across multiple datacenters over wide-area networks (WAN), standard data-parallel training suffers from severe network stalls because slow WAN transfers cannot be fully overlapped with computation.
To address this, the Google DeepMind and Google Research teams developed DiLoCo (Distributed Low-Communication), a distributed optimization algorithm designed to make training resilient to these low-bandwidth, high-latency connections. In this blog post, we discuss the core concepts of DiLoCo, how it is implemented in MaxText (Google's open-source JAX-based LLM training framework), and the performance benefits it brings to large-scale LLM training on Google Cloud.
DiLoCo was introduced by Google DeepMind in the paper DiLoCo: Distributed Low-Communication Training of Language Models. Instead of synchronizing gradients across all devices at every single training step (as in standard Data Parallelism), DiLoCo partitions training across M independent compute groups, referred to as learners, and allows each learner to perform many local optimization steps independently before communicating across groups.
DiLoCo operates on two levels:
H steps, the M learners compute the difference (delta) between their current weights and the global weights at the start of the interval. The learners average these deltas across the network via a distributed all-reduce and apply an outer optimizer (typically SGD with Nesterov momentum) to update the global weights, which then serve as the starting point for the next inner-loop interval.Despite communicating significantly less frequently (by a factor of H, e.g., 1-2 orders of magnitude less) than standard synchronous training, DiLoCo has been shown to achieve comparable convergence and model quality.
The Vanilla DiLoCo described above reduces total communication volume by a factor of H, however, all learners still transmit their weight deltas for the entire model all at once at every H-th step. On a thin network pipe, that full-model transfer creates a large periodic stall every H steps. This issue is alleviated by Streaming DiLoCo, which is described in the next section.
In multi-slice training, communication within a single slice is handled by high-speed Inter-Chip Interconnects (ICI). When scaling to multiple slices, synchronization across slices must traverse the Data Center Network (DCN) or, if the slices are geographically distributed, wide-area networks (WAN).
Within a single datacenter, DCN bandwidth is high enough that JAX and XLA can often hide standard Data Parallel all-reduce collectives by overlapping gradient communication with backward-pass computation. Across datacenters or congested links, however, bandwidth drops and latency rises sharply. Standard DP stalls at every step waiting for global gradient synchronization, collapsing Model FLOPS Utilization (MFU).
Streaming DiLoCo eliminates this synchronization stall by partitioning the model's parameters into P disjoint fragments. Over the course of H steps, every single fragment in the model is synchronized exactly once, achieving the exact same outer-loop cadence as vanilla DiLoCo without spiky bandwidth utilization.
Using a roofline model (similar to the JAX Scaling Book), we can calculate the network bandwidth required to prevent communication stalls for a model with ∣Θ∣ parameters (at b bytes per parameter, e.g., b=2 for bfloat16) and per-step compute time tstep=tfwd+tbwd. For example, the standard Data Parallel algorithm transfers 2∣Θ∣b bytes per sync via an all-reduce collective, which is a distributed reduce-scatter where each node reduces a different part, followed by an all-gather so all participants receive all parts. To achieve zero communication stalls, we overlap the reduce-scatter of the gradients with the backward pass, which produces them and takes tbwd time. Therefore, the peak required bandwidth is Bdp ≈ 2∣Θ∣b / tbwd.
Worked Example (Qwen3-32B in bfloat16, tstep≈6.2s, P=H=65):
To evaluate distributed efficiency under realistic network bottlenecks, we benchmarked Qwen3-32B (64 decoder layers, partitioned into P = 65 fragments with H = 65) across 2× TPU v5p-32 slices (dcn_diloco_parallelism=2) using built-in Linux traffic-control (tc) bandwidth shaping to limit inter-slice bandwidth across three network regimes: Unthrottled (Line Rate), 30 Gbit/s aggregate (Moderate Congestion), and 10 Gbit/s aggregate (Severe Congestion).
Key Findings:
all-reduce saturates the link. Vanilla DiLoCo (H = 65) fares much better (7.55 s/step, a 7.30× speedup), but still suffers a periodic full-model synchronization bubble every 65 steps. Streaming DiLoCo (P = 65, H = 65) maintains a flat 5.76 s/step—delivering a 9.57× speedup over DP and eliminating the step-H stall.A common question with low-communication training is whether allowing learners to train independently between synchronizations degrades model quality—especially on Mixture-of-Experts (MoE) architectures, where independent updates could cause expert routers to drift apart.
We evaluated Streaming DiLoCo against synchronous Data Parallel pre-training across two MoE architectures in MaxText (Qwen3-30B-A3B and Gemma4-26B-A4B) over 4,000 training steps:
Empirical results confirm that Streaming DiLoCo preserves—and even slightly improves—validation loss:
β = 0.9, outer learning rate η_outer between 0.3 and 0.7) across fragment pseudo-gradients stabilizes the global trajectory.MaxText provides a native JAX and Flax NNX implementation of both Vanilla and Streaming DiLoCo—requiring no external central parameter servers.
You can enable Streaming DiLoCo in your MaxText YAML configuration or CLI overrides by setting enable_diloco: true and enable_streaming_diloco: true. As a best practice, set both num_diloco_fragments (P) and diloco_sync_period (H) to N_layers + 1 so that exactly one fragment is synchronized per training step (Δh = 1):
# Enable DiLoCo and Streaming DiLoCo
enable_diloco: true
enable_streaming_diloco: true
# Partition into N_layers + 1 fragments (e.g., 36 layers + 1 embedding/head = 37 for Qwen3-8B,
# 48 + 1 = 49 for Qwen3-30B-A3B, or 64 + 1 = 65 for Qwen3-32B)
num_diloco_fragments: 37
diloco_sync_period: 37
use_sequential_layers: false # false = interleaved layers across fragments (recommended)
# Outer Nesterov SGD optimizer hyperparameters
diloco_outer_lr: 0.7
diloco_outer_momentum: 0.9
# Multi-slice topology: number of DiLoCo learner islands across DCN/WAN
dcn_diloco_parallelism: 2 # e.g., 2 TPU slices (or set to -1 to infer from num_slices)
# Optional: emulate constrained WAN bandwidth on fast clusters via Linux tc
# dcn_bandwidth_limit: "10gbit"
In MaxText, diloco.py splits each global batch across the M learners so each learner processes GBS / M tokens per step. To overlap cross-slice communication with TPU computation in Streaming DiLoCo, weight deltas for the scheduled fragment are offloaded to M-way sharded syncer workers running on the learner host CPUs. Rather than routing through a single central coordinator, the host-CPU syncer shards execute a distributed all-reduce across the network in the background to average the fragment deltas while the TPUs continue running inner-loop training steps. Each syncer shard then applies the outer Nesterov momentum update locally and writes the updated fragment weights back to its colocated TPU learner. To dive deeper or launch your first run:
Streaming DiLoCo in MaxText decouples large-scale LLM pre-training from the requirement of a single, monolithic, high-bandwidth fabric. By combining use of an inner and outer optimizer with fine-grained parameter streaming, engineering teams can pool TPU slices across congested datacenter networks or geographically distributed metros—sustaining high hardware utilization without compromising model convergence. Streaming DiLoCo is available today in MaxText v0.2.5 on PyPI, the corresponding Docker images and on GitHub main. Try it today for experimenting with distributed training workloads.