Sharding commutes with relabelling the per-example index. The batch axis is outside the per-example one, so relabelling within an example and cutting the batch do not interact.

A per-example node on every replica denotes the shard of the same node on the global batch.

…stated for relu at the whole-batch relu, the form the committed forward is written in.

The residual fan-in on every replica is the shard of the global one.

den_addVB_shard with the SECOND operand first on the right: ResNet-50's render emits addVB(body, projection), and residualProj proj body adds proj + body.

The per-example identity skip spelled addV (EfficientNet-B0's MBConv) on every replica is the shard of the global one.

Swish on every replica denotes the shard of the global swish — pointwise, like relu.

relu6 on every replica is the shard of the global relu6 — the clamp reads one cell, so cutting the batch before or after it is the same. Stated at the whole-batch relu6 the committed forwards are written in.

A per-example relabel on every replica is the shard of the relabelled global batch — e.g. MobileNetV4's head casts [N, c] ↔ [N, c, 1, 1], read at the network index.

The sync-BN site on replica r is shard r of the global-batch BatchNorm. den_bnSyncF_allReduce (P1 on the graph), carried across the mul_assoc seam: the right-hand side is bnBatchLA — what bnBatchF denotes — at N := R·N.

The reduction width a BatchNorm site needs nonzero, from the three positive dimensions.

The two-sided relu6 mask is linear in the cotangent it gates.

The three-term BatchNorm input-gradient is linear in dy — both its reductions are.

The 3×3/s2 pool's select_and_scatter is linear in the cotangent it scatters.

reassocB of a shard is the shard of the reassocB.

The sync-BN backward on replica r is shard r of the global-batch BN backward — den_bnSyncBack_allReduce (P2 on the graph) at the network index. The right-hand side is bnInB, the single-device chain's BN link, at N := R·N.

P4 at the STRIDED conv weight — den_allReduceMeanF_convWeightGradB_shard's strided peer.

P4 at the dense weight — the head's Σ_n outer product, split by replica.

P4 at the dense bias — Σ_n cot, split by replica.

(1/R)·(R·v) = v — the collective's mean against the divisor's R.

A replica family at R × the shards of COT gives the DP tie — the collective's 1/R (inv_mul_R) cancels the R the homogeneity lemmas (*_smul) carry. The same three lines for every kind below.

rowB of a shard is the shard of the rowB — a relabelling inside each example.

The divisor step. Replica r divides its smoothed-CE cotangent by B — the render's divConstB at the per-replica batch — and the single-device step at the global batch divides by R·B. At the replica's shard of the logits and targets, the replica's cotangent is R × its shard of the global one. Nothing else about the loss differs: softmax, the label-smoothing shift and the target are per example.

The row-wise input-VJP dX = W·dy (the classifier's, and the SE excite dense's) is linear in dy.

The row-wise input-VJP is batchMap of W·, so it shards like every per-example lift.

P4 at the depthwise weight — each replica's Σ_n over its own examples, averaged, is 1/R of the global batch's Σ_n.

P4 at the strided depthwise weight.

P4 at the XLA-SAME strided depthwise weight (MobileNetV2's b2 / b4 / b7 / b14, EfficientNet-B0's b2 / b4 / b6 / b12).

P4 at the XLA-SAME strided conv weight (the stem) — den_allReduceMeanF_convWeightGradB_shard's peer. Only the certificate differs from the symmetric strided one; the batch split is the same.