Example n of shard r is example (r, n) of the global batch.

⭐ batchMap commutes with sharding — the block analogue of batchSlice_batchMap. A per-example lift applied to the global batch, restricted to replica r, is the same lift applied to replica r's shard: so every conv, relu, pool, GAP and dense node in a DP render denotes batchShard r of its batch-R·N value as soon as its input does.

…and so does batchMapAux, the shape of every batched backward that recomputes from a saved per-example activation: example n is handed ITS slice of aux, which sharding respects.

A pointwise map commutes with sharding (scaleB, shiftB, divConstB, …).

A pointwise binary map commutes with sharding (addVB, subB, the relu mask selectPosB against its saved activation, …).

The replica mean of a replica-independent value is that value. bnSyncDyStatsB passes the already-global [μ ‖ m2] through the second collective, and this is why that costs nothing.

⭐⭐ The first collective IS the global mean — bnMean_shard on the channel rows.

⭐⭐ The second collective IS the global variance — Chan's parallel variance (bnVar_row_shard_chan): each replica's two-pass σ²_r plus (μ_r − μ)², averaged. This is the lemma the one-round E[x²] exchange could not have, and the reason it was replaced.

σ² + μ² at the global statistics is the global second moment — how a consumer's den, stated at (μ, m2), reads the packed [μ ‖ σ²].

⭐⭐ P1 on the graph, any R. Replica r's bnSyncF, fed by the two-round statistics subgraph over the R replicas' inputs, denotes batchShard r of the batch-R·N bnBatchTensor4 — given that each replica's operand denotes its shard (hx, the chain's induction hypothesis). den_bnSyncF_allReduce_R1 is this at R := 1.

⭐⭐ P2 on the graph, any R. Replica r's bnSyncBack, fed by the collective over the replicas' bnSyncDyStatsB (each reading the packed forward statistics), denotes batchShard r of the batch-R·N bnBatchTensor4_grad_input — given that each replica's saved activation and its incoming cotangent are its shards of the global ones (hx / hxv, hdy). den_bnSyncBack_allReduce_R1 is this at R := 1.

⭐⭐ P2γ on the graph, any R, already all-reduced. The parameter collective over the replicas' bnSyncGammaGradB — each at its shard, its shard's cotangent and the packed global statistics — is 1/R of bnPerChannel_grad_gamma at N := R·N: the committed γ gradient at the global batch. The BN γ node is the one parameter gradient sync-BN changes, because it is the one that reads x̂.

The handed-back running MEAN under sync-BN is the global batch's own — bnStatsMeanB on the packed statistics denotes what bnBatchMeanB at N := R·N denotes, so the host EMAs the global statistic on every replica.

…and so is the handed-back VARIANCE — the global batch's bnVar, what bnBatchVarB at N := R·N denotes. ⛔ No per-replica variance is ever averaged on its own: the between-shard spread (μ_r − μ)² rides along, which is what Chan's formula is.

⭐⭐ P4 for the Σ_n-shaped gradients, at the conv weight. The collective over the replicas' convWeightGradB, each on its shard at the shard-r block of the global cotangent, is 1/R of the batch-R·N node at that cotangent. den_allReduceMeanF_convWeightGradB (piece 2) is the same collective with NO relation between the replicas' inputs; this is what it becomes once P1–P3 relate them. Every other Σ_n gradient (denseWeightGradB, denseBiasGradB, the strided / depthwise / bias kinds) composes by the same three lines.

P4 at the BN β gradient: Σ dy over each shard's channel row, averaged, is 1/R of the global row's Σ dy. β reads no statistic, so this is the row split alone.

A VJP backward is linear in its cotangent — read off HasVJP.correct. This is the one step that reconciles a DP render's divConstB N with the batch-R·N step's divConstB (R·N): the per-replica cotangent is R · the shard of the global one, so every per-replica gradient is R · its shard contribution, and the collective's 1/R cancels it.