The label-smoothed loss cotangent, at a GENERAL target #

The ties of the per-example SGD-inline renders pin their top-of-chain cotangent to softmax(logits) − oneHot label — the gradient of plain cross-entropy at a hard label. The batched ImageNet renders emit a different, label-smoothed chain (below), and their step ties — ResNet-34, ResNet-50, MobileNetV2, MobileNetV4, EfficientNet-B0, ConvNeXt-T and ViT, plus the sync-BN twins of the first five — state it through smoothedLossCotGraph.

ResNet34RenderB.lean composes the head cotangent from six kit ops:

%sm  = softmaxRow(logits)          %d0 = %sm − %onehot        %lsa = α · %onehot
%d1  = %d0 + %lsa                  %d2 = %d1 − α/K            %dy  = %d2 / B

so dy = (softmax(z) − t + α·t − α/K) / B, with α = 0.1 baked (the ls0 variants set it to 0) and t arriving as the graph INPUT %onehot — which under mixup or cutmix is a soft vector drawn on the host, not a one-hot. ConvNeXtRenderB and ViTRenderB compose the same six ops. This file is the cotangent lemma those ties need: at a general target and at the smoothed form.

What is proved #

• softCE K t z = Σ_k t_k · crossEntropy K z k — cross-entropy against a target distribution, not a label. softCE_grad is its gradient, (Σ_k t_k)·softmax(z)_j − t_j, with no hypothesis on t at all: it is softmaxCE_grad under pdiv_finset_sum, and the familiar softmax − t is the Σ t = 1 case.
• smoothTarget K α t = (1−α)·t + α/K is label smoothing as a map on targets, and it preserves Σ = 1 (smoothTarget_sum, the one place 0 < K is needed).
• smoothedCE_grad: the gradient of softCE at the SMOOTHED target is exactly the expression the render emits — softmax(z)_j − t_j + α·t_j − α/K. So the six-op chain is not an approximation of the smoothed loss's gradient; it is that gradient, rearranged so that the smoothing is two extra elementwise ops on the target rather than a change to the target itself.
• smoothedLossCotGraph / smoothedLossCotGraph_den / smoothedLossCotGraph_row: the emitted graph, its denotation, and the per-example row of that denotation as the smoothed gradient divided by the batch.

What is NOT claimed #

Note: The / B is the batch mean, and it is a convention, not a theorem here. smoothedLossCotGraph_row states the row IS (1/B)·∂softCE/∂z at that example's logits; that the sum of B such rows is the gradient of the mean loss is the linearity step, and a tie against a *dp* artifact needs the replica mean on top of it (DataParallel.Node).

Note: α is a free real. The committed renders bake 0.1, and the ls0 twins bake 0; both are instances. Note: Nothing here says t is a probability vector — only Σ t = 1 is ever used, which is what mixup's convex combination of two one-hots satisfies.

References #

• Szegedy et al. 2016, Rethinking the Inception Architecture for Computer Vision (§7 label smoothing). https://arxiv.org/abs/1512.00567

The hard-label cotangent and the output-layer chain rule #

The per-example ties' top of chain: any emitted softmax − onehot cotangent denotes the plain cross-entropy gradient at a hard label (softmaxCELossCot_den), and the whole-loss gradient wrt a top dense layer's weight is the certified ∂logits/∂W contracted with that cotangent (lossWeightGrad_eq_sum, by pdiv_comp).