T3 §1 fold for ResNet-34 at TRUE BATCH-NORM — the UN-FUSED gradient ops #
ResNet34Fold.lean makes every parameter output of the per-example SGD train step
den-faithful. This is its batched peer, and one thing about it is different in kind.
⛔ r34's batched render emits *GradB, not *SgdB. Every batched ResNet-34 train step —
resnet34_sgd_train_step, the Adam family, resnet34in_mom256 and its data-parallel peers —
emits the RAW gradient and hands it to an optimizer tail (adamMNextF/adamVNextF, heavy-ball,
plain SGD). The fused θ − lr·∂Loss/∂θ op only appears in renders whose optimizer is SGD-inline,
which EfficientNet's is and r34's batched one is not. Every den = certified lemma in the repo
before this file is stated at the fused form, so none of them applies here.
⭐ That makes this tier better, not worse. A statement about the gradient covers every
optimizer variant at once: sgd, mom, momdp64, adam and adamdp128 all consume the same
*GradB node, so one lemma per op kind certifies the whole family. ⚠ The bf16 twins do NOT: a
bf16 render emits *GradBBf16, its own kind, folded in Foundation/Bf16GradNodes.lean. It is also the
form ConvNeXt's psW carve-out already had to take for a different reason (a hand-written SGD
wrap).
⭐ And no new mathematics: the *SgdB peers were already proven, and the fusion is rfl.
StableHLO.lean's *SgdB_eq_grad family (convWeightSgdB_eq_grad, …) says each fused op IS
θ − lr· applied to the un-fused one, all by rfl, and its own docstring says it exists to
"unblock a batched resnet34_adam_train_step rendered from Proofs/ — the blocker was the fusion,
never Adam." So the eight lemmas below are the per-example VJP bridge under Σ_n with no
θ − lr· wrapper, and EfficientNetFold.lean's fused lemmas are them through *SgdB_eq_grad.
⚠ Symmetric padding, not XLA-SAME. The strided lemmas here are about convStridedWeightGradB
/ convStridedBiasGradB, whose den is flatConvStride2_*; B0's peers are about the
convStridedXla* ops and flatConvStride2Xla_*. The two op families have identical types and
identical emitted shapes, so nothing but the certificate distinguishes them — and r34 is the
PyTorch-origin net, so symmetric is the shipped phase.
Honest residual (the boundary every fold carries) #
- The cotangents are free variables
cot— each lemma is∀ cot, so it holds at the actual backward-chain cotangent without naming it. Pinning each to the emitted residual-backward subgraph is the §1a tie, and isResNet34StepTieB.lean.
Batched stride-1 conv weight GRADIENT denotes the certified Σ_n weight gradient. The
un-fused peer of EnetPoC.convWB_den: same Σ_n of conv_weight_grad_bridge, with no
θ − lr· wrapper because the batched r34 render hands this node to an optimizer tail.
Batched stride-1 conv bias GRADIENT denotes the certified Σ_n bias gradient.
Batched strided conv weight GRADIENT denotes the certified Σ_n weight gradient. Generic
in the kernel size, so the one lemma certifies the 7x7 stem AND every 3x3 downsample W1 AND
every 1x1 projection Wp.
Batched strided conv bias GRADIENT denotes the certified Σ_n bias gradient.
Batched BN γ GRADIENT denotes the certified per-channel γ gradient over the merged
batch+spatial axis m = N·(h·w). γ enters affinely, so there is no batch coupling in the
PARAM gradient and this is bnPerChannel_grad_gamma_correct at that width, through the
network→oc-major reindex bnchwFwd. Generic in the free β.
Batched BN β GRADIENT denotes the certified per-channel β gradient Σ_{batch,spatial} cot
at m = N·(h·w). Carries a free v/γ — β's gradient is the channel sum and depends on
neither.
One batched BN layer's γ and β gradient nodes, tied — the pair every step tie states per
BatchNorm: the emitted bnGammaGradB / bnBetaGradB denote the certified per-channel γ and β
gradients over the merged batch+spatial axis, at the layer's pre-BN activation v and its
output cotangent cot (both in the network layout).
Equations
- One or more equations did not get rendered due to their size.
Instances For
Batched dense weight GRADIENT denotes the certified Σ_n outer product.
Batched dense bias GRADIENT denotes the certified Σ_n cotangent sum.
A stride-1 conv weight gradient node, tied (convWGradB_den).
Equations
- One or more equations did not get rendered due to their size.
Instances For
A stride-1 conv bias gradient node, tied (convBGradB_den).
Equations
- One or more equations did not get rendered due to their size.
Instances For
A stride-2 (symmetric-pad) conv weight gradient node, tied (convStridedWGradB_den).
Equations
- One or more equations did not get rendered due to their size.
Instances For
A stride-2 (symmetric-pad) conv bias gradient node, tied (convStridedBGradB_den).
Equations
- One or more equations did not get rendered due to their size.
Instances For
A stride-2 XLA-SAME conv weight gradient node, tied (EnetPoCG.convStridedXlaWGradB_den).
Equations
- One or more equations did not get rendered due to their size.
Instances For
A stride-1 depthwise weight gradient node, tied (EnetPoCG.depthwiseWGradB_den).
Equations
- One or more equations did not get rendered due to their size.
Instances For
A stride-1 depthwise bias gradient node, tied (Mnv2PaperPoCG.depthwiseBGradB_den).
Equations
- One or more equations did not get rendered due to their size.
Instances For
A stride-2 depthwise weight gradient node, tied (EnetPoCG.depthwiseStridedWGradB_den).
Equations
- One or more equations did not get rendered due to their size.