The CIFAR CNN — every parameter gradient node IS the loss's derivative, up to pool twins #
cifar_train_step_tied_certified ties each of the fourteen SGD updates to the certified per-layer
Jacobian contracted with the cotangent the emitted chain threads to it. cifar_net_lossGrad
states that the un-fused *Grad node of each layer, at the chain cotangent, is the gradient of the
loss in that parameter, for any loss L of the logits with gradient g there;
cifar_net_lossGrad_CE instantiates it at the softmax cross-entropy the render emits.
The two pools are handled as the MNIST CNN's one (CnnFold.cnn_net_lossGrad): each pool's clause
allows ties between twins, cells equal at every weight upstream of that pool (CnnFold.CnnPoolTwin
for the first, CifarPoolTwin2 for the second), and each pool's backward routes a window's
cotangent to the one cell a selection names (CnnFold.cnnChainCotW2Sel, cifarChainCotW2Sel), as
the rendered select_and_scatter does. At the first argmax of each window, the cell that op
picks, the step tie's own chains are these (CnnFold.cnnChainCotW2_eq_sel, cifarChainCotW2_eq_sel).
A parameter of the first stage sees both pools move; its
germ rewrites the outer pool first, at the true pre-activation, then the inner one.
Hypotheses. Odd kernels, every ReLU off its kink, every pool window dead or tied only between
twins, each selection naming a maximum of every window (CifarLossSmoothAt).
Scope. One example (the emitted module batch-contracts; den is per-example).
From the first pool's pre-activation to the second's: ReLU, pool, conv₃, ReLU, conv₄.
Equations
- One or more equations did not get rendered due to their size.
Instances For
The second pool's pre-activation (conv₄'s output).
Equations
- Proofs.CifarFold.cifarPre2 W₁ b₁ W₂ b₂ W₃ b₃ W₄ b₄ x = Proofs.CifarFold.cifarUp2 W₃ b₃ W₄ b₄ (Proofs.CnnFold.cnnPoolPre W₁ b₁ W₂ b₂ x)
Instances For
Two cells of the second pool's input are twins: equal at every weight of the four convs, in every channel.
Equations
- One or more equations did not get rendered due to their size.
Instances For
The conv₂-output cotangent at a first-pool selection σ₁: cifarChainCotW2 with the pool
backward routing each window's cotangent to the ONE cell σ₁ names, then the ReLU mask.
Equations
- One or more equations did not get rendered due to their size.
Instances For
The step tie's conv₂ cotangent is the capstone's at the first argmax of the first pool's
windows (maxPool2Argmax), the cell the rendered select_and_scatter picks, at every point.
The smooth-point bundle the loss gradient needs. Every ReLU off its kink; every window of each pool dead or tied only between that pool's twins; each selection naming a maximum of every window.
- pool1 : SmallParamGrad.MaxPool2SmoothUpTo (CnnFold.CnnPoolTwin c1 kH kW x) (Tensor3.unflatten (CnnFold.cnnPoolPre W₁ b₁ W₂ b₂ x))
- sel1 : SmallParamGrad.PoolSelDom σ₁ (relu (c1 * (2 * (2 * h)) * (2 * (2 * w))) (CnnFold.cnnPoolPre W₁ b₁ W₂ b₂ x))
- pool2 : SmallParamGrad.MaxPool2SmoothUpTo (CifarPoolTwin2 c1 c2 kH kW x) (Tensor3.unflatten (cifarPre2 W₁ b₁ W₂ b₂ W₃ b₃ W₄ b₄ x))
Instances For
Every CIFAR-CNN parameter node is the gradient of L in that parameter: the fourteen
un-fused nodes, each at the cotangent the chain threads to its layer (each pool routed at its
selection), stated against L of cifarCnnForward with that one parameter varied.
Equations
- One or more equations did not get rendered due to their size.
Instances For
Every CIFAR-CNN parameter node is the gradient of L in that parameter, whenever g is
L's gradient at the logits.
Hypotheses: odd kernels, and CifarLossSmoothAt — every ReLU off its kink, every window of
each pool dead or tied only between cells that are the same function of the weights upstream
of it, each selection naming a maximum of every window.
The artifact's loss: every node is the gradient of the softmax cross-entropy at label,
g the emitted loss cotangent.