Shared stem-pool and head CertLayers — batched GAP, the dense classifier, the 3×3/s2 pool #
Every conv net in the suite ends in global average pooling and a dense classifier, and both are
globally certified (GAP is linear, dense is affine), so ok := True. Written once here and shared
by MobileNetV4, ResNet-34 and ResNet-50. The ResNets' stem is here too, conv-BN-ReLU and the 3×3/s2
pool as one layer (stemReluPoolLayer), certified where every window is smooth or dead
(StemPoolSmoothAt).
Both backward graphs tie by rfl. den of .gapBackBatched is definitionally the row-wise
GAP VJP, and den of .denseRowBack is rowDenseBackFlat, which is what batchMapHasVJP
reduces to. Note: GAP's VJP does not depend on its input, and den .gapBackBatched uses that by
evaluating the backward at fun _ => 0. That is sound because GAP is linear, and it is why the tie
holds at every x.
Transport a VJP along a local equality. pdiv reads only the germ of the map at the
point (it is an fderiv), so a witness for f is one for any g that agrees with f near
x, with the same backward. How a map that is not globally a composite of certified pieces
still gets a certified VJP: the ReLU-then-pool stem is locally a gather.
Equations
- Proofs.HasVJPAt.congrOfEventuallyEq h hf = { backward := hf.backward, correct := ⋯ }
Instances For
ReLU then the 3×3/s2 pool is locally a gather. Near a pre-activation z with no zero
entry, where every window of relu z is smooth or dead (MaxPool3s2SmoothOrDead), the pooled
ReLU reads each output from the cell maxPool3s2LocalReindex (relu z) names. A smooth window
keeps its strict maximum nearby. A dead window has every pre-activation strictly negative, so
nearby its ReLUs are all 0 and any choice of cell reads 0. This is why a window of dead
ReLUs, where the pool alone has no derivative, costs the composite nothing.
The stem pool's condition, per example: every 3×3 window of the post-ReLU stem activation has its maximum at one position, or is entirely zero (a window of dead ReLUs). A tie is a property of one image's window, so the condition is stated on each row of the batch.
Real batches have many all-zero windows, and the pool alone has no derivative at any of them;
the stem layer is certified as ReLU and pool together (stemReluPoolLayer), where a dead window
is locally constant. What is still excluded is a window whose maximum is POSITIVE and attained
at two positions. On real ImageNet batches such windows occur in every batch, and they come
from two cells reading IDENTICAL input patches (flat image regions); there the stem has no
derivative in the image, so an input-gradient statement must exclude them, but the two cells
are the same function of the stem's weights, so the loss is still differentiable in the
parameters; the loss-gradient statements take the weaker StemPoolTwinAt, which allows them.
The probe script scripts/probes/stem_pool_smooth_probe.py counts all three kinds of window.
Equations
- Proofs.StemPoolSmoothAt N h w v = ∀ (r : Fin N), Proofs.MaxPool3s2SmoothOrDead (Proofs.Tensor3.unflatten (Proofs.Mat.unflatten v r))
Instances For
maxPool3s2Flat_relu_eventuallyEq over the batch, one example per row.
The render's pool backward is the gather's scatter. maxPool3s2FlatBackB (what
den maxPool3s2BackB is) sends each output cotangent to its argmax cell, which is exactly
the adjoint of the gather maxPool3s2LocalReindexB, at every input.
The batched ReLU-then-pool is differentiable at a smooth-or-dead pre-activation.
The batched ReLU-then-pool's VJP at a smooth-or-dead pre-activation: scatter to the argmax cells, then the ReLU mask.
Equations
- One or more equations did not get rendered due to their size.
Instances For
Its backward is the render's pool scatter, then the ReLU mask.
The stem pool's condition along a parameter, per example: every window of the post-ReLU
stem activation is dead, or has its maximum strictly above every cell at another position
except the positions T r twins with it (MaxPool3s2SmoothUpTo). Weaker than
StemPoolSmoothAt (stemPoolTwinAt_of_smoothAt). It is what a parameter gradient needs:
twinned cells tie at every parameter value, so the pool's choice between them does not move
(stemPoolRelu_param_eventuallyEq_select).
Equations
- Proofs.StemPoolTwinAt N h w T v = ∀ (r : Fin N), Proofs.MaxPool3s2SmoothUpTo (T r) (Proofs.Tensor3.unflatten (Proofs.Mat.unflatten v r))
Instances For
A pool selector at y: output (r, co, ho, wo) reads a cell of its own 3×3 window (example
r, channel co) that holds the window's maximum. Where a window's maximum is attained once,
every selector reads that cell; at a tie each picks one of the tied cells. The argmax gather
maxPool3s2LocalReindexB is one selector (maxPool3s2LocalReindexB_isSelect), and so is the
render's select_and_scatter with its GE select, which routes each window's cotangent to one
maximal cell (the first in window order).
Equations
- One or more equations did not get rendered due to their size.
Instances For
The argmax gather is a pool selector.
Along a parameter, ReLU then the pool is a fixed gather, for ANY pool selector σ at
θ₀. Z θ is the pre-ReLU stem activation as the parameter moves; at θ₀ it has no zero
entry, every window of its ReLU is smooth up to twins, and twinned cells are equal at EVERY
θ. Then near θ₀ the pooled ReLU reads each output from the cell σ names: a dead window
stays negative, a strict maximum stays strict, and a twin stays tied with the chosen cell.
Which of two tied twins σ picks does not matter, since they are equal near θ₀.
The gather model v ↦ relu v ∘ σ: linear after the ReLU, so differentiable at any point with
no zero entry, whatever σ.
The pooled stem agrees near x with the argmax gather of the stem's post-ReLU output.
The ResNet stem, strided conv-BN-ReLU then the 3×3/s2 pool, as ONE certified layer.
It is one layer, not cbReluStridedLayer.comp of a pool layer, because the pool alone is not
differentiable at a window of dead ReLUs, and real batches have many. Together they are:
near the input the pooled stem is the argmax gather of the ReLU output
(stemPool_eventuallyEq), so its VJP is the conv-BN-ReLU stage's after the gather's scatter,
and that scatter is what the render's maxPool3s2BackB denotes
(maxPool3s2FlatBackB_eq_reindex). The forward and the backward graph are the two-layer
composite's, so the rendered stem is unchanged; only the certification domain grows.
Equations
- One or more equations did not get rendered due to their size.
Instances For
The stem layer's VJP is the conv-BN-ReLU stage's after the render's pool scatter.