Lipschitz constants for the CNN softmax-CE loss — descent through the pool

SgdDescentMlp discharged sgd_descends' smoothness hypothesis for every MLP weight layer; this file extends the program to the Chapter-3 MNIST CNN (conv → relu → conv → relu → maxpool → dense → relu → dense → relu → dense). What's genuinely new versus the MLP:

• The dense head is free. Below the pool the CNN is an MLP at the pooled activation: the loss-of-W₅/W₄/W₃ maps are literal instances of linear_sgd_descends / mlp_hidden_sgd_descends / mlp_input_sgd_descends at x := maxPoolFlat (…). No new theorems are needed (the MLP statements are generic in the fixed activation vector).

• The max-pool needs a quantitative SELECTION margin. MaxPool2Smooth (pairwise-distinct window cells) is the qualitative off-the-kink condition; descent needs its quantitative form MaxPool2MarginQ δ (pairwise window gaps exceed 2δ): a perturbation of at most δ per entry then cannot reorder any window, so the argmax — hence the pool's routing pattern — FREEZES along the step segment (MaxPool2MarginQ.isArgmax_iff), exactly as the ReLU margins freeze the masks. The pool is also 1-Lipschitz per entry (maxPoolFlat_entry_lipschitz) and ℓ1-contractive across entries (maxPoolFlat_l1_contract — the 2×2 stride-2 windows partition the input), so drift passes through it unamplified.

• Conv layers are dense layers with weight sharing. The conv output is affine in the kernel; each output entry reads one kernel slab against bounded input values (flatConv_kernel_drift), and the ℓ1 drift picks up the spatial multiplicity h·w — each kernel entry touches every spatial position (flatConv_kernel_drift_sum).

The capstone cnn_conv2_sgd_descends mirrors mlp_input_sgd_descends: under the four margins (relu₂, pool selection, relu₃, relu₄) at the step radius and the small-step condition, one inexact SGD step on the second conv kernel provably decreases the cross-entropy loss by ≥ lr·‖∇L‖₂²/2, with the segment-Lipschitz constant explicit.

cnn_conv1_sgd_descends extends the program one layer deeper: the step now crosses conv2 AS A FUNCTION OF ITS INPUT. Conv is linear there, its Jacobian entry a single kernel tap (convTap, extracted point-free from the certified input-VJP), and its ℓ1 operator factor is LOCALITY — (channels)·kH·kW·w₂, not a spatial count. Under FIVE margins (relu₁ + the conv2 four, at conv1 radii) every routing decision freezes and the loss provably drops.

cnn_conv2_bias_sgd_descends / cnn_conv1_bias_sgd_descends close the biases: the bias-map Jacobian is a Kronecker channel indicator (conv2d_bias_pdiv, extracted from the certified bias VJP), the per-entry drift is exactly |e o| (no input bound a), and the rungs are the kernel arguments verbatim with the conv stage's a·D radii replaced by the bare D and a² ↦ 1 in the constants. EVERY parameter of the Chapter-3 CNN — both conv kernels, both conv biases, and the dense head — now has a proven descent statement.

theorem Proofs.max4_sub_abs_le {a b c d a' b' c' d' δ : ℝ} (h1 : |a - a'| ≤ δ) (h2 : |b - b'| ≤ δ) (h3 : |c - c'| ≤ δ) (h4 : |d - d'| ≤ δ) : |max (max a b) (max c d) - max (max a' b') (max c' d')| ≤ δ

A four-way max moves by at most the largest cell movement (ℓ∞).

theorem Proofs.max4_sub_abs_le_sum {a b c d a' b' c' d' : ℝ} : |max (max a b) (max c d) - max (max a' b') (max c' d')| ≤ |a - a'| + |b - b'| + |c - c'| + |d - d'|

A four-way max moves by at most the sum of the cell movements (ℓ1) — the per-window step of the pool's ℓ1 contraction.

@[reducible]

def Proofs.t3Idx {c h w : ℕ} (ci : Fin c) (hi : Fin h) (wi : Fin w) : Fin (c * h * w)

Flat index of a Tensor3 coordinate (the suite's row-major layout).

⚠ @[reducible] is load-bearing on Lean ≥ 4.33 (see planning/archive/lean_434_and_cleanup.md): t3Idx_def folds the raw encoding into the ite CONDITION below, but simp does not rewrite inside the Decidable INSTANCE argument, so the goal carries a folded condition over an unfolded instance and every if_pos/if_neg here fails to match. basisVec, which produces that ite, is @[reducible] for the same reason.

Equations

• Proofs.t3Idx ci hi wi = finProdFinEquiv (finProdFinEquiv (ci, hi), wi)

theorem Proofs.unflatten_t3Idx {c h w : ℕ} (v : Vec (c * h * w)) (ci : Fin c) (hi : Fin h) (wi : Fin w) : Tensor3.unflatten v ci hi wi = v (t3Idx ci hi wi)

t3Idx reads back through Tensor3.unflatten.

theorem Proofs.flatten_t3Idx {c h w : ℕ} (T : Tensor3 c h w) (ci : Fin c) (hi : Fin h) (wi : Fin w) : T.flatten (t3Idx ci hi wi) = T ci hi wi

Tensor3.flatten reads off at a t3Idx.

theorem Proofs.sum_t3 {c h w : ℕ} (f : Fin (c * h * w) → ℝ) : ∑ k : Fin (c * h * w), f k = ∑ ci : Fin c, ∑ hi : Fin h, ∑ wi : Fin w, f (t3Idx ci hi wi)

A flat sum is the triple tensor sum.

theorem Proofs.t3Idx_surj {c h w : ℕ} (k : Fin (c * h * w)) : ∃ (ci : Fin c) (hi : Fin h) (wi : Fin w), k = t3Idx ci hi wi

Every flat spatial index is a t3Idx — lets a per-cell bound be lifted to the whole flattened conv-output vector (∀ k), the form relu_close / maxPoolFlat_close / dense_close consume.

def Proofs.winRowEquiv (h : ℕ) : Fin h × Fin 2 ≃ Fin (2 * h)

The window-cell parameterization (out-row, sub-row) ↦ in-row is a bijection — pooled windows tile the rows.

Equations

• One or more equations did not get rendered due to their size.

def Proofs.winColEquiv (w : ℕ) : Fin w × Fin 2 ≃ Fin (2 * w)

Column version of winRowEquiv.

Equations

• One or more equations did not get rendered due to their size.

theorem Proofs.sum_window_cells {h w : ℕ} (g : Fin (2 * h) → Fin (2 * w) → ℝ) : ∑ ho : Fin h, ∑ wo : Fin w, ∑ ab : Fin 2 × Fin 2, g (winRowInv ho ab.1) (winColInv wo ab.2) = ∑ hi : Fin (2 * h), ∑ wi : Fin (2 * w), g hi wi

Summing a function over all window cells of all windows is summing it over the whole spatial grid — the 2×2 stride-2 windows partition the input.

theorem Proofs.maxPoolFlat_apply {c h w : ℕ} (u : Vec (c * (2 * h) * (2 * w))) (ci : Fin c) (ho : Fin h) (wo : Fin w) : maxPoolFlat c h w u (t3Idx ci ho wo) = max (max (u (t3Idx ci (winRowInv ho 0) (winColInv wo 0))) (u (t3Idx ci (winRowInv ho 1) (winColInv wo 0)))) (max (u (t3Idx ci (winRowInv ho 0) (winColInv wo 1))) (u (t3Idx ci (winRowInv ho 1) (winColInv wo 1))))

The pooled entry at (ci, ho, wo) is the four-way max of its window cells, in flat coordinates.

theorem Proofs.maxPoolFlat_entry_lipschitz {c h w : ℕ} (u v : Vec (c * (2 * h) * (2 * w))) {δ : ℝ} (hδ : ∀ (k : Fin (c * (2 * h) * (2 * w))), |u k - v k| ≤ δ) (q : Fin (c * h * w)) : |maxPoolFlat c h w u q - maxPoolFlat c h w v q| ≤ δ

Per-entry pool Lipschitz bound: if every input entry moves by at most δ, every pooled entry moves by at most δ.

theorem Proofs.maxPoolFlat_l1_contract {c h w : ℕ} (u v : Vec (c * (2 * h) * (2 * w))) : ∑ q : Fin (c * h * w), |maxPoolFlat c h w u q - maxPoolFlat c h w v q| ≤ ∑ k : Fin (c * (2 * h) * (2 * w)), |u k - v k|

ℓ1 contraction: the pooled drift, summed over all pooled entries, is at most the input drift summed over all input entries (windows are disjoint, max is 1-Lipschitz). The pool passes ℓ1 budgets through unamplified.

theorem Proofs.ne_of_gap_of_close {xa xb ya yb δ : ℝ} (hgap : 2 * δ < |xa - xb|) (ha : |ya - xa| ≤ δ) (hb : |yb - xb| ≤ δ) : ya ≠ yb

Two values farther apart than 2δ cannot be equalized by δ-perturbations.

theorem Proofs.lt_of_lt_gap_of_close {xa xb ya yb δ : ℝ} (hlt : 2 * δ < xb - xa) (ha : |ya - xa| ≤ δ) (hb : |yb - xb| ≤ δ) : ya < yb

Strict order survives δ-perturbations across a 2δ gap.

def Proofs.MaxPool2MarginQ {c h w : ℕ} (δ : ℝ) (x : Tensor3 c (2 * h) (2 * w)) : Prop

Quantitative pool-selection margin: every two cells of every 2×2 window differ by more than 2δ. The quantitative form of MaxPool2Smooth — a perturbation of at most δ per entry can neither create a tie nor reorder a window, so the pool's argmax routing freezes. The pool peer of the ReLU margin a·D < |zⱼ|.

Equations

• One or more equations did not get rendered due to their size.

theorem Proofs.MaxPool2MarginQ.smooth_of_close {c h w : ℕ} {δ : ℝ} {x y : Tensor3 c (2 * h) (2 * w)} (hm : MaxPool2MarginQ δ x) (hclose : ∀ (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)), |y ci hi wi - x ci hi wi| ≤ δ) : MaxPool2Smooth y

Every point within δ of a margined point is smooth (no window ties).

theorem Proofs.MaxPool2MarginQ.smooth {c h w : ℕ} {δ : ℝ} (hδ0 : 0 ≤ δ) {x : Tensor3 c (2 * h) (2 * w)} (hm : MaxPool2MarginQ δ x) : MaxPool2Smooth x

A margined point is itself smooth.

theorem Proofs.MaxPool2MarginQ.isArgmax_iff {c h w : ℕ} {δ : ℝ} {x y : Tensor3 c (2 * h) (2 * w)} (hm : MaxPool2MarginQ δ x) (hclose : ∀ (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)), |y ci hi wi - x ci hi wi| ≤ δ) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) : MaxPool2IsArgmax y ci hi wi ↔ MaxPool2IsArgmax x ci hi wi

The argmax freezes: within δ of a margined point, every window's argmax cell is the same as at the margined point.

noncomputable def Proofs.convPad {ic h w : ℕ} (kH kW : ℕ) (x : Tensor3 ic h w) (c : Fin ic) (kh : Fin kH) (kw : Fin kW) (hi : Fin h) (wi : Fin w) : ℝ

The padded input read that multiplies kernel entry (·, c, kh, kw) at output position (hi, wi) — names the dite inside conv2d so the affine-in-the-kernel structure can be stated. Depends on the input only, never the kernel.

Equations

• One or more equations did not get rendered due to their size.

theorem Proofs.conv2d_eq_convPad {ic oc h w kH kW : ℕ} (W : Kernel4 oc ic kH kW) (b : Vec oc) (x : Tensor3 ic h w) (o : Fin oc) (hi : Fin h) (wi : Fin w) : conv2d W b x o hi wi = b o + ∑ c : Fin ic, ∑ kh : Fin kH, ∑ kw : Fin kW, W o c kh kw * convPad kH kW x c kh kw hi wi

conv2d through convPad: bias plus the kernel-linear form.

theorem Proofs.abs_convPad_le {ic h w kH kW : ℕ} (x : Tensor3 ic h w) {a : ℝ} (ha : 0 ≤ a) (hx : ∀ (c : Fin ic) (i : Fin h) (j : Fin w), |x c i j| ≤ a) (c : Fin ic) (kh : Fin kH) (kw : Fin kW) (hi : Fin h) (wi : Fin w) : |convPad kH kW x c kh kw hi wi| ≤ a

Padded reads are bounded by the input bound (out-of-bounds reads are zero).

def Proofs.k4Idx {oc ic kH kW : ℕ} (o : Fin oc) (c : Fin ic) (kh : Fin kH) (kw : Fin kW) : Fin (oc * ic * kH * kW)

Flat index of a Kernel4 entry (the suite's row-major layout).

Equations

• Proofs.k4Idx o c kh kw = finProdFinEquiv (finProdFinEquiv (finProdFinEquiv (o, c), kh), kw)

theorem Proofs.unflatten_k4Idx {oc ic kH kW : ℕ} (v : Vec (oc * ic * kH * kW)) (o : Fin oc) (c : Fin ic) (kh : Fin kH) (kw : Fin kW) : Kernel4.unflatten v o c kh kw = v (k4Idx o c kh kw)

k4Idx reads back through Kernel4.unflatten.

theorem Proofs.flatten_k4Idx {oc ic kH kW : ℕ} (W : Kernel4 oc ic kH kW) (o : Fin oc) (c : Fin ic) (kh : Fin kH) (kw : Fin kW) : W.flatten (k4Idx o c kh kw) = W o c kh kw

Kernel4.flatten reads off at a k4Idx — the forward peer of unflatten_k4Idx, lifting a per-entry kernel bound to the flattened vector.

theorem Proofs.k4Idx_surj {oc ic kH kW : ℕ} (idx : Fin (oc * ic * kH * kW)) : ∃ (o : Fin oc) (c : Fin ic) (kh : Fin kH) (kw : Fin kW), idx = k4Idx o c kh kw

Every flat kernel index is a k4Idx — lets the abstract ∀ idx gradient accuracy be discharged per (o,cc,kh,kw) by cnn_conv2_grad_close.

theorem Proofs.k4Idx_inj {oc ic kH kW : ℕ} {o : Fin oc} {c c' : Fin ic} {kh kh' : Fin kH} {kw kw' : Fin kW} (hEq : k4Idx o c kh kw = k4Idx o c' kh' kw') : c = c' ∧ kh = kh' ∧ kw = kw'

theorem Proofs.sum_abs_kernel_slab_le {oc ic kH kW : ℕ} (e : Vec (oc * ic * kH * kW)) (o : Fin oc) : ∑ c : Fin ic, ∑ kh : Fin kH, ∑ kw : Fin kW, |e (k4Idx o c kh kw)| ≤ ∑ idx : Fin (oc * ic * kH * kW), |e idx|

The ℓ1 mass of one output-channel slab is at most the total ℓ1 mass — the conv analogue of a dense column being part of the flat parameter vector.

theorem Proofs.sum_abs_k4 {oc ic kH kW : ℕ} (e : Vec (oc * ic * kH * kW)) : ∑ idx : Fin (oc * ic * kH * kW), |e idx| = ∑ o : Fin oc, ∑ c : Fin ic, ∑ kh : Fin kH, ∑ kw : Fin kW, |e (k4Idx o c kh kw)|

The slabs tile the kernel: summing the slab masses over the output channels recovers the total ℓ1 mass.

def Proofs.w3Idx {ic kH kW : ℕ} (c : Fin ic) (kh : Fin kH) (kw : Fin kW) : Fin (ic * kH * kW)

Flat index of a conv window slot (c, kh, kw) — k4Idx without the output channel (row-major, fan-in ic·kH·kW).

Equations

• Proofs.w3Idx c kh kw = finProdFinEquiv (finProdFinEquiv (c, kh), kw)

theorem Proofs.sum_w3 {ic kH kW : ℕ} (g : Fin (ic * kH * kW) → ℝ) : ∑ idx : Fin (ic * kH * kW), g idx = ∑ c : Fin ic, ∑ kh : Fin kH, ∑ kw : Fin kW, g (w3Idx c kh kw)

The triple conv-window sum collapses to one flat sum over the fan-in — the conv analogue of dot being a single-index sum (mirrors sum_abs_k4, one fewer axis).

noncomputable def Proofs.convWindow {ic h w : ℕ} (kH kW : ℕ) (x : Tensor3 ic h w) (hi : Fin h) (wi : Fin w) : Vec (ic * kH * kW)

The per-output-coordinate conv window as a flat Vec over the fan-in: the (padded) input reads that the kernel slab dots against.

Equations

• One or more equations did not get rendered due to their size.

noncomputable def Proofs.convKernelMat {oc ic kH kW : ℕ} (W : Kernel4 oc ic kH kW) : Mat (ic * kH * kW) oc

The kernel as a Mat (ic·kH·kW) oc — column o is the flattened slab.

Equations

• Proofs.convKernelMat W idx o = W o (finProdFinEquiv.symm (finProdFinEquiv.symm idx).1).1 (finProdFinEquiv.symm (finProdFinEquiv.symm idx).1).2 (finProdFinEquiv.symm idx).2

@[simp]

theorem Proofs.convWindow_w3 {ic h w : ℕ} (kH kW : ℕ) (x : Tensor3 ic h w) (hi : Fin h) (wi : Fin w) (c : Fin ic) (kh : Fin kH) (kw : Fin kW) : convWindow kH kW x hi wi (w3Idx c kh kw) = convPad kH kW x c kh kw hi wi

@[simp]

theorem Proofs.convKernelMat_w3 {oc ic kH kW : ℕ} (W : Kernel4 oc ic kH kW) (o : Fin oc) (c : Fin ic) (kh : Fin kH) (kw : Fin kW) : convKernelMat W (w3Idx c kh kw) o = W o c kh kw

theorem Proofs.conv2d_eq_dense {ic oc h w kH kW : ℕ} (W : Kernel4 oc ic kH kW) (b : Vec oc) (x : Tensor3 ic h w) (o : Fin oc) (hi : Fin h) (wi : Fin w) : conv2d W b x o hi wi = dense (convKernelMat W) b (convWindow kH kW x hi wi) o

conv2d is a dense layer at the conv fan-in — conv = dense-with-sharing made exact: each output coordinate is Proofs.dense of the kernel slab against the flattened window. The structural fact that lets the float conv budget reuse dense_close.

theorem Proofs.convPad_close {ic h w kH kW : ℕ} (xt xa : Tensor3 ic h w) {e : ℝ} (he : 0 ≤ e) (hx : ∀ (c : Fin ic) (i : Fin h) (j : Fin w), |xt c i j - xa c i j| ≤ e) (c : Fin ic) (kh : Fin kH) (kw : Fin kW) (hi : Fin h) (wi : Fin w) : |convPad kH kW xt c kh kw hi wi - convPad kH kW xa c kh kw hi wi| ≤ e

Padded reads of inputs within e stay within e (the read is either a coordinate, diff ≤ e, or 0, diff 0).

noncomputable def Proofs.FloatModel.convF {ic oc h w kH kW : ℕ} (M : FloatModel) (W : Kernel4 oc ic kH kW) (b : Vec oc) (x : Tensor3 ic h w) : Tensor3 oc h w

The float conv layer — M.dense of the kernel slab against the flattened window, per output coordinate. The float peer of conv2d (every product/accumulate/bias-add rounded), in the dense form.

Equations

• M.convF W b x o hi wi = M.dense (Proofs.convKernelMat W) b (Proofs.convWindow kH kW x hi wi) o

theorem Proofs.FloatModel.convF_close {ic oc h w kH kW : ℕ} (M : FloatModel) (W : Kernel4 oc ic kH kW) (b : Vec oc) (xt xa : Tensor3 ic h w) {e : ℝ} (he : 0 ≤ e) (hx : ∀ (c : Fin ic) (i : Fin h) (j : Fin w), |xt c i j - xa c i j| ≤ e) (o : Fin oc) (hi : Fin h) (wi : Fin w) : |M.convF W b xt o hi wi - conv2d W b xa o hi wi| ≤ M.denseErr (convKernelMat W) b (convWindow kH kW xa hi wi) e o

Conv forward rounding budget (Item A). The rounded conv at a float input within e of the real activation is within the conv-fan-in denseErr of the real conv — dense_close at the flattened window. The compounded Higham factor rides the fan-in ic·kH·kW (the dense column length here), exactly as the planning doc calls for.

theorem Proofs.convKernelMat_abs_le {oc ic kH kW : ℕ} {W : Kernel4 oc ic kH kW} {w' : ℝ} (hW : ∀ (o : Fin oc) (c : Fin ic) (kh : Fin kH) (kw : Fin kW), |W o c kh kw| ≤ w') (i : Fin (ic * kH * kW)) (j : Fin oc) : |convKernelMat W i j| ≤ w'

Kernel-slab entries inherit the uniform kernel magnitude bound.

theorem Proofs.convWindow_abs_le {ic h w kH kW : ℕ} {x : Tensor3 ic h w} {a : ℝ} (ha : 0 ≤ a) (hx : ∀ (c : Fin ic) (i : Fin h) (j : Fin w), |x c i j| ≤ a) (hi : Fin h) (wi : Fin w) (idx : Fin (ic * kH * kW)) : |convWindow kH kW x hi wi idx| ≤ a

Window reads inherit the uniform input magnitude bound (padding reads 0).

theorem Proofs.conv2d_abs_le {ic oc h w kH kW : ℕ} {W : Kernel4 oc ic kH kW} {b : Vec oc} {x : Tensor3 ic h w} {w' β a : ℝ} (ha : 0 ≤ a) (hW : ∀ (o : Fin oc) (c : Fin ic) (kh : Fin kH) (kw : Fin kW), |W o c kh kw| ≤ w') (hb : ∀ (o : Fin oc), |b o| ≤ β) (hx : ∀ (c : Fin ic) (i : Fin h) (j : Fin w), |x c i j| ≤ a) (o : Fin oc) (hi : Fin h) (wi : Fin w) : |conv2d W b x o hi wi| ≤ FloatModel.layerAct (ic * kH * kW) w' β a

Conv output magnitude bound = dense_abs_le at the fan-in: conv is a dense layer, so |conv2dⱼ| ≤ layerAct (ic·kH·kW) w β a.

noncomputable def Proofs.FloatModel.flatConvF {ic oc h w kH kW : ℕ} (M : FloatModel) (W : Kernel4 oc ic kH kW) (b : Vec oc) : Vec (ic * h * w) → Vec (oc * h * w)

Vec-space float conv — the float peer of flatConv (flatten ∘ conv2d ∘ unflatten), with the rounded convF inside.

Equations

• M.flatConvF W b v = (M.convF W b (Proofs.Tensor3.unflatten v)).flatten

theorem Proofs.FloatModel.flatConvF_close {ic oc h w kH kW : ℕ} (M : FloatModel) (W : Kernel4 oc ic kH kW) (b : Vec oc) (vt va : Vec (ic * h * w)) {w' β a e : ℝ} (hw' : 0 ≤ w') (ha : 0 ≤ a) (he : 0 ≤ e) (hW : ∀ (o : Fin oc) (c : Fin ic) (kh : Fin kH) (kw : Fin kW), |W o c kh kw| ≤ w') (hb : ∀ (o : Fin oc), |b o| ≤ β) (hva : ∀ (k : Fin (ic * h * w)), |va k| ≤ a) (hvte : ∀ (k : Fin (ic * h * w)), |vt k - va k| ≤ e) (k : Fin (oc * h * w)) : |M.flatConvF W b vt k - flatConv W b va k| ≤ layerBudget M.u (ic * kH * kW) w' β a e

Vec-space conv forward budget, uniform. The rounded flatConvF at a float input within e of the real activation is within the conv-fan-in layerBudget of the real flatConv — every output coordinate, one closed form. The conv layer threads exactly like a dense layer at fan-in ic·kH·kW.

theorem Proofs.flatConv_abs_le {ic oc h w kH kW : ℕ} {W : Kernel4 oc ic kH kW} {b : Vec oc} {v : Vec (ic * h * w)} {w' β a : ℝ} (ha : 0 ≤ a) (hW : ∀ (o : Fin oc) (c : Fin ic) (kh : Fin kH) (kw : Fin kW), |W o c kh kw| ≤ w') (hb : ∀ (o : Fin oc), |b o| ≤ β) (hv : ∀ (k : Fin (ic * h * w)), |v k| ≤ a) (k : Fin (oc * h * w)) : |flatConv W b v k| ≤ FloatModel.layerAct (ic * kH * kW) w' β a

Vec-space conv magnitude bound (the activation-norm pass-through).

noncomputable def Proofs.FloatModel.mnistCnnNoBnForwardF {ic c h w d1 nClasses kH kW : ℕ} (M : FloatModel) (W₁ : Kernel4 c ic kH kW) (b₁ : Vec c) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d1) (b₃ : Vec d1) (W₄ : Mat d1 d1) (b₄ : Vec d1) (W₅ : Mat d1 nClasses) (b₅ : Vec nClasses) : Vec (ic * (2 * h) * (2 * w)) → Vec nClasses

The float MNIST-CNN (no BN) forward — the float peer of mnistCnnNoBnForward: rounded conv (flatConvF) and rounded dense (M.dense); relu and maxPoolFlat appear bare (exact in float).

Equations

• One or more equations did not get rendered due to their size.

theorem Proofs.FloatModel.cnn_float_close {ic c h w d1 nClasses kH kW : ℕ} (M : FloatModel) (W₁ : Kernel4 c ic kH kW) (b₁ : Vec c) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d1) (b₃ : Vec d1) (W₄ : Mat d1 d1) (b₄ : Vec d1) (W₅ : Mat d1 nClasses) (b₅ : Vec nClasses) (x : Vec (ic * (2 * h) * (2 * w))) {w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ a : ℝ} (hw₁ : 0 ≤ w₁) (hβ₁ : 0 ≤ β₁) (hw₂ : 0 ≤ w₂) (hβ₂ : 0 ≤ β₂) (hw₃ : 0 ≤ w₃) (hβ₃ : 0 ≤ β₃) (hw₄ : 0 ≤ w₄) (hβ₄ : 0 ≤ β₄) (hw₅ : 0 ≤ w₅) (ha : 0 ≤ a) (hW₁ : ∀ (o : Fin c) (cc : Fin ic) (kh : Fin kH) (kw : Fin kW), |W₁ o cc kh kw| ≤ w₁) (hb₁ : ∀ (o : Fin c), |b₁ o| ≤ β₁) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hb₂ : ∀ (o : Fin c), |b₂ o| ≤ β₂) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d1), |W₃ i j| ≤ w₃) (hb₃ : ∀ (j : Fin d1), |b₃ j| ≤ β₃) (hW₄ : ∀ (i j : Fin d1), |W₄ i j| ≤ w₄) (hb₄ : ∀ (j : Fin d1), |b₄ j| ≤ β₄) (hW₅ : ∀ (i : Fin d1) (j : Fin nClasses), |W₅ i j| ≤ w₅) (hb₅ : ∀ (j : Fin nClasses), |b₅ j| ≤ β₅) (hx : ∀ (i : Fin (ic * (2 * h) * (2 * w))), |x i| ≤ a) (k : Fin nClasses) : |M.mnistCnnNoBnForwardF W₁ b₁ W₂ b₂ W₃ b₃ W₄ b₄ W₅ b₅ x k - mnistCnnNoBnForward W₁ b₁ W₂ b₂ W₃ b₃ W₄ b₄ W₅ b₅ x k| ≤ layerBudget M.u d1 w₅ β₅ (layerAct d1 w₄ β₄ (layerAct (c * h * w) w₃ β₃ (layerAct (c * kH * kW) w₂ β₂ (layerAct (ic * kH * kW) w₁ β₁ a)))) (layerBudget M.u d1 w₄ β₄ (layerAct (c * h * w) w₃ β₃ (layerAct (c * kH * kW) w₂ β₂ (layerAct (ic * kH * kW) w₁ β₁ a))) (layerBudget M.u (c * h * w) w₃ β₃ (layerAct (c * kH * kW) w₂ β₂ (layerAct (ic * kH * kW) w₁ β₁ a)) (layerBudget M.u (c * kH * kW) w₂ β₂ (layerAct (ic * kH * kW) w₁ β₁ a) (layerBudget M.u (ic * kH * kW) w₁ β₁ a 0))))

Whole-net MNIST-CNN forward rounding budget (Item A capstone). The rounded forward is within an explicit closed-form layerBudget of the real conv→relu→conv→relu→maxpool→dense→relu→dense→relu→dense forward, per output logit — the binary32 forward-error bound for the Chapter-3 CNN.

Each weight layer threads identically: conv layers as dense at their fan-in (ic·kH·kW, then c·kH·kW), the dense head at c·h·w / d1; relu and maxpool pass error through exactly (no rounding, no amplification). The budget is the mlp_float_close_uniform nest extended to the CNN's six layers — norm_num-evaluable at a concrete net and magnitude profile.

theorem Proofs.conv2d_kernel_sub {ic oc h w kH kW : ℕ} (b : Vec oc) (x : Tensor3 ic h w) (v e : Vec (oc * ic * kH * kW)) (o : Fin oc) (hi : Fin h) (wi : Fin w) : conv2d (Kernel4.unflatten (v + e)) b x o hi wi - conv2d (Kernel4.unflatten v) b x o hi wi = ∑ c : Fin ic, ∑ kh : Fin kH, ∑ kw : Fin kW, e (k4Idx o c kh kw) * convPad kH kW x c kh kw hi wi

theorem Proofs.conv2d_kernel_drift {ic oc h w kH kW : ℕ} (b : Vec oc) (x : Tensor3 ic h w) {a : ℝ} (ha : 0 ≤ a) (hx : ∀ (c : Fin ic) (i : Fin h) (j : Fin w), |x c i j| ≤ a) (v e : Vec (oc * ic * kH * kW)) (o : Fin oc) (hi : Fin h) (wi : Fin w) : |conv2d (Kernel4.unflatten (v + e)) b x o hi wi - conv2d (Kernel4.unflatten v) b x o hi wi| ≤ a * ∑ c : Fin ic, ∑ kh : Fin kH, ∑ kw : Fin kW, |e (k4Idx o c kh kw)|

Per-entry conv drift, slab-refined: a kernel perturbation moves the output entry (o, hi, wi) by at most a times the ℓ1 mass of the channel-o slab (each output reads only its own slab).

theorem Proofs.conv2d_kernel_drift_total {ic oc h w kH kW : ℕ} (b : Vec oc) (x : Tensor3 ic h w) {a : ℝ} (ha : 0 ≤ a) (hx : ∀ (c : Fin ic) (i : Fin h) (j : Fin w), |x c i j| ≤ a) (v e : Vec (oc * ic * kH * kW)) (o : Fin oc) (hi : Fin h) (wi : Fin w) : |conv2d (Kernel4.unflatten (v + e)) b x o hi wi - conv2d (Kernel4.unflatten v) b x o hi wi| ≤ a * ∑ idx : Fin (oc * ic * kH * kW), |e idx|

Per-entry conv drift against the TOTAL ℓ1 mass — the form the relu margins consume.

theorem Proofs.conv2d_kernel_drift_sum {ic oc h w kH kW : ℕ} (b : Vec oc) (x : Tensor3 ic h w) {a : ℝ} (ha : 0 ≤ a) (hx : ∀ (c : Fin ic) (i : Fin h) (j : Fin w), |x c i j| ≤ a) (v e : Vec (oc * ic * kH * kW)) : ∑ o : Fin oc, ∑ hi : Fin h, ∑ wi : Fin w, |conv2d (Kernel4.unflatten (v + e)) b x o hi wi - conv2d (Kernel4.unflatten v) b x o hi wi| ≤ ↑(h * w) * (a * ∑ idx : Fin (oc * ic * kH * kW), |e idx|)

ℓ1 conv drift: summed over all output entries, the drift is at most (h·w)·a·‖e‖₁ — the spatial multiplicity h·w is the price of weight sharing (each kernel entry touches every spatial position).

theorem Proofs.MaxPool2MarginQ.pdiv3_eq {c h w : ℕ} {δ : ℝ} (hδ0 : 0 ≤ δ) {x y : Tensor3 c (2 * h) (2 * w)} (hm : MaxPool2MarginQ δ x) (hclose : ∀ (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)), |y ci hi wi - x ci hi wi| ≤ δ) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) (co : Fin c) (ho : Fin h) (wo : Fin w) : pdiv3 maxPool2 y ci hi wi co ho wo = pdiv3 maxPool2 x ci hi wi co ho wo

The pool's routing pattern is frozen: under the margin, the pdiv3 Jacobian of the pool is entry-for-entry IDENTICAL at the margined point and at any δ-close point. This is what lets the pool behave as a fixed linear selector along the whole step segment.

theorem Proofs.MaxPool2MarginQ.poolBack_close {c h w : ℕ} {δ : ℝ} {x y : Tensor3 c (2 * h) (2 * w)} (hm : MaxPool2MarginQ δ x) (hclose : ∀ (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)), |y ci hi wi - x ci hi wi| ≤ δ) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) {ay ax e : ℝ} (ha : |ay - ax| ≤ e) : |(if MaxPool2IsArgmax y ci hi wi then ay else 0) - if MaxPool2IsArgmax x ci hi wi then ax else 0| ≤ e

Float pool-backward closeness (Increment 1 keystone). Under the pool margin the float post-relu argmax matches the real one (isArgmax_iff), so the pool's backward selector 𝟙[(ci,hi,wi) is its window's argmax]·(pooled cotangent) differs from the certified one only through the pooled cotangent value — an indicator pass-through (indicator ∈ {0,1}), the pool peer of reluMask_close. The two cotangent values ay (float) / ax (real) enter only via their closeness |ay − ax| ≤ e.

theorem Proofs.t3Idx_def {c h w : ℕ} (ci : Fin c) (hi : Fin h) (wi : Fin w) : finProdFinEquiv (finProdFinEquiv (ci, hi), wi) = t3Idx ci hi wi

Folds the raw finProdFinEquiv encoding back into t3Idx.

theorem Proofs.t3Idx_inj {c h w : ℕ} {ci ci' : Fin c} {hi hi' : Fin h} {wi wi' : Fin w} (hEq : t3Idx ci hi wi = t3Idx ci' hi' wi') : ci = ci' ∧ hi = hi' ∧ wi = wi'

t3Idx is injective componentwise — the spatial peer of k4Idx_inj.

theorem Proofs.ce_head3_differentiableAt {p d₃ d₄ nC : ℕ} (W₃ : Mat p d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (u : Vec p) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ u l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ u)) q ≠ 0) : DifferentiableAt ℝ (fun (y : Vec p) (x : Fin 1) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ y))))) label) u

The 3-dense head CE ∘ d₅ ∘ relu ∘ d₄ ∘ relu ∘ d₃ is differentiable at any point whose two ReLU pre-activations are off the kinks.

theorem Proofs.ce_head3_input_grad {p d₃ d₄ nC : ℕ} (W₃ : Mat p d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (u : Vec p) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ u l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ u)) q ≠ 0) (j : Fin p) : pdiv (fun (y : Vec p) (x : Fin 1) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ y))))) label) u j 0 = ∑ l : Fin d₃, W₃ j l * ((if dense W₃ b₃ u l > 0 then 1 else 0) * ∑ q : Fin d₄, W₄ l q * ((if dense W₄ b₄ (relu d₃ (dense W₃ b₃ u)) q > 0 then 1 else 0) * ∑ k : Fin nC, W₅ q k * (softmax nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ u))))) k - oneHot nC label k)))

Loss input-gradient of the 3-dense head CE∘d₅∘relu∘d₄∘relu∘d₃ at the pooled vector — one pdiv_comp hop (peel dense W₃) on top of ce_head2_input_grad, exactly as ce_head2 was one hop on ce_head_relu. Note there is NO leading mask: the pool output feeds dense W₃ directly.

theorem Proofs.pool_head_differentiableAt {c h w d₃ d₄ nC : ℕ} (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hc : 0 < c) (hh : 0 < h) (hw : 0 < w) (z₂ : Vec (c * (2 * h) * (2 * w))) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), z₂ k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₂))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) z₂)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) z₂)))) q ≠ 0) : DifferentiableAt ℝ (fun (y : Vec (c * (2 * h) * (2 * w))) (x : Fin 1) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) y))))))) label) z₂

The whole head above the conv output — CE∘head3∘maxPoolFlat∘relu — is differentiable at any point with the relu₂ pre-activation off the kinks, no pool ties (POST-relu), and the two head masks off the kinks.

theorem Proofs.pool_relu_input_grad {c h w d₃ d₄ nC : ℕ} (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (z₂ : Vec (c * (2 * h) * (2 * w))) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), z₂ k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₂))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) z₂)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) z₂)))) q ≠ 0) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) : pdiv (fun (y : Vec (c * (2 * h) * (2 * w))) (x : Fin 1) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) y))))))) label) z₂ (t3Idx ci hi wi) 0 = (if z₂ (t3Idx ci hi wi) > 0 then 1 else 0) * if MaxPool2IsArgmax (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₂)) ci hi wi then ∑ l : Fin d₃, W₃ (t3Idx ci (winRow hi) (winCol wi)) l * ((if dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) z₂)) l > 0 then 1 else 0) * ∑ q : Fin d₄, W₄ l q * ((if dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) z₂)))) q > 0 then 1 else 0) * ∑ k : Fin nC, W₅ q k * (softmax nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) z₂))))))) k - oneHot nC label k))) else 0

Loss input-gradient at the conv output — the key glue of the conv rung. The chain pdivs through the relu (mask) and the pool (frozen selector): at a smooth point the sum over pooled coordinates collapses to the single argmax term, so

∂(CE∘head3∘pool∘relu)/∂z₂[ci,hi,wi] = relu'(z₂[ci,hi,wi]) · 𝟙[(ci,hi,wi) is its window's argmax] · head3grad(window(ci,hi,wi)).

NB the pool acts on the POST-relu activation, so the smoothness and argmax conditions are stated on relu z₂, not z₂.

theorem Proofs.conv2d_weight_pdiv {ic oc h w kH kW : ℕ} (b : Vec oc) (x : Tensor3 ic h w) (v : Vec (oc * ic * kH * kW)) (o : Fin oc) (cc : Fin ic) (kh : Fin kH) (kw : Fin kW) (co : Fin oc) (hi : Fin h) (wi : Fin w) : pdiv (fun (v' : Vec (oc * ic * kH * kW)) => (conv2d (Kernel4.unflatten v') b x).flatten) v (k4Idx o cc kh kw) (t3Idx co hi wi) = if co = o then convPad kH kW x cc kh kw hi wi else 0

Closed form of the conv weight-map pdiv — extracted from the certified VJP (conv2d_weight_grad_has_vjp) by contracting its .correct field against a basis vector. Kernel entry (o,cc,kh,kw) touches output (co,hi,wi) iff co = o, with coefficient the padded input read convPad. NB the right-hand side does not mention v: the weight map is affine, so its Jacobian is point-free — this is what lets the gradient difference along a step segment collapse to the head drift alone.

theorem Proofs.conv2d_weight_pdiv_row_l1 {ic oc h w kH kW : ℕ} (b : Vec oc) (x : Tensor3 ic h w) {a : ℝ} (ha : 0 ≤ a) (hx : ∀ (c : Fin ic) (i : Fin h) (j : Fin w), |x c i j| ≤ a) (v : Vec (oc * ic * kH * kW)) (o : Fin oc) (cc : Fin ic) (kh : Fin kH) (kw : Fin kW) : ∑ k : Fin (oc * h * w), |pdiv (fun (v' : Vec (oc * ic * kH * kW)) => (conv2d (Kernel4.unflatten v') b x).flatten) v (k4Idx o cc kh kw) k| ≤ ↑(h * w) * a

The ℓ1 mass of one Jacobian row of the conv weight map: kernel entry (o,cc,kh,kw) touches the (h·w) outputs of its slab, each with a padded read bounded by a — the quantitative form of "weight sharing costs a spatial multiplicity".

theorem Proofs.sum_s2 {h w : ℕ} (g : Fin (h * w) → ℝ) : ∑ s : Fin (h * w), g s = ∑ hi : Fin h, ∑ wi : Fin w, g (finProdFinEquiv (hi, wi))

The spatial (hi, wi) sum collapses to one flat sum over Fin (h·w).

noncomputable def Proofs.convPadWin {ic h w : ℕ} (kH kW : ℕ) (x : Tensor3 ic h w) (cc : Fin ic) (kh : Fin kH) (kw : Fin kW) : Vec (h * w)

The padded-input window for a fixed kernel slot, flattened over the (hi, wi) spatial grid — the left operand of the conv weight-grad dot.

Equations

• Proofs.convPadWin kH kW x cc kh kw s = Proofs.convPad kH kW x cc kh kw (finProdFinEquiv.symm s).1 (finProdFinEquiv.symm s).2

noncomputable def Proofs.cotWin {oc h w : ℕ} (cot : Tensor3 oc h w) (o : Fin oc) : Vec (h * w)

The cotangent slab for a fixed output channel, flattened over (hi, wi).

Equations

• Proofs.cotWin cot o s = cot o (finProdFinEquiv.symm s).1 (finProdFinEquiv.symm s).2

@[simp]

theorem Proofs.convPadWin_apply {ic h w : ℕ} (kH kW : ℕ) (x : Tensor3 ic h w) (cc : Fin ic) (kh : Fin kH) (kw : Fin kW) (hi : Fin h) (wi : Fin w) : convPadWin kH kW x cc kh kw (finProdFinEquiv (hi, wi)) = convPad kH kW x cc kh kw hi wi

@[simp]

theorem Proofs.cotWin_apply {oc h w : ℕ} (cot : Tensor3 oc h w) (o : Fin oc) (hi : Fin h) (wi : Fin w) : cotWin cot o (finProdFinEquiv (hi, wi)) = cot o hi wi

theorem Proofs.convWeightGrad_eq_dot {ic oc h w kH kW : ℕ} (x : Tensor3 ic h w) (cot : Tensor3 oc h w) (o : Fin oc) (cc : Fin ic) (kh : Fin kH) (kw : Fin kW) : ∑ s : Fin (h * w), convPadWin kH kW x cc kh kw s * cotWin cot o s = ∑ hi : Fin h, ∑ wi : Fin w, convPad kH kW x cc kh kw hi wi * cot o hi wi

The conv weight gradient is the spatial dot Σ_{hi,wi} convPad · cot (the contraction conv2d_weight_pdiv certifies as ∂L/∂W_{o,cc,kh,kw}), re-expressed as a flat Fin (h·w) dot of the padded-input window against the cotangent slab — the form the float dot rounds.

theorem Proofs.convBiasGrad_eq_sum {oc h w : ℕ} (cot : Tensor3 oc h w) (o : Fin oc) : ∑ s : Fin (h * w), cotWin cot o s = ∑ hi : Fin h, ∑ wi : Fin w, cot o hi wi

The conv bias gradient is the spatial sum Σ_{hi,wi} cot.

theorem Proofs.FloatModel.cnn_convW_step_float_close {ic oc h w kH kW : ℕ} (M : FloatModel) (W : Kernel4 oc ic kH kW) (x : Tensor3 ic h w) (cot : Tensor3 oc h w) {lr G : ℝ} (o : Fin oc) (cc : Fin ic) (kh : Fin kH) (kw : Fin kW) (hG : |∑ s : Fin (h * w), convPadWin kH kW x cc kh kw s * cotWin cot o s| ≤ G) (hlr : 0 ≤ lr) : |M.sub (W o cc kh kw) (M.mul lr (M.dot (convPadWin kH kW x cc kh kw) (cotWin cot o))) - (W o cc kh kw - lr * ∑ s : Fin (h * w), convPadWin kH kW x cc kh kw s * cotWin cot o s)| ≤ sgdErr M.u lr |W o cc kh kw| G (((1 + M.u) ^ (h * w + 1) - 1) * ∑ s : Fin (h * w), |convPadWin kH kW x cc kh kw s * cotWin cot o s|)

Rounded conv weight update (Item B). The float update fl(Wₒ,cc,kh,kw − fl(lr·fl(convPadWin · cotWin))) — the conv weight gradient is a correlation, a dot over the h·w spatial positions — is within sgdErr of the real step W − lr·(Σ_{hi,wi} convPad·cot), the dot's Higham γ (fan-in h·w) as the gradient-error slot. Reuses the generic dotSgd_step_close; the cotangent is supplied (the loss-head exp accuracy lives in cotErr).

theorem Proofs.FloatModel.cnn_convb_step_float_close {oc h w : ℕ} (M : FloatModel) (b : Vec oc) (cot : Tensor3 oc h w) {lr G : ℝ} (o : Fin oc) (hG : |∑ s : Fin (h * w), cotWin cot o s| ≤ G) (hlr : 0 ≤ lr) : |M.sub (b o) (M.mul lr (M.sum (cotWin cot o))) - (b o - lr * ∑ s : Fin (h * w), cotWin cot o s)| ≤ sgdErr M.u lr |b o| G (((1 + M.u) ^ (h * w + 1) - 1) * ∑ s : Fin (h * w), |cotWin cot o s|)

Rounded conv bias update (Item B) — the bias gradient is the spatial sum Σ cot, so the rounded update reduces to sumSgd_step_close.

theorem Proofs.FloatModel.mnist_cnn_convW_step_float_budget (M : FloatModel) (hMu : M.u ≤ u32) (W : Kernel4 32 32 3 3) (act cot : Tensor3 32 28 28) {a g : ℝ} (ha : 0 ≤ a) (hg : 0 ≤ g) (hW : ∀ (o cc : Fin 32) (kh kw : Fin 3), |W o cc kh kw| ≤ 3 / 5) (hact : ∀ (c : Fin 32) (i j : Fin 28), |act c i j| ≤ a) (hcot : ∀ (o : Fin 32) (i j : Fin 28), |cot o i j| ≤ g) (o cc : Fin 32) (kh kw : Fin 3) : |M.sub (W o cc kh kw) (M.mul (1 / 10) (M.dot (convPadWin 3 3 act cc kh kw) (cotWin cot o))) - (W o cc kh kw - 1 / 10 * ∑ s : Fin (28 * 28), convPadWin 3 3 act cc kh kw s * cotWin cot o s)| ≤ a * g / 250 + 1 / 10000000

Numeric conv-weight-step capstone at the committed MNIST-CNN dims (Item C). The Chapter-3 conv2 is 32→32, 3×3, at 28×28 (the conv output grid, before maxpool), so the weight gradient is a dot over 28·28 = 784 spatial positions. At binary32 (u ≤ 2⁻²⁴), lr = 1/10, kernel |W| ≤ 3/5 (the trained-magnitude bound, matching the MLP capstone), every rounded conv2 weight SGD entry is within (a·g)/250 + 10⁻⁷ of the certified real step — where a bounds the conv2-input activation and g the conv2 cotangent magnitude.

Both a and g are a-posteriori / measured quantities (the conv input and back-propagated cotangent are not intrinsically ≤ 1, unlike the softmax−onehot loss head), supplied as hypotheses — the same worst-case→ measured hand-off as the forward δ. The decimal rate 1/250 ≈ 0.4% is dominated by lr·γ₇₈₅ (the gradient's Higham error at learning-rate scale): the conv weight step is as accurate as the gradient itself, no worse.

theorem Proofs.FloatModel.mnist_cnn_convb_step_float_budget (M : FloatModel) (hMu : M.u ≤ u32) (b : Vec 32) (cot : Tensor3 32 28 28) {g : ℝ} (hg : 0 ≤ g) (hb : ∀ (o : Fin 32), |b o| ≤ 3 / 5) (hcot : ∀ (o : Fin 32) (i j : Fin 28), |cot o i j| ≤ g) (o : Fin 32) : |M.sub (b o) (M.mul (1 / 10) (M.sum (cotWin cot o))) - (b o - 1 / 10 * ∑ s : Fin (28 * 28), cotWin cot o s)| ≤ g / 250 + 1 / 10000000

Numeric conv-BIAS-step capstone at the committed MNIST-CNN dims — the bias peer of mnist_cnn_convW_step_float_budget. The conv2 bias gradient is the spatial SUM ∑ₛ cotWin cot o over the 28·28 = 784 conv-output positions (no input window, so no a factor — the bias Jacobian is the bare channel indicator). At binary32 (u ≤ 2⁻²⁴), lr = 1/10, |b| ≤ 3/5, every rounded conv2 bias SGD entry is within g/250 + 10⁻⁷ of the certified real step — where g bounds the conv2 cotangent magnitude. The same 1/250 rate as the weight step (it is lr·γ₇₈₅, the gradient's Higham error at learning-rate scale), with a·g ↦ g — the bias step is as accurate as the gradient, no worse.

theorem Proofs.cnn_conv2_loss_differentiableAt {c h w d₃ d₄ nC kH kW : ℕ} (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hc : 0 < c) (hh : 0 < h) (hw : 0 < w) (v : Vec (c * c * kH * kW)) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d (Kernel4.unflatten v) b₂ x₁).flatten k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)))) q ≠ 0) : DifferentiableAt ℝ (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) v

The loss-of-conv2-kernel map is differentiable wherever the relu₂ pre-activation is off the kinks, no pool window ties (POST-relu), and the two head pre-activations are off the kinks.

theorem Proofs.cnn_conv2_loss_gradAt {c h w d₃ d₄ nC kH kW : ℕ} (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hh : 0 < h) (hw : 0 < w) (v : Vec (c * c * kH * kW)) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d (Kernel4.unflatten v) b₂ x₁).flatten k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)))) q ≠ 0) (o cc : Fin c) (kh : Fin kH) (kw : Fin kW) : gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) v (k4Idx o cc kh kw) = ∑ ci : Fin c, ∑ hi : Fin (2 * h), ∑ wi : Fin (2 * w), (if ci = o then convPad kH kW x₁ cc kh kw hi wi else 0) * ((if (conv2d (Kernel4.unflatten v) b₂ x₁).flatten (t3Idx ci hi wi) > 0 then 1 else 0) * if MaxPool2IsArgmax (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)) ci hi wi then ∑ l : Fin d₃, W₃ (t3Idx ci (winRow hi) (winCol wi)) l * ((if dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)) l > 0 then 1 else 0) * ∑ q : Fin d₄, W₄ l q * ((if dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)))) q > 0 then 1 else 0) * ∑ k : Fin nC, W₅ q k * (softmax nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten))))))) k - oneHot nC label k))) else 0)

Closed form of the conv2 loss gradient at any four-margin point — the EXISTING fold conv_total_loss_grad_fold (generic in the downstream G) contracted with the pool-collapsed head gradient (pool_relu_input_grad) and the point-free conv weight Jacobian (conv2d_weight_pdiv). The conv-layer peer of mlp_input_loss_gradAt; the spatial triple sum (vs the MLP's Kronecker collapse) is weight sharing.

theorem Proofs.dense_transpose_eq {p n : ℕ} (W : Mat p n) (c : Vec n) (l : Fin p) : ∑ k : Fin n, W l k * c k = dense (fun (j : Fin n) (i' : Fin p) => W i' j) (fun (x : Fin p) => 0) c l

The unmasked peer of reluMask_dense_transpose_eq: a bare Wᵀ contraction ∑ₖ Wₗₖ·cₖ = dense (transpose W) 0 c l. The pool feeds dense W₃ with no leading ReLU mask, so the W₃ contraction in the certified conv-2 gradient collapses through this, where the masked W₄/W₅ contractions collapse through reluMask_dense_transpose_eq. NB this is generic in (W, c), so fire it only where the goal has no other matrix contraction (e.g. the spatial ∑ convPad·cot) — see head3_cot_reluMask.

theorem Proofs.head3_cot_reluMask {p d₃ d₄ nC : ℕ} (W₃ : Mat p d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (u : Vec p) (j : Fin p) : ∑ l : Fin d₃, W₃ j l * ((if dense W₃ b₃ u l > 0 then 1 else 0) * ∑ q : Fin d₄, W₄ l q * ((if dense W₄ b₄ (relu d₃ (dense W₃ b₃ u)) q > 0 then 1 else 0) * ∑ k : Fin nC, W₅ q k * (softmax nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ u))))) k - oneHot nC label k))) = dense (fun (j' : Fin d₃) (i' : Fin p) => W₃ i' j') (fun (x : Fin p) => 0) (FloatModel.reluMask (dense W₃ b₃ u) (dense (fun (j' : Fin d₄) (i' : Fin d₃) => W₄ i' j') (fun (x : Fin d₃) => 0) (FloatModel.reluMask (dense W₄ b₄ (relu d₃ (dense W₃ b₃ u))) (dense (fun (j' : Fin nC) (i' : Fin d₄) => W₅ i' j') (fun (x : Fin d₄) => 0) fun (k : Fin nC) => softmax nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ u))))) k - oneHot nC label k)))) j

The 3-dense head cotangent in dense/reluMask form. The raw nested ∑ₗ W₃·(𝟙[z₃]·∑_q W₄·(𝟙[z₄]·∑_k W₅·(softmax−onehot))) that pool_relu_input_grad / cnn_conv2_loss_gradAt leave at the pooled vector u equals dense W₃ᵀ 0 (mask z₃ (dense W₄ᵀ 0 (mask z₄ (dense W₅ᵀ 0 (softmax−onehot))))) — the two masked contractions via reluMask_dense_transpose_eq, the unmasked W₃ via dense_transpose_eq. Stated head-locally (no spatial sum) so the generic dense_transpose_eq fires only on the W₃ row. The head peer the conv grad-close bounds against via dense_close (W₃) and cot_step_close (W₄/W₅).

theorem Proofs.cnn_conv2_loss_gradAt_reluMask {c h w d₃ d₄ nC kH kW : ℕ} (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hh : 0 < h) (hw : 0 < w) (v : Vec (c * c * kH * kW)) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d (Kernel4.unflatten v) b₂ x₁).flatten k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)))) q ≠ 0) (o cc : Fin c) (kh : Fin kH) (kw : Fin kW) : gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) v (k4Idx o cc kh kw) = ∑ s : Fin (2 * h * (2 * w)), convPadWin kH kW x₁ cc kh kw s * cotWin (fun (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) => (if (conv2d (Kernel4.unflatten v) b₂ x₁).flatten (t3Idx ci hi wi) > 0 then 1 else 0) * if MaxPool2IsArgmax (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)) ci hi wi then dense (fun (j : Fin d₃) (i' : Fin (c * h * w)) => W₃ i' j) (fun (x : Fin (c * h * w)) => 0) (FloatModel.reluMask (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten))) (dense (fun (j : Fin d₄) (i' : Fin d₃) => W₄ i' j) (fun (x : Fin d₃) => 0) (FloatModel.reluMask (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten))))) (dense (fun (j : Fin nC) (i' : Fin d₄) => W₅ i' j) (fun (x : Fin d₄) => 0) fun (k : Fin nC) => softmax nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten))))))) k - oneHot nC label k)))) (t3Idx ci (winRow hi) (winCol wi)) else 0) o s

The certified conv-2 loss gradient, head restated in dense/reluMask form — the conv peer of mlp_input_loss_gradAt_reluMask (Increment 1 keystone). The two head Wᵀ contractions (under the d₄/d₃ ReLU masks) collapse via reluMask_dense_transpose_eq, the unmasked W₃ contraction via dense_transpose_eq; the conv-output ReLU mask 𝟙[z₂>0] and the pool argmax selector are kept explicit (their float closeness is handled by reluMask_close and MaxPool2MarginQ.poolBack_close). The whole conv gradient is then packaged as the spatial dot ∑ₛ convPadWin·cotWin (convWeightGrad_eq_dot) — the exact quantity the rendered trainer's float conv-weight dot rounds, so the conv grad-close bounds against this.

theorem Proofs.mask_scalar_close {zt z xt x ez ex : ℝ} (hz : |zt - z| ≤ ez) (hm : ez < |z|) (hx : |xt - x| ≤ ex) (hex : 0 ≤ ex) : |(if zt > 0 then 1 else 0) * xt - (if z > 0 then 1 else 0) * x| ≤ ex

Scalar ReLU-mask freeze — the (if z>0 then 1 else 0)·x peer of reluMask_close. Under the sign margin ez < |z| the float and real masks agree, so the masked value is 1-Lipschitz in x. The conv-output ReLU mask 𝟙[z₂>0] in the conv-2 grad-close sits on a scalar cell (not a Vec), so it needs this rather than the vector reluMask_close.

theorem Proofs.FloatModel.dot_perturbed_close {n : ℕ} (M : FloatModel) (A Bt B : Vec n) {a Ct eB : ℝ} (ha : 0 ≤ a) (hA : ∀ (i : Fin n), |A i| ≤ a) (hBt : ∀ (i : Fin n), |Bt i| ≤ Ct) (hB : ∀ (i : Fin n), |Bt i - B i| ≤ eB) : |M.dot A Bt - ∑ i : Fin n, A i * B i| ≤ ((1 + M.u) ^ (n + 1) - 1) * (↑n * (a * Ct)) + ↑n * (a * eB)

Float dot against a perturbed cotangent — the conv-2 grad-close's final contraction (the conv peer of the MLP's scalar mul_close: a dot, because of weight sharing). M.dot A B̃ (exact left operand A, float cotangent B̃) vs the certified ∑ Aᵢ·Bᵢ splits into the Higham dot rounding on A·B̃ (dot_close, fan-in n) plus the per-entry cotangent drift |B̃ᵢ − Bᵢ| ≤ eB. With |Aᵢ| ≤ a and |B̃ᵢ| ≤ Ct, the bound is closed-form and norm_num-evaluable.

noncomputable def Proofs.FloatModel.cnnConv2FloatGrad {c h w d₃ d₄ nC kH kW : ℕ} (M : FloatModel) (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (fexp : ℝ → ℝ) (label : Fin nC) (v : Vec (c * c * kH * kW)) : Vec (c * c * kH * kW)

The binary32 conv-2 weight gradient the rendered trainer computes — the conv peer of mlpInputFloatGrad. At kernel entry (o,cc,kh,kw) it is the float dot of the (exact) padded-input window convPadWin against the float conv-output cotangent slab cotWin c̃Conv o, where the float cotangent c̃Conv rounds every step of the backward — conv-output ReLU mask 𝟙[z̃₂>0], pool argmax selector (read on the FLOAT post-relu), and the head W₃ᵀ·mask(z̃₃)·W₄ᵀ·mask(z̃₄)·W₅ᵀ·(float softmax−onehot) at the float pre-activations. The M-free reluMask/maxPoolFlat/relu are exact in float; M.convF/M.dense/M.softmaxCECotF carry the rounding.

Equations

• One or more equations did not get rendered due to their size.

@[simp]

theorem Proofs.FloatModel.cnnConv2FloatGrad_apply {c h w d₃ d₄ nC kH kW : ℕ} (M : FloatModel) (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (fexp : ℝ → ℝ) (label : Fin nC) (v : Vec (c * c * kH * kW)) (o cc : Fin c) (kh : Fin kH) (kw : Fin kW) : M.cnnConv2FloatGrad b₂ x₁ W₃ b₃ W₄ b₄ W₅ b₅ fexp label v (k4Idx o cc kh kw) = M.dot (convPadWin kH kW x₁ cc kh kw) (cotWin (fun (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) => (if (M.convF (Kernel4.unflatten v) b₂ x₁).flatten (t3Idx ci hi wi) > 0 then 1 else 0) * if MaxPool2IsArgmax (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (M.convF (Kernel4.unflatten v) b₂ x₁).flatten)) ci hi wi then M.dense (fun (j : Fin d₃) (i' : Fin (c * h * w)) => W₃ i' j) (fun (x : Fin (c * h * w)) => 0) (reluMask (M.dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (M.convF (Kernel4.unflatten v) b₂ x₁).flatten))) (M.dense (fun (j : Fin d₄) (i' : Fin d₃) => W₄ i' j) (fun (x : Fin d₃) => 0) (reluMask (M.dense W₄ b₄ (relu d₃ (M.dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (M.convF (Kernel4.unflatten v) b₂ x₁).flatten))))) (M.dense (fun (j : Fin nC) (i' : Fin d₄) => W₅ i' j) (fun (x : Fin d₄) => 0) (M.softmaxCECotF fexp (M.dense W₅ b₅ (relu d₄ (M.dense W₄ b₄ (relu d₃ (M.dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (M.convF (Kernel4.unflatten v) b₂ x₁).flatten))))))) label))))) (t3Idx ci (winRow hi) (winCol wi)) else 0) o)

noncomputable def Proofs.FloatModel.cnnConv2GradBudget (M : FloatModel) (c h w d₃ d₄ nC kH kW : ℕ) (a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp : ℝ) : ℝ

The conv-2 float-backward grad-close budget — the closed-form η the rounded W₂ gradient stays within of the certified one. Bottom-up: the forward rounding nest (Econv → E₃ → E₄ → δlogit, conv at fan-in c·kH·kW, the dense head at c·h·w / d₃) feeds the head cotErr; the backward then rides two cot_step layerBudgets (W₅/W₄) and the unmasked W₃ layerBudget to econv; finally the spatial dot (fan-in (2h)·(2w)) contributes its Higham γ on the float-cotangent magnitude Ctilde plus the per-entry cotangent drift econv. The conv peer of the MLP's mulErr/layerBudget/cotErr nest, deeper by the pool + the dot.

Equations

• One or more equations did not get rendered due to their size.

theorem Proofs.cnn_conv2_grad_close {c h w d₃ d₄ nC kH kW : ℕ} (M : FloatModel) (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (fexp : ℝ → ℝ) (v : Vec (c * c * kH * kW)) {a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp : ℝ} (hh : 0 < h) (hw : 0 < w) (ha : 0 ≤ a) (hw₂ : 0 ≤ w₂) (hβ₂ : 0 ≤ β₂) (hw₃ : 0 ≤ w₃) (hβ₃ : 0 ≤ β₃) (hw₄ : 0 ≤ w₄) (hβ₄ : 0 ≤ β₄) (hw₅ : 0 ≤ w₅) (hβ₅ : 0 ≤ β₅) (heexp0 : 0 ≤ eexp) (heexp1 : eexp ≤ 1) (hfexp : ∀ (t : ℝ), |fexp t - Real.exp t| ≤ eexp * Real.exp t) (hρ1 : FloatModel.smRho M.u eexp nC < 1) (hx₁ : ∀ (ci : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₁ ci i j| ≤ a) (hv2 : ∀ (idx : Fin (c * c * kH * kW)), |v idx| ≤ w₂) (hb₂ : ∀ (o : Fin c), |b₂ o| ≤ β₂) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hb₃ : ∀ (j : Fin d₃), |b₃ j| ≤ β₃) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hb₄ : ∀ (j : Fin d₄), |b₄ j| ≤ β₄) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (hb₅ : ∀ (j : Fin nC), |b₅ j| ≤ β₅) (hmarginConv : ∀ (k : Fin (c * (2 * h) * (2 * w))), FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0 < |(conv2d (Kernel4.unflatten v) b₂ x₁).flatten k|) (hmarginPool : MaxPool2MarginQ (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten))) (hmargin3 : ∀ (l : Fin d₃), FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ a) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)) l|) (hmargin4 : ∀ (q : Fin d₄), FloatModel.layerBudget M.u d₃ w₄ β₄ (FloatModel.layerAct (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ a)) (FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ a) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0)) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)))) q|) (o cc : Fin c) (kh : Fin kH) (kw : Fin kW) : |M.cnnConv2FloatGrad b₂ x₁ W₃ b₃ W₄ b₄ W₅ b₅ fexp label v (k4Idx o cc kh kw) - gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) v (k4Idx o cc kh kw)| ≤ M.cnnConv2GradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp

The binary32 conv-2 weight gradient is within an explicit budget of the certified one (Increment 2 capstone) — the conv-layer peer of mlp_w0_grad_close, the project's deepest float-backward grad-close. With the conv-2 input x₁ exact, the rendered trainer's W₂ gradient M.cnnConv2FloatGrad … stays within cnnConv2GradBudget of the certified gradAt. The chain: float forward (convF_close → dense_close×3, relu and pool error-transparent) ⟶ head (softmax_ce_cot_close) ⟶ two masked Wᵀ cot_step_close (W₅ under z̃₄, W₄ under z̃₃) ⟶ unmasked W₃ dense_close ⟶ pool-backward freeze (poolBack_close) ⟶ conv-output ReLU mask freeze (mask_scalar_close) ⟶ the spatial dot (dot_perturbed_close). Four quantitative margins are carried (conv-output Econv, pool Econv POST-relu, z̃₃ E₃, z̃₄ E₄); the bridge cnn_conv2_loss_gradAt_reluMask turns the gradAt into the dot the float gradient rounds.

theorem Proofs.smul_l1_mass {n : ℕ} (e : Vec n) {t : ℝ} (ht0 : 0 ≤ t) : ∑ idx : Fin n, |(t • e) idx| = t * ∑ idx : Fin n, |e idx|

The ℓ1 mass of a scaled step.

theorem Proofs.smul_l1_mass_le {n : ℕ} (e : Vec n) {t D : ℝ} (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (he : ∑ idx : Fin n, |e idx| ≤ D) : ∑ idx : Fin n, |(t • e) idx| ≤ D

A t-scaled step stays inside the step radius for t ∈ [0,1].

theorem Proofs.dense_input_drift {m n : ℕ} (W : Mat m n) (b : Vec n) {wb : ℝ} (hW : ∀ (i : Fin m) (j : Fin n), |W i j| ≤ wb) (u u' : Vec m) (j : Fin n) : |dense W b u' j - dense W b u j| ≤ wb * ∑ i : Fin m, |u' i - u i|

A dense layer's output moves by at most w·‖Δinput‖₁ per entry — the ℓ1→ℓ∞ operator bound used at every dense crossing of the chain.

theorem Proofs.conv2d_flat_kernel_drift_total {ic oc h w kH kW : ℕ} (b : Vec oc) (x : Tensor3 ic h w) {a : ℝ} (ha : 0 ≤ a) (hx : ∀ (c : Fin ic) (i : Fin h) (j : Fin w), |x c i j| ≤ a) (v e : Vec (oc * ic * kH * kW)) (k : Fin (oc * h * w)) : |(conv2d (Kernel4.unflatten (v + e)) b x).flatten k - (conv2d (Kernel4.unflatten v) b x).flatten k| ≤ a * ∑ idx : Fin (oc * ic * kH * kW), |e idx|

Per-entry conv drift, flat-index form of conv2d_kernel_drift_total.

theorem Proofs.conv2d_flat_kernel_drift_sum {ic oc h w kH kW : ℕ} (b : Vec oc) (x : Tensor3 ic h w) {a : ℝ} (ha : 0 ≤ a) (hx : ∀ (c : Fin ic) (i : Fin h) (j : Fin w), |x c i j| ≤ a) (v e : Vec (oc * ic * kH * kW)) : ∑ k : Fin (oc * h * w), |(conv2d (Kernel4.unflatten (v + e)) b x).flatten k - (conv2d (Kernel4.unflatten v) b x).flatten k| ≤ ↑(h * w) * (a * ∑ idx : Fin (oc * ic * kH * kW), |e idx|)

ℓ1 conv drift, flat-index form of conv2d_kernel_drift_sum.

theorem Proofs.cnn_pool_l1_drift {c h w kH kW : ℕ} (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) {a : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₁ cc i j| ≤ a) (v e : Vec (c * c * kH * kW)) : ∑ q : Fin (c * h * w), |maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (v + e)) b₂ x₁).flatten) q - maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten) q| ≤ ↑(2 * h * (2 * w)) * (a * ∑ idx : Fin (c * c * kH * kW), |e idx|)

Pooled ℓ1 drift: kernel perturbation → conv (ℓ1, spatial multiplicity) → relu (contraction) → pool (contraction).

theorem Proofs.cnn_postrelu_close {c h w kH kW : ℕ} (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) {a : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₁ cc i j| ≤ a) (v e : Vec (c * c * kH * kW)) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) : |Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (v + e)) b₂ x₁).flatten) ci hi wi - Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten) ci hi wi| ≤ a * ∑ idx : Fin (c * c * kH * kW), |e idx|

Per-entry POST-relu tensor drift — the form the pool margin (MaxPool2MarginQ) consumes.

theorem Proofs.cnn_z3_drift {c h w d₃ kH kW : ℕ} (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) {a w₃ : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₁ cc i j| ≤ a) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (v e : Vec (c * c * kH * kW)) (l : Fin d₃) : |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (v + e)) b₂ x₁).flatten)) l - dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)) l| ≤ w₃ * (↑(2 * h * (2 * w)) * (a * ∑ idx : Fin (c * c * kH * kW), |e idx|))

Per-entry drift of the relu₃ pre-activation.

theorem Proofs.cnn_z4_drift {c h w d₃ d₄ kH kW : ℕ} (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) {a w₃ w₄ : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₁ cc i j| ≤ a) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (v e : Vec (c * c * kH * kW)) (q : Fin d₄) : |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (v + e)) b₂ x₁).flatten)))) q - dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)))) q| ≤ w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (a * ∑ idx : Fin (c * c * kH * kW), |e idx|))))

Per-entry drift of the relu₄ pre-activation.

theorem Proofs.cnn_conv2_logit_drift {c h w d₃ d₄ nC kH kW : ℕ} (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) {a w₃ w₄ w₅ : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₁ cc i j| ≤ a) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hw₅ : 0 ≤ w₅) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (v e : Vec (c * c * kH * kW)) (k : Fin nC) : |dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (v + e)) b₂ x₁).flatten)))))) k - dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)))))) k| ≤ w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (a * ∑ idx : Fin (c * c * kH * kW), |e idx|))))))

Logit drift through the whole conv2 chain: kernel perturbation → conv → relu → pool → d₃ → relu → d₄ → relu → d₅. Each dense crossing contributes its ℓ1→ℓ1 operator factor dᵢ·wᵢ; the conv contributes the weight-sharing multiplicity (2h)·(2w).

theorem Proofs.cnn_margin2_keeps_offkink {c h w kH kW : ℕ} (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) {a D : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₁ cc i j| ≤ a) (v e : Vec (c * c * kH * kW)) (he : ∑ idx : Fin (c * c * kH * kW), |e idx| ≤ D) (hm : ∀ (k : Fin (c * (2 * h) * (2 * w))), a * D < |(conv2d (Kernel4.unflatten v) b₂ x₁).flatten k|) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (k : Fin (c * (2 * h) * (2 * w))) : (conv2d (Kernel4.unflatten (v + t • e)) b₂ x₁).flatten k ≠ 0 ∧ (0 < (conv2d (Kernel4.unflatten (v + t • e)) b₂ x₁).flatten k ↔ 0 < (conv2d (Kernel4.unflatten v) b₂ x₁).flatten k)

The relu₂ margin keeps the conv pre-activation off the kink, same sign, along the whole step segment.

theorem Proofs.cnn_postrelu_close_seg {c h w kH kW : ℕ} (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) {a D : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₁ cc i j| ≤ a) (v e : Vec (c * c * kH * kW)) (he : ∑ idx : Fin (c * c * kH * kW), |e idx| ≤ D) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) : |Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (v + t • e)) b₂ x₁).flatten) ci hi wi - Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten) ci hi wi| ≤ a * D

The POST-relu tensor stays within the pool margin radius a·D along the whole step segment — what MaxPool2MarginQ.{smooth_of_close, isArgmax_iff, pdiv3_eq} consume.

theorem Proofs.cnn_margin3_keeps_offkink {c h w d₃ kH kW : ℕ} (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) {a w₃ D : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₁ cc i j| ≤ a) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (v e : Vec (c * c * kH * kW)) (he : ∑ idx : Fin (c * c * kH * kW), |e idx| ≤ D) (hm : ∀ (l : Fin d₃), w₃ * (↑(2 * h * (2 * w)) * (a * D)) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)) l|) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (l : Fin d₃) : dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (v + t • e)) b₂ x₁).flatten)) l ≠ 0 ∧ (0 < dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (v + t • e)) b₂ x₁).flatten)) l ↔ 0 < dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)) l)

The relu₃ margin keeps the first head pre-activation off the kink, same sign, along the whole step segment.

theorem Proofs.cnn_margin4_keeps_offkink {c h w d₃ d₄ kH kW : ℕ} (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) {a w₃ w₄ D : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₁ cc i j| ≤ a) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (v e : Vec (c * c * kH * kW)) (he : ∑ idx : Fin (c * c * kH * kW), |e idx| ≤ D) (hm : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (a * D)))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)))) q|) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (q : Fin d₄) : dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (v + t • e)) b₂ x₁).flatten)))) q ≠ 0 ∧ (0 < dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (v + t • e)) b₂ x₁).flatten)))) q ↔ 0 < dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)))) q)

The relu₄ margin keeps the second head pre-activation off the kink, same sign, along the whole step segment.

theorem Proofs.head3_sum_drift {p d₃ d₄ nC : ℕ} (W₃ : Mat p d₃) (W₄ : Mat d₃ d₄) (W₅ : Mat d₄ nC) {w₃ w₄ w₅ Δ : ℝ} (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin p) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hw₅ : 0 ≤ w₅) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (m₃ : Fin d₃ → ℝ) (hm₃ : ∀ (l : Fin d₃), |m₃ l| ≤ 1) (m₄ : Fin d₄ → ℝ) (hm₄ : ∀ (r : Fin d₄), |m₄ r| ≤ 1) (s s' oh : Vec nC) (hs : ∀ (k : Fin nC), |s' k - s k| ≤ Δ) (q : Fin p) : |∑ l : Fin d₃, W₃ q l * (m₃ l * ∑ r : Fin d₄, W₄ l r * (m₄ r * ∑ k : Fin nC, W₅ r k * (s' k - oh k))) - ∑ l : Fin d₃, W₃ q l * (m₃ l * ∑ r : Fin d₄, W₄ l r * (m₄ r * ∑ k : Fin nC, W₅ r k * (s k - oh k)))| ≤ ↑d₃ * (w₃ * (↑d₄ * (w₄ * (↑nC * (w₅ * Δ)))))

Frozen-mask head-gradient drift: with the two head masks frozen (0/1-valued, shared between the two points) and the softmax drifting by at most Δ, the head3 gradient closed form drifts by at most d₃·w₃·d₄·w₄·nC·w₅·Δ — the oneHot cancels in the difference.

theorem Proofs.cnn_conv2_loss_grad_lipschitz {c h w d₃ d₄ nC kH kW : ℕ} (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hh : 0 < h) (hw : 0 < w) {a w₃ w₄ w₅ D : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₁ cc i j| ≤ a) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hw₅ : 0 ≤ w₅) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (v d : Vec (c * c * kH * kW)) (hd : ∑ idx : Fin (c * c * kH * kW), |d idx| ≤ D) (hm2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), a * D < |(conv2d (Kernel4.unflatten v) b₂ x₁).flatten k|) (hmq : MaxPool2MarginQ (a * D) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten))) (hm3 : ∀ (l : Fin d₃), w₃ * (↑(2 * h * (2 * w)) * (a * D)) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)) l|) (hm4 : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (a * D)))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v) b₂ x₁).flatten)))) q|) (hsmall : 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (a * D))))))) < 1) (t : ℝ) (ht : t ∈ Set.Icc 0 1) (idx : Fin (c * c * kH * kW)) : |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) (v + t • d) idx - gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) v idx| ≤ 2 * ↑nC * ↑(2 * h * (2 * w)) ^ 2 * ↑d₃ ^ 2 * ↑d₄ ^ 2 * w₃ ^ 2 * w₄ ^ 2 * w₅ ^ 2 * a ^ 2 / (1 - 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (a * D)))))))) * (t * D)

Segment-Lipschitz gradient for the conv2-kernel loss, explicit constant. Under the four margins at step radius D — relu₂ (a·D), pool selection (MaxPool2MarginQ (a·D) of the POST-relu tensor), relu₃ (w₃·4hw·a·D), relu₄ (w₄·d₃·w₃·4hw·a·D) — every routing decision (masks AND pool argmaxes) freezes along [v, v+d], the point-free conv Jacobian factors out, and the difference collapses to the softmax drift exactly as in mlp_input_loss_grad_lipschitz. The conv-layer peer of that theorem; the constant picks up the weight-sharing multiplicity ((2h)·(2w))².

theorem Proofs.cnn_conv2_sgd_descends {c h w d₃ d₄ nC kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (gh : Vec (c * c * kH * kW)) (hc : 0 < c) (hh : 0 < h) (hw : 0 < w) {lr η a w₃ w₄ w₅ : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₁ cc i j| ≤ a) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hw₅ : 0 ≤ w₅) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (hlr : 0 ≤ lr) (hη : 0 ≤ η) (hgh : ∀ (idx : Fin (c * c * kH * kW)), |gh idx - gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| ≤ η) (hm2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), a * (lr * (∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| + ↑(c * c * kH * kW) * η)) < |(conv2d W₂ b₂ x₁).flatten k|) (hmq : MaxPool2MarginQ (a * (lr * (∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| + ↑(c * c * kH * kW) * η))) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten))) (hm3 : ∀ (l : Fin d₃), w₃ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| + ↑(c * c * kH * kW) * η)))) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten)) l|) (hm4 : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| + ↑(c * c * kH * kW) * η)))))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten)))) q|) (hsmall : 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| + ↑(c * c * kH * kW) * η))))))))) < 1) (h1 : lr * η * ∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| ≤ (lr * ∑ idx : Fin (c * c * kH * kW), gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx ^ 2) / 4) (h2 : 2 * ↑nC * ↑(2 * h * (2 * w)) ^ 2 * ↑d₃ ^ 2 * ↑d₄ ^ 2 * w₃ ^ 2 * w₄ ^ 2 * w₅ ^ 2 * a ^ 2 / (1 - 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| + ↑(c * c * kH * kW) * η)))))))))) * (lr * (∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| + ↑(c * c * kH * kW) * η)) ^ 2 ≤ (lr * ∑ idx : Fin (c * c * kH * kW), gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx ^ 2) / 4) : crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (W₂.flatten - lr • gh)) b₂ x₁).flatten))))))) label ≤ crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten W₂.flatten) b₂ x₁).flatten))))))) label - (lr * ∑ idx : Fin (c * c * kH * kW), gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx ^ 2) / 2

One inexact SGD step on the CNN's second conv kernel provably decreases the cross-entropy loss. All of sgd_descends' hypotheses discharged for the loss-of-conv2-kernel map: differentiability along the segment and the segment-Lipschitz constant both come from the FOUR margin hypotheses at the step radius D = lr·(‖∇L‖₁ + |kernel|·η) — relu₂, the pool-selection margin (POST-relu), relu₃, relu₄ — which freeze every mask and the pool's entire routing pattern along the step. Remaining hypotheses are checkable arithmetic: the oracle accuracy η, the margins, the small-step condition, and the two dominance conditions. Conclusion: the loss drops by ≥ lr·‖∇L‖₂²/2. The conv-layer peer of mlp_input_sgd_descends; the descent program now reaches through weight sharing and max-pooling.

theorem Proofs.cnn_conv2_float_sgd_descends {c h w d₃ d₄ nC kH kW : ℕ} (M : FloatModel) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (fexp : ℝ → ℝ) {lr a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp : ℝ} (hc : 0 < c) (hh : 0 < h) (hw : 0 < w) (ha : 0 ≤ a) (hw₂ : 0 ≤ w₂) (hβ₂ : 0 ≤ β₂) (hw₃ : 0 ≤ w₃) (hβ₃ : 0 ≤ β₃) (hw₄ : 0 ≤ w₄) (hβ₄ : 0 ≤ β₄) (hw₅ : 0 ≤ w₅) (hβ₅ : 0 ≤ β₅) (hlr : 0 ≤ lr) (heexp0 : 0 ≤ eexp) (heexp1 : eexp ≤ 1) (hfexp : ∀ (t : ℝ), |fexp t - Real.exp t| ≤ eexp * Real.exp t) (hρ1 : FloatModel.smRho M.u eexp nC < 1) (hx : ∀ (cc : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₁ cc i j| ≤ a) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hb₂ : ∀ (o : Fin c), |b₂ o| ≤ β₂) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hb₃ : ∀ (j : Fin d₃), |b₃ j| ≤ β₃) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hb₄ : ∀ (j : Fin d₄), |b₄ j| ≤ β₄) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (hb₅ : ∀ (j : Fin nC), |b₅ j| ≤ β₅) (hmarginConv : ∀ (k : Fin (c * (2 * h) * (2 * w))), FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0 < |(conv2d W₂ b₂ x₁).flatten k|) (hmarginPool : MaxPool2MarginQ (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten))) (hmargin3 : ∀ (l : Fin d₃), FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ a) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten)) l|) (hmargin4 : ∀ (q : Fin d₄), FloatModel.layerBudget M.u d₃ w₄ β₄ (FloatModel.layerAct (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ a)) (FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ a) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0)) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten)))) q|) (hm2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), a * (lr * (∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| + ↑(c * c * kH * kW) * M.cnnConv2GradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)) < |(conv2d W₂ b₂ x₁).flatten k|) (hmq : MaxPool2MarginQ (a * (lr * (∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| + ↑(c * c * kH * kW) * M.cnnConv2GradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp))) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten))) (hm3 : ∀ (l : Fin d₃), w₃ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| + ↑(c * c * kH * kW) * M.cnnConv2GradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)))) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten)) l|) (hm4 : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| + ↑(c * c * kH * kW) * M.cnnConv2GradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)))))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten)))) q|) (hsmall : 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| + ↑(c * c * kH * kW) * M.cnnConv2GradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp))))))))) < 1) (h1 : lr * M.cnnConv2GradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp * ∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| ≤ (lr * ∑ idx : Fin (c * c * kH * kW), gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx ^ 2) / 4) (h2 : 2 * ↑nC * ↑(2 * h * (2 * w)) ^ 2 * ↑d₃ ^ 2 * ↑d₄ ^ 2 * w₃ ^ 2 * w₄ ^ 2 * w₅ ^ 2 * a ^ 2 / (1 - 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| + ↑(c * c * kH * kW) * M.cnnConv2GradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)))))))))) * (lr * (∑ idx : Fin (c * c * kH * kW), |gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx| + ↑(c * c * kH * kW) * M.cnnConv2GradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)) ^ 2 ≤ (lr * ∑ idx : Fin (c * c * kH * kW), gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx ^ 2) / 4) : crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (W₂.flatten - lr • M.cnnConv2FloatGrad b₂ x₁ W₃ b₃ W₄ b₄ W₅ b₅ fexp label W₂.flatten)) b₂ x₁).flatten))))))) label ≤ crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten W₂.flatten) b₂ x₁).flatten))))))) label - (lr * ∑ idx : Fin (c * c * kH * kW), gradAt (fun (v' : Vec (c * c * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten v') b₂ x₁).flatten))))))) label) W₂.flatten idx ^ 2) / 2

One binary32 SGD step on the CNN's second conv kernel provably decreases the cross-entropy loss — with NO abstract gradient-accuracy parameter (Increment 3, the conv-2 rung capstone). The conv peer of mlp_input_float_sgd_descends: the gradient is the actual binary32 W₂ gradient M.cnnConv2FloatGrad …, and its accuracy is proven by cnn_conv2_grad_close (η := cnnConv2GradBudget, discharged per kernel entry via k4Idx_surj), not assumed. The two rounding-margin families are carried as hypotheses (the honest first cut): the per-layer ROUND margins (hmarginConv/Pool/3/4, feeding the grad-close) and the gradient-radius STEP margins + hsmall/h1/h2 (feeding cnn_conv2_sgd_descends's drift-freeze and the descent geometry). The conv-2 input x₁ is exact.

theorem Proofs.sum_pinned_le {n : ℕ} {X : ℝ} (hX : 0 ≤ X) (P : Fin n → Prop) [DecidablePred P] (huniq : ∀ (i j : Fin n), P i → P j → i = j) : (∑ i : Fin n, if P i then X else 0) ≤ X

A 0/1-pinned sum is at most its pinned value: if P holds for at most one index, ∑ i, (if P i then X else 0) ≤ X.

theorem Proofs.sum_swap_12_3 {α : Type u_1} {β : Type u_2} {γ : Type u_3} [Fintype α] [Fintype β] [Fintype γ] (f : α → β → γ → ℝ) : ∑ a : α, ∑ b : β, ∑ c : γ, f a b c = ∑ c : γ, ∑ a : α, ∑ b : β, f a b c

Rotate the innermost summation index of a triple sum to the front.

theorem Proofs.sum_swap_pair_pair {α : Type u_1} {β : Type u_2} {γ : Type u_3} {δ : Type u_4} [Fintype α] [Fintype β] [Fintype γ] [Fintype δ] (f : α → β → γ → δ → ℝ) : ∑ a : α, ∑ b : β, ∑ c : γ, ∑ d : δ, f a b c d = ∑ c : γ, ∑ d : δ, ∑ a : α, ∑ b : β, f a b c d

Swap the two index pairs of a quadruple sum.

theorem Proofs.abs_triple_sum_sub_le {α : Type u_1} {β : Type u_2} {γ : Type u_3} [Fintype α] [Fintype β] [Fintype γ] (f g : α → β → γ → ℝ) : |∑ a : α, ∑ b : β, ∑ c : γ, f a b c - ∑ a : α, ∑ b : β, ∑ c : γ, g a b c| ≤ ∑ a : α, ∑ b : β, ∑ c : γ, |f a b c - g a b c|

Triangle inequality for a difference of triple sums.

noncomputable def Proofs.convTap {ic oc h w kH kW : ℕ} (W : Kernel4 oc ic kH kW) (ci : Fin ic) (hi : Fin h) (wi : Fin w) (co : Fin oc) (ho : Fin h) (wo : Fin w) : ℝ

The kernel tap that multiplies input entry (ci,hi,wi) in output entry (co,ho,wo) — the input-side Jacobian entry of conv2d. Depends on the kernel only, never the input (conv is linear in its input). Deliberately let-free, like convPad.

Equations

• One or more equations did not get rendered due to their size.

theorem Proofs.convTap_abs_le {ic oc h w kH kW : ℕ} {W : Kernel4 oc ic kH kW} {w' : ℝ} (hw' : 0 ≤ w') (hW : ∀ (o : Fin oc) (c : Fin ic) (kh : Fin kH) (kw : Fin kW), |W o c kh kw| ≤ w') (ci : Fin ic) (hi : Fin h) (wi : Fin w) (co : Fin oc) (ho : Fin h) (wo : Fin w) : |convTap W ci hi wi co ho wo| ≤ w'

A single conv tap is bounded by the kernel magnitude (out-of-pad taps are zero) — the per-entry bound the conv-2 backward dot_perturbed_close uses.

theorem Proofs.abs_convTap_expand {ic oc h w kH kW : ℕ} (W : Kernel4 oc ic kH kW) (ci : Fin ic) (hi : Fin h) (wi : Fin w) (co : Fin oc) (ho : Fin h) (wo : Fin w) : |convTap W ci hi wi co ho wo| = ∑ kh : Fin kH, ∑ kw : Fin kW, if ↑kh + ↑ho = ↑hi + (kH - 1) / 2 ∧ ↑kw + ↑wo = ↑wi + (kW - 1) / 2 then |W co ci kh kw| else 0

The tap as a kernel-offset indicator sum: |convTap| is the sum over kernel offsets (kh,kw) of |W co ci kh kw| pinned to the unique offset aligning input (hi,wi) with output (ho,wo). The workhorse for both mass bounds: summing it over OUTPUTS pins (ho,wo) per offset, summing it over INPUTS pins (hi,wi).

theorem Proofs.convTap_out_l1 {ic oc h w kH kW : ℕ} (W : Kernel4 oc ic kH kW) {wK : ℝ} (hW : ∀ (o : Fin oc) (c : Fin ic) (kh : Fin kH) (kw : Fin kW), |W o c kh kw| ≤ wK) (ci : Fin ic) (hi : Fin h) (wi : Fin w) : ∑ co : Fin oc, ∑ ho : Fin h, ∑ wo : Fin w, |convTap W ci hi wi co ho wo| ≤ ↑(oc * kH * kW) * wK

Output-side tap mass: one input entry feeds at most oc·kH·kW outputs, each through a tap bounded by wK — the ℓ1→ℓ1 operator factor of a conv crossing as a function of its input.

theorem Proofs.convTap_in_l1 {ic oc h w kH kW : ℕ} (W : Kernel4 oc ic kH kW) {wK : ℝ} (hW : ∀ (o : Fin oc) (c : Fin ic) (kh : Fin kH) (kw : Fin kW), |W o c kh kw| ≤ wK) (co : Fin oc) (ho : Fin h) (wo : Fin w) : ∑ ci : Fin ic, ∑ hi : Fin h, ∑ wi : Fin w, |convTap W ci hi wi co ho wo| ≤ ↑(ic * kH * kW) * wK

Input-side tap mass: one output entry reads at most ic·kH·kW inputs, each through a tap bounded by wK.

theorem Proofs.conv2d_input_pdiv3 {ic oc h w kH kW : ℕ} (W : Kernel4 oc ic kH kW) (b : Vec oc) (x : Tensor3 ic h w) (ci : Fin ic) (hi : Fin h) (wi : Fin w) (co : Fin oc) (ho : Fin h) (wo : Fin w) : pdiv3 (conv2d W b) x ci hi wi co ho wo = convTap W ci hi wi co ho wo

Closed form of the conv input-map pdiv3 — extracted from the certified input-VJP (conv2d_has_vjp3) by contracting its .correct field against a basis cotangent. Point-free in x: conv is linear in its input.

theorem Proofs.conv2d_flat_input_pdiv {ic oc h w kH kW : ℕ} (W : Kernel4 oc ic kH kW) (b : Vec oc) (y : Vec (ic * h * w)) (ci : Fin ic) (hi : Fin h) (wi : Fin w) (co : Fin oc) (ho : Fin h) (wo : Fin w) : pdiv (fun (u : Vec (ic * h * w)) => (conv2d W b (Tensor3.unflatten u)).flatten) y (t3Idx ci hi wi) (t3Idx co ho wo) = convTap W ci hi wi co ho wo

Flat-coordinate form of conv2d_input_pdiv3 — the shape the chain rule through flatConv W₂ b₂ consumes.

theorem Proofs.abs_convPad_sub_le {ic h w kH kW : ℕ} (x x' : Tensor3 ic h w) {δ : ℝ} (hδ : 0 ≤ δ) (hclose : ∀ (c : Fin ic) (i : Fin h) (j : Fin w), |x' c i j - x c i j| ≤ δ) (c : Fin ic) (kh : Fin kH) (kw : Fin kW) (hi : Fin h) (wi : Fin w) : |convPad kH kW x' c kh kw hi wi - convPad kH kW x c kh kw hi wi| ≤ δ

Padded reads move no more than the input entries.

theorem Proofs.conv2d_input_entry_drift {ic oc h w kH kW : ℕ} (W : Kernel4 oc ic kH kW) (b : Vec oc) (x x' : Tensor3 ic h w) {wK δ : ℝ} (hwK : 0 ≤ wK) (hW : ∀ (o : Fin oc) (c : Fin ic) (kh : Fin kH) (kw : Fin kW), |W o c kh kw| ≤ wK) (hδ : 0 ≤ δ) (hclose : ∀ (c : Fin ic) (i : Fin h) (j : Fin w), |x' c i j - x c i j| ≤ δ) (o : Fin oc) (ho : Fin h) (wo : Fin w) : |conv2d W b x' o ho wo - conv2d W b x o ho wo| ≤ ↑(ic * kH * kW) * (wK * δ)

Per-entry conv input drift: each output reads ic·kH·kW padded inputs through taps bounded by wK.

theorem Proofs.abs_convPad_sub_expand {ic h w kH kW : ℕ} (x x' : Tensor3 ic h w) (c : Fin ic) (kh : Fin kH) (kw : Fin kW) (ho : Fin h) (wo : Fin w) : |convPad kH kW x' c kh kw ho wo - convPad kH kW x c kh kw ho wo| = ∑ i : Fin h, ∑ j : Fin w, if ↑kh + ↑ho = ↑i + (kH - 1) / 2 ∧ ↑kw + ↑wo = ↑j + (kW - 1) / 2 then |x' c i j - x c i j| else 0

The padded-read drift as a position-pinned indicator sum — the input-side peer of abs_convTap_expand, for the ℓ1 bound.

theorem Proofs.conv2d_input_l1_drift {ic oc h w kH kW : ℕ} (W : Kernel4 oc ic kH kW) (b : Vec oc) (x x' : Tensor3 ic h w) {wK : ℝ} (hwK : 0 ≤ wK) (hW : ∀ (o : Fin oc) (c : Fin ic) (kh : Fin kH) (kw : Fin kW), |W o c kh kw| ≤ wK) : ∑ o : Fin oc, ∑ ho : Fin h, ∑ wo : Fin w, |conv2d W b x' o ho wo - conv2d W b x o ho wo| ≤ ↑(oc * kH * kW) * (wK * ∑ c : Fin ic, ∑ i : Fin h, ∑ j : Fin w, |x' c i j - x c i j|)

ℓ1 conv input drift: each input entry feeds at most oc·kH·kW outputs, so the total output drift is at most oc·kH·kW·wK times the total input drift — locality, not a spatial count.

theorem Proofs.cnn1_postrelu1_close {ic c h w kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) {a : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (u e : Vec (c * ic * kH * kW)) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) : |Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (u + e)) b₁ x₀).flatten) ci hi wi - Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten) ci hi wi| ≤ a * ∑ idx : Fin (c * ic * kH * kW), |e idx|

POST-relu₁ tensor drift under a conv1 kernel perturbation.

theorem Proofs.cnn1_z2_entry_drift {ic c h w kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) {a w₂ : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (u e : Vec (c * ic * kH * kW)) (k : Fin (c * (2 * h) * (2 * w))) : |(conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (u + e)) b₁ x₀).flatten))).flatten k - (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten k| ≤ ↑(c * kH * kW) * (w₂ * (a * ∑ idx : Fin (c * ic * kH * kW), |e idx|))

Per-entry conv2-preactivation drift under a conv1 kernel perturbation: the perturbation crosses conv2 as a function of its INPUT, picking up the locality factor c·kH·kW·w₂.

theorem Proofs.cnn1_postrelu2_close {ic c h w kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) {a w₂ : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (u e : Vec (c * ic * kH * kW)) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) : |Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (u + e)) b₁ x₀).flatten))).flatten) ci hi wi - Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten) ci hi wi| ≤ ↑(c * kH * kW) * (w₂ * (a * ∑ idx : Fin (c * ic * kH * kW), |e idx|))

POST-relu₂ tensor drift under a conv1 kernel perturbation — what the pool margin consumes on the conv1 rung.

theorem Proofs.cnn1_pool_l1_drift {ic c h w kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) {a w₂ : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (u e : Vec (c * ic * kH * kW)) : ∑ q : Fin (c * h * w), |maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (u + e)) b₁ x₀).flatten))).flatten) q - maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten) q| ≤ ↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * ∑ idx : Fin (c * ic * kH * kW), |e idx|)))

Pooled ℓ1 drift under a conv1 kernel perturbation: conv1 (ℓ1, spatial multiplicity) → relu → conv2-as-input (ℓ1, LOCALITY multiplicity c·kH·kW) → relu → pool.

theorem Proofs.cnn1_z3_drift {ic c h w d₃ kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) {a w₂ w₃ : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (u e : Vec (c * ic * kH * kW)) (l : Fin d₃) : |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (u + e)) b₁ x₀).flatten))).flatten)) l - dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)) l| ≤ w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * ∑ idx : Fin (c * ic * kH * kW), |e idx|))))

Per-entry drift of the relu₃ pre-activation, conv1 rung.

theorem Proofs.cnn1_z4_drift {ic c h w d₃ d₄ kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) {a w₂ w₃ w₄ : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (u e : Vec (c * ic * kH * kW)) (q : Fin d₄) : |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (u + e)) b₁ x₀).flatten))).flatten)))) q - dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)))) q| ≤ w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * ∑ idx : Fin (c * ic * kH * kW), |e idx|))))))

Per-entry drift of the relu₄ pre-activation, conv1 rung.

theorem Proofs.cnn1_logit_drift {ic c h w d₃ d₄ nC kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) {a w₂ w₃ w₄ w₅ : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hw₅ : 0 ≤ w₅) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (u e : Vec (c * ic * kH * kW)) (k : Fin nC) : |dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (u + e)) b₁ x₀).flatten))).flatten)))))) k - dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)))))) k| ≤ w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * ∑ idx : Fin (c * ic * kH * kW), |e idx|))))))))

Logit drift through the whole conv1 chain.

theorem Proofs.cnn1_margin1_keeps_offkink {ic c h w kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) {a D : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (u e : Vec (c * ic * kH * kW)) (he : ∑ idx : Fin (c * ic * kH * kW), |e idx| ≤ D) (hm : ∀ (k : Fin (c * (2 * h) * (2 * w))), a * D < |(conv2d (Kernel4.unflatten u) b₁ x₀).flatten k|) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (k : Fin (c * (2 * h) * (2 * w))) : (conv2d (Kernel4.unflatten (u + t • e)) b₁ x₀).flatten k ≠ 0 ∧ (0 < (conv2d (Kernel4.unflatten (u + t • e)) b₁ x₀).flatten k ↔ 0 < (conv2d (Kernel4.unflatten u) b₁ x₀).flatten k)

The relu₁ margin keeps the conv1 pre-activation off the kink.

theorem Proofs.cnn1_margin2_keeps_offkink {ic c h w kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) {a w₂ D : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (u e : Vec (c * ic * kH * kW)) (he : ∑ idx : Fin (c * ic * kH * kW), |e idx| ≤ D) (hm : ∀ (k : Fin (c * (2 * h) * (2 * w))), ↑(c * kH * kW) * (w₂ * (a * D)) < |(conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten k|) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (k : Fin (c * (2 * h) * (2 * w))) : (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (u + t • e)) b₁ x₀).flatten))).flatten k ≠ 0 ∧ (0 < (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (u + t • e)) b₁ x₀).flatten))).flatten k ↔ 0 < (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten k)

The relu₂ margin (at the conv1 radius) keeps the conv2 pre-activation off the kink.

theorem Proofs.cnn1_postrelu2_close_seg {ic c h w kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) {a w₂ D : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (u e : Vec (c * ic * kH * kW)) (he : ∑ idx : Fin (c * ic * kH * kW), |e idx| ≤ D) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) : |Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (u + t • e)) b₁ x₀).flatten))).flatten) ci hi wi - Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten) ci hi wi| ≤ ↑(c * kH * kW) * (w₂ * (a * D))

The POST-relu₂ tensor stays within the conv1-rung pool margin radius along the whole step segment.

theorem Proofs.cnn1_margin3_keeps_offkink {ic c h w d₃ kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) {a w₂ w₃ D : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (u e : Vec (c * ic * kH * kW)) (he : ∑ idx : Fin (c * ic * kH * kW), |e idx| ≤ D) (hm : ∀ (l : Fin d₃), w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * D)))) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)) l|) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (l : Fin d₃) : dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (u + t • e)) b₁ x₀).flatten))).flatten)) l ≠ 0 ∧ (0 < dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (u + t • e)) b₁ x₀).flatten))).flatten)) l ↔ 0 < dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)) l)

The relu₃ margin (at the conv1 radius) keeps the first head pre-activation off the kink.

theorem Proofs.cnn1_margin4_keeps_offkink {ic c h w d₃ d₄ kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) {a w₂ w₃ w₄ D : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (u e : Vec (c * ic * kH * kW)) (he : ∑ idx : Fin (c * ic * kH * kW), |e idx| ≤ D) (hm : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * D)))))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)))) q|) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (q : Fin d₄) : dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (u + t • e)) b₁ x₀).flatten))).flatten)))) q ≠ 0 ∧ (0 < dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (u + t • e)) b₁ x₀).flatten))).flatten)))) q ↔ 0 < dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)))) q)

The relu₄ margin (at the conv1 radius) keeps the second head pre-activation off the kink.

theorem Proofs.cnn1_pool_head_differentiableAt {c h w d₃ d₄ nC kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hc : 0 < c) (hh : 0 < h) (hw : 0 < w) (z₁ : Vec (c * (2 * h) * (2 * w))) (hz1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), z₁ k ≠ 0) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₁))).flatten k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₁))).flatten))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₁))).flatten)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₁))).flatten)))) q ≠ 0) : DifferentiableAt ℝ (fun (y : Vec (c * (2 * h) * (2 * w))) (x : Fin 1) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) y))).flatten))))))) label) z₁

The whole head above the conv1 output — CE∘head3∘pool∘relu∘ (flatConv W₂ b₂)∘relu — is differentiable at any five-condition point.

theorem Proofs.cnn1_pool_head_input_grad {c h w d₃ d₄ nC kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (z₁ : Vec (c * (2 * h) * (2 * w))) (hz1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), z₁ k ≠ 0) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₁))).flatten k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₁))).flatten))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₁))).flatten)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₁))).flatten)))) q ≠ 0) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) : pdiv (fun (y : Vec (c * (2 * h) * (2 * w))) (x : Fin 1) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) y))).flatten))))))) label) z₁ (t3Idx ci hi wi) 0 = (if z₁ (t3Idx ci hi wi) > 0 then 1 else 0) * ∑ co : Fin c, ∑ ho : Fin (2 * h), ∑ wo : Fin (2 * w), convTap W₂ ci hi wi co ho wo * ((if (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₁))).flatten (t3Idx co ho wo) > 0 then 1 else 0) * if MaxPool2IsArgmax (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₁))).flatten)) co ho wo then ∑ l : Fin d₃, W₃ (t3Idx co (winRow ho) (winCol wo)) l * ((if dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₁))).flatten)) l > 0 then 1 else 0) * ∑ q : Fin d₄, W₄ l q * ((if dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₁))).flatten)))) q > 0 then 1 else 0) * ∑ k : Fin nC, W₅ q k * (softmax nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) z₁))).flatten))))))) k - oneHot nC label k))) else 0)

Loss input-gradient at the conv1 output — the conv1 peer of pool_relu_input_grad. One more relu mask and one conv-as-input crossing: the chain picks up relu'(z₁) and contracts the point-free tap Jacobian of conv2 with the pool-collapsed conv2-rung gradient.

theorem Proofs.cnn_conv1_loss_differentiableAt {ic c h w d₃ d₄ nC kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hc : 0 < c) (hh : 0 < h) (hw : 0 < w) (u : Vec (c * ic * kH * kW)) (hz1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d (Kernel4.unflatten u) b₁ x₀).flatten k ≠ 0) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)))) q ≠ 0) : DifferentiableAt ℝ (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) u

The loss-of-conv1-kernel map is differentiable at any five-condition point.

theorem Proofs.cnn_conv1_loss_gradAt {ic c h w d₃ d₄ nC kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hh : 0 < h) (hw : 0 < w) (u : Vec (c * ic * kH * kW)) (hz1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d (Kernel4.unflatten u) b₁ x₀).flatten k ≠ 0) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)))) q ≠ 0) (o : Fin c) (cc : Fin ic) (kh : Fin kH) (kw : Fin kW) : gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) u (k4Idx o cc kh kw) = ∑ ci : Fin c, ∑ hi : Fin (2 * h), ∑ wi : Fin (2 * w), (if ci = o then convPad kH kW x₀ cc kh kw hi wi else 0) * ((if (conv2d (Kernel4.unflatten u) b₁ x₀).flatten (t3Idx ci hi wi) > 0 then 1 else 0) * ∑ co : Fin c, ∑ ho : Fin (2 * h), ∑ wo : Fin (2 * w), convTap W₂ ci hi wi co ho wo * ((if (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten (t3Idx co ho wo) > 0 then 1 else 0) * if MaxPool2IsArgmax (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)) co ho wo then ∑ l : Fin d₃, W₃ (t3Idx co (winRow ho) (winCol wo)) l * ((if dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)) l > 0 then 1 else 0) * ∑ q : Fin d₄, W₄ l q * ((if dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)))) q > 0 then 1 else 0) * ∑ k : Fin nC, W₅ q k * (softmax nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten))))))) k - oneHot nC label k))) else 0))

Closed form of the conv1 loss gradient at any five-margin point — the same fold, contracted with the conv1 head gradient (cnn1_pool_head_input_grad): the conv1 weight Jacobian (convPad reads of the IMAGE) times relu₁'s mask times the point-free conv2 tap Jacobian times the pool-collapsed head. Two spatial triple-sums: weight sharing at conv1, locality at conv2.

theorem Proofs.cnn_conv1_loss_gradAt_reluMask {ic c h w d₃ d₄ nC kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hh : 0 < h) (hw : 0 < w) (u : Vec (c * ic * kH * kW)) (hz1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d (Kernel4.unflatten u) b₁ x₀).flatten k ≠ 0) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)))) q ≠ 0) (o : Fin c) (cc : Fin ic) (kh : Fin kH) (kw : Fin kW) : gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) u (k4Idx o cc kh kw) = ∑ s : Fin (2 * h * (2 * w)), convPadWin kH kW x₀ cc kh kw s * cotWin (fun (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) => (if (conv2d (Kernel4.unflatten u) b₁ x₀).flatten (t3Idx ci hi wi) > 0 then 1 else 0) * ∑ co : Fin c, ∑ ho : Fin (2 * h), ∑ wo : Fin (2 * w), convTap W₂ ci hi wi co ho wo * ((if (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten (t3Idx co ho wo) > 0 then 1 else 0) * if MaxPool2IsArgmax (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)) co ho wo then dense (fun (j : Fin d₃) (i' : Fin (c * h * w)) => W₃ i' j) (fun (x : Fin (c * h * w)) => 0) (FloatModel.reluMask (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten))) (dense (fun (j : Fin d₄) (i' : Fin d₃) => W₄ i' j) (fun (x : Fin d₃) => 0) (FloatModel.reluMask (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten))))) (dense (fun (j : Fin nC) (i' : Fin d₄) => W₅ i' j) (fun (x : Fin d₄) => 0) fun (k : Fin nC) => softmax nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten))))))) k - oneHot nC label k)))) (t3Idx co (winRow ho) (winCol wo)) else 0)) o s

The certified conv-1 loss gradient, head restated in dense/reluMask form — the conv-1 peer of cnn_conv2_loss_gradAt_reluMask (Increment 4 keystone). One conv-backward deeper than conv-2: the conv-1-output cotangent is 𝟙[z₁>0] · ∑_{co,ho,wo} convTap·(conv-2-output cotangent), with the 3-dense head collapsed by head3_cot_reluMask exactly as in conv-2. The conv-1 ReLU mask, the conv-2 backward tap (convTap, the point-free conv-2 input Jacobian), the conv-2 ReLU mask and the pool selector all stay explicit (their float closeness is mask_scalar_close / dot_perturbed_close / poolBack_close). Packaged as the spatial dot ∑ₛ convPadWin x₀·cotWin (convWeightGrad_eq_dot) the float conv-1 weight dot rounds.

noncomputable def Proofs.FloatModel.cnnConv1FloatGrad {ic c h w d₃ d₄ nC kH kW : ℕ} (M : FloatModel) (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (fexp : ℝ → ℝ) (label : Fin nC) (u : Vec (c * ic * kH * kW)) : Vec (c * ic * kH * kW)

The binary32 conv-1 weight gradient the rendered trainer computes — the conv-1 peer of cnnConv2FloatGrad, one conv-backward deeper. At kernel entry (o,cc,kh,kw) it is the float dot of the (exact) padded-input window convPadWin x₀ against the float conv-1-output cotangent slab; that cotangent is the conv-1 ReLU mask 𝟙[z̃₁>0] times the float conv-2 backward M.dot (convTap W₂ slab) (float conv-2-output cotangent slab), where the conv-2 cotangent is exactly cnnConv2FloatGrad's, but at the FLOAT conv-2 input relu(z̃₁) (a function of the conv-1 kernel u). All M-ops carry the rounding; reluMask/maxPoolFlat/relu/convTap are exact.

Equations

• One or more equations did not get rendered due to their size.

@[simp]

theorem Proofs.FloatModel.cnnConv1FloatGrad_apply {ic c h w d₃ d₄ nC kH kW : ℕ} (M : FloatModel) (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (fexp : ℝ → ℝ) (label : Fin nC) (u : Vec (c * ic * kH * kW)) (o : Fin c) (cc : Fin ic) (kh : Fin kH) (kw : Fin kW) : M.cnnConv1FloatGrad b₁ x₀ W₂ b₂ W₃ b₃ W₄ b₄ W₅ b₅ fexp label u (k4Idx o cc kh kw) = M.dot (convPadWin kH kW x₀ cc kh kw) (cotWin (fun (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) => (if (M.convF (Kernel4.unflatten u) b₁ x₀).flatten (t3Idx ci hi wi) > 0 then 1 else 0) * M.dot (Tensor3.flatten fun (co : Fin c) (ho : Fin (2 * h)) (wo : Fin (2 * w)) => convTap W₂ ci hi wi co ho wo) (Tensor3.flatten fun (co : Fin c) (ho : Fin (2 * h)) (wo : Fin (2 * w)) => (if (M.convF W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (M.convF (Kernel4.unflatten u) b₁ x₀).flatten))).flatten (t3Idx co ho wo) > 0 then 1 else 0) * if MaxPool2IsArgmax (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (M.convF W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (M.convF (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)) co ho wo then M.dense (fun (j : Fin d₃) (i' : Fin (c * h * w)) => W₃ i' j) (fun (x : Fin (c * h * w)) => 0) (reluMask (M.dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (M.convF W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (M.convF (Kernel4.unflatten u) b₁ x₀).flatten))).flatten))) (M.dense (fun (j : Fin d₄) (i' : Fin d₃) => W₄ i' j) (fun (x : Fin d₃) => 0) (reluMask (M.dense W₄ b₄ (relu d₃ (M.dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (M.convF W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (M.convF (Kernel4.unflatten u) b₁ x₀).flatten))).flatten))))) (M.dense (fun (j : Fin nC) (i' : Fin d₄) => W₅ i' j) (fun (x : Fin d₄) => 0) (M.softmaxCECotF fexp (M.dense W₅ b₅ (relu d₄ (M.dense W₄ b₄ (relu d₃ (M.dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (M.convF W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (M.convF (Kernel4.unflatten u) b₁ x₀).flatten))).flatten))))))) label))))) (t3Idx co (winRow ho) (winCol wo)) else 0)) o)

noncomputable def Proofs.FloatModel.cnnConv1GradBudget (M : FloatModel) (ic c h w d₃ d₄ nC kH kW : ℕ) (a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp : ℝ) : ℝ

The conv-1 float-backward grad-close budget — the conv-2 budget (cnnConv2GradBudget-shaped) deepened by one conv layer at the bottom (the forward nest now starts at conv-1 fan-in ic·kH·kW, the conv-2 input carries the conv-1 rounding E₁) and one conv-backward at the top: the conv-2 backward Higham γ over the slab c·(2h)·(2w) against the float-cotangent magnitude C2t plus the per-entry conv-2-cotangent drift e₂ gives eback; the conv-1 spatial dot (fan-in (2h)·(2w)) then rides eback and the float conv-1-cotangent magnitude C1t.

Equations

• One or more equations did not get rendered due to their size.

noncomputable def Proofs.FloatModel.cnnConv2CotBudget (M : FloatModel) (c h w d₃ d₄ nC kH kW : ℕ) (aX2 eX2 w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp : ℝ) : ℝ

The conv-2-output cotangent error budget, as a function of the conv-2 input magnitude aX2 and rounding eX2 — the e₂ of cnnConv2GradBudget (where aX2 = a, eX2 = 0) and the e₂ inside cnnConv1GradBudget (where aX2 = A₁, eX2 = E₁). Factored so the conv-1 rung reuses the conv-2 cotangent chain at a FLOAT conv-2 input.

Equations

• One or more equations did not get rendered due to their size.

noncomputable def Proofs.FloatModel.cnnConv2CotMag (d₃ d₄ nC : ℕ) (w₃ w₄ w₅ : ℝ) : ℝ

The real conv-2-output cotangent magnitude bound — aX2/eX2-independent (the head cotangent and the two masked Wᵀ steps are magnitude-frozen).

Equations

• Proofs.FloatModel.cnnConv2CotMag d₃ d₄ nC w₃ w₄ w₅ = Proofs.FloatModel.layerAct d₃ w₃ 0 (Proofs.FloatModel.layerAct d₄ w₄ 0 (Proofs.FloatModel.layerAct nC w₅ 0 1))

theorem Proofs.cnn_conv2_cot_close {c h w d₃ d₄ nC kH kW : ℕ} (M : FloatModel) (X2 X2F : Tensor3 c (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (fexp : ℝ → ℝ) {aX2 eX2 w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp : ℝ} (haX2 : 0 ≤ aX2) (heX2 : 0 ≤ eX2) (hw₂ : 0 ≤ w₂) (hβ₂ : 0 ≤ β₂) (hw₃ : 0 ≤ w₃) (hβ₃ : 0 ≤ β₃) (hw₄ : 0 ≤ w₄) (hβ₄ : 0 ≤ β₄) (hw₅ : 0 ≤ w₅) (hβ₅ : 0 ≤ β₅) (heexp0 : 0 ≤ eexp) (heexp1 : eexp ≤ 1) (hfexp : ∀ (t : ℝ), |fexp t - Real.exp t| ≤ eexp * Real.exp t) (hρ1 : FloatModel.smRho M.u eexp nC < 1) (hX2 : ∀ (co : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |X2F co i j - X2 co i j| ≤ eX2) (hX2mag : ∀ (co : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |X2 co i j| ≤ aX2) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hb₂ : ∀ (o : Fin c), |b₂ o| ≤ β₂) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hb₃ : ∀ (j : Fin d₃), |b₃ j| ≤ β₃) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hb₄ : ∀ (j : Fin d₄), |b₄ j| ≤ β₄) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (hb₅ : ∀ (j : Fin nC), |b₅ j| ≤ β₅) (hmarginConv : ∀ (k : Fin (c * (2 * h) * (2 * w))), FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ aX2 eX2 < |(conv2d W₂ b₂ X2).flatten k|) (hmarginPool : MaxPool2MarginQ (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ aX2 eX2) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ X2).flatten))) (hmargin3 : ∀ (l : Fin d₃), FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ aX2) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ aX2 eX2) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ X2).flatten)) l|) (hmargin4 : ∀ (q : Fin d₄), FloatModel.layerBudget M.u d₃ w₄ β₄ (FloatModel.layerAct (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ aX2)) (FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ aX2) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ aX2 eX2)) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ X2).flatten)))) q|) (co : Fin c) (ho : Fin (2 * h)) (wo : Fin (2 * w)) : |((if (M.convF W₂ b₂ X2F).flatten (t3Idx co ho wo) > 0 then 1 else 0) * if MaxPool2IsArgmax (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (M.convF W₂ b₂ X2F).flatten)) co ho wo then M.dense (fun (j : Fin d₃) (i' : Fin (c * h * w)) => W₃ i' j) (fun (x : Fin (c * h * w)) => 0) (FloatModel.reluMask (M.dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (M.convF W₂ b₂ X2F).flatten))) (M.dense (fun (j : Fin d₄) (i' : Fin d₃) => W₄ i' j) (fun (x : Fin d₃) => 0) (FloatModel.reluMask (M.dense W₄ b₄ (relu d₃ (M.dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (M.convF W₂ b₂ X2F).flatten))))) (M.dense (fun (j : Fin nC) (i' : Fin d₄) => W₅ i' j) (fun (x : Fin d₄) => 0) (M.softmaxCECotF fexp (M.dense W₅ b₅ (relu d₄ (M.dense W₄ b₄ (relu d₃ (M.dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (M.convF W₂ b₂ X2F).flatten))))))) label))))) (t3Idx co (winRow ho) (winCol wo)) else 0) - (if (conv2d W₂ b₂ X2).flatten (t3Idx co ho wo) > 0 then 1 else 0) * if MaxPool2IsArgmax (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ X2).flatten)) co ho wo then dense (fun (j : Fin d₃) (i' : Fin (c * h * w)) => W₃ i' j) (fun (x : Fin (c * h * w)) => 0) (FloatModel.reluMask (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ X2).flatten))) (dense (fun (j : Fin d₄) (i' : Fin d₃) => W₄ i' j) (fun (x : Fin d₃) => 0) (FloatModel.reluMask (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ X2).flatten))))) (dense (fun (j : Fin nC) (i' : Fin d₄) => W₅ i' j) (fun (x : Fin d₄) => 0) fun (k : Fin nC) => softmax nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ X2).flatten))))))) k - oneHot nC label k)))) (t3Idx co (winRow ho) (winCol wo)) else 0| ≤ M.cnnConv2CotBudget c h w d₃ d₄ nC kH kW aX2 eX2 w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp

The conv-2-output cotangent is float-close at a float conv-2 input (Increment 4 keystone) — Increment 2's conv-2 cotangent chain, factored to take the conv-2 input (X2, X2F) with |X2F − X2| ≤ eX2, |X2| ≤ aX2. The conv-1 rung instantiates X2 = relu(z₁), X2F = relu(z̃₁), eX2 = E₁. The chain: float forward from X2 (convF_close → dense_close×3) → head (softmax_ce_cot_close) → cot_step_close×2 → unmasked W₃ dense_close → pool-back (poolBack_close) → conv-2 ReLU mask (mask_scalar_close).

theorem Proofs.cnn_conv2_cot_real_abs_le {c h w d₃ d₄ nC kH kW : ℕ} (X2 : Tensor3 c (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) {w₃ w₄ w₅ : ℝ} (hw₃ : 0 ≤ w₃) (hw₄ : 0 ≤ w₄) (hw₅ : 0 ≤ w₅) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (co : Fin c) (ho : Fin (2 * h)) (wo : Fin (2 * w)) : |(if (conv2d W₂ b₂ X2).flatten (t3Idx co ho wo) > 0 then 1 else 0) * if MaxPool2IsArgmax (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ X2).flatten)) co ho wo then dense (fun (j : Fin d₃) (i' : Fin (c * h * w)) => W₃ i' j) (fun (x : Fin (c * h * w)) => 0) (FloatModel.reluMask (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ X2).flatten))) (dense (fun (j : Fin d₄) (i' : Fin d₃) => W₄ i' j) (fun (x : Fin d₃) => 0) (FloatModel.reluMask (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ X2).flatten))))) (dense (fun (j : Fin nC) (i' : Fin d₄) => W₅ i' j) (fun (x : Fin d₄) => 0) fun (k : Fin nC) => softmax nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ X2).flatten))))))) k - oneHot nC label k)))) (t3Idx co (winRow ho) (winCol wo)) else 0| ≤ FloatModel.cnnConv2CotMag d₃ d₄ nC w₃ w₄ w₅

The real conv-2-output cotangent is magnitude-bounded by cnnConv2CotMag — the aX2/eX2-independent ℓ∞ bound (the conv-2 ReLU mask and pool selector only shrink, the head cotangent is in [−1,1], the two masked Wᵀ steps and the unmasked W₃ ride layerAct). Used to bound the real conv-1 cotangent ∑ convTap·c₂ in the conv-1 rung.

theorem Proofs.abs_le_of_close {a b e C : ℝ} (h1 : |a - b| ≤ e) (h2 : |b| ≤ C) : |a| ≤ C + e

|a| ≤ C + e from |a − b| ≤ e and |b| ≤ C — lifts a closeness + a base magnitude to a float magnitude (the float cotangent bound from the real bound plus the drift).

theorem Proofs.convTap_back_close {c h w kH kW : ℕ} (M : FloatModel) (W₂ : Kernel4 c c kH kW) (c2F c2R : Tensor3 c (2 * h) (2 * w)) {w₂ C2t e2 : ℝ} (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hc2F : ∀ (co : Fin c) (ho : Fin (2 * h)) (wo : Fin (2 * w)), |c2F co ho wo| ≤ C2t) (hc2close : ∀ (co : Fin c) (ho : Fin (2 * h)) (wo : Fin (2 * w)), |c2F co ho wo - c2R co ho wo| ≤ e2) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) : |M.dot (Tensor3.flatten fun (co : Fin c) (ho : Fin (2 * h)) (wo : Fin (2 * w)) => convTap W₂ ci hi wi co ho wo) c2F.flatten - ∑ co : Fin c, ∑ ho : Fin (2 * h), ∑ wo : Fin (2 * w), convTap W₂ ci hi wi co ho wo * c2R co ho wo| ≤ ((1 + M.u) ^ (c * (2 * h) * (2 * w) + 1) - 1) * (↑(c * (2 * h) * (2 * w)) * (w₂ * C2t)) + ↑(c * (2 * h) * (2 * w)) * (w₂ * e2)

The float conv-2 backward (transpose conv) against a perturbed cotangent. The rounded M.dot of the exact convTap slab against the float conv-2-output cotangent c2F, vs the certified ∑ convTap·c2R — the convTap-flattening (sum_t3) plus dot_perturbed_close (fan-in c·(2h)·(2w), per-entry tap bound w₂, float-cotangent magnitude C2t, drift e₂). Generic in (c2F, c2R) so the conv-1 rung passes the conv-2 cotangent tensors abstractly.

theorem Proofs.convTap_back_abs_le {c h w kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (c2R : Tensor3 c (2 * h) (2 * w)) {w₂ CP : ℝ} (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hc2R : ∀ (co : Fin c) (ho : Fin (2 * h)) (wo : Fin (2 * w)), |c2R co ho wo| ≤ CP) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) : |∑ co : Fin c, ∑ ho : Fin (2 * h), ∑ wo : Fin (2 * w), convTap W₂ ci hi wi co ho wo * c2R co ho wo| ≤ ↑(c * (2 * h) * (2 * w)) * (w₂ * CP)

The real conv-2 backward ∑ convTap·c2R is magnitude-bounded by the tap ℓ∞-mass (c·(2h)·(2w))·w₂ times the cotangent bound CP — the (loose, uniform) bound on the real conv-1 cotangent.

theorem Proofs.cnn_conv1_grad_close {ic c h w d₃ d₄ nC kH kW : ℕ} (M : FloatModel) (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (fexp : ℝ → ℝ) (u : Vec (c * ic * kH * kW)) {a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp : ℝ} (hh : 0 < h) (hw : 0 < w) (ha : 0 ≤ a) (hw₁ : 0 ≤ w₁) (hβ₁ : 0 ≤ β₁) (hw₂ : 0 ≤ w₂) (hβ₂ : 0 ≤ β₂) (hw₃ : 0 ≤ w₃) (hβ₃ : 0 ≤ β₃) (hw₄ : 0 ≤ w₄) (hβ₄ : 0 ≤ β₄) (hw₅ : 0 ≤ w₅) (hβ₅ : 0 ≤ β₅) (heexp0 : 0 ≤ eexp) (heexp1 : eexp ≤ 1) (hfexp : ∀ (t : ℝ), |fexp t - Real.exp t| ≤ eexp * Real.exp t) (hρ1 : FloatModel.smRho M.u eexp nC < 1) (hx₀ : ∀ (ci : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ ci i j| ≤ a) (hu1 : ∀ (idx : Fin (c * ic * kH * kW)), |u idx| ≤ w₁) (hb₁ : ∀ (o : Fin c), |b₁ o| ≤ β₁) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hb₂ : ∀ (o : Fin c), |b₂ o| ≤ β₂) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hb₃ : ∀ (j : Fin d₃), |b₃ j| ≤ β₃) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hb₄ : ∀ (j : Fin d₄), |b₄ j| ≤ β₄) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (hb₅ : ∀ (j : Fin nC), |b₅ j| ≤ β₅) (hmargin1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0 < |(conv2d (Kernel4.unflatten u) b₁ x₀).flatten k|) (hmargin2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0) < |(conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten k|) (hmarginPool : MaxPool2MarginQ (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0)) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten))) (hmargin3 : ∀ (l : Fin d₃), FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a)) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0)) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)) l|) (hmargin4 : ∀ (q : Fin d₄), FloatModel.layerBudget M.u d₃ w₄ β₄ (FloatModel.layerAct (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a))) (FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a)) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)))) q|) (o : Fin c) (cc : Fin ic) (kh : Fin kH) (kw : Fin kW) : |M.cnnConv1FloatGrad b₁ x₀ W₂ b₂ W₃ b₃ W₄ b₄ W₅ b₅ fexp label u (k4Idx o cc kh kw) - gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) u (k4Idx o cc kh kw)| ≤ M.cnnConv1GradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp

The binary32 conv-1 weight gradient is within an explicit budget of the certified one (Increment 4 capstone) — the conv-1 peer of cnn_conv2_grad_close, one conv-backward deeper. With x₀ exact, the rendered trainer's W₁ gradient M.cnnConv1FloatGrad … stays within cnnConv1GradBudget. The conv-2 cotangent chain is reused at a FLOAT conv-2 input relu(z̃₁) (cnn_conv2_cot_close); the conv-2 backward is a rounded dot of the (exact) convTap slab against the float conv-2 cotangent slab (dot_perturbed_close over c·(2h)·(2w)); the conv-1 ReLU mask freezes (mask_scalar_close); the conv-1 weight dot rounds it (dot_perturbed_close over (2h)·(2w)). Five quantitative margins are carried; the bridge cnn_conv1_loss_gradAt_reluMask turns the gradAt into the dot.

theorem Proofs.cnn_conv1_loss_grad_lipschitz {ic c h w d₃ d₄ nC kH kW : ℕ} (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hh : 0 < h) (hw : 0 < w) {a w₂ w₃ w₄ w₅ D : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hw₅ : 0 ≤ w₅) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (u d : Vec (c * ic * kH * kW)) (hd : ∑ idx : Fin (c * ic * kH * kW), |d idx| ≤ D) (hm1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), a * D < |(conv2d (Kernel4.unflatten u) b₁ x₀).flatten k|) (hm2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), ↑(c * kH * kW) * (w₂ * (a * D)) < |(conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten k|) (hmq : MaxPool2MarginQ (↑(c * kH * kW) * (w₂ * (a * D))) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten))) (hm3 : ∀ (l : Fin d₃), w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * D)))) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)) l|) (hm4 : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * D)))))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u) b₁ x₀).flatten))).flatten)))) q|) (hsmall : 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * D))))))))) < 1) (t : ℝ) (ht : t ∈ Set.Icc 0 1) (idx : Fin (c * ic * kH * kW)) : |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) (u + t • d) idx - gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) u idx| ≤ 2 * ↑nC * ↑(2 * h * (2 * w)) ^ 2 * ↑(c * kH * kW) ^ 2 * ↑d₃ ^ 2 * ↑d₄ ^ 2 * w₂ ^ 2 * w₃ ^ 2 * w₄ ^ 2 * w₅ ^ 2 * a ^ 2 / (1 - 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * D)))))))))) * (t * D)

Segment-Lipschitz gradient for the conv1-kernel loss, explicit constant. Under the FIVE margins at step radius D — relu₁ (a·D), relu₂ (c·kH·kW·w₂·a·D), pool selection (same radius, POST-relu₂), relu₃, relu₄ — every routing decision freezes along [u, u+d], BOTH conv Jacobians factor out point-free, and the difference collapses to the softmax drift. The constant picks up the conv1 weight-sharing multiplicity ((2h)·(2w))² AND the conv2 locality multiplicity (c·kH·kW)²·w₂².

theorem Proofs.cnn_conv1_sgd_descends {ic c h w d₃ d₄ nC kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (gh : Vec (c * ic * kH * kW)) (hc : 0 < c) (hh : 0 < h) (hw : 0 < w) {lr η a w₂ w₃ w₄ w₅ : ℝ} (ha : 0 ≤ a) (hx : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hw₅ : 0 ≤ w₅) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (hlr : 0 ≤ lr) (hη : 0 ≤ η) (hgh : ∀ (idx : Fin (c * ic * kH * kW)), |gh idx - gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| ≤ η) (hm1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), a * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * η)) < |(conv2d W₁ b₁ x₀).flatten k|) (hm2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), ↑(c * kH * kW) * (w₂ * (a * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * η)))) < |(conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten k|) (hmq : MaxPool2MarginQ (↑(c * kH * kW) * (w₂ * (a * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * η))))) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten))) (hm3 : ∀ (l : Fin d₃), w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * η)))))) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten)) l|) (hm4 : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * η)))))))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten)))) q|) (hsmall : 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * η))))))))))) < 1) (h1 : lr * η * ∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| ≤ (lr * ∑ idx : Fin (c * ic * kH * kW), gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx ^ 2) / 4) (h2 : 2 * ↑nC * ↑(2 * h * (2 * w)) ^ 2 * ↑(c * kH * kW) ^ 2 * ↑d₃ ^ 2 * ↑d₄ ^ 2 * w₂ ^ 2 * w₃ ^ 2 * w₄ ^ 2 * w₅ ^ 2 * a ^ 2 / (1 - 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * η)))))))))))) * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * η)) ^ 2 ≤ (lr * ∑ idx : Fin (c * ic * kH * kW), gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx ^ 2) / 4) : crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (W₁.flatten - lr • gh)) b₁ x₀).flatten))).flatten))))))) label ≤ crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten W₁.flatten) b₁ x₀).flatten))).flatten))))))) label - (lr * ∑ idx : Fin (c * ic * kH * kW), gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx ^ 2) / 2

One inexact SGD step on the CNN's FIRST conv kernel provably decreases the cross-entropy loss. The deepest rung: the step crosses relu₁, conv2 (as a function of its input — the point-free tap Jacobian with locality factor c·kH·kW·w₂), relu₂, the pool, and the 3-dense head. Under the FIVE margins at the step radius D = lr·(‖∇L‖₁ + |kernel|·η), every mask and the pool's routing pattern freeze along the step, and the loss drops by ≥ lr·‖∇L‖₂²/2. With this, every conv kernel of the Chapter-3 CNN has a proven descent statement.

theorem Proofs.cnn_conv1_float_sgd_descends {ic c h w d₃ d₄ nC kH kW : ℕ} (M : FloatModel) (W₁ : Kernel4 c ic kH kW) (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (fexp : ℝ → ℝ) {lr a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp : ℝ} (hc : 0 < c) (hh : 0 < h) (hw : 0 < w) (ha : 0 ≤ a) (hw₁ : 0 ≤ w₁) (hβ₁ : 0 ≤ β₁) (hw₂ : 0 ≤ w₂) (hβ₂ : 0 ≤ β₂) (hw₃ : 0 ≤ w₃) (hβ₃ : 0 ≤ β₃) (hw₄ : 0 ≤ w₄) (hβ₄ : 0 ≤ β₄) (hw₅ : 0 ≤ w₅) (hβ₅ : 0 ≤ β₅) (hlr : 0 ≤ lr) (heexp0 : 0 ≤ eexp) (heexp1 : eexp ≤ 1) (hfexp : ∀ (t : ℝ), |fexp t - Real.exp t| ≤ eexp * Real.exp t) (hρ1 : FloatModel.smRho M.u eexp nC < 1) (hx₀ : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (hW₁ : ∀ (o : Fin c) (cc : Fin ic) (kh : Fin kH) (kw : Fin kW), |W₁ o cc kh kw| ≤ w₁) (hb₁ : ∀ (o : Fin c), |b₁ o| ≤ β₁) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hb₂ : ∀ (o : Fin c), |b₂ o| ≤ β₂) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hb₃ : ∀ (j : Fin d₃), |b₃ j| ≤ β₃) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hb₄ : ∀ (j : Fin d₄), |b₄ j| ≤ β₄) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (hb₅ : ∀ (j : Fin nC), |b₅ j| ≤ β₅) (hr1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0 < |(conv2d W₁ b₁ x₀).flatten k|) (hr2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0) < |(conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten k|) (hrPool : MaxPool2MarginQ (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0)) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten))) (hr3 : ∀ (l : Fin d₃), FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a)) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0)) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten)) l|) (hr4 : ∀ (q : Fin d₄), FloatModel.layerBudget M.u d₃ w₄ β₄ (FloatModel.layerAct (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a))) (FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a)) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten)))) q|) (hm1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), a * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * M.cnnConv1GradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)) < |(conv2d W₁ b₁ x₀).flatten k|) (hm2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), ↑(c * kH * kW) * (w₂ * (a * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * M.cnnConv1GradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)))) < |(conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten k|) (hmq : MaxPool2MarginQ (↑(c * kH * kW) * (w₂ * (a * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * M.cnnConv1GradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp))))) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten))) (hm3 : ∀ (l : Fin d₃), w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * M.cnnConv1GradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)))))) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten)) l|) (hm4 : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * M.cnnConv1GradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)))))))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten)))) q|) (hsmall : 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * M.cnnConv1GradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp))))))))))) < 1) (h1 : lr * M.cnnConv1GradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp * ∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| ≤ (lr * ∑ idx : Fin (c * ic * kH * kW), gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx ^ 2) / 4) (h2 : 2 * ↑nC * ↑(2 * h * (2 * w)) ^ 2 * ↑(c * kH * kW) ^ 2 * ↑d₃ ^ 2 * ↑d₄ ^ 2 * w₂ ^ 2 * w₃ ^ 2 * w₄ ^ 2 * w₅ ^ 2 * a ^ 2 / (1 - 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (a * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * M.cnnConv1GradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)))))))))))) * (lr * (∑ idx : Fin (c * ic * kH * kW), |gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx| + ↑(c * ic * kH * kW) * M.cnnConv1GradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)) ^ 2 ≤ (lr * ∑ idx : Fin (c * ic * kH * kW), gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx ^ 2) / 4) : crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten (W₁.flatten - lr • M.cnnConv1FloatGrad b₁ x₀ W₂ b₂ W₃ b₃ W₄ b₄ W₅ b₅ fexp label W₁.flatten)) b₁ x₀).flatten))).flatten))))))) label ≤ crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten W₁.flatten) b₁ x₀).flatten))).flatten))))))) label - (lr * ∑ idx : Fin (c * ic * kH * kW), gradAt (fun (u' : Vec (c * ic * kH * kW)) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d (Kernel4.unflatten u') b₁ x₀).flatten))).flatten))))))) label) W₁.flatten idx ^ 2) / 2

One binary32 SGD step on the CNN's FIRST conv kernel provably decreases the cross-entropy loss — with NO abstract gradient-accuracy parameter (Increment 4 capstone, the deepest descent statement). The conv-1 peer of cnn_conv2_float_sgd_descends: the gradient is the actual binary32 W₁ gradient M.cnnConv1FloatGrad …, accuracy proven by cnn_conv1_grad_close (η := cnnConv1GradBudget, discharged per kernel entry via k4Idx_surj), wired into the abstract cnn_conv1_sgd_descends. Five per-layer ROUND margins feed the grad-close; the gradient-radius STEP margins + hsmall/h1/h2 feed the drift-freeze and descent geometry. Every conv kernel of the Chapter-3 CNN now has a float-faithful descent statement.

theorem Proofs.conv2d_bias_sub {ic oc h w kH kW : ℕ} (W : Kernel4 oc ic kH kW) (x : Tensor3 ic h w) (b e : Vec oc) (o : Fin oc) (hi : Fin h) (wi : Fin w) : conv2d W (b + e) x o hi wi - conv2d W b x o hi wi = e o

The conv output difference under a bias perturbation, exactly: output (o,hi,wi) moves by e o — conv2d is affine in the bias.

theorem Proofs.conv2d_flat_bias_drift_total {ic oc h w kH kW : ℕ} (W : Kernel4 oc ic kH kW) (x : Tensor3 ic h w) (b e : Vec oc) (k : Fin (oc * h * w)) : |(conv2d W (b + e) x).flatten k - (conv2d W b x).flatten k| ≤ ∑ idx : Fin oc, |e idx|

Per-entry conv drift under a bias perturbation: the perturbation's own entry — no a factor, no kernel mass. Flat-index form.

theorem Proofs.conv2d_flat_bias_drift_sum {ic oc h w kH kW : ℕ} (W : Kernel4 oc ic kH kW) (x : Tensor3 ic h w) (b e : Vec oc) : ∑ k : Fin (oc * h * w), |(conv2d W (b + e) x).flatten k - (conv2d W b x).flatten k| ≤ ↑(h * w) * ∑ idx : Fin oc, |e idx|

ℓ1 conv bias drift: summed over all output entries, at most (h·w)·‖e‖₁ — one bias entry feeds every spatial position of its channel.

theorem Proofs.conv2d_bias_pdiv {ic oc h w kH kW : ℕ} (W : Kernel4 oc ic kH kW) (x : Tensor3 ic h w) (b : Vec oc) (o co : Fin oc) (hi : Fin h) (wi : Fin w) : pdiv (fun (b' : Vec oc) => (conv2d W b' x).flatten) b o (t3Idx co hi wi) = if co = o then 1 else 0

Closed form of the conv bias-map pdiv — extracted from the certified VJP (conv2d_bias_grad_has_vjp) by contracting its .correct field against a basis vector, exactly as conv2d_weight_pdiv. Bias entry o touches output (co,hi,wi) iff co = o, with coefficient 1 — the Kronecker channel indicator. Point-free (the bias map is affine), so along a step segment only the head gradient moves.

theorem Proofs.cnnb2_pool_l1_drift {c h w kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (x₁ : Tensor3 c (2 * h) (2 * w)) (b e : Vec c) : ∑ q : Fin (c * h * w), |maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ (b + e) x₁).flatten) q - maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten) q| ≤ ↑(2 * h * (2 * w)) * ∑ idx : Fin c, |e idx|

Pooled ℓ1 drift under a conv2 bias perturbation: conv (ℓ1, spatial multiplicity, no a) → relu (contraction) → pool (contraction).

theorem Proofs.cnnb2_postrelu_close {c h w kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (x₁ : Tensor3 c (2 * h) (2 * w)) (b e : Vec c) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) : |Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ (b + e) x₁).flatten) ci hi wi - Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten) ci hi wi| ≤ ∑ idx : Fin c, |e idx|

Per-entry POST-relu tensor drift under a conv2 bias perturbation — the form the pool margin consumes.

theorem Proofs.cnnb2_z3_drift {c h w d₃ kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) {w₃ : ℝ} (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (b e : Vec c) (l : Fin d₃) : |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ (b + e) x₁).flatten)) l - dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)) l| ≤ w₃ * (↑(2 * h * (2 * w)) * ∑ idx : Fin c, |e idx|)

Per-entry drift of the relu₃ pre-activation, conv2-bias rung.

theorem Proofs.cnnb2_z4_drift {c h w d₃ d₄ kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) {w₃ w₄ : ℝ} (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (b e : Vec c) (q : Fin d₄) : |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ (b + e) x₁).flatten)))) q - dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)))) q| ≤ w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * ∑ idx : Fin c, |e idx|)))

Per-entry drift of the relu₄ pre-activation, conv2-bias rung.

theorem Proofs.cnnb2_logit_drift {c h w d₃ d₄ nC kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) {w₃ w₄ w₅ : ℝ} (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hw₅ : 0 ≤ w₅) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (b e : Vec c) (k : Fin nC) : |dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ (b + e) x₁).flatten)))))) k - dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)))))) k| ≤ w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * ∑ idx : Fin c, |e idx|)))))

Logit drift through the whole conv2-bias chain.

theorem Proofs.cnnb2_margin2_keeps_offkink {c h w kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (x₁ : Tensor3 c (2 * h) (2 * w)) {D : ℝ} (b e : Vec c) (he : ∑ idx : Fin c, |e idx| ≤ D) (hm : ∀ (k : Fin (c * (2 * h) * (2 * w))), D < |(conv2d W₂ b x₁).flatten k|) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (k : Fin (c * (2 * h) * (2 * w))) : (conv2d W₂ (b + t • e) x₁).flatten k ≠ 0 ∧ (0 < (conv2d W₂ (b + t • e) x₁).flatten k ↔ 0 < (conv2d W₂ b x₁).flatten k)

The relu₂ margin (at the bias radius D) keeps the conv pre-activation off the kink along the whole step segment.

theorem Proofs.cnnb2_postrelu_close_seg {c h w kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (x₁ : Tensor3 c (2 * h) (2 * w)) {D : ℝ} (b e : Vec c) (he : ∑ idx : Fin c, |e idx| ≤ D) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) : |Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ (b + t • e) x₁).flatten) ci hi wi - Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten) ci hi wi| ≤ D

The POST-relu tensor stays within the bias-rung pool margin radius D along the whole step segment.

theorem Proofs.cnnb2_margin3_keeps_offkink {c h w d₃ kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) {w₃ D : ℝ} (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (b e : Vec c) (he : ∑ idx : Fin c, |e idx| ≤ D) (hm : ∀ (l : Fin d₃), w₃ * (↑(2 * h * (2 * w)) * D) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)) l|) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (l : Fin d₃) : dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ (b + t • e) x₁).flatten)) l ≠ 0 ∧ (0 < dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ (b + t • e) x₁).flatten)) l ↔ 0 < dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)) l)

The relu₃ margin (at the bias radius) keeps the first head pre-activation off the kink along the whole step segment.

theorem Proofs.cnnb2_margin4_keeps_offkink {c h w d₃ d₄ kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) {w₃ w₄ D : ℝ} (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (b e : Vec c) (he : ∑ idx : Fin c, |e idx| ≤ D) (hm : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * D))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)))) q|) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (q : Fin d₄) : dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ (b + t • e) x₁).flatten)))) q ≠ 0 ∧ (0 < dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ (b + t • e) x₁).flatten)))) q ↔ 0 < dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)))) q)

The relu₄ margin (at the bias radius) keeps the second head pre-activation off the kink along the whole step segment.

theorem Proofs.cnn_conv2_bias_loss_differentiableAt {c h w d₃ d₄ nC kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hc : 0 < c) (hh : 0 < h) (hw : 0 < w) (b : Vec c) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d W₂ b x₁).flatten k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)))) q ≠ 0) : DifferentiableAt ℝ (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b

The loss-of-conv2-bias map is differentiable at any four-condition point.

theorem Proofs.cnn_conv2_bias_loss_gradAt {c h w d₃ d₄ nC kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hh : 0 < h) (hw : 0 < w) (b : Vec c) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d W₂ b x₁).flatten k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)))) q ≠ 0) (o : Fin c) : gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b o = ∑ ci : Fin c, ∑ hi : Fin (2 * h), ∑ wi : Fin (2 * w), (if ci = o then 1 else 0) * ((if (conv2d W₂ b x₁).flatten (t3Idx ci hi wi) > 0 then 1 else 0) * if MaxPool2IsArgmax (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)) ci hi wi then ∑ l : Fin d₃, W₃ (t3Idx ci (winRow hi) (winCol wi)) l * ((if dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)) l > 0 then 1 else 0) * ∑ q : Fin d₄, W₄ l q * ((if dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)))) q > 0 then 1 else 0) * ∑ k : Fin nC, W₅ q k * (softmax nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten))))))) k - oneHot nC label k))) else 0)

Closed form of the conv2 bias loss gradient at any four-margin point — the EXISTING fold conv_bias_total_loss_grad_fold contracted with the pool-collapsed head gradient (pool_relu_input_grad, reused verbatim) and the Kronecker bias Jacobian (conv2d_bias_pdiv).

theorem Proofs.cnn_conv2_bias_loss_grad_lipschitz {c h w d₃ d₄ nC kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hh : 0 < h) (hw : 0 < w) {w₃ w₄ w₅ D : ℝ} (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hw₅ : 0 ≤ w₅) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (b d : Vec c) (hd : ∑ idx : Fin c, |d idx| ≤ D) (hm2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), D < |(conv2d W₂ b x₁).flatten k|) (hmq : MaxPool2MarginQ D (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten))) (hm3 : ∀ (l : Fin d₃), w₃ * (↑(2 * h * (2 * w)) * D) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)) l|) (hm4 : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * D))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)))) q|) (hsmall : 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * D)))))) < 1) (t : ℝ) (ht : t ∈ Set.Icc 0 1) (o : Fin c) : |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) (b + t • d) o - gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b o| ≤ 2 * ↑nC * ↑(2 * h * (2 * w)) ^ 2 * ↑d₃ ^ 2 * ↑d₄ ^ 2 * w₃ ^ 2 * w₄ ^ 2 * w₅ ^ 2 / (1 - 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * D))))))) * (t * D)

Segment-Lipschitz gradient for the conv2-bias loss, explicit constant. The kernel-rung argument with the conv stage's a·D radius replaced by the bare D — the bias Jacobian is a Kronecker indicator with row mass (2h)·(2w), no input bound. Constant: the kernel constant with a² ↦ 1.

theorem Proofs.cnn_conv2_bias_sgd_descends {c h w d₃ d₄ nC kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (gh : Vec c) (hc : 0 < c) (hh : 0 < h) (hw : 0 < w) {lr η w₃ w₄ w₅ : ℝ} (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hw₅ : 0 ≤ w₅) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (hlr : 0 ≤ lr) (hη : 0 ≤ η) (hgh : ∀ (o : Fin c), |gh o - gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| ≤ η) (hm2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), lr * (∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| + ↑c * η) < |(conv2d W₂ b₂ x₁).flatten k|) (hmq : MaxPool2MarginQ (lr * (∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| + ↑c * η)) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten))) (hm3 : ∀ (l : Fin d₃), w₃ * (↑(2 * h * (2 * w)) * (lr * (∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| + ↑c * η))) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten)) l|) (hm4 : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (lr * (∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| + ↑c * η))))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten)))) q|) (hsmall : 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (lr * (∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| + ↑c * η)))))))) < 1) (h1 : lr * η * ∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| ≤ (lr * ∑ o : Fin c, gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o ^ 2) / 4) (h2 : 2 * ↑nC * ↑(2 * h * (2 * w)) ^ 2 * ↑d₃ ^ 2 * ↑d₄ ^ 2 * w₃ ^ 2 * w₄ ^ 2 * w₅ ^ 2 / (1 - 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (lr * (∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| + ↑c * η))))))))) * (lr * (∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| + ↑c * η)) ^ 2 ≤ (lr * ∑ o : Fin c, gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o ^ 2) / 4) : crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ (b₂ - lr • gh) x₁).flatten))))))) label ≤ crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten))))))) label - (lr * ∑ o : Fin c, gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o ^ 2) / 2

One inexact SGD step on the CNN's second conv BIAS provably decreases the cross-entropy loss. The conv2-kernel capstone with the bias-rung radii: the four margins at the step radius D = lr·(‖∇L‖₁ + c·η) carry no input bound a (the bias Jacobian is a Kronecker indicator), and the parameter needs no flatten/unflatten plumbing — the bias IS a vector.

theorem Proofs.cnnb1_postrelu1_close {ic c h w kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) (b e : Vec c) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) : |Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b + e) x₀).flatten) ci hi wi - Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten) ci hi wi| ≤ ∑ idx : Fin c, |e idx|

POST-relu₁ tensor drift under a conv1 bias perturbation.

theorem Proofs.cnnb1_z2_entry_drift {ic c h w kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) {w₂ : ℝ} (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (b e : Vec c) (k : Fin (c * (2 * h) * (2 * w))) : |(conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b + e) x₀).flatten))).flatten k - (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten k| ≤ ↑(c * kH * kW) * (w₂ * ∑ idx : Fin c, |e idx|)

Per-entry conv2-preactivation drift under a conv1 bias perturbation: the perturbation crosses conv2 as a function of its INPUT, picking up the locality factor c·kH·kW·w₂.

theorem Proofs.cnnb1_postrelu2_close {ic c h w kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) {w₂ : ℝ} (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (b e : Vec c) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) : |Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b + e) x₀).flatten))).flatten) ci hi wi - Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten) ci hi wi| ≤ ↑(c * kH * kW) * (w₂ * ∑ idx : Fin c, |e idx|)

POST-relu₂ tensor drift under a conv1 bias perturbation — what the pool margin consumes on the conv1-bias rung.

theorem Proofs.cnnb1_pool_l1_drift {ic c h w kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) {w₂ : ℝ} (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (b e : Vec c) : ∑ q : Fin (c * h * w), |maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b + e) x₀).flatten))).flatten) q - maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten) q| ≤ ↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * ∑ idx : Fin c, |e idx|))

Pooled ℓ1 drift under a conv1 bias perturbation: conv1 (ℓ1, spatial multiplicity, no a) → relu → conv2-as-input (ℓ1, LOCALITY multiplicity c·kH·kW) → relu → pool.

theorem Proofs.cnnb1_z3_drift {ic c h w d₃ kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) {w₂ w₃ : ℝ} (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (b e : Vec c) (l : Fin d₃) : |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b + e) x₀).flatten))).flatten)) l - dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)) l| ≤ w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * ∑ idx : Fin c, |e idx|)))

Per-entry drift of the relu₃ pre-activation, conv1-bias rung.

theorem Proofs.cnnb1_z4_drift {ic c h w d₃ d₄ kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) {w₂ w₃ w₄ : ℝ} (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (b e : Vec c) (q : Fin d₄) : |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b + e) x₀).flatten))).flatten)))) q - dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)))) q| ≤ w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * ∑ idx : Fin c, |e idx|)))))

Per-entry drift of the relu₄ pre-activation, conv1-bias rung.

theorem Proofs.cnnb1_logit_drift {ic c h w d₃ d₄ nC kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) {w₂ w₃ w₄ w₅ : ℝ} (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hw₅ : 0 ≤ w₅) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (b e : Vec c) (k : Fin nC) : |dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b + e) x₀).flatten))).flatten)))))) k - dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)))))) k| ≤ w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * ∑ idx : Fin c, |e idx|)))))))

Logit drift through the whole conv1-bias chain.

theorem Proofs.cnnb1_margin1_keeps_offkink {ic c h w kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) {D : ℝ} (b e : Vec c) (he : ∑ idx : Fin c, |e idx| ≤ D) (hm : ∀ (k : Fin (c * (2 * h) * (2 * w))), D < |(conv2d W₁ b x₀).flatten k|) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (k : Fin (c * (2 * h) * (2 * w))) : (conv2d W₁ (b + t • e) x₀).flatten k ≠ 0 ∧ (0 < (conv2d W₁ (b + t • e) x₀).flatten k ↔ 0 < (conv2d W₁ b x₀).flatten k)

The relu₁ margin (at the bias radius D) keeps the conv1 pre-activation off the kink.

theorem Proofs.cnnb1_margin2_keeps_offkink {ic c h w kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) {w₂ D : ℝ} (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (b e : Vec c) (he : ∑ idx : Fin c, |e idx| ≤ D) (hm : ∀ (k : Fin (c * (2 * h) * (2 * w))), ↑(c * kH * kW) * (w₂ * D) < |(conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten k|) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (k : Fin (c * (2 * h) * (2 * w))) : (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b + t • e) x₀).flatten))).flatten k ≠ 0 ∧ (0 < (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b + t • e) x₀).flatten))).flatten k ↔ 0 < (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten k)

The relu₂ margin (at the conv1-bias radius) keeps the conv2 pre-activation off the kink.

theorem Proofs.cnnb1_postrelu2_close_seg {ic c h w kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) {w₂ D : ℝ} (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (b e : Vec c) (he : ∑ idx : Fin c, |e idx| ≤ D) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) : |Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b + t • e) x₀).flatten))).flatten) ci hi wi - Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten) ci hi wi| ≤ ↑(c * kH * kW) * (w₂ * D)

The POST-relu₂ tensor stays within the conv1-bias pool margin radius along the whole step segment.

theorem Proofs.cnnb1_margin3_keeps_offkink {ic c h w d₃ kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) {w₂ w₃ D : ℝ} (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (b e : Vec c) (he : ∑ idx : Fin c, |e idx| ≤ D) (hm : ∀ (l : Fin d₃), w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * D))) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)) l|) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (l : Fin d₃) : dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b + t • e) x₀).flatten))).flatten)) l ≠ 0 ∧ (0 < dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b + t • e) x₀).flatten))).flatten)) l ↔ 0 < dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)) l)

The relu₃ margin (at the conv1-bias radius) keeps the first head pre-activation off the kink.

theorem Proofs.cnnb1_margin4_keeps_offkink {ic c h w d₃ d₄ kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) {w₂ w₃ w₄ D : ℝ} (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (b e : Vec c) (he : ∑ idx : Fin c, |e idx| ≤ D) (hm : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * D))))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)))) q|) (t : ℝ) (ht0 : 0 ≤ t) (ht1 : t ≤ 1) (q : Fin d₄) : dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b + t • e) x₀).flatten))).flatten)))) q ≠ 0 ∧ (0 < dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b + t • e) x₀).flatten))).flatten)))) q ↔ 0 < dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)))) q)

The relu₄ margin (at the conv1-bias radius) keeps the second head pre-activation off the kink.

theorem Proofs.cnn_conv1_bias_loss_differentiableAt {ic c h w d₃ d₄ nC kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hc : 0 < c) (hh : 0 < h) (hw : 0 < w) (b : Vec c) (hz1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d W₁ b x₀).flatten k ≠ 0) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)))) q ≠ 0) : DifferentiableAt ℝ (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b

The loss-of-conv1-bias map is differentiable at any five-condition point.

theorem Proofs.cnn_conv1_bias_loss_gradAt {ic c h w d₃ d₄ nC kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hh : 0 < h) (hw : 0 < w) (b : Vec c) (hz1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d W₁ b x₀).flatten k ≠ 0) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)))) q ≠ 0) (o : Fin c) : gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b o = ∑ ci : Fin c, ∑ hi : Fin (2 * h), ∑ wi : Fin (2 * w), (if ci = o then 1 else 0) * ((if (conv2d W₁ b x₀).flatten (t3Idx ci hi wi) > 0 then 1 else 0) * ∑ co : Fin c, ∑ ho : Fin (2 * h), ∑ wo : Fin (2 * w), convTap W₂ ci hi wi co ho wo * ((if (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten (t3Idx co ho wo) > 0 then 1 else 0) * if MaxPool2IsArgmax (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)) co ho wo then ∑ l : Fin d₃, W₃ (t3Idx co (winRow ho) (winCol wo)) l * ((if dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)) l > 0 then 1 else 0) * ∑ q : Fin d₄, W₄ l q * ((if dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)))) q > 0 then 1 else 0) * ∑ k : Fin nC, W₅ q k * (softmax nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten))))))) k - oneHot nC label k))) else 0))

Closed form of the conv1 bias loss gradient at any five-margin point — the bias fold at conv1, contracted with the conv1 head gradient (cnn1_pool_head_input_grad, reused verbatim): the Kronecker bias Jacobian times relu₁'s mask times the point-free conv2 tap Jacobian times the pool-collapsed head.

theorem Proofs.cnn_conv1_bias_loss_grad_lipschitz {ic c h w d₃ d₄ nC kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hh : 0 < h) (hw : 0 < w) {w₂ w₃ w₄ w₅ D : ℝ} (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hw₅ : 0 ≤ w₅) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (b d : Vec c) (hd : ∑ idx : Fin c, |d idx| ≤ D) (hm1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), D < |(conv2d W₁ b x₀).flatten k|) (hm2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), ↑(c * kH * kW) * (w₂ * D) < |(conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten k|) (hmq : MaxPool2MarginQ (↑(c * kH * kW) * (w₂ * D)) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten))) (hm3 : ∀ (l : Fin d₃), w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * D))) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)) l|) (hm4 : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * D))))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)))) q|) (hsmall : 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * D)))))))) < 1) (t : ℝ) (ht : t ∈ Set.Icc 0 1) (o : Fin c) : |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) (b + t • d) o - gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b o| ≤ 2 * ↑nC * ↑(2 * h * (2 * w)) ^ 2 * ↑(c * kH * kW) ^ 2 * ↑d₃ ^ 2 * ↑d₄ ^ 2 * w₂ ^ 2 * w₃ ^ 2 * w₄ ^ 2 * w₅ ^ 2 / (1 - 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * D))))))))) * (t * D)

Segment-Lipschitz gradient for the conv1-bias loss, explicit constant. The conv1-kernel argument with the conv1 stage's a·D radii replaced by the bare D — the bias Jacobian is a Kronecker indicator with row mass (2h)·(2w). Constant: the conv1-kernel constant with a² ↦ 1.

theorem Proofs.cnn_conv1_bias_sgd_descends {ic c h w d₃ d₄ nC kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (gh : Vec c) (hc : 0 < c) (hh : 0 < h) (hw : 0 < w) {lr η w₂ w₃ w₄ w₅ : ℝ} (hw₂ : 0 ≤ w₂) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hw₃ : 0 ≤ w₃) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hw₄ : 0 ≤ w₄) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hw₅ : 0 ≤ w₅) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (hlr : 0 ≤ lr) (hη : 0 ≤ η) (hgh : ∀ (idx : Fin c), |gh idx - gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| ≤ η) (hm1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * η) < |(conv2d W₁ b₁ x₀).flatten k|) (hm2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), ↑(c * kH * kW) * (w₂ * (lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * η))) < |(conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten k|) (hmq : MaxPool2MarginQ (↑(c * kH * kW) * (w₂ * (lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * η)))) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten))) (hm3 : ∀ (l : Fin d₃), w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * η))))) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten)) l|) (hm4 : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * η))))))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten)))) q|) (hsmall : 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * η)))))))))) < 1) (h1 : lr * η * ∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| ≤ (lr * ∑ idx : Fin c, gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx ^ 2) / 4) (h2 : 2 * ↑nC * ↑(2 * h * (2 * w)) ^ 2 * ↑(c * kH * kW) ^ 2 * ↑d₃ ^ 2 * ↑d₄ ^ 2 * w₂ ^ 2 * w₃ ^ 2 * w₄ ^ 2 * w₅ ^ 2 / (1 - 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * η))))))))))) * (lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * η)) ^ 2 ≤ (lr * ∑ idx : Fin c, gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx ^ 2) / 4) : crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b₁ - lr • gh) x₀).flatten))).flatten))))))) label ≤ crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten))))))) label - (lr * ∑ idx : Fin c, gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx ^ 2) / 2

One inexact SGD step on the CNN's FIRST conv BIAS provably decreases the cross-entropy loss. The conv1-kernel capstone with the bias-rung radii: the FIVE margins at the step radius D = lr·(‖∇L‖₁ + c·η) carry no input bound a (the bias Jacobian is a Kronecker indicator) and the parameter needs no flatten/unflatten plumbing. With this theorem every parameter of the Chapter-3 CNN — both conv kernels, both conv biases, and the three dense layers (weights and biases via the MLP rungs) — has a proven descent statement.

theorem Proofs.FloatModel.sum_perturbed_close {n : ℕ} (M : FloatModel) (Bt B : Vec n) {Ct eB : ℝ} (hBt : ∀ (i : Fin n), |Bt i| ≤ Ct) (hB : ∀ (i : Fin n), |Bt i - B i| ≤ eB) : |M.sum Bt - ∑ i : Fin n, B i| ≤ ((1 + M.u) ^ (n + 1) - 1) * (↑n * Ct) + ↑n * eB

Float sum against a perturbed summand — the conv-BIAS grad-close's final reduction, the M.sum peer of dot_perturbed_close (no left operand A: the bias Jacobian is the bare channel indicator, so the contraction is a plain reduction, not a dot). M.sum B̃ vs the certified ∑ Bᵢ splits into the Higham sum rounding on B̃ (sum_close, fan-in n) plus the per-entry drift |B̃ᵢ − Bᵢ| ≤ eB. With |B̃ᵢ| ≤ Ct the bound is closed-form.

noncomputable def Proofs.FloatModel.cnnConv2BiasFloatGrad {c h w d₃ d₄ nC kH kW : ℕ} (M : FloatModel) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (fexp : ℝ → ℝ) (label : Fin nC) : Vec c

The binary32 conv-2 bias gradient the rendered trainer computes — the bias peer of cnnConv2FloatGrad: at output channel o it is the float SUM M.sum (cotWin c̃Conv o) of the same float conv-2-output cotangent slab (the bias Jacobian is the channel indicator, so there is no convPadWin left operand and no per-slot kernel index — one entry per channel).

Equations

• One or more equations did not get rendered due to their size.

theorem Proofs.cnn_conv2_bias_loss_gradAt_reluMask {c h w d₃ d₄ nC kH kW : ℕ} (W₂ : Kernel4 c c kH kW) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hh : 0 < h) (hw : 0 < w) (b : Vec c) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d W₂ b x₁).flatten k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)))) q ≠ 0) (o : Fin c) : gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b o = ∑ s : Fin (2 * h * (2 * w)), cotWin (fun (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) => (if (conv2d W₂ b x₁).flatten (t3Idx ci hi wi) > 0 then 1 else 0) * if MaxPool2IsArgmax (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten)) ci hi wi then dense (fun (j : Fin d₃) (i' : Fin (c * h * w)) => W₃ i' j) (fun (x : Fin (c * h * w)) => 0) (FloatModel.reluMask (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten))) (dense (fun (j : Fin d₄) (i' : Fin d₃) => W₄ i' j) (fun (x : Fin d₃) => 0) (FloatModel.reluMask (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten))))) (dense (fun (j : Fin nC) (i' : Fin d₄) => W₅ i' j) (fun (x : Fin d₄) => 0) fun (k : Fin nC) => softmax nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b x₁).flatten))))))) k - oneHot nC label k)))) (t3Idx ci (winRow hi) (winCol wi)) else 0) o s

The certified conv-2 BIAS loss gradient, restated as the spatial SUM of the reluMask-form cotangent — the bias peer of cnn_conv2_loss_gradAt_reluMask. The 3-dense head collapses via head3_cot_reluMask; the channel-Kronecker Jacobian if ci = o collapses the ∑ ci to ci = o (Finset.sum_eq_single); the remaining spatial ∑ hi wi is packaged as ∑ s, cotWin c o s (convBiasGrad_eq_sum) — the quantity the rendered trainer's float bias SUM rounds.

noncomputable def Proofs.FloatModel.cnnConv2BiasGradBudget (M : FloatModel) (c h w d₃ d₄ nC kH kW : ℕ) (a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp : ℝ) : ℝ

The conv-2 bias-gradient grad-close budget — cnnConv2GradBudget with the a· input factor stripped (the bias Jacobian carries no input window): the spatial sum's Higham γ over (2h)·(2w) against the float-cotangent magnitude (cnnConv2CotMag + cnnConv2CotBudget) plus the per-entry cotangent drift cnnConv2CotBudget. The cotangent chain is the exact conv-2-input (aX2 = a, eX2 = 0) instance of the factored cnnConv2CotBudget / cnnConv2CotMag.

Equations

• One or more equations did not get rendered due to their size.

theorem Proofs.cnn_conv2_bias_grad_close {c h w d₃ d₄ nC kH kW : ℕ} (M : FloatModel) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (fexp : ℝ → ℝ) {a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp : ℝ} (hh : 0 < h) (hw : 0 < w) (ha : 0 ≤ a) (hw₂ : 0 ≤ w₂) (hβ₂ : 0 ≤ β₂) (hw₃ : 0 ≤ w₃) (hβ₃ : 0 ≤ β₃) (hw₄ : 0 ≤ w₄) (hβ₄ : 0 ≤ β₄) (hw₅ : 0 ≤ w₅) (hβ₅ : 0 ≤ β₅) (heexp0 : 0 ≤ eexp) (heexp1 : eexp ≤ 1) (hfexp : ∀ (t : ℝ), |fexp t - Real.exp t| ≤ eexp * Real.exp t) (hρ1 : FloatModel.smRho M.u eexp nC < 1) (hx₁ : ∀ (ci : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₁ ci i j| ≤ a) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hb₂ : ∀ (o : Fin c), |b₂ o| ≤ β₂) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hb₃ : ∀ (j : Fin d₃), |b₃ j| ≤ β₃) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hb₄ : ∀ (j : Fin d₄), |b₄ j| ≤ β₄) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (hb₅ : ∀ (j : Fin nC), |b₅ j| ≤ β₅) (hmarginConv : ∀ (k : Fin (c * (2 * h) * (2 * w))), FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0 < |(conv2d W₂ b₂ x₁).flatten k|) (hmarginPool : MaxPool2MarginQ (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten))) (hmargin3 : ∀ (l : Fin d₃), FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ a) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten)) l|) (hmargin4 : ∀ (q : Fin d₄), FloatModel.layerBudget M.u d₃ w₄ β₄ (FloatModel.layerAct (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ a)) (FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ a) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0)) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten)))) q|) (o : Fin c) : |M.cnnConv2BiasFloatGrad W₂ b₂ x₁ W₃ b₃ W₄ b₄ W₅ b₅ fexp label o - gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| ≤ M.cnnConv2BiasGradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp

The binary32 conv-2 BIAS gradient is within an explicit budget of the certified one — the bias peer of cnn_conv2_grad_close, built on the factored conv-2 cotangent chain at the exact conv-2 input x₁ (cnn_conv2_cot_close with aX2 = a, eX2 = 0) and the spatial-SUM core sum_perturbed_close. The bridge cnn_conv2_bias_loss_gradAt_reluMask turns the gradAt into the sum the float bias gradient rounds. Four quantitative margins (conv-output, pool POST-relu, z̃₃, z̃₄) freeze the routing.

theorem Proofs.cnn_conv2_bias_float_sgd_descends {c h w d₃ d₄ nC kH kW : ℕ} (M : FloatModel) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (x₁ : Tensor3 c (2 * h) (2 * w)) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (fexp : ℝ → ℝ) {lr a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp : ℝ} (hc : 0 < c) (hh : 0 < h) (hw : 0 < w) (ha : 0 ≤ a) (hw₂ : 0 ≤ w₂) (hβ₂ : 0 ≤ β₂) (hw₃ : 0 ≤ w₃) (hβ₃ : 0 ≤ β₃) (hw₄ : 0 ≤ w₄) (hβ₄ : 0 ≤ β₄) (hw₅ : 0 ≤ w₅) (hβ₅ : 0 ≤ β₅) (hlr : 0 ≤ lr) (heexp0 : 0 ≤ eexp) (heexp1 : eexp ≤ 1) (hfexp : ∀ (t : ℝ), |fexp t - Real.exp t| ≤ eexp * Real.exp t) (hρ1 : FloatModel.smRho M.u eexp nC < 1) (hx : ∀ (cc : Fin c) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₁ cc i j| ≤ a) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hb₂ : ∀ (o : Fin c), |b₂ o| ≤ β₂) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hb₃ : ∀ (j : Fin d₃), |b₃ j| ≤ β₃) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hb₄ : ∀ (j : Fin d₄), |b₄ j| ≤ β₄) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (hb₅ : ∀ (j : Fin nC), |b₅ j| ≤ β₅) (hmarginConv : ∀ (k : Fin (c * (2 * h) * (2 * w))), FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0 < |(conv2d W₂ b₂ x₁).flatten k|) (hmarginPool : MaxPool2MarginQ (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten))) (hmargin3 : ∀ (l : Fin d₃), FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ a) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten)) l|) (hmargin4 : ∀ (q : Fin d₄), FloatModel.layerBudget M.u d₃ w₄ β₄ (FloatModel.layerAct (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ a)) (FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ a) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ a 0)) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten)))) q|) (hm2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), lr * (∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| + ↑c * M.cnnConv2BiasGradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp) < |(conv2d W₂ b₂ x₁).flatten k|) (hmq : MaxPool2MarginQ (lr * (∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| + ↑c * M.cnnConv2BiasGradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten))) (hm3 : ∀ (l : Fin d₃), w₃ * (↑(2 * h * (2 * w)) * (lr * (∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| + ↑c * M.cnnConv2BiasGradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp))) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten)) l|) (hm4 : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (lr * (∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| + ↑c * M.cnnConv2BiasGradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp))))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten)))) q|) (hsmall : 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (lr * (∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| + ↑c * M.cnnConv2BiasGradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)))))))) < 1) (h1 : lr * M.cnnConv2BiasGradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp * ∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| ≤ (lr * ∑ o : Fin c, gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o ^ 2) / 4) (h2 : 2 * ↑nC * ↑(2 * h * (2 * w)) ^ 2 * ↑d₃ ^ 2 * ↑d₄ ^ 2 * w₃ ^ 2 * w₄ ^ 2 * w₅ ^ 2 / (1 - 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(2 * h * (2 * w)) * (lr * (∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| + ↑c * M.cnnConv2BiasGradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp))))))))) * (lr * (∑ o : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o| + ↑c * M.cnnConv2BiasGradBudget c h w d₃ d₄ nC kH kW a w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)) ^ 2 ≤ (lr * ∑ o : Fin c, gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o ^ 2) / 4) : crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ (b₂ - lr • M.cnnConv2BiasFloatGrad W₂ b₂ x₁ W₃ b₃ W₄ b₄ W₅ b₅ fexp label) x₁).flatten))))))) label ≤ crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ x₁).flatten))))))) label - (lr * ∑ o : Fin c, gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b' x₁).flatten))))))) label) b₂ o ^ 2) / 2

One inexact SGD step on the CNN's second conv BIAS provably decreases the cross-entropy loss — the bias peer of cnn_conv2_float_sgd_descends: the gradient is the actual binary32 bias gradient M.cnnConv2BiasFloatGrad …, and its accuracy is proven by cnn_conv2_bias_grad_close (η := cnnConv2BiasGradBudget, discharged per output channel — the bias IS a vector, so no flatten/unflatten plumbing), not assumed. The two rounding-margin families are carried as the honest first cut, exactly as the weight rungs.

noncomputable def Proofs.FloatModel.cnnConv1BiasFloatGrad {ic c h w d₃ d₄ nC kH kW : ℕ} (M : FloatModel) (W₁ : Kernel4 c ic kH kW) (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (fexp : ℝ → ℝ) (label : Fin nC) : Vec c

Equations

• One or more equations did not get rendered due to their size.

theorem Proofs.cnn_conv1_bias_loss_gradAt_reluMask {ic c h w d₃ d₄ nC kH kW : ℕ} (W₁ : Kernel4 c ic kH kW) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (hh : 0 < h) (hw : 0 < w) (b : Vec c) (hz1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d W₁ b x₀).flatten k ≠ 0) (hz2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten k ≠ 0) (hmp : MaxPool2Smooth (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten))) (hz3 : ∀ (l : Fin d₃), dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)) l ≠ 0) (hz4 : ∀ (q : Fin d₄), dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)))) q ≠ 0) (o : Fin c) : gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b o = ∑ s : Fin (2 * h * (2 * w)), cotWin (fun (ci : Fin c) (hi : Fin (2 * h)) (wi : Fin (2 * w)) => (if (conv2d W₁ b x₀).flatten (t3Idx ci hi wi) > 0 then 1 else 0) * ∑ co : Fin c, ∑ ho : Fin (2 * h), ∑ wo : Fin (2 * w), convTap W₂ ci hi wi co ho wo * ((if (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten (t3Idx co ho wo) > 0 then 1 else 0) * if MaxPool2IsArgmax (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten)) co ho wo then dense (fun (j : Fin d₃) (i' : Fin (c * h * w)) => W₃ i' j) (fun (x : Fin (c * h * w)) => 0) (FloatModel.reluMask (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten))) (dense (fun (j : Fin d₄) (i' : Fin d₃) => W₄ i' j) (fun (x : Fin d₃) => 0) (FloatModel.reluMask (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten))))) (dense (fun (j : Fin nC) (i' : Fin d₄) => W₅ i' j) (fun (x : Fin d₄) => 0) fun (k : Fin nC) => softmax nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b x₀).flatten))).flatten))))))) k - oneHot nC label k)))) (t3Idx co (winRow ho) (winCol wo)) else 0)) o s

The certified conv-1 BIAS loss gradient, restated as the spatial SUM of the reluMask-form cotangent — the bias peer of cnn_conv1_loss_gradAt_reluMask, one conv-backward deeper. The head collapses via head3_cot_reluMask, the conv-2 backward stays the explicit ∑ convTap·c₂, and the channel-Kronecker conv-1 Jacobian collapses the ∑ ci (Finset.sum_eq_single + convBiasGrad_eq_sum) to the spatial SUM the float bias gradient rounds.

noncomputable def Proofs.FloatModel.cnnConv1BiasGradBudget (M : FloatModel) (ic c h w d₃ d₄ nC kH kW : ℕ) (a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp : ℝ) : ℝ

The conv-1 bias-gradient grad-close budget — cnnConv1GradBudget with the a· input factors stripped (the bias Jacobian carries no input window): the spatial sum's Higham γ over (2h)·(2w) against the float conv-1 cotangent magnitude C1t = c(2h)(2w)·w₂·CP + eback plus the per-entry drift eback (the conv-2 backward budget). Reuses the factored cnnConv2CotMag /cnnConv2CotBudget at the FLOAT conv-2 input (aX2 = A₁, eX2 = E₁).

Equations

• One or more equations did not get rendered due to their size.

theorem Proofs.cnn_conv1_bias_grad_close {ic c h w d₃ d₄ nC kH kW : ℕ} (M : FloatModel) (W₁ : Kernel4 c ic kH kW) (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (fexp : ℝ → ℝ) {a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp : ℝ} (hh : 0 < h) (hw : 0 < w) (ha : 0 ≤ a) (hw₁ : 0 ≤ w₁) (hβ₁ : 0 ≤ β₁) (hw₂ : 0 ≤ w₂) (hβ₂ : 0 ≤ β₂) (hw₃ : 0 ≤ w₃) (hβ₃ : 0 ≤ β₃) (hw₄ : 0 ≤ w₄) (hβ₄ : 0 ≤ β₄) (hw₅ : 0 ≤ w₅) (hβ₅ : 0 ≤ β₅) (heexp0 : 0 ≤ eexp) (heexp1 : eexp ≤ 1) (hfexp : ∀ (t : ℝ), |fexp t - Real.exp t| ≤ eexp * Real.exp t) (hρ1 : FloatModel.smRho M.u eexp nC < 1) (hx₀ : ∀ (ci : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ ci i j| ≤ a) (hW₁ : ∀ (o : Fin c) (cc : Fin ic) (kh : Fin kH) (kw : Fin kW), |W₁ o cc kh kw| ≤ w₁) (hb₁ : ∀ (o : Fin c), |b₁ o| ≤ β₁) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hb₂ : ∀ (o : Fin c), |b₂ o| ≤ β₂) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hb₃ : ∀ (j : Fin d₃), |b₃ j| ≤ β₃) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hb₄ : ∀ (j : Fin d₄), |b₄ j| ≤ β₄) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (hb₅ : ∀ (j : Fin nC), |b₅ j| ≤ β₅) (hmargin1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0 < |(conv2d W₁ b₁ x₀).flatten k|) (hmargin2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0) < |(conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten k|) (hmarginPool : MaxPool2MarginQ (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0)) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten))) (hmargin3 : ∀ (l : Fin d₃), FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a)) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0)) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten)) l|) (hmargin4 : ∀ (q : Fin d₄), FloatModel.layerBudget M.u d₃ w₄ β₄ (FloatModel.layerAct (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a))) (FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a)) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten)))) q|) (o : Fin c) : |M.cnnConv1BiasFloatGrad W₁ b₁ x₀ W₂ b₂ W₃ b₃ W₄ b₄ W₅ b₅ fexp label o - gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ o| ≤ M.cnnConv1BiasGradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp

The binary32 conv-1 BIAS gradient is within an explicit budget of the certified one — the bias peer of cnn_conv1_grad_close, built on the factored conv-2 cotangent chain at the FLOAT conv-2 input relu(z̃₁) (cnn_conv2_cot_close), the conv-2 backward convTap_back_close, the conv-1 ReLU-mask freeze (mask_scalar_close), and the spatial-SUM core sum_perturbed_close. Five quantitative margins freeze the routing; the bridge cnn_conv1_bias_loss_gradAt_reluMask turns the gradAt into the sum.

theorem Proofs.cnn_conv1_bias_float_sgd_descends {ic c h w d₃ d₄ nC kH kW : ℕ} (M : FloatModel) (W₁ : Kernel4 c ic kH kW) (b₁ : Vec c) (x₀ : Tensor3 ic (2 * h) (2 * w)) (W₂ : Kernel4 c c kH kW) (b₂ : Vec c) (W₃ : Mat (c * h * w) d₃) (b₃ : Vec d₃) (W₄ : Mat d₃ d₄) (b₄ : Vec d₄) (W₅ : Mat d₄ nC) (b₅ : Vec nC) (label : Fin nC) (fexp : ℝ → ℝ) {lr a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp : ℝ} (hc : 0 < c) (hh : 0 < h) (hw : 0 < w) (ha : 0 ≤ a) (hw₁ : 0 ≤ w₁) (hβ₁ : 0 ≤ β₁) (hw₂ : 0 ≤ w₂) (hβ₂ : 0 ≤ β₂) (hw₃ : 0 ≤ w₃) (hβ₃ : 0 ≤ β₃) (hw₄ : 0 ≤ w₄) (hβ₄ : 0 ≤ β₄) (hw₅ : 0 ≤ w₅) (hβ₅ : 0 ≤ β₅) (hlr : 0 ≤ lr) (heexp0 : 0 ≤ eexp) (heexp1 : eexp ≤ 1) (hfexp : ∀ (t : ℝ), |fexp t - Real.exp t| ≤ eexp * Real.exp t) (hρ1 : FloatModel.smRho M.u eexp nC < 1) (hx : ∀ (cc : Fin ic) (i : Fin (2 * h)) (j : Fin (2 * w)), |x₀ cc i j| ≤ a) (hW₁ : ∀ (o : Fin c) (cc : Fin ic) (kh : Fin kH) (kw : Fin kW), |W₁ o cc kh kw| ≤ w₁) (hb₁ : ∀ (o : Fin c), |b₁ o| ≤ β₁) (hW₂ : ∀ (o cc : Fin c) (kh : Fin kH) (kw : Fin kW), |W₂ o cc kh kw| ≤ w₂) (hb₂ : ∀ (o : Fin c), |b₂ o| ≤ β₂) (hW₃ : ∀ (i : Fin (c * h * w)) (j : Fin d₃), |W₃ i j| ≤ w₃) (hb₃ : ∀ (j : Fin d₃), |b₃ j| ≤ β₃) (hW₄ : ∀ (i : Fin d₃) (j : Fin d₄), |W₄ i j| ≤ w₄) (hb₄ : ∀ (j : Fin d₄), |b₄ j| ≤ β₄) (hW₅ : ∀ (i : Fin d₄) (j : Fin nC), |W₅ i j| ≤ w₅) (hb₅ : ∀ (j : Fin nC), |b₅ j| ≤ β₅) (hmargin1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0 < |(conv2d W₁ b₁ x₀).flatten k|) (hmargin2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0) < |(conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten k|) (hmarginPool : MaxPool2MarginQ (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0)) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten))) (hmargin3 : ∀ (l : Fin d₃), FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a)) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0)) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten)) l|) (hmargin4 : ∀ (q : Fin d₄), FloatModel.layerBudget M.u d₃ w₄ β₄ (FloatModel.layerAct (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a))) (FloatModel.layerBudget M.u (c * h * w) w₃ β₃ (FloatModel.layerAct (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a)) (FloatModel.layerBudget M.u (c * kH * kW) w₂ β₂ (FloatModel.layerAct (ic * kH * kW) w₁ β₁ a) (FloatModel.layerBudget M.u (ic * kH * kW) w₁ β₁ a 0))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten)))) q|) (hm1 : ∀ (k : Fin (c * (2 * h) * (2 * w))), lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * M.cnnConv1BiasGradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp) < |(conv2d W₁ b₁ x₀).flatten k|) (hm2 : ∀ (k : Fin (c * (2 * h) * (2 * w))), ↑(c * kH * kW) * (w₂ * (lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * M.cnnConv1BiasGradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp))) < |(conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten k|) (hmq : MaxPool2MarginQ (↑(c * kH * kW) * (w₂ * (lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * M.cnnConv1BiasGradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)))) (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten))) (hm3 : ∀ (l : Fin d₃), w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * M.cnnConv1BiasGradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp))))) < |dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten)) l|) (hm4 : ∀ (q : Fin d₄), w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * M.cnnConv1BiasGradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp))))))) < |dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten)))) q|) (hsmall : 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * M.cnnConv1BiasGradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)))))))))) < 1) (h1 : lr * M.cnnConv1BiasGradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp * ∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| ≤ (lr * ∑ idx : Fin c, gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx ^ 2) / 4) (h2 : 2 * ↑nC * ↑(2 * h * (2 * w)) ^ 2 * ↑(c * kH * kW) ^ 2 * ↑d₃ ^ 2 * ↑d₄ ^ 2 * w₂ ^ 2 * w₃ ^ 2 * w₄ ^ 2 * w₅ ^ 2 / (1 - 2 * (w₅ * (↑d₄ * (w₄ * (↑d₃ * (w₃ * (↑(c * kH * kW) * (w₂ * (↑(2 * h * (2 * w)) * (lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * M.cnnConv1BiasGradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp))))))))))) * (lr * (∑ idx : Fin c, |gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx| + ↑c * M.cnnConv1BiasGradBudget ic c h w d₃ d₄ nC kH kW a w₁ β₁ w₂ β₂ w₃ β₃ w₄ β₄ w₅ β₅ eexp)) ^ 2 ≤ (lr * ∑ idx : Fin c, gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx ^ 2) / 4) : crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ (b₁ - lr • M.cnnConv1BiasFloatGrad W₁ b₁ x₀ W₂ b₂ W₃ b₃ W₄ b₄ W₅ b₅ fexp label) x₀).flatten))).flatten))))))) label ≤ crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b₁ x₀).flatten))).flatten))))))) label - (lr * ∑ idx : Fin c, gradAt (fun (b' : Vec c) => crossEntropy nC (dense W₅ b₅ (relu d₄ (dense W₄ b₄ (relu d₃ (dense W₃ b₃ (maxPoolFlat c h w (relu (c * (2 * h) * (2 * w)) (conv2d W₂ b₂ (Tensor3.unflatten (relu (c * (2 * h) * (2 * w)) (conv2d W₁ b' x₀).flatten))).flatten))))))) label) b₁ idx ^ 2) / 2

One inexact SGD step on the CNN's first conv BIAS provably decreases the cross-entropy loss — the bias peer of cnn_conv1_float_sgd_descends, the deepest descent rung: the gradient is the actual binary32 bias gradient M.cnnConv1BiasFloatGrad …, accuracy proven by cnn_conv1_bias_grad_close (η := cnnConv1BiasGradBudget, discharged per output channel — the bias IS a vector), not assumed. With this, every conv weight AND bias of the Chapter-3 CNN is a float-faithful descent step.