CROWN: a linear-relaxation bound on the seam IBP already certifies through #
IntervalBound.lean concretizes to an interval after every layer, so the box
grows multiplicatively with depth: each layer throws away the correlations
between neurons. CROWN (Zhang et al. 2018) never concretizes in the middle. It
carries a linear function of the input backward through the net — relaxing
each unstable ReLU by a linear lower/upper envelope — and concretizes once,
at the end.
Why that is tighter, in one line: IBP bounds each pre-activation with
⟨W1ᵢ, x₀⟩ ∓ ε‖W1ᵢ‖₁ and then combines; CROWN forms the composite row
A = Σₜ aₜ · W1ₜ first and takes ε‖A‖₁ once. Cancellation between the
rows of W1 survives in A and is destroyed by the per-row ‖·‖₁.
CROWN-IBP, which is what this file supports: the per-neuron pre-activation
bounds [l, u] that decide each ReLU's relaxation are taken from IBP — here
literally denseLo W1 … / denseHi W1 …, the box IntervalBound.lean already
proves — and only the output pass is CROWN. (Full CROWN would re-derive [l,u]
by running itself on every prefix: strictly tighter, much more work.)
What is certified, and why it is a MARGIN and not a box #
certified_of_boxSound (IntervalBound.lean) consumes a bracket on the logit
vector and asks the boxes to separate. CROWN does not produce one usefully:
bounding f · y and f · j independently and then separating discards the
correlation between them, which is a large fraction of the available tightening.
So the capstone here is stated on the margin f · y − f · j directly, via
certified_of_marginPos — the same seam idea (the certificate never inspects
how the bound was obtained), one rung lower. certified_of_marginPos is
generic: a margin-direct interval bound plugs into it too, with no CROWN.
Relaxation, and the rational-size question #
Both envelopes are stated so a generator can discharge them by norm_num on
rationals, and — critically — so the slope may be rounded:
- lower:
α · z ≤ relu zfor ANYα ∈ [0,1](relu_lower_envelope) — soundness is insensitive toα, so it is purely a tightness knob; - upper:
relu z ≤ s · (z − l)on[l,u]wheneverl ≤ 0,0 ≤ sandu ≤ s · (u − l)(relu_upper_envelope). The hypothesis is stated multiplicatively, NOT ass = u/(u−l), sosmay be any rational at or above the chord slope — in particular the chord ROUNDED UP to a/2^kgrid.
That rounding is what keeps the coefficients small. Left unrounded, u/(u−l)
carries the layer-1 denominators into every entry of A and reproduces the
LipSDP tier's ~230-digit blow-up. Measured (scripts/crown_ibp_probe.py,
planning/archive/crown_ibp.md §5.5): at k = 8 the rounding costs ZERO images on both
trained nets at every radius, so the coefficients stay at the same /256 scale
as the weights themselves.
Depth: this file does the single backward step a dense ∘ relu ∘ dense net
needs, where crownRow IS the matrix-level back-substitution. A general
multi-layer LinSound would be real machinery with no consumer yet — the conv
tier stays on IBP (max-pool is not an elementwise nonlinearity), and the trained
dense nets are two-layer.
Everything is elementary and closes under propext / Classical.choice / Quot.sound. Engine only; the generated instance is a separate file.
Margin positivity ⇒ L∞ certificate. If for every wrong class the
margin f · y − f · j is strictly positive at every point of the box
x ∓ ε, then y is the strict argmax under every perturbation with
|δ i| ≤ ε coordinatewise.
The peer of certified_of_boxSound one rung lower: it says nothing about
HOW the margin was bounded, so a CROWN bound and a margin-direct interval
bound discharge it the same way. Bounding the margin rather than the two
logits separately is itself a tightening — it keeps the correlation between
f · y and f · j that a box separation throws away.
Lower envelope. α · z ≤ relu z for ANY α ∈ [0,1], with no reference
to [l,u] at all. Soundness is therefore insensitive to the choice of α:
optimizing it (α-CROWN) is a tightness question, never a correctness one,
and a rounded α needs no re-verification.
Upper envelope. On [l, u] with l ≤ 0, any nonneg s at or above the
chord slope dominates relu. The chord condition is stated as
u ≤ s * (u - l) rather than s = u / (u - l), which is what lets a
generator ROUND s up to a /2^k grid and still discharge it by
norm_num — the rational-size lever (see the header).
Proof: g z = s·(z−l) − relu z is affine on [0,u] with g 0 = −s·l ≥ 0
and g u = s·(u−l) − u ≥ 0, hence nonneg between; below 0 it is trivial.
a · z + c is a sound linear lower bound for v · relu z on [l, u].
One neuron's contribution to a margin, relaxed. v is that neuron's
coefficient in the margin row W2 y · − W2 j ·; its SIGN decides which
envelope may be used, which is why the instances below split on it and not
on the neuron alone.
Equations
Instances For
A neuron that is dead on the whole box (u ≤ 0) contributes exactly 0 —
both envelopes are exact and the coefficient vanishes.
A neuron that is active on the whole box (0 ≤ l) is EXACT: relu z = z,
so it passes its margin coefficient through unrelaxed. This is where CROWN
keeps everything IBP would have widened.
Unstable neuron, NEGATIVE margin coefficient: multiplying by v ≤ 0 flips
the inequality, so the upper envelope is what lower-bounds the
contribution. This is the only branch that carries a constant, and the only
one whose slope must be verified against the chord.
The relaxation coefficient, with the branch chosen by the box and the sign
of the margin coefficient v.
Equations
Instances For
All four branches in one lemma, uniform in v. The chord condition is
required only where it is used — on a genuinely unstable neuron — so a
stable neuron discharges it vacuously and may carry s = 0.
This is what keeps a generated instance affordable: the relaxation is proved once per NEURON (16 facts), not once per (class, neuron) pair (144), because nothing here depends on which wrong class is being separated.
The CROWN row. Back-substitute the per-neuron relaxation coefficients
through the first layer: A = Σₜ aₜ · W1ₜ. Forming this composite BEFORE
taking any norm is the entire mechanism — cancellation between the rows of
W1 survives here and is destroyed by IBP's per-row ‖·‖₁.
Equations
- Proofs.LipschitzCertDemo.crownRow a W1 i = ∑ t : Fin h, a t * W1 t i
Instances For
Back-substitution is exact: relaxing in pre-activation space and then
substituting z = W1 x' is the same linear function of x' as the CROWN
row applied directly.
Concretization, once. On the uniform box x ∓ ε a linear functional
bottoms out at ⟨A, x₀⟩ − ε‖A‖₁. This is denseLo_uniform read at the
one-row matrix A — reused, not re-derived. ONE ℓ1 norm per
(image, class), where IBP pays one per (image, neuron).
The two-layer net, evaluated one logit at a time — denseE/reluE peeled
exactly once, so the inner denseE W1 x' stays folded (unfolding it would
put a 784-term sum under every max).
The margin bound. Summing the per-neuron relaxations gives a linear
lower bound on f · y − f · j in pre-activation space. z is abstract
here: the bound is a statement about the relaxation, not about the net.
CROWN-IBP L∞ certificate for dense ∘ relu ∘ dense.
For each wrong class j, the generator supplies relaxation coefficients
a j and constants cc j for the margin row W2 y · − W2 j ·, discharged
against the IBP pre-activation box by the reluLB_* instances; the
certificate fires when the ONE concretized linear bound is positive:
⟨A, x₀⟩ − ε‖A‖₁ + Σ cc > 0, A = crownRow (a j) W1.
Note the box in hrelax is literally denseLo W1 … / denseHi W1 … — the
interval box IntervalBound.lean proves — not a float recomputation. That
is forced by the statement, which is the point: CROWN-IBP is sound only if
the [l,u] you relax against are the ones that actually hold on the box the
certificate quantifies over.
‖A‖₁ = Σᵢ |Σₜ aₜ·W1ₜᵢ| does NOT decompose over t — the absolute value is
taken after the combination, which is exactly the point of CROWN. So it needs a
fact of its own, and the naive route (emit A's 784 numerators per
(image, class)) makes the exhibit enormous (gotcha 2).
Instead the kernel forms A from the weight rows the corpus already commits:
a generator emits the 16 coefficient numerators, and absSumZ (combZ …) folds
the 16×784 combination and the absolute sum in one decide +kernel.
⟨A, x₀⟩ needs no new fact at all — crownRow_dot turns it back into
Σₜ aₜ·⟨W1ₜ, x₀⟩, i.e. a 16-term rational sum over the committed hpre dots.
Entrywise c · row + acc.
Equations
- Proofs.LipschitzCertDemo.scaleAddZ c row acc = List.zipWith (fun (r a : ℤ) => c * r + a) row acc
Instances For
Σₜ csₜ · rowsₜ, entrywise: the CROWN row A, at integer scale.
Equations
- Proofs.LipschitzCertDemo.combZ n [] x✝ = List.replicate n 0
- Proofs.LipschitzCertDemo.combZ n (head :: tail) [] = List.replicate n 0
- Proofs.LipschitzCertDemo.combZ n (c :: cs) (r :: rs) = Proofs.LipschitzCertDemo.scaleAddZ c r (Proofs.LipschitzCertDemo.combZ n cs rs)
Instances For
The CROWN row, in one kernel-checkable object. With coefficients
aₜ = csₜ / dc and weights W1ₜᵢ = rowsₜᵢ / dw, the ℝ-level crownRow
is the integer combZ over dc·dw.
‖A‖₁ from one absSumZ … := by decide +kernel fact. One ℓ1 norm per
(image, class), where IBP pays one per (image, neuron) — and the 784
entries of A are never emitted, only folded.