Documentation

LeanMlir.Proofs.Codegen.ConvNeXtRender

ConvNeXt-T train step rendered from the verified AST, with declared carve-outs #

The per-example ConvNeXt render. It writes one artifact, the SGD-inline verified_mlir/convnext_train_step.mlir (convNextTrainStepText), and holds what ConvNeXtRenderB reuses: the stage tables (CnxDims, cnxTiny/cnxSmall/cnxBase), the parameter list (cnxAllParams), the forward render convNextFwdText, the shared traversal convNextBackAll, and the AdamW tail convNextAdamTrainStepText. Every other ConvNeXt artifact is written by the batched chain in ConvNeXtRenderB, whose Proofs tier is ConvNeXtFoldGB. This chain stays because the batched traversal has no fused-SGD arm and ConvNeXtStepTie is stated at this file's bytes. TestConvNeXtFwdBTie checks the two chains against each other.

The train step is the full [3,3,9,3] ConvNeXt-T (BS=32, 3×224²→10): forward, backward cotangent chain, and the parameter updates. All 182 parameter gradients are SHlo nodes. The stem 4×4/s4 weight gradient is .convStride4WeightGrad, whose den is flatConvStride4WeightGradHasVJP; the three 2×2/s2 downsample weight gradients are .convStridedWeightGrad, whose den is the kernel-generic flatConvStride2WeightGradHasVJP (the emitter splits odd/even kernels in StableHLO.sWGradGeom).

Hand-written text in the rendered module, outside any SHlo term:

In the SGD render every other parameter is updated by a fused SHlo SGD op (gradient and θ − lr·g in one node), which denotes θ − lr·(certified per-layer Jacobian · its cotangent) (ConvNeXtFold). Render is value-independent (skel erases values), so placeholders + lr:=0 are passed; the emitted lrStr/epsStr literals carry the real values.

References #

Equations
Instances For

    The SPATIAL table, and it is the one thing that is NOT a size parameter: 224 with a 4×4/s4 patchify stem gives 56/28/14/7 at every ConvNeXt size. T, S and B differ in depth and width only, so this stays a constant and the 7s in the GAP backward stay literals.

    Equations
    Instances For

      The ConvNeXt size — depths and channel dims TOGETHER, in one record.

      They are bundled deliberately. A bare Array Nat of depths beside a bare array of dims admits (depths := S, dims := T), which is not a ConvNeXt of any size but type-checks, renders and trains. B moves BOTH, so two bare arrays would make the mismatch reachable in the direction that matters. One record, three instances, and a caller cannot spell a net that does not exist. Same reasoning as VitDims, arrived at from the opposite direction: ViT bundled to keep d = heads * hd definitional, this bundles to keep two tables from drifting.

      Both tables are Vector Nat 4: ConvNeXt has four stages at every size, and the renders read exactly stages 0–3. With a bare Array a fifth depth entry renders the same bytes while cnxDropTotal counts it, and a short one reads 0 through !.

      deriving DecidableEq is what ConvNeXtRenderB's #guards on its restatement use, and cnxModelName matches on the whole record rather than on the block count — which is what stops B (36 blocks, like S) from introducing itself as an S.

      Instances For
        Equations
        Instances For
          Equations
          • One or more equations did not get rendered due to their size.
          Instances For

            ConvNeXt-T: [3,3,9,3] @ [96,192,384,768], 28.6 M at K=1000. The DEFAULT everywhere, so every pre-existing call site is unchanged and every committed artifact re-renders byte-identical.

            Equations
            Instances For

              ConvNeXt-S: T deepened — stage 3 goes 9 → 27, dims UNCHANGED. 50.2 M at K=1000.

              Equations
              Instances For

                ConvNeXt-B: S's depth AND a wider net — [128,256,512,1024]. 88.6 M at K=1000. B is the size that made the dims a parameter: it shares S's depth table exactly, so anything keying on block count alone cannot tell them apart.

                Equations
                Instances For

                  Total ConvNeXt blocks = the reference's totalDrop, i.e. the ramp DENOMINATOR is this minus 1. Derived from the stage table the render traverses, not restated beside it — which is what makes it follow D to 36 at ConvNeXt-S without a second edit.

                  Equations
                  Instances For

                    The ramp index of stage si's block j — the reference's dbi, which counts blocks over the WHOLE net. The single source for the drop-site numbering: the forward calls it walking the stages forwards, the backward walking them backwards, and they cannot disagree.

                    Equations
                    Instances For

                      The number of per-example drop-path scale inputs an SD ConvNeXt render takes: one per block.

                      Equations
                      Instances For

                        The model name a render's banner claims, DERIVED from the stage table it was rendered at.

                        A banner is a render's own description of what it did, and the one thing worse than not having one is having one that lies — cnxDropFwdBanner's own note says exactly that about reusing a drop-free banner on an SD artifact. Passing the name as a second parameter beside D would be two writers for one fact. So it is read off D.

                        It matches on the WHOLE RECORD, not on the block count, and that is not fussiness. B is [3,3,27,3] like S, so a name keyed on cnxDropTotal (36 blocks) names S and B alike and every B artifact would open by calling itself a ConvNeXt-S. The guards below enumerate all three rather than spot-checking one. An unrecognised table gets a name that SAYS it is unrecognised, rather than falling back to "ConvNeXt-T" the way a getD would.

                        Equations
                        • One or more equations did not get rendered due to their size.
                        Instances For

                          The %dp<i>: tensor<Bxf32> inputs an SD render appends to its signature — one per block, in ramp-index order, which is the order the driver's dropScales writes them into the blob. Empty when off, which is what keeps every committed artifact byte-identical.

                          Equations
                          Instances For

                            The SGD update wrap for ConvNeXt's stem weight, taking the gradient's SSA name explicitly: the certified SHlo ops emit a fresh %vN.

                            Equations
                            • One or more equations did not get rendered due to their size.
                            Instances For

                              The channel LayerNorm. ConvNeXt's channel_layer_norm takes h·w statistics per example, each over the c channels at one spatial position, with a per-channel [c] affine. That is a different function from .bnF (bnForward n ε γ β: one mean and one variance over the whole c·h·w map, scalar γ/β). Matching a reduction's literal across dimensions = [1] against the reference's axis=1 does not check this: the artifact's tensor is rank-2 [B, c·h·w] and the reference's is rank-4 NCHW.

                              No new SHlo op. ConvNeXt's channel-LN IS ViT's row-LN under a transpose — view an example as [c, s] with s = h·w, transpose to [s, c], and each row is one spatial position holding its c channels, which is exactly what rowLNFlat normalises. Checked on device (lake build channel-ln): forward and all three backward pieces tie the closed form at rel 0, the .bnF control fires at rel 0.82, and the transposes measure free against the whole-net LN (XLA folds a transpose into the consumer's layout).

                              Nat multiplication is not definitionally associative, and the ambient index here is c*h*h = (c*h)*h while the transpose needs c*(h*h). reassoc/unassoc (IndexCast) are castIdx along Nat.mul_assoc; they are casts on the index, not on the value, so pretty walks the same tree.

                              The %one/%zero constants the channel-LN chain binds lnRowF/lnRowBack's SCALAR γ/β to — the real per-channel affine is rowScaleF/rowBiasF downstream, exactly as ViT does it. Emitted once per module body.

                              Wire the operand and forget the prelude, and the artifact uses an SSA name nothing defines — iree-compile/XLA say "use of undeclared SSA value name" and nothing before them says anything. regen_verified_mlir.sh check's prelude audit is what catches it.

                              Equations
                              • One or more equations did not get rendered due to their size.
                              Instances For
                                def Proofs.StableHLO.cnxLnFwdSite (cBS : ℕ) (gN btN xin : String) (c h : ℕ) :

                                One LayerNorm forward site — ConvNeXt's real channel-LN: transpose to [h·w, c], normalise each spatial row over its channels at the scalar identities %one/%zero, apply the real [c] affine (vecLnSite at m = h·w), transpose back.

                                Equations
                                • One or more equations did not get rendered due to their size.
                                Instances For

                                  The HEAD LN forward site — the paper's norm(x.mean([-2,-1])).

                                  It is cnxLnFwdSite with the two transposes DELETED, and that is the whole difference: after GAP the tensor is a single [d] row, so "normalise each spatial row over its channels" and "normalise the feature vector" are the same function at m = 1. Mirrors Proofs.StableHLO.headLNGraph op for op.

                                  It is vecLnSite at m = 1, so every operand is annotated Vec (1 * d), never Vec d. 1 * d does NOT reduce for a VARIABLE d (Nat.mul recurses on its second argument), and d here is V.dims[3]!, which moves with the ConvNeXt size. It is the same annotation convNextBackAll's smoothing chain carries, for the same reason.

                                  Equations
                                  Instances For
                                    def Proofs.StableHLO.headLnBackSite (cBS : ℕ) (gN xName cot : String) (d : ℕ) :

                                    The head LN's input-VJP: dx = lnRowBack γ=1 (rowScale γ dy). xName is the saved LN input — the GAP output — which lnRowBack re-normalises from rather than saving x̂/istd.

                                    Equations
                                    • One or more equations did not get rendered due to their size.
                                    Instances For
                                      def Proofs.StableHLO.headLnGammaTail (cBS : ℕ) (adam : Bool) (gN xName cot : String) (d : ℕ) :

                                      The head LN's γ tail — veclnGamma{Grad,Sgd} at N = 1.

                                      Equations
                                      • One or more equations did not get rendered due to their size.
                                      Instances For
                                        def Proofs.StableHLO.headLnBetaTail (cBS : ℕ) (adam : Bool) (btN cot : String) (d : ℕ) :

                                        The head LN's β tail — rowDenseBias{Grad,Sgd} at N = 1, reducing dims = [0,1], i.e. contracting the batch (and the single row) as a shared [d] parameter needs.

                                        Equations
                                        • One or more equations did not get rendered due to their size.
                                        Instances For
                                          def Proofs.StableHLO.lnBackSite (cBS : ℕ) (gN xName cot : String) (c h : ℕ) :

                                          One LayerNorm input-VJP site: dx = transposeᵀ (lnRowBack γ=1 (rowScale γ dyᵀ)). xName is the saved LN INPUT — lnRowBack recomputes x̂/istd from it rather than saving them.

                                          Equations
                                          • One or more equations did not get rendered due to their size.
                                          Instances For
                                            def Proofs.StableHLO.lnGammaTail (cBS : ℕ) (adam : Bool) (gN xName cot : String) (c h : ℕ) :

                                            The γ tail for one LN site — the per-channel veclnGamma{Grad,Sgd}. The transposes of xName/cot are re-emitted here rather than threaded from lnBackSite: pretty has no CSE, but XLA does, and XLA collapses exactly this kind of duplicated subtree.

                                            Equations
                                            • One or more equations did not get rendered due to their size.
                                            Instances For
                                              def Proofs.StableHLO.lnBetaTail (cBS : ℕ) (adam : Bool) (btN cot : String) (c h : ℕ) :

                                              The β tail — the per-channel rowDenseBias{Grad,Sgd}. It reduces dims = [0,1], i.e. it contracts the BATCH as well as the spatial rows, which is what a shared [c] parameter needs.

                                              Equations
                                              • One or more equations did not get rendered due to their size.
                                              Instances For
                                                Instances For
                                                  def Proofs.StableHLO.cnxFwdBlock (gf : GeluForm) (cBS : ℕ) (pfx xin : String) (c e h : ℕ) :
                                                  Equations
                                                  • One or more equations did not get rendered due to their size.
                                                  Instances For
                                                    Equations
                                                    • One or more equations did not get rendered due to their size.
                                                    Instances For
                                                      def Proofs.StableHLO.cnxFwdDown (cBS : ℕ) (pfx xin : String) (ci co h2 : ℕ) :
                                                      Equations
                                                      • One or more equations did not get rendered due to their size.
                                                      Instances For
                                                        def Proofs.StableHLO.bwdDown (cBS : ℕ) (pfx dy xin : String) (ci co h2 : ℕ) :
                                                        Equations
                                                        • One or more equations did not get rendered due to their size.
                                                        Instances For

                                                          Every leaf below is one of the *Sgd/*Grad pairs whose dens differ by exactly θ − lr · · (*Sgd_eq_grad, all rfl) and whose emits differ by exactly the const-lr/multiply/subtract tail (tests/TestBatchedEmitTie.lean, byte-PREFIX). So one traversal serves both renders.

                                                          ConvNeXt's param tails are factored out of the cotangent traversal — bwdBlock computes cotangents and nothing else. lrStr is threaded but unused in adam mode: AdamW's learning rate is the runtime %lr argument, not a baked literal.

                                                          def Proofs.StableHLO.blockParamSgd (cBS : ℕ) (adam : Bool) (pfx : String) (b : FNames) (cot_p cot_e cot_n cot_d dy : String) (c e h : ℕ) :
                                                          Equations
                                                          • One or more equations did not get rendered due to their size.
                                                          Instances For
                                                            def Proofs.StableHLO.downParamSgd (cBS : ℕ) (adam : Bool) (pfx downLn downIn cot_n dy : String) (ci co h2 : ℕ) :
                                                            Equations
                                                            • One or more equations did not get rendered due to their size.
                                                            Instances For

                                                              Shapes are List Nat rather than rendered tensor<…> strings because the AdamW render needs both forms: %{nm}/%{nm}m/%{nm}v for the moment slots and the raw dimensions for the emitted Adam ops.

                                                              The LN γ/β are [c] — the real channel-LN's per-channel affine. This list is the func signature AND the return types, so getting it wrong is not silent — the body uses %…ng at tensor<{c}xf32> and the compiler rejects a signature that disagrees with itself.

                                                              Equations
                                                              • One or more equations did not get rendered due to their size.
                                                              Instances For
                                                                def Proofs.StableHLO.allParams (nClasses : ℕ := 10) (V : CnxDims := cnxTiny) :

                                                                nClasses defaults to 10 (Imagenette) so every existing call site is unchanged; the ImageNet renders pass 1000. Only the head moves — 180 of the 182 entries are class-independent.

                                                                Equations
                                                                • One or more equations did not get rendered due to their size.
                                                                Instances For
                                                                  def Proofs.StableHLO.cnxWdCounts (nClasses : ℕ := 10) (V : CnxDims := cnxTiny) :

                                                                  The decayed / excluded split, as the renderer computes it.

                                                                  Equations
                                                                  • One or more equations did not get rendered due to their size.
                                                                  Instances For

                                                                    A namespaced public alias of allParams (this file's signature source, itself public) for the one thing outside it that needs the list — tests/TestWdExcludeTie.lean, which must read the SAME names and shapes the renderer chose %wd/%wdz from.

                                                                    Equations
                                                                    Instances For

                                                                      Every SSA name the ConvNeXt-T forward produces. convNextFwdText returns just logits; the train steps additionally consume the block records, the two downsample names and the head's gap/hn on the way back.

                                                                      Instances For
                                                                        def Proofs.StableHLO.convNextFwdChain (gf : GeluForm) (cBS : ℕ) (nClasses : ℕ := 10) (V : CnxDims := cnxTiny) :

                                                                        The full ConvNeXt-T [3,3,9,3] forward as pretty of the verified AST. 4×4/s4 patchify stem (3→96, 224→56) → 4 stages at 56/28/14/7 with 2×2/s2 downsamples between them → GAP(7×7) → head LN → dense(768→10). Every emitted line is pretty of a verified SHlo node.

                                                                        There is no train/eval mode here, and there should not be — ConvNeXt normalises with LayerNorm, which reduces over the channel/spatial axes of ONE example and never over the batch. So the forward is already class-batch-independent: train == eval, and @convnext_fwd is the only forward artifact this net needs (unlike the BN nets, which need a frozen-stats peer).

                                                                        Equations
                                                                        • One or more equations did not get rendered due to their size.
                                                                        Instances For
                                                                          def Proofs.StableHLO.convNextFwdText (gf : GeluForm) (funcName : String := "convnext_fwd") (nClasses : ℕ := 10) (V : CnxDims := cnxTiny) (cBS : ℕ := 32) :

                                                                          @convnext_fwd from the per-example chain — convNextFwdChain, the forward both train steps in this file emit, followed by return logits. Takes %x plus the 182 params of allParams in func-arg order (183 inputs) and returns logits [cBS, nClasses]. Around the chain the text is the signature, a banner comment and chLnPrelude.

                                                                          The committed verified_mlir/convnext_fwd.mlir is written by ConvNeXtRenderB (convNextFwdRenderB); TestConvNeXtFwdBTie checks this render against it byte for byte, and scripts/regen_verified_mlir.sh check checks that its body is a prefix of convnext_train_step.mlir's.

                                                                          Equations
                                                                          • One or more equations did not get rendered due to their size.
                                                                          Instances For
                                                                            def Proofs.StableHLO.convNextBackAll (gf : GeluForm) (adam : Bool) (smooth : Option (String × String × String) := none) (nClasses : ℕ := 10) (V : CnxDims := cnxTiny) (cBS : ℕ := 32) :

                                                                            The whole-net forward + cotangent + backward traversal, SHARED by the SGD and AdamW renders.

                                                                            Returns (code, params, softmax): every emitted line; one SSA per parameter — the updated param at adam := false, the un-fused gradient at adam := true; and the softmax SSA, which the AdamW render's report-only %loss reads.

                                                                            smooth is the cotangent recipe. none → the SGD render's (softmax − onehot)/B, unchanged down to the hand-written %dy divide. some (α, −α/K, B) → the label-smoothed ((softmax − onehot) + α·onehot − α/K)/B composed from kit ops, which is what the AdamW recipe trains on. ConvNeXt already spells the ÷B explicitly (unlike ViT/R34, which fold the mean into lr), so here the smoothing is the only difference between the two cotangents.

                                                                            The softmax is prettyd on its own line rather than nested inside the .sub so that %loss can read it; .operand is a leaf that emits nothing, so the fresh-name sequence is the same either way.

                                                                            Equations
                                                                            • One or more equations did not get rendered due to their size.
                                                                            Instances For
                                                                              def Proofs.StableHLO.convNextTrainStepText (gf : GeluForm) (funcName : String := "convnext_train_step") (nClasses : ℕ := 10) (V : CnxDims := cnxTiny) (cBS : ℕ := 32) :

                                                                              ConvNeXt-T (full [3,3,9,3]) SGD train step rendered from the verified AST, the writer of verified_mlir/convnext_train_step.mlir: %x, the 182 params and %onehot in (184 inputs), the 182 updated params out. All 182 parameter gradients are SHlo nodes, and every parameter but the stem weight psW is updated by a fused SHlo SGD op whose output is the updated parameter. Hand-written text: the signature, the %sc/%bsc constants, chLnPrelude, the %dy divide, the GAP-backward block, and psW's update (sgdOf around the certified .convStride4WeightGrad).

                                                                              The cotangent is plain CE with an explicit ÷B — unlike ViT/R34, which fold the batch mean into lr — so the committed cLR = 0.1 is an effective 0.1, the house convention spelled differently.

                                                                              Equations
                                                                              • One or more equations did not get rendered due to their size.
                                                                              Instances For
                                                                                def Proofs.StableHLO.cnxAdamVariant (replicas : ℕ) (ema wdExclude clip sd bf16 : Bool := false) (gf : GeluForm := GeluForm.tanh) (acc : ℕ := 1) :

                                                                                The driver's variant slug for a given replica count: the artifact is verified_mlir/convnext_<variant>_train_step.mlir, the entry point is @convnext_<variant>_train_step, and LEAN_MLIR_VARIANT selects it. All three must agree — the shim checks the entry name and refuses a mismatch outright ("entry mismatch") rather than running the wrong graph. The #guards at the bottom pin the literal #eval paths against this.

                                                                                Unlike mnv2AdamVariant/r34AdamVariant there is no batch suffix.

                                                                                The EMA renders get their OWN slugs (ema/emadp), for the reason the RMSProp ones do: a render carrying a fourth [θ|m|v|ema] region must never be able to overwrite the artifact the AdamW trainer runs, whose blob has three: two writers for one path, where the last one wins, and here it would also be an arity mismatch the driver could not survive.

                                                                                Equations
                                                                                • One or more equations did not get rendered due to their size.
                                                                                Instances For
                                                                                  def Proofs.StableHLO.convnextAdamConsts (nClasses : ℕ) (V : CnxDims) (wdExclude : Bool := false) (wdStr : String := "0.0001") (opt : OptRecipe := OptRecipe.adamw) :

                                                                                  β₁/β₂/ε/wd as graph constants — the committed ConvNeXt-T AdamW recipe.

                                                                                  wdStr is a parameter for the reason ViTRender.vitAdamConsts' is: the Imagenette and ImageNet configs disagree on it by 500×. convnextVerified's recipe is the baked 1e-4; convnextTinyImagenetConfig.weightDecay := 0.05. The default is unchanged, so every committed artifact keeps its bytes; only the ImageNet wx render passes 0.05.

                                                                                  Equations
                                                                                  • One or more equations did not get rendered due to their size.
                                                                                  Instances For

                                                                                    The hand-written block the AdamW train-step banners name beside trainStepHandNote's list: the GAP backward (%dgi … %dgapf) on the gradient path.

                                                                                    Equations
                                                                                    Instances For
                                                                                      def Proofs.StableHLO.convNextAdamTrainStepText (gf : GeluForm) (alphaStr negAlphaKStr bStr : String) (replicas : ℕ := 1) (nClasses : ℕ := 10) (slug : String := "convnext") (ema wdExclude : Bool := false) (wdStr : String := "0.0001") (clip : Bool := false) (clipStr : String := "1.0") (traversal : Option (StateM EmitS (String × List (String × String) × String)) := none) (sd : Bool := false) (V : CnxDims := cnxTiny) (bf16 : Bool := false) (cBS : ℕ := 32) (acc : ℕ := 1) :

                                                                                      ConvNeXt AdamW train step rendered from the verified AST — the wrapper ConvNeXtRenderB calls (with traversal set to its batched chain) to write convnext_adam_train_step.mlir and the other ConvNeXt AdamW/EMA artifacts.

                                                                                      Same backward as convnext_train_step (convNextBackAll, one traversal) but taking the un-fused gradients, each fed to the proven AdamW triple. The cotangent adds label smoothing (α = 0.1, K = nClasses); ConvNeXt already spelled the ÷B explicitly, so that is the only difference.

                                                                                      Interface at ConvNeXt-T, ema := false, no drop-path: 551 in (%x, 182 θ, 182 m, 182 v, %lr/%bc1/%bc2, %onehot) / 549 out (182 θ', 182 m', 182 v', %loss/%bc1/%bc2), the packed [θ|m|v] layout the AdamW trainer reads.

                                                                                      All 182 parameter gradients are SHlo nodes, and each update is the proven AdamW triple. Hand-written text: the signature and constants, the GAP-backward block (in convNextBackAll), the %bc1/%bc2 passthroughs, and the report-only %loss (on no gradient path).

                                                                                      replicas > 1 renders the DATA-PARALLEL variant to its own entry name and artifact path via cnxAdamVariant, so producing it can never clobber the one the trainer runs. The only difference is one all_reduce(add)/N per parameter gradient, between the certified gradient and the certified AdamW triple: certified gradient → trusted collective → certified AdamW. See optOne for the carve-out.

                                                                                      Equations
                                                                                      • One or more equations did not get rendered due to their size.
                                                                                      Instances For