Optional drop sites — stochastic depth and classifier dropout as Option binders #
A verified render carries stochastic depth and classifier dropout as HOST-DRAWN mask inputs
(Proofs.Training.DropPath: the keep probability is folded into the mask, the op is a diagonal
scale, its VJP is itself). Whether a site is RENDERED is a renderer flag (sd, cd), so a graph
or chain statement that covers both the drop-free artifact and the *drop* / *do* one takes
the site as an Option: none emits no node and denotes the identity, some s emits the
dropPathB / dropoutB node at the mask s. The forward graphs state their sites this way
(LeanMlir.Proofs.Nets.EfficientNet.EfficientNetFullB0Drop's efficientnetFwdGraphBFullDrop); the step, sync and
loss-gradient ties thread the same binder through their cotangent chains
(Proofs.MobileNetV2TieB's classifier dropout first), so the drop-free tie is the none
instance verbatim and the some instance is the artifact that trained.
dropPathOpt/dropoutOpt— the real-valued site;_noneisrfl,_onesthe eval identity (dropPath_ones_id,dropout_ones_id).dropPathOptG/dropoutOptG— the graph node;den_*by cases on the site.dropoutOptHasVJP/dropPathOptHasVJP— the VJP, its backward THE OP ITSELF at the same mask at either value (dropout_vjp_is_selfoneOptionup), stated as thebackwardfield so a chain lemma that isrflat the drop-free chain staysrflwith the site in; the differentiability each tie'sHasGradAt.compneeds beside it.dropoutOpt_smul,dropoutOpt_shard— what a data-parallel tie needs: the site is linear in the cotangent and commutes with the batch cut when replicarholds shardrof the mask, as the DP renders' per-replica%doinputs are.dropPathOpt_smul,dropPathOpt_shardare the per-example-mask peers: a stochastic-depth mask is aVec Nof scalars, cut byexampleShard(batchShard's cut at width one), anddropPathOptFamis the replica family a DP chain feeds a drop site, with its_shardand_scaled(the invariant the sync ties carry block to block).
Drop-path at a site that may be absent: none is the identity.
Equations
- Proofs.dropPathOpt N n none = id
- Proofs.dropPathOpt N n (some s) = Proofs.dropPath N n s
Instances For
Dropout at a site that may be absent: none is the identity.
Equations
Instances For
At the all-ones scale drop-path is the identity — the masks the driver passes at eval.
At the all-ones mask dropout is the identity.
The backward is the forward at the same mask, at either value — stated as the witness's
backward field, not derived by cases, so (dropoutOptHasVJP cd).backward x dy unfolds to
dropoutOpt cd dy for a symbolic cd.
Equations
- Proofs.dropoutOptHasVJP cd = { backward := fun (x dy : Proofs.Vec m) => Proofs.dropoutOpt cd dy, correct := ⋯ }
Instances For
dropoutOptHasVJP one rank down — the per-example scale.
Equations
- Proofs.dropPathOptHasVJP N n sd = { backward := fun (x dy : Proofs.Vec (N * n)) => Proofs.dropPathOpt N n sd dy, correct := ⋯ }
Instances For
The site is linear in what flows through it.
Replica r applying ITS shard of the mask to ITS shard of a value is shard r of the global
site — batchShard_zipWith at the site's multiply.
Replica r's block of a per-example scalar family laid out [R·N]: example (r, n) of the
global batch is example n of shard r — batchShard's cut at width one, for the
stochastic-depth masks (Vec N per site, not Vec (N * 1)). The DP renders' per-replica
%dp<i> inputs are these.
Equations
- Proofs.exampleShard R N S r n = S (finProdFinEquiv (r, n))
Instances For
The per-example site is linear in what flows through it.
Replica r scaling ITS shard of a value by ITS shard of the per-example mask is shard r of
the global site: the example a cell belongs to is the same on both sides of the cut
(Equiv.symm_apply_apply at batchShard's index).
dropPathOpt_shard at a rendered site.
The replica family a data-parallel chain feeds a drop site: replica r's cotangent through
replica r's shard of the mask.
Equations
- Proofs.dropPathOptFam R N n sd dys r = Proofs.dropPathOpt N n (Option.map (fun (S : Proofs.Vec (R * N)) => Proofs.exampleShard R N S r) sd) (dys r)
Instances For
The scaled-shard invariant passes a drop site: replicas at R × the shards of a global
cotangent hand the branch R × the shards of the global branch cotangent.
A dropPathB node when the site is rendered, nothing otherwise.
Equations
- Proofs.StableHLO.dropPathOptG mN none x✝ = x✝
- Proofs.StableHLO.dropPathOptG mN (some s) x✝ = Proofs.StableHLO.SHlo.dropPathB mN s x✝
Instances For
A dropoutB node when the site is rendered, nothing otherwise.
Equations
- Proofs.StableHLO.dropoutOptG mN none x✝ = x✝
- Proofs.StableHLO.dropoutOptG mN (some m) x✝ = Proofs.StableHLO.SHlo.dropoutB mN m x✝