fix(runtime): rate-limit the supervision reprovision — deferred, not dropped (review F3)

The default decider restarts a faulting child immediately and without limit, and
the hook makes every restart do ShutdownAsync+InitializeAsync+SubscribeAsync, so a
driver faulting on every value became a reconnect storm. Per-driver last-reprovision
stamp + DefaultReprovisionBackoff (10s, matched to DefaultReconnectInterval);
a held request is re-delivered by a single per-driver timer carrying the original
incarnation ref, so the F2 identity check still applies.

Seen RED with the backoff disabled: InitializeCount == 3 (second applied
immediately) where the test requires 2-then-3.

Also: the _children.Keys union-arm test (seeder gained taglessDrivers) and the
_desiredByDriver xmldoc note that it lingers on last-known-good after a failed apply.
This commit is contained in:
Joseph Doherty
2026-08-20 10:39:13 -04:00
parent 2b7100cf19
commit 0793b0cba8
4 changed files with 212 additions and 13 deletions
+18
View File
@@ -201,6 +201,24 @@ Things to know before touching any of this:
remove a driver; a bounded retry count would silently stop one after N crashes. Both change failure
semantics for every driver in the fleet and deserve their own plan and live gate. The reprovision hook
makes the *current* infinite-restart behaviour correct instead of dead.
- ⚠️ **…which is exactly why the reprovision is RATE-LIMITED.** The default decider restarts a faulting
child **immediately and without limit**, and the hook now makes every one of those restarts do
`ShutdownAsync` + `InitializeAsync` + `SubscribeAsync` — so a driver that faults on every value would
hammer its device as fast as it can crash. `DriverHostActor` keeps a per-driver last-reprovision stamp
and holds anything inside `DefaultReprovisionBackoff` (10 s, matched to
`DriverInstanceActor.DefaultReconnectInterval` because it drives the same connect work). The held
request is **deferred, not dropped** — one `StartSingleTimer` per driver, so a burst collapses to a
single pending reprovision — because dropping it would leave a child whose *final* crash fell inside
the window un-provisioned forever, which is the 03/S16 failure this hook exists to close. Each
deferral logs a Warning naming the crash type, so a crash loop is visible rather than merely quiet.
- **A reprovision is routed by INCARNATION, not by driver id.** `ReprovisionRequested` is delivered
asynchronously, so one can still be in flight from an old incarnation after `HandleRestartDriver`
stopped that child and spawned a replacement under the same id. Delivered to the new child — which is
in `Connecting`, where `InitializeRequested` *is* handled — it would start a second, concurrent
`InitializeAsync`, racing the dispose-before-overwrite that #527's teardown exists to serialise
(`ModbusDriver._transport` / `MqttDriver._connection` each leak a transport on that race). The handler
therefore checks `Sender` against the child it currently holds and drops anything else; the deferred
path carries the original ref so the same check still applies when the timer re-delivers it.
- **Ordering on a respawning deploy, and its one accepted cost.** `ApplyDriverSpecs` (which spawns) runs
before the pass that rebuilds `_desiredByDriver`, so a child respawned by a deploy — raw tags are folded
into the merged `DriverConfig`, so a tag-set edit changes the spec and trips the #516 config respawn —