fix(runtime): rate-limit the supervision reprovision — deferred, not dropped (review F3)
The default decider restarts a faulting child immediately and without limit, and the hook makes every restart do ShutdownAsync+InitializeAsync+SubscribeAsync, so a driver faulting on every value became a reconnect storm. Per-driver last-reprovision stamp + DefaultReprovisionBackoff (10s, matched to DefaultReconnectInterval); a held request is re-delivered by a single per-driver timer carrying the original incarnation ref, so the F2 identity check still applies. Seen RED with the backoff disabled: InitializeCount == 3 (second applied immediately) where the test requires 2-then-3. Also: the _children.Keys union-arm test (seeder gained taglessDrivers) and the _desiredByDriver xmldoc note that it lingers on last-known-good after a failed apply.
This commit is contained in:
@@ -201,6 +201,24 @@ Things to know before touching any of this:
|
||||
remove a driver; a bounded retry count would silently stop one after N crashes. Both change failure
|
||||
semantics for every driver in the fleet and deserve their own plan and live gate. The reprovision hook
|
||||
makes the *current* infinite-restart behaviour correct instead of dead.
|
||||
- ⚠️ **…which is exactly why the reprovision is RATE-LIMITED.** The default decider restarts a faulting
|
||||
child **immediately and without limit**, and the hook now makes every one of those restarts do
|
||||
`ShutdownAsync` + `InitializeAsync` + `SubscribeAsync` — so a driver that faults on every value would
|
||||
hammer its device as fast as it can crash. `DriverHostActor` keeps a per-driver last-reprovision stamp
|
||||
and holds anything inside `DefaultReprovisionBackoff` (10 s, matched to
|
||||
`DriverInstanceActor.DefaultReconnectInterval` because it drives the same connect work). The held
|
||||
request is **deferred, not dropped** — one `StartSingleTimer` per driver, so a burst collapses to a
|
||||
single pending reprovision — because dropping it would leave a child whose *final* crash fell inside
|
||||
the window un-provisioned forever, which is the 03/S16 failure this hook exists to close. Each
|
||||
deferral logs a Warning naming the crash type, so a crash loop is visible rather than merely quiet.
|
||||
- **A reprovision is routed by INCARNATION, not by driver id.** `ReprovisionRequested` is delivered
|
||||
asynchronously, so one can still be in flight from an old incarnation after `HandleRestartDriver`
|
||||
stopped that child and spawned a replacement under the same id. Delivered to the new child — which is
|
||||
in `Connecting`, where `InitializeRequested` *is* handled — it would start a second, concurrent
|
||||
`InitializeAsync`, racing the dispose-before-overwrite that #527's teardown exists to serialise
|
||||
(`ModbusDriver._transport` / `MqttDriver._connection` each leak a transport on that race). The handler
|
||||
therefore checks `Sender` against the child it currently holds and drops anything else; the deferred
|
||||
path carries the original ref so the same check still applies when the timer re-delivers it.
|
||||
- **Ordering on a respawning deploy, and its one accepted cost.** `ApplyDriverSpecs` (which spawns) runs
|
||||
before the pass that rebuilds `_desiredByDriver`, so a child respawned by a deploy — raw tags are folded
|
||||
into the merged `DriverConfig`, so a tag-set edit changes the spec and trips the #516 config respawn —
|
||||
|
||||
Reference in New Issue
Block a user