Merge branch 'fix/r3-03-restart-resubscribe-watchdog' — R3-03: SpawnChild always sends the desired set (operator Restart + supervision restart re-provision, stale-alarm-set re-subscribe, incarnation check, 10 s backoff); watchdog grace = max(LastSuccessfulRead, connectedSince) (archreview 03/S16, S17, U6)

This commit is contained in:
Joseph Doherty
2026-08-20 10:40:11 -04:00
9 changed files with 1699 additions and 46 deletions
@@ -452,3 +452,321 @@ T8 anytime; T9 last (after T3/T5/T6/T7 merge-ready)
```
Overall effort: **~0.75–1 dev-day code+unit, plus the rig session.** The one HIGH-RISK production slice (T3, which changes what every child receives at spawn) lands behind the full 25-suite `DriverHostActor` regression before any new test stacks on it, and behind T2's explicitly behaviour-preserving refactor.
---
## Execution notes
Branch `fix/r3-03-restart-resubscribe-watchdog`, worktree
`.claude/worktrees/agent-a5bcdd61bfba22e10`, off master `5fd65d19`. TDD throughout: every
behaviour task's test was written first, RUN, and observed red for the predicted reason before the
production edit landed. Red observations are recorded per task below.
### T1 — 03/S17 watchdog grace (`??` → max)
**RED observed (as predicted).** With the two new tests in place against unmodified
`DriverInstanceActor`:
```
DriverInstanceActorCommsLossWatchdogTests.A_reconnected_driver_gets_a_full_grace_window_before_the_watchdog_trips_again [FAIL]
Failed: Expected no messages during 00:00:00.4000000, instead we received
<ConnectivityChanged { DriverInstanceId = stub-driver-1, Connected = False }> after 00:00:00.0496590
Failed! - Failed: 1, Passed: 9, Total: 10
```
The re-trip landed **49.7 ms** after the reconnect — one health tick (`FastPoll` = 50 ms) — against a
`Grace` of 1500 ms, i.e. exactly the "churn at health-tick cadence instead of the designed grace"
the finding describes. `A_half_broken_device_is_still_declared_lost_once_the_grace_elapses` passed
in the same run, which is what makes it the anti-vacuity control: it cannot distinguish the fix from
the bug, so the delay assertion is the one carrying the finding.
**GREEN after the one-line max at `:574`** (plus the "grace is per connection" remarks paragraph and
the reciprocal falsification-partner xmldoc on the existing
`A_degraded_driver_that_is_still_reading_successfully_is_left_alone`): 10/10 passed, 20 s.
`A_reconnect_attempt_tears_the_dead_transport_down_before_re_initialising` stayed green — as the plan
anticipated, each of its trip cycles now costs a full `FastTimeout` (250 ms) instead of one 50 ms
tick, and its `AwaitCondition(InitializeCount >= 2, 5 s)` still has ample headroom. No budget change
was needed.
### T2 — desired-subscription payload authority
Behaviour-preserving refactor, so there is no red observation to record by design — the gate is that
**no test moves**. `--filter "FullyQualifiedName~DriverHostActor"` → **114/114 passed**, unchanged
from the base commit.
⚠️ One run of that filter reported `DriverHostActorNativeAlarmAckRoutingTests.RouteNativeAlarmAck_on_non_primary_is_dropped`
failing on an `AwaitAssert`. Re-run in isolation: **3/3 passed**; re-run of the whole 114-test filter:
**114/114 passed**. Treated as the known #500 flake set for this assembly (~13 %/run), not a
regression — it touches none of the code this task edits (the alarm-ack Primary gate is a different
method entirely).
Deviation from the plan's literal step text (cosmetic, recorded for the reviewer): the plan's Finding-1
step 1 says `_desiredByDriver` *replaces* `_lastComposition` at `:347-349` while T3 step 2 says T3
deletes `_lastComposition`. Keeping T2 compilable means the two fields coexist for one commit;
`_lastComposition` and its assignment are deleted in T3 as T3's own step text specifies.
### T3 — `SpawnChild` always sends the desired set; `_lastComposition` deleted
The falsification for this production slice is **T5**, written and observed red afterwards (below) —
T3 lands the fix first because the plan sequences it that way and the ordering guard needs the whole
25-suite regression before any new test is trusted on top of it.
**HIGH-RISK regression gate (every child's spawn-time state changes, deploy path included):** the
FULL `Runtime.Tests` project, not just the `DriverHostActor` filter —
**639 passed, 3 skipped, 0 failed, 55 s**. The 3 skips are pre-existing (`DeploymentArtifact*` parity
tests, skipped on the base commit too).
`_lastComposition` is gone: `grep -c _lastComposition DriverHostActor.cs` → **0**. Its xmldoc had also
drifted (it claimed the field was set at the end of `PushDesiredSubscriptions`; the assignment was in
`ApplyDesiredSubscriptions`) — deleted with the field rather than corrected. The stale
"dropped-driver fallback" comment on the old `SendAuthoredOnly` local function was rewritten as the
`SendDesiredSubscriptions` remarks naming the three real consumers.
### T4 — test harness (seeder `enabled` flag + recording driver factory)
Test-only, additive, source-compatible: every pre-existing `SeedV3Deployment` / `SpawnHostAndApply`
call site compiles and behaves identically (the new parameters are trailing and default to today's
values). Gate: `--filter "…~DriverHostActor|…~DriverInstanceActor"` → **179/179 passed** (the plan
named `--filter "…~V3"`, which matches nothing — the seeder's four consumers are all `DriverHostActor*`
classes; the wider filter is the same set plus more).
**Deviation (recorded).** The plan lists one new seeder parameter (`enabled`); a second was needed and
added: `IReadOnlyDictionary<string, string>? tagConfigByTag = null`. The seeder hard-codes
`TagConfig = "{}"` on every tag, and a raw tag's alarm is derived *only* from `TagConfig.alarm` via
`TagConfigIntent.Parse` (`AddressSpaceComposer.cs:769-772`) — so without it there is no way to seed the
alarm-bearing raw tag that `Operator_restart_re_subscribes_alarm_refs_too` (a test the plan itself
specifies) requires. It is trailing and optional, so it is the same source-compatible shape as
`enabled`.
### T5 — `DriverHostActorRestartTests` (operator-restart falsification, 03/U6's first half)
**RED observed against a T3-reverted build** (the `SendDesiredSubscriptions(child, …)` call in
`SpawnChild` commented out, everything else on the branch): **all 4 FAILED, 0 passed**, each at the
bar the plan predicts —
```
Operator_restart_re_subscribes_the_respawned_child [FAIL]
factory.Created[1].SubscribeCount should be greater than or equal to 1 but was 0
Operator_restart_re_subscribes_alarm_refs_too [FAIL]
factory.Created[1].SubscribeAlarmsCount should be greater than or equal to 1 but was 0
Restart_of_a_driver_with_an_empty_value_set_still_sends_the_set [FAIL]
respawned.SubscribeAlarmsCount should be greater than or equal to 1 but was 0
A_child_spawned_by_a_fresh_deploy_ends_with_the_NEW_refs [FAIL]
factory.Created[2].LastSubscribedRefs should be ["Plant/Modbus/dev1/temp"] but was null
```
The call was then restored; **4/4 pass**, and the class was run **20× in isolation, 20/20 green** (#500
discipline for this assembly).
**Two deviations, both found by running the tests rather than by reading:**
1. ⚠️ **The plan's claim that the transient spawn-time set "is never acted on" is FALSE, and the
production comment that repeated it has been corrected.** The plan reasons that the child is still
`Connecting` when the stale set arrives, so it is only stored. But the child runs **concurrently**
with the rest of `HandleArtifactForApply` — `Context.ActorOf` returns and the child starts
processing while the host is still building `refsByDriver` — so a driver whose `InitializeAsync`
completes instantly reaches Connected and subscribes the stale set *before* the bulk push arrives.
Measured directly: `SubscribeCount == 2` on the respawned child, first with revision A's ref and
then with B's. What Akka's per-pair ordering does guarantee is **convergence**, and
`HandleSubscribeAsync` drops the prior subscription before establishing the new one, so the cost is
one extra subscribe round-trip on a *respawning* deploy — a real device driver, whose initialise
opens a socket, rarely wins that race. Accepted rather than "fixed": withholding the spawn-time send
is the 03/S16 defect, and the operator-Restart and supervision paths have no bulk push coming at
all. The `SpawnChild` comment now states this instead of the plan's claim, and the test asserts
convergence (the plan's own stated acceptance — "the child's live subscription carries the NEW
refs") rather than pinning the race.
2. **`A_child_spawned_by_a_fresh_deploy_ends_with_the_NEW_refs` had to be re-shaped.** The plan
describes revision B as one "which respawns nothing but re-pushes". It respawns: the raw tags are
folded into the merged `DriverConfig`, so a tag-set edit changes the `DriverInstanceSpec` and trips
the #516 config respawn. That is the *stronger* arrangement for this test's subject (a brand-new
child spawned while the authority still holds A), so the test keeps the same name and asserts the
respawned child converges on B — plus a subsequent operator Restart, which pins that the authority
is repopulated **whole** per apply rather than accumulated.
3. **`Restart_of_a_driver_with_no_authored_tags_sends_an_empty_set_not_nothing` was renamed
`Restart_of_a_driver_with_an_empty_value_set_still_sends_the_set`, with a real positive control.**
The seeder derives `DriverInstances` from the raw-tag rows, so a driver with *zero* tags is not
expressible, and — more importantly — an all-empty set has **no driver-side observable at all**
(the child stores it and, having no handle, does nothing), which would have made the test vacuous:
"it did not subscribe" passes equally against a build that sends the respawned child nothing, i.e.
against the defect itself. The subject is therefore a driver whose only raw tag is alarm-bearing:
its value half is genuinely empty while its alarm half is not, and **both halves ride in one
message** — so `SubscribeAlarmsCount >= 1` proves the message ARRIVED and `SubscribeCount == 0`
then proves the empty value half was honoured. The falsification run above confirms the control
works: with T3 reverted this test fails on the *positive control*, not on the absence.
**Not covered, and worth a reviewer's eye:** the `_desiredByDriver` union's `_children.Keys` arm
(a running child absent from BOTH ref maps) is not reachable through this seeder and is untested.
### T6 — supervision reprovision hook
`DriverInstanceActor.PostRestart` (calling `base.PostRestart(reason)` first, so `PreStart` re-attaches
the rediscovery / host-status sources and restarts the health timer) → `ReprovisionRequested` →
`DriverHostActor.HandleReprovisionRequested`, registered in **all three** `Receive` blocks that already
accept `RestartDriver`. The falsification is T7, below.
Gate: `--filter "…~DriverHostActor|…~DriverInstanceActor"` → **183/183 passed** (179 before this task
plus T5's four), no test moved.
### T7 — first supervision-restart test + the force-reconnect verified negative (03/U6)
**Crash seam that landed: the planned one** — `RecordingSubscribableDriver.FireMalformedDataChange`
raises `OnDataChange` with a null `Snapshot`; `DriverInstanceActor.OnDataChangeForward` dereferences
`msg.Snapshot.SourceTimestampUtc` on the actor thread and NREs; Akka's default decider Restarts. No
fallback (`OneForOneStrategy` + an internal test message) was needed, so no production surface was
added for the test.
**RED observed against a T6-reverted build** (the `Context.Parent.Tell(new ReprovisionRequested(…))`
in `PostRestart` commented out):
```
Supervision_restart_reinitialises_and_re_subscribes_the_child [FAIL]
driver.InitializeCount should be 2 but was 1
"the restarted child was never re-initialised — it is sitting in Connecting around a driver
its own PostStop already shut down (03/S6, the worse half of 03/S16)"
```
**Deviation, and the reason for it — the first red run was AMBIGUOUS and the test was strengthened.**
`InitializeCount == 1` is equally consistent with "the child restarted and was not re-provisioned"
and with "the child never crashed at all", so a seam that silently stopped working would read as a
passing regression test forever. A preceding `AwaitAssert(ShutdownCount >= 1)` was added — `PreRestart`
runs `PostStop`, which calls `ShutdownAsync`, so that is the evidence a restart genuinely happened.
On the re-run of the falsification that assertion **passed** and the failure landed squarely on the
re-initialise, which is what makes the red observation above meaningful.
`Force_reconnect_keeps_the_desired_set_and_re_subscribes` passed in **both** the red and green runs —
correct, and the point: it is the verified negative (`ForceReconnect` reaches the same live actor, so
`_desiredRefs` survive and the reconnect's Connected entry re-applies them). It carries a 30 s budget
rather than `PresenceBudget`, because `SpawnChild` gives the child the production
`DefaultReconnectInterval` (10 s) and the retry is a periodic timer, so the reconnect cannot land
sooner; a budget is only spent on failure.
After restoring the hook: **6/6 pass**, and the class was run **20× in isolation, 20/20 green**.
### T8 — docs
`docs/DriverLifecycle.md`:
- New §"What a (re)spawn re-provisions" (placed before §IDriverHealthPublisher): `_desiredByDriver` as
the single authority, a table of the three incarnation routes plus the verified-negative Force
Reconnect, the supervision route's invisibility, why replacing the default decider is out of scope,
the respawning-deploy ordering **with the measured caveat that the transient is not inert**, and the
assert-a-function-not-existence rule naming the weaker existing test.
- §"The comms-loss watchdog": conjunct 2 restated as the max, plus a new paragraph on the
per-connection grace — why `??` was wrong, that it **paces** rather than ends the half-broken-device
cycle, that the watchdog is not disabled by the change, and the two tests that pin each half.
### Final verification
- `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → **0 errors** (904 warnings, all pre-existing analyzer
`OTOPCUA0001` noise in unrelated driver test projects).
- **Full `Runtime.Tests`: 645 passed, 3 skipped, 0 failed** (1 m 9 s). Baseline on the branch point was
637 + 3 skipped; the +8 are this plan's new tests (T1 ×2, T5 ×4, T7 ×2). The 3 skips are pre-existing
`DeploymentArtifact*` parity tests.
- The two other test projects that reference `ZB.MOM.WW.OtOpcUa.Runtime` were run as well, since T3
changes what every driver child receives at spawn: `ControlPlane.Tests` **262/262**, `Host.Tests`
**106/106**.
- T9 (live gate) deliberately left **pending** — run by the orchestrator.
### Summary of deviations (for the reviewer)
1. **The plan's "the transient spawn-time set is never acted on" is false** — measured, corrected in
both the production comment and the docs, and accepted with its cost stated (T5, deviation 1). This
is the one behavioural claim in the plan that did not survive contact with the code.
2. `A_child_spawned_by_a_fresh_deploy_ends_with_the_NEW_refs` re-shaped — revision B *does* respawn the
child (raw tags fold into the merged `DriverConfig`) (T5, deviation 2).
3. `Restart_of_a_driver_with_no_authored_tags_sends_an_empty_set_not_nothing` renamed +
re-based on an alarm-only driver, because an all-empty set has no driver-side observable and the
planned test would have been vacuous; the `_children.Keys` union arm remains untested (T5,
deviation 3).
4. Seeder gained a second optional parameter (`tagConfigByTag`) — the only way to seed an alarm-bearing
raw tag, which one of the plan's own tests needs (T4).
5. The supervision test gained a preceding `ShutdownCount >= 1` assertion, because the plan's red
observation was ambiguous with "the child never crashed" (T7).
6. `Force_reconnect_keeps_the_desired_set_and_re_subscribes` uses a 30 s budget, not `PresenceBudget` —
`SpawnChild` gives the child the production 10 s `DefaultReconnectInterval` (T7).
7. `_lastComposition` and `_desiredByDriver` coexist for one commit (T2) so T2 stays compilable; the
deletion lands in T3, as T3's own step text specifies.
---
## Review outcome
Verdict **FIX-FIRST**, 3 items. All three confirmed against the code, fixed TDD (test first, seen red
for the predicted reason, then fixed), one commit each.
### F1 [HIGH regression] — a changed alarm set latched (`edd01fcc`)
**Confirmed.** `SubscribeDesiredAlarms` and `HandleSubscribeAlarmsAsync` both gated on
`_alarmSubscriptionHandle is not null`, which is the right guard for an idempotent re-push and the wrong
one for a deploy that changes the set. The value side converges by construction
(`StoreDesiredSubscriptions` is a full replace; `HandleSubscribeAsync` drops the prior subscription
first) — the alarm side did not. **T3 is what made it reachable**: a tag-set edit forces a respawn, and
a fast-initialising driver subscribes the transient spawn-time set before the bulk push, so the
PREVIOUS revision's alarm refs latched for the connection's life. `sourceNodeIds`-filtering drivers
(OPC UA Client, AB CIP, FOCAS) would silently never deliver the newly-added alarm.
**RED:** `A_deploy_that_widens_the_alarm_set_re_subscribes_the_new_refs` — after deploying
alarm `{alarmX}` then `{alarmX, alarmY}`, `LastSubscribedAlarmRefs` was still `["Plant/Modbus/dev1/alarmX"]`.
**Fix:** new `_subscribedAlarmRefs` records what the live handle was established with;
`AlarmSubscriptionIsStale()` (no handle **or** the desired refs differ) replaces the bare null check in
both places; a stale handle is **released before** re-subscribing so a filtering driver cannot
accumulate two overlapping subscriptions. The release clears the handle *first*, so a throwing
unsubscribe cannot leave the staleness check treating a dead handle as live, and it is best-effort
(logged, non-fatal — the re-subscribe is what matters). Unsubscribe-on-**empty** is still deliberately
absent, unchanged. Two notes: the release had to be wrapped in `_invoker.ExecuteAsync` or the repo's
`OTOPCUA0001` analyzer fails the build; and the Connected-handler comment that stated the
value-only convergence claim was corrected, as was the matching bullet in `docs/DriverLifecycle.md`.
### F2 [MEDIUM] — reprovision routed by id, not incarnation (`2b7100cf`)
**Confirmed.** Fixed with `if (!Equals(Sender, entry.Actor)) return;` (plus a Debug line naming the
superseded ref).
⚠️ **The first version of the test PASSED against the unfixed code — the arrangement mattered.** With
the child already **Connected**, `InitializeRequested` is unhandled, so the stale reprovision was
swallowed for the wrong reason and the test was vacuous. The defect only bites a child in
`Connecting`, which is exactly where `InitializeRequested` *is* handled. `RecordingSubscribableDriver`
therefore gained an `InitializeGate` (a `TaskCompletionSource` its `InitializeAsync` awaits) so the
test can hold the child in `Connecting`. **RED with that arrangement:** `InitializeCount == 2` — the
second concurrent `InitializeAsync` racing the #527 dispose-before-overwrite. The absence assertion
carries a positive control (the child is proven alive at `InitializeCount == 1` first) and a
`GetDiagnostics` round-trip so the dropped message is provably processed before the absence is read.
### F3 [IMPORTANT] — no backoff on the reprovision (`0793b0cb`)
**Confirmed.** Added `DefaultReprovisionBackoff` (10 s, matched to
`DriverInstanceActor.DefaultReconnectInterval` because the reprovision drives the same connect work),
a per-driver `_lastReprovisionUtc` stamp, and a `reprovisionBackoff` Props parameter following the
existing `applyWedgeTimeout` precedent so a test need not wait the production window.
A held request is **deferred, not dropped**: one `StartSingleTimer` per driver re-delivers a
`ReprovisionDeferred` carrying the **original incarnation ref**, so F2's identity check still applies
on re-delivery, and a burst collapses to a single pending reprovision rather than a queue. Each
deferral logs a Warning naming the crash type, so a crash loop is visible rather than quiet.
**RED** (backoff disabled): `InitializeCount == 3` where the test requires 2-then-3 — i.e. the second
reprovision applied immediately. The test drives **two genuine supervision restarts** (not synthetic
messages), waits for the re-subscribe before arming the second crash, and asserts `ShutdownCount >= 2`
so "the second crash tripped supervision" is proven before the backoff is read.
### Optional adds (both done)
- `A_running_driver_with_no_authored_tags_is_handed_an_empty_set` — the `_children.Keys` union arm,
previously untested and called out as a gap in the T5 notes. It needed a driver with no tags at all,
which the seeder could not express, so `SeedV3Deployment` gained `taglessDrivers`. The tagless child
is asserted initialised-but-never-subscribed, with its tag-owning sibling's subscribe as the positive
control that the push pass ran.
- `_desiredByDriver` xmldoc now records that it is rebuilt only by a **successful** apply, so it
lingers on the last-known-good composition after a failed one — consistent with the address space,
drivers and subscriptions that also keep serving (#485/#486).
### Verification after the three fixes
- `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → **0 errors**.
- **Full `Runtime.Tests`: 649 passed, 3 skipped, 0 failed** (up 4 from 645 — F1, F2, F3 and the
union-arm test).
- **`ControlPlane.Tests`: 262/262.**
- `DriverHostActorRestartTests` (now 10 cases) run **10× in isolation, 10/10 green**.
- T9 (live gate) still **pending** — the orchestrator runs it. Worth noting for that session: F1 is
directly observable on the rig, by deploying a second alarm-bearing raw tag to a driver that already
has one and confirming a transition on the NEW tag reaches `/alerts`.
@@ -5,19 +5,19 @@
{
"id": "T1",
"subject": "Comms-loss grace is per connection: ?? -> max(LastSuccessfulRead, _connectedSinceUtc) + falsification pair (03/S17)",
"status": "pending",
"status": "done",
"blockedBy": []
},
{
"id": "T2",
"subject": "Desired-subscription payload authority: _desiredByDriver + SendDesiredSubscriptions (behaviour-preserving refactor)",
"status": "pending",
"status": "done",
"blockedBy": []
},
{
"id": "T3",
"subject": "SpawnChild always sends the desired set; delete the write-only _lastComposition (03/S16)",
"status": "pending",
"status": "done",
"blockedBy": [
"T2"
]
@@ -25,13 +25,13 @@
{
"id": "T4",
"subject": "Test harness: V3ArtifactSeeder enabled flag + RecordingSubscribableDriver(Factory) with a crash seam",
"status": "pending",
"status": "done",
"blockedBy": []
},
{
"id": "T5",
"subject": "DriverHostActorRestartTests: operator restart re-subscribes the respawned child (falsifiable; 03/S16, U6)",
"status": "pending",
"status": "done",
"blockedBy": [
"T3",
"T4"
@@ -40,7 +40,7 @@
{
"id": "T6",
"subject": "Supervision reprovision hook: DriverInstanceActor.PostRestart -> ReprovisionRequested -> host re-init + re-subscribe (03/S6, S16)",
"status": "pending",
"status": "done",
"blockedBy": [
"T3"
]
@@ -48,7 +48,7 @@
{
"id": "T7",
"subject": "First real supervision-restart test + force-reconnect verified-negative regression (03/U6)",
"status": "pending",
"status": "done",
"blockedBy": [
"T4",
"T6"
@@ -57,7 +57,7 @@
{
"id": "T8",
"subject": "docs/DriverLifecycle.md: what a (re)spawn re-provisions + per-connection watchdog grace",
"status": "pending",
"status": "done",
"blockedBy": []
},
{
+109 -1
View File
@@ -164,6 +164,93 @@ mostly reusable — what was always missing is the trigger.
---
## What a (re)spawn re-provisions
A driver child can come into existence by **three** routes, and until 2026-08-20 only the first of them
handed the child everything it needs to be a working driver. That is archreview **03/S16** (and its worse
half, **03/S6**): a child that exists without its desired subscription set is a *dead* driver — it
connects, reports `Connected`, publishes nothing and raises no alarms, and the #488 periodic reconcile
cannot rescue it (it early-returns on an empty desired set). The fix makes the desired set **part of what
it means to have a child** rather than a later step in the apply pipeline.
**`DriverHostActor._desiredByDriver` is the single authority.** It is a per-driver
`(ValueRefs, AlarmRefs)` pair, cleared and repopulated **whole** on every apply in the same pass that
rebuilds the routing maps — keyed over the union of the value-ref map, the alarm-ref map and the running
`_children`, so a driver with no authored tags gets an explicit **empty** set rather than being absent.
Everything that hands a child a `SetDesiredSubscriptions` goes through the one helper
`SendDesiredSubscriptions(actor, driverId)`. (An empty set is meaningful, not a no-op to skip: the child
routes it to `Unsubscribe`, dropping a stale handle.)
| Route | Trigger | What re-provisions it |
|---|---|---|
| **Deploy reconcile** | `ApplyDriverSpecs` → `SpawnChild` | `SpawnChild` sends `InitializeRequested` then the desired set; the apply's bulk push then sends the (freshly rebuilt) set to every child. |
| **Operator Restart** | `/hosts` → Restart → `HandleRestartDriver` | Stops the child, respawns from the cached spec — and `SpawnChild` sends the desired set, because it always does. **This is the 03/S16 fix**: there is no bulk push on this path, and before it the respawned child subscribed to nothing until the next deployment. |
| **Supervision restart** | a child throws ⇒ Akka's **default decider** ⇒ `Restart` | `DriverInstanceActor.PostRestart` Tells the host `ReprovisionRequested`; `HandleReprovisionRequested` logs a Warning naming the crash type, re-sends `InitializeRequested` (non-stub children) and the desired set. |
| **Force Reconnect** | `/hosts` → Force Reconnect → `HandleReconnectDriver` | **Nothing — verified.** It Tells `ForceReconnect` to the *same* child actor, so the instance and its `_desiredRefs`/`_desiredAlarmRefs` survive and the reconnect's Connected entry re-applies them itself. Pinned by `Force_reconnect_keeps_the_desired_set_and_re_subscribes` so a future edit to the restart paths cannot break the one that was already correct. |
Things to know before touching any of this:
- ⚠️ **The supervision route is the invisible one, and it was the worst.** There is **no**
`SupervisorStrategy` anywhere under `src/Server`, so a child that throws gets Akka's default `Restart`.
A restart recreates the actor object around the **same** `IDriver` — the Props closure captured it —
and its `PreRestart` runs `PostStop`, which calls `IDriver.ShutdownAsync`. It does **not** run
`SpawnChild`, and the child's `IActorRef` is unchanged, so the host had no notification a new
incarnation existed at all. Net effect before the hook: the child sat in `Connecting` forever around a
shut-down driver, with its desired set gone, and nothing logged the non-recovery.
- **Replacing the default decider is deliberately out of scope.** A `Stop` directive would silently
remove a driver; a bounded retry count would silently stop one after N crashes. Both change failure
semantics for every driver in the fleet and deserve their own plan and live gate. The reprovision hook
makes the *current* infinite-restart behaviour correct instead of dead.
- ⚠️ **…which is exactly why the reprovision is RATE-LIMITED.** The default decider restarts a faulting
child **immediately and without limit**, and the hook now makes every one of those restarts do
`ShutdownAsync` + `InitializeAsync` + `SubscribeAsync` — so a driver that faults on every value would
hammer its device as fast as it can crash. `DriverHostActor` keeps a per-driver last-reprovision stamp
and holds anything inside `DefaultReprovisionBackoff` (10 s, matched to
`DriverInstanceActor.DefaultReconnectInterval` because it drives the same connect work). The held
request is **deferred, not dropped** — one `StartSingleTimer` per driver, so a burst collapses to a
single pending reprovision — because dropping it would leave a child whose *final* crash fell inside
the window un-provisioned forever, which is the 03/S16 failure this hook exists to close. Each
deferral logs a Warning naming the crash type, so a crash loop is visible rather than merely quiet.
- **A reprovision is routed by INCARNATION, not by driver id.** `ReprovisionRequested` is delivered
asynchronously, so one can still be in flight from an old incarnation after `HandleRestartDriver`
stopped that child and spawned a replacement under the same id. Delivered to the new child — which is
in `Connecting`, where `InitializeRequested` *is* handled — it would start a second, concurrent
`InitializeAsync`, racing the dispose-before-overwrite that #527's teardown exists to serialise
(`ModbusDriver._transport` / `MqttDriver._connection` each leak a transport on that race). The handler
therefore checks `Sender` against the child it currently holds and drops anything else; the deferred
path carries the original ref so the same check still applies when the timer re-delivers it.
- **Ordering on a respawning deploy, and its one accepted cost.** `ApplyDriverSpecs` (which spawns) runs
before the pass that rebuilds `_desiredByDriver`, so a child respawned by a deploy — raw tags are folded
into the merged `DriverConfig`, so a tag-set edit changes the spec and trips the #516 config respawn —
is first handed the **previous** apply's set, then the new one by the bulk push. Akka's per-pair
ordering guarantees it **converges**; it does not guarantee the transient is inert, because the child
runs concurrently with the rest of the apply handler and a fast-initialising driver can reach
`Connected` and subscribe the stale set first. The cost is one extra subscribe round-trip
(`HandleSubscribeAsync` drops the prior subscription before establishing the new one); a real device
driver, whose initialise opens a socket, rarely wins that race. Withholding the spawn-time send to avoid
it would simply restore 03/S16, since the Restart and supervision routes have no bulk push coming.
- ⚠️ **Convergence is not free on the ALARM side, and "a subscription already exists" is the wrong guard
there.** The value path converges by construction — `StoreDesiredSubscriptions` is a full replace and
`HandleSubscribeAsync` drops the prior subscription before establishing the new one — but
`SubscribeDesiredAlarms` / `HandleSubscribeAlarmsAsync` used to gate purely on
`_alarmSubscriptionHandle is not null`, so once a handle existed the alarm set **latched for the life of
the connection**. Combined with the transient above, a deploy widening a driver's alarm refs `{X}` →
`{X, Y}` could leave a fast-initialising driver subscribed to `{X}` forever, and the drivers that filter
their feed on the subscribed `sourceNodeIds` (OPC UA Client, AB CIP, FOCAS) would simply never deliver
the new alarm — silently. The guard is now "the desired refs differ from the ones the live handle was
established with" (`_subscribedAlarmRefs`), and a changed set **releases the old subscription before
re-subscribing** so a filtering driver does not accumulate two overlapping ones. There is still
deliberately no unsubscribe-on-**empty**: a removed alarm tag's condition node is torn down on the
address-space rebuild and `ForwardNativeAlarm` drops any transition whose `ConditionId` no longer maps.
- **Restart tests must assert a FUNCTION, never existence.** `RestartDriver_respawns_the_child_without_an_actor_name_collision`
asserts only that a child exists after a restart, and it passed green against the dead driver 03/S16
describes. It is kept — its own subject, the `InvalidActorNameException`, is real — but the load-bearing
suite is `DriverHostActorRestartTests`, which asserts that the new incarnation *subscribed* and
*re-initialised*. Its file remarks also record the supervision crash seam and why the obvious
alternatives (a throwing health publisher, a throwing `SubscribeAsync`, `Kill.Instance`) do not work.
---
## IDriverHealthPublisher — health pub/sub sink
[`Core.Abstractions/IDriverHealthPublisher.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverHealthPublisher.cs)
@@ -236,7 +323,8 @@ The `Connected`-state health tick now also evaluates a watchdog, and self-sends
1. the driver reports a state other than `Healthy`, **and**
2. `DriverHealth.LastSuccessfulRead` has not advanced for `DefaultCommsLossTimeout` (90 s = three ticks),
measured from the Connected entry when the driver has never reported a successful read.
measured from **the later of that read stamp and the current Connected entry** (see "the grace is per
connection" below).
**The conjunction is the whole design, and "non-Healthy for N polls" is the wrong rule.** `Degraded` means
*connected but some data is bad* — the enum says so. Several drivers publish health **per tag inside their
@@ -250,6 +338,26 @@ still needed as the second conjunct, because a `Healthy` driver with nothing sub
reads and must not be declared dead for being idle. Both halves are pinned by their own test in
`DriverInstanceActorCommsLossWatchdogTests`, each of which goes red when the *other* conjunct is deleted.
**The grace is per CONNECTION — the baseline is `max(LastSuccessfulRead, connectedSince)`, not
`LastSuccessfulRead ?? connectedSince`** (archreview 03/S17, fixed 2026-08-20). The two spellings differ
for exactly the case that matters, and the `??` one collapsed the grace to a single health tick. Because
`LastSuccessfulRead` is *preserved* across `Degraded`/`Faulted` writes — that preservation is the design
described above — a device that accepts a TCP connection but never serves data (a half-broken PLC, a wrong
unit id, a firewalled read path) reconnects carrying a read stamp frozen from before the outage, already
far older than the threshold. Measuring from that stamp alone re-trips on the **first** tick of every new
connection, so the reconnect cycle ran at health-tick cadence (~30 s in production) rather than at the
designed 90 s. Taking the *later* of the two gives each connection its own full window, which is what
re-stamping `connectedSince` on every Connected entry was always for; `??` only ever helped a driver that
had never read at all, and the max still covers that case.
What this does **not** change: the conjunction, the disable-at-zero hatch, and a driver that stops reading
*within* a stable connection (its read stamp is the later term, so it still trips on schedule measured
from that read). And it **paces** the half-broken-device cycle rather than ending it — expect at most one
reconnect attempt per threshold per connection, not silence. Pinned by
`A_reconnected_driver_gets_a_full_grace_window_before_the_watchdog_trips_again` (the delay) and
`A_half_broken_device_is_still_declared_lost_once_the_grace_elapses` (the anti-vacuity half — without it,
a "fix" that simply disabled the watchdog after the first connection would look correct).
Two consequences worth knowing:
- **`CalculationDriver` can never trip it, correctly.** It reports `Healthy` whenever it is running and has
@@ -82,6 +82,14 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
/// </remarks>
public static readonly TimeSpan DefaultApplyWedgeTimeout = TimeSpan.FromMinutes(5);
/// <summary>Minimum interval between two supervision reprovisions of the SAME driver child (archreview
/// review F3). Akka's default decider restarts a faulting child immediately and without limit, and the
/// reprovision hook makes every restart do ShutdownAsync + InitializeAsync + SubscribeAsync — so an
/// un-limited hook turns a driver that faults on every value into a reconnect storm against its device.
/// Matched to <see cref="DriverInstanceActor.DefaultReconnectInterval"/>: the reprovision drives the same
/// connect work, so it should not run more often than an ordinary reconnect attempt.</summary>
public static readonly TimeSpan DefaultReprovisionBackoff = TimeSpan.FromSeconds(10);
/// <summary>
/// Cache key used when an artifact carries no cluster scoping (single-cluster or unscoped
/// deployments). A literal rather than an empty string so the row is visibly deliberate when
@@ -344,9 +352,25 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
/// the config (#516). Bounded by the number of driver children with a delta in flight.</summary>
private readonly Dictionary<CorrelationId, DriverInstanceSpec> _pendingDelta = new();
/// <summary>The composition from the most-recent apply (set at the END of
/// <see cref="PushDesiredSubscriptions"/>). Null until the first apply.</summary>
private AddressSpaceComposition? _lastComposition;
/// <summary>One driver child's desired subscription payload — exactly the
/// <see cref="DriverInstanceActor.SetDesiredSubscriptions"/> the apply's bulk push sent it.</summary>
/// <param name="ValueRefs">The non-alarm raw tags' RawPaths this driver should keep subscribed.</param>
/// <param name="AlarmRefs">The alarm-bearing raw tags' RawPaths that un-gate this driver's
/// <see cref="IAlarmSource"/> feed.</param>
private readonly record struct DesiredSubscriptionSet(
IReadOnlyList<string> ValueRefs,
IReadOnlyList<string> AlarmRefs);
/// <summary>Per-driver desired subscription sets from the most-recent apply. THE authority: the bulk
/// push, <see cref="SpawnChild"/> and the supervision-reprovision handler all send from here, so a
/// child obtained by ANY route carries the same set. Cleared and repopulated whole on every apply,
/// alongside the routing maps, so renames/removals can never leave a stale entry. Empty until the
/// first apply — a child spawned before any apply is handed nothing, which is what it should have.
/// <para>It is only ever rebuilt by a SUCCESSFUL apply, so after a failed one it deliberately lingers on
/// the last-good composition — matching the address space, drivers and subscriptions that also keep
/// serving through a failed apply (#485/#486). A respawn or reprovision in that window therefore hands
/// the child the last-known-good set, not an empty one.</para></summary>
private readonly Dictionary<string, DesiredSubscriptionSet> _desiredByDriver = new(StringComparer.Ordinal);
/// <summary>
/// Cached local <see cref="RedundancyRole"/> from the latest <see cref="RedundancyStateChanged"/>
@@ -401,6 +425,15 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
/// <see cref="DefaultApplyWedgeTimeout"/>).</summary>
private readonly TimeSpan _applyWedgeTimeout;
/// <summary>Per-driver rate limit for the supervision-reprovision hook (see
/// <see cref="DefaultReprovisionBackoff"/>).</summary>
private readonly TimeSpan _reprovisionBackoff;
/// <summary>When each driver was last reprovisioned after a supervision restart, so a crash loop is
/// paced rather than amplified. Bounded by the number of driver children; an entry for a driver that is
/// later removed simply stops being consulted (the handler resolves <c>_children</c> first).</summary>
private readonly Dictionary<string, DateTime> _lastReprovisionUtc = new(StringComparer.Ordinal);
/// <summary>
/// Host of this node's LocalDb replication partner — the only node that holds a copy of this
/// node's alarm queue, and therefore the only node it may ever stand down in favour of.
@@ -542,7 +575,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
IDriverPrimaryRoleSource? primaryRoleSource = null,
IHistorizationRoleView? historizationRoleView = null,
Func<CommonsNodeId, bool>? sharesHistorizationScopeWithLocal = null,
TimeSpan? applyWedgeTimeout = null) =>
TimeSpan? applyWedgeTimeout = null,
TimeSpan? reprovisionBackoff = null) =>
// WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
// expression tree. Six IActorRef? parameters and several interface-typed ones mean a
// mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
@@ -554,7 +588,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider,
deploymentArtifactCache, redundancyRoleView, replicationPeerHost, fetchAndCacheMode, artifactFetcher,
alarmStateStore, telemetryHub, primaryRoleSource, historizationRoleView,
sharesHistorizationScopeWithLocal, applyWedgeTimeout));
sharesHistorizationScopeWithLocal, applyWedgeTimeout, reprovisionBackoff));
/// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
/// <param name="dbFactory">Database context factory for configuration database access.</param>
@@ -640,9 +674,11 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
IDriverPrimaryRoleSource? primaryRoleSource = null,
IHistorizationRoleView? historizationRoleView = null,
Func<CommonsNodeId, bool>? sharesHistorizationScopeWithLocal = null,
TimeSpan? applyWedgeTimeout = null)
TimeSpan? applyWedgeTimeout = null,
TimeSpan? reprovisionBackoff = null)
{
_applyWedgeTimeout = applyWedgeTimeout ?? DefaultApplyWedgeTimeout;
_reprovisionBackoff = reprovisionBackoff ?? DefaultReprovisionBackoff;
_historizationRoleView = historizationRoleView;
_sharesHistorizationScopeWithLocal = sharesHistorizationScopeWithLocal;
_telemetryHub = telemetryHub;
@@ -1076,6 +1112,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
Receive<DriverInstanceActor.DeltaApplied>(HandleDeltaApplied);
Receive<DriverInstanceActor.ApplyResult>(HandleApplyResult);
Receive<RestartDriver>(HandleRestartDriver);
Receive<DriverInstanceActor.ReprovisionRequested>(HandleReprovisionRequested);
Receive<ReprovisionDeferred>(m => ReprovisionChild(m.Request, m.Child));
Receive<ReconnectDriver>(HandleReconnectDriver);
Receive<RouteNodeWrite>(HandleRouteNodeWrite);
Receive<RouteNativeAlarmAck>(HandleRouteNativeAlarmAck);
@@ -1131,6 +1169,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
Receive<DriverInstanceActor.DeltaApplied>(HandleDeltaApplied);
Receive<DriverInstanceActor.ApplyResult>(HandleApplyResult);
Receive<RestartDriver>(HandleRestartDriver);
Receive<DriverInstanceActor.ReprovisionRequested>(HandleReprovisionRequested);
Receive<ReprovisionDeferred>(m => ReprovisionChild(m.Request, m.Child));
Receive<ReconnectDriver>(HandleReconnectDriver);
Receive<RouteNodeWrite>(HandleRouteNodeWrite);
Receive<RouteNativeAlarmAck>(HandleRouteNativeAlarmAck);
@@ -1713,6 +1753,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
Receive<GetDiagnostics>(HandleGetDiagnostics);
Receive<RetryConfigDbConnection>(_ => TryRecoverFromStale());
Receive<RestartDriver>(HandleRestartDriver);
Receive<DriverInstanceActor.ReprovisionRequested>(HandleReprovisionRequested);
Receive<ReprovisionDeferred>(m => ReprovisionChild(m.Request, m.Child));
Receive<ReconnectDriver>(HandleReconnectDriver);
Receive<RedundancyStateChanged>(OnRedundancyStateChanged);
// The redundancy-state topic also carries OpcUaProbeResult (OpcUaPublishActor peer-probes).
@@ -2652,23 +2694,25 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
}
}
// One authored-only push (value refs + alarm refs from the maps built above), shared by the bulk loop AND
// the dropped-driver fallback so the two CANNOT drift: the fallback's correctness depends on sending the
// SAME payload the bulk loop would have, so it's a shared helper (structural), not a comment-maintained
// invariant. An EMPTY set is valid — the child's Connected handler routes it to Unsubscribe (dropping a
// stale handle) rather than a spurious subscribe. Returns the value-ref count for the bulk-loop log total.
int SendAuthoredOnly(IActorRef actor, string driverId)
// Cache the per-driver payload as THE authority, in the same clear-and-repopulate pass as the routing
// maps above so a rename/removal can never leave a stale entry behind. Keyed over the UNION of the two
// ref maps AND the running children: a child with no authored tags must get an explicit EMPTY set (which
// it routes to Unsubscribe), not be absent from the authority — absence and empty must not differ here.
_desiredByDriver.Clear();
foreach (var driverId in refsByDriver.Keys
.Concat(alarmRefsByDriver.Keys)
.Concat(_children.Keys)
.Distinct(StringComparer.Ordinal))
{
var refs = refsByDriver.TryGetValue(driverId, out var r) ? r : Array.Empty<string>();
var alarmRefs = alarmRefsByDriver.TryGetValue(driverId, out var ar) ? ar : Array.Empty<string>();
actor.Tell(new DriverInstanceActor.SetDesiredSubscriptions(refs, SubscriptionPublishingInterval, alarmRefs));
return refs.Count;
_desiredByDriver[driverId] = new DesiredSubscriptionSet(
refsByDriver.TryGetValue(driverId, out var r) ? r : Array.Empty<string>(),
alarmRefsByDriver.TryGetValue(driverId, out var ar) ? ar : Array.Empty<string>());
}
var total = 0;
foreach (var (driverId, entry) in _children)
{
total += SendAuthoredOnly(entry.Actor, driverId);
total += SendDesiredSubscriptions(entry.Actor, driverId);
}
if (total > 0)
@@ -2703,11 +2747,34 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
_log.Info("DriverHost {Node}: applied {Count} Equipment ScriptedAlarm(s) to the ScriptedAlarm host",
_localNode, composition.EquipmentScriptedAlarms.Count);
}
}
// Cache the applied composition LAST. Set here (not in HandleArtifactForApply) so both the fresh-apply and
// bootstrap-restore paths — which both route through this method — leave a current composition.
_lastComposition = composition;
/// <summary>
/// Send <paramref name="driverId"/>'s desired subscription payload to <paramref name="actor"/>.
/// </summary>
/// <remarks>
/// The ONE place a <see cref="DriverInstanceActor.SetDesiredSubscriptions"/> is built, so the apply's
/// bulk push, a respawn (<see cref="SpawnChild"/>) and a supervision reprovision
/// (<c>HandleReprovisionRequested</c>) cannot drift. That is structural rather than
/// comment-maintained on purpose: a second sender that assembled its own payload is exactly how
/// 03/S16 happened — the restart path got a child without the set that makes a child useful.
/// <para>
/// An EMPTY set is valid and MEANINGFUL — the child routes it to <c>Unsubscribe</c> (dropping a
/// stale handle) rather than to a spurious subscribe — so a driver missing from
/// <see cref="_desiredByDriver"/> is sent an empty set, never skipped.
/// </para>
/// </remarks>
/// <param name="actor">The driver child to hand the set to.</param>
/// <param name="driverId">The driver instance whose set to send.</param>
/// <returns>The value-ref count, for the bulk loop's log total.</returns>
private int SendDesiredSubscriptions(IActorRef actor, string driverId)
{
var set = _desiredByDriver.TryGetValue(driverId, out var s)
? s
: new DesiredSubscriptionSet(Array.Empty<string>(), Array.Empty<string>());
actor.Tell(new DriverInstanceActor.SetDesiredSubscriptions(
set.ValueRefs, SubscriptionPublishingInterval, set.AlarmRefs));
return set.ValueRefs.Count;
}
private void SpawnChild(DriverInstanceSpec spec)
@@ -2781,6 +2848,29 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
_children[spec.DriverInstanceId] = new ChildEntry(child, spec, stub);
// EVERY spawn hands the child its desired set — deploy reconcile, operator Restart (03/S16), and the
// supervision reprovision. Sending it HERE rather than at each call site is the point: a child that
// exists without its subscription set is a dead driver, so the two facts are established together.
// Before this, the only sender was the apply's bulk push, so an operator's Restart produced a child
// that connected and then read nothing until the next deployment — the #488 periodic reconcile cannot
// help, because it early-returns on an empty desired set.
//
// ORDERING, and its one accepted cost: ApplyDriverSpecs (which spawns) runs BEFORE
// ApplyDesiredSubscriptions (which rebuilds _desiredByDriver), so on a deploy that RESPAWNS a child
// (#516 — the merged DriverConfig folds in the raw tags, so a tag-set edit changes the spec) the new
// child is handed the PREVIOUS apply's set, and nothing at all on the first deploy. That transient is
// superseded a moment later by the bulk push, on the same sender→receiver pair, so Akka's per-pair
// ordering guarantees the child CONVERGES on the new set — but it is NOT guaranteed to be inert on
// the way there: the child runs concurrently with the rest of this handler, so a driver whose
// InitializeAsync completes quickly can reach Connected and subscribe the stale set before the bulk
// push arrives, then re-subscribe. The cost is one extra subscribe/unsubscribe round-trip on a
// respawning deploy (HandleSubscribeAsync drops the prior subscription before establishing the new
// one, so it converges); a real device driver, whose initialise opens a socket, rarely wins that race.
// Deliberately not "fixed" by withholding the send: a child that never gets a set is the 03/S16
// defect, and the operator-Restart and supervision paths have no bulk push coming at all.
// Sent after InitializeRequested above, which keeps init-before-subscribe ordering.
SendDesiredSubscriptions(child, spec.DriverInstanceId);
// v3 (WP7): if the (real, non-stub) driver consumes other tags' live values, spawn a mux-adapter
// child that registers its dependency refs on the per-node dependency mux and forwards changes
// into the driver. Requires a wired mux (dev/None deployments have none — then calc inputs simply
@@ -2931,6 +3021,87 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
SpawnChild(entry.Spec);
}
/// <summary>
/// Re-provision a driver child that Akka's supervision just RESTARTED in place (archreview 03/S6,
/// 03/S16) — the third and least visible of the three incarnation routes.
/// </summary>
/// <remarks>
/// A restart recreates the actor object around the SAME driver instance (the Props closure captured
/// it) without going through <see cref="SpawnChild"/>, so neither the <c>InitializeRequested</c> nor
/// the desired-subscription send that a spawn performs ever happens — and the child's
/// <see cref="IActorRef"/> is unchanged, so <c>_children</c> needs no maintenance and the host has no
/// other notification that a new incarnation exists. Its <c>PreRestart</c> ran <c>PostStop</c>, which
/// called <c>ShutdownAsync</c> on the driver, so re-initialising is not optional: without it the
/// child sits in <c>Connecting</c> forever around a shut-down driver.
/// <para>
/// The dependency-consumer mux adapter is a SEPARATE child of this host and is unaffected by the
/// instance actor restarting, so it is deliberately not touched here.
/// </para>
/// </remarks>
/// <param name="msg">The restarted child's self-announcement.</param>
private void HandleReprovisionRequested(DriverInstanceActor.ReprovisionRequested msg) =>
ReprovisionChild(msg, Sender);
/// <summary>A reprovision held back by <see cref="_reprovisionBackoff"/>, carrying the incarnation it
/// was requested by so the identity check still applies when it is re-delivered by the timer.</summary>
/// <param name="Request">The original request.</param>
/// <param name="Child">The incarnation that asked; re-validated against <c>_children</c> on delivery,
/// so a driver restarted or removed in the meantime discards it.</param>
private sealed record ReprovisionDeferred(
DriverInstanceActor.ReprovisionRequested Request, IActorRef Child);
private static string ReprovisionTimerKey(string driverInstanceId) => "reprovision-" + driverInstanceId;
private void ReprovisionChild(DriverInstanceActor.ReprovisionRequested msg, IActorRef child)
{
// Already stopped (a restart racing a reconcile that removed the driver) — nothing to provision.
if (!_children.TryGetValue(msg.DriverInstanceId, out var entry)) return;
// Route by INCARNATION, not just by id. This message is delivered asynchronously, so one can still
// be in flight from an OLD incarnation after HandleRestartDriver stopped that child and spawned a
// replacement under the same DriverInstanceId. Delivered to the new child — which is in Connecting,
// where InitializeRequested IS handled — it would start a second, concurrent InitializeAsync,
// racing the dispose-before-overwrite that #527's teardown exists to serialise
// (ModbusDriver._transport, MqttDriver._connection each leak a transport on that race).
if (!Equals(child, entry.Actor))
{
_log.Debug(
"DriverHost {Node}: dropping a reprovision for {Id} from a superseded incarnation ({Stale})",
_localNode, msg.DriverInstanceId, child);
return;
}
// Rate-limit per driver (review F3). DEFERRED, never dropped: the last request must still be
// honoured, or a child whose final crash fell inside the window is left un-provisioned forever —
// exactly the 03/S16 failure this hook exists to close. One timer key per driver, so a burst
// collapses to a single pending reprovision rather than a queue of them.
var now = DateTime.UtcNow;
if (_lastReprovisionUtc.TryGetValue(msg.DriverInstanceId, out var last) &&
now - last < _reprovisionBackoff)
{
var delay = _reprovisionBackoff - (now - last);
_log.Warning(
"DriverHost {Node}: driver child {Id} was restarted by supervision again after {Reason} " +
"within {Backoff} — deferring the re-provision by {Delay} (crash loop?)",
_localNode, msg.DriverInstanceId, msg.Reason, _reprovisionBackoff, delay);
Timers.StartSingleTimer(
ReprovisionTimerKey(msg.DriverInstanceId), new ReprovisionDeferred(msg, child), delay);
return;
}
_lastReprovisionUtc[msg.DriverInstanceId] = now;
// A crash that Akka silently restarted is otherwise invisible to operators; name it.
_log.Warning(
"DriverHost {Node}: driver child {Id} was restarted by supervision after {Reason} — re-provisioning",
_localNode, msg.DriverInstanceId, msg.Reason);
// A stub child has no real driver to initialise (its Stubbed behaviour no-ops InitializeRequested
// anyway), but it still stores a desired set, so only the initialise is conditional.
if (!entry.Stubbed) entry.Actor.Tell(new DriverInstanceActor.InitializeRequested(entry.Spec.DriverConfig));
SendDesiredSubscriptions(entry.Actor, msg.DriverInstanceId);
}
private void HandleReconnectDriver(ReconnectDriver msg)
{
// DPS broadcast — only act if this node hosts the requested instance.
@@ -42,6 +42,25 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// restored → Good) — independently of alarm transitions, since a comms-lost driver emits no alarm
/// events. Fire-and-forget, mirroring <see cref="DeltaApplied"/>.</summary>
public sealed record ConnectivityChanged(string DriverInstanceId, bool Connected);
/// <summary>
/// Sent to the parent from <see cref="PostRestart"/> so the host re-provisions THIS incarnation
/// (archreview 03/S6, 03/S16).
/// </summary>
/// <remarks>
/// A supervision restart recreates the actor object in place: the desired subscription sets are gone
/// (they are instance fields), and <c>PreRestart</c> has already run <see cref="PostStop"/> — which
/// called <see cref="IDriver.ShutdownAsync"/> on the driver object the Props closure re-uses. Nothing
/// else re-sends <see cref="InitializeRequested"/>: its only sender is
/// <c>DriverHostActor.SpawnChild</c>, and a restart does NOT respawn (the child's
/// <see cref="IActorRef"/> is unchanged, so the host is not otherwise notified that a new incarnation
/// exists at all). Without this the restarted child sits in <c>Connecting</c> forever around a
/// shut-down driver, with nothing subscribed — silently, since Akka logs the fault but the recovery
/// is invisible.
/// </remarks>
/// <param name="DriverInstanceId">The driver instance that was restarted.</param>
/// <param name="Reason">The type name of the exception that tripped supervision, for the host's log.</param>
public sealed record ReprovisionRequested(string DriverInstanceId, string Reason);
public sealed record ApplyDelta(string DriverConfigJson, CorrelationId Correlation);
public sealed record ApplyResult(bool Success, string? Reason, CorrelationId Correlation);
/// <summary>
@@ -252,6 +271,15 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// idempotent across repeated <see cref="SetDesiredSubscriptions"/> pushes.</summary>
private IAlarmSubscriptionHandle? _alarmSubscriptionHandle;
/// <summary>The alarm references the live <see cref="_alarmSubscriptionHandle"/> was actually established
/// with, so a CHANGED desired set can be told apart from a repeat of the same one.
/// <para>Without this the subscription latched: both <see cref="SubscribeDesiredAlarms"/> and
/// <see cref="HandleSubscribeAlarmsAsync"/> gate on "a handle already exists", which is the right guard
/// for an idempotent re-push and the WRONG one for a deploy that widens or narrows the alarm set —
/// drivers that filter on <c>sourceNodeIds</c> (OPC UA Client, AB CIP, FOCAS) would then never deliver a
/// newly-added alarm for the life of the connection. Empty whenever no handle is held.</para></summary>
private IReadOnlyList<string> _subscribedAlarmRefs = Array.Empty<string>();
/// <summary>
/// Gets or sets the timer scheduler for scheduling reconnection attempts.
/// </summary>
@@ -375,6 +403,20 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
Timers.StartPeriodicTimer("health-poll", HealthPollTick.Instance, _healthPollInterval);
}
/// <inheritdoc />
/// <remarks>
/// Ask the host to re-provision this incarnation — see <see cref="ReprovisionRequested"/> for why a
/// supervision restart otherwise leaves a dead child. <c>base.PostRestart</c> runs first because it
/// runs <see cref="PreStart"/>, which re-attaches the rediscovery / host-status sources and restarts
/// the health timer; the reprovision is the part <see cref="PreStart"/> cannot do, since only the
/// host knows this driver's config and desired set.
/// </remarks>
protected override void PostRestart(Exception reason)
{
base.PostRestart(reason);
Context.Parent.Tell(new ReprovisionRequested(_driverInstanceId, reason.GetType().Name));
}
private void Stubbed()
{
// Stubbed drivers accept the standard message contracts but return deterministic
@@ -488,10 +530,14 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
// Native-alarm analogue: un-gate the IAlarmSource feed when alarm tags are (now) present. The
// common live path — a deploy delivers SetDesiredSubscriptions while the driver is already
// Connected — flows through HERE, so the alarm subscribe must happen on this message, not only
// on Connected entry. Unlike the value path above, there is deliberately no unsubscribe-on-empty:
// a removed alarm tag's condition node is torn down on the address-space rebuild and
// DriverHostActor.ForwardNativeAlarm drops any transition whose ConditionId no longer maps, so a
// lingering session-less alarm subscription can never surface a removed alarm.
// on Connected entry. A CHANGED set re-subscribes (release-then-subscribe inside
// HandleSubscribeAlarmsAsync): unlike the value path, "a subscription already exists" is NOT
// sufficient, because several drivers filter their feed on the subscribed sourceNodeIds, so a
// widened set that never re-subscribed would silently never deliver the new alarm.
// Still deliberately no unsubscribe-on-EMPTY: a removed alarm tag's condition node is torn down
// on the address-space rebuild and DriverHostActor.ForwardNativeAlarm drops any transition whose
// ConditionId no longer maps, so a lingering session-less alarm subscription can never surface a
// removed alarm.
SubscribeDesiredAlarms();
});
ReceiveAsync<SubscribeAlarms>(HandleSubscribeAlarmsAsync);
@@ -561,6 +607,19 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// state on sustained failure: the nine poll drivers via Degraded, MQTT via its live connection
/// state, Galaxy via its reconnect supervisor.
/// </para>
/// <para>
/// <b>The grace is per CONNECTION (03/S17).</b> The silence baseline is
/// <c>max(LastSuccessfulRead, _connectedSinceUtc)</c>, not <c>LastSuccessfulRead ?? _connectedSinceUtc</c>.
/// Because <c>LastSuccessfulRead</c> is <em>preserved</em> across Degraded/Faulted writes (see above),
/// a device that accepts a connection but serves no data reconnects carrying a stamp that is already
/// older than the threshold, so a <c>??</c> baseline re-trips on the FIRST health tick of every new
/// connection — a reconnect cycle at health-tick cadence rather than the designed threshold. Taking
/// the later of the two gives each connection its own full window, which is exactly what re-stamping
/// <c>_connectedSinceUtc</c> on every Connected entry was for. Note this <b>paces</b> the cycle, it
/// does not end it: a permanently half-broken device still reconnects once per threshold, and a
/// driver that stops reading <i>within</i> a stable connection still trips on schedule (its read
/// stamp is the later term, exactly as before).
/// </para>
/// </remarks>
/// <param name="health">The health observed on this tick, or null when <see cref="IDriver.GetHealth"/>
/// threw — a failed observation is not evidence of comms loss, so it never advances the watchdog.</param>
@@ -570,8 +629,15 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
if (health is not { } observed) return;
if (observed.State == DriverState.Healthy) return;
// A driver that has never reported a successful read is measured from the current Connected entry.
var lastEvidence = observed.LastSuccessfulRead ?? _connectedSinceUtc;
// Measure silence from the LATER of the last successful read and THIS connect — the grace is per
// connection. `??` was wrong: drivers PRESERVE LastSuccessfulRead across Degraded/Faulted writes (that
// preservation is the #527 design), so after a trip + reconnect to a device that accepts the connection
// but still serves no data, the stamp is already older than the threshold and the FIRST tick of the new
// connection re-trips — churn at health-tick cadence instead of the designed grace. `??` only ever helped
// a driver that had never read at all.
var lastEvidence = observed.LastSuccessfulRead is { } read && read > _connectedSinceUtc
? read
: _connectedSinceUtc;
var silence = DateTime.UtcNow - lastEvidence;
if (silence < _commsLossTimeout) return;
@@ -1114,6 +1180,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
// (GalaxyDriver.SubscribeAlarmsAsync clears the set before adding), so handles no longer accumulate
// across reconnects. The gate (Count > 0) and the one-shot fan-out are unchanged.
_alarmSubscriptionHandle = null;
_subscribedAlarmRefs = Array.Empty<string>();
}
/// <summary>Establish the native-alarm subscription that un-gates an <see cref="IAlarmSource"/> driver's
@@ -1123,10 +1190,17 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// desired, and no subscription is yet established.</summary>
private void SubscribeDesiredAlarms()
{
if (_driver is IAlarmSource && _desiredAlarmRefs.Count > 0 && _alarmSubscriptionHandle is null)
if (_driver is IAlarmSource && _desiredAlarmRefs.Count > 0 && AlarmSubscriptionIsStale())
Self.Tell(new SubscribeAlarms());
}
/// <summary>True when the desired alarm refs are not the ones the live handle was established with —
/// either because there is no handle yet, or because a deploy changed the set. The second case is the
/// one a bare "handle is null" check misses; see <see cref="_subscribedAlarmRefs"/>.</summary>
private bool AlarmSubscriptionIsStale() =>
_alarmSubscriptionHandle is null ||
!_subscribedAlarmRefs.SequenceEqual(_desiredAlarmRefs, StringComparer.Ordinal);
/// <summary>Calls the driver's <see cref="IAlarmSource.SubscribeAlarmsAsync"/> (bounded) to register the
/// alarm subscription that un-gates its native feed, caching the returned handle. Re-checks the guard
/// (the desired set may have cleared, or another SubscribeAlarms may have already established a handle,
@@ -1134,10 +1208,36 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
/// stays gated until then.</summary>
private async Task HandleSubscribeAlarmsAsync(SubscribeAlarms _)
{
if (_driver is not IAlarmSource src || _desiredAlarmRefs.Count == 0 || _alarmSubscriptionHandle is not null)
if (_driver is not IAlarmSource src || _desiredAlarmRefs.Count == 0 || !AlarmSubscriptionIsStale())
return;
var refs = _desiredAlarmRefs;
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(10));
// A handle held against a DIFFERENT set must be released before re-subscribing, or a
// sourceNodeIds-filtering driver accumulates two overlapping subscriptions. Best-effort and
// deliberately non-fatal: the re-subscribe below is what actually matters, and a driver whose
// unsubscribe throws must not be left on the stale set because of it. Cleared FIRST so a throw
// cannot leave a handle that the staleness check would keep treating as live.
if (_alarmSubscriptionHandle is { } stale)
{
_alarmSubscriptionHandle = null;
_subscribedAlarmRefs = Array.Empty<string>();
try
{
await _invoker.ExecuteAsync(
DriverCapability.AlarmSubscribe,
_driverInstanceId,
async ct => await src.UnsubscribeAlarmsAsync(stale, ct),
cts.Token);
}
catch (Exception ex)
{
_log.Warning(ex,
"DriverInstance {Id}: releasing the previous native-alarm subscription ({Diag}) threw; re-subscribing anyway",
_driverInstanceId, stale.DiagnosticId);
}
}
try
{
_alarmSubscriptionHandle = await _invoker.ExecuteAsync(
@@ -1145,6 +1245,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
_driverInstanceId,
async ct => await src.SubscribeAlarmsAsync(refs, ct),
cts.Token);
_subscribedAlarmRefs = refs;
_log.Info("DriverInstance {Id}: native-alarm subscription established for {Count} alarm ref(s) ({Diag})",
_driverInstanceId, refs.Count, _alarmSubscriptionHandle.DiagnosticId);
}
@@ -0,0 +1,630 @@
using Akka.Actor;
using Microsoft.EntityFrameworkCore;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers;
/// <summary>
/// The repo's first driver-child <b>restart</b> suite (archreview 03/S16, 03/U6): what a child obtained
/// by each of the three incarnation routes is actually handed.
/// </summary>
/// <remarks>
/// <para>
/// <b>The falsification bar, and why this file exists at all.</b> An actor-restart test must assert a
/// FUNCTION the new incarnation performs — it subscribed, it re-initialised — never that the
/// incarnation EXISTS. Existence is exactly what
/// <c>DriverHostActorReconcileTests.RestartDriver_respawns_the_child_without_an_actor_name_collision</c>
/// asserts (no name collision, factory created a second driver, one child remains), and that test
/// passes green against the 03/S16 defect: a respawned child that connects and then reads nothing
/// forever. It is kept, because its own subject — the <c>InvalidActorNameException</c> — is real; it
/// is simply the weaker half of a pair, and this file is the other half.
/// </para>
/// <para>
/// <b>The three incarnation routes, and what re-provisions each.</b> (1) <i>Deploy reconcile</i> —
/// <c>ApplyDriverSpecs</c> → <c>SpawnChild</c>, then the apply's bulk desired-subscription push.
/// (2) <i>Operator Restart</i> — <c>HandleRestartDriver</c> stops the child and calls
/// <c>SpawnChild</c> with the cached spec; before 03/S16 that was the whole story, and the new child
/// started with an EMPTY desired set that nothing re-sent until the next deployment (the #488
/// periodic reconcile early-returns on an empty set, so it is structurally inert here).
/// (3) <i>Supervision restart</i> — Akka's default decider recreates the actor object around the
/// SAME driver instance without going through <c>SpawnChild</c> at all, so nothing re-sent
/// <c>InitializeRequested</c> either and the child sat in <c>Connecting</c> forever around a driver
/// its own <c>PostStop</c> had already shut down. A fourth route, <i>Force Reconnect</i>, re-uses the
/// live actor instance and needs nothing — a verified negative, pinned here so a future edit to the
/// restart path cannot break the path that was already correct.
/// </para>
/// <para>
/// <b>The supervision crash seam, and why the obvious ones do not work.</b> The seam used is the
/// driver's own <c>OnDataChange</c> carrying a NULL <c>Snapshot</c>
/// (<see cref="RecordingSubscribableDriver.FireMalformedDataChange"/>): <c>DataChangeEventArgs</c> is
/// a positional record with no null validation, and <c>DriverInstanceActor.OnDataChangeForward</c>
/// dereferences <c>msg.Snapshot</c> unguarded on the ACTOR thread, so it NREs inside the child and
/// the default decider Restarts it. Rejected alternatives, each for a specific reason: a throwing
/// <c>IDriverHealthPublisher</c> (<c>PublishHealthSnapshot</c> wraps the publish in a
/// <c>catch (Exception)</c>, so it never escapes); a throwing <c>SubscribeAsync</c> (also swallowed —
/// the stub already models it via <c>SubscribeFailuresBeforeSuccess</c>); <c>Kill.Instance</c> (the
/// default decider <b>Stops</b> on <c>ActorKilledException</c>, a different directive and therefore a
/// different code path); and an <c>internal</c> "throw this" test message (a production surface added
/// purely for a test). The fault is arbitrary by design — what is under test is the RECOVERY. If
/// <c>DataChangeEventArgs</c> ever gains null validation, the fallback is an explicit
/// <c>OneForOneStrategy(Directive.Restart)</c> on the host plus an internal test-only throw message.
/// </para>
/// </remarks>
public sealed class DriverHostActorRestartTests : RuntimeActorTestBase
{
private static readonly RevisionHash RevA = RevisionHash.Parse(new string('a', 64));
private static readonly RevisionHash RevB = RevisionHash.Parse(new string('b', 64));
private static readonly NodeId TestNode = NodeId.Parse("node-1:4053");
private const string DriverId = "DI-1";
private const string SpeedPath = "Plant/Modbus/dev1/speed";
private const string TempPath = "Plant/Modbus/dev1/temp";
private const string AlarmXPath = "Plant/Modbus/dev1/alarmX";
private const string AlarmYPath = "Plant/Modbus/dev1/alarmY";
/// <summary>TagConfig marking a raw tag as alarm-bearing (parsed by <c>TagConfigIntent</c>).</summary>
private const string AlarmConfig = """{"alarm":{"alarmType":"AlarmCondition","severity":500}}""";
/// <summary>Seeds + applies a v3 deployment whose single Modbus driver is ENABLED, so a real child is
/// spawned around a <see cref="RecordingSubscribableDriver"/>, and waits until that child has connected
/// and established its first subscription.</summary>
private (IActorRef Host, RecordingSubscribableDriverFactory Factory) DeployOneDriver(
IDbContextFactory<OtOpcUaConfigDbContext> db,
(string Driver, string DriverName, string Device, string Tag, string DataType, bool Writable)[] rawTags,
IReadOnlyDictionary<string, string>? tagConfigByTag = null)
{
var factory = new RecordingSubscribableDriverFactory("Modbus");
var deploymentId = V3ArtifactSeeder.SeedV3Deployment(
db, RevA, rawTags, Array.Empty<(string, string, string, string, string?)>(),
enabled: true, tagConfigByTag: tagConfigByTag);
var (host, _, _) = V3ArtifactSeeder.SpawnHostAndApply(this, db, TestNode, deploymentId, RevA, factory);
return (host, factory);
}
/// <summary>
/// <b>The 03/S16 falsification.</b> An operator's Restart is supposed to recover a sick driver; before
/// the fix it silently killed it — the respawned child came up with <c>_desiredRefs</c> empty, and the
/// only <c>SetDesiredSubscriptions</c> sender in the repo was the apply's bulk push, so the driver
/// subscribed to nothing until the next deployment dispatch.
/// <para>
/// Deliberately NOT the pass criterion: that a second driver was created, or that one child
/// remains. Both are true against the defect.
/// </para>
/// </summary>
[Fact]
public void Operator_restart_re_subscribes_the_respawned_child()
{
var db = NewInMemoryDbFactory();
var (host, factory) = DeployOneDriver(db, new[] { (DriverId, "Modbus", "dev1", "speed", "Float", false) });
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(1);
factory.Created[0].SubscribeCount.ShouldBe(1);
factory.Created[0].LastSubscribedRefs.ShouldBe(new[] { SpeedPath });
},
duration: PresenceBudget);
host.Tell(new RestartDriver("c1", DriverId, "tester", Guid.NewGuid()));
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(2, "the restart did not respawn the child around a new driver");
factory.Created[1].SubscribeCount.ShouldBeGreaterThanOrEqualTo(
1,
"the RESPAWNED child never subscribed — it is Connected and reading nothing, which is the " +
"03/S16 defect (an operator pressed Restart to recover a driver and silently killed it)");
factory.Created[1].LastSubscribedRefs.ShouldBe(new[] { SpeedPath });
},
duration: PresenceBudget);
}
/// <summary>
/// The value refs and the alarm refs ride in ONE <c>SetDesiredSubscriptions</c> but are applied by two
/// different child code paths (the value set self-Tells <c>Subscribe</c>; the alarm set runs
/// <c>SubscribeDesiredAlarms</c> against the <c>IAlarmSource</c> feed, which stays gated shut until it
/// is called). Sending the message is therefore not evidence that both halves landed, so the alarm
/// half is pinned separately.
/// </summary>
[Fact]
public void Operator_restart_re_subscribes_alarm_refs_too()
{
var db = NewInMemoryDbFactory();
var (host, factory) = DeployOneDriver(
db,
new[]
{
(DriverId, "Modbus", "dev1", "speed", "Float", false),
(DriverId, "Modbus", "dev1", "temp", "Float", false),
},
// An alarm-bearing raw tag is EXCLUDED from the value set and carried in the alarm set instead.
new Dictionary<string, string>
{
["temp"] = """{"alarm":{"alarmType":"AlarmCondition","severity":500}}""",
});
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(1);
factory.Created[0].SubscribeAlarmsCount.ShouldBeGreaterThanOrEqualTo(1);
factory.Created[0].LastSubscribedAlarmRefs.ShouldBe(new[] { TempPath });
},
duration: PresenceBudget);
host.Tell(new RestartDriver("c1", DriverId, "tester", Guid.NewGuid()));
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(2);
factory.Created[1].SubscribeAlarmsCount.ShouldBeGreaterThanOrEqualTo(
1,
"the respawned child never re-subscribed its alarm refs — the IAlarmSource feed stays " +
"gated shut, so no native alarm transition can ever reach the host again");
factory.Created[1].LastSubscribedAlarmRefs.ShouldBe(new[] { TempPath });
// The value half must survive the alarm half being present.
factory.Created[1].LastSubscribedRefs.ShouldBe(new[] { SpeedPath });
},
duration: PresenceBudget);
}
/// <summary>
/// The ordering guard for the spawn-time send. <c>ApplyDriverSpecs</c> (which spawns) runs BEFORE
/// <c>ApplyDesiredSubscriptions</c> (which rebuilds the authority), so a child spawned by a fresh
/// deploy is momentarily handed the PREVIOUS apply's set. That transient must never be acted on: it
/// is superseded by the bulk push on the same sender→receiver pair (Akka guarantees per-pair
/// ordering) while the child is still <c>Connecting</c>, where <c>SetDesiredSubscriptions</c> only
/// STORES; the subscribe happens on the Connected entry, from whatever was stored last.
/// <para>
/// Revision B swaps the driver's tag, so A's and B's sets are distinguishable. Note the raw tags
/// are folded into the merged <c>DriverConfig</c>, so a tag-set change DOES change the driver
/// spec and therefore respawns the child (#516) — which is precisely the arrangement under test:
/// a brand-new child, spawned while the authority still holds A, must end up subscribed to B.
/// </para>
/// <para>
/// <b>What is asserted is CONVERGENCE, not inertness</b>, and the difference was measured rather
/// than assumed. The child runs concurrently with the rest of the apply handler, so a driver
/// whose <c>InitializeAsync</c> completes instantly (this stub does) can reach Connected and
/// subscribe the transient stale set BEFORE the bulk push arrives — observed here as
/// <c>SubscribeCount == 2</c>. Akka's per-pair ordering still guarantees the final state is B's,
/// and <c>HandleSubscribeAsync</c> drops the prior subscription before establishing the new one,
/// so the cost is one extra round-trip on a respawning deploy. Pinning the count would pin a
/// race; the final refs are the invariant.
/// </para>
/// </summary>
[Fact]
public void A_child_spawned_by_a_fresh_deploy_ends_with_the_NEW_refs()
{
var db = NewInMemoryDbFactory();
var (host, factory) = DeployOneDriver(db, new[] { (DriverId, "Modbus", "dev1", "speed", "Float", false) });
AwaitAssert(
() => factory.Created[0].LastSubscribedRefs.ShouldBe(new[] { SpeedPath }),
duration: PresenceBudget);
var revB = V3ArtifactSeeder.SeedV3Deployment(
db, RevB,
new[] { (DriverId, "Modbus", "dev1", "temp", "Float", false) },
Array.Empty<(string, string, string, string, string?)>(),
enabled: true);
host.Tell(new Commons.Messages.Deploy.DispatchDeployment(revB, RevB, CorrelationId.NewId()));
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(2, "the tag-set change re-spawns the child (#516 config respawn)");
factory.Created[1].LastSubscribedRefs.ShouldBe(
new[] { TempPath },
"the newly-spawned child is on the PREVIOUS revision's refs — the spawn-time set was not superseded");
},
duration: PresenceBudget);
// A Restart on top of that must hand the new child B's set, not A's — i.e. the authority really is
// repopulated WHOLE on every apply rather than accumulated.
host.Tell(new RestartDriver("c1", DriverId, "tester", Guid.NewGuid()));
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(3);
factory.Created[2].LastSubscribedRefs.ShouldBe(new[] { TempPath });
},
duration: PresenceBudget);
}
/// <summary>
/// The empty-set semantics are load-bearing and must not degrade into "skip the send": an EMPTY value
/// set is what tells the child to <c>Unsubscribe</c> and drop a stale handle, so it must be SENT.
/// <para>
/// The subject is a driver whose only raw tag is alarm-bearing: alarm tags are excluded from the
/// value set and carried in the alarm one, so its value half is genuinely empty while its alarm
/// half is not — and both halves ride in ONE <c>SetDesiredSubscriptions</c>. That is what makes
/// the absence assertion non-vacuous: <c>SubscribeAlarmsCount >= 1</c> proves the message
/// ARRIVED, and <c>SubscribeCount == 0</c> then proves the empty value half was honoured rather
/// than the whole send being skipped. Asserting only "it did not subscribe" would pass equally
/// against a build that sent the respawned child nothing at all — the 03/S16 defect itself.
/// </para>
/// </summary>
[Fact]
public void Restart_of_a_driver_with_an_empty_value_set_still_sends_the_set()
{
var db = NewInMemoryDbFactory();
var (host, factory) = DeployOneDriver(
db,
new[] { (DriverId, "Modbus", "dev1", "temp", "Float", false) },
new Dictionary<string, string>
{
["temp"] = """{"alarm":{"alarmType":"AlarmCondition","severity":500}}""",
});
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(1);
factory.Created[0].SubscribeAlarmsCount.ShouldBeGreaterThanOrEqualTo(1);
},
duration: PresenceBudget);
host.Tell(new RestartDriver("c1", DriverId, "tester", Guid.NewGuid()));
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(2);
var respawned = factory.Created[1];
respawned.InitializeCount.ShouldBe(1, "the respawned child was never initialised");
// Positive control: the SetDesiredSubscriptions genuinely reached the respawned child.
respawned.SubscribeAlarmsCount.ShouldBeGreaterThanOrEqualTo(
1, "the respawned child received no desired set at all — the absence below would be vacuous");
respawned.LastSubscribedAlarmRefs.ShouldBe(new[] { TempPath });
respawned.SubscribeCount.ShouldBe(
0, "an EMPTY value set must route to Unsubscribe / no-op, never to a spurious subscribe");
},
duration: PresenceBudget);
}
/// <summary>
/// <b>Review finding F1 — the alarm subscription must FOLLOW a changed desired set, not latch.</b>
/// The value side converges by construction (<c>HandleSubscribeAsync</c> drops the prior subscription
/// before establishing the new one, and <c>StoreDesiredSubscriptions</c> is a full replace), but the
/// alarm side is guarded by <c>_alarmSubscriptionHandle is not null</c> in BOTH
/// <c>SubscribeDesiredAlarms</c> and <c>HandleSubscribeAlarmsAsync</c> — so once a handle exists it
/// was never re-established for the life of the connection.
/// <para>
/// That became reachable when every spawn started carrying a desired set: a tag-set edit forces a
/// respawn (raw tags fold into the merged <c>DriverConfig</c>, #516), and a fast-initialising
/// driver connects and subscribes the transient spawn-time set — the PREVIOUS revision's alarm
/// refs — before the bulk push lands. The bulk push then early-returns, latching the old set for
/// the connection's life. Drivers that FILTER on <c>sourceNodeIds</c> (OPC UA Client, AB CIP,
/// FOCAS) would silently never deliver the newly-added alarm.
/// </para>
/// </summary>
[Fact]
public void A_deploy_that_widens_the_alarm_set_re_subscribes_the_new_refs()
{
var db = NewInMemoryDbFactory();
var (host, factory) = DeployOneDriver(
db,
new[] { (DriverId, "Modbus", "dev1", "alarmX", "Float", false) },
new Dictionary<string, string> { ["alarmX"] = AlarmConfig });
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(1);
factory.Created[0].LastSubscribedAlarmRefs.ShouldBe(new[] { AlarmXPath }, ignoreOrder: true);
},
duration: PresenceBudget);
// Revision B WIDENS the alarm set: X stays, Y is added.
var revB = V3ArtifactSeeder.SeedV3Deployment(
db, RevB,
new[]
{
(DriverId, "Modbus", "dev1", "alarmX", "Float", false),
(DriverId, "Modbus", "dev1", "alarmY", "Float", false),
},
Array.Empty<(string, string, string, string, string?)>(),
enabled: true,
tagConfigByTag: new Dictionary<string, string>
{
["alarmX"] = AlarmConfig,
["alarmY"] = AlarmConfig,
});
host.Tell(new Commons.Messages.Deploy.DispatchDeployment(revB, RevB, CorrelationId.NewId()));
AwaitAssert(
() =>
{
var current = factory.Created[^1];
current.LastSubscribedAlarmRefs.ShouldBe(
new[] { AlarmXPath, AlarmYPath },
ignoreOrder: true,
"the driver is still subscribed to the PREVIOUS revision's alarm refs — the alarm " +
"subscription latched on the transient spawn-time set and never followed the new one");
},
duration: PresenceBudget);
}
/// <summary>
/// <b>The repo's first genuine Akka SUPERVISION-restart test (03/U6).</b> Not a stop, not a stub: the
/// driver raises a malformed <c>OnDataChange</c>, the actor NREs on its own thread, and Akka's default
/// decider Restarts the child in place.
/// <para>
/// A restart is strictly worse than the operator-Restart case 03/S16 names. It recreates the actor
/// object around the SAME driver instance without going through <c>SpawnChild</c>, so nothing
/// re-sends <c>InitializeRequested</c> — while <c>PreRestart</c> has already run <c>PostStop</c>,
/// which called <c>ShutdownAsync</c> on that very driver. The child therefore sat in
/// <c>Connecting</c> forever around a shut-down driver, with its desired set gone too, and
/// silently: Akka logs the fault, nothing logs the non-recovery.
/// </para>
/// <para>
/// The assertions are FUNCTIONS of the new incarnation — it re-initialised, it re-subscribed the
/// deployed refs — never that it exists. <c>Created.Count == 1</c> is what distinguishes this from
/// the operator path: a restart must NOT produce a second driver object.
/// </para>
/// </summary>
[Fact]
public void Supervision_restart_reinitialises_and_re_subscribes_the_child()
{
var db = NewInMemoryDbFactory();
var (_, factory) = DeployOneDriver(db, new[] { (DriverId, "Modbus", "dev1", "speed", "Float", false) });
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(1);
factory.Created[0].SubscribeCount.ShouldBe(1);
},
duration: PresenceBudget);
var driver = factory.Created[0];
driver.InitializeCount.ShouldBe(1);
// The crash. Subscribing is what attaches the actor's OnDataChange handler, so this only reaches the
// actor thread because the arrangement above got as far as a live subscription.
driver.FireMalformedDataChange(SpeedPath);
// FIRST: prove the crash seam actually tripped supervision, before asserting anything about the
// recovery. PreRestart runs PostStop, which calls ShutdownAsync on the driver — so a raised
// ShutdownCount is the evidence that a restart happened at all. Without this the assertions below
// would be ambiguous: "the child was not re-provisioned" and "the child never crashed" look
// identical from the outside, and a seam that silently stopped working would read as a passing
// regression test forever.
AwaitAssert(
() => driver.ShutdownCount.ShouldBeGreaterThanOrEqualTo(
1, "the crash seam did not trip supervision — nothing below is testing a restart"),
duration: PresenceBudget);
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(
1, "a supervision restart must re-use the same driver object, not respawn the child");
driver.InitializeCount.ShouldBe(
2,
"the restarted child was never re-initialised — it is sitting in Connecting around a " +
"driver its own PostStop already shut down (03/S6, the worse half of 03/S16)");
driver.SubscribeCount.ShouldBeGreaterThanOrEqualTo(
2, "the restarted child never re-subscribed — its desired set died with the old instance");
driver.LastSubscribedRefs.ShouldBe(new[] { SpeedPath });
},
duration: PresenceBudget);
}
/// <summary>
/// <b>Review finding F2 — a reprovision must be routed by INCARNATION, not just by driver id.</b>
/// <c>ReprovisionRequested</c> is delivered asynchronously, so one can still be in flight from an old
/// incarnation when <c>HandleRestartDriver</c> has already stopped that child and swapped in a new
/// one under the same <c>DriverInstanceId</c>. Routing on the id alone delivers it to the NEW child,
/// which is in <c>Connecting</c> around a healthy driver: it triggers a second, concurrent
/// <c>InitializeAsync</c> that races the very dispose-before-overwrite the #527 teardown exists to
/// serialise (<c>ModbusDriver._transport</c>, <c>MqttDriver._connection</c>).
/// <para>
/// A stale sender is simulated directly — a probe standing in for the dead incarnation — because
/// provoking the real race would be timing-dependent. The assertion is an ABSENCE, so it carries
/// a positive control: the current child is proven alive and provisioned first, and its
/// <c>InitializeCount</c> must then not move.
/// </para>
/// </summary>
[Fact]
public void A_reprovision_from_a_stale_incarnation_is_ignored()
{
var db = NewInMemoryDbFactory();
// The child must be in CONNECTING for this defect to bite: InitializeRequested is unhandled in the
// Connected behaviour, so an already-connected child would swallow the stale reprovision for the
// wrong reason and the test would pass vacuously. The gated init holds it in Connecting.
var factory = new RecordingSubscribableDriverFactory("Modbus", gateInitialize: true);
var deploymentId = V3ArtifactSeeder.SeedV3Deployment(
db, RevA,
new[] { (DriverId, "Modbus", "dev1", "speed", "Float", false) },
Array.Empty<(string, string, string, string, string?)>(),
enabled: true);
var (host, _, _) = V3ArtifactSeeder.SpawnHostAndApply(this, db, TestNode, deploymentId, RevA, factory);
// Positive control: the current child is alive and has begun its one and only initialise.
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(1);
factory.Created[0].InitializeCount.ShouldBe(1);
},
duration: PresenceBudget);
var driver = factory.Created[0];
// A ReprovisionRequested that did NOT come from the child the host currently holds.
var staleIncarnation = CreateTestProbe();
host.Tell(
new DriverInstanceActor.ReprovisionRequested(DriverId, "NullReferenceException"),
staleIncarnation.Ref);
// Round-trip a message the host must answer, so the reprovision above is provably processed before
// the absence is asserted — otherwise "nothing happened yet" would pass for "it was ignored".
var coordinator = CreateTestProbe();
host.Tell(new Commons.Messages.Fleet.GetDiagnostics(CorrelationId.NewId()), coordinator.Ref);
coordinator.ExpectMsg<Commons.Interfaces.NodeDiagnosticsSnapshot>(PresenceBudget);
driver.InitializeCount.ShouldBe(
1,
"a reprovision from a dead incarnation reached the CURRENT child and started a second, " +
"concurrent InitializeAsync on a driver that was already connected");
}
/// <summary>
/// <b>Review finding F3 - a crash-looping driver must not become a reconnect storm.</b> There is no
/// <c>SupervisorStrategy</c> anywhere under <c>src/Server</c>, so the default decider restarts a
/// faulting child <b>immediately and without limit</b>. That was survivable while a restart did
/// nothing; the reprovision hook now makes every one of them do
/// <c>ShutdownAsync</c> + <c>InitializeAsync</c> + <c>SubscribeAsync</c>, so a driver that faults on
/// every value would hammer its device as fast as it can crash.
/// <para>
/// The reprovision is therefore rate-limited per driver. Crucially it is <b>deferred, not
/// dropped</b> - the last request must still be honoured, or a child whose final crash fell
/// inside the window would be left un-provisioned forever, which is the very failure this whole
/// plan exists to close. Two genuine back-to-back supervision restarts: the first reprovision
/// applies immediately, the second is held and then lands.
/// </para>
/// </summary>
[Fact]
public void A_second_reprovision_inside_the_backoff_window_is_deferred_not_dropped()
{
var db = NewInMemoryDbFactory();
// Generous relative to the in-memory step time below, so "inside the window" is a decision rather
// than a race, while still far under the presence budget the deferred landing waits on.
var backoff = TimeSpan.FromSeconds(3);
var factory = new RecordingSubscribableDriverFactory("Modbus");
var deploymentId = V3ArtifactSeeder.SeedV3Deployment(
db, RevA,
new[] { (DriverId, "Modbus", "dev1", "speed", "Float", false) },
Array.Empty<(string, string, string, string, string?)>(),
enabled: true);
V3ArtifactSeeder.SpawnHostAndApply(
this, db, TestNode, deploymentId, RevA, factory, reprovisionBackoff: backoff);
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(1);
factory.Created[0].SubscribeCount.ShouldBe(1);
},
duration: PresenceBudget);
var driver = factory.Created[0];
// Crash 1: the first reprovision has no prior stamp, so it applies immediately.
driver.FireMalformedDataChange(SpeedPath);
AwaitAssert(
() =>
{
driver.InitializeCount.ShouldBe(2);
driver.SubscribeCount.ShouldBe(2, "the re-subscribe must land before the second crash is armed");
},
duration: PresenceBudget);
// Crash 2, immediately after - well inside the window.
driver.FireMalformedDataChange(SpeedPath);
AwaitAssert(
() => driver.ShutdownCount.ShouldBeGreaterThanOrEqualTo(
2, "the second crash seam did not trip supervision - nothing below is testing a backoff"),
duration: PresenceBudget);
driver.InitializeCount.ShouldBe(
2, "the second reprovision was applied immediately - the backoff is not rate-limiting anything");
// ...but it must still land once the window elapses.
AwaitAssert(
() => driver.InitializeCount.ShouldBe(
3, "the deferred reprovision was DROPPED, not deferred"),
duration: PresenceBudget);
}
/// <summary>
/// The <c>_children.Keys</c> arm of the desired-set authority's union. A driver with NO authored tags
/// is absent from both the value- and alarm-ref maps, so keying the authority off those maps alone
/// would leave it with no entry at all — and "absent" must not be allowed to differ from "empty",
/// because an empty set is what tells a child to <c>Unsubscribe</c> and drop a stale handle.
/// <para>
/// Asserted the only way it is observable: the tagless child is initialised (so it is provably
/// alive and was provisioned) and never subscribes, while its tag-owning sibling in the same
/// deployment does — the sibling is the positive control proving the apply's push pass actually
/// ran, so "it did not subscribe" cannot pass for "nothing happened".
/// </para>
/// </summary>
[Fact]
public void A_running_driver_with_no_authored_tags_is_handed_an_empty_set()
{
var db = NewInMemoryDbFactory();
var factory = new RecordingSubscribableDriverFactory("Modbus");
var deploymentId = V3ArtifactSeeder.SeedV3Deployment(
db, RevA,
new[] { (DriverId, "Modbus", "dev1", "speed", "Float", false) },
Array.Empty<(string, string, string, string, string?)>(),
enabled: true,
taglessDrivers: new[] { "DI-TAGLESS" });
V3ArtifactSeeder.SpawnHostAndApply(this, db, TestNode, deploymentId, RevA, factory);
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(2);
// Positive control: the sibling that DOES have a tag subscribed, so the push pass ran.
factory.Created.Single(d => d.DriverInstanceId == DriverId)
.SubscribeCount.ShouldBe(1);
factory.Created.Single(d => d.DriverInstanceId == "DI-TAGLESS")
.InitializeCount.ShouldBe(1);
},
duration: PresenceBudget);
factory.Created.Single(d => d.DriverInstanceId == "DI-TAGLESS")
.SubscribeCount.ShouldBe(
0, "a driver with no authored tags must be handed an EMPTY set, never a spurious subscribe");
}
/// <summary>
/// The verified negative, pinned so this plan's edits to the restart paths cannot break the one that
/// was already correct. Force Reconnect (<c>ReconnectDriver</c>) Tells <c>ForceReconnect</c> to the
/// SAME child actor: the instance survives, so <c>_desiredRefs</c>/<c>_desiredAlarmRefs</c> are intact
/// and the reconnect's Connected entry re-applies them by itself. Nothing needs re-provisioning, and
/// no new driver object is created.
/// </summary>
/// <remarks>
/// The budget here is deliberately wider than <see cref="RuntimeActorTestBase.PresenceBudget"/>:
/// <c>SpawnChild</c> gives the child the production <c>DefaultReconnectInterval</c> (10 s) and the
/// retry is a periodic timer, so the reconnect cannot land sooner than that. A budget is spent only
/// when the assertion fails, so widening it costs nothing on the happy path.
/// </remarks>
[Fact]
public void Force_reconnect_keeps_the_desired_set_and_re_subscribes()
{
var db = NewInMemoryDbFactory();
var (host, factory) = DeployOneDriver(db, new[] { (DriverId, "Modbus", "dev1", "speed", "Float", false) });
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(1);
factory.Created[0].SubscribeCount.ShouldBe(1);
},
duration: PresenceBudget);
var driver = factory.Created[0];
host.Tell(new ReconnectDriver("c1", DriverId, "tester", Guid.NewGuid()));
AwaitAssert(
() =>
{
factory.Created.Count.ShouldBe(1, "Force Reconnect must NOT respawn the child");
driver.SubscribeCount.ShouldBeGreaterThanOrEqualTo(
2, "the reconnected driver never re-subscribed its (surviving) desired set");
driver.LastSubscribedRefs.ShouldBe(new[] { SpeedPath });
},
duration: TimeSpan.FromSeconds(30));
}
}
@@ -40,6 +40,10 @@ public sealed class DriverInstanceActorCommsLossWatchdogTests : RuntimeActorTest
private static readonly TimeSpan FastPoll = TimeSpan.FromMilliseconds(50);
private static readonly TimeSpan FastTimeout = TimeSpan.FromMilliseconds(250);
/// <summary>The threshold used by the per-connection grace pair (03/S17). Deliberately an order of
/// magnitude above <see cref="FastPoll"/> so the absence window below is a decision, not a race.</summary>
private static readonly TimeSpan Grace = TimeSpan.FromMilliseconds(1500);
/// <summary>Drives the actor to Connected and returns once the connect has been announced.</summary>
private (IActorRef Actor, Akka.TestKit.TestProbe Parent) ConnectDriver(
HealthControlledStubDriver driver,
@@ -87,6 +91,14 @@ public sealed class DriverInstanceActorCommsLossWatchdogTests : RuntimeActorTest
/// <c>ModbusDriver.ReadAsync</c> leaves latched whenever the bad tag is read last — but it is still
/// reading successfully, so <c>LastSuccessfulRead</c> keeps advancing. It must be left alone.
/// A state-only watchdog would tear this driver down on every threshold, forever.
/// <para>
/// <b>Falsification partner (03/S17):</b>
/// <see cref="A_reconnected_driver_gets_a_full_grace_window_before_the_watchdog_trips_again"/>.
/// The silence baseline is <c>max(LastSuccessfulRead, _connectedSinceUtc)</c>, and each test kills
/// one term of that max — this one goes red if the <c>LastSuccessfulRead</c> term is dropped (its
/// connect ages far past the threshold while its reads keep succeeding), that one goes red if the
/// <c>_connectedSinceUtc</c> term is dropped (i.e. if the original <c>??</c> is restored).
/// </para>
/// </summary>
[Fact]
public void A_degraded_driver_that_is_still_reading_successfully_is_left_alone()
@@ -258,6 +270,77 @@ public sealed class DriverInstanceActorCommsLossWatchdogTests : RuntimeActorTest
parent.ExpectNoMsg(TimeSpan.FromMilliseconds(200));
}
/// <summary>
/// 03/S17 — <b>the grace is per connection.</b> Drivers deliberately <em>preserve</em>
/// <see cref="DriverHealth.LastSuccessfulRead"/> across Degraded/Faulted health writes (that
/// preservation is the #527 design), so for a device that accepts a TCP connection but never serves
/// data the stamp stays frozen from before the outage — already older than the threshold. Measuring
/// silence from that stamp alone (<c>??</c>) makes the <b>first</b> health tick of every new
/// connection re-trip, so the reconnect cycle runs at health-tick cadence (~30 s in production)
/// instead of the designed grace. Measuring from <c>max(LastSuccessfulRead, _connectedSinceUtc)</c>
/// gives each connection its own full window, which is exactly what the <c>_connectedSinceUtc</c>
/// re-stamp on every Connected entry already promises in its comment.
/// <para>
/// <b>Falsification partner:</b>
/// <see cref="A_degraded_driver_that_is_still_reading_successfully_is_left_alone"/> — see its
/// remarks for how the pair covers both terms of the max.
/// </para>
/// <para>
/// The absence window carries the house positive control: a proven number of health-poll ticks
/// processed <em>after</em> the reconnect, so "the watchdog did not re-trip" cannot pass because
/// the actor stopped ticking. It is also deliberately far shorter than <see cref="Grace"/>, so
/// it asserts a delay, not a disablement — <see cref="A_half_broken_device_is_still_declared_lost_once_the_grace_elapses"/>
/// is the anti-vacuity half.
/// </para>
/// </summary>
[Fact]
public void A_reconnected_driver_gets_a_full_grace_window_before_the_watchdog_trips_again()
{
var driver = new HealthControlledStubDriver
{
// The half-broken device: the socket connects, every read fails, and the last successful read is
// frozen well before the outage — preserved across the Degraded writes, as every driver does.
Health = new DriverHealth(DriverState.Degraded, DateTime.UtcNow.AddMinutes(-5), "reads time out"),
};
var (_, parent) = ConnectDriver(driver, commsLossTimeout: Grace, reconnectInterval: TimeSpan.FromMilliseconds(100));
// The first trip is correct and expected — the device really has been silent for five minutes.
parent.ExpectMsg<DriverInstanceActor.ConnectivityChanged>(PresenceBudget)
.Connected.ShouldBeFalse();
// The retry re-initialises successfully (the socket still accepts), so a NEW connection starts here.
parent.ExpectMsg<DriverInstanceActor.ConnectivityChanged>(PresenceBudget)
.Connected.ShouldBeTrue();
var ticksAtReconnect = driver.HealthReadCount;
parent.ExpectNoMsg(TimeSpan.FromMilliseconds(400));
(driver.HealthReadCount - ticksAtReconnect).ShouldBeGreaterThanOrEqualTo(
4,
"health-poll ticks did not run after the reconnect — the absence assertion above would be vacuous");
}
/// <summary>
/// The anti-vacuity half of the 03/S17 pair: the per-connection grace <b>delays</b> the watchdog, it
/// does not disable it. The same half-broken device is still declared lost once the grace elapses —
/// without this, a "fix" that simply stopped evaluating the watchdog after the first connection would
/// pass <see cref="A_reconnected_driver_gets_a_full_grace_window_before_the_watchdog_trips_again"/>
/// and look correct.
/// </summary>
[Fact]
public void A_half_broken_device_is_still_declared_lost_once_the_grace_elapses()
{
var driver = new HealthControlledStubDriver
{
Health = new DriverHealth(DriverState.Degraded, DateTime.UtcNow.AddMinutes(-5), "reads time out"),
};
var (_, parent) = ConnectDriver(driver, commsLossTimeout: Grace, reconnectInterval: TimeSpan.FromMilliseconds(100));
parent.ExpectMsg<DriverInstanceActor.ConnectivityChanged>(PresenceBudget).Connected.ShouldBeFalse();
parent.ExpectMsg<DriverInstanceActor.ConnectivityChanged>(PresenceBudget).Connected.ShouldBeTrue();
// The second connection serves no data either, so once ITS grace elapses the watchdog trips again.
parent.ExpectMsg<DriverInstanceActor.ConnectivityChanged>(PresenceBudget).Connected.ShouldBeFalse();
}
}
/// <summary>
@@ -188,3 +188,215 @@ internal sealed class StubHandle : ISubscriptionHandle
/// <summary>Gets the diagnostic ID of the subscription.</summary>
public string DiagnosticId => "stub-sub";
}
/// <summary>
/// A per-instance recording driver for the restart / re-subscribe suite (03/S16, U6): unlike
/// <see cref="SubscribableStubDriver"/> it carries the <b>spec's</b> <see cref="DriverInstanceId"/>
/// (<see cref="StubDriver"/> hard-codes a constant on a non-virtual member), and it counts and records the
/// alarm subscription alongside the value one — the two travel in a single
/// <c>SetDesiredSubscriptions</c> but through two different child code paths, so both need observing.
/// </summary>
/// <remarks>
/// <para>
/// <b>The falsification bar these counters exist for:</b> a restart test must assert a FUNCTION the new
/// incarnation performs — it subscribed, it re-initialised — never that the incarnation exists.
/// Existence is what
/// <c>DriverHostActorReconcileTests.RestartDriver_respawns_the_child_without_an_actor_name_collision</c>
/// asserts, and that test passes against the very defect 03/S16 names.
/// </para>
/// <para>
/// <b><see cref="FireMalformedDataChange"/> is the supervision crash seam</b> — see
/// <c>DriverHostActorRestartTests</c>'s file remarks for why the obvious alternatives do not work.
/// </para>
/// </remarks>
internal sealed class RecordingSubscribableDriver : IDriver, ISubscribable, IAlarmSource
{
private readonly Lock _gate = new();
private readonly StubHandle _handle = new();
private readonly RecordingAlarmHandle _alarmHandle = new();
private int _initializeCount;
private int _shutdownCount;
private int _subscribeCount;
private int _subscribeAlarmsCount;
private IReadOnlyList<string>? _lastSubscribedRefs;
private IReadOnlyList<string>? _lastSubscribedAlarmRefs;
/// <summary>Initializes a new recording driver stamped with the spec's identity.</summary>
/// <param name="driverInstanceId">The driver instance id from the spec that created it.</param>
/// <param name="driverType">The driver type from the spec that created it.</param>
public RecordingSubscribableDriver(string driverInstanceId, string driverType)
{
DriverInstanceId = driverInstanceId;
DriverType = driverType;
}
/// <inheritdoc />
public string DriverInstanceId { get; }
/// <inheritdoc />
public string DriverType { get; }
/// <summary>Number of <see cref="InitializeAsync"/> calls. A supervision restart re-uses the SAME driver
/// object (the Props closure captured it), so this going 1 to 2 is how the reprovision is observed.</summary>
public int InitializeCount => Volatile.Read(ref _initializeCount);
/// <summary>Number of <see cref="ShutdownAsync"/> calls — a restart's <c>PostStop</c> raises it.</summary>
public int ShutdownCount => Volatile.Read(ref _shutdownCount);
/// <summary>Number of <see cref="SubscribeAsync"/> calls.</summary>
public int SubscribeCount => Volatile.Read(ref _subscribeCount);
/// <summary>Number of <see cref="SubscribeAlarmsAsync"/> calls.</summary>
public int SubscribeAlarmsCount => Volatile.Read(ref _subscribeAlarmsCount);
/// <summary>The refs passed to the most recent <see cref="SubscribeAsync"/>, or null if never called.</summary>
public IReadOnlyList<string>? LastSubscribedRefs
{
get { lock (_gate) return _lastSubscribedRefs; }
}
/// <summary>The refs passed to the most recent <see cref="SubscribeAlarmsAsync"/>, or null if never called.</summary>
public IReadOnlyList<string>? LastSubscribedAlarmRefs
{
get { lock (_gate) return _lastSubscribedAlarmRefs; }
}
/// <summary>
/// When set, <see cref="InitializeAsync"/> awaits it instead of completing inline — so the actor
/// stays in <c>Connecting</c> for as long as the test wants. Needed because several defects only
/// bite a child that has NOT yet connected (<c>InitializeRequested</c> is unhandled in
/// <c>Connected</c>, so a test built on an already-connected child passes vacuously).
/// </summary>
public TaskCompletionSource? InitializeGate { get; set; }
/// <inheritdoc />
public async Task InitializeAsync(string driverConfigJson, CancellationToken cancellationToken)
{
Interlocked.Increment(ref _initializeCount);
if (InitializeGate is { } gate) await gate.Task.ConfigureAwait(false);
}
/// <inheritdoc />
public Task ReinitializeAsync(string driverConfigJson, CancellationToken cancellationToken) => Task.CompletedTask;
/// <inheritdoc />
public Task ShutdownAsync(CancellationToken cancellationToken)
{
Interlocked.Increment(ref _shutdownCount);
return Task.CompletedTask;
}
/// <inheritdoc />
public DriverHealth GetHealth() => new(DriverState.Healthy, DateTime.UtcNow, null);
/// <inheritdoc />
public event EventHandler<DataChangeEventArgs>? OnDataChange;
/// <inheritdoc />
public Task<ISubscriptionHandle> SubscribeAsync(
IReadOnlyList<string> fullReferences, TimeSpan publishingInterval, CancellationToken cancellationToken)
{
Interlocked.Increment(ref _subscribeCount);
lock (_gate) _lastSubscribedRefs = fullReferences.ToArray();
return Task.FromResult<ISubscriptionHandle>(_handle);
}
/// <inheritdoc />
public Task UnsubscribeAsync(ISubscriptionHandle handle, CancellationToken cancellationToken) => Task.CompletedTask;
/// <inheritdoc />
public event EventHandler<AlarmEventArgs>? OnAlarmEvent;
/// <inheritdoc />
public Task<IAlarmSubscriptionHandle> SubscribeAlarmsAsync(
IReadOnlyList<string> sourceNodeIds, CancellationToken cancellationToken)
{
Interlocked.Increment(ref _subscribeAlarmsCount);
lock (_gate) _lastSubscribedAlarmRefs = sourceNodeIds.ToArray();
return Task.FromResult<IAlarmSubscriptionHandle>(_alarmHandle);
}
/// <inheritdoc />
public Task UnsubscribeAlarmsAsync(IAlarmSubscriptionHandle handle, CancellationToken cancellationToken) => Task.CompletedTask;
/// <inheritdoc />
public Task AcknowledgeAsync(
IReadOnlyList<AlarmAcknowledgeRequest> acknowledgements, CancellationToken cancellationToken) => Task.CompletedTask;
/// <summary>Fires a well-formed data change (the ordinary production path).</summary>
/// <param name="fullRef">The reference whose value changed.</param>
/// <param name="value">The new value.</param>
public void FireDataChange(string fullRef, object? value) =>
OnDataChange?.Invoke(this, new DataChangeEventArgs(
_handle, fullRef, new DataValueSnapshot(value, 0u, DateTime.UtcNow, DateTime.UtcNow)));
/// <summary>
/// Raises <see cref="OnDataChange"/> with a NULL <c>Snapshot</c> — the supervision crash seam.
/// <c>DataChangeEventArgs</c> is a positional record with no null validation, and the actor's
/// <c>OnDataChangeForward</c> dereferences <c>msg.Snapshot</c> unguarded ON THE ACTOR THREAD, so this
/// produces a genuine unhandled exception inside the child, which Akka's default decider turns into a
/// Restart. The fault itself is arbitrary; what is under test is the RECOVERY.
/// </summary>
/// <param name="fullRef">The reference to name in the malformed event.</param>
public void FireMalformedDataChange(string fullRef) =>
OnDataChange?.Invoke(this, new DataChangeEventArgs(_handle, fullRef, null!));
/// <summary>Raises <see cref="OnAlarmEvent"/>; present so the event is not merely written-to-never-read.</summary>
/// <param name="conditionId">The condition id to raise.</param>
public void FireAlarm(string conditionId) =>
OnAlarmEvent?.Invoke(this, new AlarmEventArgs(
_alarmHandle, conditionId, conditionId, "AlarmCondition", "test", AlarmSeverity.High, DateTime.UtcNow));
private sealed class RecordingAlarmHandle : IAlarmSubscriptionHandle
{
/// <inheritdoc />
public string DiagnosticId => "recording-alarm-sub";
}
}
/// <summary>
/// An <see cref="IDriverFactory"/> that hands out <see cref="RecordingSubscribableDriver"/>s and KEEPS
/// every one it created, in creation order. The list — not a single "the driver" reference — is the point:
/// an operator Restart tears the child down and spawns a new one around a NEW driver object, so the
/// assertion subject is <c>Created[1]</c>, while a SUPERVISION restart re-uses <c>Created[0]</c> (the Props
/// closure captured it) and part of the assertion is that <c>Created.Count</c> stayed 1.
/// </summary>
internal sealed class RecordingSubscribableDriverFactory : IDriverFactory
{
private readonly string _supportedType;
private readonly Lock _gate = new();
private readonly List<RecordingSubscribableDriver> _created = new();
private readonly bool _gateInitialize;
/// <summary>Initializes a factory that materialises exactly one driver type.</summary>
/// <param name="supportedType">The driver type this factory answers for.</param>
/// <param name="gateInitialize">When true, every created driver gets an un-completed
/// <see cref="RecordingSubscribableDriver.InitializeGate"/>, so its child actor stays in
/// <c>Connecting</c> until the test completes it.</param>
public RecordingSubscribableDriverFactory(string supportedType, bool gateInitialize = false)
{
_supportedType = supportedType;
_gateInitialize = gateInitialize;
}
/// <summary>Every driver this factory created, in creation order.</summary>
public IReadOnlyList<RecordingSubscribableDriver> Created
{
get { lock (_gate) return _created.ToArray(); }
}
/// <inheritdoc />
public IDriver? TryCreate(string driverType, string driverInstanceId, string driverConfigJson)
{
if (!string.Equals(driverType, _supportedType, StringComparison.Ordinal)) return null;
var driver = new RecordingSubscribableDriver(driverInstanceId, driverType);
if (_gateInitialize)
driver.InitializeGate = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
lock (_gate) _created.Add(driver);
return driver;
}
/// <inheritdoc />
public IReadOnlyCollection<string> SupportedTypes => new[] { _supportedType };
}
@@ -9,6 +9,7 @@ using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
@@ -36,16 +37,35 @@ public static class V3ArtifactSeeder
/// <param name="rev">The revision hash the deployment is sealed under.</param>
/// <param name="rawTags">The raw tags to materialise, one per (driver, device, tag).</param>
/// <param name="unsRefs">UNS references projecting a raw tag (by <c>BackingTag</c>) into an equipment.</param>
/// <param name="enabled">Whether the seeded <c>DriverInstances</c> are Enabled — i.e. whether applying this
/// artifact actually SPAWNS driver children. Defaults to <c>false</c>, the shape every pre-existing caller
/// relies on: the value-routing suites inject values straight into the host and want no children at all.
/// The restart/re-subscribe suite (03/S16) passes <c>true</c>, because its whole subject is what a spawned
/// child is handed.</param>
/// <param name="tagConfigByTag">Optional per-tag <c>TagConfig</c> JSON, keyed by tag name; tags not named
/// here keep the default <c>"{}"</c>. The only way to seed an <b>alarm-bearing</b> raw tag (the alarm intent
/// is parsed out of <c>TagConfig.alarm</c> by <c>TagConfigIntent</c>), which the alarm-ref half of the
/// restart suite needs — an alarm-bearing raw tag is excluded from the VALUE subscription set and carried
/// in the alarm one instead, so the two travel through different child code paths.</param>
/// <param name="taglessDrivers">Extra <c>DriverInstance</c> ids carrying NO devices and NO tags. The
/// only way to express a driver that is a running child yet absent from both the value- and alarm-ref
/// maps — the <c>_children.Keys</c> arm of <c>ApplyDesiredSubscriptions</c>'s union, which must still
/// hand that child an explicit EMPTY set rather than skipping it.</param>
/// <returns>The seeded deployment's id.</returns>
public static DeploymentId SeedV3Deployment(
IDbContextFactory<OtOpcUaConfigDbContext> db, RevisionHash rev,
(string Driver, string DriverName, string Device, string Tag, string DataType, bool Writable)[] rawTags,
(string Area, string Line, string Equip, string BackingTag, string? Effective)[] unsRefs)
(string Area, string Line, string Equip, string BackingTag, string? Effective)[] unsRefs,
bool enabled = false,
IReadOnlyDictionary<string, string>? tagConfigByTag = null,
string[]? taglessDrivers = null)
{
var drivers = rawTags
.Select(t => (t.Driver, t.DriverName))
.Distinct()
.Select(d => new { DriverInstanceId = d.Driver, RawFolderId = "rf-plant", Name = d.DriverName, DriverType = "Modbus", DriverConfig = "{}", ClusterId = "c1", Enabled = false })
.Select(d => new { DriverInstanceId = d.Driver, RawFolderId = "rf-plant", Name = d.DriverName, DriverType = "Modbus", DriverConfig = "{}", ClusterId = "c1", Enabled = enabled })
.Concat((taglessDrivers ?? Array.Empty<string>())
.Select(id => new { DriverInstanceId = id, RawFolderId = "rf-plant", Name = id, DriverType = "Modbus", DriverConfig = "{}", ClusterId = "c1", Enabled = enabled }))
.ToArray();
var devices = rawTags
.Select(t => (t.Driver, t.Device))
@@ -60,7 +80,7 @@ public static class V3ArtifactSeeder
Name = t.Tag,
DataType = t.DataType,
AccessLevel = t.Writable ? 1 : 0, // TagAccessLevel.ReadWrite = 1
TagConfig = "{}",
TagConfig = tagConfigByTag is not null && tagConfigByTag.TryGetValue(t.Tag, out var tc) ? tc : "{}",
}).ToArray();
// UNS topology: one area/line per distinct (Area, Line); one equipment per (Area, Line, Equip).
@@ -114,13 +134,21 @@ public static class V3ArtifactSeeder
/// <param name="localNode">The host's local node id.</param>
/// <param name="deploymentId">The deployment to dispatch.</param>
/// <param name="rev">The revision hash the deployment was sealed under.</param>
/// <param name="driverFactory">Optional driver factory forwarded to <c>DriverHostActor.Props</c>. Null (the
/// default, preserving every existing call site) leaves the host on its own default, which materialises no
/// real drivers. Supply one — with <c>enabled: true</c> on the seed — to actually spawn driver children.</param>
/// <param name="reprovisionBackoff">Optional per-driver supervision-reprovision rate limit forwarded to
/// <c>DriverHostActor.Props</c>; null keeps the production default. Exists so a test can observe the
/// deferral without waiting the production window.</param>
/// <returns>The host actor plus the publish and mux probes wired into it.</returns>
public static (IActorRef Actor, TestProbe Publish, TestProbe Mux) SpawnHostAndApply(
TestKitBase kit, IDbContextFactory<OtOpcUaConfigDbContext> db,
NodeId localNode, DeploymentId deploymentId, RevisionHash rev)
NodeId localNode, DeploymentId deploymentId, RevisionHash rev,
IDriverFactory? driverFactory = null,
TimeSpan? reprovisionBackoff = null)
{
var publish = kit.CreateTestProbe();
var (actor, mux, coordinator) = Spawn(kit, db, localNode, publish.Ref);
var (actor, mux, coordinator) = Spawn(kit, db, localNode, publish.Ref, driverFactory, reprovisionBackoff);
Dispatch(actor, coordinator, deploymentId, rev);
publish.ExpectMsg<OpcUaPublishActor.RebuildAddressSpace>(RuntimeActorTestBase.PresenceBudget);
@@ -140,14 +168,14 @@ public static class V3ArtifactSeeder
TestKitBase kit, IDbContextFactory<OtOpcUaConfigDbContext> db,
NodeId localNode, DeploymentId deploymentId, RevisionHash rev)
{
var (actor, mux, coordinator) = Spawn(kit, db, localNode, opcUaPublishActor: null);
var (actor, mux, coordinator) = Spawn(kit, db, localNode, opcUaPublishActor: null, driverFactory: null, reprovisionBackoff: null);
Dispatch(actor, coordinator, deploymentId, rev);
return (actor, mux);
}
private static (IActorRef Actor, TestProbe Mux, TestProbe Coordinator) Spawn(
TestKitBase kit, IDbContextFactory<OtOpcUaConfigDbContext> db, NodeId localNode,
IActorRef? opcUaPublishActor)
IActorRef? opcUaPublishActor, IDriverFactory? driverFactory, TimeSpan? reprovisionBackoff)
{
var coordinator = kit.CreateTestProbe();
var mux = kit.CreateTestProbe();
@@ -155,11 +183,13 @@ public static class V3ArtifactSeeder
var actor = kit.Sys.ActorOf(DriverHostActor.Props(
db, localNode, coordinator.Ref,
driverFactory: driverFactory,
localRoles: new HashSet<string> { "driver" },
dependencyMux: mux.Ref,
opcUaPublishActor: opcUaPublishActor,
virtualTagEvaluator: NullVirtualTagEvaluator.Instance,
virtualTagHostOverride: vtHost.Ref));
virtualTagHostOverride: vtHost.Ref,
reprovisionBackoff: reprovisionBackoff));
return (actor, mux, coordinator);
}