lmxopcua/docs/DriverLifecycle.md

# Driver Lifecycle & Server Infrastructure Contracts

Reference for the server-side infrastructure interfaces that surround a
driver but are **not** driver *capabilities* (read/write/subscribe/etc.,
documented in [ReadWriteOperations.md](ReadWriteOperations.md) and the
per-driver pages). These contracts live in
[`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/)
so they carry no behavior — concrete implementations live in the driver
projects, the Runtime, and the ControlPlane. Each subsection below gives the
purpose, the key members, and where it is implemented/used.

The capability interfaces a driver opts into (`IReadable`, `IWritable`,
`ITagDiscovery`, `ISubscribable`, `IAlarmSource`,
`IHostConnectivityProbe`, `IPerCallHostResolver`, `IRediscoverable`) are
covered elsewhere and discovered by the server via `is`-checks on the
`IDriver` instance. The interfaces here are the *plumbing* the server uses to
**create**, **probe**, **supervise**, **report on**, and **configure** those
drivers, plus the server-side historian read surface.

---

## IDriverFactory — creating drivers from config rows

[`Core.Abstractions/IDriverFactory.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverFactory.cs)

Abstraction over the process-wide driver registry. The Runtime consumes this
instead of the concrete registry so the Runtime project does not pull in
`ZB.MOM.WW.OtOpcUa.Core` (which would drag in Polly + driver hosting).

Members:

- `IDriver? TryCreate(string driverType, string driverInstanceId, string driverConfigJson)`
  — returns a new driver for the given type, or `null` when no factory is
  registered for that type (missing assembly, typo). The `DriverHostActor`
  logs and skips the row rather than failing the whole apply.
- `IReadOnlyCollection<string> SupportedTypes` — driver-type names this
  factory can materialise; mostly for diagnostics and logs.

Implementations:

- `NullDriverFactory` (same file) returns `null` from every `TryCreate` and
  exposes zero supported types. Bound when no concrete driver assemblies have
  been registered (Mac dev path, smoke tests); the deployment becomes a no-op.
- `DriverFactoryRegistry`
  ([`Core/Hosting/DriverFactoryRegistry.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core/Hosting/DriverFactoryRegistry.cs))
  is the real process-singleton registry keyed by `DriverInstance.DriverType`
  (case-insensitive). Each driver project ships a `Register(...)` extension;
  `Register` records the factory **and** the driver's stability
  [`DriverTier`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverTier.cs)
  (defaults to Tier A). Registering the same type twice throws.
- `DriverFactoryRegistryAdapter`
  ([`Core/Hosting/DriverFactoryRegistryAdapter.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core/Hosting/DriverFactoryRegistryAdapter.cs))
  bridges the registry to the `IDriverFactory` abstraction.

Wiring: `DriverFactoryBootstrap.AddOtOpcUaDriverFactories`
([`Host/Drivers/DriverFactoryBootstrap.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs))
registers the singleton registry, runs every driver assembly's `Register`
extension, then binds `IDriverFactory` to the adapter. It must run **before**
`AddAkka` so the Runtime can resolve `IDriverFactory` when spawning the
`DriverHostActor`
([`Runtime/Drivers/DriverHostActor.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs)).
The registry is skipped on admin-only nodes (they never run drivers); the
probe set is the exception — see [IDriverProbe](#idriverprobe--test-connect).

---

## IDriverProbe — Test Connect

[`Core.Abstractions/IDriverProbe.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverProbe.cs)

A cheap test-connect probe for one driver type, backing the AdminUI **Test
Connect** button. An implementation deserializes a driver-config JSON, attempts
a cheap connection (TCP open, OPC UA session, gRPC ping — whatever the driver's
native protocol supports), and reports success/failure with latency. **Probes
must not mutate persistent state**: the AdminUI invokes them against the
transient config in the typed form, not against the persisted `DriverInstance`
row.

Members:

- `string DriverType { get; }` — the `DriverInstance.DriverType` string this
  probe handles; used for DI lookup.
- `Task<DriverProbeResult> ProbeAsync(string configJson, TimeSpan timeout, CancellationToken ct)`
  — never throws on connection failure; returns a result with `Ok = false`
  and a message instead.
- `DriverProbeResult(bool Ok, string? Message, TimeSpan? Latency)` — outcome
  record (`Message` is `null` on success; `Latency` is `null` on failure).

Implementations: every driver ships a `*DriverProbe` in its driver project
(e.g.
[`Driver.Modbus/ModbusDriverProbe.cs`](../src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriverProbe.cs)
does a bare socket open/close). The historian backend is the external
HistorianGateway (consumed as a gRPC client package, not an `IDriver`), so it
has no driver probe.

Flow: the AdminUI's `AdminProbeService`
([`AdminUI/Clients/AdminProbeService.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Clients/AdminProbeService.cs))
dispatches a `TestDriverConnect` message through `IAdminOperationsClient` to the
cluster-singleton `AdminOperationsActor`
([`ControlPlane/AdminOperations/AdminOperationsActor.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs)),
which holds the probes keyed by `DriverType` and invokes the matching one
(timeout clamped to `[1, 60]` seconds). Because the admin singleton is
admin-pinned, the probe set must be registered on admin nodes too — `Program.cs`
calls `AddOtOpcUaDriverProbes` in the `hasAdmin` block, and
`AddOtOpcUaDriverFactories` registers it for fused admin+driver nodes.

---

## IDriverSupervisor / process recycle — REMOVED (Gitea #522)

`IDriverSupervisor`, `MemoryRecycle`, `MemoryTracking` and
`ScheduledRecycleScheduler` **no longer exist.** This section used to describe
them as the Tier C protection layer.

They were removed because the thing they acted on is gone. "Tier C" meant a
driver running **out-of-process behind a supervisor that could restart its Host**
without tearing down the OPC UA session or co-hosted drivers. No such process
remains anywhere: Galaxy reaches MXAccess over gRPC to the external
`mxaccessgw` sidecar (PR 7.2 retired the in-process `Galaxy.Host` / `Proxy` /
`Shared` projects), and FOCAS has run in-process since its managed wire client
landed 2026-04-24. Consistent with that, `IDriverSupervisor` had **zero
implementations**, and `MemoryTracking` / `MemoryRecycle` /
`ScheduledRecycleScheduler` were constructed **only in their own unit tests** —
so this was not dormant-but-ready code, it was code with no path to running.

The operator-facing half went too: `RecycleIntervalSeconds` was authorable
through the AdminUI's driver resilience section and validated by the parser
while configuring nothing. A stored `ResilienceConfig` still carrying the key
parses cleanly (unknown keys are ignored) and the AdminUI preserves it rather
than silently rewriting an operator's config.

**`DriverTier` itself survives and is load-bearing** — do not delete it while
following this note. `DriverResilienceOptions.GetTierDefaults(tier)` supplies
the real per-capability timeout / retry / breaker policies, resolved via
`DriverFactoryRegistry.GetTier` and applied by `DriverCapabilityInvokerFactory`.
Every driver runs Tier A because no factory passes a tier, so today that means
one uniform policy set.

`IDriver` **no longer carries `GetMemoryFootprint()` or
`FlushOptionalCachesAsync()`** (Gitea **#525**). They existed to feed
`MemoryTracking`, deleted with the rest of the driver-tier recycle machinery in
**#522**; after that nothing called them, and their doc-comments still claimed
Core polled the footprint every 30 s, which had not been true since v3. The
sweep removed them from the interface, all 12 drivers, and every test stub,
along with the support code that existed only to feed them — MTConnect's
per-entry byte weights and its cached data-item count, OpcUaClient's
`_discoveredNodeCount`, and MQTT's per-tag byte estimate.

Two consequences worth knowing:

- `AbCipTemplateCache` now has **no production reset path**. Its only caller was
  AB CIP's flush — which itself had no caller, so nothing has actually changed
  in behaviour, only in how visible the fact is. A UDT redefined on a live
  controller keeps its stale shape until the driver instance is rebuilt.
- MTConnect's `/probe` cache is still dropped, by `InvalidateProbeModel()` on an
  Agent restart. The tests that used to reach it through the flush now drive
  that path instead, so they cover a case that actually occurs.

If per-driver memory protection is wanted again, build it for the architecture
as it is now: a process-wide watchdog that can ask drivers to flush, not a
per-driver tier whose only remedy requires a supervisor that cannot exist
in-process. The deleted per-driver implementations are in git history and were
mostly reusable — what was always missing is the trigger.

---

## What a (re)spawn re-provisions

A driver child can come into existence by **three** routes, and until 2026-08-20 only the first of them
handed the child everything it needs to be a working driver. That is archreview **03/S16** (and its worse
half, **03/S6**): a child that exists without its desired subscription set is a *dead* driver — it
connects, reports `Connected`, publishes nothing and raises no alarms, and the #488 periodic reconcile
cannot rescue it (it early-returns on an empty desired set). The fix makes the desired set **part of what
it means to have a child** rather than a later step in the apply pipeline.

**`DriverHostActor._desiredByDriver` is the single authority.** It is a per-driver
`(ValueRefs, AlarmRefs)` pair, cleared and repopulated **whole** on every apply in the same pass that
rebuilds the routing maps — keyed over the union of the value-ref map, the alarm-ref map and the running
`_children`, so a driver with no authored tags gets an explicit **empty** set rather than being absent.
Everything that hands a child a `SetDesiredSubscriptions` goes through the one helper
`SendDesiredSubscriptions(actor, driverId)`. (An empty set is meaningful, not a no-op to skip: the child
routes it to `Unsubscribe`, dropping a stale handle.)

| Route | Trigger | What re-provisions it |
|---|---|---|
| **Deploy reconcile** | `ApplyDriverSpecs` → `SpawnChild` | `SpawnChild` sends `InitializeRequested` then the desired set; the apply's bulk push then sends the (freshly rebuilt) set to every child. |
| **Operator Restart** | `/hosts` → Restart → `HandleRestartDriver` | Stops the child, respawns from the cached spec — and `SpawnChild` sends the desired set, because it always does. **This is the 03/S16 fix**: there is no bulk push on this path, and before it the respawned child subscribed to nothing until the next deployment. |
| **Supervision restart** | a child throws ⇒ Akka's **default decider** ⇒ `Restart` | `DriverInstanceActor.PostRestart` Tells the host `ReprovisionRequested`; `HandleReprovisionRequested` logs a Warning naming the crash type, re-sends `InitializeRequested` (non-stub children) and the desired set. |
| **Force Reconnect** | `/hosts` → Force Reconnect → `HandleReconnectDriver` | **Nothing — verified.** It Tells `ForceReconnect` to the *same* child actor, so the instance and its `_desiredRefs`/`_desiredAlarmRefs` survive and the reconnect's Connected entry re-applies them itself. Pinned by `Force_reconnect_keeps_the_desired_set_and_re_subscribes` so a future edit to the restart paths cannot break the one that was already correct. |

Things to know before touching any of this:

- ⚠️ **The supervision route is the invisible one, and it was the worst.** There is **no**
  `SupervisorStrategy` anywhere under `src/Server`, so a child that throws gets Akka's default `Restart`.
  A restart recreates the actor object around the **same** `IDriver` — the Props closure captured it —
  and its `PreRestart` runs `PostStop`, which calls `IDriver.ShutdownAsync`. It does **not** run
  `SpawnChild`, and the child's `IActorRef` is unchanged, so the host had no notification a new
  incarnation existed at all. Net effect before the hook: the child sat in `Connecting` forever around a
  shut-down driver, with its desired set gone, and nothing logged the non-recovery.
- **Replacing the default decider is deliberately out of scope.** A `Stop` directive would silently
  remove a driver; a bounded retry count would silently stop one after N crashes. Both change failure
  semantics for every driver in the fleet and deserve their own plan and live gate. The reprovision hook
  makes the *current* infinite-restart behaviour correct instead of dead.
- ⚠️ **…which is exactly why the reprovision is RATE-LIMITED.** The default decider restarts a faulting
  child **immediately and without limit**, and the hook now makes every one of those restarts do
  `ShutdownAsync` + `InitializeAsync` + `SubscribeAsync` — so a driver that faults on every value would
  hammer its device as fast as it can crash. `DriverHostActor` keeps a per-driver last-reprovision stamp
  and holds anything inside `DefaultReprovisionBackoff` (10 s, matched to
  `DriverInstanceActor.DefaultReconnectInterval` because it drives the same connect work). The held
  request is **deferred, not dropped** — one `StartSingleTimer` per driver, so a burst collapses to a
  single pending reprovision — because dropping it would leave a child whose *final* crash fell inside
  the window un-provisioned forever, which is the 03/S16 failure this hook exists to close. Each
  deferral logs a Warning naming the crash type, so a crash loop is visible rather than merely quiet.
- **A reprovision is routed by INCARNATION, not by driver id.** `ReprovisionRequested` is delivered
  asynchronously, so one can still be in flight from an old incarnation after `HandleRestartDriver`
  stopped that child and spawned a replacement under the same id. Delivered to the new child — which is
  in `Connecting`, where `InitializeRequested` *is* handled — it would start a second, concurrent
  `InitializeAsync`, racing the dispose-before-overwrite that #527's teardown exists to serialise
  (`ModbusDriver._transport` / `MqttDriver._connection` each leak a transport on that race). The handler
  therefore checks `Sender` against the child it currently holds and drops anything else; the deferred
  path carries the original ref so the same check still applies when the timer re-delivers it.
- **Ordering on a respawning deploy, and its one accepted cost.** `ApplyDriverSpecs` (which spawns) runs
  before the pass that rebuilds `_desiredByDriver`, so a child respawned by a deploy — raw tags are folded
  into the merged `DriverConfig`, so a tag-set edit changes the spec and trips the #516 config respawn —
  is first handed the **previous** apply's set, then the new one by the bulk push. Akka's per-pair
  ordering guarantees it **converges**; it does not guarantee the transient is inert, because the child
  runs concurrently with the rest of the apply handler and a fast-initialising driver can reach
  `Connected` and subscribe the stale set first. The cost is one extra subscribe round-trip
  (`HandleSubscribeAsync` drops the prior subscription before establishing the new one); a real device
  driver, whose initialise opens a socket, rarely wins that race. Withholding the spawn-time send to avoid
  it would simply restore 03/S16, since the Restart and supervision routes have no bulk push coming.
- ⚠️ **Convergence is not free on the ALARM side, and "a subscription already exists" is the wrong guard
  there.** The value path converges by construction — `StoreDesiredSubscriptions` is a full replace and
  `HandleSubscribeAsync` drops the prior subscription before establishing the new one — but
  `SubscribeDesiredAlarms` / `HandleSubscribeAlarmsAsync` used to gate purely on
  `_alarmSubscriptionHandle is not null`, so once a handle existed the alarm set **latched for the life of
  the connection**. Combined with the transient above, a deploy widening a driver's alarm refs `{X}` →
  `{X, Y}` could leave a fast-initialising driver subscribed to `{X}` forever, and the drivers that filter
  their feed on the subscribed `sourceNodeIds` (OPC UA Client, AB CIP, FOCAS) would simply never deliver
  the new alarm — silently. The guard is now "the desired refs differ from the ones the live handle was
  established with" (`_subscribedAlarmRefs`), and a changed set **releases the old subscription before
  re-subscribing** so a filtering driver does not accumulate two overlapping ones. There is still
  deliberately no unsubscribe-on-**empty**: a removed alarm tag's condition node is torn down on the
  address-space rebuild and `ForwardNativeAlarm` drops any transition whose `ConditionId` no longer maps.
- ⚠️ **A data re-subscribe must not touch the alarm feed (#662).** Every deploy's bulk push re-sends
  `SetDesiredSubscriptions` to every kept driver, and `HandleSubscribeAsync` drops the prior DATA
  subscription first. That teardown used to be the full `DetachSubscription`, which also unhooked
  `OnAlarmEvent` and forgot the alarm handle — so on any deploy that did **not** respawn a driver (e.g. one
  that only disabled another driver) its alarm subscription kept logging "established" while every event
  reached nobody, until a reconnect or respawn. The data path now uses `DetachDataSubscription`; only a lost
  connection (`DisconnectObserved` / `ForceReconnect`) and `PostStop` detach both. The **#488 periodic
  reconcile covers alarms too** (`ReconcileAlarmSubscription`): it re-attaches a missing `OnAlarmEvent`
  handler and re-subscribes when the live alarm handle does not match the desired set, so an alarm subscribe
  that failed on (re)connect — a session not yet ready after an outage — is retried every health-poll tick
  instead of gating the feed for the life of the connection. OpcUaClient removes a half-built alarm
  subscription from its session when the subscribe throws, so those retries do not accumulate.
  End-to-end pin: `NativeAlarmFeedSurvivesDeployTests`.
- **There is no in-place config path — a config change is a respawn (#665).** Since #516 any DriverConfig,
  DriverType or ResilienceConfig change is stop + respawn, so the factory is the single parse authority. The old
  in-place seam (host `ApplyChildDelta` → actor `ApplyDelta` → `ReinitializeAsync`, plus the mid-connect
  `AdoptConfigDuringInit`, `ApplyResult` and `DeltaApplied`) had no producer and was deleted after two latent
  bugs were found in it: a Connected reinit kept subscription handles the driver had dropped (#663), and the
  mid-connect path re-ran `InitializeAsync(newConfig)` on two drivers that ignored the JSON (#664). Kept from that
  work: Calculation's `InitializeAsync` adopts a config with a body and Galaxy's refuses a non-equivalent one,
  because the `IDriver` contract is that a driver serves the document it is handed. `ReinitializeAsync` stays as a
  per-driver contract the runtime does not call.
- **Restart tests must assert a FUNCTION, never existence.** `RestartDriver_respawns_the_child_without_an_actor_name_collision`
  asserts only that a child exists after a restart, and it passed green against the dead driver 03/S16
  describes. It is kept — its own subject, the `InvalidActorNameException`, is real — but the load-bearing
  suite is `DriverHostActorRestartTests`, which asserts that the new incarnation *subscribed* and
  *re-initialised*. Its file remarks also record the supervision crash seam and why the obvious
  alternatives (a throwing health publisher, a throwing `SubscribeAsync`, `Kill.Instance`) do not work.

---

## IDriverHealthPublisher — health pub/sub sink

[`Core.Abstractions/IDriverHealthPublisher.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverHealthPublisher.cs)

A sink for driver-health state-change notifications. Implementations must be
non-blocking and safe to call from any thread.

Member:

- `void Publish(string clusterId, string driverInstanceId, DriverHealth health, int errorCount5Min)`

Implementations:

- `NullDriverHealthPublisher` (same file) is the drop-in no-op for tests and
  dev-stub paths. A `DriverInstanceActor` defaults to it when no publisher is
  supplied.
- `AkkaDriverHealthPublisher`
  ([`Runtime/Drivers/AkkaDriverHealthPublisher.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/AkkaDriverHealthPublisher.cs))
  is the production binding: it forwards each transition as a
  `DriverHealthChanged` message onto the cluster-wide `driver-health`
  Akka DistributedPubSub topic.

Producer: `DriverInstanceActor`
([`Runtime/Drivers/DriverInstanceActor.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs))
calls `Publish` when a driver's health transitions. The published snapshot is
consumed AdminUI-side and surfaced through the driver-status panel (read
in-process by the AdminUI bridge rather than dialing its own hub).

### Writing health around a teardown (Gitea #517)

Two rules, and the second is the counter-intuitive one:

1. **`GetHealth()` must report actual state, never a constant.** A hard-coded `Healthy` reports a
   healthy driver before initialize and after shutdown alike. `CalculationDriver` did exactly this —
   it was the fleet's one driver reporting Healthy with its timers stopped, deterministically rather
   than as a race. A driver with no transport still has a running / not-running distinction.

2. **Land the terminal health value *after* the teardown — and, where the window is observable, write
   it *before* as well.** #517's original suggestion was to move the write earlier so the race window
   leans conservative. Moving it is wrong: teardown drains in-flight poll/probe loops, and those loops
   write health as they unwind, so a driver that only wrote first could have a late loop overwrite the
   shutdown state with `Healthy` and leave *that* as the resting value — with every connection closed.
   The safe shape is a **bracket**: conservative value before, terminal value after.

Where the window is actually observable is worth knowing before adding brackets everywhere:

| Path | Caller | Can `GetHealth()` land mid-teardown? |
|---|---|---|
| `ShutdownAsync` | `DriverInstanceActor.PostStop` | **No** — actor thread, after the mailbox drains, so the health poll is serialised against it. |
| `InitializeAsync` failure | `DriverInstanceActor.InitializeAsync` | **Yes** — invoked on a fire-and-forget `Task.Run`, so the health-poll tick keeps calling `GetHealth()` throughout. |

So a driver whose `InitializeAsync` disposes anything on its failure path must bracket; only
`MTConnectDriver` does today, because it is the only one that tears down inside a `catch`. The
shutdown paths deliberately keep the single trailing write.

### The quality contract

> **A value consumer never reports a quality better than its source's.** A node derived from a driver
> value — the raw node, every UNS mirror, a Calculation output, a virtual tag, a scripted condition's
> quality — carries, at worst, the worst status among its inputs: a Bad input makes the derived node Bad
> **with that input's sub-code** (the last-known value is kept server-side, but see the note below: a client never sees it beside a Bad status); an Uncertain input makes it Uncertain; a source
> that has stopped communicating is a Bad input (`BadNoCommunication`) for everything downstream of it,
> **including the external OPC UA data plane**. Recovery is a *publish*, never an assumption: the first
> post-recovery value must land even when it equals the value held before the outage, so every dedup on
> the path — driver caches, the node's own status, a consumer's last-published memory — must be keyed on
> **(value, status)**, not value alone.
>
> **"Value retained" is server-side state, not wire-observable.** The node keeps its last value under the
> Bad status (that is what lets the next Good re-publish carry it and what the `(value, status)` dedup
> compares against), but the OPC UA SDK's `NodeState.ReadAttribute` nulls the value of any Bad-status read
> and monitored-item sample, so a client sees `Value = null` with the Bad status and never the last-known
> value. Pinned server-side by `OtOpcUaNodeManagerWriteStatusesTests`; observation: `archreview/plans/R3-02-stale-good-family-plan.md`.

This is one rule with one implementation of its comparison,
`ZB.MOM.WW.OtOpcUa.Core.Abstractions.InputQuality` (`WorstOf` / `IsUsable`), used by every derived-value
seam. It was written down because the same lie had been fixed four times in four places and re-opened in
a fifth (archreview round 3, theme "the stale-Good family"). The seams that implement it:

| Seam | What it does | Where |
|---|---|---|
| Driver comms loss → **value nodes** | one status-only batch stamps `BadNoCommunication` on the driver's raw NodeIds and every referencing UNS NodeId, **value retained server-side** (clients see a null value with the Bad status) | `DriverHostActor.FanCommsLossToValueNodes` (below) |
| Driver comms loss → dependency mux | synthetic Bad per ref, so scripts and alarms see the source die | `DriverHostActor.FanCommsLossToDependencyMux` |
| Driver comms loss → Part 9 conditions | condition quality annotated out of band | `DriverHostActor.OnDriverConnectivityChanged` |
| MQTT / Sparkplug | connection loss stales observed refs; every (re)subscribe pushes the cached snapshot | [`docs/drivers/Mqtt.md`](drivers/Mqtt.md#quality-on-connection-loss-and-re-subscribe) |
| Calculation driver | holds on Bad, propagates the worst input status | [`docs/drivers/Calculation.md`](drivers/Calculation.md#input-quality) |
| Virtual tags | holds on Bad, inherits the worst input status, reassert carries it | [`docs/ScriptEditor.md`](ScriptEditor.md#input-quality-what-a-script-sees-and-when-it-runs-at-all) |
| Scripted alarms | worst-of-input quality annotated out of band; Bad inputs hold the condition | [`docs/ScriptedAlarms.md`](ScriptedAlarms.md) |

⚠️ **Consumer-visible:** value nodes of a comms-lost driver now go **Bad**, where they used to serve
their last value at Good. A consumer that treats a Bad value as "drop the tag" (ScadaBridge's data
connection layer, say) will see more Bads than before — that is the fix working, not a regression.

### The comms-loss watchdog (Gitea #527) — `GetHealth()` is now load-bearing

`GetHealth()` used to be **reporting-only**: `DriverInstanceActor.PublishHealthSnapshot` read it every 30 s
and forwarded it to the AdminUI, and nothing ever acted on a driver that reported itself unhealthy. That
left `DisconnectObserved` with **no production sender at all** — the only paths into `Reconnecting` were a
failed *initial* connect and an operator's Force Reconnect. A driver that connected and then lost its
device stayed `Connected` forever: publishing Bad for every tag, never re-initialising, and **not
recovering when the device came back** (live-confirmed against a stopped Modbus fixture, which needed a
node restart). Both comms-loss quality paths — #477's native-condition annotation and #481's
scripted-alarm input fan-out — hang off `ConnectivityChanged`, so both were wired to a signal that never
fired.

The `Connected`-state health tick now also evaluates a watchdog, and self-sends `DisconnectObserved` when
**both** of these hold:

1. the driver reports a state other than `Healthy`, **and**
2. `DriverHealth.LastSuccessfulRead` has not advanced for `DefaultCommsLossTimeout` (90 s = three ticks),
   measured from **the later of that read stamp and the current Connected entry** (see "the grace is per
   connection" below).

**The conjunction is the whole design, and "non-Healthy for N polls" is the wrong rule.** `Degraded` means
*connected but some data is bad* — the enum says so. Several drivers publish health **per tag inside their
read loop** (`ModbusDriver.ReadAsync` writes `Healthy` or `Degraded` once per reference), so a single
unreadable address leaves the driver latched `Degraded` whenever it happens to be read last, indefinitely,
against a device that is perfectly reachable. A state-only watchdog would tear that driver down every 90 s
forever — strictly worse than the bug it fixes. `LastSuccessfulRead` is the honest liveness evidence: every
driver advances it on any successful read and *preserves* it across `Degraded`/`Faulted` writes, so one bad
tag among good ones keeps it moving while a device that is genuinely gone freezes it. The state check is
still needed as the second conjunct, because a `Healthy` driver with nothing subscribed legitimately never
reads and must not be declared dead for being idle. Both halves are pinned by their own test in
`DriverInstanceActorCommsLossWatchdogTests`, each of which goes red when the *other* conjunct is deleted.

**The grace is per CONNECTION — the baseline is `max(LastSuccessfulRead, connectedSince)`, not
`LastSuccessfulRead ?? connectedSince`** (archreview 03/S17, fixed 2026-08-20). The two spellings differ
for exactly the case that matters, and the `??` one collapsed the grace to a single health tick. Because
`LastSuccessfulRead` is *preserved* across `Degraded`/`Faulted` writes — that preservation is the design
described above — a device that accepts a TCP connection but never serves data (a half-broken PLC, a wrong
unit id, a firewalled read path) reconnects carrying a read stamp frozen from before the outage, already
far older than the threshold. Measuring from that stamp alone re-trips on the **first** tick of every new
connection, so the reconnect cycle ran at health-tick cadence (~30 s in production) rather than at the
designed 90 s. Taking the *later* of the two gives each connection its own full window, which is what
re-stamping `connectedSince` on every Connected entry was always for; `??` only ever helped a driver that
had never read at all, and the max still covers that case.

What this does **not** change: the conjunction, the disable-at-zero hatch, and a driver that stops reading
*within* a stable connection (its read stamp is the later term, so it still trips on schedule measured
from that read). And it **paces** the half-broken-device cycle rather than ending it — expect at most one
reconnect attempt per threshold per connection, not silence. Pinned by
`A_reconnected_driver_gets_a_full_grace_window_before_the_watchdog_trips_again` (the delay) and
`A_half_broken_device_is_still_declared_lost_once_the_grace_elapses` (the anti-vacuity half — without it,
a "fix" that simply disabled the watchdog after the first connection would look correct).

Two consequences worth knowing:

- **`CalculationDriver` can never trip it, correctly.** It reports `Healthy` whenever it is running and has
  no transport to be degraded about. Every other driver reaches a non-Healthy state on sustained failure:
  the nine poll drivers via `Degraded`, MQTT via its live connection state, Galaxy via its reconnect
  supervisor. This was surveyed across all 12 before relying on it — it is the assumption the whole
  mechanism rests on.
- **A device that accepts a connection but serves no data will cycle**, reconnecting once per threshold.
  That is the correct reading of a broken device, it is rate-limited, and it is visible in the health
  snapshot's error count — but do not mistake it for a bug when reading logs.
- ⚠️ **"Reaches a non-Healthy state" is an obligation on each driver, and a driver can fail to meet it
  without failing anything.** Galaxy's entry in the survey above is "via its reconnect supervisor" — which
  holds only for the paths that actually *reach* the supervisor. Its event stream could also end the way a
  closed or evicted gateway session ends it: the gRPC call completes **without an error**, so nothing
  throws, the supervisor was never told, health stayed `Healthy` — and a `Healthy` driver can never satisfy
  conjunct 1, so the watchdog could not fire either. The driver went permanently deaf with nothing
  reporting it anywhere (fixed in `EventPump`; see [Galaxy.md](drivers/Galaxy.md) §Operational Notes).
  The general shape is worth carrying to the next driver: this watchdog only covers failures a driver
  **notices**, so a silent-success termination path — a stream that ends cleanly, a poll loop that exits
  without throwing — bypasses it entirely.

**The reconnect retry now tears down before re-initialising.** `Reconnecting`'s `RetryConnect` calls
`ShutdownAsync` before `InitializeAsync` — the sequence every driver's own `ReinitializeAsync` already
uses — because several drivers **overwrite** an `IAsyncDisposable` transport field inside `InitializeAsync`
without disposing the previous one (`ModbusDriver._transport`, `MqttDriver._connection`). Re-initialising
without a teardown leaks one socket per attempt. That was latent while only a manual Force Reconnect
re-entered this path; the watchdog makes it automatic and unbounded, so a device down for an hour would
have leaked ~360 transports at the 10 s retry interval. The shutdown is best-effort: a driver that throws
on it still gets its initialise attempt.

**`InitializeAsync` itself releases the previous device/connection state first (#630).** The retry above is not
the only way in: the `IDriver` contract allows `InitializeAsync` on a live driver with *no* teardown, and a retry
tick can overlap a still-running attempt, so a driver that does `_devices[host] = new DeviceState(...)` (or `Session = ...`, `Plc = ...`) orphans
the previous state's probe / recycle / fixed-tree loops and its client. FOCAS, AbLegacy, AbCip, TwinCAT, S7 and
OpcUaClient now cancel, bounded-join (5 s) and dispose that state at the top of `InitializeAsync`
(`ReleaseDeviceStatesAsync` / `ReleaseConnectionAsync` / `ReleaseSessionAsync`), alongside Modbus and MQTT.
It is deliberately the device-state half of `ShutdownAsync` only: the poll engine and subscriptions are *not*
dropped: the release is scoped to device state so it cannot take state the shutdown path owns. The actor re-subscribes values and alarms after every successful init, so driver-side subscriptions kept across a no-teardown re-init are retained leftovers, not needed for data flow. A new driver must do the same.

#### The fan-out that watchdog triggers: indexed per driver, and one batch

`DisconnectObserved` raises `ConnectivityChanged`, and `DriverHostActor` answers it by pushing
`BadNoCommunication` at every ref the flapping driver owns — **three** fans: the value nodes (the
external OPC UA surface), the dependency mux, and the publish actor's condition quality. Both used to answer "which refs are this driver's?"
by **scanning the node's entire ref map and filtering** — the maps are keyed by the composite
`(DriverInstanceId, RawPath)`, so one driver's flap walked all ~50k keys on a large node to find its own
few thousand. Each now has a secondary index, `DriverInstanceId → [(RawPath, node set)]`
(`_valueRefsByDriver`, `_alarmRefsByDriver`), so the fan touches only that driver's refs.

The index cannot drift from its primary, because it is never maintained incrementally: both are cleared
and rebuilt whole inside `ApplyDesiredSubscriptions`, in the same pass and the same clear as the maps
they index. Two details are load-bearing if you touch that loop — an entry is added **only on set
creation** (so a repeated `(DriverInstanceId, RawPath)` contributes exactly one), and the node
`HashSet` is held **by reference, not copied**, so nodes registered against the same raw tag later in the
same pass (a UNS reference variable, say) are visible through the index with no second bookkeeping step.

⚠️ **The value fan is now a single message.** It sends one
`DriverInstanceActor.AttributeValueBatchPublished` carrying one item per RawPath, where it previously sent
one `AttributeValuePublished` per ref — 5 000 messages onto the mux that also carries the node's steady
value ingress, every time one driver flapped. Delivery is unchanged: the mux filters the batch per item
against the same registered-interest map the per-value receive consulted, so only the framing moved. The
**alarm** fan is indexed but deliberately not batched — one driver ref can back several condition NodeIds,
so it still emits one `AlarmQualityUpdate` per condition.

**Consequence worth knowing: `AttributeValuePublished` now has no production sender at all.** This fan was
its last one — everything on the subscription path was already wrapped in `AttributeValueBatchPublished`.
`DependencyMuxActor` still registers the per-value receive, but in production it is dead code; it stays
for tests and direct senders.

#### The third fan: the value nodes (archreview 03/S15)

The two fans above reach *internal* consumers and *conditions*. Until archreview round 3 the driver's raw
value variables and their UNS mirrors — what an OPC UA client actually subscribes to — got nothing, so a
client saw the last value at **Good** quality for the whole outage. That was invisible on Modbus (which
degrades per tag within one poll interval on its own) and total on every driver whose failure shape is
"the reader stops producing": a coalesced-read throw, a dropped push session, or simply a driver whose
last act was a Good publish.

`FanCommsLossToValueNodes` closes it with **one status-only batch** — a new sink primitive,
`IOpcUaAddressSpaceSink.WriteStatuses` → `OtOpcUaNodeManager.WriteStatuses` — that changes each node's
StatusCode and Timestamp and **leaves its Value alone**. Status-only is not an optimisation: the host
holds no value cache (v3's single-source design deliberately removed the independent buffer), so only the
node itself can retain the last known value, and a Bad node still carrying its last reading is the OPC UA
Part 8 convention a SCADA consumer expects. Two details:

- It is **ungated by redundancy role**, like the condition annotation: a Secondary keeps its address space
  honest for failover.
- An unmaterialised NodeId is **skipped, never created** — a status carries nothing to create a node from,
  and the per-driver index can legitimately name a node a surgical remove has just dropped.

**Nothing is written on reconnect.** Publishing Good there would assert a quality the host has not
observed. Recovery is the driver's own re-publish: for a poll driver the watchdog's `DetachSubscription`
forces a fresh `PollGroupEngine` subscription whose initial poll runs `forceRaise: true`; for MQTT it is
the initial push added by 05/STAB-18. What the host does instead is **audit** it — `CommsRecoveryAudit`
fires `DefaultCommsRecoveryAuditDelay` (30 s) after the driver reconnects and reports any ref still
latched Bad, as a Warning naming the driver plus the `otopcua.driver.comms_latched_refs` counter (tagged
`driver_instance`, `driver_type`). That counter going non-zero means a driver reconnected without re-publishing —
a push driver missing its initial push, or a value-only dedup swallowing an unchanged value. It is a
tripwire, not a repair: nothing downstream depends on it.

### The connect path feeds the circuit breaker (R2-10 T10 closure)

`IDriver.InitializeAsync` is routed through the capability invoker as
**`DriverCapability.Initialize`**, keyed on the driver-instance id (init spans the whole driver,
like a bulk subscribe). Before this, the connect path was the ONE capability call the invoker never
saw: a driver that could not connect contributed **zero** breaker input, so the AdminUI resilience
panel read *"All targets healthy"* against a dead endpoint — live-proven during the R2-10 T10 gate,
and the reason that gate was structurally unsatisfiable for a year of attempts. Things to know
before touching it:

- **The Initialize policy never retries** — this actor's `RetryConnect` timer *is* the retry loop,
  and pipeline retries would stack inside each timer tick (and stretch one attempt past the timer
  interval so attempts overlap). The parser forces a `retryCount` override back to 0, same funnel
  as the write-shaped rule.
- ⚠️ **The breaker threshold is 2, and raising it silently re-breaks the feature.** The breaker
  samples over 30 s and the reconnect timer fires every 10 s, so at most **three** init attempts fit
  one sampling window — a Read-sized threshold of 5 can literally never open on connect failures,
  which reads exactly like "the panel doesn't work" with nothing logged. Pinned by
  `Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence` and, behaviourally, by
  `Initialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt`.
- **An open breaker cannot wedge the reconnect loop.** A fast-failed attempt
  (`BrokenCircuitException`) lands in the same catch as a real connect failure →
  `InitializeFailed` → the timer keeps ticking. The 15 s break window is shorter than two 10 s
  ticks, so recovery costs at most one extra tick. The teardown `ShutdownAsync` stays deliberately
  UNWRAPPED — best-effort cleanup of an already-dead transport must not count toward the breaker.
- **A respawn resets the instance's tracker snapshots** (`DriverResilienceStatusTracker.Invalidate`,
  called by `DriverCapabilityInvokerFactory.Create` alongside the pipeline invalidation). The
  pipelines carry the actual breaker state; once they are discarded, a retained "Breaker OPEN"
  snapshot could never self-correct — only the discarded pipeline's `OnClosed` stamps the close —
  so a deploy during an outage would otherwise strand a false OPEN chip forever. This was
  unreachable while breakers never opened; the Initialize wrap made it reachable.
- `ConsecutiveFailures` stays 0 during a connect outage **by design** — the tracker counts retry
  attempts, and Initialize has none. The breaker-OPEN chip is the operator signal.

---

## IDriverConfigEditor — custom AdminUI config editor (plug-point)

[`Core.Abstractions/IDriverConfigEditor.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverConfigEditor.cs)

An **optional** plug-point a driver can implement to provide a custom AdminUI
editor for its `DriverConfig` JSON. Drivers that don't implement it fall back to
the generic JSON editor with schema-driven validation. This is the contract
between the driver and the Admin Blazor app; the Admin app discovers
implementations and slots them into the Driver Detail screen.

Members:

- `string DriverType { get; }` — the driver type this editor handles.
- `Type EditorComponentType { get; }` — the Razor component type that renders
  the editor (returned as `Type` so `Core.Abstractions` needs no Blazor
  reference).

Status: this is a forward-looking plug-point. No driver ships a concrete
`IDriverConfigEditor` today — every driver uses the generic JSON editor — so
the interface currently has the contract defined but no implementations.

---

## IHistorianDataSource — server-side historian read surface

[`Core.Abstractions/Historian/IHistorianDataSource.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs)

The server-side historian read surface. Registered with the server's history
router and resolved **per OPC UA namespace**, independent of any driver's
lifecycle. It is a *server registration*, not a driver capability: drivers carry
no history surface (there is no `IHistoryProvider`; deleted in Gitea #655). The
server resolves the data source by namespace and calls it directly, so one
historian (the HistorianGateway) can serve many drivers' nodes, and drivers can
restart without dropping history availability.

The interface is `: IDisposable` and declares the full read surface as
**required** members:

- `ReadRawAsync(fullReference, startUtc, endUtc, maxValuesPerNode, ct)` — raw
  historical samples over a time range.
- `ReadProcessedAsync(fullReference, startUtc, endUtc, interval, aggregate, ct)`
  — interval-bucketed aggregates (average/min/max/count); an empty bucket
  returns a `BadNoData` sample.
- `ReadAtTimeAsync(fullReference, timestampsUtc, ct)` — one sample per requested
  timestamp (OPC UA HistoryReadAtTime); the returned list matches the requested
  length and order, gaps as Bad-quality snapshots.
- `ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, ct)` — historical
  alarm/event records (OPC UA HistoryReadEvents); `sourceName` is `null` to
  return all sources. `maxEvents` is a signed `int` so a non-positive value is a
  "use the backend's default cap" sentinel.
- `GetHealthSnapshot()` — point-in-time health snapshot for diagnostics and
  dashboards; pure observation, never blocks on backend I/O.

All values use the shared `DataValueSnapshot` / `HistoricalEvent` shapes;
backend-specific quality/type encodings are translated to OPC UA `StatusCode`
uints inside the data source.

Implementation:

- `GatewayHistorianDataSource`
  ([`Driver.Historian.Gateway/GatewayHistorianDataSource.cs`](../src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianDataSource.cs))
  — the read backend that talks gRPC to the external `ZB.MOM.WW.HistorianGateway`
  (via the `ZB.MOM.WW.HistorianGateway.Client` package, behind the
  `IHistorianGatewayClient` seam). The alarm-event drain target is the separate
  `GatewayAlarmHistorianWriter` (the gateway `SendEvent` path; see
  [AlarmHistorian.md](AlarmHistorian.md)).

The HistorianGateway is the sole historian backend; its config keys and
deployment prerequisites are in [Historian.md](Historian.md).

---

## PollGroupEngine — poll pacing and teardown

[`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/PollGroupEngine.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/PollGroupEngine.cs)

The shared poll loop behind the seven polling drivers (Modbus, S7, AbCip, AbLegacy, FOCAS, Sql, TwinCAT).
Note it lives in `Core.Abstractions`, not under `src/Drivers/`.

**Pacing is fixed-rate, not fixed-delay.** The loop used to `await Task.Delay(interval)` *after* each
poll, so the real cadence was `interval + read time` and drifted with device latency — an 80 ms read on a
100 ms interval produced a 180 ms cycle, and a subscription's configured publishing interval was a floor
nobody actually got. Each poll is now timed (in a `finally`, so a *failed* poll is measured too) and the
next wait is `max(interval − elapsed, 0)`.

- **An overrun does not skip a cycle and does not burst to catch up.** When the remaining time is zero or
  negative the loop does not call `Task.Delay` at all — it does a `Task.Yield()`, re-checks cancellation,
  and starts the next poll immediately. The rate simply degrades to whatever the device sustains. The
  `Yield` is load-bearing: without it a synchronously-completing reader turns a zero delay into a
  thread-pool-starving tight loop that never observes cancellation.
- **The failure path is exempt.** When `backoffCap` is set *and* there have been consecutive failures, the
  capped exponential `ConnectionBackoff.ComputeDelay` wins outright — a dead device must not be hammered
  at fixed rate. (Previously the backoff formula was consulted whenever a cap was configured, including at
  zero failures where it just returns the base interval.)

**`OverrunCount` is the signal that a device cannot keep up.** It counts poll cycles whose measured
duration exceeded that subscription's publishing interval, engine-wide across every subscription,
monotonic and never reset. A non-zero and *climbing* value means the loop is already running flat out.

- ⚠️ **It conflates a slow success with a slow failure** — a poll that overran because the device was slow
  and one that overran because it timed out increment the same counter. Read it alongside the health
  snapshot's error count, not on its own.
- **There is no meter and no log**, because `Core.Abstractions` deliberately carries no metrics
  dependency. The surface is the `OverrunCount` property plus an optional `onOverrun` constructor
  callback. **No driver wires either today** — a driver-level meter is a recorded follow-up, so treat the
  counter as latent instrumentation rather than something you can read off a dashboard.

**Unsubscribe is awaited, not blocked on.** `Unsubscribe` drained the in-flight poll with a synchronous
`task.Wait(5s)` — and every driver's `UnsubscribeAsync` called it and returned `Task.CompletedTask`, so
that block landed *inside a `DriverInstanceActor` mailbox turn*, stalling the actor for up to five
seconds. `UnsubscribeAsync` now awaits `task.WaitAsync(DrainTimeout)` instead. The synchronous
`Unsubscribe` is retained for compatibility and has no production caller left; prefer the async form on
any path inside an actor turn or a request pipeline. Semantics are identical either way — the loop is
cancelled and awaited *before* the CTS is disposed, so no change callback fires after teardown and the
CTS is never disposed while a `Task.Delay` still holds its token.

---

## Commons — shared cross-cutting primitives

[`src/Core/ZB.MOM.WW.OtOpcUa.Commons/`](../src/Core/ZB.MOM.WW.OtOpcUa.Commons/)

`ZB.MOM.WW.OtOpcUa.Commons` is the low-level shared library that the Runtime,
ControlPlane, AdminUI, and OPC UA server projects all reference. It holds
cross-cutting primitives with no driver- or host-specific behavior, so the
heavier projects can share message contracts and value types without taking a
dependency on each other. It references only `Akka` and the internal
`ZB.MOM.WW.Audit` package.

Folders:

- **`Messages/`** — Akka message contracts grouped by concern (`Admin`,
  `Alerts`, `Deploy`, `Drivers`, `Fleet`, `Logging`, `Redundancy`). These are
  the wire/inter-actor messages — e.g. `Messages/Admin/TestDriverConnect.cs`
  (Test Connect request, see [IDriverProbe](#idriverprobe--test-connect)) and
  `Messages/Drivers/DriverHealthChanged.cs` (the driver-health pub/sub payload,
  see [IDriverHealthPublisher](#idriverhealthpublisher--health-pubsub-sink)).
- **`Interfaces/`** — cluster-facing client contracts such as
  `IAdminOperationsClient`, `IClusterRoleInfo`, and `IFleetDiagnosticsClient`.
- **`Types/`** — strongly-typed identifier value types: `CorrelationId`,
  `DeploymentId`, `ExecutionId`, `NodeId`, `RevisionHash`.
- **`Browsing/`** — live-browse abstractions (`BrowseNode`, `IBrowseSession`,
  `IDriverBrowser`) backing the AdminUI address pickers.
- **`Engines/`** — evaluator seams (`IScriptedAlarmEvaluator`,
  `IVirtualTagEvaluator`, `IAlarmActorStateStore`) consumed by the
  [VirtualTags](VirtualTags.md) / [ScriptedAlarms](ScriptedAlarms.md) engines.
- **`OpcUa/`** — deferred-publish seams (`IOpcUaAddressSpaceSink`,
  `IServiceLevelPublisher` and their `Deferred*` no-op stand-ins) so address-space
  and [ServiceLevel](Redundancy.md) writes can be wired late.
- **`Observability/`** — `OtOpcUaTelemetry` (the shared ActivitySource/metrics
  surface).

---

## See also

- [ReadWriteOperations.md](ReadWriteOperations.md) — the driver *capability*
  interfaces (read/write/subscribe) and resilience pipeline.
- [ServiceHosting.md](ServiceHosting.md) — role gating, the Akka cluster, and
  the external HistorianGateway backend.
- [AlarmHistorian.md](AlarmHistorian.md) — the store-and-forward SQLite alarm
  sink that drains to `IAlarmHistorianWriter`.
- [Redundancy.md](Redundancy.md) — driver stability tiers in the redundancy
  context.