Joseph Doherty
36 KiB
Driver Lifecycle & Server Infrastructure Contracts
Reference for the server-side infrastructure interfaces that surround a
driver but are not driver capabilities (read/write/subscribe/etc.,
documented in ReadWriteOperations.md and the
per-driver pages). These contracts live in
src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/
so they carry no behavior — concrete implementations live in the driver
projects, the Runtime, and the ControlPlane. Each subsection below gives the
purpose, the key members, and where it is implemented/used.
The capability interfaces a driver opts into (IReadable, IWritable,
ITagDiscovery, ISubscribable, IAlarmSource, IHistoryProvider,
IHostConnectivityProbe, IPerCallHostResolver, IRediscoverable) are
covered elsewhere and discovered by the server via is-checks on the
IDriver instance. The interfaces here are the plumbing the server uses to
create, probe, supervise, report on, and configure those
drivers, plus the server-side historian read surface.
IDriverFactory — creating drivers from config rows
Core.Abstractions/IDriverFactory.cs
Abstraction over the process-wide driver registry. The Runtime consumes this
instead of the concrete registry so the Runtime project does not pull in
ZB.MOM.WW.OtOpcUa.Core (which would drag in Polly + driver hosting).
Members:
IDriver? TryCreate(string driverType, string driverInstanceId, string driverConfigJson)— returns a new driver for the given type, ornullwhen no factory is registered for that type (missing assembly, typo). TheDriverHostActorlogs and skips the row rather than failing the whole apply.IReadOnlyCollection<string> SupportedTypes— driver-type names this factory can materialise; mostly for diagnostics and logs.
Implementations:
NullDriverFactory(same file) returnsnullfrom everyTryCreateand exposes zero supported types. Bound when no concrete driver assemblies have been registered (Mac dev path, smoke tests); the deployment becomes a no-op.DriverFactoryRegistry(Core/Hosting/DriverFactoryRegistry.cs) is the real process-singleton registry keyed byDriverInstance.DriverType(case-insensitive). Each driver project ships aRegister(...)extension;Registerrecords the factory and the driver's stabilityDriverTier(defaults to Tier A). Registering the same type twice throws.DriverFactoryRegistryAdapter(Core/Hosting/DriverFactoryRegistryAdapter.cs) bridges the registry to theIDriverFactoryabstraction.
Wiring: DriverFactoryBootstrap.AddOtOpcUaDriverFactories
(Host/Drivers/DriverFactoryBootstrap.cs)
registers the singleton registry, runs every driver assembly's Register
extension, then binds IDriverFactory to the adapter. It must run before
AddAkka so the Runtime can resolve IDriverFactory when spawning the
DriverHostActor
(Runtime/Drivers/DriverHostActor.cs).
The registry is skipped on admin-only nodes (they never run drivers); the
probe set is the exception — see IDriverProbe.
IDriverProbe — Test Connect
Core.Abstractions/IDriverProbe.cs
A cheap test-connect probe for one driver type, backing the AdminUI Test
Connect button. An implementation deserializes a driver-config JSON, attempts
a cheap connection (TCP open, OPC UA session, gRPC ping — whatever the driver's
native protocol supports), and reports success/failure with latency. Probes
must not mutate persistent state: the AdminUI invokes them against the
transient config in the typed form, not against the persisted DriverInstance
row.
Members:
string DriverType { get; }— theDriverInstance.DriverTypestring this probe handles; used for DI lookup.Task<DriverProbeResult> ProbeAsync(string configJson, TimeSpan timeout, CancellationToken ct)— never throws on connection failure; returns a result withOk = falseand a message instead.DriverProbeResult(bool Ok, string? Message, TimeSpan? Latency)— outcome record (Messageisnullon success;Latencyisnullon failure).
Implementations: every driver ships a *DriverProbe in its driver project
(e.g.
Driver.Modbus/ModbusDriverProbe.cs
does a bare socket open/close). The historian backend is the external
HistorianGateway (consumed as a gRPC client package, not an IDriver), so it
has no driver probe.
Flow: the AdminUI's AdminProbeService
(AdminUI/Clients/AdminProbeService.cs)
dispatches a TestDriverConnect message through IAdminOperationsClient to the
cluster-singleton AdminOperationsActor
(ControlPlane/AdminOperations/AdminOperationsActor.cs),
which holds the probes keyed by DriverType and invokes the matching one
(timeout clamped to [1, 60] seconds). Because the admin singleton is
admin-pinned, the probe set must be registered on admin nodes too — Program.cs
calls AddOtOpcUaDriverProbes in the hasAdmin block, and
AddOtOpcUaDriverFactories registers it for fused admin+driver nodes.
IDriverSupervisor / process recycle — REMOVED (Gitea #522)
IDriverSupervisor, MemoryRecycle, MemoryTracking and
ScheduledRecycleScheduler no longer exist. This section used to describe
them as the Tier C protection layer.
They were removed because the thing they acted on is gone. "Tier C" meant a
driver running out-of-process behind a supervisor that could restart its Host
without tearing down the OPC UA session or co-hosted drivers. No such process
remains anywhere: Galaxy reaches MXAccess over gRPC to the external
mxaccessgw sidecar (PR 7.2 retired the in-process Galaxy.Host / Proxy /
Shared projects), and FOCAS has run in-process since its managed wire client
landed 2026-04-24. Consistent with that, IDriverSupervisor had zero
implementations, and MemoryTracking / MemoryRecycle /
ScheduledRecycleScheduler were constructed only in their own unit tests —
so this was not dormant-but-ready code, it was code with no path to running.
The operator-facing half went too: RecycleIntervalSeconds was authorable
through the AdminUI's driver resilience section and validated by the parser
while configuring nothing. A stored ResilienceConfig still carrying the key
parses cleanly (unknown keys are ignored) and the AdminUI preserves it rather
than silently rewriting an operator's config.
DriverTier itself survives and is load-bearing — do not delete it while
following this note. DriverResilienceOptions.GetTierDefaults(tier) supplies
the real per-capability timeout / retry / breaker policies, resolved via
DriverFactoryRegistry.GetTier and applied by DriverCapabilityInvokerFactory.
Every driver runs Tier A because no factory passes a tier, so today that means
one uniform policy set.
IDriver no longer carries GetMemoryFootprint() or
FlushOptionalCachesAsync() (Gitea #525). They existed to feed
MemoryTracking, deleted with the rest of the driver-tier recycle machinery in
#522; after that nothing called them, and their doc-comments still claimed
Core polled the footprint every 30 s, which had not been true since v3. The
sweep removed them from the interface, all 12 drivers, and every test stub,
along with the support code that existed only to feed them — MTConnect's
per-entry byte weights and its cached data-item count, OpcUaClient's
_discoveredNodeCount, and MQTT's per-tag byte estimate.
Two consequences worth knowing:
AbCipTemplateCachenow has no production reset path. Its only caller was AB CIP's flush — which itself had no caller, so nothing has actually changed in behaviour, only in how visible the fact is. A UDT redefined on a live controller keeps its stale shape until the driver instance is rebuilt.- MTConnect's
/probecache is still dropped, byInvalidateProbeModel()on an Agent restart. The tests that used to reach it through the flush now drive that path instead, so they cover a case that actually occurs.
If per-driver memory protection is wanted again, build it for the architecture as it is now: a process-wide watchdog that can ask drivers to flush, not a per-driver tier whose only remedy requires a supervisor that cannot exist in-process. The deleted per-driver implementations are in git history and were mostly reusable — what was always missing is the trigger.
IDriverHealthPublisher — health pub/sub sink
Core.Abstractions/IDriverHealthPublisher.cs
A sink for driver-health state-change notifications. Implementations must be non-blocking and safe to call from any thread.
Member:
void Publish(string clusterId, string driverInstanceId, DriverHealth health, int errorCount5Min)
Implementations:
NullDriverHealthPublisher(same file) is the drop-in no-op for tests and dev-stub paths. ADriverInstanceActordefaults to it when no publisher is supplied.AkkaDriverHealthPublisher(Runtime/Drivers/AkkaDriverHealthPublisher.cs) is the production binding: it forwards each transition as aDriverHealthChangedmessage onto the cluster-widedriver-healthAkka DistributedPubSub topic.
Producer: DriverInstanceActor
(Runtime/Drivers/DriverInstanceActor.cs)
calls Publish when a driver's health transitions. The published snapshot is
consumed AdminUI-side and surfaced through the driver-status panel (read
in-process by the AdminUI bridge rather than dialing its own hub).
Writing health around a teardown (Gitea #517)
Two rules, and the second is the counter-intuitive one:
-
GetHealth()must report actual state, never a constant. A hard-codedHealthyreports a healthy driver before initialize and after shutdown alike.CalculationDriverdid exactly this — it was the fleet's one driver reporting Healthy with its timers stopped, deterministically rather than as a race. A driver with no transport still has a running / not-running distinction. -
Land the terminal health value after the teardown — and, where the window is observable, write it before as well. #517's original suggestion was to move the write earlier so the race window leans conservative. Moving it is wrong: teardown drains in-flight poll/probe loops, and those loops write health as they unwind, so a driver that only wrote first could have a late loop overwrite the shutdown state with
Healthyand leave that as the resting value — with every connection closed. The safe shape is a bracket: conservative value before, terminal value after.
Where the window is actually observable is worth knowing before adding brackets everywhere:
| Path | Caller | Can GetHealth() land mid-teardown? |
|---|---|---|
ShutdownAsync |
DriverInstanceActor.PostStop |
No — actor thread, after the mailbox drains, so the health poll is serialised against it. |
InitializeAsync failure |
DriverInstanceActor.InitializeAsync |
Yes — invoked on a fire-and-forget Task.Run, so the health-poll tick keeps calling GetHealth() throughout. |
So a driver whose InitializeAsync disposes anything on its failure path must bracket; only
MTConnectDriver does today, because it is the only one that tears down inside a catch. The
shutdown paths deliberately keep the single trailing write.
The quality contract
A value consumer never reports a quality better than its source's. A node derived from a driver value — the raw node, every UNS mirror, a Calculation output, a virtual tag, a scripted condition's quality — carries, at worst, the worst status among its inputs: a Bad input makes the derived node Bad with that input's sub-code and the last-known value; an Uncertain input makes it Uncertain; a source that has stopped communicating is a Bad input (
BadNoCommunication) for everything downstream of it, including the external OPC UA data plane. Recovery is a publish, never an assumption: the first post-recovery value must land even when it equals the value held before the outage, so every dedup on the path — driver caches, the node's own status, a consumer's last-published memory — must be keyed on (value, status), not value alone.
This is one rule with one implementation of its comparison,
ZB.MOM.WW.OtOpcUa.Core.Abstractions.InputQuality (WorstOf / IsUsable), used by every derived-value
seam. It was written down because the same lie had been fixed four times in four places and re-opened in
a fifth (archreview round 3, theme "the stale-Good family"). The seams that implement it:
| Seam | What it does | Where |
|---|---|---|
| Driver comms loss → value nodes | one status-only batch stamps BadNoCommunication on the driver's raw NodeIds and every referencing UNS NodeId, value retained |
DriverHostActor.FanCommsLossToValueNodes (below) |
| Driver comms loss → dependency mux | synthetic Bad per ref, so scripts and alarms see the source die | DriverHostActor.FanCommsLossToDependencyMux |
| Driver comms loss → Part 9 conditions | condition quality annotated out of band | DriverHostActor.OnDriverConnectivityChanged |
| MQTT / Sparkplug | connection loss stales observed refs; every (re)subscribe pushes the cached snapshot | docs/drivers/Mqtt.md |
| Calculation driver | holds on Bad, propagates the worst input status | docs/drivers/Calculation.md |
| Virtual tags | holds on Bad, inherits the worst input status, reassert carries it | docs/ScriptEditor.md |
| Scripted alarms | worst-of-input quality annotated out of band; Bad inputs hold the condition | docs/ScriptedAlarms.md |
⚠️ Consumer-visible: value nodes of a comms-lost driver now go Bad, where they used to serve their last value at Good. A consumer that treats a Bad value as "drop the tag" (ScadaBridge's data connection layer, say) will see more Bads than before — that is the fix working, not a regression.
The comms-loss watchdog (Gitea #527) — GetHealth() is now load-bearing
GetHealth() used to be reporting-only: DriverInstanceActor.PublishHealthSnapshot read it every 30 s
and forwarded it to the AdminUI, and nothing ever acted on a driver that reported itself unhealthy. That
left DisconnectObserved with no production sender at all — the only paths into Reconnecting were a
failed initial connect and an operator's Force Reconnect. A driver that connected and then lost its
device stayed Connected forever: publishing Bad for every tag, never re-initialising, and not
recovering when the device came back (live-confirmed against a stopped Modbus fixture, which needed a
node restart). Both comms-loss quality paths — #477's native-condition annotation and #481's
scripted-alarm input fan-out — hang off ConnectivityChanged, so both were wired to a signal that never
fired.
The Connected-state health tick now also evaluates a watchdog, and self-sends DisconnectObserved when
both of these hold:
- the driver reports a state other than
Healthy, and DriverHealth.LastSuccessfulReadhas not advanced forDefaultCommsLossTimeout(90 s = three ticks), measured from the Connected entry when the driver has never reported a successful read.
The conjunction is the whole design, and "non-Healthy for N polls" is the wrong rule. Degraded means
connected but some data is bad — the enum says so. Several drivers publish health per tag inside their
read loop (ModbusDriver.ReadAsync writes Healthy or Degraded once per reference), so a single
unreadable address leaves the driver latched Degraded whenever it happens to be read last, indefinitely,
against a device that is perfectly reachable. A state-only watchdog would tear that driver down every 90 s
forever — strictly worse than the bug it fixes. LastSuccessfulRead is the honest liveness evidence: every
driver advances it on any successful read and preserves it across Degraded/Faulted writes, so one bad
tag among good ones keeps it moving while a device that is genuinely gone freezes it. The state check is
still needed as the second conjunct, because a Healthy driver with nothing subscribed legitimately never
reads and must not be declared dead for being idle. Both halves are pinned by their own test in
DriverInstanceActorCommsLossWatchdogTests, each of which goes red when the other conjunct is deleted.
Two consequences worth knowing:
CalculationDrivercan never trip it, correctly. It reportsHealthywhenever it is running and has no transport to be degraded about. Every other driver reaches a non-Healthy state on sustained failure: the nine poll drivers viaDegraded, MQTT via its live connection state, Galaxy via its reconnect supervisor. This was surveyed across all 12 before relying on it — it is the assumption the whole mechanism rests on.- A device that accepts a connection but serves no data will cycle, reconnecting once per threshold. That is the correct reading of a broken device, it is rate-limited, and it is visible in the health snapshot's error count — but do not mistake it for a bug when reading logs.
- ⚠️ "Reaches a non-Healthy state" is an obligation on each driver, and a driver can fail to meet it
without failing anything. Galaxy's entry in the survey above is "via its reconnect supervisor" — which
holds only for the paths that actually reach the supervisor. Its event stream could also end the way a
closed or evicted gateway session ends it: the gRPC call completes without an error, so nothing
throws, the supervisor was never told, health stayed
Healthy— and aHealthydriver can never satisfy conjunct 1, so the watchdog could not fire either. The driver went permanently deaf with nothing reporting it anywhere (fixed inEventPump; see Galaxy.md §Operational Notes). The general shape is worth carrying to the next driver: this watchdog only covers failures a driver notices, so a silent-success termination path — a stream that ends cleanly, a poll loop that exits without throwing — bypasses it entirely.
The reconnect retry now tears down before re-initialising. Reconnecting's RetryConnect calls
ShutdownAsync before InitializeAsync — the sequence every driver's own ReinitializeAsync already
uses — because several drivers overwrite an IAsyncDisposable transport field inside InitializeAsync
without disposing the previous one (ModbusDriver._transport, MqttDriver._connection). Re-initialising
without a teardown leaks one socket per attempt. That was latent while only a manual Force Reconnect
re-entered this path; the watchdog makes it automatic and unbounded, so a device down for an hour would
have leaked ~360 transports at the 10 s retry interval. The shutdown is best-effort: a driver that throws
on it still gets its initialise attempt.
The fan-out that watchdog triggers: indexed per driver, and one batch
DisconnectObserved raises ConnectivityChanged, and DriverHostActor answers it by pushing
BadNoCommunication at every ref the flapping driver owns — three fans: the value nodes (the
external OPC UA surface), the dependency mux, and the publish actor's condition quality. Both used to answer "which refs are this driver's?"
by scanning the node's entire ref map and filtering — the maps are keyed by the composite
(DriverInstanceId, RawPath), so one driver's flap walked all ~50k keys on a large node to find its own
few thousand. Each now has a secondary index, DriverInstanceId → [(RawPath, node set)]
(_valueRefsByDriver, _alarmRefsByDriver), so the fan touches only that driver's refs.
The index cannot drift from its primary, because it is never maintained incrementally: both are cleared
and rebuilt whole inside ApplyDesiredSubscriptions, in the same pass and the same clear as the maps
they index. Two details are load-bearing if you touch that loop — an entry is added only on set
creation (so a repeated (DriverInstanceId, RawPath) contributes exactly one), and the node
HashSet is held by reference, not copied, so nodes registered against the same raw tag later in the
same pass (a UNS reference variable, say) are visible through the index with no second bookkeeping step.
⚠️ The value fan is now a single message. It sends one
DriverInstanceActor.AttributeValueBatchPublished carrying one item per RawPath, where it previously sent
one AttributeValuePublished per ref — 5 000 messages onto the mux that also carries the node's steady
value ingress, every time one driver flapped. Delivery is unchanged: the mux filters the batch per item
against the same registered-interest map the per-value receive consulted, so only the framing moved. The
alarm fan is indexed but deliberately not batched — one driver ref can back several condition NodeIds,
so it still emits one AlarmQualityUpdate per condition.
Consequence worth knowing: AttributeValuePublished now has no production sender at all. This fan was
its last one — everything on the subscription path was already wrapped in AttributeValueBatchPublished.
DependencyMuxActor still registers the per-value receive, but in production it is dead code; it stays
for tests and direct senders.
The third fan: the value nodes (archreview 03/S15)
The two fans above reach internal consumers and conditions. Until archreview round 3 the driver's raw value variables and their UNS mirrors — what an OPC UA client actually subscribes to — got nothing, so a client saw the last value at Good quality for the whole outage. That was invisible on Modbus (which degrades per tag within one poll interval on its own) and total on every driver whose failure shape is "the reader stops producing": a coalesced-read throw, a dropped push session, or simply a driver whose last act was a Good publish.
FanCommsLossToValueNodes closes it with one status-only batch — a new sink primitive,
IOpcUaAddressSpaceSink.WriteStatuses → OtOpcUaNodeManager.WriteStatuses — that changes each node's
StatusCode and Timestamp and leaves its Value alone. Status-only is not an optimisation: the host
holds no value cache (v3's single-source design deliberately removed the independent buffer), so only the
node itself can retain the last known value, and a Bad node still carrying its last reading is the OPC UA
Part 8 convention a SCADA consumer expects. Two details:
- It is ungated by redundancy role, like the condition annotation: a Secondary keeps its address space honest for failover.
- An unmaterialised NodeId is skipped, never created — a status carries nothing to create a node from, and the per-driver index can legitimately name a node a surgical remove has just dropped.
Nothing is written on reconnect. Publishing Good there would assert a quality the host has not
observed. Recovery is the driver's own re-publish: for a poll driver the watchdog's DetachSubscription
forces a fresh PollGroupEngine subscription whose initial poll runs forceRaise: true; for MQTT it is
the initial push added by 05/STAB-18. What the host does instead is audit it — CommsRecoveryAudit
fires DefaultCommsRecoveryAuditDelay (30 s) after the driver reconnects and reports any ref still
latched Bad, as a Warning naming the driver plus the otopcua.driver.comms_latched_refs counter (tagged
driver, driver_type). That counter going non-zero means a driver reconnected without re-publishing —
a push driver missing its initial push, or a value-only dedup swallowing an unchanged value. It is a
tripwire, not a repair: nothing downstream depends on it.
The connect path feeds the circuit breaker (R2-10 T10 closure)
IDriver.InitializeAsync is routed through the capability invoker as
DriverCapability.Initialize, keyed on the driver-instance id (init spans the whole driver,
like a bulk subscribe). Before this, the connect path was the ONE capability call the invoker never
saw: a driver that could not connect contributed zero breaker input, so the AdminUI resilience
panel read "All targets healthy" against a dead endpoint — live-proven during the R2-10 T10 gate,
and the reason that gate was structurally unsatisfiable for a year of attempts. Things to know
before touching it:
- The Initialize policy never retries — this actor's
RetryConnecttimer is the retry loop, and pipeline retries would stack inside each timer tick (and stretch one attempt past the timer interval so attempts overlap). The parser forces aretryCountoverride back to 0, same funnel as the write-shaped rule. - ⚠️ The breaker threshold is 2, and raising it silently re-breaks the feature. The breaker
samples over 30 s and the reconnect timer fires every 10 s, so at most three init attempts fit
one sampling window — a Read-sized threshold of 5 can literally never open on connect failures,
which reads exactly like "the panel doesn't work" with nothing logged. Pinned by
Initialize_BreakerThreshold_IsReachableAtTheReconnectCadenceand, behaviourally, byInitialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt. - An open breaker cannot wedge the reconnect loop. A fast-failed attempt
(
BrokenCircuitException) lands in the same catch as a real connect failure →InitializeFailed→ the timer keeps ticking. The 15 s break window is shorter than two 10 s ticks, so recovery costs at most one extra tick. The teardownShutdownAsyncstays deliberately UNWRAPPED — best-effort cleanup of an already-dead transport must not count toward the breaker. - A respawn resets the instance's tracker snapshots (
DriverResilienceStatusTracker.Invalidate, called byDriverCapabilityInvokerFactory.Createalongside the pipeline invalidation). The pipelines carry the actual breaker state; once they are discarded, a retained "Breaker OPEN" snapshot could never self-correct — only the discarded pipeline'sOnClosedstamps the close — so a deploy during an outage would otherwise strand a false OPEN chip forever. This was unreachable while breakers never opened; the Initialize wrap made it reachable. ConsecutiveFailuresstays 0 during a connect outage by design — the tracker counts retry attempts, and Initialize has none. The breaker-OPEN chip is the operator signal.
IDriverConfigEditor — custom AdminUI config editor (plug-point)
Core.Abstractions/IDriverConfigEditor.cs
An optional plug-point a driver can implement to provide a custom AdminUI
editor for its DriverConfig JSON. Drivers that don't implement it fall back to
the generic JSON editor with schema-driven validation. This is the contract
between the driver and the Admin Blazor app; the Admin app discovers
implementations and slots them into the Driver Detail screen.
Members:
string DriverType { get; }— the driver type this editor handles.Type EditorComponentType { get; }— the Razor component type that renders the editor (returned asTypesoCore.Abstractionsneeds no Blazor reference).
Status: this is a forward-looking plug-point. No driver ships a concrete
IDriverConfigEditor today — every driver uses the generic JSON editor — so
the interface currently has the contract defined but no implementations.
IHistorianDataSource — server-side historian read surface
Core.Abstractions/Historian/IHistorianDataSource.cs
The server-side historian read surface. Registered with the server's history
router and resolved per OPC UA namespace, independent of any driver's
lifecycle. This is distinct from the driver capability IHistoryProvider:
IHistoryProvideris a driver capability — the server dispatches to it via the driver instance.IHistorianDataSourceis a server registration — the server resolves it by namespace and calls it directly, so one historian (the HistorianGateway) can serve many drivers' nodes, and drivers can restart without dropping history availability.
The interface is : IDisposable and declares the full read surface as
required members (unlike IHistoryProvider, where at-time/event reads are
optional default-impl methods so legacy drivers can stay raw-only):
ReadRawAsync(fullReference, startUtc, endUtc, maxValuesPerNode, ct)— raw historical samples over a time range.ReadProcessedAsync(fullReference, startUtc, endUtc, interval, aggregate, ct)— interval-bucketed aggregates (average/min/max/count); an empty bucket returns aBadNoDatasample.ReadAtTimeAsync(fullReference, timestampsUtc, ct)— one sample per requested timestamp (OPC UA HistoryReadAtTime); the returned list matches the requested length and order, gaps as Bad-quality snapshots.ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, ct)— historical alarm/event records (OPC UA HistoryReadEvents);sourceNameisnullto return all sources.maxEventsis a signedintso a non-positive value is a "use the backend's default cap" sentinel.GetHealthSnapshot()— point-in-time health snapshot for diagnostics and dashboards; pure observation, never blocks on backend I/O.
All values use the shared DataValueSnapshot / HistoricalEvent shapes;
backend-specific quality/type encodings are translated to OPC UA StatusCode
uints inside the data source.
Implementation:
GatewayHistorianDataSource(Driver.Historian.Gateway/GatewayHistorianDataSource.cs) — the read backend that talks gRPC to the externalZB.MOM.WW.HistorianGateway(via theZB.MOM.WW.HistorianGateway.Clientpackage, behind theIHistorianGatewayClientseam). The alarm-event drain target is the separateGatewayAlarmHistorianWriter(the gatewaySendEventpath; see AlarmHistorian.md).
The HistorianGateway is the sole historian backend; its config keys and deployment prerequisites are in Historian.md.
PollGroupEngine — poll pacing and teardown
src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/PollGroupEngine.cs
The shared poll loop behind the seven polling drivers (Modbus, S7, AbCip, AbLegacy, FOCAS, Sql, TwinCAT).
Note it lives in Core.Abstractions, not under src/Drivers/.
Pacing is fixed-rate, not fixed-delay. The loop used to await Task.Delay(interval) after each
poll, so the real cadence was interval + read time and drifted with device latency — an 80 ms read on a
100 ms interval produced a 180 ms cycle, and a subscription's configured publishing interval was a floor
nobody actually got. Each poll is now timed (in a finally, so a failed poll is measured too) and the
next wait is max(interval − elapsed, 0).
- An overrun does not skip a cycle and does not burst to catch up. When the remaining time is zero or
negative the loop does not call
Task.Delayat all — it does aTask.Yield(), re-checks cancellation, and starts the next poll immediately. The rate simply degrades to whatever the device sustains. TheYieldis load-bearing: without it a synchronously-completing reader turns a zero delay into a thread-pool-starving tight loop that never observes cancellation. - The failure path is exempt. When
backoffCapis set and there have been consecutive failures, the capped exponentialConnectionBackoff.ComputeDelaywins outright — a dead device must not be hammered at fixed rate. (Previously the backoff formula was consulted whenever a cap was configured, including at zero failures where it just returns the base interval.)
OverrunCount is the signal that a device cannot keep up. It counts poll cycles whose measured
duration exceeded that subscription's publishing interval, engine-wide across every subscription,
monotonic and never reset. A non-zero and climbing value means the loop is already running flat out.
- ⚠️ It conflates a slow success with a slow failure — a poll that overran because the device was slow and one that overran because it timed out increment the same counter. Read it alongside the health snapshot's error count, not on its own.
- There is no meter and no log, because
Core.Abstractionsdeliberately carries no metrics dependency. The surface is theOverrunCountproperty plus an optionalonOverrunconstructor callback. No driver wires either today — a driver-level meter is a recorded follow-up, so treat the counter as latent instrumentation rather than something you can read off a dashboard.
Unsubscribe is awaited, not blocked on. Unsubscribe drained the in-flight poll with a synchronous
task.Wait(5s) — and every driver's UnsubscribeAsync called it and returned Task.CompletedTask, so
that block landed inside a DriverInstanceActor mailbox turn, stalling the actor for up to five
seconds. UnsubscribeAsync now awaits task.WaitAsync(DrainTimeout) instead. The synchronous
Unsubscribe is retained for compatibility and has no production caller left; prefer the async form on
any path inside an actor turn or a request pipeline. Semantics are identical either way — the loop is
cancelled and awaited before the CTS is disposed, so no change callback fires after teardown and the
CTS is never disposed while a Task.Delay still holds its token.
Commons — shared cross-cutting primitives
src/Core/ZB.MOM.WW.OtOpcUa.Commons/
ZB.MOM.WW.OtOpcUa.Commons is the low-level shared library that the Runtime,
ControlPlane, AdminUI, and OPC UA server projects all reference. It holds
cross-cutting primitives with no driver- or host-specific behavior, so the
heavier projects can share message contracts and value types without taking a
dependency on each other. It references only Akka and the internal
ZB.MOM.WW.Audit package.
Folders:
Messages/— Akka message contracts grouped by concern (Admin,Alerts,Deploy,Drivers,Fleet,Logging,Redundancy). These are the wire/inter-actor messages — e.g.Messages/Admin/TestDriverConnect.cs(Test Connect request, see IDriverProbe) andMessages/Drivers/DriverHealthChanged.cs(the driver-health pub/sub payload, see IDriverHealthPublisher).Interfaces/— cluster-facing client contracts such asIAdminOperationsClient,IClusterRoleInfo, andIFleetDiagnosticsClient.Types/— strongly-typed identifier value types:CorrelationId,DeploymentId,ExecutionId,NodeId,RevisionHash.Browsing/— live-browse abstractions (BrowseNode,IBrowseSession,IDriverBrowser) backing the AdminUI address pickers.Engines/— evaluator seams (IScriptedAlarmEvaluator,IVirtualTagEvaluator,IAlarmActorStateStore) consumed by the VirtualTags / ScriptedAlarms engines.OpcUa/— deferred-publish seams (IOpcUaAddressSpaceSink,IServiceLevelPublisherand theirDeferred*no-op stand-ins) so address-space and ServiceLevel writes can be wired late.Observability/—OtOpcUaTelemetry(the shared ActivitySource/metrics surface).
See also
- ReadWriteOperations.md — the driver capability interfaces (read/write/subscribe) and resilience pipeline.
- ServiceHosting.md — role gating, the Akka cluster, and the external HistorianGateway backend.
- AlarmHistorian.md — the store-and-forward SQLite alarm
sink that drains to
IAlarmHistorianWriter. - Redundancy.md — driver stability tiers in the redundancy context.