4769044f88
Closes the two follow-ups left open by the MxGateway 0.2.0 bump. Reading the
gateway's own contract first changed what each of them is, and one of my earlier
claims was simply wrong -- both corrections are recorded below.
Replay gap
----------
A sentinel MxEvent carries family = UNSPECIFIED, so it fell through EventPump's
family switch into the default drop: we silently ignored a documented MUST ("you
missed events -- discard local state and re-snapshot"). It is now intercepted and
answered by re-issuing SubscribeBulk for every tracked subscription, since
MXAccess delivers an initial value per item on subscribe -- re-subscribing IS the
re-snapshot. It does not restart the event pump (the stream carrying the sentinel
is healthy, and the handler runs on that stream's own read loop, so disposing the
pump from there would deadlock on itself), it does not block the read loop, and
re-entry is gated so a burst collapses to one pass instead of a subscribe storm
per sentinel.
Handled on the READ loop, not in the dispatch loop, because everything past that
point is subject to newest-dropped channel pressure -- and "you missed events" is
the one signal whose loss is unrecoverable. Pinned by a test that fills the
channel and still expects the sentinel.
⚠️ This cannot fire in the shipped configuration, and I did not pretend otherwise
or paper over it by enabling resume. The gateway emits the sentinel only for a
stream opened with after_worker_sequence > 0; GatewayGalaxySubscriber passes 0,
which the gateway treats as "fresh stream, not a resume -- plain attach, no
replay, no sentinel". Enabling resume is a larger and genuinely delicate change:
the sequence is only meaningful within ONE gateway session, our reconnect
recreates the session, and replaying a remembered sequence against a fresh
session sets the server's live watermark ABOVE every event it is about to send --
silencing the stream instead of recovering it. That deserves its own change with
a live gate, which I cannot run from here.
Sparse arrays -- correcting the last commit message
---------------------------------------------------
I wrote that the bump "makes handling it possible" on the decode path. That is
wrong: MxSparseArray is documented write-only, the gateway expands it before
forwarding, and "the worker never receives or produces it". So the decode path
is unreachable by contract and the write path has no use for it either -- it is a
wire compaction for a WHOLE-array write, not a partial update (unmentioned
indices are RESET, deliberately, because the gateway cannot read-modify-write
without racing the provider). It is therefore the wrong primitive for an OPC UA
IndexRange write, which expects untouched elements preserved -- and WriteRequest
carries no IndexRange in the first place.
It is decoded anyway, as contract defence: the alternative if a future gateway
ever emits one is a silent null published under Good quality. Fill semantics
mirror the gateway's own SparseArrayExpander exactly (epoch for Time, empty
string, Integer widening to long[] when any element is int64) so a value that
round-trips through a write reads back as the same CLR array.
Also: the decoder is now total by contract, since it runs inside the dispatch
loop where an escaping exception kills dispatch for the life of the pump. A
malformed payload degrades instead of throwing.
A real defect found in the same code
------------------------------------
A stream the GATEWAY completes -- a closed or evicted session -- ends the RPC
without an error, so nothing threw, the supervisor was never notified, and the
pump exited silently. GetHealth() kept reporting Healthy, which also means the
#527 comms-loss watchdog could never trip, because it requires a non-Healthy
state. The driver went permanently deaf with no signal anywhere. This one is
live, unlike the two above. Now reported as a fault; the falsification control is
the existing CleanShutdown_DoesNotInvokeOnStreamFault, since our own disposal
also ends the stream and must NOT be reported.
Value kinds we cannot represent are now counted and warned once rather than
published as an indistinguishable null. The status is deliberately NOT downgraded
to Bad: nothing can reach that branch today, and changing what an OPC UA client
sees for a live shape is not a change to make without a gate. The counter exists
so the next person has evidence instead of a guess.
Verified: 330 Galaxy tests green (12 new), plus Galaxy.Browser 21, Runtime 558,
Core 351; full solution builds 0 errors with the warning census unchanged. Every
new test was falsified first -- disabling the gap interception reds 3 of 4 (the
throwing-callback test passes vacuously when the callback never runs, by
construction), disabling sparse decode plus the representability signal reds all
7 decoder tests, and both were restored and re-verified green.
Docs: the Galaxy.Performance invariant was stated as received = dispatched +
dropped + in-flight, which this breaks -- a sentinel is received but never enters
the channel -- so it now reads + replay_gaps. Galaxy.md gains the three
operational notes; DriverLifecycle.md records that "reaches a non-Healthy state"
is an obligation a driver can silently fail to meet, which is the general shape
of the clean-end bug.
441 lines
26 KiB
Markdown
441 lines
26 KiB
Markdown
# Driver Lifecycle & Server Infrastructure Contracts
|
|
|
|
Reference for the server-side infrastructure interfaces that surround a
|
|
driver but are **not** driver *capabilities* (read/write/subscribe/etc.,
|
|
documented in [ReadWriteOperations.md](ReadWriteOperations.md) and the
|
|
per-driver pages). These contracts live in
|
|
[`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/)
|
|
so they carry no behavior — concrete implementations live in the driver
|
|
projects, the Runtime, and the ControlPlane. Each subsection below gives the
|
|
purpose, the key members, and where it is implemented/used.
|
|
|
|
The capability interfaces a driver opts into (`IReadable`, `IWritable`,
|
|
`ITagDiscovery`, `ISubscribable`, `IAlarmSource`, `IHistoryProvider`,
|
|
`IHostConnectivityProbe`, `IPerCallHostResolver`, `IRediscoverable`) are
|
|
covered elsewhere and discovered by the server via `is`-checks on the
|
|
`IDriver` instance. The interfaces here are the *plumbing* the server uses to
|
|
**create**, **probe**, **supervise**, **report on**, and **configure** those
|
|
drivers, plus the server-side historian read surface.
|
|
|
|
---
|
|
|
|
## IDriverFactory — creating drivers from config rows
|
|
|
|
[`Core.Abstractions/IDriverFactory.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverFactory.cs)
|
|
|
|
Abstraction over the process-wide driver registry. The Runtime consumes this
|
|
instead of the concrete registry so the Runtime project does not pull in
|
|
`ZB.MOM.WW.OtOpcUa.Core` (which would drag in Polly + driver hosting).
|
|
|
|
Members:
|
|
|
|
- `IDriver? TryCreate(string driverType, string driverInstanceId, string driverConfigJson)`
|
|
— returns a new driver for the given type, or `null` when no factory is
|
|
registered for that type (missing assembly, typo). The `DriverHostActor`
|
|
logs and skips the row rather than failing the whole apply.
|
|
- `IReadOnlyCollection<string> SupportedTypes` — driver-type names this
|
|
factory can materialise; mostly for diagnostics and logs.
|
|
|
|
Implementations:
|
|
|
|
- `NullDriverFactory` (same file) returns `null` from every `TryCreate` and
|
|
exposes zero supported types. Bound when no concrete driver assemblies have
|
|
been registered (Mac dev path, smoke tests); the deployment becomes a no-op.
|
|
- `DriverFactoryRegistry`
|
|
([`Core/Hosting/DriverFactoryRegistry.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core/Hosting/DriverFactoryRegistry.cs))
|
|
is the real process-singleton registry keyed by `DriverInstance.DriverType`
|
|
(case-insensitive). Each driver project ships a `Register(...)` extension;
|
|
`Register` records the factory **and** the driver's stability
|
|
[`DriverTier`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverTier.cs)
|
|
(defaults to Tier A). Registering the same type twice throws.
|
|
- `DriverFactoryRegistryAdapter`
|
|
([`Core/Hosting/DriverFactoryRegistryAdapter.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core/Hosting/DriverFactoryRegistryAdapter.cs))
|
|
bridges the registry to the `IDriverFactory` abstraction.
|
|
|
|
Wiring: `DriverFactoryBootstrap.AddOtOpcUaDriverFactories`
|
|
([`Host/Drivers/DriverFactoryBootstrap.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs))
|
|
registers the singleton registry, runs every driver assembly's `Register`
|
|
extension, then binds `IDriverFactory` to the adapter. It must run **before**
|
|
`AddAkka` so the Runtime can resolve `IDriverFactory` when spawning the
|
|
`DriverHostActor`
|
|
([`Runtime/Drivers/DriverHostActor.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs)).
|
|
The registry is skipped on admin-only nodes (they never run drivers); the
|
|
probe set is the exception — see [IDriverProbe](#idriverprobe--test-connect).
|
|
|
|
---
|
|
|
|
## IDriverProbe — Test Connect
|
|
|
|
[`Core.Abstractions/IDriverProbe.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverProbe.cs)
|
|
|
|
A cheap test-connect probe for one driver type, backing the AdminUI **Test
|
|
Connect** button. An implementation deserializes a driver-config JSON, attempts
|
|
a cheap connection (TCP open, OPC UA session, gRPC ping — whatever the driver's
|
|
native protocol supports), and reports success/failure with latency. **Probes
|
|
must not mutate persistent state**: the AdminUI invokes them against the
|
|
transient config in the typed form, not against the persisted `DriverInstance`
|
|
row.
|
|
|
|
Members:
|
|
|
|
- `string DriverType { get; }` — the `DriverInstance.DriverType` string this
|
|
probe handles; used for DI lookup.
|
|
- `Task<DriverProbeResult> ProbeAsync(string configJson, TimeSpan timeout, CancellationToken ct)`
|
|
— never throws on connection failure; returns a result with `Ok = false`
|
|
and a message instead.
|
|
- `DriverProbeResult(bool Ok, string? Message, TimeSpan? Latency)` — outcome
|
|
record (`Message` is `null` on success; `Latency` is `null` on failure).
|
|
|
|
Implementations: every driver ships a `*DriverProbe` in its driver project
|
|
(e.g.
|
|
[`Driver.Modbus/ModbusDriverProbe.cs`](../src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriverProbe.cs)
|
|
does a bare socket open/close). The historian backend is the external
|
|
HistorianGateway (consumed as a gRPC client package, not an `IDriver`), so it
|
|
has no driver probe.
|
|
|
|
Flow: the AdminUI's `AdminProbeService`
|
|
([`AdminUI/Clients/AdminProbeService.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Clients/AdminProbeService.cs))
|
|
dispatches a `TestDriverConnect` message through `IAdminOperationsClient` to the
|
|
cluster-singleton `AdminOperationsActor`
|
|
([`ControlPlane/AdminOperations/AdminOperationsActor.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs)),
|
|
which holds the probes keyed by `DriverType` and invokes the matching one
|
|
(timeout clamped to `[1, 60]` seconds). Because the admin singleton is
|
|
admin-pinned, the probe set must be registered on admin nodes too — `Program.cs`
|
|
calls `AddOtOpcUaDriverProbes` in the `hasAdmin` block, and
|
|
`AddOtOpcUaDriverFactories` registers it for fused admin+driver nodes.
|
|
|
|
---
|
|
|
|
## IDriverSupervisor / process recycle — REMOVED (Gitea #522)
|
|
|
|
`IDriverSupervisor`, `MemoryRecycle`, `MemoryTracking` and
|
|
`ScheduledRecycleScheduler` **no longer exist.** This section used to describe
|
|
them as the Tier C protection layer.
|
|
|
|
They were removed because the thing they acted on is gone. "Tier C" meant a
|
|
driver running **out-of-process behind a supervisor that could restart its Host**
|
|
without tearing down the OPC UA session or co-hosted drivers. No such process
|
|
remains anywhere: Galaxy reaches MXAccess over gRPC to the external
|
|
`mxaccessgw` sidecar (PR 7.2 retired the in-process `Galaxy.Host` / `Proxy` /
|
|
`Shared` projects), and FOCAS has run in-process since its managed wire client
|
|
landed 2026-04-24. Consistent with that, `IDriverSupervisor` had **zero
|
|
implementations**, and `MemoryTracking` / `MemoryRecycle` /
|
|
`ScheduledRecycleScheduler` were constructed **only in their own unit tests** —
|
|
so this was not dormant-but-ready code, it was code with no path to running.
|
|
|
|
The operator-facing half went too: `RecycleIntervalSeconds` was authorable
|
|
through the AdminUI's driver resilience section and validated by the parser
|
|
while configuring nothing. A stored `ResilienceConfig` still carrying the key
|
|
parses cleanly (unknown keys are ignored) and the AdminUI preserves it rather
|
|
than silently rewriting an operator's config.
|
|
|
|
**`DriverTier` itself survives and is load-bearing** — do not delete it while
|
|
following this note. `DriverResilienceOptions.GetTierDefaults(tier)` supplies
|
|
the real per-capability timeout / retry / breaker policies, resolved via
|
|
`DriverFactoryRegistry.GetTier` and applied by `DriverCapabilityInvokerFactory`.
|
|
Every driver runs Tier A because no factory passes a tier, so today that means
|
|
one uniform policy set.
|
|
|
|
`IDriver` **no longer carries `GetMemoryFootprint()` or
|
|
`FlushOptionalCachesAsync()`** (Gitea **#525**). They existed to feed
|
|
`MemoryTracking`, deleted with the rest of the driver-tier recycle machinery in
|
|
**#522**; after that nothing called them, and their doc-comments still claimed
|
|
Core polled the footprint every 30 s, which had not been true since v3. The
|
|
sweep removed them from the interface, all 12 drivers, and every test stub,
|
|
along with the support code that existed only to feed them — MTConnect's
|
|
per-entry byte weights and its cached data-item count, OpcUaClient's
|
|
`_discoveredNodeCount`, and MQTT's per-tag byte estimate.
|
|
|
|
Two consequences worth knowing:
|
|
|
|
- `AbCipTemplateCache` now has **no production reset path**. Its only caller was
|
|
AB CIP's flush — which itself had no caller, so nothing has actually changed
|
|
in behaviour, only in how visible the fact is. A UDT redefined on a live
|
|
controller keeps its stale shape until the driver instance is rebuilt.
|
|
- MTConnect's `/probe` cache is still dropped, by `InvalidateProbeModel()` on an
|
|
Agent restart. The tests that used to reach it through the flush now drive
|
|
that path instead, so they cover a case that actually occurs.
|
|
|
|
If per-driver memory protection is wanted again, build it for the architecture
|
|
as it is now: a process-wide watchdog that can ask drivers to flush, not a
|
|
per-driver tier whose only remedy requires a supervisor that cannot exist
|
|
in-process. The deleted per-driver implementations are in git history and were
|
|
mostly reusable — what was always missing is the trigger.
|
|
|
|
---
|
|
|
|
## IDriverHealthPublisher — health pub/sub sink
|
|
|
|
[`Core.Abstractions/IDriverHealthPublisher.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverHealthPublisher.cs)
|
|
|
|
A sink for driver-health state-change notifications. Implementations must be
|
|
non-blocking and safe to call from any thread.
|
|
|
|
Member:
|
|
|
|
- `void Publish(string clusterId, string driverInstanceId, DriverHealth health, int errorCount5Min)`
|
|
|
|
Implementations:
|
|
|
|
- `NullDriverHealthPublisher` (same file) is the drop-in no-op for tests and
|
|
dev-stub paths. A `DriverInstanceActor` defaults to it when no publisher is
|
|
supplied.
|
|
- `AkkaDriverHealthPublisher`
|
|
([`Runtime/Drivers/AkkaDriverHealthPublisher.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/AkkaDriverHealthPublisher.cs))
|
|
is the production binding: it forwards each transition as a
|
|
`DriverHealthChanged` message onto the cluster-wide `driver-health`
|
|
Akka DistributedPubSub topic.
|
|
|
|
Producer: `DriverInstanceActor`
|
|
([`Runtime/Drivers/DriverInstanceActor.cs`](../src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs))
|
|
calls `Publish` when a driver's health transitions. The published snapshot is
|
|
consumed AdminUI-side and surfaced through the driver-status panel (read
|
|
in-process by the AdminUI bridge rather than dialing its own hub).
|
|
|
|
### Writing health around a teardown (Gitea #517)
|
|
|
|
Two rules, and the second is the counter-intuitive one:
|
|
|
|
1. **`GetHealth()` must report actual state, never a constant.** A hard-coded `Healthy` reports a
|
|
healthy driver before initialize and after shutdown alike. `CalculationDriver` did exactly this —
|
|
it was the fleet's one driver reporting Healthy with its timers stopped, deterministically rather
|
|
than as a race. A driver with no transport still has a running / not-running distinction.
|
|
|
|
2. **Land the terminal health value *after* the teardown — and, where the window is observable, write
|
|
it *before* as well.** #517's original suggestion was to move the write earlier so the race window
|
|
leans conservative. Moving it is wrong: teardown drains in-flight poll/probe loops, and those loops
|
|
write health as they unwind, so a driver that only wrote first could have a late loop overwrite the
|
|
shutdown state with `Healthy` and leave *that* as the resting value — with every connection closed.
|
|
The safe shape is a **bracket**: conservative value before, terminal value after.
|
|
|
|
Where the window is actually observable is worth knowing before adding brackets everywhere:
|
|
|
|
| Path | Caller | Can `GetHealth()` land mid-teardown? |
|
|
|---|---|---|
|
|
| `ShutdownAsync` | `DriverInstanceActor.PostStop` | **No** — actor thread, after the mailbox drains, so the health poll is serialised against it. |
|
|
| `InitializeAsync` failure | `DriverInstanceActor.InitializeAsync` | **Yes** — invoked on a fire-and-forget `Task.Run`, so the health-poll tick keeps calling `GetHealth()` throughout. |
|
|
|
|
So a driver whose `InitializeAsync` disposes anything on its failure path must bracket; only
|
|
`MTConnectDriver` does today, because it is the only one that tears down inside a `catch`. The
|
|
shutdown paths deliberately keep the single trailing write.
|
|
|
|
### The comms-loss watchdog (Gitea #527) — `GetHealth()` is now load-bearing
|
|
|
|
`GetHealth()` used to be **reporting-only**: `DriverInstanceActor.PublishHealthSnapshot` read it every 30 s
|
|
and forwarded it to the AdminUI, and nothing ever acted on a driver that reported itself unhealthy. That
|
|
left `DisconnectObserved` with **no production sender at all** — the only paths into `Reconnecting` were a
|
|
failed *initial* connect and an operator's Force Reconnect. A driver that connected and then lost its
|
|
device stayed `Connected` forever: publishing Bad for every tag, never re-initialising, and **not
|
|
recovering when the device came back** (live-confirmed against a stopped Modbus fixture, which needed a
|
|
node restart). Both comms-loss quality paths — #477's native-condition annotation and #481's
|
|
scripted-alarm input fan-out — hang off `ConnectivityChanged`, so both were wired to a signal that never
|
|
fired.
|
|
|
|
The `Connected`-state health tick now also evaluates a watchdog, and self-sends `DisconnectObserved` when
|
|
**both** of these hold:
|
|
|
|
1. the driver reports a state other than `Healthy`, **and**
|
|
2. `DriverHealth.LastSuccessfulRead` has not advanced for `DefaultCommsLossTimeout` (90 s = three ticks),
|
|
measured from the Connected entry when the driver has never reported a successful read.
|
|
|
|
**The conjunction is the whole design, and "non-Healthy for N polls" is the wrong rule.** `Degraded` means
|
|
*connected but some data is bad* — the enum says so. Several drivers publish health **per tag inside their
|
|
read loop** (`ModbusDriver.ReadAsync` writes `Healthy` or `Degraded` once per reference), so a single
|
|
unreadable address leaves the driver latched `Degraded` whenever it happens to be read last, indefinitely,
|
|
against a device that is perfectly reachable. A state-only watchdog would tear that driver down every 90 s
|
|
forever — strictly worse than the bug it fixes. `LastSuccessfulRead` is the honest liveness evidence: every
|
|
driver advances it on any successful read and *preserves* it across `Degraded`/`Faulted` writes, so one bad
|
|
tag among good ones keeps it moving while a device that is genuinely gone freezes it. The state check is
|
|
still needed as the second conjunct, because a `Healthy` driver with nothing subscribed legitimately never
|
|
reads and must not be declared dead for being idle. Both halves are pinned by their own test in
|
|
`DriverInstanceActorCommsLossWatchdogTests`, each of which goes red when the *other* conjunct is deleted.
|
|
|
|
Two consequences worth knowing:
|
|
|
|
- **`CalculationDriver` can never trip it, correctly.** It reports `Healthy` whenever it is running and has
|
|
no transport to be degraded about. Every other driver reaches a non-Healthy state on sustained failure:
|
|
the nine poll drivers via `Degraded`, MQTT via its live connection state, Galaxy via its reconnect
|
|
supervisor. This was surveyed across all 12 before relying on it — it is the assumption the whole
|
|
mechanism rests on.
|
|
- **A device that accepts a connection but serves no data will cycle**, reconnecting once per threshold.
|
|
That is the correct reading of a broken device, it is rate-limited, and it is visible in the health
|
|
snapshot's error count — but do not mistake it for a bug when reading logs.
|
|
- ⚠️ **"Reaches a non-Healthy state" is an obligation on each driver, and a driver can fail to meet it
|
|
without failing anything.** Galaxy's entry in the survey above is "via its reconnect supervisor" — which
|
|
holds only for the paths that actually *reach* the supervisor. Its event stream could also end the way a
|
|
closed or evicted gateway session ends it: the gRPC call completes **without an error**, so nothing
|
|
throws, the supervisor was never told, health stayed `Healthy` — and a `Healthy` driver can never satisfy
|
|
conjunct 1, so the watchdog could not fire either. The driver went permanently deaf with nothing
|
|
reporting it anywhere (fixed in `EventPump`; see [Galaxy.md](drivers/Galaxy.md) §Operational Notes).
|
|
The general shape is worth carrying to the next driver: this watchdog only covers failures a driver
|
|
**notices**, so a silent-success termination path — a stream that ends cleanly, a poll loop that exits
|
|
without throwing — bypasses it entirely.
|
|
|
|
**The reconnect retry now tears down before re-initialising.** `Reconnecting`'s `RetryConnect` calls
|
|
`ShutdownAsync` before `InitializeAsync` — the sequence every driver's own `ReinitializeAsync` already
|
|
uses — because several drivers **overwrite** an `IAsyncDisposable` transport field inside `InitializeAsync`
|
|
without disposing the previous one (`ModbusDriver._transport`, `MqttDriver._connection`). Re-initialising
|
|
without a teardown leaks one socket per attempt. That was latent while only a manual Force Reconnect
|
|
re-entered this path; the watchdog makes it automatic and unbounded, so a device down for an hour would
|
|
have leaked ~360 transports at the 10 s retry interval. The shutdown is best-effort: a driver that throws
|
|
on it still gets its initialise attempt.
|
|
|
|
### The connect path feeds the circuit breaker (R2-10 T10 closure)
|
|
|
|
`IDriver.InitializeAsync` is routed through the capability invoker as
|
|
**`DriverCapability.Initialize`**, keyed on the driver-instance id (init spans the whole driver,
|
|
like a bulk subscribe). Before this, the connect path was the ONE capability call the invoker never
|
|
saw: a driver that could not connect contributed **zero** breaker input, so the AdminUI resilience
|
|
panel read *"All targets healthy"* against a dead endpoint — live-proven during the R2-10 T10 gate,
|
|
and the reason that gate was structurally unsatisfiable for a year of attempts. Things to know
|
|
before touching it:
|
|
|
|
- **The Initialize policy never retries** — this actor's `RetryConnect` timer *is* the retry loop,
|
|
and pipeline retries would stack inside each timer tick (and stretch one attempt past the timer
|
|
interval so attempts overlap). The parser forces a `retryCount` override back to 0, same funnel
|
|
as the write-shaped rule.
|
|
- ⚠️ **The breaker threshold is 2, and raising it silently re-breaks the feature.** The breaker
|
|
samples over 30 s and the reconnect timer fires every 10 s, so at most **three** init attempts fit
|
|
one sampling window — a Read-sized threshold of 5 can literally never open on connect failures,
|
|
which reads exactly like "the panel doesn't work" with nothing logged. Pinned by
|
|
`Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence` and, behaviourally, by
|
|
`Initialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt`.
|
|
- **An open breaker cannot wedge the reconnect loop.** A fast-failed attempt
|
|
(`BrokenCircuitException`) lands in the same catch as a real connect failure →
|
|
`InitializeFailed` → the timer keeps ticking. The 15 s break window is shorter than two 10 s
|
|
ticks, so recovery costs at most one extra tick. The teardown `ShutdownAsync` stays deliberately
|
|
UNWRAPPED — best-effort cleanup of an already-dead transport must not count toward the breaker.
|
|
- **A respawn resets the instance's tracker snapshots** (`DriverResilienceStatusTracker.Invalidate`,
|
|
called by `DriverCapabilityInvokerFactory.Create` alongside the pipeline invalidation). The
|
|
pipelines carry the actual breaker state; once they are discarded, a retained "Breaker OPEN"
|
|
snapshot could never self-correct — only the discarded pipeline's `OnClosed` stamps the close —
|
|
so a deploy during an outage would otherwise strand a false OPEN chip forever. This was
|
|
unreachable while breakers never opened; the Initialize wrap made it reachable.
|
|
- `ConsecutiveFailures` stays 0 during a connect outage **by design** — the tracker counts retry
|
|
attempts, and Initialize has none. The breaker-OPEN chip is the operator signal.
|
|
|
|
---
|
|
|
|
## IDriverConfigEditor — custom AdminUI config editor (plug-point)
|
|
|
|
[`Core.Abstractions/IDriverConfigEditor.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/IDriverConfigEditor.cs)
|
|
|
|
An **optional** plug-point a driver can implement to provide a custom AdminUI
|
|
editor for its `DriverConfig` JSON. Drivers that don't implement it fall back to
|
|
the generic JSON editor with schema-driven validation. This is the contract
|
|
between the driver and the Admin Blazor app; the Admin app discovers
|
|
implementations and slots them into the Driver Detail screen.
|
|
|
|
Members:
|
|
|
|
- `string DriverType { get; }` — the driver type this editor handles.
|
|
- `Type EditorComponentType { get; }` — the Razor component type that renders
|
|
the editor (returned as `Type` so `Core.Abstractions` needs no Blazor
|
|
reference).
|
|
|
|
Status: this is a forward-looking plug-point. No driver ships a concrete
|
|
`IDriverConfigEditor` today — every driver uses the generic JSON editor — so
|
|
the interface currently has the contract defined but no implementations.
|
|
|
|
---
|
|
|
|
## IHistorianDataSource — server-side historian read surface
|
|
|
|
[`Core.Abstractions/Historian/IHistorianDataSource.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs)
|
|
|
|
The server-side historian read surface. Registered with the server's history
|
|
router and resolved **per OPC UA namespace**, independent of any driver's
|
|
lifecycle. This is distinct from the driver capability `IHistoryProvider`:
|
|
|
|
- `IHistoryProvider` is a *driver capability* — the server dispatches to it via
|
|
the driver instance.
|
|
- `IHistorianDataSource` is a *server registration* — the server resolves it by
|
|
namespace and calls it directly, so one historian (the HistorianGateway) can
|
|
serve many drivers' nodes, and drivers can restart without dropping history
|
|
availability.
|
|
|
|
The interface is `: IDisposable` and declares the full read surface as
|
|
**required** members (unlike `IHistoryProvider`, where at-time/event reads are
|
|
optional default-impl methods so legacy drivers can stay raw-only):
|
|
|
|
- `ReadRawAsync(fullReference, startUtc, endUtc, maxValuesPerNode, ct)` — raw
|
|
historical samples over a time range.
|
|
- `ReadProcessedAsync(fullReference, startUtc, endUtc, interval, aggregate, ct)`
|
|
— interval-bucketed aggregates (average/min/max/count); an empty bucket
|
|
returns a `BadNoData` sample.
|
|
- `ReadAtTimeAsync(fullReference, timestampsUtc, ct)` — one sample per requested
|
|
timestamp (OPC UA HistoryReadAtTime); the returned list matches the requested
|
|
length and order, gaps as Bad-quality snapshots.
|
|
- `ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, ct)` — historical
|
|
alarm/event records (OPC UA HistoryReadEvents); `sourceName` is `null` to
|
|
return all sources. `maxEvents` is a signed `int` so a non-positive value is a
|
|
"use the backend's default cap" sentinel.
|
|
- `GetHealthSnapshot()` — point-in-time health snapshot for diagnostics and
|
|
dashboards; pure observation, never blocks on backend I/O.
|
|
|
|
All values use the shared `DataValueSnapshot` / `HistoricalEvent` shapes;
|
|
backend-specific quality/type encodings are translated to OPC UA `StatusCode`
|
|
uints inside the data source.
|
|
|
|
Implementation:
|
|
|
|
- `GatewayHistorianDataSource`
|
|
([`Driver.Historian.Gateway/GatewayHistorianDataSource.cs`](../src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianDataSource.cs))
|
|
— the read backend that talks gRPC to the external `ZB.MOM.WW.HistorianGateway`
|
|
(via the `ZB.MOM.WW.HistorianGateway.Client` package, behind the
|
|
`IHistorianGatewayClient` seam). The alarm-event drain target is the separate
|
|
`GatewayAlarmHistorianWriter` (the gateway `SendEvent` path; see
|
|
[AlarmHistorian.md](AlarmHistorian.md)).
|
|
|
|
The HistorianGateway is the sole historian backend; its config keys and
|
|
deployment prerequisites are in [Historian.md](Historian.md).
|
|
|
|
---
|
|
|
|
## Commons — shared cross-cutting primitives
|
|
|
|
[`src/Core/ZB.MOM.WW.OtOpcUa.Commons/`](../src/Core/ZB.MOM.WW.OtOpcUa.Commons/)
|
|
|
|
`ZB.MOM.WW.OtOpcUa.Commons` is the low-level shared library that the Runtime,
|
|
ControlPlane, AdminUI, and OPC UA server projects all reference. It holds
|
|
cross-cutting primitives with no driver- or host-specific behavior, so the
|
|
heavier projects can share message contracts and value types without taking a
|
|
dependency on each other. It references only `Akka` and the internal
|
|
`ZB.MOM.WW.Audit` package.
|
|
|
|
Folders:
|
|
|
|
- **`Messages/`** — Akka message contracts grouped by concern (`Admin`,
|
|
`Alerts`, `Deploy`, `Drivers`, `Fleet`, `Logging`, `Redundancy`). These are
|
|
the wire/inter-actor messages — e.g. `Messages/Admin/TestDriverConnect.cs`
|
|
(Test Connect request, see [IDriverProbe](#idriverprobe--test-connect)) and
|
|
`Messages/Drivers/DriverHealthChanged.cs` (the driver-health pub/sub payload,
|
|
see [IDriverHealthPublisher](#idriverhealthpublisher--health-pubsub-sink)).
|
|
- **`Interfaces/`** — cluster-facing client contracts such as
|
|
`IAdminOperationsClient`, `IClusterRoleInfo`, and `IFleetDiagnosticsClient`.
|
|
- **`Types/`** — strongly-typed identifier value types: `CorrelationId`,
|
|
`DeploymentId`, `ExecutionId`, `NodeId`, `RevisionHash`.
|
|
- **`Browsing/`** — live-browse abstractions (`BrowseNode`, `IBrowseSession`,
|
|
`IDriverBrowser`) backing the AdminUI address pickers.
|
|
- **`Engines/`** — evaluator seams (`IScriptedAlarmEvaluator`,
|
|
`IVirtualTagEvaluator`, `IAlarmActorStateStore`) consumed by the
|
|
[VirtualTags](VirtualTags.md) / [ScriptedAlarms](ScriptedAlarms.md) engines.
|
|
- **`OpcUa/`** — deferred-publish seams (`IOpcUaAddressSpaceSink`,
|
|
`IServiceLevelPublisher` and their `Deferred*` no-op stand-ins) so address-space
|
|
and [ServiceLevel](Redundancy.md) writes can be wired late.
|
|
- **`Observability/`** — `OtOpcUaTelemetry` (the shared ActivitySource/metrics
|
|
surface).
|
|
|
|
---
|
|
|
|
## See also
|
|
|
|
- [ReadWriteOperations.md](ReadWriteOperations.md) — the driver *capability*
|
|
interfaces (read/write/subscribe) and resilience pipeline.
|
|
- [ServiceHosting.md](ServiceHosting.md) — role gating, the Akka cluster, and
|
|
the external HistorianGateway backend.
|
|
- [AlarmHistorian.md](AlarmHistorian.md) — the store-and-forward SQLite alarm
|
|
sink that drains to `IAlarmHistorianWriter`.
|
|
- [Redundancy.md](Redundancy.md) — driver stability tiers in the redundancy
|
|
context.
|