Joseph Doherty
4769044f88
Closes the two follow-ups left open by the MxGateway 0.2.0 bump. Reading the
gateway's own contract first changed what each of them is, and one of my earlier
claims was simply wrong -- both corrections are recorded below.
Replay gap
----------
A sentinel MxEvent carries family = UNSPECIFIED, so it fell through EventPump's
family switch into the default drop: we silently ignored a documented MUST ("you
missed events -- discard local state and re-snapshot"). It is now intercepted and
answered by re-issuing SubscribeBulk for every tracked subscription, since
MXAccess delivers an initial value per item on subscribe -- re-subscribing IS the
re-snapshot. It does not restart the event pump (the stream carrying the sentinel
is healthy, and the handler runs on that stream's own read loop, so disposing the
pump from there would deadlock on itself), it does not block the read loop, and
re-entry is gated so a burst collapses to one pass instead of a subscribe storm
per sentinel.
Handled on the READ loop, not in the dispatch loop, because everything past that
point is subject to newest-dropped channel pressure -- and "you missed events" is
the one signal whose loss is unrecoverable. Pinned by a test that fills the
channel and still expects the sentinel.
⚠️ This cannot fire in the shipped configuration, and I did not pretend otherwise
or paper over it by enabling resume. The gateway emits the sentinel only for a
stream opened with after_worker_sequence > 0; GatewayGalaxySubscriber passes 0,
which the gateway treats as "fresh stream, not a resume -- plain attach, no
replay, no sentinel". Enabling resume is a larger and genuinely delicate change:
the sequence is only meaningful within ONE gateway session, our reconnect
recreates the session, and replaying a remembered sequence against a fresh
session sets the server's live watermark ABOVE every event it is about to send --
silencing the stream instead of recovering it. That deserves its own change with
a live gate, which I cannot run from here.
Sparse arrays -- correcting the last commit message
---------------------------------------------------
I wrote that the bump "makes handling it possible" on the decode path. That is
wrong: MxSparseArray is documented write-only, the gateway expands it before
forwarding, and "the worker never receives or produces it". So the decode path
is unreachable by contract and the write path has no use for it either -- it is a
wire compaction for a WHOLE-array write, not a partial update (unmentioned
indices are RESET, deliberately, because the gateway cannot read-modify-write
without racing the provider). It is therefore the wrong primitive for an OPC UA
IndexRange write, which expects untouched elements preserved -- and WriteRequest
carries no IndexRange in the first place.
It is decoded anyway, as contract defence: the alternative if a future gateway
ever emits one is a silent null published under Good quality. Fill semantics
mirror the gateway's own SparseArrayExpander exactly (epoch for Time, empty
string, Integer widening to long[] when any element is int64) so a value that
round-trips through a write reads back as the same CLR array.
Also: the decoder is now total by contract, since it runs inside the dispatch
loop where an escaping exception kills dispatch for the life of the pump. A
malformed payload degrades instead of throwing.
A real defect found in the same code
------------------------------------
A stream the GATEWAY completes -- a closed or evicted session -- ends the RPC
without an error, so nothing threw, the supervisor was never notified, and the
pump exited silently. GetHealth() kept reporting Healthy, which also means the
#527 comms-loss watchdog could never trip, because it requires a non-Healthy
state. The driver went permanently deaf with no signal anywhere. This one is
live, unlike the two above. Now reported as a fault; the falsification control is
the existing CleanShutdown_DoesNotInvokeOnStreamFault, since our own disposal
also ends the stream and must NOT be reported.
Value kinds we cannot represent are now counted and warned once rather than
published as an indistinguishable null. The status is deliberately NOT downgraded
to Bad: nothing can reach that branch today, and changing what an OPC UA client
sees for a live shape is not a change to make without a gate. The counter exists
so the next person has evidence instead of a guess.
Verified: 330 Galaxy tests green (12 new), plus Galaxy.Browser 21, Runtime 558,
Core 351; full solution builds 0 errors with the warning census unchanged. Every
new test was falsified first -- disabling the gap interception reds 3 of 4 (the
throwing-callback test passes vacuously when the callback never runs, by
construction), disabling sparse decode plus the representability signal reds all
7 decoder tests, and both were restored and re-verified green.
Docs: the Galaxy.Performance invariant was stated as received = dispatched +
dropped + in-flight, which this breaks -- a sentinel is received but never enters
the channel -- so it now reads + replay_gaps. Galaxy.md gains the three
operational notes; DriverLifecycle.md records that "reaches a non-Healthy state"
is an obligation a driver can silently fail to meet, which is the general shape
of the clean-end bug.
116 lines
13 KiB
Markdown
116 lines
13 KiB
Markdown
# Galaxy Driver
|
|
|
|
The Galaxy driver bridges OtOpcUa to AVEVA System Platform (Wonderware) Galaxies. It is a **Tier-A in-process driver** that runs in the OtOpcUa server's .NET 10 AnyCPU process and speaks gRPC to a separately installed `mxaccessgw` server (sibling repo at `c:\Users\dohertj2\Desktop\mxaccessgw\`). The gateway owns the MXAccess COM apartment, the STA + Win32 message pump, the Galaxy Repository SQL reader, and the Historian SDK — all the bits that need x86 / .NET Framework 4.8 / COM interop. The driver itself is platform-agnostic and contains no COM, no STA thread, and no x86 bitness constraint.
|
|
|
|
For the driver spec (capability surface, config shape, addressing), see [docs/v2/driver-specs.md §1](../v2/driver-specs.md). For the gateway setup recipe, see [docs/v2/Galaxy.ParityRig.md](../v2/Galaxy.ParityRig.md). For tracing, metrics, and soak profile, see [docs/v2/Galaxy.Performance.md](../v2/Galaxy.Performance.md).
|
|
|
|
> **Note**: the related docs [`Galaxy-Repository.md`](../v1/drivers/Galaxy-Repository.md) and [`Galaxy-Test-Fixture.md`](../v1/drivers/Galaxy-Test-Fixture.md) describe the previous v1 / out-of-process topology and now live under `docs/v1/drivers/`. For current testing use [`Galaxy.ParityRig.md`](../v2/Galaxy.ParityRig.md) and the `mxaccessgw` repo.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
+---------------------------------------+
|
|
| OtOpcUa.Server (.NET 10 AnyCPU) |
|
|
| GalaxyDriver (in-process) |
|
|
| ITagDiscovery / IReadable / |
|
|
| IWritable / ISubscribable / |
|
|
| IRediscoverable / |
|
|
| IHostConnectivityProbe / |
|
|
| IAlarmSource |
|
|
+-------------------+-------------------+
|
|
|
|
|
gRPC (default http://localhost:5120)
|
|
|
|
|
v
|
|
+---------------------------------------+
|
|
| mxaccessgw (sibling repo) |
|
|
| +-------------------------------+ |
|
|
| | MxGateway.Worker (x86 net48) | |
|
|
| | STA + WM_APP pump | |
|
|
| | ArchestrA.MxAccess COM | |
|
|
| | Galaxy Repository SQL | |
|
|
| | Wonderware Historian SDK | |
|
|
| +-------------------------------+ |
|
|
+---------------------------------------+
|
|
```
|
|
|
|
History reads moved server-side in PR 7.2 (`IHistoryRouter`). Galaxy no longer implements `IHistoryProvider` of its own.
|
|
|
|
`IAlarmSource` was retired with PR 7.2 and **restored in PR B.2** of the
|
|
alarms-over-gateway epic ([docs/plans/alarms-over-gateway.md](../plans/alarms-over-gateway.md)).
|
|
Alarm transitions arrive on the same gateway `StreamEvents` channel as
|
|
data-change events under the new `MX_EVENT_FAMILY_ON_ALARM_TRANSITION`
|
|
family; acknowledgements route through the gateway's
|
|
`AcknowledgeAlarm` RPC. The previous value-driven sub-attribute path
|
|
remains as a fallback for Galaxy templates without `$Alarm*`
|
|
extensions — the server-side `AlarmConditionService` dedups when both
|
|
paths fire on the same condition. See [docs/AlarmTracking.md](../AlarmTracking.md)
|
|
for the v2-final architecture.
|
|
|
|
## Project Layout
|
|
|
|
The driver ships as a single project: `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/` (.NET 10, AnyCPU). Sub-folders:
|
|
|
|
| Folder | Role |
|
|
|--------|------|
|
|
| `Browse/` | Static-side discovery: `GalaxyDiscoverer` walks the gateway's hierarchy + attribute-set RPCs, `DataTypeMap` and `SecurityMap` translate Galaxy types and security classifications into OPC UA equivalents, `AlarmRefBuilder` extracts alarm-bearing attribute references for the server-layer alarm engine. `IGalaxyHierarchySource` + `GatewayGalaxyHierarchySource` + `TracedGalaxyHierarchySource` decorate the gateway browse RPC; `IGalaxyDeployWatchSource` + `GatewayGalaxyDeployWatchSource` + `DeployWatcher` drive `IRediscoverable`. |
|
|
| `Runtime/` | Live data path: `EventPump` runs the gateway's `StreamEvents` RPC and fans out to subscribers via a bounded channel; `GalaxyMxSession` is the read-side handle; `GatewayGalaxySubscriber` + `GatewayGalaxyDataWriter` (each with a `Traced*` decorator) implement `ISubscribable` / `IWritable`; `SubscriptionRegistry` tracks subscription state for replay; `ReconnectSupervisor` owns the backoff loop and triggers `ReplaySubscriptions` on session loss; `StatusCodeMap` translates gateway StatusCodes to OPC UA; `MxValueDecoder` / `MxValueEncoder` handle scalar + array marshalling; `GalaxyTelemetry` + `GalaxySubscriptionHandle` round out the surface. |
|
|
| `Health/` | `HostStatusAggregator` rolls per-platform probe state into the driver's `IHostConnectivityProbe` view; `PerPlatformProbeWatcher` listens on the gateway's per-host status stream; `HostConnectivityForwarder` pushes transitions out to the server's connectivity bus. |
|
|
| `Config/` | `GalaxyDriverOptions` and the four nested option records (`GalaxyGatewayOptions`, `GalaxyMxAccessOptions`, `GalaxyRepositoryOptions`, `GalaxyReconnectOptions`). |
|
|
|
|
Project root files:
|
|
|
|
- `GalaxyDriver.cs` — `IDriver` + capability-interface implementation; composes the Browse / Runtime / Health collaborators.
|
|
- `GalaxyDriverFactoryExtensions.cs` — DI registration helper used by the server's driver bootstrap.
|
|
|
|
## Capability Surface
|
|
|
|
`GalaxyDriver : IDriver, ITagDiscovery, IReadable, IWritable, ISubscribable, IRediscoverable, IHostConnectivityProbe, IAlarmSource, IDisposable, IAsyncDisposable`.
|
|
|
|
| Capability | Implementation entry point |
|
|
|------------|---------------------------|
|
|
| `ITagDiscovery` | `Browse/GalaxyDiscoverer.cs` |
|
|
| `IRediscoverable` | `Browse/DeployWatcher.cs` — consumed as an **operator prompt**: a Galaxy redeploy raises a "re-browse" chip on `/hosts`. ⚠️ It does **not** rebuild the address space — re-browse the device under `/raw` and commit ([#518](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/518)) |
|
|
| `IReadable` | `Runtime/GalaxyMxSession.cs` |
|
|
| `IWritable` | `Runtime/GatewayGalaxyDataWriter.cs` |
|
|
| `ISubscribable` | `Runtime/GatewayGalaxySubscriber.cs` (driven by `EventPump`) |
|
|
| `IHostConnectivityProbe` | `Health/HostStatusAggregator.cs` |
|
|
| `IAlarmSource` | `Runtime/GatewayGalaxyAlarmFeed.cs` (transitions) + `Runtime/GatewayGalaxyAlarmAcknowledger.cs` (acks) |
|
|
|
|
## Configuration
|
|
|
|
`DriverConfig` JSON binds to `Config/GalaxyDriverOptions.cs`. The four sections are:
|
|
|
|
- **`Gateway`** — endpoint, API key secret ref, TLS knobs, connect/call/stream timeouts. `StreamTimeoutSeconds = 0` keeps the long-lived `StreamEvents` RPC open for the driver's lifetime.
|
|
- **`MxAccess`** — `ClientName` (must be unique per OtOpcUa instance — redundancy pairs enforce uniqueness at install time), `PublishingIntervalMs` (forwarded as `buffered_update_interval_ms` on subscribe), `WriteUserId` for ArchestrA secured-write, `EventPumpChannelCapacity` (default 50_000 — one second of headroom at 50k tags / 1Hz; tune via the `galaxy.events.dropped` metric).
|
|
- **`Repository`** — `DiscoverPageSize`, `WatchDeployEvents`.
|
|
- **`Reconnect`** — `InitialBackoffMs`, `MaxBackoffMs`, `ReplayOnSessionLost` (calls the gateway's `ReplaySubscriptions` RPC after reconnect rather than re-issuing subscribe-bulk for every tag).
|
|
|
|
Full per-field descriptions live in `Config/GalaxyDriverOptions.cs`. The full JSON skeleton is reproduced in [docs/v2/driver-specs.md §1](../v2/driver-specs.md).
|
|
|
|
## Reconnect + Replay
|
|
|
|
`ReconnectSupervisor` owns an exponential-backoff loop bounded by `Reconnect.InitialBackoffMs` / `MaxBackoffMs`. On session loss it calls `GalaxyDriver.ReopenAsync`, which invokes `GalaxyMxSession.RecreateAsync` to dispose the stale/faulted session and client before rebuilding (`OpenSessionAsync` + `RegisterAsync`). Previously `ConnectAsync` was a no-op when a stale session handle was still present, so the reopen supervisor looped forever without recovering. After a successful reopen — when `ReplayOnSessionLost = true` — the supervisor calls the gateway's `ReplaySubscriptions` RPC with the cached subscription set from `SubscriptionRegistry` instead of re-subscribing tag-by-tag. The gateway's worker then re-issues `AdviseSupervisory` server-side under the apartment lock.
|
|
|
|
The session-less alarm feed (`GatewayGalaxyAlarmFeed`) and alarm acknowledger (`GatewayGalaxyAlarmAcknowledger`) run on a separate always-on `_ownedMxClient` that is intentionally **not** recreated when `ReconnectSupervisor` rebuilds the worker session. The feed has its own re-invoke loop (~5 s backoff) that reopens `StreamAlarms` after any stream fault, and gRPC.NET's channel auto-reconnect recovers the underlying HTTP/2 connection after a gateway restart — so both the alarm stream and individual ack calls recover without client replacement. The acknowledger is completely stateless between calls: each `AcknowledgeAsync` issues a single unary RPC and returns; no dead-client state is latched on failure. Channel-level keepalive hardening (TCP keep-alive intervals, gRPC ping frames) would require exposing additional knobs on `MxGatewayClientOptions` in the sibling `mxaccessgw` repo — a future option if flaky long-lived connections are observed in production.
|
|
|
|
## Testing
|
|
|
|
- **Unit tests**: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests/` — fakes the gateway gRPC surface; covers Browse, Runtime, Health, and Config in isolation.
|
|
- **Parity rig + dev-rig walkthrough**: see [docs/v2/Galaxy.ParityRig.md](../v2/Galaxy.ParityRig.md). The rig stands up a real `mxaccessgw` against a live Galaxy and exercises the full read / write / subscribe / rediscover path.
|
|
- **Performance + soak**: see [docs/v2/Galaxy.Performance.md](../v2/Galaxy.Performance.md).
|
|
|
|
## Browse Tree Shape
|
|
|
|
`GalaxyDiscoverer` builds the OPC UA address space as a **true nested tree** by `parent_gobject_id`: each gobject folder is parented under its Galaxy parent folder rather than being emitted flat under the driver root. Parentage falls back gracefully — a gobject with `parent_gobject_id` of 0, a parent not present in the returned set, a self-referencing parent, or a cycle attaches directly to the driver root. This matches the Galaxy `ArchestrA` object hierarchy as seen in the IDE.
|
|
|
|
## Operational Notes
|
|
|
|
- **MXAccess `ClientName` collisions**: two OtOpcUa instances sharing a `ClientName` cause the older Wonderware session to lose subscription state. Redundancy pairs (decision #149) enforce uniqueness via install scripts.
|
|
- **Channel saturation**: `galaxy.events.dropped > 0` indicates `EventPump` is back-pressured. Raise `EventPumpChannelCapacity` or investigate downstream slowness in the server-side fan-out.
|
|
- **Connectivity surface**: per-platform probe state is exposed through `IHostConnectivityProbe` and aggregated by the server's connectivity bus — there is no driver-private dashboard surface anymore. The Admin UI's Host Status panel is the consumer.
|
|
- **A stream the gateway ends cleanly is treated as a fault.** A closed or evicted gateway session completes the `StreamEvents` RPC *without an error*, so nothing throws. `EventPump` reports that to the `ReconnectSupervisor` exactly as it reports a transport fault. Before this it exited silently: the supervisor was never told, `GetHealth()` kept reporting `Healthy`, and because the [comms-loss watchdog](../DriverLifecycle.md) requires a **non-Healthy** state to trip, it never fired either — the driver went permanently deaf with no signal anywhere. Nothing else in the system can observe this, so do not remove the notification on the grounds that "a clean end isn't an error".
|
|
- **Replay gaps** (`galaxy.events.replay_gaps`): the gateway signals that events were evicted from its replay ring before we could read them. The driver responds by re-issuing `SubscribeBulk` for every tracked subscription — MXAccess delivers an initial value per item on subscribe, so re-subscribing *is* the re-snapshot the contract asks for. It does **not** restart the event pump (the stream carrying the sentinel is healthy) and it honours `Reconnect:ReplayOnSessionLost=false` by logging and standing down, on the grounds that an operator who delegated recovery to the gateway did not ask for a re-subscribe on a different trigger.
|
|
⚠️ This cannot currently fire: the sentinel is only emitted on a stream opened with `after_worker_sequence > 0`, and the driver opens with `0`, which the gateway treats as "fresh stream, not a resume". Enabling resume is a bigger change than it looks — the sequence is only meaningful **within one gateway session**, and the reconnect path recreates the session, so a remembered sequence replayed against a fresh session would set the server's live watermark *above* every event it is about to send, silencing the stream rather than recovering it.
|
|
- **Undecodable values** (`galaxy.events.undecodable`): a value kind this driver has no CLR representation for is published as a **null under the event's own quality**, not as Bad. The status is deliberately left alone — nothing can reach this branch today, so the counter exists to produce evidence before anyone changes what an OPC UA client sees.
|