Files
lmxopcua/docs/drivers/Galaxy.md
T
Joseph Doherty 4769044f88
v2-ci / build (push) Successful in 4m39s
v2-ci / unit-tests (push) Failing after 19m35s
feat(galaxy): handle the replay-gap sentinel and sparse-array values
Closes the two follow-ups left open by the MxGateway 0.2.0 bump. Reading the
gateway's own contract first changed what each of them is, and one of my earlier
claims was simply wrong -- both corrections are recorded below.

Replay gap
----------
A sentinel MxEvent carries family = UNSPECIFIED, so it fell through EventPump's
family switch into the default drop: we silently ignored a documented MUST ("you
missed events -- discard local state and re-snapshot"). It is now intercepted and
answered by re-issuing SubscribeBulk for every tracked subscription, since
MXAccess delivers an initial value per item on subscribe -- re-subscribing IS the
re-snapshot. It does not restart the event pump (the stream carrying the sentinel
is healthy, and the handler runs on that stream's own read loop, so disposing the
pump from there would deadlock on itself), it does not block the read loop, and
re-entry is gated so a burst collapses to one pass instead of a subscribe storm
per sentinel.

Handled on the READ loop, not in the dispatch loop, because everything past that
point is subject to newest-dropped channel pressure -- and "you missed events" is
the one signal whose loss is unrecoverable. Pinned by a test that fills the
channel and still expects the sentinel.

⚠️ This cannot fire in the shipped configuration, and I did not pretend otherwise
or paper over it by enabling resume. The gateway emits the sentinel only for a
stream opened with after_worker_sequence > 0; GatewayGalaxySubscriber passes 0,
which the gateway treats as "fresh stream, not a resume -- plain attach, no
replay, no sentinel". Enabling resume is a larger and genuinely delicate change:
the sequence is only meaningful within ONE gateway session, our reconnect
recreates the session, and replaying a remembered sequence against a fresh
session sets the server's live watermark ABOVE every event it is about to send --
silencing the stream instead of recovering it. That deserves its own change with
a live gate, which I cannot run from here.

Sparse arrays -- correcting the last commit message
---------------------------------------------------
I wrote that the bump "makes handling it possible" on the decode path. That is
wrong: MxSparseArray is documented write-only, the gateway expands it before
forwarding, and "the worker never receives or produces it". So the decode path
is unreachable by contract and the write path has no use for it either -- it is a
wire compaction for a WHOLE-array write, not a partial update (unmentioned
indices are RESET, deliberately, because the gateway cannot read-modify-write
without racing the provider). It is therefore the wrong primitive for an OPC UA
IndexRange write, which expects untouched elements preserved -- and WriteRequest
carries no IndexRange in the first place.

It is decoded anyway, as contract defence: the alternative if a future gateway
ever emits one is a silent null published under Good quality. Fill semantics
mirror the gateway's own SparseArrayExpander exactly (epoch for Time, empty
string, Integer widening to long[] when any element is int64) so a value that
round-trips through a write reads back as the same CLR array.

Also: the decoder is now total by contract, since it runs inside the dispatch
loop where an escaping exception kills dispatch for the life of the pump. A
malformed payload degrades instead of throwing.

A real defect found in the same code
------------------------------------
A stream the GATEWAY completes -- a closed or evicted session -- ends the RPC
without an error, so nothing threw, the supervisor was never notified, and the
pump exited silently. GetHealth() kept reporting Healthy, which also means the
#527 comms-loss watchdog could never trip, because it requires a non-Healthy
state. The driver went permanently deaf with no signal anywhere. This one is
live, unlike the two above. Now reported as a fault; the falsification control is
the existing CleanShutdown_DoesNotInvokeOnStreamFault, since our own disposal
also ends the stream and must NOT be reported.

Value kinds we cannot represent are now counted and warned once rather than
published as an indistinguishable null. The status is deliberately NOT downgraded
to Bad: nothing can reach that branch today, and changing what an OPC UA client
sees for a live shape is not a change to make without a gate. The counter exists
so the next person has evidence instead of a guess.

Verified: 330 Galaxy tests green (12 new), plus Galaxy.Browser 21, Runtime 558,
Core 351; full solution builds 0 errors with the warning census unchanged. Every
new test was falsified first -- disabling the gap interception reds 3 of 4 (the
throwing-callback test passes vacuously when the callback never runs, by
construction), disabling sparse decode plus the representability signal reds all
7 decoder tests, and both were restored and re-verified green.

Docs: the Galaxy.Performance invariant was stated as received = dispatched +
dropped + in-flight, which this breaks -- a sentinel is received but never enters
the channel -- so it now reads + replay_gaps. Galaxy.md gains the three
operational notes; DriverLifecycle.md records that "reaches a non-Healthy state"
is an obligation a driver can silently fail to meet, which is the general shape
of the clean-end bug.
2026-08-12 02:56:17 -04:00

13 KiB

Galaxy Driver

The Galaxy driver bridges OtOpcUa to AVEVA System Platform (Wonderware) Galaxies. It is a Tier-A in-process driver that runs in the OtOpcUa server's .NET 10 AnyCPU process and speaks gRPC to a separately installed mxaccessgw server (sibling repo at c:\Users\dohertj2\Desktop\mxaccessgw\). The gateway owns the MXAccess COM apartment, the STA + Win32 message pump, the Galaxy Repository SQL reader, and the Historian SDK — all the bits that need x86 / .NET Framework 4.8 / COM interop. The driver itself is platform-agnostic and contains no COM, no STA thread, and no x86 bitness constraint.

For the driver spec (capability surface, config shape, addressing), see docs/v2/driver-specs.md §1. For the gateway setup recipe, see docs/v2/Galaxy.ParityRig.md. For tracing, metrics, and soak profile, see docs/v2/Galaxy.Performance.md.

Note

: the related docs Galaxy-Repository.md and Galaxy-Test-Fixture.md describe the previous v1 / out-of-process topology and now live under docs/v1/drivers/. For current testing use Galaxy.ParityRig.md and the mxaccessgw repo.

Architecture

                    +---------------------------------------+
                    |  OtOpcUa.Server (.NET 10 AnyCPU)      |
                    |   GalaxyDriver (in-process)           |
                    |    ITagDiscovery / IReadable /        |
                    |    IWritable / ISubscribable /        |
                    |    IRediscoverable /                  |
                    |    IHostConnectivityProbe /           |
                    |    IAlarmSource                       |
                    +-------------------+-------------------+
                                        |
                                       gRPC (default http://localhost:5120)
                                        |
                                        v
                    +---------------------------------------+
                    |  mxaccessgw  (sibling repo)           |
                    |    +-------------------------------+  |
                    |    | MxGateway.Worker (x86 net48)  |  |
                    |    |   STA + WM_APP pump           |  |
                    |    |   ArchestrA.MxAccess COM      |  |
                    |    |   Galaxy Repository SQL       |  |
                    |    |   Wonderware Historian SDK    |  |
                    |    +-------------------------------+  |
                    +---------------------------------------+

History reads moved server-side in PR 7.2 (IHistoryRouter). Galaxy no longer implements IHistoryProvider of its own.

IAlarmSource was retired with PR 7.2 and restored in PR B.2 of the alarms-over-gateway epic (docs/plans/alarms-over-gateway.md). Alarm transitions arrive on the same gateway StreamEvents channel as data-change events under the new MX_EVENT_FAMILY_ON_ALARM_TRANSITION family; acknowledgements route through the gateway's AcknowledgeAlarm RPC. The previous value-driven sub-attribute path remains as a fallback for Galaxy templates without $Alarm* extensions — the server-side AlarmConditionService dedups when both paths fire on the same condition. See docs/AlarmTracking.md for the v2-final architecture.

Project Layout

The driver ships as a single project: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/ (.NET 10, AnyCPU). Sub-folders:

Folder Role
Browse/ Static-side discovery: GalaxyDiscoverer walks the gateway's hierarchy + attribute-set RPCs, DataTypeMap and SecurityMap translate Galaxy types and security classifications into OPC UA equivalents, AlarmRefBuilder extracts alarm-bearing attribute references for the server-layer alarm engine. IGalaxyHierarchySource + GatewayGalaxyHierarchySource + TracedGalaxyHierarchySource decorate the gateway browse RPC; IGalaxyDeployWatchSource + GatewayGalaxyDeployWatchSource + DeployWatcher drive IRediscoverable.
Runtime/ Live data path: EventPump runs the gateway's StreamEvents RPC and fans out to subscribers via a bounded channel; GalaxyMxSession is the read-side handle; GatewayGalaxySubscriber + GatewayGalaxyDataWriter (each with a Traced* decorator) implement ISubscribable / IWritable; SubscriptionRegistry tracks subscription state for replay; ReconnectSupervisor owns the backoff loop and triggers ReplaySubscriptions on session loss; StatusCodeMap translates gateway StatusCodes to OPC UA; MxValueDecoder / MxValueEncoder handle scalar + array marshalling; GalaxyTelemetry + GalaxySubscriptionHandle round out the surface.
Health/ HostStatusAggregator rolls per-platform probe state into the driver's IHostConnectivityProbe view; PerPlatformProbeWatcher listens on the gateway's per-host status stream; HostConnectivityForwarder pushes transitions out to the server's connectivity bus.
Config/ GalaxyDriverOptions and the four nested option records (GalaxyGatewayOptions, GalaxyMxAccessOptions, GalaxyRepositoryOptions, GalaxyReconnectOptions).

Project root files:

  • GalaxyDriver.cs — IDriver + capability-interface implementation; composes the Browse / Runtime / Health collaborators.
  • GalaxyDriverFactoryExtensions.cs — DI registration helper used by the server's driver bootstrap.

Capability Surface

GalaxyDriver : IDriver, ITagDiscovery, IReadable, IWritable, ISubscribable, IRediscoverable, IHostConnectivityProbe, IAlarmSource, IDisposable, IAsyncDisposable.

Capability Implementation entry point
ITagDiscovery Browse/GalaxyDiscoverer.cs
IRediscoverable Browse/DeployWatcher.cs — consumed as an operator prompt: a Galaxy redeploy raises a "re-browse" chip on /hosts. ⚠️ It does not rebuild the address space — re-browse the device under /raw and commit (#518)
IReadable Runtime/GalaxyMxSession.cs
IWritable Runtime/GatewayGalaxyDataWriter.cs
ISubscribable Runtime/GatewayGalaxySubscriber.cs (driven by EventPump)
IHostConnectivityProbe Health/HostStatusAggregator.cs
IAlarmSource Runtime/GatewayGalaxyAlarmFeed.cs (transitions) + Runtime/GatewayGalaxyAlarmAcknowledger.cs (acks)

Configuration

DriverConfig JSON binds to Config/GalaxyDriverOptions.cs. The four sections are:

  • Gateway — endpoint, API key secret ref, TLS knobs, connect/call/stream timeouts. StreamTimeoutSeconds = 0 keeps the long-lived StreamEvents RPC open for the driver's lifetime.
  • MxAccess — ClientName (must be unique per OtOpcUa instance — redundancy pairs enforce uniqueness at install time), PublishingIntervalMs (forwarded as buffered_update_interval_ms on subscribe), WriteUserId for ArchestrA secured-write, EventPumpChannelCapacity (default 50_000 — one second of headroom at 50k tags / 1Hz; tune via the galaxy.events.dropped metric).
  • Repository — DiscoverPageSize, WatchDeployEvents.
  • Reconnect — InitialBackoffMs, MaxBackoffMs, ReplayOnSessionLost (calls the gateway's ReplaySubscriptions RPC after reconnect rather than re-issuing subscribe-bulk for every tag).

Full per-field descriptions live in Config/GalaxyDriverOptions.cs. The full JSON skeleton is reproduced in docs/v2/driver-specs.md §1.

Reconnect + Replay

ReconnectSupervisor owns an exponential-backoff loop bounded by Reconnect.InitialBackoffMs / MaxBackoffMs. On session loss it calls GalaxyDriver.ReopenAsync, which invokes GalaxyMxSession.RecreateAsync to dispose the stale/faulted session and client before rebuilding (OpenSessionAsync + RegisterAsync). Previously ConnectAsync was a no-op when a stale session handle was still present, so the reopen supervisor looped forever without recovering. After a successful reopen — when ReplayOnSessionLost = true — the supervisor calls the gateway's ReplaySubscriptions RPC with the cached subscription set from SubscriptionRegistry instead of re-subscribing tag-by-tag. The gateway's worker then re-issues AdviseSupervisory server-side under the apartment lock.

The session-less alarm feed (GatewayGalaxyAlarmFeed) and alarm acknowledger (GatewayGalaxyAlarmAcknowledger) run on a separate always-on _ownedMxClient that is intentionally not recreated when ReconnectSupervisor rebuilds the worker session. The feed has its own re-invoke loop (~5 s backoff) that reopens StreamAlarms after any stream fault, and gRPC.NET's channel auto-reconnect recovers the underlying HTTP/2 connection after a gateway restart — so both the alarm stream and individual ack calls recover without client replacement. The acknowledger is completely stateless between calls: each AcknowledgeAsync issues a single unary RPC and returns; no dead-client state is latched on failure. Channel-level keepalive hardening (TCP keep-alive intervals, gRPC ping frames) would require exposing additional knobs on MxGatewayClientOptions in the sibling mxaccessgw repo — a future option if flaky long-lived connections are observed in production.

Testing

  • Unit tests: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests/ — fakes the gateway gRPC surface; covers Browse, Runtime, Health, and Config in isolation.
  • Parity rig + dev-rig walkthrough: see docs/v2/Galaxy.ParityRig.md. The rig stands up a real mxaccessgw against a live Galaxy and exercises the full read / write / subscribe / rediscover path.
  • Performance + soak: see docs/v2/Galaxy.Performance.md.

Browse Tree Shape

GalaxyDiscoverer builds the OPC UA address space as a true nested tree by parent_gobject_id: each gobject folder is parented under its Galaxy parent folder rather than being emitted flat under the driver root. Parentage falls back gracefully — a gobject with parent_gobject_id of 0, a parent not present in the returned set, a self-referencing parent, or a cycle attaches directly to the driver root. This matches the Galaxy ArchestrA object hierarchy as seen in the IDE.

Operational Notes

  • MXAccess ClientName collisions: two OtOpcUa instances sharing a ClientName cause the older Wonderware session to lose subscription state. Redundancy pairs (decision #149) enforce uniqueness via install scripts.
  • Channel saturation: galaxy.events.dropped > 0 indicates EventPump is back-pressured. Raise EventPumpChannelCapacity or investigate downstream slowness in the server-side fan-out.
  • Connectivity surface: per-platform probe state is exposed through IHostConnectivityProbe and aggregated by the server's connectivity bus — there is no driver-private dashboard surface anymore. The Admin UI's Host Status panel is the consumer.
  • A stream the gateway ends cleanly is treated as a fault. A closed or evicted gateway session completes the StreamEvents RPC without an error, so nothing throws. EventPump reports that to the ReconnectSupervisor exactly as it reports a transport fault. Before this it exited silently: the supervisor was never told, GetHealth() kept reporting Healthy, and because the comms-loss watchdog requires a non-Healthy state to trip, it never fired either — the driver went permanently deaf with no signal anywhere. Nothing else in the system can observe this, so do not remove the notification on the grounds that "a clean end isn't an error".
  • Replay gaps (galaxy.events.replay_gaps): the gateway signals that events were evicted from its replay ring before we could read them. The driver responds by re-issuing SubscribeBulk for every tracked subscription — MXAccess delivers an initial value per item on subscribe, so re-subscribing is the re-snapshot the contract asks for. It does not restart the event pump (the stream carrying the sentinel is healthy) and it honours Reconnect:ReplayOnSessionLost=false by logging and standing down, on the grounds that an operator who delegated recovery to the gateway did not ask for a re-subscribe on a different trigger. ⚠️ This cannot currently fire: the sentinel is only emitted on a stream opened with after_worker_sequence > 0, and the driver opens with 0, which the gateway treats as "fresh stream, not a resume". Enabling resume is a bigger change than it looks — the sequence is only meaningful within one gateway session, and the reconnect path recreates the session, so a remembered sequence replayed against a fresh session would set the server's live watermark above every event it is about to send, silencing the stream rather than recovering it.
  • Undecodable values (galaxy.events.undecodable): a value kind this driver has no CLR representation for is published as a null under the event's own quality, not as Bad. The status is deliberately left alone — nothing can reach this branch today, so the counter exists to produce evidence before anyone changes what an OPC UA client sees.