fix(galaxy): stranded subscriptions after a no-replay reconnect go Bad + Degraded (#862)
With Reconnect.ReplayOnSessionLost=false a recreated gateway session carries none of the old advises, so every pre-reconnect subscription went silently deaf: nodes kept their last value as Good and health returned to Healthy (only a Warning log). ReplayAsync now records the stranded subscriptions and publishes BadNoCommunication (StatusCodeMap.BadNoCommunication, 0x80310000; value null — no driver value cache) for every tag of each, straight to OnDataChange so LastSuccessfulRead is not stamped. GetHealth reports Degraded naming the setting while any stranded subscription is still registered; it clears on UnsubscribeAsync (the host's re-subscribe path) or runtime teardown, and pruned ids never pin it. A subscribe after the reconnect is an ordinary new-session subscription. With nothing else flowing, the comms-loss watchdog re-initializes the driver after 90 s (a delayed replay). Tests: Runtime/StrandedSubscriptionTests (6). Docs: drivers/Galaxy.md, which also no longer claims the replay uses the gateway's ReplaySubscriptions RPC. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
+11
-2
@@ -84,7 +84,7 @@ Project root files:
|
||||
- **`Gateway`** — endpoint, API key secret ref (`secret:galaxy/<name>` · `secret:otopcua/galaxy/<name>` · `env:OTOPCUA_GALAXY_<NAME>` · legacy `env:GALAXY_MXGW_<NAME>` · `dev:KEY` · literal; anything else, incl. `file:`, is refused — #815 (an already-deployed driver keeps resolving a pre-#815 ref on init and every gateway reconnect in deprecated warn-only mode until its next deploy), [`docs/security.md`](../security.md) §"Credential references are namespaced per driver"), TLS knobs, connect/call/stream timeouts. `StreamTimeoutSeconds = 0` keeps the long-lived `StreamEvents` RPC open for the driver's lifetime.
|
||||
- **`MxAccess`** — `ClientName` (must be unique per OtOpcUa instance — redundancy pairs enforce uniqueness at install time), `PublishingIntervalMs` (forwarded as `buffered_update_interval_ms` on subscribe), `WriteUserId` for ArchestrA secured-write, `EventPumpChannelCapacity` (default 50_000 — one second of headroom at 50k tags / 1Hz; tune via the `galaxy.events.dropped` metric).
|
||||
- **`Repository`** — `DiscoverPageSize`, `WatchDeployEvents`.
|
||||
- **`Reconnect`** — `InitialBackoffMs`, `MaxBackoffMs`, `ReplayOnSessionLost` (calls the gateway's `ReplaySubscriptions` RPC after reconnect rather than re-issuing subscribe-bulk for every tag).
|
||||
- **`Reconnect`** — `InitialBackoffMs`, `MaxBackoffMs`, `ReplayOnSessionLost` (default `true`: after a gateway session is recreated, re-issue `SubscribeBulk` for every tracked subscription; `false` leaves them stranded — Bad + Degraded, see [Reconnect + Replay](#reconnect--replay)).
|
||||
- **`AlarmAck`** (optional, lmxopcua#749) — `FallbackUser`, `FallbackDomain`, `FallbackPasswordSecretRef`. See [Alarm acknowledge credential](#alarm-acknowledge-credential).
|
||||
|
||||
### Alarm acknowledge credential
|
||||
@@ -147,7 +147,16 @@ Full per-field descriptions live in `Config/GalaxyDriverOptions.cs`. The full JS
|
||||
|
||||
## Reconnect + Replay
|
||||
|
||||
`ReconnectSupervisor` owns an exponential-backoff loop bounded by `Reconnect.InitialBackoffMs` / `MaxBackoffMs`. On session loss it calls `GalaxyDriver.ReopenAsync`, which invokes `GalaxyMxSession.RecreateAsync` to dispose the stale/faulted session and client before rebuilding (`OpenSessionAsync` + `RegisterAsync`). Previously `ConnectAsync` was a no-op when a stale session handle was still present, so the reopen supervisor looped forever without recovering. After a successful reopen — when `ReplayOnSessionLost = true` — the supervisor calls the gateway's `ReplaySubscriptions` RPC with the cached subscription set from `SubscriptionRegistry` instead of re-subscribing tag-by-tag. The gateway's worker then re-issues `AdviseSupervisory` server-side under the apartment lock.
|
||||
`ReconnectSupervisor` owns an exponential-backoff loop bounded by `Reconnect.InitialBackoffMs` / `MaxBackoffMs`. On session loss it calls `GalaxyDriver.ReopenAsync`, which invokes `GalaxyMxSession.RecreateAsync` to dispose the stale/faulted session and client before rebuilding (`OpenSessionAsync` + `RegisterAsync`). Previously `ConnectAsync` was a no-op when a stale session handle was still present, so the reopen supervisor looped forever without recovering. After a successful reopen — when `ReplayOnSessionLost = true` (the default) — `ReplayAsync` re-issues `SubscribeBulk` for every subscription still holding the previous session's handles and rebinds the registry with the new ones (the gateway's batched `ReplaySubscriptions` RPC is not used).
|
||||
|
||||
**`ReplayOnSessionLost = false` strands the subscriptions, visibly (#862).** A recreated gateway session carries none of the old session's advises, so with replay disabled every subscription that existed before the reconnect stays registered but receives nothing. That used to be silent — each node kept its last value as Good and health returned to `Healthy`; the only trace was a Warning. Now, following the [quality contract](../DriverLifecycle.md):
|
||||
|
||||
- every tag of every stranded subscription is published **`BadNoCommunication`** once per reconnect, under that subscription's own handle (value null — the driver keeps no value cache, and a Bad sample's value is not client-observable; the next Good value comes from MXAccess's initial value when the tag is advised again);
|
||||
- `GetHealth()` reports **`Degraded`** with a message naming `Reconnect.ReplayOnSessionLost=false` and the stranded count, after the supervisor's own reconnect Degraded has ended;
|
||||
- `LastSuccessfulRead` is **not** advanced by those publishes. If no other data flows (no other live subscription), the host's [comms-loss watchdog](../DriverLifecycle.md) sees Degraded + a frozen `LastSuccessfulRead` and re-initializes the driver after 90 s — which re-subscribes everything, i.e. a delayed replay. Set `ReplayOnSessionLost = true` to recover immediately instead;
|
||||
- **Degraded clears** when the last stranded subscription is unsubscribed (the host unsubscribes the old handle on every data re-subscribe, e.g. a deploy, and on re-init) or the runtime is torn down. A `SubscribeAsync` issued after the reconnect is an ordinary new-session subscription: it receives data normally and does not count as stranded.
|
||||
|
||||
The per-platform `ScanState` probe watcher is not replayed when `ReplayOnSessionLost = false` either (unchanged), so `/hosts` per-platform status also stops updating until a re-init.
|
||||
|
||||
The session-less alarm feed (`GatewayGalaxyAlarmFeed`) and alarm acknowledger (`GatewayGalaxyAlarmAcknowledger`) run on a separate always-on `_ownedMxClient` that is intentionally **not** recreated when `ReconnectSupervisor` rebuilds the worker session. The feed has its own re-invoke loop (~5 s backoff) that reopens `StreamAlarms` after any stream fault, and gRPC.NET's channel auto-reconnect recovers the underlying HTTP/2 connection after a gateway restart — so both the alarm stream and individual ack calls recover without client replacement. The acknowledger is completely stateless between calls: each `AcknowledgeAsync` issues a single unary RPC and returns; no dead-client state is latched on failure. Channel-level keepalive hardening (TCP keep-alive intervals, gRPC ping frames) would require exposing additional knobs on `MxGatewayClientOptions` in the sibling `mxaccessgw` repo — a future option if flaky long-lived connections are observed in production.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user