fix(galaxy): confirm unconfirmed native alarm acks against Galaxy's acknowledged transition (#869)

mxaccessgw answers AcknowledgeAlarm OK both when the AckMsg write was
applied and when it was only sent (no OnWriteComplete; the reply then
carries a diagnostic). EvaluateReply treated both as success, so the H6
live gate saw a condition go Acknowledged while Galaxy never applied it.

- EvaluateReply returns Applied (no diagnostic) or Unconfirmed (diagnostic)
  and logs the gateway diagnostic on every branch that carries one.
- GalaxyAckConfirmationTracker watches the alarm feed from before the RPC
  and, for an unconfirmed reply, waits up to AlarmAck.ConfirmTimeoutMs
  (default 10 s, 0 = old behaviour) for an Acknowledge/ActiveAcknowledged
  transition; on timeout it refuses (BadTimeout + diagnostic).
- New optional IAlarmAcknowledgeConfirmation: DriverInstanceActor awaits it
  after AcknowledgeAsync, off its mailbox and outside the resilience
  pipeline; a failure is the existing #760 AlarmAckFailed revert.
- AdminUI Galaxy form carries the new knob; docs updated.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
Joseph Doherty
2026-11-11 13:03:04 -05:00
parent e857135e8b
commit 4e492e6550
19 changed files with 1046 additions and 34 deletions
+35 -1
View File
@@ -85,7 +85,7 @@ Project root files:
- **`MxAccess`** — `ClientName` (must be unique per OtOpcUa instance — redundancy pairs enforce uniqueness at install time), `PublishingIntervalMs` (forwarded as `buffered_update_interval_ms` on subscribe), `WriteUserId` for ArchestrA secured-write, `EventPumpChannelCapacity` (default 50_000 — one second of headroom at 50k tags / 1Hz; tune via the `galaxy.events.dropped` metric).
- **`Repository`** — `DiscoverPageSize`, `WatchDeployEvents`.
- **`Reconnect`** — `InitialBackoffMs`, `MaxBackoffMs`, `ReplayOnSessionLost` (default `true`: after a gateway session is recreated, re-issue `SubscribeBulk` for every tracked subscription; `false` leaves them stranded — Bad + Degraded, see [Reconnect + Replay](#reconnect--replay)).
- **`AlarmAck`** (optional, lmxopcua#749) — `FallbackUser`, `FallbackDomain`, `FallbackPasswordSecretRef`. See [Alarm acknowledge credential](#alarm-acknowledge-credential).
- **`AlarmAck`** (optional, lmxopcua#749) — `FallbackUser`, `FallbackDomain`, `FallbackPasswordSecretRef`. See [Alarm acknowledge credential](#alarm-acknowledge-credential). `ConfirmTimeoutMs` (default `10000`, max `120000`, `0` = do not wait, negative = default; #869) — see [Ack confirmation](#ack-confirmation-869).
### Alarm acknowledge credential
@@ -132,6 +132,40 @@ Galaxy password, so `GalaxyAckCredentialResolver` picks the account per acknowle
`Retain=true`) and logs the reason. A driver transition projected after the ack wins over a late
failure.
### Ack confirmation (#869)
The gateway answers `AcknowledgeAlarm` **OK with `hresult` 0 in two cases**: the `AckMsg` write's
`OnWriteComplete` reported success (*applied* — empty `diagnostic_message`), or the write was sent and no
completion arrived inside the worker's window (*sent but unconfirmed* — `diagnostic_message` says so,
e.g. `AckMsg write to '…' sent; OnWriteComplete not observed within 5000 ms.`). The H6 live gate
(2026-11-11, VD03) caught the second case: OtOpcUa showed the condition Acknowledged while Galaxy's
`.Acked` stayed False and no ACK row was ever historized.
- `GatewayGalaxyAlarmAcknowledger.EvaluateReply` returns **Applied** for an OK reply with no diagnostic and
**Unconfirmed** for one with a diagnostic (any unexplained diagnostic counts — the safe direction), and
logs the gateway diagnostic on every branch that carries one.
- `GalaxyAckConfirmationTracker` registers a watch for the alarm **before** the RPC (the alarm feed can
deliver the confirming transition ahead of the unary reply) and, for an unconfirmed reply, waits up to
`AlarmAck.ConfirmTimeoutMs` for an `Acknowledge` / `ActiveAcknowledged` transition of that alarm on the
alarm feed (live or snapshot re-statement; the reference match is case-insensitive). Whoever acked it,
Galaxy now shows it acknowledged — that is the truth the condition must match.
- The wait is the optional `IAlarmAcknowledgeConfirmation` capability. `DriverInstanceActor` calls it after
`AcknowledgeAsync` returned, **off its mailbox** (a detached task) and **outside the resilience pipeline**
(a wait on an upstream event: it must not be cut by the Tier-A 5 s `AlarmAcknowledge` timeout nor count
toward the breaker). The OPC UA method thread and the node-manager `Lock` are never involved — the method
already returned when the fire-and-forget route started.
- **On timeout** the confirmation throws `DriverOperationRefusedException` (`… not confirm … (BadTimeout;
gateway: <diagnostic>)`), the driver logs a Warning with the diagnostic, and the actor reports
`AlarmAckFailed`: the host re-projects the condition **unacknowledged** — the same #760 revert path.
**What the OPC UA client sees:** the `Acknowledge` call has already returned `Good` (it cannot be changed
after the fact, exactly as for every other #760 revert); the failure reaches the client as a fresh
condition event with `AckedState = false`, and the `BadTimeout` verdict is recorded in the driver and host
Warning logs. A confirming transition that arrives after the revert simply re-acknowledges the condition.
A confirmation in flight when the driver actor stops is abandoned and reverted the same way.
- `ConfirmTimeoutMs = 0` restores the pre-#869 behaviour (an unconfirmed ack counts as acknowledged, with a
Warning) — an escape hatch only. Why the gateway-side ack did not stick on VD03 is tracked upstream
(mxaccessgw#226).
The comment written to `AckMsg` is suffixed with the OPC UA identity —
`"<comment> [OPC UA: <principal>, key <keyId>]"` or `"[OPC UA: anonymous]"` — so the client is not
lost when the Historian's Operator is a shared fallback account. The password is never logged: the