diff --git a/docs/drivers/evidence/galaxy-h6-ack-2026-11-12.md b/docs/drivers/evidence/galaxy-h6-ack-2026-11-12.md new file mode 100644 index 000000000..4b4aefd26 --- /dev/null +++ b/docs/drivers/evidence/galaxy-h6-ack-2026-11-12.md @@ -0,0 +1,154 @@ +# Galaxy H6 (#614): OPC UA ack of a real Galaxy native alarm to AVEVA history, 2026-11-12 (re-run) + +Gitea #614 item H6: "an OPC UA Acknowledge of a REAL Galaxy native alarm reaches AVEVA event history". + +**Result: PASS.** One OPC UA `Acknowledge` sent through OtOpcUa to the DEV mxaccessgw on wonder-app-vd03 +reached every layer: the OPC UA condition, the gateway alarm feed, Galaxy (`.Acked` True, `.AckMsg` holds +the comment) and AVEVA event history (an `ACK_ALM` row with the right operator, domain, node and comment). + +Exactly **one** acknowledge was sent (owner-approved write, 2026-11-13T04:17:22Z, i.e. 2026-11-12 23:17 +EST). No retry was needed. No remote command ran on any wonder box: Galaxy was reached only through the +gateway's gRPC API, and the Historian only through read-only SQL. + +The failed first run is recorded in `galaxy-h6-ack-2026-11-11.md`; this run reused its recipe. + +## Why the 2026-11-11 ack didn't apply (pre-ack analysis) + +The question was whether mxaccessgw#226 (alarm times 4–5 h late) or #228 (`Write2` time argument read as +local) could explain the failure. **Neither does**, from the mxaccessgw source: + +- The gateway does not match an ack by alarm time. `GatewayAlarmMonitor` turns `alarm_full_reference` + `Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert` into `AcknowledgeAlarmByName` (provider `Galaxy`, group + `CVDAisle_1`, alarm `TableAlarms_008.RuntimeAlert`). The worker's `WnWrapAlarmConsumer.ResolveAckTarget*` + then maps that to the item `TableAlarms_008.RuntimeAlert.AckMsg` by name only. No timestamp is involved. +- `AlarmAckWriter` writes the comment with a plain MXAccess `Write(..., userId)` after `AuthenticateUser`. + It does not use `Write2`, so #228 does not touch it. + +**The actual cause was the gateway build.** At 17:37:55Z on 2026-11-11, vd03 was still running mxgw `4f931e4` +(2026-09-03). The vd03 upgrade to `2b13564e` happened later that day, between about 21:37Z and 21:51Z (see +`.claude/skills/sister-project-history/otopcua.md` in scadaproj, and mxgw `docs/deploy/`). `4f931e4` +predates the operator-authenticated ack (`488649c`, PR #162, 2026-10-20). In that build, an ack was +`wwAlarmConsumerClass.AlarmAckByName`/`AlarmAckByGUID`. mxgw's own source and +`docs/AlarmProbeFindings.md` say that path returns `rc=0` when the call is *accepted*, not when it is +*applied*. It carries an operator name but no authenticated user id, so under Galaxy security it cannot +satisfy the permission check and does nothing. That matches what 11-11 saw: OK/hresult 0 and no change in +Galaxy. The OPC UA proto's `operator_password` field didn't exist in `4f931e4`, so that build never saw the +password. + +vd03 now runs `76a71bc8` (mxaccessgw#229, deployed 2026-11-12 ~02:19Z), which has the `AuthenticateUser` + +`AckMsg` path. No other un-fixed cause made a second failure certain, so the ack went ahead. The remaining +risks (bad credentials, no ack permission, no `OnWriteComplete`) all fail visibly now, and OtOpcUa reverts +them after #869. + +## Setup (docker-dev, temporary, all removed afterwards) + +| Item | Value | +|---|---| +| OtOpcUa image | `otopcua-host:dev` `sha256:5094b682cef5` (built 2026-11-12 16:14Z). It contains the #869 fix: `GalaxyAckConfirmationTracker` and `AlarmAck.ConfirmTimeoutMs` are present in `ZB.MOM.WW.OtOpcUa.Driver.Galaxy.dll`. Not rebuilt for this run. | +| Gateway | `wonder-app-vd03.zmr.zimmer.com:5120` (h2c), mxgw main `76a71bc8`; `/health/ready` Healthy before the run | +| Node | SITE-B `site-b-1` (OPC UA `opc.tcp://localhost:4844`). `site-b-2` was stopped for the run, so site-b-1 was the sole (Primary) node. | +| Driver | `h6gal`, DriverType `GalaxyMxGateway`, seeded straight into the ConfigDb. `Gateway.Endpoint` = the vd03 gateway, `UseTls=false`, `ApiKeySecretRef=env:GALAXY_MXGW_API_KEY`, `MxAccess.ClientName=OtOpcUa-H6-gate`, `Repository.WatchDeployEvents=false`, `AlarmAck.FallbackUser`/`FallbackDomain` = the owner-provided account (domain `NAM`), `FallbackPasswordSecretRef=env:OTOPCUA_GALAXY_ACKPW`. `ConfirmTimeoutMs` was left at the default (10 s). | +| Device / tag | device `vd03`, tag `RuntimeAlert`, Boolean, Read, `attributeRef` `Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert`, `alarm` = `AlarmCondition`, severity 400, `historizeToAveva:false` (so OtOpcUa's own alarm historian wrote nothing to AVEVA) | +| Secrets | The gateway key and the ack password came from the owner's local env file. They were injected into site-b-1 as runtime env only, through a scratch compose override holding `${VAR}` references. Nothing secret went into the repo, the compose files, the ConfigDb (only the `env:` reference) or this file. | +| OPC UA client | `otopcua-cli` built from `85db1dbf`. UserName `apikey` + a temporary data-plane key (`h6-gate`, keyId `a33c3b973768fbf5`, principal `h6gate`, 4 h expiry). The key had a SITE-B `Cluster` grant of 399: Browse, Read, Subscribe, HistoryRead, AlarmRead and AlarmAcknowledge. | +| Deploy | `POST /api/deployments` (`b95c3609`). site-b-1 Applied. The deployment stayed `AwaitingApplyAcks` because site-b-2 was stopped, as on 11-11. | + +Driver start (site-b-1, 04:16:32Z): `GalaxyMxSession connected — clientName=OtOpcUa-H6-gate`, +`DriverInstance h6gal: connected`, `native-alarm subscription established for 1 alarm ref(s)`. + +## Target, before the ack + +`Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert` was still active and unacknowledged, so it was used again. +Z28061 was not touched. + +| Check (before) | Value | +|---|---| +| `QueryActiveAlarms` | `ALARM_CONDITION_STATE_ACTIVE`, severity 400, `lastTransitionTimestamp 2026-07-28T20:02:46.830Z`. This is now correct UTC: #226 is visible end to end, and on 11-11 the value was `01:02:46.830Z` on 07-29. | +| `.Acked` / `.AckMsg` / `.InAlarm` (read-only gateway session, 04:14:47Z) | `False` / `""` / `True` (since 2026-07-28T20:02:46.830Z), quality 192 | +| `v_AlarmEventHistory2` | only the 07-28 `UNACK_ALM` and later `UNACK_RTN` rows; no ACK row | +| OPC UA `alarms --refresh` | `[2026-07-28T20:02:46.8300000Z] h6gal/vd03/RuntimeAlert`, `Active, Unacknowledged`, EventId `4966AAB18408A24F9BB3CEC025FCA77E` | + +## The single ack + +``` +2026-11-13T04:17:22Z +otopcua-cli ack -u opc.tcp://localhost:4844 -n "ns=2;s=h6gal/vd03/RuntimeAlert" -e 4966AAB18408A24F9BB3CEC025FCA77E -c "OtOpcUa #614 H6 gate" -U apikey -P +Acknowledge successful: ns=2;s=h6gal/vd03/RuntimeAlert +``` + +## Results per layer + +### 1. OPC UA (PASS) + +The live subscriber saw the condition go `Active, Acknowledged` at 04:17:22.546Z (EventId +`E260F1DB097FC543A90AE478FF5E928C`). It saw no later revert to Unacknowledged in the 90 s window. + +Driver log (account name redacted): + +``` +[04:17:22 INF] Galaxy driver h6gal: acknowledging Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert as Galaxy user NAM\ (Fallback) for OPC UA principal h6gate (key a33c3b973768fbf5) +[04:17:25 INF] DriverInstance h6gal: acknowledged native condition h6gal/vd03/RuntimeAlert by h6gate +``` + +There was no gateway diagnostic, no `Unconfirmed` and no `BadTimeout` revert. The ~3 s between the two lines +is the gateway waiting on MXAccess `OnWriteComplete`. On 11-11 the same pair of lines was logged within the +same second. + +### 2. Gateway alarm feed (PASS) + +`QueryActiveAlarms` at 04:19:22Z: + +``` +"currentState": "ALARM_CONDITION_STATE_ACTIVE_ACKED", "severity": 400, +"lastTransitionTimestamp": "2026-11-13T04:17:25.173Z", +"operatorUser": "", +"operatorComment": "OtOpcUa #614 H6 gate [OPC UA: h6gate, key a33c3b973768fbf5]" +``` + +### 3. Galaxy (PASS) + +Read-only gateway session (`OpenSession` → `Register` → `ReadBulk` → `CloseSession`) at 04:19:20Z: + +| Attribute | Value | Quality | +|---|---|---| +| `TableAlarms_008.RuntimeAlert.Acked` | **True** | 192 | +| `TableAlarms_008.RuntimeAlert.AckMsg` | **`OtOpcUa #614 H6 gate [OPC UA: h6gate, key a33c3b973768fbf5]`** | 192 | +| `TableAlarms_008.RuntimeAlert.InAlarm` | True (since 2026-07-28T20:02:46.830Z) | 192 | + +### 4. AVEVA event history (PASS) + +`Runtime.dbo.v_AlarmEventHistory2` on wonder-sql-vd03, `EventStamp` 2026-11-12 22:00 to 2026-11-13 02:00 local +(read-only). `Operator` is the ack account (redacted here), and `UserFullName` (redacted) is that account's +directory name: + +| EventStamp (local) | EventStampUTC | AlarmState | TagName | Operator | DomainName | OperatorNode | Description / Value | +|---|---|---|---|---|---|---|---| +| 2026-11-12 23:17:25.173 | 2026-11-13 04:17:25.173 | `ACK_ALM` | `TableAlarms_008.RuntimeAlert` | `` | `nam` | `MxGateway` | Description = `OtOpcUa #614 H6 gate [OPC UA: h6gate, key a33c3b973768fbf5]`, Priority 400 | +| 2026-11-12 23:17:25.173 | 2026-11-13 04:17:25.173 | NULL (operator write event, Type `OPR`) | `TableAlarms_008.RuntimeAlert.AckMsg` | `` | `nam` | `MxGateway` | `Write success -`, Value = the comment | + +So the Historian holds the real operator (not a service identity) and the comment. The OPC UA principal and +key id are carried inside the comment. + +## Observations + +- **OPC UA shows Acknowledged before Galaxy confirms.** The subscriber saw `Active, Acknowledged` at + 04:17:22.5Z, about 2.6 s before the gateway reply and Galaxy's `04:17:25.173Z` transition. That is the #869 + design: the condition goes Acknowledged optimistically and reverts on a failure or confirm timeout. No + revert happened here. +- **Severity is still 500 in OPC UA** against Galaxy priority 400 and authored severity 400, as on 11-11. Not + investigated further (not part of H6). +- The gateway and Historian timestamps for the ack agree to the millisecond (`04:17:25.173Z`). After #226 + the raise time agrees too. + +## Cleanup + +- Deleted from the ConfigDb: `Tag h6gal-rtalert`, `Device h6gal-vd03`, `DriverInstance h6gal`, + `NodeAcl h6gate-acl` and `ApiKey h6-gate` (`a33c3b973768fbf5`). +- Cleanup deployed (`h6-gate-cleanup`, `840ff8db`). site-b-1 logged `GalaxyDriver h6gal shutting down` and + `disposed`. +- Deleted the gate `Deployment` row `b95c3609` and its `NodeDeploymentState` rows, so the account name is not + left in an artifact blob. No remaining deployment artifact mentions `h6gal`. +- site-b-1 was recreated from the plain compose file with the same image: the ack password is absent and the + gateway key is back to the committed default. site-b-2 was restarted. +- The scratch key file and override were removed, and the shell env was unset. +- The site-b-1 LocalDb cache may still hold a gate-era artifact until its next replacement.