Files
lmxopcua/docs/drivers/evidence/galaxy-h6-ack-2026-11-12.md
T
Joseph Doherty 8e846f0e62
v2-ci / unit-tests (push) Has been cancelled
docs(#614): H6 PASS — OPC UA ack of a real Galaxy alarm reached Galaxy and AVEVA history
One owner-approved ack of CVDAisle_1.TableAlarms_008.RuntimeAlert at
2026-11-13T04:17:22Z through docker-dev site-b-1 -> vd03 mxaccessgw 76a71bc8.
.Acked True, .AckMsg = comment, gateway feed ACTIVE_ACKED, v_AlarmEventHistory2
ACK_ALM with the NAM operator and OperatorNode MxGateway. The 11-11 failure
was the vd03 gateway build (4f931e4, pre-AckMsg wnwrap AlarmAckByName no-op),
not #226/#228.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:21:47 -05:00

10 KiB
Raw Blame History

Galaxy H6 (#614): OPC UA ack of a real Galaxy native alarm to AVEVA history, 2026-11-12 (re-run)

Gitea #614 item H6: "an OPC UA Acknowledge of a REAL Galaxy native alarm reaches AVEVA event history".

Result: PASS. One OPC UA Acknowledge sent through OtOpcUa to the DEV mxaccessgw on wonder-app-vd03 reached every layer: the OPC UA condition, the gateway alarm feed, Galaxy (.Acked True, .AckMsg holds the comment) and AVEVA event history (an ACK_ALM row with the right operator, domain, node and comment).

Exactly one acknowledge was sent (owner-approved write, 2026-11-13T04:17:22Z, i.e. 2026-11-12 23:17 EST). No retry was needed. No remote command ran on any wonder box: Galaxy was reached only through the gateway's gRPC API, and the Historian only through read-only SQL.

The failed first run is recorded in galaxy-h6-ack-2026-11-11.md; this run reused its recipe.

Why the 2026-11-11 ack didn't apply (pre-ack analysis)

The question was whether mxaccessgw#226 (alarm times 4–5 h late) or #228 (Write2 time argument read as local) could explain the failure. Neither does, from the mxaccessgw source:

  • The gateway does not match an ack by alarm time. GatewayAlarmMonitor turns alarm_full_reference Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert into AcknowledgeAlarmByName (provider Galaxy, group CVDAisle_1, alarm TableAlarms_008.RuntimeAlert). The worker's WnWrapAlarmConsumer.ResolveAckTarget* then maps that to the item TableAlarms_008.RuntimeAlert.AckMsg by name only. No timestamp is involved.
  • AlarmAckWriter writes the comment with a plain MXAccess Write(..., userId) after AuthenticateUser. It does not use Write2, so #228 does not touch it.

The actual cause was the gateway build. At 17:37:55Z on 2026-11-11, vd03 was still running mxgw 4f931e4 (2026-09-03). The vd03 upgrade to 2b13564e happened later that day, between about 21:37Z and 21:51Z (see .claude/skills/sister-project-history/otopcua.md in scadaproj, and mxgw docs/deploy/). 4f931e4 predates the operator-authenticated ack (488649c, PR #162, 2026-10-20). In that build, an ack was wwAlarmConsumerClass.AlarmAckByName/AlarmAckByGUID. mxgw's own source and docs/AlarmProbeFindings.md say that path returns rc=0 when the call is accepted, not when it is applied. It carries an operator name but no authenticated user id, so under Galaxy security it cannot satisfy the permission check and does nothing. That matches what 11-11 saw: OK/hresult 0 and no change in Galaxy. The OPC UA proto's operator_password field didn't exist in 4f931e4, so that build never saw the password.

vd03 now runs 76a71bc8 (mxaccessgw#229, deployed 2026-11-12 ~02:19Z), which has the AuthenticateUser + AckMsg path. No other un-fixed cause made a second failure certain, so the ack went ahead. The remaining risks (bad credentials, no ack permission, no OnWriteComplete) all fail visibly now, and OtOpcUa reverts them after #869.

Setup (docker-dev, temporary, all removed afterwards)

Item Value
OtOpcUa image otopcua-host:dev sha256:5094b682cef5 (built 2026-11-12 16:14Z). It contains the #869 fix: GalaxyAckConfirmationTracker and AlarmAck.ConfirmTimeoutMs are present in ZB.MOM.WW.OtOpcUa.Driver.Galaxy.dll. Not rebuilt for this run.
Gateway wonder-app-vd03.zmr.zimmer.com:5120 (h2c), mxgw main 76a71bc8; /health/ready Healthy before the run
Node SITE-B site-b-1 (OPC UA opc.tcp://localhost:4844). site-b-2 was stopped for the run, so site-b-1 was the sole (Primary) node.
Driver h6gal, DriverType GalaxyMxGateway, seeded straight into the ConfigDb. Gateway.Endpoint = the vd03 gateway, UseTls=false, ApiKeySecretRef=env:GALAXY_MXGW_API_KEY, MxAccess.ClientName=OtOpcUa-H6-gate, Repository.WatchDeployEvents=false, AlarmAck.FallbackUser/FallbackDomain = the owner-provided account (domain NAM), FallbackPasswordSecretRef=env:OTOPCUA_GALAXY_ACKPW. ConfirmTimeoutMs was left at the default (10 s).
Device / tag device vd03, tag RuntimeAlert, Boolean, Read, attributeRef Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert, alarm = AlarmCondition, severity 400, historizeToAveva:false (so OtOpcUa's own alarm historian wrote nothing to AVEVA)
Secrets The gateway key and the ack password came from the owner's local env file. They were injected into site-b-1 as runtime env only, through a scratch compose override holding ${VAR} references. Nothing secret went into the repo, the compose files, the ConfigDb (only the env: reference) or this file.
OPC UA client otopcua-cli built from 85db1dbf. UserName apikey + a temporary data-plane key (h6-gate, keyId a33c3b973768fbf5, principal h6gate, 4 h expiry). The key had a SITE-B Cluster grant of 399: Browse, Read, Subscribe, HistoryRead, AlarmRead and AlarmAcknowledge.
Deploy POST /api/deployments (b95c3609). site-b-1 Applied. The deployment stayed AwaitingApplyAcks because site-b-2 was stopped, as on 11-11.

Driver start (site-b-1, 04:16:32Z): GalaxyMxSession connected — clientName=OtOpcUa-H6-gate, DriverInstance h6gal: connected, native-alarm subscription established for 1 alarm ref(s).

Target, before the ack

Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert was still active and unacknowledged, so it was used again. Z28061 was not touched.

Check (before) Value
QueryActiveAlarms ALARM_CONDITION_STATE_ACTIVE, severity 400, lastTransitionTimestamp 2026-07-28T20:02:46.830Z. This is now correct UTC: #226 is visible end to end, and on 11-11 the value was 01:02:46.830Z on 07-29.
.Acked / .AckMsg / .InAlarm (read-only gateway session, 04:14:47Z) False / "" / True (since 2026-07-28T20:02:46.830Z), quality 192
v_AlarmEventHistory2 only the 07-28 UNACK_ALM and later UNACK_RTN rows; no ACK row
OPC UA alarms --refresh [2026-07-28T20:02:46.8300000Z] h6gal/vd03/RuntimeAlert, Active, Unacknowledged, EventId 4966AAB18408A24F9BB3CEC025FCA77E

The single ack

2026-11-13T04:17:22Z
otopcua-cli ack -u opc.tcp://localhost:4844 -n "ns=2;s=h6gal/vd03/RuntimeAlert" -e 4966AAB18408A24F9BB3CEC025FCA77E -c "OtOpcUa #614 H6 gate" -U apikey -P <temp key>
Acknowledge successful: ns=2;s=h6gal/vd03/RuntimeAlert

Results per layer

1. OPC UA (PASS)

The live subscriber saw the condition go Active, Acknowledged at 04:17:22.546Z (EventId E260F1DB097FC543A90AE478FF5E928C). It saw no later revert to Unacknowledged in the 90 s window.

Driver log (account name redacted):

[04:17:22 INF] Galaxy driver h6gal: acknowledging Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert as Galaxy user NAM\<ack account> (Fallback) for OPC UA principal h6gate (key a33c3b973768fbf5)
[04:17:25 INF] DriverInstance h6gal: acknowledged native condition h6gal/vd03/RuntimeAlert by h6gate

There was no gateway diagnostic, no Unconfirmed and no BadTimeout revert. The ~3 s between the two lines is the gateway waiting on MXAccess OnWriteComplete. On 11-11 the same pair of lines was logged within the same second.

2. Gateway alarm feed (PASS)

QueryActiveAlarms at 04:19:22Z:

"currentState": "ALARM_CONDITION_STATE_ACTIVE_ACKED", "severity": 400,
"lastTransitionTimestamp": "2026-11-13T04:17:25.173Z",
"operatorUser": "<ack account>",
"operatorComment": "OtOpcUa #614 H6 gate [OPC UA: h6gate, key a33c3b973768fbf5]"

3. Galaxy (PASS)

Read-only gateway session (OpenSession → Register → ReadBulk → CloseSession) at 04:19:20Z:

Attribute Value Quality
TableAlarms_008.RuntimeAlert.Acked True 192
TableAlarms_008.RuntimeAlert.AckMsg OtOpcUa #614 H6 gate [OPC UA: h6gate, key a33c3b973768fbf5] 192
TableAlarms_008.RuntimeAlert.InAlarm True (since 2026-07-28T20:02:46.830Z) 192

4. AVEVA event history (PASS)

Runtime.dbo.v_AlarmEventHistory2 on wonder-sql-vd03, EventStamp 2026-11-12 22:00 to 2026-11-13 02:00 local (read-only). Operator is the ack account (redacted here), and UserFullName (redacted) is that account's directory name:

EventStamp (local) EventStampUTC AlarmState TagName Operator DomainName OperatorNode Description / Value
2026-11-12 23:17:25.173 2026-11-13 04:17:25.173 ACK_ALM TableAlarms_008.RuntimeAlert <ack account> nam MxGateway Description = OtOpcUa #614 H6 gate [OPC UA: h6gate, key a33c3b973768fbf5], Priority 400
2026-11-12 23:17:25.173 2026-11-13 04:17:25.173 NULL (operator write event, Type OPR) TableAlarms_008.RuntimeAlert.AckMsg <ack account> nam MxGateway Write success -, Value = the comment

So the Historian holds the real operator (not a service identity) and the comment. The OPC UA principal and key id are carried inside the comment.

Observations

  • OPC UA shows Acknowledged before Galaxy confirms. The subscriber saw Active, Acknowledged at 04:17:22.5Z, about 2.6 s before the gateway reply and Galaxy's 04:17:25.173Z transition. That is the #869 design: the condition goes Acknowledged optimistically and reverts on a failure or confirm timeout. No revert happened here.
  • Severity is still 500 in OPC UA against Galaxy priority 400 and authored severity 400, as on 11-11. Not investigated further (not part of H6).
  • The gateway and Historian timestamps for the ack agree to the millisecond (04:17:25.173Z). After #226 the raise time agrees too.

Cleanup

  • Deleted from the ConfigDb: Tag h6gal-rtalert, Device h6gal-vd03, DriverInstance h6gal, NodeAcl h6gate-acl and ApiKey h6-gate (a33c3b973768fbf5).
  • Cleanup deployed (h6-gate-cleanup, 840ff8db). site-b-1 logged GalaxyDriver h6gal shutting down and disposed.
  • Deleted the gate Deployment row b95c3609 and its NodeDeploymentState rows, so the account name is not left in an artifact blob. No remaining deployment artifact mentions h6gal.
  • site-b-1 was recreated from the plain compose file with the same image: the ack password is absent and the gateway key is back to the committed default. site-b-2 was restarted.
  • The scratch key file and override were removed, and the shell env was unset.
  • The site-b-1 LocalDb cache may still hold a gate-era artifact until its next replacement.