One owner-approved ack of CVDAisle_1.TableAlarms_008.RuntimeAlert at 2026-11-13T04:17:22Z through docker-dev site-b-1 -> vd03 mxaccessgw 76a71bc8. .Acked True, .AckMsg = comment, gateway feed ACTIVE_ACKED, v_AlarmEventHistory2 ACK_ALM with the NAM operator and OperatorNode MxGateway. The 11-11 failure was the vd03 gateway build (4f931e4, pre-AckMsg wnwrap AlarmAckByName no-op), not #226/#228. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
10 KiB
Galaxy H6 (#614): OPC UA ack of a real Galaxy native alarm to AVEVA history, 2026-11-12 (re-run)
Gitea #614 item H6: "an OPC UA Acknowledge of a REAL Galaxy native alarm reaches AVEVA event history".
Result: PASS. One OPC UA Acknowledge sent through OtOpcUa to the DEV mxaccessgw on wonder-app-vd03
reached every layer: the OPC UA condition, the gateway alarm feed, Galaxy (.Acked True, .AckMsg holds
the comment) and AVEVA event history (an ACK_ALM row with the right operator, domain, node and comment).
Exactly one acknowledge was sent (owner-approved write, 2026-11-13T04:17:22Z, i.e. 2026-11-12 23:17 EST). No retry was needed. No remote command ran on any wonder box: Galaxy was reached only through the gateway's gRPC API, and the Historian only through read-only SQL.
The failed first run is recorded in galaxy-h6-ack-2026-11-11.md; this run reused its recipe.
Why the 2026-11-11 ack didn't apply (pre-ack analysis)
The question was whether mxaccessgw#226 (alarm times 4–5 h late) or #228 (Write2 time argument read as
local) could explain the failure. Neither does, from the mxaccessgw source:
- The gateway does not match an ack by alarm time.
GatewayAlarmMonitorturnsalarm_full_referenceGalaxy!CVDAisle_1.TableAlarms_008.RuntimeAlertintoAcknowledgeAlarmByName(providerGalaxy, groupCVDAisle_1, alarmTableAlarms_008.RuntimeAlert). The worker'sWnWrapAlarmConsumer.ResolveAckTarget*then maps that to the itemTableAlarms_008.RuntimeAlert.AckMsgby name only. No timestamp is involved. AlarmAckWriterwrites the comment with a plain MXAccessWrite(..., userId)afterAuthenticateUser. It does not useWrite2, so #228 does not touch it.
The actual cause was the gateway build. At 17:37:55Z on 2026-11-11, vd03 was still running mxgw 4f931e4
(2026-09-03). The vd03 upgrade to 2b13564e happened later that day, between about 21:37Z and 21:51Z (see
.claude/skills/sister-project-history/otopcua.md in scadaproj, and mxgw docs/deploy/). 4f931e4
predates the operator-authenticated ack (488649c, PR #162, 2026-10-20). In that build, an ack was
wwAlarmConsumerClass.AlarmAckByName/AlarmAckByGUID. mxgw's own source and
docs/AlarmProbeFindings.md say that path returns rc=0 when the call is accepted, not when it is
applied. It carries an operator name but no authenticated user id, so under Galaxy security it cannot
satisfy the permission check and does nothing. That matches what 11-11 saw: OK/hresult 0 and no change in
Galaxy. The OPC UA proto's operator_password field didn't exist in 4f931e4, so that build never saw the
password.
vd03 now runs 76a71bc8 (mxaccessgw#229, deployed 2026-11-12 ~02:19Z), which has the AuthenticateUser +
AckMsg path. No other un-fixed cause made a second failure certain, so the ack went ahead. The remaining
risks (bad credentials, no ack permission, no OnWriteComplete) all fail visibly now, and OtOpcUa reverts
them after #869.
Setup (docker-dev, temporary, all removed afterwards)
| Item | Value |
|---|---|
| OtOpcUa image | otopcua-host:dev sha256:5094b682cef5 (built 2026-11-12 16:14Z). It contains the #869 fix: GalaxyAckConfirmationTracker and AlarmAck.ConfirmTimeoutMs are present in ZB.MOM.WW.OtOpcUa.Driver.Galaxy.dll. Not rebuilt for this run. |
| Gateway | wonder-app-vd03.zmr.zimmer.com:5120 (h2c), mxgw main 76a71bc8; /health/ready Healthy before the run |
| Node | SITE-B site-b-1 (OPC UA opc.tcp://localhost:4844). site-b-2 was stopped for the run, so site-b-1 was the sole (Primary) node. |
| Driver | h6gal, DriverType GalaxyMxGateway, seeded straight into the ConfigDb. Gateway.Endpoint = the vd03 gateway, UseTls=false, ApiKeySecretRef=env:GALAXY_MXGW_API_KEY, MxAccess.ClientName=OtOpcUa-H6-gate, Repository.WatchDeployEvents=false, AlarmAck.FallbackUser/FallbackDomain = the owner-provided account (domain NAM), FallbackPasswordSecretRef=env:OTOPCUA_GALAXY_ACKPW. ConfirmTimeoutMs was left at the default (10 s). |
| Device / tag | device vd03, tag RuntimeAlert, Boolean, Read, attributeRef Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert, alarm = AlarmCondition, severity 400, historizeToAveva:false (so OtOpcUa's own alarm historian wrote nothing to AVEVA) |
| Secrets | The gateway key and the ack password came from the owner's local env file. They were injected into site-b-1 as runtime env only, through a scratch compose override holding ${VAR} references. Nothing secret went into the repo, the compose files, the ConfigDb (only the env: reference) or this file. |
| OPC UA client | otopcua-cli built from 85db1dbf. UserName apikey + a temporary data-plane key (h6-gate, keyId a33c3b973768fbf5, principal h6gate, 4 h expiry). The key had a SITE-B Cluster grant of 399: Browse, Read, Subscribe, HistoryRead, AlarmRead and AlarmAcknowledge. |
| Deploy | POST /api/deployments (b95c3609). site-b-1 Applied. The deployment stayed AwaitingApplyAcks because site-b-2 was stopped, as on 11-11. |
Driver start (site-b-1, 04:16:32Z): GalaxyMxSession connected — clientName=OtOpcUa-H6-gate,
DriverInstance h6gal: connected, native-alarm subscription established for 1 alarm ref(s).
Target, before the ack
Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert was still active and unacknowledged, so it was used again.
Z28061 was not touched.
| Check (before) | Value |
|---|---|
QueryActiveAlarms |
ALARM_CONDITION_STATE_ACTIVE, severity 400, lastTransitionTimestamp 2026-07-28T20:02:46.830Z. This is now correct UTC: #226 is visible end to end, and on 11-11 the value was 01:02:46.830Z on 07-29. |
.Acked / .AckMsg / .InAlarm (read-only gateway session, 04:14:47Z) |
False / "" / True (since 2026-07-28T20:02:46.830Z), quality 192 |
v_AlarmEventHistory2 |
only the 07-28 UNACK_ALM and later UNACK_RTN rows; no ACK row |
OPC UA alarms --refresh |
[2026-07-28T20:02:46.8300000Z] h6gal/vd03/RuntimeAlert, Active, Unacknowledged, EventId 4966AAB18408A24F9BB3CEC025FCA77E |
The single ack
2026-11-13T04:17:22Z
otopcua-cli ack -u opc.tcp://localhost:4844 -n "ns=2;s=h6gal/vd03/RuntimeAlert" -e 4966AAB18408A24F9BB3CEC025FCA77E -c "OtOpcUa #614 H6 gate" -U apikey -P <temp key>
Acknowledge successful: ns=2;s=h6gal/vd03/RuntimeAlert
Results per layer
1. OPC UA (PASS)
The live subscriber saw the condition go Active, Acknowledged at 04:17:22.546Z (EventId
E260F1DB097FC543A90AE478FF5E928C). It saw no later revert to Unacknowledged in the 90 s window.
Driver log (account name redacted):
[04:17:22 INF] Galaxy driver h6gal: acknowledging Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert as Galaxy user NAM\<ack account> (Fallback) for OPC UA principal h6gate (key a33c3b973768fbf5)
[04:17:25 INF] DriverInstance h6gal: acknowledged native condition h6gal/vd03/RuntimeAlert by h6gate
There was no gateway diagnostic, no Unconfirmed and no BadTimeout revert. The ~3 s between the two lines
is the gateway waiting on MXAccess OnWriteComplete. On 11-11 the same pair of lines was logged within the
same second.
2. Gateway alarm feed (PASS)
QueryActiveAlarms at 04:19:22Z:
"currentState": "ALARM_CONDITION_STATE_ACTIVE_ACKED", "severity": 400,
"lastTransitionTimestamp": "2026-11-13T04:17:25.173Z",
"operatorUser": "<ack account>",
"operatorComment": "OtOpcUa #614 H6 gate [OPC UA: h6gate, key a33c3b973768fbf5]"
3. Galaxy (PASS)
Read-only gateway session (OpenSession → Register → ReadBulk → CloseSession) at 04:19:20Z:
| Attribute | Value | Quality |
|---|---|---|
TableAlarms_008.RuntimeAlert.Acked |
True | 192 |
TableAlarms_008.RuntimeAlert.AckMsg |
OtOpcUa #614 H6 gate [OPC UA: h6gate, key a33c3b973768fbf5] |
192 |
TableAlarms_008.RuntimeAlert.InAlarm |
True (since 2026-07-28T20:02:46.830Z) | 192 |
4. AVEVA event history (PASS)
Runtime.dbo.v_AlarmEventHistory2 on wonder-sql-vd03, EventStamp 2026-11-12 22:00 to 2026-11-13 02:00 local
(read-only). Operator is the ack account (redacted here), and UserFullName (redacted) is that account's
directory name:
| EventStamp (local) | EventStampUTC | AlarmState | TagName | Operator | DomainName | OperatorNode | Description / Value |
|---|---|---|---|---|---|---|---|
| 2026-11-12 23:17:25.173 | 2026-11-13 04:17:25.173 | ACK_ALM |
TableAlarms_008.RuntimeAlert |
<ack account> |
nam |
MxGateway |
Description = OtOpcUa #614 H6 gate [OPC UA: h6gate, key a33c3b973768fbf5], Priority 400 |
| 2026-11-12 23:17:25.173 | 2026-11-13 04:17:25.173 | NULL (operator write event, Type OPR) |
TableAlarms_008.RuntimeAlert.AckMsg |
<ack account> |
nam |
MxGateway |
Write success -, Value = the comment |
So the Historian holds the real operator (not a service identity) and the comment. The OPC UA principal and key id are carried inside the comment.
Observations
- OPC UA shows Acknowledged before Galaxy confirms. The subscriber saw
Active, Acknowledgedat 04:17:22.5Z, about 2.6 s before the gateway reply and Galaxy's04:17:25.173Ztransition. That is the #869 design: the condition goes Acknowledged optimistically and reverts on a failure or confirm timeout. No revert happened here. - Severity is still 500 in OPC UA against Galaxy priority 400 and authored severity 400, as on 11-11. Not investigated further (not part of H6).
- The gateway and Historian timestamps for the ack agree to the millisecond (
04:17:25.173Z). After #226 the raise time agrees too.
Cleanup
- Deleted from the ConfigDb:
Tag h6gal-rtalert,Device h6gal-vd03,DriverInstance h6gal,NodeAcl h6gate-aclandApiKey h6-gate(a33c3b973768fbf5). - Cleanup deployed (
h6-gate-cleanup,840ff8db). site-b-1 loggedGalaxyDriver h6gal shutting downanddisposed. - Deleted the gate
Deploymentrowb95c3609and itsNodeDeploymentStaterows, so the account name is not left in an artifact blob. No remaining deployment artifact mentionsh6gal. - site-b-1 was recreated from the plain compose file with the same image: the ack password is absent and the gateway key is back to the committed default. site-b-2 was restarted.
- The scratch key file and override were removed, and the shell env was unset.
- The site-b-1 LocalDb cache may still hold a gate-era artifact until its next replacement.