Files
lmxopcua/docs/drivers/evidence/galaxy-h6-ack-2026-11-11.md
T

10 KiB

Galaxy H6 (#614): OPC UA ack of a real Galaxy native alarm to AVEVA history, 2026-11-11

Gitea #614 item H6: "an OPC UA Acknowledge of a REAL Galaxy native alarm reaches AVEVA event history".

Result: FAIL — the item cannot be ticked. The OPC UA path worked end to end inside OtOpcUa: the Part 9 Acknowledge returned Good, the client saw the condition go Active, Acknowledged, and the driver sent the ack to mxaccessgw as the owner-provided Galaxy account with no error. The acknowledgement never reached Galaxy. After the ack, Galaxy's own .Acked was still False, .AckMsg was still empty, the gateway feed still showed the alarm as unacknowledged, and AVEVA event history has no ACK row. So OtOpcUa now shows the condition acknowledged while Galaxy and the Historian do not (a silent-divergence defect, below).

Exactly one acknowledge was sent (approved write, 2026-11-11T17:37:55Z). It was not retried. No remote command ran on any wonder box. Galaxy access was through the gateway's gRPC API and read-only SQL only.

Target

Alarm Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert (object TableAlarms_008, contained as Right_004.TableAlarms), priority 400, "Run Time Exceeded"
Gateway wonder-app-vd03.zmr.zimmer.com:5120 (h2c)
Galaxy security OS/domain accounts (ZB.dbo.user_profile). Every 2026 ack in the Historian before this gate is a NAM\… operator from an InTouch node; none came through the gateway.
Historian wonder-sql-vd03 → Runtime.dbo.v_AlarmEventHistory2. Queries need an EventStamp range. There is no A2ALMDB or WWALMDB.

Setup (docker-dev, temporary, all removed afterwards)

Item Value
Node SITE-B site-b-1 (OPC UA opc.tcp://localhost:4844). Its partner site-b-2 was stopped for the run so site-b-1 was the sole (Primary) node.
Driver h6gal, DriverType GalaxyMxGateway, seeded straight into the ConfigDb. Gateway.Endpoint = the VD03 gateway, UseTls=false, ApiKeySecretRef=env:GALAXY_MXGW_API_KEY, MxAccess.ClientName=OtOpcUa-H6-gate, Repository.WatchDeployEvents=false. AlarmAck.FallbackUser/FallbackDomain = the owner-provided account (domain NAM); FallbackPasswordSecretRef=env:OTOPCUA_GALAXY_ACKPW
Device / tag device vd03 ({}), tag RuntimeAlert, Boolean, Read, TagConfig = {"attributeRef":"Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert","alarm":{"alarmType":"AlarmCondition","severity":400,"historizeToAveva":false}}. historizeToAveva:false keeps OtOpcUa's own alarm historian from writing anything to AVEVA.
Secrets the gateway key and the ack password were injected into the site-b-1 container as runtime env only, through a scratch compose override that held ${VAR} references, never values. Nothing secret went into the repo, the compose files, the ConfigDb or this file.
OPC UA client otopcua-cli (src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI), UserName apikey + a temporary data-plane key (h6-gate, principal h6gate, 4 h expiry) with a SITE-B Cluster grant: Browse, Read, Subscribe, HistoryRead, AlarmRead and AlarmAcknowledge (399). LDAP was not used.
Deploy POST /api/deployments. The first attempt used DriverType Galaxy, which is wrong: the registered name is GalaxyMxGateway. It was corrected and deployed again. Deployments to SITE-B end TimedOut while site-b-2 is stopped (the same as the Heidenhain gate); site-b-1 applied each one.

Driver start (site-b-1): GalaxyMxSession connected — clientName=OtOpcUa-H6-gate, DriverInstance h6gal: connected, native-alarm subscription established for 1 alarm ref(s).

Step 1: OPC UA, condition and Acknowledge (PASS within OtOpcUa)

alarms --refresh before the ack:

[2026-07-29T01:02:46.8300000Z] ALARM  h6gal/vd03/RuntimeAlert
  Condition: RuntimeAlert
  State:     Active, Unacknowledged
  Severity:  500
  Retain:    True
  ConditionNodeId: ns=2;s=h6gal/vd03/RuntimeAlert
  EventId:   8DB71E793FBD7040993404E30808FADC

The single ack, at 17:37:55Z:

otopcua-cli ack -u opc.tcp://localhost:4844 -n "ns=2;s=h6gal/vd03/RuntimeAlert" -e 8DB71E793FBD7040993404E30808FADC -c "OtOpcUa #614 H6 gate" -U apikey -P <temp key>
Acknowledge successful: ns=2;s=h6gal/vd03/RuntimeAlert

The live subscriber running during the ack saw:

[2026-11-11T17:37:55.8648690Z] ALARM  h6gal/vd03/RuntimeAlert
  State:     Active, Acknowledged
  Severity:  500
  Retain:    True
  EventId:   0F15D34C9D0B6D43B7A4B08F3A72D047

Driver log (account name redacted):

[17:37:55 INF] Galaxy driver h6gal: acknowledging Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert as Galaxy user NAM\<ack account> (Fallback) for OPC UA principal h6gate (key fa1d95d325ab9ab5)
[17:37:55 INF] DriverInstance h6gal: acknowledged native condition h6gal/vd03/RuntimeAlert by h6gate

No warning, refusal or revert followed. So the gateway answered PROTOCOL_STATUS_CODE_OK with hresult 0 (or absent): a non-OK status or a non-zero hresult throws and is logged. The comment the driver sent was OtOpcUa #614 H6 gate [OPC UA: h6gate, key fa1d95d325ab9ab5].

Step 2: Galaxy and the gateway alarm feed (FAIL)

QueryActiveAlarms (prefix = the alarm reference) about 30 s after the ack, unchanged from before:

"currentState": "ALARM_CONDITION_STATE_ACTIVE", "severity": 400,
"lastTransitionTimestamp": "2026-07-29T01:02:46.830Z", "operatorComment": "Run Time Exceeded"

Read-only gateway session (OpenSession → Register → ReadBulk → CloseSession), at 17:39:24Z and again at 17:40:18Z:

Attribute Value Quality
TableAlarms_008.RuntimeAlert.Acked False 192
TableAlarms_008.RuntimeAlert.AckMsg "" (empty) 192
TableAlarms_008.RuntimeAlert.InAlarm True (since 2026-07-28T20:02:46.830Z) 192

On the windev rig, mxaccessgw's gateway-level verification (AlarmProbeFindings.md) showed a successful ack turning .Acked True and leaving the comment in .AckMsg. Neither happened here.

Step 3: AVEVA event history (FAIL)

v_AlarmEventHistory2, EventStamp 2026-11-11 12:00 to 23:59 local, queried at 17:38, 17:39 and 17:46 UTC. The Historian is live: it stored other events at 17:00 UTC. Nothing for this alarm:

2026-11-11 12:00:30.922 | 17:00:30.922 | UNACK_ALM | DevWinPlatform.CPULoadHiAlarm | NULL | NULL
2026-11-11 12:00:50.918 | 17:00:50.918 | UNACK_RTN | DevWinPlatform.CPULoadHiAlarm | NULL | NULL

There is no ACK_ALM or ACK_RTN row for TableAlarms_008.RuntimeAlert and no row with OperatorNode=MxGateway.

Defect: OtOpcUa reports Acknowledged on an ack Galaxy never applied (reported, not fixed)

mxaccessgw returns OK/hresult 0 both for "write applied" and for "write sent, no OnWriteComplete within the wait (unconfirmed, with a diagnostic)" (mxaccessgw docs/Grpc.md, AcknowledgeAlarm outcome table). Its own docs say that "a client that must be certain should watch for the resulting transition on StreamAlarms".

GatewayGalaxyAlarmAcknowledger.EvaluateReply (src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Runtime/GatewayGalaxyAlarmAcknowledger.cs) treats OK + hresult 0 as success. It never reads or logs AcknowledgeAlarmReply.diagnostic_message, and nothing in the driver or host waits for the upstream Acknowledge transition. So the #760 revert never fires, and the OPC UA condition stays Acknowledged while Galaxy holds .Acked=False. A ScadaBridge or operator client would believe the alarm was handled.

Why the gateway-side write did not take effect cannot be determined from here: the gateway's logs are on wonder-app-vd03, and remote commands are barred. Candidates:

  • an unconfirmed AckMsg write that was later dropped or rejected;
  • the account lacking acknowledge permission on the object's security group;
  • a VD03 gateway build that predates the AckMsg path.

Not attempted again (one-ack rule). Suggested follow-ups (not filed, no Gitea per brief):

  • OtOpcUa: log diagnostic_message on every ack reply. Treat an unconfirmed ack as pending until the feed shows the Acknowledge transition, and revert on timeout.
  • mxaccessgw: expose applied vs unconfirmed as a field rather than only in a diagnostic.
  • Owner: check the VD03 gateway log around 2026-11-11T17:37:55Z for the ack's diagnostic.

Other observations (not filed)

  • The gateway snapshot timestamp is 5 h off. It reports lastTransitionTimestamp 2026-07-29T01:02:46.830Z. MXAccess's own InAlarm timestamp and the Historian's EventStampUTC both say 2026-07-28T20:02:46.830Z. Cross-repo, mxaccessgw.
  • The gateway snapshot and the Historian disagree on state. The gateway says ACTIVE since July, and .InAlarm agrees. The Historian logged five later "set off scan" UNACK_RTN rows (latest 2026-10-22) with no new UNACK_ALM.
  • Severity differs between the three views. The OPC UA condition shows Severity 500. The authored alarm.severity was 400, and so is the Galaxy priority.
  • site-b-2 never acknowledged the cleanup deploys. After cleanup, the standby site-b-2 (restarted and restored from its LocalDb at the same revision e221b179…) did not ack the re-dispatched deployment, so the reseal deploy also ended TimedOut (site-b-1 Applied).
  • The rig needed a nudge after the OrbStack restart. otopcua-dev-sql-1 and otopcua-dev-traefik-1 stayed Exited. I started them and restarted central-1 and central-2 (#868 ordering) before seeding.

Cleanup

  • Deleted from the ConfigDb:
    • Tag h6gal-rtalert, Device h6gal-vd03 and DriverInstance h6gal;
    • NodeAcl h6gate-acl and ApiKey h6-gate (fa1d95d325ab9ab5);
    • the two gate Deployment rows that carried the driver config, and their NodeDeploymentState rows, so the account name is not left in an artifact blob.
  • Cleanup deployed (h6-gate-cleanup, h6-gate-cleanup-reseal). site-b-1 logged GalaxyDriver h6gal shutting down … disposed.
  • site-b-1 recreated from the plain compose file: the ack password is absent and the gateway key is back to the committed default. site-b-2 restarted.
  • The scratch key file was deleted. The shell env was unset.
  • The site-b-1 LocalDb cache may still hold a gate-era artifact until its next replacement.