Files
lmxopcua/docs/drivers/evidence/galaxy-h6-ack-2026-11-11.md
T

171 lines
10 KiB
Markdown

# Galaxy H6 (#614): OPC UA ack of a real Galaxy native alarm to AVEVA history, 2026-11-11
Gitea #614 item H6: "an OPC UA Acknowledge of a REAL Galaxy native alarm reaches AVEVA event history".
**Result: FAIL — the item cannot be ticked.** The OPC UA path worked end to end inside OtOpcUa: the
Part 9 `Acknowledge` returned Good, the client saw the condition go `Active, Acknowledged`, and the driver
sent the ack to mxaccessgw as the owner-provided Galaxy account with no error. **The acknowledgement
never reached Galaxy.** After the ack, Galaxy's own `.Acked` was still `False`, `.AckMsg` was still
empty, the gateway feed still showed the alarm as unacknowledged, and AVEVA event history has no ACK
row. So OtOpcUa now shows the condition acknowledged while Galaxy and the Historian do not (a
silent-divergence defect, below).
Exactly **one** acknowledge was sent (approved write, 2026-11-11T17:37:55Z). It was not retried. No
remote command ran on any wonder box. Galaxy access was through the gateway's gRPC API and read-only
SQL only.
## Target
| | |
|---|---|
| Alarm | `Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert` (object `TableAlarms_008`, contained as `Right_004.TableAlarms`), priority 400, "Run Time Exceeded" |
| Gateway | `wonder-app-vd03.zmr.zimmer.com:5120` (h2c) |
| Galaxy security | OS/domain accounts (`ZB.dbo.user_profile`). Every 2026 ack in the Historian before this gate is a `NAM\…` operator from an InTouch node; none came through the gateway. |
| Historian | `wonder-sql-vd03` → `Runtime.dbo.v_AlarmEventHistory2`. Queries need an `EventStamp` range. There is no `A2ALMDB` or `WWALMDB`. |
## Setup (docker-dev, temporary, all removed afterwards)
| Item | Value |
|---|---|
| Node | SITE-B `site-b-1` (OPC UA `opc.tcp://localhost:4844`). Its partner `site-b-2` was stopped for the run so site-b-1 was the sole (Primary) node. |
| Driver | `h6gal`, DriverType `GalaxyMxGateway`, seeded straight into the ConfigDb. `Gateway.Endpoint` = the VD03 gateway, `UseTls=false`, `ApiKeySecretRef=env:GALAXY_MXGW_API_KEY`, `MxAccess.ClientName=OtOpcUa-H6-gate`, `Repository.WatchDeployEvents=false`. `AlarmAck.FallbackUser`/`FallbackDomain` = the owner-provided account (domain `NAM`); `FallbackPasswordSecretRef=env:OTOPCUA_GALAXY_ACKPW` |
| Device / tag | device `vd03` (`{}`), tag `RuntimeAlert`, Boolean, Read, `TagConfig` = `{"attributeRef":"Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert","alarm":{"alarmType":"AlarmCondition","severity":400,"historizeToAveva":false}}`. `historizeToAveva:false` keeps OtOpcUa's own alarm historian from writing anything to AVEVA. |
| Secrets | the gateway key and the ack password were injected into the site-b-1 container as runtime env only, through a scratch compose override that held `${VAR}` references, never values. Nothing secret went into the repo, the compose files, the ConfigDb or this file. |
| OPC UA client | `otopcua-cli` (`src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI`), UserName `apikey` + a temporary data-plane key (`h6-gate`, principal `h6gate`, 4 h expiry) with a SITE-B `Cluster` grant: Browse, Read, Subscribe, HistoryRead, AlarmRead and AlarmAcknowledge (399). LDAP was not used. |
| Deploy | `POST /api/deployments`. The first attempt used DriverType `Galaxy`, which is wrong: the registered name is `GalaxyMxGateway`. It was corrected and deployed again. Deployments to SITE-B end `TimedOut` while `site-b-2` is stopped (the same as the Heidenhain gate); `site-b-1` applied each one. |
Driver start (site-b-1): `GalaxyMxSession connected — clientName=OtOpcUa-H6-gate`, `DriverInstance h6gal: connected`,
`native-alarm subscription established for 1 alarm ref(s)`.
## Step 1: OPC UA, condition and Acknowledge (PASS within OtOpcUa)
`alarms --refresh` before the ack:
```
[2026-07-29T01:02:46.8300000Z] ALARM h6gal/vd03/RuntimeAlert
Condition: RuntimeAlert
State: Active, Unacknowledged
Severity: 500
Retain: True
ConditionNodeId: ns=2;s=h6gal/vd03/RuntimeAlert
EventId: 8DB71E793FBD7040993404E30808FADC
```
The single ack, at 17:37:55Z:
```
otopcua-cli ack -u opc.tcp://localhost:4844 -n "ns=2;s=h6gal/vd03/RuntimeAlert" -e 8DB71E793FBD7040993404E30808FADC -c "OtOpcUa #614 H6 gate" -U apikey -P <temp key>
Acknowledge successful: ns=2;s=h6gal/vd03/RuntimeAlert
```
The live subscriber running during the ack saw:
```
[2026-11-11T17:37:55.8648690Z] ALARM h6gal/vd03/RuntimeAlert
State: Active, Acknowledged
Severity: 500
Retain: True
EventId: 0F15D34C9D0B6D43B7A4B08F3A72D047
```
Driver log (account name redacted):
```
[17:37:55 INF] Galaxy driver h6gal: acknowledging Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert as Galaxy user NAM\<ack account> (Fallback) for OPC UA principal h6gate (key fa1d95d325ab9ab5)
[17:37:55 INF] DriverInstance h6gal: acknowledged native condition h6gal/vd03/RuntimeAlert by h6gate
```
No warning, refusal or revert followed. So the gateway answered `PROTOCOL_STATUS_CODE_OK` with
`hresult` 0 (or absent): a non-OK status or a non-zero hresult throws and is logged. The comment the
driver sent was `OtOpcUa #614 H6 gate [OPC UA: h6gate, key fa1d95d325ab9ab5]`.
## Step 2: Galaxy and the gateway alarm feed (FAIL)
`QueryActiveAlarms` (prefix = the alarm reference) about 30 s after the ack, unchanged from before:
```
"currentState": "ALARM_CONDITION_STATE_ACTIVE", "severity": 400,
"lastTransitionTimestamp": "2026-07-29T01:02:46.830Z", "operatorComment": "Run Time Exceeded"
```
Read-only gateway session (`OpenSession` → `Register` → `ReadBulk` → `CloseSession`), at 17:39:24Z and again at 17:40:18Z:
| Attribute | Value | Quality |
|---|---|---|
| `TableAlarms_008.RuntimeAlert.Acked` | **False** | 192 |
| `TableAlarms_008.RuntimeAlert.AckMsg` | **""** (empty) | 192 |
| `TableAlarms_008.RuntimeAlert.InAlarm` | True (since 2026-07-28T20:02:46.830Z) | 192 |
On the windev rig, mxaccessgw's gateway-level verification (`AlarmProbeFindings.md`) showed a
successful ack turning `.Acked` True and leaving the comment in `.AckMsg`. Neither happened here.
## Step 3: AVEVA event history (FAIL)
`v_AlarmEventHistory2`, `EventStamp` 2026-11-11 12:00 to 23:59 local, queried at 17:38, 17:39 and 17:46 UTC.
The Historian is live: it stored other events at 17:00 UTC. Nothing for this alarm:
```
2026-11-11 12:00:30.922 | 17:00:30.922 | UNACK_ALM | DevWinPlatform.CPULoadHiAlarm | NULL | NULL
2026-11-11 12:00:50.918 | 17:00:50.918 | UNACK_RTN | DevWinPlatform.CPULoadHiAlarm | NULL | NULL
```
There is no `ACK_ALM` or `ACK_RTN` row for `TableAlarms_008.RuntimeAlert` and no row with `OperatorNode=MxGateway`.
## Defect: OtOpcUa reports Acknowledged on an ack Galaxy never applied (reported, not fixed)
mxaccessgw returns `OK`/`hresult 0` both for "write applied" and for "write sent, no `OnWriteComplete`
within the wait (unconfirmed, with a diagnostic)" (mxaccessgw `docs/Grpc.md`, `AcknowledgeAlarm`
outcome table). Its own docs say that "a client that must be certain should watch for the resulting
transition on `StreamAlarms`".
`GatewayGalaxyAlarmAcknowledger.EvaluateReply`
(`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Runtime/GatewayGalaxyAlarmAcknowledger.cs`) treats OK +
hresult 0 as success. It never reads or logs `AcknowledgeAlarmReply.diagnostic_message`, and nothing
in the driver or host waits for the upstream Acknowledge transition. So the #760 revert never fires,
and the OPC UA condition stays Acknowledged while Galaxy holds `.Acked=False`. A ScadaBridge or
operator client would believe the alarm was handled.
Why the gateway-side write did not take effect cannot be determined from here: the gateway's logs are
on wonder-app-vd03, and remote commands are barred. Candidates:
- an unconfirmed `AckMsg` write that was later dropped or rejected;
- the account lacking acknowledge permission on the object's security group;
- a VD03 gateway build that predates the `AckMsg` path.
Not attempted again (one-ack rule). Suggested follow-ups (not filed, no Gitea per brief):
- **OtOpcUa:** log `diagnostic_message` on every ack reply. Treat an unconfirmed ack as pending until
the feed shows the Acknowledge transition, and revert on timeout.
- **mxaccessgw:** expose applied vs unconfirmed as a field rather than only in a diagnostic.
- **Owner:** check the VD03 gateway log around 2026-11-11T17:37:55Z for the ack's diagnostic.
## Other observations (not filed)
- **The gateway snapshot timestamp is 5 h off.** It reports `lastTransitionTimestamp
2026-07-29T01:02:46.830Z`. MXAccess's own `InAlarm` timestamp and the Historian's `EventStampUTC`
both say 2026-07-28T20:02:46.830Z. Cross-repo, mxaccessgw.
- **The gateway snapshot and the Historian disagree on state.** The gateway says ACTIVE since July,
and `.InAlarm` agrees. The Historian logged five later "set off scan" `UNACK_RTN` rows (latest
2026-10-22) with no new `UNACK_ALM`.
- **Severity differs between the three views.** The OPC UA condition shows Severity 500. The authored
`alarm.severity` was 400, and so is the Galaxy priority.
- **site-b-2 never acknowledged the cleanup deploys.** After cleanup, the standby `site-b-2` (restarted
and restored from its LocalDb at the same revision `e221b179…`) did not ack the re-dispatched
deployment, so the reseal deploy also ended `TimedOut` (site-b-1 Applied).
- **The rig needed a nudge after the OrbStack restart.** `otopcua-dev-sql-1` and
`otopcua-dev-traefik-1` stayed Exited. I started them and restarted central-1 and central-2 (#868
ordering) before seeding.
## Cleanup
- Deleted from the ConfigDb:
- `Tag h6gal-rtalert`, `Device h6gal-vd03` and `DriverInstance h6gal`;
- `NodeAcl h6gate-acl` and `ApiKey h6-gate` (`fa1d95d325ab9ab5`);
- the two gate `Deployment` rows that carried the driver config, and their `NodeDeploymentState` rows,
so the account name is not left in an artifact blob.
- Cleanup deployed (`h6-gate-cleanup`, `h6-gate-cleanup-reseal`). site-b-1 logged `GalaxyDriver h6gal shutting down … disposed`.
- site-b-1 recreated from the plain compose file: the ack password is absent and the gateway key is back
to the committed default. site-b-2 restarted.
- The scratch key file was deleted. The shell env was unset.
- The site-b-1 LocalDb cache may still hold a gate-era artifact until its next replacement.