docs(#614): H6 PASS — OPC UA ack of a real Galaxy alarm reached Galaxy and AVEVA history
v2-ci / unit-tests (push) Has been cancelled
v2-ci / unit-tests (push) Has been cancelled
One owner-approved ack of CVDAisle_1.TableAlarms_008.RuntimeAlert at 2026-11-13T04:17:22Z through docker-dev site-b-1 -> vd03 mxaccessgw 76a71bc8. .Acked True, .AckMsg = comment, gateway feed ACTIVE_ACKED, v_AlarmEventHistory2 ACK_ALM with the NAM operator and OperatorNode MxGateway. The 11-11 failure was the vd03 gateway build (4f931e4, pre-AckMsg wnwrap AlarmAckByName no-op), not #226/#228. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,154 @@
|
||||
# Galaxy H6 (#614): OPC UA ack of a real Galaxy native alarm to AVEVA history, 2026-11-12 (re-run)
|
||||
|
||||
Gitea #614 item H6: "an OPC UA Acknowledge of a REAL Galaxy native alarm reaches AVEVA event history".
|
||||
|
||||
**Result: PASS.** One OPC UA `Acknowledge` sent through OtOpcUa to the DEV mxaccessgw on wonder-app-vd03
|
||||
reached every layer: the OPC UA condition, the gateway alarm feed, Galaxy (`.Acked` True, `.AckMsg` holds
|
||||
the comment) and AVEVA event history (an `ACK_ALM` row with the right operator, domain, node and comment).
|
||||
|
||||
Exactly **one** acknowledge was sent (owner-approved write, 2026-11-13T04:17:22Z, i.e. 2026-11-12 23:17
|
||||
EST). No retry was needed. No remote command ran on any wonder box: Galaxy was reached only through the
|
||||
gateway's gRPC API, and the Historian only through read-only SQL.
|
||||
|
||||
The failed first run is recorded in `galaxy-h6-ack-2026-11-11.md`; this run reused its recipe.
|
||||
|
||||
## Why the 2026-11-11 ack didn't apply (pre-ack analysis)
|
||||
|
||||
The question was whether mxaccessgw#226 (alarm times 4–5 h late) or #228 (`Write2` time argument read as
|
||||
local) could explain the failure. **Neither does**, from the mxaccessgw source:
|
||||
|
||||
- The gateway does not match an ack by alarm time. `GatewayAlarmMonitor` turns `alarm_full_reference`
|
||||
`Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert` into `AcknowledgeAlarmByName` (provider `Galaxy`, group
|
||||
`CVDAisle_1`, alarm `TableAlarms_008.RuntimeAlert`). The worker's `WnWrapAlarmConsumer.ResolveAckTarget*`
|
||||
then maps that to the item `TableAlarms_008.RuntimeAlert.AckMsg` by name only. No timestamp is involved.
|
||||
- `AlarmAckWriter` writes the comment with a plain MXAccess `Write(..., userId)` after `AuthenticateUser`.
|
||||
It does not use `Write2`, so #228 does not touch it.
|
||||
|
||||
**The actual cause was the gateway build.** At 17:37:55Z on 2026-11-11, vd03 was still running mxgw `4f931e4`
|
||||
(2026-09-03). The vd03 upgrade to `2b13564e` happened later that day, between about 21:37Z and 21:51Z (see
|
||||
`.claude/skills/sister-project-history/otopcua.md` in scadaproj, and mxgw `docs/deploy/`). `4f931e4`
|
||||
predates the operator-authenticated ack (`488649c`, PR #162, 2026-10-20). In that build, an ack was
|
||||
`wwAlarmConsumerClass.AlarmAckByName`/`AlarmAckByGUID`. mxgw's own source and
|
||||
`docs/AlarmProbeFindings.md` say that path returns `rc=0` when the call is *accepted*, not when it is
|
||||
*applied*. It carries an operator name but no authenticated user id, so under Galaxy security it cannot
|
||||
satisfy the permission check and does nothing. That matches what 11-11 saw: OK/hresult 0 and no change in
|
||||
Galaxy. The OPC UA proto's `operator_password` field didn't exist in `4f931e4`, so that build never saw the
|
||||
password.
|
||||
|
||||
vd03 now runs `76a71bc8` (mxaccessgw#229, deployed 2026-11-12 ~02:19Z), which has the `AuthenticateUser` +
|
||||
`AckMsg` path. No other un-fixed cause made a second failure certain, so the ack went ahead. The remaining
|
||||
risks (bad credentials, no ack permission, no `OnWriteComplete`) all fail visibly now, and OtOpcUa reverts
|
||||
them after #869.
|
||||
|
||||
## Setup (docker-dev, temporary, all removed afterwards)
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| OtOpcUa image | `otopcua-host:dev` `sha256:5094b682cef5` (built 2026-11-12 16:14Z). It contains the #869 fix: `GalaxyAckConfirmationTracker` and `AlarmAck.ConfirmTimeoutMs` are present in `ZB.MOM.WW.OtOpcUa.Driver.Galaxy.dll`. Not rebuilt for this run. |
|
||||
| Gateway | `wonder-app-vd03.zmr.zimmer.com:5120` (h2c), mxgw main `76a71bc8`; `/health/ready` Healthy before the run |
|
||||
| Node | SITE-B `site-b-1` (OPC UA `opc.tcp://localhost:4844`). `site-b-2` was stopped for the run, so site-b-1 was the sole (Primary) node. |
|
||||
| Driver | `h6gal`, DriverType `GalaxyMxGateway`, seeded straight into the ConfigDb. `Gateway.Endpoint` = the vd03 gateway, `UseTls=false`, `ApiKeySecretRef=env:GALAXY_MXGW_API_KEY`, `MxAccess.ClientName=OtOpcUa-H6-gate`, `Repository.WatchDeployEvents=false`, `AlarmAck.FallbackUser`/`FallbackDomain` = the owner-provided account (domain `NAM`), `FallbackPasswordSecretRef=env:OTOPCUA_GALAXY_ACKPW`. `ConfirmTimeoutMs` was left at the default (10 s). |
|
||||
| Device / tag | device `vd03`, tag `RuntimeAlert`, Boolean, Read, `attributeRef` `Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert`, `alarm` = `AlarmCondition`, severity 400, `historizeToAveva:false` (so OtOpcUa's own alarm historian wrote nothing to AVEVA) |
|
||||
| Secrets | The gateway key and the ack password came from the owner's local env file. They were injected into site-b-1 as runtime env only, through a scratch compose override holding `${VAR}` references. Nothing secret went into the repo, the compose files, the ConfigDb (only the `env:` reference) or this file. |
|
||||
| OPC UA client | `otopcua-cli` built from `85db1dbf`. UserName `apikey` + a temporary data-plane key (`h6-gate`, keyId `a33c3b973768fbf5`, principal `h6gate`, 4 h expiry). The key had a SITE-B `Cluster` grant of 399: Browse, Read, Subscribe, HistoryRead, AlarmRead and AlarmAcknowledge. |
|
||||
| Deploy | `POST /api/deployments` (`b95c3609`). site-b-1 Applied. The deployment stayed `AwaitingApplyAcks` because site-b-2 was stopped, as on 11-11. |
|
||||
|
||||
Driver start (site-b-1, 04:16:32Z): `GalaxyMxSession connected — clientName=OtOpcUa-H6-gate`,
|
||||
`DriverInstance h6gal: connected`, `native-alarm subscription established for 1 alarm ref(s)`.
|
||||
|
||||
## Target, before the ack
|
||||
|
||||
`Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert` was still active and unacknowledged, so it was used again.
|
||||
Z28061 was not touched.
|
||||
|
||||
| Check (before) | Value |
|
||||
|---|---|
|
||||
| `QueryActiveAlarms` | `ALARM_CONDITION_STATE_ACTIVE`, severity 400, `lastTransitionTimestamp 2026-07-28T20:02:46.830Z`. This is now correct UTC: #226 is visible end to end, and on 11-11 the value was `01:02:46.830Z` on 07-29. |
|
||||
| `.Acked` / `.AckMsg` / `.InAlarm` (read-only gateway session, 04:14:47Z) | `False` / `""` / `True` (since 2026-07-28T20:02:46.830Z), quality 192 |
|
||||
| `v_AlarmEventHistory2` | only the 07-28 `UNACK_ALM` and later `UNACK_RTN` rows; no ACK row |
|
||||
| OPC UA `alarms --refresh` | `[2026-07-28T20:02:46.8300000Z] h6gal/vd03/RuntimeAlert`, `Active, Unacknowledged`, EventId `4966AAB18408A24F9BB3CEC025FCA77E` |
|
||||
|
||||
## The single ack
|
||||
|
||||
```
|
||||
2026-11-13T04:17:22Z
|
||||
otopcua-cli ack -u opc.tcp://localhost:4844 -n "ns=2;s=h6gal/vd03/RuntimeAlert" -e 4966AAB18408A24F9BB3CEC025FCA77E -c "OtOpcUa #614 H6 gate" -U apikey -P <temp key>
|
||||
Acknowledge successful: ns=2;s=h6gal/vd03/RuntimeAlert
|
||||
```
|
||||
|
||||
## Results per layer
|
||||
|
||||
### 1. OPC UA (PASS)
|
||||
|
||||
The live subscriber saw the condition go `Active, Acknowledged` at 04:17:22.546Z (EventId
|
||||
`E260F1DB097FC543A90AE478FF5E928C`). It saw no later revert to Unacknowledged in the 90 s window.
|
||||
|
||||
Driver log (account name redacted):
|
||||
|
||||
```
|
||||
[04:17:22 INF] Galaxy driver h6gal: acknowledging Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert as Galaxy user NAM\<ack account> (Fallback) for OPC UA principal h6gate (key a33c3b973768fbf5)
|
||||
[04:17:25 INF] DriverInstance h6gal: acknowledged native condition h6gal/vd03/RuntimeAlert by h6gate
|
||||
```
|
||||
|
||||
There was no gateway diagnostic, no `Unconfirmed` and no `BadTimeout` revert. The ~3 s between the two lines
|
||||
is the gateway waiting on MXAccess `OnWriteComplete`. On 11-11 the same pair of lines was logged within the
|
||||
same second.
|
||||
|
||||
### 2. Gateway alarm feed (PASS)
|
||||
|
||||
`QueryActiveAlarms` at 04:19:22Z:
|
||||
|
||||
```
|
||||
"currentState": "ALARM_CONDITION_STATE_ACTIVE_ACKED", "severity": 400,
|
||||
"lastTransitionTimestamp": "2026-11-13T04:17:25.173Z",
|
||||
"operatorUser": "<ack account>",
|
||||
"operatorComment": "OtOpcUa #614 H6 gate [OPC UA: h6gate, key a33c3b973768fbf5]"
|
||||
```
|
||||
|
||||
### 3. Galaxy (PASS)
|
||||
|
||||
Read-only gateway session (`OpenSession` → `Register` → `ReadBulk` → `CloseSession`) at 04:19:20Z:
|
||||
|
||||
| Attribute | Value | Quality |
|
||||
|---|---|---|
|
||||
| `TableAlarms_008.RuntimeAlert.Acked` | **True** | 192 |
|
||||
| `TableAlarms_008.RuntimeAlert.AckMsg` | **`OtOpcUa #614 H6 gate [OPC UA: h6gate, key a33c3b973768fbf5]`** | 192 |
|
||||
| `TableAlarms_008.RuntimeAlert.InAlarm` | True (since 2026-07-28T20:02:46.830Z) | 192 |
|
||||
|
||||
### 4. AVEVA event history (PASS)
|
||||
|
||||
`Runtime.dbo.v_AlarmEventHistory2` on wonder-sql-vd03, `EventStamp` 2026-11-12 22:00 to 2026-11-13 02:00 local
|
||||
(read-only). `Operator` is the ack account (redacted here), and `UserFullName` (redacted) is that account's
|
||||
directory name:
|
||||
|
||||
| EventStamp (local) | EventStampUTC | AlarmState | TagName | Operator | DomainName | OperatorNode | Description / Value |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 2026-11-12 23:17:25.173 | 2026-11-13 04:17:25.173 | `ACK_ALM` | `TableAlarms_008.RuntimeAlert` | `<ack account>` | `nam` | `MxGateway` | Description = `OtOpcUa #614 H6 gate [OPC UA: h6gate, key a33c3b973768fbf5]`, Priority 400 |
|
||||
| 2026-11-12 23:17:25.173 | 2026-11-13 04:17:25.173 | NULL (operator write event, Type `OPR`) | `TableAlarms_008.RuntimeAlert.AckMsg` | `<ack account>` | `nam` | `MxGateway` | `Write success -`, Value = the comment |
|
||||
|
||||
So the Historian holds the real operator (not a service identity) and the comment. The OPC UA principal and
|
||||
key id are carried inside the comment.
|
||||
|
||||
## Observations
|
||||
|
||||
- **OPC UA shows Acknowledged before Galaxy confirms.** The subscriber saw `Active, Acknowledged` at
|
||||
04:17:22.5Z, about 2.6 s before the gateway reply and Galaxy's `04:17:25.173Z` transition. That is the #869
|
||||
design: the condition goes Acknowledged optimistically and reverts on a failure or confirm timeout. No
|
||||
revert happened here.
|
||||
- **Severity is still 500 in OPC UA** against Galaxy priority 400 and authored severity 400, as on 11-11. Not
|
||||
investigated further (not part of H6).
|
||||
- The gateway and Historian timestamps for the ack agree to the millisecond (`04:17:25.173Z`). After #226
|
||||
the raise time agrees too.
|
||||
|
||||
## Cleanup
|
||||
|
||||
- Deleted from the ConfigDb: `Tag h6gal-rtalert`, `Device h6gal-vd03`, `DriverInstance h6gal`,
|
||||
`NodeAcl h6gate-acl` and `ApiKey h6-gate` (`a33c3b973768fbf5`).
|
||||
- Cleanup deployed (`h6-gate-cleanup`, `840ff8db`). site-b-1 logged `GalaxyDriver h6gal shutting down` and
|
||||
`disposed`.
|
||||
- Deleted the gate `Deployment` row `b95c3609` and its `NodeDeploymentState` rows, so the account name is not
|
||||
left in an artifact blob. No remaining deployment artifact mentions `h6gal`.
|
||||
- site-b-1 was recreated from the plain compose file with the same image: the ack password is absent and the
|
||||
gateway key is back to the committed default. site-b-2 was restarted.
|
||||
- The scratch key file and override were removed, and the shell env was unset.
|
||||
- The site-b-1 LocalDb cache may still hold a gate-era artifact until its next replacement.
|
||||
Reference in New Issue
Block a user