7a27ef1f67
v2-ci / unit-tests (push) Waiting to run
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
171 lines
10 KiB
Markdown
171 lines
10 KiB
Markdown
# Galaxy H6 (#614): OPC UA ack of a real Galaxy native alarm to AVEVA history, 2026-11-11
|
|
|
|
Gitea #614 item H6: "an OPC UA Acknowledge of a REAL Galaxy native alarm reaches AVEVA event history".
|
|
|
|
**Result: FAIL — the item cannot be ticked.** The OPC UA path worked end to end inside OtOpcUa: the
|
|
Part 9 `Acknowledge` returned Good, the client saw the condition go `Active, Acknowledged`, and the driver
|
|
sent the ack to mxaccessgw as the owner-provided Galaxy account with no error. **The acknowledgement
|
|
never reached Galaxy.** After the ack, Galaxy's own `.Acked` was still `False`, `.AckMsg` was still
|
|
empty, the gateway feed still showed the alarm as unacknowledged, and AVEVA event history has no ACK
|
|
row. So OtOpcUa now shows the condition acknowledged while Galaxy and the Historian do not (a
|
|
silent-divergence defect, below).
|
|
|
|
Exactly **one** acknowledge was sent (approved write, 2026-11-11T17:37:55Z). It was not retried. No
|
|
remote command ran on any wonder box. Galaxy access was through the gateway's gRPC API and read-only
|
|
SQL only.
|
|
|
|
## Target
|
|
|
|
| | |
|
|
|---|---|
|
|
| Alarm | `Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert` (object `TableAlarms_008`, contained as `Right_004.TableAlarms`), priority 400, "Run Time Exceeded" |
|
|
| Gateway | `wonder-app-vd03.zmr.zimmer.com:5120` (h2c) |
|
|
| Galaxy security | OS/domain accounts (`ZB.dbo.user_profile`). Every 2026 ack in the Historian before this gate is a `NAM\…` operator from an InTouch node; none came through the gateway. |
|
|
| Historian | `wonder-sql-vd03` → `Runtime.dbo.v_AlarmEventHistory2`. Queries need an `EventStamp` range. There is no `A2ALMDB` or `WWALMDB`. |
|
|
|
|
## Setup (docker-dev, temporary, all removed afterwards)
|
|
|
|
| Item | Value |
|
|
|---|---|
|
|
| Node | SITE-B `site-b-1` (OPC UA `opc.tcp://localhost:4844`). Its partner `site-b-2` was stopped for the run so site-b-1 was the sole (Primary) node. |
|
|
| Driver | `h6gal`, DriverType `GalaxyMxGateway`, seeded straight into the ConfigDb. `Gateway.Endpoint` = the VD03 gateway, `UseTls=false`, `ApiKeySecretRef=env:GALAXY_MXGW_API_KEY`, `MxAccess.ClientName=OtOpcUa-H6-gate`, `Repository.WatchDeployEvents=false`. `AlarmAck.FallbackUser`/`FallbackDomain` = the owner-provided account (domain `NAM`); `FallbackPasswordSecretRef=env:OTOPCUA_GALAXY_ACKPW` |
|
|
| Device / tag | device `vd03` (`{}`), tag `RuntimeAlert`, Boolean, Read, `TagConfig` = `{"attributeRef":"Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert","alarm":{"alarmType":"AlarmCondition","severity":400,"historizeToAveva":false}}`. `historizeToAveva:false` keeps OtOpcUa's own alarm historian from writing anything to AVEVA. |
|
|
| Secrets | the gateway key and the ack password were injected into the site-b-1 container as runtime env only, through a scratch compose override that held `${VAR}` references, never values. Nothing secret went into the repo, the compose files, the ConfigDb or this file. |
|
|
| OPC UA client | `otopcua-cli` (`src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI`), UserName `apikey` + a temporary data-plane key (`h6-gate`, principal `h6gate`, 4 h expiry) with a SITE-B `Cluster` grant: Browse, Read, Subscribe, HistoryRead, AlarmRead and AlarmAcknowledge (399). LDAP was not used. |
|
|
| Deploy | `POST /api/deployments`. The first attempt used DriverType `Galaxy`, which is wrong: the registered name is `GalaxyMxGateway`. It was corrected and deployed again. Deployments to SITE-B end `TimedOut` while `site-b-2` is stopped (the same as the Heidenhain gate); `site-b-1` applied each one. |
|
|
|
|
Driver start (site-b-1): `GalaxyMxSession connected — clientName=OtOpcUa-H6-gate`, `DriverInstance h6gal: connected`,
|
|
`native-alarm subscription established for 1 alarm ref(s)`.
|
|
|
|
## Step 1: OPC UA, condition and Acknowledge (PASS within OtOpcUa)
|
|
|
|
`alarms --refresh` before the ack:
|
|
|
|
```
|
|
[2026-07-29T01:02:46.8300000Z] ALARM h6gal/vd03/RuntimeAlert
|
|
Condition: RuntimeAlert
|
|
State: Active, Unacknowledged
|
|
Severity: 500
|
|
Retain: True
|
|
ConditionNodeId: ns=2;s=h6gal/vd03/RuntimeAlert
|
|
EventId: 8DB71E793FBD7040993404E30808FADC
|
|
```
|
|
|
|
The single ack, at 17:37:55Z:
|
|
|
|
```
|
|
otopcua-cli ack -u opc.tcp://localhost:4844 -n "ns=2;s=h6gal/vd03/RuntimeAlert" -e 8DB71E793FBD7040993404E30808FADC -c "OtOpcUa #614 H6 gate" -U apikey -P <temp key>
|
|
Acknowledge successful: ns=2;s=h6gal/vd03/RuntimeAlert
|
|
```
|
|
|
|
The live subscriber running during the ack saw:
|
|
|
|
```
|
|
[2026-11-11T17:37:55.8648690Z] ALARM h6gal/vd03/RuntimeAlert
|
|
State: Active, Acknowledged
|
|
Severity: 500
|
|
Retain: True
|
|
EventId: 0F15D34C9D0B6D43B7A4B08F3A72D047
|
|
```
|
|
|
|
Driver log (account name redacted):
|
|
|
|
```
|
|
[17:37:55 INF] Galaxy driver h6gal: acknowledging Galaxy!CVDAisle_1.TableAlarms_008.RuntimeAlert as Galaxy user NAM\<ack account> (Fallback) for OPC UA principal h6gate (key fa1d95d325ab9ab5)
|
|
[17:37:55 INF] DriverInstance h6gal: acknowledged native condition h6gal/vd03/RuntimeAlert by h6gate
|
|
```
|
|
|
|
No warning, refusal or revert followed. So the gateway answered `PROTOCOL_STATUS_CODE_OK` with
|
|
`hresult` 0 (or absent): a non-OK status or a non-zero hresult throws and is logged. The comment the
|
|
driver sent was `OtOpcUa #614 H6 gate [OPC UA: h6gate, key fa1d95d325ab9ab5]`.
|
|
|
|
## Step 2: Galaxy and the gateway alarm feed (FAIL)
|
|
|
|
`QueryActiveAlarms` (prefix = the alarm reference) about 30 s after the ack, unchanged from before:
|
|
|
|
```
|
|
"currentState": "ALARM_CONDITION_STATE_ACTIVE", "severity": 400,
|
|
"lastTransitionTimestamp": "2026-07-29T01:02:46.830Z", "operatorComment": "Run Time Exceeded"
|
|
```
|
|
|
|
Read-only gateway session (`OpenSession` → `Register` → `ReadBulk` → `CloseSession`), at 17:39:24Z and again at 17:40:18Z:
|
|
|
|
| Attribute | Value | Quality |
|
|
|---|---|---|
|
|
| `TableAlarms_008.RuntimeAlert.Acked` | **False** | 192 |
|
|
| `TableAlarms_008.RuntimeAlert.AckMsg` | **""** (empty) | 192 |
|
|
| `TableAlarms_008.RuntimeAlert.InAlarm` | True (since 2026-07-28T20:02:46.830Z) | 192 |
|
|
|
|
On the windev rig, mxaccessgw's gateway-level verification (`AlarmProbeFindings.md`) showed a
|
|
successful ack turning `.Acked` True and leaving the comment in `.AckMsg`. Neither happened here.
|
|
|
|
## Step 3: AVEVA event history (FAIL)
|
|
|
|
`v_AlarmEventHistory2`, `EventStamp` 2026-11-11 12:00 to 23:59 local, queried at 17:38, 17:39 and 17:46 UTC.
|
|
The Historian is live: it stored other events at 17:00 UTC. Nothing for this alarm:
|
|
|
|
```
|
|
2026-11-11 12:00:30.922 | 17:00:30.922 | UNACK_ALM | DevWinPlatform.CPULoadHiAlarm | NULL | NULL
|
|
2026-11-11 12:00:50.918 | 17:00:50.918 | UNACK_RTN | DevWinPlatform.CPULoadHiAlarm | NULL | NULL
|
|
```
|
|
|
|
There is no `ACK_ALM` or `ACK_RTN` row for `TableAlarms_008.RuntimeAlert` and no row with `OperatorNode=MxGateway`.
|
|
|
|
## Defect: OtOpcUa reports Acknowledged on an ack Galaxy never applied (reported, not fixed)
|
|
|
|
mxaccessgw returns `OK`/`hresult 0` both for "write applied" and for "write sent, no `OnWriteComplete`
|
|
within the wait (unconfirmed, with a diagnostic)" (mxaccessgw `docs/Grpc.md`, `AcknowledgeAlarm`
|
|
outcome table). Its own docs say that "a client that must be certain should watch for the resulting
|
|
transition on `StreamAlarms`".
|
|
|
|
`GatewayGalaxyAlarmAcknowledger.EvaluateReply`
|
|
(`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Runtime/GatewayGalaxyAlarmAcknowledger.cs`) treats OK +
|
|
hresult 0 as success. It never reads or logs `AcknowledgeAlarmReply.diagnostic_message`, and nothing
|
|
in the driver or host waits for the upstream Acknowledge transition. So the #760 revert never fires,
|
|
and the OPC UA condition stays Acknowledged while Galaxy holds `.Acked=False`. A ScadaBridge or
|
|
operator client would believe the alarm was handled.
|
|
|
|
Why the gateway-side write did not take effect cannot be determined from here: the gateway's logs are
|
|
on wonder-app-vd03, and remote commands are barred. Candidates:
|
|
|
|
- an unconfirmed `AckMsg` write that was later dropped or rejected;
|
|
- the account lacking acknowledge permission on the object's security group;
|
|
- a VD03 gateway build that predates the `AckMsg` path.
|
|
|
|
Not attempted again (one-ack rule). Suggested follow-ups (not filed, no Gitea per brief):
|
|
|
|
- **OtOpcUa:** log `diagnostic_message` on every ack reply. Treat an unconfirmed ack as pending until
|
|
the feed shows the Acknowledge transition, and revert on timeout.
|
|
- **mxaccessgw:** expose applied vs unconfirmed as a field rather than only in a diagnostic.
|
|
- **Owner:** check the VD03 gateway log around 2026-11-11T17:37:55Z for the ack's diagnostic.
|
|
|
|
## Other observations (not filed)
|
|
|
|
- **The gateway snapshot timestamp is 5 h off.** It reports `lastTransitionTimestamp
|
|
2026-07-29T01:02:46.830Z`. MXAccess's own `InAlarm` timestamp and the Historian's `EventStampUTC`
|
|
both say 2026-07-28T20:02:46.830Z. Cross-repo, mxaccessgw.
|
|
- **The gateway snapshot and the Historian disagree on state.** The gateway says ACTIVE since July,
|
|
and `.InAlarm` agrees. The Historian logged five later "set off scan" `UNACK_RTN` rows (latest
|
|
2026-10-22) with no new `UNACK_ALM`.
|
|
- **Severity differs between the three views.** The OPC UA condition shows Severity 500. The authored
|
|
`alarm.severity` was 400, and so is the Galaxy priority.
|
|
- **site-b-2 never acknowledged the cleanup deploys.** After cleanup, the standby `site-b-2` (restarted
|
|
and restored from its LocalDb at the same revision `e221b179…`) did not ack the re-dispatched
|
|
deployment, so the reseal deploy also ended `TimedOut` (site-b-1 Applied).
|
|
- **The rig needed a nudge after the OrbStack restart.** `otopcua-dev-sql-1` and
|
|
`otopcua-dev-traefik-1` stayed Exited. I started them and restarted central-1 and central-2 (#868
|
|
ordering) before seeding.
|
|
|
|
## Cleanup
|
|
|
|
- Deleted from the ConfigDb:
|
|
- `Tag h6gal-rtalert`, `Device h6gal-vd03` and `DriverInstance h6gal`;
|
|
- `NodeAcl h6gate-acl` and `ApiKey h6-gate` (`fa1d95d325ab9ab5`);
|
|
- the two gate `Deployment` rows that carried the driver config, and their `NodeDeploymentState` rows,
|
|
so the account name is not left in an artifact blob.
|
|
- Cleanup deployed (`h6-gate-cleanup`, `h6-gate-cleanup-reseal`). site-b-1 logged `GalaxyDriver h6gal shutting down … disposed`.
|
|
- site-b-1 recreated from the plain compose file: the ack password is absent and the gateway key is back
|
|
to the committed default. site-b-2 restarted.
|
|
- The scratch key file was deleted. The shell env was unset.
|
|
- The site-b-1 LocalDb cache may still hold a gate-era artifact until its next replacement.
|