Files
lmxopcua/docs/drivers/evidence/galaxy-g6-e2e-2026-11-12.md
T
Joseph Doherty 09b373e4db
v2-ci / unit-tests (push) Has been cancelled
fix(address-space): no UNS value Variable for a reference to an alarm-bearing tag (#896)
A UnsTagReference to a raw tag that carries an "alarm" object produced a UNS
value Variable that read BadWaitingForInitialData forever (G6 live gate,
TestAlarm002Condition on testmachine-015). BuildUnsReferenceVariables now skips
those references; the equipment gets the alarm only as the raw condition wired
as an event notifier of its folder, which is unchanged (the reference still
feeds RawTagPlan.ReferencingEquipmentPaths).

Why condition-only and not "feed the mirror":
- The raw side has no value node for such a tag: AddressSpaceApplier
  materialises an alarm tag as a condition OR a value variable, never both.
  There is no raw value for the UNS mirror to mirror.
- DriverHostActor skips alarm tags when it builds the value fan-out and write
  routing maps (`continue` before the UNS registration), and the driver leaves
  them out of its value subscription ("subscribed to 5 refs" with 6 tags in
  G6). Feeding the mirror would mean subscribing a value the design says the
  tag does not have.
- Docs already state condition-only: ScriptedAlarms.md ("instead of a value
  variable"), AlarmTracking.md ("only the condition, with no sibling value
  variable"), the v3 design (native alarm = one condition at the raw tag,
  surfaced at equipment via notifiers).
- The composer tests that asserted the mirror were incidental: f4f3e17e
  (v3 B4-WP1) put the alarm on the fixture's Speed tag to test raw alarm
  attachment, and later tests reused Speed for writable / historian / array
  checks of the mirror. No commit or doc chose a value mirror for alarm tags.
- ScadaBridge does not read it: its OtOpcUa alarm path is event subscriptions
  on the equipment/raw folders (v3 alarm B/C live gate), and a node that never
  had a value cannot be a working binding. G6 itself authored the alarm's
  Boolean as a separate value tag (TestAlarm002), which is the supported way to
  read it.

Tests: the composer fixture moves the alarm onto its own HighTemp tag
(Speed is now a plain historized value tag), with a new test asserting no UNS
Variable for the alarm reference, with positive controls (the raw tag still
carries the equipment path; the value references still emit). New real-SDK
test Alarm_bearing_reference_surfaces_as_the_condition_only_against_real_SDK:
no Variable at the UNS path, the Speed mirror present, one condition, the
equipment folder wired as its notifier. Both new tests fail with the skip
removed. Docs: Uns.md, ScriptedAlarms.md, G6 evidence note.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:55:49 -05:00

255 lines
17 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Galaxy G6 (#615): end to end against a live Galaxy, 2026-11-12
Gitea #615 gate G6 (`docs/plans/v2-ga-lab-gates-plan.md` §G6): "at least one end-to-end
integration run against live Galaxy succeeds".
**Result: PASS.** Every G6 pass criterion was met, plus write and native-alarm checks that the plan
does not require:
| G6 criterion (plan §G6) | Result |
|---|---|
| Browse returns a non-empty tree that mirrors the Galaxy hierarchy | PASS. Both subtrees: raw `galaxy-windev/dev/TestArea/TestMachine_015/…`, `…/TestArea3/TestMachine_036/…`; UNS `testarea/line-1/testmachine-015/…`, `testarea3/line-1/testmachine-036/…` |
| Read returns `Good` with a non-null value | PASS. Every authored value tag, through both NodeIds |
| Subscribe receives at least one data change within 5 s | PASS. First notification inside the first second, then one per second (1 s publishing) through both NodeIds |
| No `BadNoCommunication` / `BadTimeout` in the server log | PASS, with one exception. The only `BadNoCommunication` line is the planned driver respawn during a redeploy (`driver g6gal disconnected — 10 value node(s) marked BadNoCommunication (value retained)`), and the driver reconnected 10 s later. There are no `BadTimeout` lines, and no `ERR`/`FTL` lines in either run |
The extra checks:
| Extra step | Result |
|---|---|
| Write a TestMachine attribute, then read it back | PASS through raw and UNS, cross-checked through an independent gateway session. The anonymous negative control was refused `BadUserAccessDenied` |
| Native alarm becomes a Part 9 condition, then Ack, then Clear | PASS, with the ack confirmed in Galaxy (`.Acked=True`, `.AckMsg` = the comment) |
| Native alarm authored the way the AdminUI picker authors it | **FAIL on master `00a5be26`** (the condition never went active). Fixed in `e113faf5` (#895) and **PASS live** after the fix |
| DeployWatcher on a Galaxy redeploy | **NOT RUN.** See below |
Two defects were filed: **#895** (fixed in this branch) and **#896** (open).
## Target and versions
| | |
|---|---|
| Galaxy | windev `DESKTOP-6JL3KKO` test Galaxy `DEV` (GR DB `ZB`). AVEVA System Platform 2020 / Application Server 2020 (20.0.000). **129 deployed instances**, including all 40 `$TestMachine` objects (`TestMachine_001..040`) under areas `TestArea`, `TestArea2` and `TestArea3`. Last Galaxy deploy 2026-10-19T15:58:30Z (`GetLastDeployTime`) |
| Gateway | mxaccessgw main `447070a5`, NSSM `MxAccessGw`, gRPC h2c `http://10.100.0.48:5120`. Not restarted or reconfigured. `http://10.100.0.48:5130/health/ready` returned 200 before and after the run |
| Gateway key | temporary `g6-otopcua-e2e` (`session:open,session:close,invoke:read,invoke:write,events:read,metadata:read`, 1-day expiry), minted with the gateway admin CLI and revoked afterwards |
| OtOpcUa | `origin/master` `00a5be26` for the main run. The #895 re-check ran the same tree plus the fix (committed as `e113faf5`) |
| OPC UA client | `otopcua-cli` (`src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI`), security None |
**Clock note.** windev's clock is about **9 min 12 s behind** the Mac (04:07:57 against 04:17:09 UTC,
checked during the run). Galaxy `SourceTimestamp`s therefore read about 9 min behind `ServerTimestamp`
throughout. This is box skew and not a defect: the gap stayed constant while values changed every second.
## Setup (local, temporary, removed afterwards)
The docker-dev rig was in use by other agents and its image commit could not be identified, so a single
OtOpcUa Host ran on the Mac from this worktree:
- Fused `admin,driver` single node, Akka self-seeded on `localhost:4063`.
- OPC UA on `opc.tcp://localhost:4850`, AdminUI and REST on `http://127.0.0.1:9310`.
- Its own ConfigDb `OtOpcUaG6` on the rig's SQL container (`localhost:14330`), migrated with `dotnet ef database update`.
- Cluster `G6`: `NodeCount=1`, `RedundancyMode=None`, one `ClusterNode localhost:4063`.
- Committed docker-dev dev secrets (KEK, pepper, JWT, LDAP); `DisableLogin=true`; `Security:Acl:Enabled=false`.
Authored config, seeded into the ConfigDb and deployed with `POST /api/deployments` (`X-Api-Key`):
| Row | Value |
|---|---|
| DriverInstance | `g6gal` "galaxy-windev", `GalaxyMxGateway`, `Gateway.Endpoint=http://10.100.0.48:5120`, `UseTls=false`, `ApiKeySecretRef=env:OTOPCUA_GALAXY_APIKEY`, `MxAccess.ClientName=OtOpcUa-G6-gate`, `Repository.WatchDeployEvents=true`, `AlarmAck.FallbackUser=Administrator` with `FallbackPasswordSecretRef=dev:` (empty password, the windev rig's documented identity, mxaccessgw `docs/AlarmProbeFindings.md`) |
| Device / groups | device `dev`; TagGroups `TestArea/TestMachine_015` and `TestArea3/TestMachine_036` (mirroring Galaxy area → object) |
| Tags | `TestMachine_015`: `TestChangingInt` (Int32 R), `TestDouble` (Double RW), `TestAlarm002` (Boolean RW, value), `TestAlarm002Condition` (`attributeRef "Galaxy!TestArea.TestMachine_015.TestAlarm002"` + `alarm` AlarmCondition 500); `TestMachine_036`: `TestChangingInt`, `TestHistoryValue` (Int32 R). Later added: `TestAlarm003` (`attributeRef "TestMachine_015.TestAlarm003"` + `alarm` OffNormalAlarm 700, the picker's form) |
| UNS | areas `testarea` / `testarea3` → `line-1` → equipment `testmachine-015` (`EQ-a6f600000000`) / `testmachine-036` (`EQ-b6f600000000`), with all six tags referenced |
| Data-plane key | `g6-gate` (principal `g6gate`, 8 h), `NodeAcl` Cluster grant 927 (Browse/Read/Subscribe/HistoryRead/WriteOperate/AlarmRead/AlarmAck/AlarmConfirm) |
Deploys `8ee64aa9…` (driver), `0646dee1…` (key and ACL), `20affbc8…` (TestAlarm003). Each one was
`Sealed`, with node `localhost:4063` `Applied`. A first key deploy was correctly refused
`422 NodeAclScopeIdUnexpected`: a Cluster grant must carry no ScopeId.
Driver start:
```
DriverHost localhost:4063: spawned GalaxyMxGateway driver g6gal (stub=False)
GalaxyMxSession connected — clientName=OtOpcUa-G6-gate serverHandle=1
GalaxyDriver g6gal initialized — endpoint=http://10.100.0.48:5120 clientName=OtOpcUa-G6-gate
DriverInstance g6gal: subscribed to 5 refs (galaxy-sub-1)
DriverInstance g6gal: native-alarm subscription established for 1 alarm ref(s)
```
## Step 1: browse both subtrees (PASS)
`browse -r -d 8` (excerpt; condition children elided):
```
[Object] OtOpcUa (NodeId: ns=2;s=OtOpcUa)
[Object] testarea (NodeId: ns=3;s=g6-area-test)
[Object] line-1 (NodeId: ns=3;s=g6-line-a)
[Object] testmachine-015 (NodeId: ns=3;s=EQ-a6f600000000)
[Variable] TestChangingInt (NodeId: ns=3;s=testarea/line-1/testmachine-015/TestChangingInt)
[Variable] TestDouble (NodeId: ns=3;s=testarea/line-1/testmachine-015/TestDouble)
[Variable] TestAlarm002 (NodeId: ns=3;s=testarea/line-1/testmachine-015/TestAlarm002)
[Variable] TestAlarm002Condition (NodeId: ns=3;s=testarea/line-1/testmachine-015/TestAlarm002Condition) <- #896
[Object] TestAlarm002Condition (NodeId: ns=2;s=galaxy-windev/dev/TestArea/TestMachine_015/TestAlarm002Condition)
[Object] testarea3 (NodeId: ns=3;s=g6-area-test3)
[Object] line-1 (NodeId: ns=3;s=g6-line-b)
[Object] testmachine-036 (NodeId: ns=3;s=EQ-b6f600000000)
[Variable] TestChangingInt (NodeId: ns=3;s=testarea3/line-1/testmachine-036/TestChangingInt)
[Variable] TestHistoryValue (NodeId: ns=3;s=testarea3/line-1/testmachine-036/TestHistoryValue)
[Object] galaxy-windev (NodeId: ns=2;s=galaxy-windev)
[Object] dev (NodeId: ns=2;s=galaxy-windev/dev)
[Object] TestArea (NodeId: ns=2;s=galaxy-windev/dev/TestArea)
[Object] TestMachine_015 (NodeId: ns=2;s=galaxy-windev/dev/TestArea/TestMachine_015)
[Variable] TestAlarm002 (…/TestMachine_015/TestAlarm002)
[Object] TestAlarm002Condition (…/TestMachine_015/TestAlarm002Condition) (+ the full Part 9 child set: ActiveState, AckedState, Acknowledge, Confirm, ShelvingState, …)
[Variable] TestChangingInt (…/TestMachine_015/TestChangingInt)
[Variable] TestDouble (…/TestMachine_015/TestDouble)
[Object] TestArea3 (NodeId: ns=2;s=galaxy-windev/dev/TestArea3)
[Object] TestMachine_036 (NodeId: ns=2;s=galaxy-windev/dev/TestArea3/TestMachine_036)
[Variable] TestChangingInt (…/TestMachine_036/TestChangingInt)
[Variable] TestHistoryValue (…/TestMachine_036/TestHistoryValue)
```
The raw subtree uses `ns=2;s=<RawPath>` and the UNS subtree uses `ns=3;s=<Area>/<Line>/<Equipment>/<Signal>`.
The Part 9 condition appears under both the raw TagGroup and the referencing equipment folder (notifier fan-out).
## Step 2: read through both NodeIds (PASS)
| NodeId | Value | Status |
|---|---|---|
| raw `…/TestMachine_015/TestChangingInt` | 841 | Good |
| raw `…/TestMachine_015/TestDouble` | 0 | Good |
| raw `…/TestMachine_015/TestAlarm002` | False | Good |
| raw `…/TestMachine_036/TestChangingInt` | 479 | Good |
| raw `…/TestMachine_036/TestHistoryValue` | 0 | Good |
| uns `testarea/line-1/testmachine-015/TestChangingInt` | 904 | Good |
| uns `testarea/line-1/testmachine-015/TestDouble` | 0 | Good |
| uns `testarea3/line-1/testmachine-036/TestChangingInt` | 330 | Good |
| `Server.ServiceLevel` (i=2267) | 250 | Good |
| uns `…/testmachine-015/TestAlarm002Condition` | (none) | **BadWaitingForInitialData**, permanently. #896 |
## Step 3: subscribe (PASS)
Raw `…/TestMachine_015/TestChangingInt`, 1000 ms, 10 s: 9 Good notifications, the first inside the first
second (`49, 474, 899, 325, 750, 176, 601, 27, 452`). UNS equipment `EQ-b6f600000000` recursive: 2 variables,
8 notifications in 8 s (`TestChangingInt` every second; `TestHistoryValue` initial value only, since the
attribute is static). Summaries: `Ever went BAD during window: 0`, `No update received at all: 0`.
## Step 4: write and read back (PASS)
```
anonymous write raw TestDouble = 42.125 -> Write failed: BadUserAccessDenied (negative control; anonymous has no WriteOperate)
apikey g6gate write raw TestDouble = 42.125 -> Write successful
read raw …/TestMachine_015/TestDouble -> 42.125 Good
read uns testarea/line-1/testmachine-015/TestDouble -> 42.125 Good
apikey g6gate write uns TestDouble = 17.5 -> Write successful
read raw …/TestMachine_015/TestDouble -> 17.5 Good
```
Independent readback through a separate gateway session (mxgw-dotnet `smoke --item TestMachine_015.TestDouble`):
`OnDataChange doubleValue: 17.5, quality 192`. TestDouble was then written back to its original 0.
Before the key deploy, a write with the new key was refused (`AclGate: … key … is no longer enabled or has
expired in generation 1`). That is expected: data-plane key usability follows the deployed generation, so a
key inserted after a deploy needs a redeploy.
## Step 5: native alarm to Part 9, ack and clear (PASS)
The alarm was raised through OPC UA: `write raw …/TestMachine_015/TestAlarm002 = True` (key `g6gate`). The live
`alarms --refresh` subscriber (key `g6gate`) saw:
```
[2026-11-13T04:11:05.2390000Z] ALARM galaxy-windev/dev/TestArea/TestMachine_015/TestAlarm002Condition
State: Active, Unacknowledged
Severity: 750
Retain: True
EventId: 9503F17B7A8F82498FBCEDD868C4AD98
```
```
otopcua-cli ack -n "ns=2;s=…/TestAlarm002Condition" -e 9503F17B… -c "OtOpcUa #615 G6 gate" -U apikey -P <key>
Acknowledge successful
[2026-11-13T04:20:34.4620500Z] … State: Active, Acknowledged Retain: True
```
Driver log:
```
Galaxy driver g6gal: acknowledging Galaxy!TestArea.TestMachine_015.TestAlarm002 as Galaxy user Administrator (Fallback) for OPC UA principal g6gate (key 8e142bf2a2b70df8)
DriverInstance g6gal: acknowledged native condition galaxy-windev/dev/TestArea/TestMachine_015/TestAlarm002Condition by g6gate
```
Galaxy side, read through a separate gateway session after the ack:
| Attribute | Value |
|---|---|
| `TestMachine_015.TestAlarm002.Acked` | **True** (q 192) |
| `TestMachine_015.TestAlarm002.AckMsg` | **`OtOpcUa #615 G6 gate [OPC UA: g6gate, key 8e142bf2a2b70df8]`** |
| `TestMachine_015.TestAlarm002.InAlarm` | True |
| gateway `QueryActiveAlarms` | `ALARM_CONDITION_STATE_ACTIVE_ACKED`, `operatorUser: Administrator`, the same comment |
This is the leg H6 could not get on vd03 (`galaxy-h6-ack-2026-11-11.md`): the ack reached Galaxy.
The alarm was cleared through the UNS NodeId: `write uns testarea/line-1/testmachine-015/TestAlarm002 = False`.
Galaxy `TestAlarm002`/`.InAlarm` went False at once. The gateway feed's return-to-normal arrived between 40 s
and 100 s later; the gateway `ReconcileIntervalSeconds` is 30. The condition then went
`Inactive, Acknowledged`, `Retain: False`. The clear event's `Time` carried the earlier transition timestamp,
not the return-to-normal time. That latency and timestamp are gateway-side behaviour, so they are recorded
here but not filed against OtOpcUa.
**Severity (#894).** The gateway reported `severity 500` (AVEVA priority 500, High band 251–500), and the
condition read **750**. The authored `alarm.severity` was 500 for TestAlarm002 and 700 for TestAlarm003; the
live feed value overrode both. This is the #887 band mapping working live (750, as #894 expects), on another
alarm and another gateway than #894's vd03 target.
## Step 6: picker-form native alarm (FAIL on master, PASS with #895)
`TestAlarm003` was authored as the AdminUI Galaxy picker would author it: `attributeRef "TestMachine_015.TestAlarm003"`
plus the seeded `{"alarmType":"OffNormalAlarm","severity":700}`. It was raised by a supervisory write through a
separate gateway session. A plain `Write` is refused `SecurityError 1008`, because `TestAlarm003` is `MxSecurityOperate`.
- **Master `00a5be26`:** the gateway feed reported `Galaxy!TestArea.TestMachine_015.TestAlarm003` as
`ALARM_CONDITION_STATE_ACTIVE`. The OPC UA condition stayed `Inactive, Acknowledged`. Cause:
`OnAlarmFeedTransition` mapped only the feed's full reference to the authored attributeRef. The
`sourceObjectReference` the feed also sends (`TestMachine_015.TestAlarm003`) was ignored, so the transition got
a ConditionId matching no condition. → **#895**.
- **With the fix (`e113faf5`):** after a host restart on the same deployment, which the host restored from
LocalDb on bootstrap, ConditionRefresh returned
`TestAlarm003 State: Active, Unacknowledged Severity: 750 Retain: True`. The OPC UA `Acknowledge` dialled
`Galaxy!TestArea.TestMachine_015.TestAlarm003` (log line as above), and the condition went `Active, Acknowledged`.
Galaxy then read `.Acked = True` and `.AckMsg = "OtOpcUa #615 G6 gate picker-form (#895) [OPC UA: g6gate, key 8e142bf2a2b70df8]"`.
After the supervisory clear the condition went `Inactive, Acknowledged`, `Retain: False`, and the gateway
showed no active alarms.
## Not run: DeployWatcher on a Galaxy redeploy
This step was not run for two reasons:
- **No way to trigger a redeploy.** A Galaxy redeploy needs the ArchestrA IDE or a GRAccess script on windev.
No GRAccess assembly was found in the ArchestrA Framework bin directory.
- **Risk to another agent.** The test Galaxy was serving another agent's MXAccess captures, and redeploying
its objects would have disturbed them.
Expected behaviour if it is run later: `docs/drivers/Galaxy.md` says the `IRediscoverable` / `DeployWatcher`
path raises a "re-browse" prompt on `/hosts` and does **not** rebuild the address space (#518). The live check
would be a `/hosts` chip, not a tree rebuild. `Repository.WatchDeployEvents=true` was set and the driver started
cleanly with it.
## Defects
- **#895 (fixed here, `e113faf5`).** A native alarm authored by the AdminUI picker (`attributeRef tag.Attr`)
never went active, and its ack would dial a reference the gateway refuses. Three unit tests were added to
`GalaxyDriverRawPathKeyingTests`. The main one fails with the fix reverted, which was checked by reverting
the fix and re-running it.
- **#896 (fixed after this run).** The UNS mirror of an alarm-bearing raw tag gets a value Variable that stays
`BadWaitingForInitialData` forever, against the documented "condition instead of value variable" contract.
It was not fixed here because the composer test fixtures assert that mirror.
Fixed afterwards: the composer no longer emits a UNS Variable for a reference to an alarm-bearing tag, so the
equipment shows only the condition (the `<- #896` Variable in the Step 1 browse is gone).
## Cleanup
- **windev:** gateway key `g6-otopcua-e2e` revoked. No other key was touched. `MxAccessGw` was never stopped,
restarted or reconfigured, and `/health/ready` returned 200 throughout. Temporary `.ps1` files in
`C:\Users\Public` were deleted.
- **Galaxy:** TestDouble restored to 0. TestAlarm002 and TestAlarm003 are both back to False, and the gateway
shows no active alarms. Their raise/ack/clear rows stay in the Galaxy's own alarm history.
- **Mac:** local Host stopped. Database `OtOpcUaG6` dropped from the rig SQL container. The docker-dev rig
containers and their `OtOpcUa` DB were untouched. Scratch key files deleted.