Files
lmxopcua/docs/drivers/evidence/galaxy-g6-e2e-2026-11-12.md
T
Joseph Doherty 09b373e4db
v2-ci / unit-tests (push) Has been cancelled
fix(address-space): no UNS value Variable for a reference to an alarm-bearing tag (#896)
A UnsTagReference to a raw tag that carries an "alarm" object produced a UNS
value Variable that read BadWaitingForInitialData forever (G6 live gate,
TestAlarm002Condition on testmachine-015). BuildUnsReferenceVariables now skips
those references; the equipment gets the alarm only as the raw condition wired
as an event notifier of its folder, which is unchanged (the reference still
feeds RawTagPlan.ReferencingEquipmentPaths).

Why condition-only and not "feed the mirror":
- The raw side has no value node for such a tag: AddressSpaceApplier
  materialises an alarm tag as a condition OR a value variable, never both.
  There is no raw value for the UNS mirror to mirror.
- DriverHostActor skips alarm tags when it builds the value fan-out and write
  routing maps (`continue` before the UNS registration), and the driver leaves
  them out of its value subscription ("subscribed to 5 refs" with 6 tags in
  G6). Feeding the mirror would mean subscribing a value the design says the
  tag does not have.
- Docs already state condition-only: ScriptedAlarms.md ("instead of a value
  variable"), AlarmTracking.md ("only the condition, with no sibling value
  variable"), the v3 design (native alarm = one condition at the raw tag,
  surfaced at equipment via notifiers).
- The composer tests that asserted the mirror were incidental: f4f3e17e
  (v3 B4-WP1) put the alarm on the fixture's Speed tag to test raw alarm
  attachment, and later tests reused Speed for writable / historian / array
  checks of the mirror. No commit or doc chose a value mirror for alarm tags.
- ScadaBridge does not read it: its OtOpcUa alarm path is event subscriptions
  on the equipment/raw folders (v3 alarm B/C live gate), and a node that never
  had a value cannot be a working binding. G6 itself authored the alarm's
  Boolean as a separate value tag (TestAlarm002), which is the supported way to
  read it.

Tests: the composer fixture moves the alarm onto its own HighTemp tag
(Speed is now a plain historized value tag), with a new test asserting no UNS
Variable for the alarm reference, with positive controls (the raw tag still
carries the equipment path; the value references still emit). New real-SDK
test Alarm_bearing_reference_surfaces_as_the_condition_only_against_real_SDK:
no Variable at the UNS path, the Speed mirror present, one condition, the
equipment folder wired as its notifier. Both new tests fail with the skip
removed. Docs: Uns.md, ScriptedAlarms.md, G6 evidence note.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 23:55:49 -05:00

17 KiB
Raw Blame History

Galaxy G6 (#615): end to end against a live Galaxy, 2026-11-12

Gitea #615 gate G6 (docs/plans/v2-ga-lab-gates-plan.md §G6): "at least one end-to-end integration run against live Galaxy succeeds".

Result: PASS. Every G6 pass criterion was met, plus write and native-alarm checks that the plan does not require:

G6 criterion (plan §G6) Result
Browse returns a non-empty tree that mirrors the Galaxy hierarchy PASS. Both subtrees: raw galaxy-windev/dev/TestArea/TestMachine_015/…, …/TestArea3/TestMachine_036/…; UNS testarea/line-1/testmachine-015/…, testarea3/line-1/testmachine-036/…
Read returns Good with a non-null value PASS. Every authored value tag, through both NodeIds
Subscribe receives at least one data change within 5 s PASS. First notification inside the first second, then one per second (1 s publishing) through both NodeIds
No BadNoCommunication / BadTimeout in the server log PASS, with one exception. The only BadNoCommunication line is the planned driver respawn during a redeploy (driver g6gal disconnected — 10 value node(s) marked BadNoCommunication (value retained)), and the driver reconnected 10 s later. There are no BadTimeout lines, and no ERR/FTL lines in either run

The extra checks:

Extra step Result
Write a TestMachine attribute, then read it back PASS through raw and UNS, cross-checked through an independent gateway session. The anonymous negative control was refused BadUserAccessDenied
Native alarm becomes a Part 9 condition, then Ack, then Clear PASS, with the ack confirmed in Galaxy (.Acked=True, .AckMsg = the comment)
Native alarm authored the way the AdminUI picker authors it FAIL on master 00a5be26 (the condition never went active). Fixed in e113faf5 (#895) and PASS live after the fix
DeployWatcher on a Galaxy redeploy NOT RUN. See below

Two defects were filed: #895 (fixed in this branch) and #896 (open).

Target and versions

Galaxy windev DESKTOP-6JL3KKO test Galaxy DEV (GR DB ZB). AVEVA System Platform 2020 / Application Server 2020 (20.0.000). 129 deployed instances, including all 40 $TestMachine objects (TestMachine_001..040) under areas TestArea, TestArea2 and TestArea3. Last Galaxy deploy 2026-10-19T15:58:30Z (GetLastDeployTime)
Gateway mxaccessgw main 447070a5, NSSM MxAccessGw, gRPC h2c http://10.100.0.48:5120. Not restarted or reconfigured. http://10.100.0.48:5130/health/ready returned 200 before and after the run
Gateway key temporary g6-otopcua-e2e (session:open,session:close,invoke:read,invoke:write,events:read,metadata:read, 1-day expiry), minted with the gateway admin CLI and revoked afterwards
OtOpcUa origin/master 00a5be26 for the main run. The #895 re-check ran the same tree plus the fix (committed as e113faf5)
OPC UA client otopcua-cli (src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI), security None

Clock note. windev's clock is about 9 min 12 s behind the Mac (04:07:57 against 04:17:09 UTC, checked during the run). Galaxy SourceTimestamps therefore read about 9 min behind ServerTimestamp throughout. This is box skew and not a defect: the gap stayed constant while values changed every second.

Setup (local, temporary, removed afterwards)

The docker-dev rig was in use by other agents and its image commit could not be identified, so a single OtOpcUa Host ran on the Mac from this worktree:

  • Fused admin,driver single node, Akka self-seeded on localhost:4063.
  • OPC UA on opc.tcp://localhost:4850, AdminUI and REST on http://127.0.0.1:9310.
  • Its own ConfigDb OtOpcUaG6 on the rig's SQL container (localhost:14330), migrated with dotnet ef database update.
  • Cluster G6: NodeCount=1, RedundancyMode=None, one ClusterNode localhost:4063.
  • Committed docker-dev dev secrets (KEK, pepper, JWT, LDAP); DisableLogin=true; Security:Acl:Enabled=false.

Authored config, seeded into the ConfigDb and deployed with POST /api/deployments (X-Api-Key):

Row Value
DriverInstance g6gal "galaxy-windev", GalaxyMxGateway, Gateway.Endpoint=http://10.100.0.48:5120, UseTls=false, ApiKeySecretRef=env:OTOPCUA_GALAXY_APIKEY, MxAccess.ClientName=OtOpcUa-G6-gate, Repository.WatchDeployEvents=true, AlarmAck.FallbackUser=Administrator with FallbackPasswordSecretRef=dev: (empty password, the windev rig's documented identity, mxaccessgw docs/AlarmProbeFindings.md)
Device / groups device dev; TagGroups TestArea/TestMachine_015 and TestArea3/TestMachine_036 (mirroring Galaxy area → object)
Tags TestMachine_015: TestChangingInt (Int32 R), TestDouble (Double RW), TestAlarm002 (Boolean RW, value), TestAlarm002Condition (attributeRef "Galaxy!TestArea.TestMachine_015.TestAlarm002" + alarm AlarmCondition 500); TestMachine_036: TestChangingInt, TestHistoryValue (Int32 R). Later added: TestAlarm003 (attributeRef "TestMachine_015.TestAlarm003" + alarm OffNormalAlarm 700, the picker's form)
UNS areas testarea / testarea3 → line-1 → equipment testmachine-015 (EQ-a6f600000000) / testmachine-036 (EQ-b6f600000000), with all six tags referenced
Data-plane key g6-gate (principal g6gate, 8 h), NodeAcl Cluster grant 927 (Browse/Read/Subscribe/HistoryRead/WriteOperate/AlarmRead/AlarmAck/AlarmConfirm)

Deploys 8ee64aa9… (driver), 0646dee1… (key and ACL), 20affbc8… (TestAlarm003). Each one was Sealed, with node localhost:4063 Applied. A first key deploy was correctly refused 422 NodeAclScopeIdUnexpected: a Cluster grant must carry no ScopeId.

Driver start:

DriverHost localhost:4063: spawned GalaxyMxGateway driver g6gal (stub=False)
GalaxyMxSession connected — clientName=OtOpcUa-G6-gate serverHandle=1
GalaxyDriver g6gal initialized — endpoint=http://10.100.0.48:5120 clientName=OtOpcUa-G6-gate
DriverInstance g6gal: subscribed to 5 refs (galaxy-sub-1)
DriverInstance g6gal: native-alarm subscription established for 1 alarm ref(s)

Step 1: browse both subtrees (PASS)

browse -r -d 8 (excerpt; condition children elided):

[Object] OtOpcUa (NodeId: ns=2;s=OtOpcUa)
  [Object] testarea (NodeId: ns=3;s=g6-area-test)
    [Object] line-1 (NodeId: ns=3;s=g6-line-a)
      [Object] testmachine-015 (NodeId: ns=3;s=EQ-a6f600000000)
        [Variable] TestChangingInt (NodeId: ns=3;s=testarea/line-1/testmachine-015/TestChangingInt)
        [Variable] TestDouble (NodeId: ns=3;s=testarea/line-1/testmachine-015/TestDouble)
        [Variable] TestAlarm002 (NodeId: ns=3;s=testarea/line-1/testmachine-015/TestAlarm002)
        [Variable] TestAlarm002Condition (NodeId: ns=3;s=testarea/line-1/testmachine-015/TestAlarm002Condition)   <- #896
        [Object] TestAlarm002Condition (NodeId: ns=2;s=galaxy-windev/dev/TestArea/TestMachine_015/TestAlarm002Condition)
  [Object] testarea3 (NodeId: ns=3;s=g6-area-test3)
    [Object] line-1 (NodeId: ns=3;s=g6-line-b)
      [Object] testmachine-036 (NodeId: ns=3;s=EQ-b6f600000000)
        [Variable] TestChangingInt (NodeId: ns=3;s=testarea3/line-1/testmachine-036/TestChangingInt)
        [Variable] TestHistoryValue (NodeId: ns=3;s=testarea3/line-1/testmachine-036/TestHistoryValue)
  [Object] galaxy-windev (NodeId: ns=2;s=galaxy-windev)
    [Object] dev (NodeId: ns=2;s=galaxy-windev/dev)
      [Object] TestArea (NodeId: ns=2;s=galaxy-windev/dev/TestArea)
        [Object] TestMachine_015 (NodeId: ns=2;s=galaxy-windev/dev/TestArea/TestMachine_015)
          [Variable] TestAlarm002 (…/TestMachine_015/TestAlarm002)
          [Object] TestAlarm002Condition (…/TestMachine_015/TestAlarm002Condition)   (+ the full Part 9 child set: ActiveState, AckedState, Acknowledge, Confirm, ShelvingState, …)
          [Variable] TestChangingInt (…/TestMachine_015/TestChangingInt)
          [Variable] TestDouble (…/TestMachine_015/TestDouble)
      [Object] TestArea3 (NodeId: ns=2;s=galaxy-windev/dev/TestArea3)
        [Object] TestMachine_036 (NodeId: ns=2;s=galaxy-windev/dev/TestArea3/TestMachine_036)
          [Variable] TestChangingInt (…/TestMachine_036/TestChangingInt)
          [Variable] TestHistoryValue (…/TestMachine_036/TestHistoryValue)

The raw subtree uses ns=2;s=<RawPath> and the UNS subtree uses ns=3;s=<Area>/<Line>/<Equipment>/<Signal>. The Part 9 condition appears under both the raw TagGroup and the referencing equipment folder (notifier fan-out).

Step 2: read through both NodeIds (PASS)

NodeId Value Status
raw …/TestMachine_015/TestChangingInt 841 Good
raw …/TestMachine_015/TestDouble 0 Good
raw …/TestMachine_015/TestAlarm002 False Good
raw …/TestMachine_036/TestChangingInt 479 Good
raw …/TestMachine_036/TestHistoryValue 0 Good
uns testarea/line-1/testmachine-015/TestChangingInt 904 Good
uns testarea/line-1/testmachine-015/TestDouble 0 Good
uns testarea3/line-1/testmachine-036/TestChangingInt 330 Good
Server.ServiceLevel (i=2267) 250 Good
uns …/testmachine-015/TestAlarm002Condition (none) BadWaitingForInitialData, permanently. #896

Step 3: subscribe (PASS)

Raw …/TestMachine_015/TestChangingInt, 1000 ms, 10 s: 9 Good notifications, the first inside the first second (49, 474, 899, 325, 750, 176, 601, 27, 452). UNS equipment EQ-b6f600000000 recursive: 2 variables, 8 notifications in 8 s (TestChangingInt every second; TestHistoryValue initial value only, since the attribute is static). Summaries: Ever went BAD during window: 0, No update received at all: 0.

Step 4: write and read back (PASS)

anonymous write raw TestDouble = 42.125        -> Write failed: BadUserAccessDenied   (negative control; anonymous has no WriteOperate)
apikey g6gate write raw TestDouble = 42.125    -> Write successful
  read raw  …/TestMachine_015/TestDouble       -> 42.125 Good
  read uns  testarea/line-1/testmachine-015/TestDouble -> 42.125 Good
apikey g6gate write uns TestDouble = 17.5      -> Write successful
  read raw  …/TestMachine_015/TestDouble       -> 17.5 Good

Independent readback through a separate gateway session (mxgw-dotnet smoke --item TestMachine_015.TestDouble): OnDataChange doubleValue: 17.5, quality 192. TestDouble was then written back to its original 0.

Before the key deploy, a write with the new key was refused (AclGate: … key … is no longer enabled or has expired in generation 1). That is expected: data-plane key usability follows the deployed generation, so a key inserted after a deploy needs a redeploy.

Step 5: native alarm to Part 9, ack and clear (PASS)

The alarm was raised through OPC UA: write raw …/TestMachine_015/TestAlarm002 = True (key g6gate). The live alarms --refresh subscriber (key g6gate) saw:

[2026-11-13T04:11:05.2390000Z] ALARM  galaxy-windev/dev/TestArea/TestMachine_015/TestAlarm002Condition
  State:     Active, Unacknowledged
  Severity:  750
  Retain:    True
  EventId:   9503F17B7A8F82498FBCEDD868C4AD98
otopcua-cli ack -n "ns=2;s=…/TestAlarm002Condition" -e 9503F17B… -c "OtOpcUa #615 G6 gate" -U apikey -P <key>
Acknowledge successful
[2026-11-13T04:20:34.4620500Z] … State: Active, Acknowledged   Retain: True

Driver log:

Galaxy driver g6gal: acknowledging Galaxy!TestArea.TestMachine_015.TestAlarm002 as Galaxy user Administrator (Fallback) for OPC UA principal g6gate (key 8e142bf2a2b70df8)
DriverInstance g6gal: acknowledged native condition galaxy-windev/dev/TestArea/TestMachine_015/TestAlarm002Condition by g6gate

Galaxy side, read through a separate gateway session after the ack:

Attribute Value
TestMachine_015.TestAlarm002.Acked True (q 192)
TestMachine_015.TestAlarm002.AckMsg OtOpcUa #615 G6 gate [OPC UA: g6gate, key 8e142bf2a2b70df8]
TestMachine_015.TestAlarm002.InAlarm True
gateway QueryActiveAlarms ALARM_CONDITION_STATE_ACTIVE_ACKED, operatorUser: Administrator, the same comment

This is the leg H6 could not get on vd03 (galaxy-h6-ack-2026-11-11.md): the ack reached Galaxy.

The alarm was cleared through the UNS NodeId: write uns testarea/line-1/testmachine-015/TestAlarm002 = False. Galaxy TestAlarm002/.InAlarm went False at once. The gateway feed's return-to-normal arrived between 40 s and 100 s later; the gateway ReconcileIntervalSeconds is 30. The condition then went Inactive, Acknowledged, Retain: False. The clear event's Time carried the earlier transition timestamp, not the return-to-normal time. That latency and timestamp are gateway-side behaviour, so they are recorded here but not filed against OtOpcUa.

Severity (#894). The gateway reported severity 500 (AVEVA priority 500, High band 251–500), and the condition read 750. The authored alarm.severity was 500 for TestAlarm002 and 700 for TestAlarm003; the live feed value overrode both. This is the #887 band mapping working live (750, as #894 expects), on another alarm and another gateway than #894's vd03 target.

Step 6: picker-form native alarm (FAIL on master, PASS with #895)

TestAlarm003 was authored as the AdminUI Galaxy picker would author it: attributeRef "TestMachine_015.TestAlarm003" plus the seeded {"alarmType":"OffNormalAlarm","severity":700}. It was raised by a supervisory write through a separate gateway session. A plain Write is refused SecurityError 1008, because TestAlarm003 is MxSecurityOperate.

  • Master 00a5be26: the gateway feed reported Galaxy!TestArea.TestMachine_015.TestAlarm003 as ALARM_CONDITION_STATE_ACTIVE. The OPC UA condition stayed Inactive, Acknowledged. Cause: OnAlarmFeedTransition mapped only the feed's full reference to the authored attributeRef. The sourceObjectReference the feed also sends (TestMachine_015.TestAlarm003) was ignored, so the transition got a ConditionId matching no condition. → #895.
  • With the fix (e113faf5): after a host restart on the same deployment, which the host restored from LocalDb on bootstrap, ConditionRefresh returned TestAlarm003 State: Active, Unacknowledged Severity: 750 Retain: True. The OPC UA Acknowledge dialled Galaxy!TestArea.TestMachine_015.TestAlarm003 (log line as above), and the condition went Active, Acknowledged. Galaxy then read .Acked = True and .AckMsg = "OtOpcUa #615 G6 gate picker-form (#895) [OPC UA: g6gate, key 8e142bf2a2b70df8]". After the supervisory clear the condition went Inactive, Acknowledged, Retain: False, and the gateway showed no active alarms.

Not run: DeployWatcher on a Galaxy redeploy

This step was not run for two reasons:

  • No way to trigger a redeploy. A Galaxy redeploy needs the ArchestrA IDE or a GRAccess script on windev. No GRAccess assembly was found in the ArchestrA Framework bin directory.
  • Risk to another agent. The test Galaxy was serving another agent's MXAccess captures, and redeploying its objects would have disturbed them.

Expected behaviour if it is run later: docs/drivers/Galaxy.md says the IRediscoverable / DeployWatcher path raises a "re-browse" prompt on /hosts and does not rebuild the address space (#518). The live check would be a /hosts chip, not a tree rebuild. Repository.WatchDeployEvents=true was set and the driver started cleanly with it.

Defects

  • #895 (fixed here, e113faf5). A native alarm authored by the AdminUI picker (attributeRef tag.Attr) never went active, and its ack would dial a reference the gateway refuses. Three unit tests were added to GalaxyDriverRawPathKeyingTests. The main one fails with the fix reverted, which was checked by reverting the fix and re-running it.
  • #896 (fixed after this run). The UNS mirror of an alarm-bearing raw tag gets a value Variable that stays BadWaitingForInitialData forever, against the documented "condition instead of value variable" contract. It was not fixed here because the composer test fixtures assert that mirror. Fixed afterwards: the composer no longer emits a UNS Variable for a reference to an alarm-bearing tag, so the equipment shows only the condition (the <- #896 Variable in the Step 1 browse is gone).

Cleanup

  • windev: gateway key g6-otopcua-e2e revoked. No other key was touched. MxAccessGw was never stopped, restarted or reconfigured, and /health/ready returned 200 throughout. Temporary .ps1 files in C:\Users\Public were deleted.
  • Galaxy: TestDouble restored to 0. TestAlarm002 and TestAlarm003 are both back to False, and the gateway shows no active alarms. Their raise/ack/clear rows stay in the Galaxy's own alarm history.
  • Mac: local Host stopped. Database OtOpcUaG6 dropped from the rig SQL container. The docker-dev rig containers and their OtOpcUa DB were untouched. Scratch key files deleted.