Files

T

Joseph Doherty 06c56d7719 fix(cluster,acl): OTOPCUA_ROLES fallback + one string-identity rule (#532 , #533 )

Two independent defects surfaced in review of #520 stage 3.

#532 — Cluster:Roles had no OTOPCUA_ROLES fallback, despite two doc sites
promising one. A node configured only through the env var — which drives ALL of
Program.cs's conditional wiring, so it reads as the authoritative knob — bound an
empty Roles array and joined the cluster carrying no Akka roles at all. It ran,
served, and logged nothing wrong, while central reported "no driver-role nodes
are Up" and every deployment to it timed out. Not hypothetical: the field fix on
wonder-app-vd03 was to hand-add Cluster__Roles__0/1 to a Windows service's
registry environment. Anyone reading the code to diagnose that box would have
concluded from the XML doc that the fallback was already applied.

AddOtOpcUaCluster now post-configures AkkaClusterOptions through the shared
RoleParser.ResolveEffectiveRoles, so BuildClusterOptions (Akka's own member
roles), ClusterRoleInfo, the ControlPlane singleton scope and
ConfigDbApiKeyStore's ClusterId all see one answer. Two sibling validators
(ConfigSource, Telemetry) read Cluster:Roles directly and were exempting the
exact node they exist for — an env-only driver-only node has no ConfigDb and
could sit on ConfigSource:Mode=Direct unchallenged. They resolve the same way now.

Also collapsed the four open-coded `Roles.FirstOrDefault(IsClusterRole)` +
ClusterIdFromRole copies into RoleParser.ClusterIdFromRoles. They had drifted:
three classified the cluster role ordinally, so a verbatim-bound `Cluster-SITE-A`
yielded a null ClusterId — and a null ClusterId applies NO cluster filter, which
made #528's cross-cluster over-grant guard fail OPEN. IsClusterRole is now
case-insensitive; the ClusterId suffix keeps its authored spelling, because
normalizing it would silently rename the Akka role of a running deployment.

#533 — three string-identity rules in one authorization path. AclScopeRef used
default (ordinal) equality while the trie builder filtered ClusterId and hashed
principals OrdinalIgnoreCase; the deploy gate's cross-cluster check was ordinal
too, which was the worse half — it could reject a whole deploy over a ClusterId
case difference the runtime would have accepted. It only worked because resolver
and builder were handed the same NodeAcl instances.

The rule is now stated once, on AclScopeRef, and applied throughout:
identifiers (ScopeId, ClusterId, Principal) compare OrdinalIgnoreCase; path
segments compare Ordinal. AclScopePathResolver re-keys its own maps rather than
documenting a comparer contract for callers, so the two independent producers —
EF at the gate, artifact JSON on the node — cannot diverge from the outside.
PermissionTrieNode.Children stays Ordinal: those are browse names, and Speed /
speed are two nodes that can coexist under one device.

Split PermissionTrieBuildDiagnosticReason.RetiredScopeKind out of
UnresolvedScope. A legacy FolderSegment row and a typo'd id had the same message
and different remedies.

Falsified: five mutations, each reddening only its own guard — remove the
PostConfigure (3 tests), revert IsClusterRole to ordinal (3), revert AclScopeRef
equality (1), revert the resolver's map comparer (6), revert the gate's
comparisons (2), remove the FolderSegment short-circuit (1).

Test isolation: OTOPCUA_ROLES is process-global and xUnit runs classes in
parallel, so the three validator suites that now resolve effective roles share a
non-parallel collection. Without it, cases passing no roles would intermittently
observe another class's env var.

Full solution build clean; full suite green apart from two pre-existing failures
confirmed identical on 3a6b2a1d (PrimaryGateFailoverTests, and
DriverProbeRegistrationTests expecting 10 probe types where 12 now register)
plus the AbLegacy integration suite, which needs a docker fixture.

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo

2026-08-01 01:51:22 -04:00

7.1 KiB

Raw Blame History

Deployment record — wonder-app-vd03 · Makino Pro 5 (Z-34184) FOCAS

Date configured: 2026-06-25 Host: wonder-app-vd03.zmr.zimmer.com (OtOpcUaHost Windows service, single fused admin+driver node) Status: Configured + deployed + served. Live values blocked by the FOCAS PDU-v3 driver gap — see docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md.

Equipment

Field	Value
Machine	Makino Pro 5 (FANUC 30i-B control)
IP / FOCAS endpoint	`10.201.31.5:8193` (FOCAS Ethernet, TCP) — reachable from the host (and from VPN clients)
ZTag	`Z-34184`
Manufacturer / Model	`Makino` / `Pro 5`
CNC series (driver)	`Thirty_i` (30i)

OtOpcUa host endpoints

AdminUI: http://wonder-app-vd03.zmr.zimmer.com:9000 (login disabled — Security__Auth__DisableLogin=true)
OPC UA: opc.tcp://wonder-app-vd03.zmr.zimmer.com:4840/OtOpcUa
Akka cluster: :4053 · ConfigDb: SQL OtOpcUaConfig on :1433

Deployed configuration (cluster `DEV`, Enterprise `zb` / Site `wonder-app-vd03`)

First config ever deployed on this node. All authored via the AdminUI.

Object	Value
Namespace	`dev-equipment` (Equipment), URI `urn:zb:wonder-app-vd03:equipment`
UNS path	`zb / wonder-app-vd03 / machining / makino`
FOCAS driver instance	`focas-z-34184` ("Makino Pro 5 Z-34184 (FOCAS)"), backend `wire`, FixedTree enabled
Device	`10.201.31.5:8193`, series `Thirty_i`
Equipment	`z-34184` → `EquipmentId = EQ-3686c0272279`, MachineCode `Z-34184`, ZTag `Z-34184`
Tags	`parts-count` → `MACRO:3901` (Float64/Double, Read); `parts-required` → `MACRO:3902` (Float64/Double, Read)
Deployment	`0c2db588` (rev `924b59097eba…`) — Sealed / "In sync"

OPC UA node IDs (verified served via the OtOpcUa CLI client):

Equipment: ns=2;s=EQ-3686c0272279
ns=2;s=EQ-3686c0272279/parts-count
ns=2;s=EQ-3686c0272279/parts-required

Host change required to deploy at all (Akka cluster roles)

This node was joining the Akka cluster role-less, so no deployment could ever complete ("a task was canceled"; Fleet status: "no driver-role nodes are Up"). Root cause: OTOPCUA_ROLES (set to admin,driver) drives Program.cs actor wiring but does not populate the Akka member roles — those come from config Cluster:Roles, which was unset (no Cluster__Roles env var; no Cluster section in any appsettings). No code bridges OTOPCUA_ROLES → AkkaClusterOptions.Roles.

Fix applied to the host service (HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaHost → Environment): added Cluster__Roles__0=admin and Cluster__Roles__1=driver, then restarted the service. The member now joins UP with roles ADMIN+DRIVER and deployments seal. Prior env backed up on the host at E:\ApiInstall\OtOpcUa\_envbak-20260625T145303.txt.

Product follow-up: either wire OTOPCUA_ROLES → AkkaClusterOptions.Roles, or bake Cluster:Roles into the deploy template, so a node redeploy doesn't regress to role-less.

RESOLVED 2026-08-01 (Gitea #532). The first option shipped: AddOtOpcUaCluster post-configures AkkaClusterOptions, so an empty Cluster:Roles now falls back to OTOPCUA_ROLES through RoleParser.ResolveEffectiveRoles. A node in this box's original configuration would join UP with roles ADMIN+DRIVER without the registry edit above, so a redeploy that drops the hand-added Cluster__Roles__0/1 no longer regresses to role-less. The Cluster__Roles entries are still worth keeping — the two sources normalize case differently (see docs/Configuration.md → Cluster:Roles), and an explicit list is the one that survives someone changing the env var for a different reason.

Worth recording for the next incident of this shape: this was documented behaviour that did not exist. AkkaClusterOptions.Roles' XML doc and SplitTopologyTransportValidator both described the fallback, so anyone reading the code to diagnose this box would have concluded the roles were already being applied and looked elsewhere.

Live verification (OtOpcUa CLI client → server)

connect + browse confirm the equipment + both tags are served at the node IDs above.

Driver fix shipped (2026-06-25): the wire client now accepts PDU v3 (see the linked plan). Validated directly against 10.201.31.5 — MACRO:3901/3902 read Good from the live 30i-B. However, the OtOpcUaHost on this box is still running the pre-v3 driver binary, so the live OPC UA tags will keep returning Bad_WaitingForInitialData until the rebuilt ZB.MOM.WW.OtOpcUa.Driver.FOCAS.dll (or a fresh self-contained host publish) is deployed to E:\ApiInstall\OtOpcUa\ and OtOpcUaHost is restarted. Once redeployed, parts-count/parts-required should go Good (FixedTree + PMC/Parameter still pending the follow-on v3 command work).

FixedTree under the Equipment node (feature built 2026-06-26)

The FOCAS FixedTree (Identity / Axes / Spindle / Program / Timers) now surfaces under the equipment as read-only value nodes, via a generic post-connect ITagDiscovery injection feature (branch feat/focas-fixedtree-equipment-injection; design + plan at docs/plans/2026-06-26-otopcua-fixedtree-equipment-injection-design.md and …-injection.md). After the driver connects and its FixedTreeCache populates (~0–2 s), nodes are grafted at e.g. ns=2;s=EQ-3686c0272279/FOCAS/Identity/SeriesNumber and …/FOCAS/Axes/X/AbsolutePosition, carrying live values through the same path as the authored parts-count/parts-required tags, and survive redeploys.

✅ LIVE-VALIDATED 2026-06-26. A full self-contained Host overlay built from feat/focas-fixedtree-equipment-injection @ 37cac5de was deployed to E:\ApiInstall\OtOpcUa (stop → full backup E:\ApiInstall\OtOpcUa_bak-20260626111416 → robocopy overlay preserving appsettings*.json + pki\ → restart; /healthz 200 Healthy, member Up ADMIN+DRIVER, OPC :4840 listening). After the FOCAS driver reconnected, the host log recorded DriverHost …: injected 57 discovered node(s) for driver focas-z-34184 under EQ-3686c0272279 and AddressSpaceApplier: discovered nodes materialised under EQ-3686c0272279 (folders=14, vars=57) — no Exception/NotSupportedException. Verified via the OtOpcUa CLI from a remote client:

browse --recursive ns=2;s=EQ-3686c0272279 → new FOCAS subtree (Identity, Axes X/Y/Z/B/C/AA
- Actual, Spindle/S1, Program, OperationMode, Timers), stable + idempotent across repeat browses (no churn); the device-host folder (10.201.31.5:8193) was collapsed as designed.
Sample reads, all Good (Status 0x0) with fresh/advancing source times: FOCAS/Identity/SeriesNumber=G431, CncType=31, AxisCount=7, FOCAS/Axes/X/AbsolutePosition=2801574 (live), FOCAS/OperationMode/ModeText=TJOG.
Authored parts-count/parts-required still Good — no regression.

Rollback point retained at E:\ApiInstall\OtOpcUa_bak-20260626111416.

7.1 KiB Raw Blame History Unescape Escape