- deferment.md: node ACLs no longer authored-but-inert (#520 shipped through stage 7, opt-in Security:Acl:Enabled); failure-visibility trio mostly closed (01/S-1 + 03/S4 fixed, 06/S-1 partial); bulkhead + WriteIdempotentAttribute closed by R2-02; Sql Postgres+Oracle dialects shipped (d54f3c83); akka-hosting partials verified superseded - archreview/00-OVERALL.md: staleness banner on the carried-open block (04/C-1→R2-05, 03/S2+S3→R2-08, 03/P1→R2-07, driver fleet→R2-09/R2-11/#457, hygiene→R2-12, trio per above; what genuinely remains from theme 4) - archreview/plans/STATUS.md: carved out the true exceptions to "every Round-2 live gate GREEN" (R2-03 T11/T12 deferred-live, R2-10 T10 breaker-OPEN never forced); #459 closed by mesh Phase 7 drill D4 (c50ebcf7) - archreview/plans/INTEGRATION-SWEEP-STATUS.md: "follow-ups unstarted" → all 7 shipped as PRs #446–#452 - archreview/06-gateway-integrations.md: 06/S-1 STILL OPEN → PARTIAL (fail-closed advise + unconfirmed meter; residual cross-repo) - 2026-07-31-apikey-acl-enforcement-design.md: status → implemented through stage 7, #520 CLOSED 2026-08-01 - 2026-07-21-per-cluster-mesh-design.md: "Phases 1–7 not started" → PROGRAM COMPLETE 2026-07-24 (Phase 7c50ebcf7) - 2026-07-20-localdb-phase2-live-gate.md: merged as 06b5d81b; open design question shipped by mesh Phase 6 (checks 3/4 still never re-run on the three-mesh topology) - 2026-07-20-localdb-phase1-progress.md: Tasks 10–16 REMAIN → Phase 1 complete, live-gated 8/8, mergedc957db52- 2026-08-06-clusterclient-to-grpc-tracking.md: defect 4 OPEN → fixed byc0475f2d+e54883a8, merged @b360239e- 2026-08-06-clusterclient-to-grpc.md.tasks.json: Task 15 pending → completed (Phase 4 gate PASS 8/8 @0b194003, recordd0ac6fc5) - 2026-07-22-mesh-phase2-live-gate.md: DriverStatusPanel orphan → wired into /hosts by #546 (9420f8ef) - 2026-07-21-issue-485-live-gate.md: pointer from the "NOT fixed here" finding to the #486 fix (DriverHostActor.cs:1696-1717 + FetchAndCache mirror) - 2026-08-05-sql-postgres-oracle.md: chrome/AdminUI gate not deferred — PASSED 2026-08-05, found + fixed the Test-Connect provider gate (4e461951) - v2-ga-lab-gates-plan.md / v2-release-readiness.md / live-hardware-validation-runbooks.md: FOCAS live-CNC smoke #54 RESOLVED on a real 31i-B 2026-06-25 (cnc_rdparam caveat preserved) - Mqtt-Test-Fixture.md: STATE/DCMD "not implemented" → shipped + live-gated (#5082a7d1167, #511857a10b4) - 2026-08-01-continuous-historization-live-gate.md: poison-tag FIFO pin "filed, not fixed" → #538 fixed same day (e7fbe977, gate 4bdce910; Capacity default now 500k) - 2026-05-26-akka-hosting-alignment-plan.md.tasks.json: F8/F9/F10/F13/F14 partial → completed, superseded by the v3 address-space rework (Phase7Composer/ScriptedAlarmActor deleted; evaluators/SDK sinks/security policies production-wired) Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
34 KiB
API-key identity + node ACL enforcement (Gitea #520)
Status: implemented through stage 7 — #520 CLOSED 2026-08-01. Stage merges: fbe7403c
(stages 1-2, 4: API-key identity + keys/ACLs via the artifact), 3a6b2a1d (stage 3: variable-depth
scope model + scopePaths producer), 1ccc7400/f692f6af (stage 5: enforcement on write, alarms,
HistoryRead), d4389f57/00aa4866 (stage 6: enforcement on read, browse, subscribe), 3796c0bf
(stage 7: ACL scope picker + authoring-time resolution). Enforcement is opt-in via
Security:Acl:Enabled (default false). (Was: "design agreed 2026-07-31, not yet implemented" —
corrected 2026-08-07, truth sweep.)
Supersedes: docs/v2/acl-design.md (LDAP-group-keyed model, never wired).
Why this exists
IPermissionEvaluator + TriePermissionEvaluator + PermissionTrieCache + PermissionTrieBuilder
have lived in src/Core/ZB.MOM.WW.OtOpcUa.Core/Authorization/ with 30 passing unit tests and zero
production call sites since v2. Operators can author NodeAcl rows at /clusters/{id}/acls;
ConfigComposer snapshots every row into every deployment artifact, so an ACL edit shifts the
artifact RevisionHash and ships fleet-wide — and DeploymentArtifact never deserializes them. The
bytes arrive and are dropped.
Actual enforcement today is two server-wide role checks: WriteOperate on Value writes (either
realm) and AlarmAck on the five Part 9 alarm methods. Read, Browse, HistoryRead and
CreateMonitoredItems have no authorization check at all, so any admitted session — including
Anonymous — reads the entire address space.
Re-verified on master 2026-07-31 (all six claims still true). Both AdminUI warning banners and the
corrected docs from 2026-07-27 are in place, so the trap is signposted but still a trap.
Decisions taken (owner, 2026-07-31)
| # | Decision | Consequence |
|---|---|---|
| D1 | The OPC UA data plane is machine-to-machine. Identity is an API key presented with a static username, not an LDAP user. | NodeAcl keys on a principal, not an LDAP group — blocker #2 dissolves rather than being solved. |
| D2 | Replace LDAP entirely on the OPC UA data plane. | LdapOpcUaUserAuthenticator and the OPC UA half of Security:Ldap are deleted. AdminUI login keeps its own LDAP — untouched. |
| D3 | Anonymous is a normal, grantable principal, gated by a new enable setting. | When enabled, Anonymous resolves to the reserved principal anonymous and can be granted read or write like any key. When disabled, anonymous session activation is rejected. |
D3 is the neat one: it removes Anonymous as a special case in the evaluator. There is no "anonymous bypass" branch to get wrong — an ungranted anonymous session is denied by the same fail-closed path as an ungranted key.
Assumption I am making explicit (not asked)
Enforcement ships behind Security:Acl:Enabled, default false. This is the dark-switch
discipline MeshTransport / ConfigSource / Telemetry all shipped under: both sides are wired in
both modes, the flip is a config change rather than a redeploy. Without it, the first deploy carrying
this code would silently start denying every client that has no key — including ScadaBridge's Data
Connection Layer, which consumes this address space from another repo. The authenticator swap (D2)
is not dark-switchable in the same way and is a hard cutover; that asymmetry is deliberate and
called out in the rollout stage.
Identity model
OPC UA UserName token Anonymous token
username = <static> (no credential)
password = <api key secret>
| |
v v
ApiKeyOpcUaUserAuthenticator Security:Acl:AllowAnonymous ?
| |
v v
principal set <------------- "anonymous"
|
v
PermissionTrie (per cluster, per artifact revision)
- Key format mirrors the family convention (
histgw_<id>_<secret>in HistorianGateway,apikey create-keyin mxaccessgw):otua_<keyId>_<secret>. ThekeyIdprefix makes lookup an index hit rather than a scan over every stored hash. - Storage is a hash, never the secret — HMAC-SHA256 under a server pepper, compared with
CryptographicOperations.FixedTimeEquals. The plaintext is shown once at creation in AdminUI and is unrecoverable afterwards. - A key maps to one or more named principals, and
NodeAclrows key on the principal name. The indirection is deliberate: many keys (one per consuming system, independently revocable) share one grant set, and rotating a key never touches an ACL row.
NodeAcl.LdapGroup → NodeAcl.Principal
A rename + migration, not a new column. The trie walker is unchanged — it already matches an opaque string; only what produces that string changes.
The four blockers, re-scoped
| # | #520's blocker | Resolution under this design |
|---|---|---|
| 1 | Raw-realm nodes have no representable NodeScope (NodeHierarchyKind has one member, Equipment) |
Add NodeHierarchyKind.Raw. ⚠️ This cell was wrong about the shape — it described the path as the five-tuple Folder/Driver/Device/TagGroup/Tag, but folders and tag groups both self-nest, so the real path is folder… / Driver / Device / group… / Tag with unbounded depth on two axes. The fix was therefore not an enum member but a variable-depth NodeScope + segment-loop walk. FolderSegment was not repurposed as the raw-folder level; it is retired, and RawFolder/RawDriver/RawDevice/RawTagGroup were appended instead. Correctly identified as the one genuine design item. |
| 2 | Session carries roles, not LDAP groups | Dissolved by D1 — the principal comes from the key, not from a directory. |
| 3 | PermissionTrieBuilder's scopePaths has no producer |
Done in stage 3 — AclScopePathResolver, built node-side from the artifact's raw + UNS topology (no DB read, so a driver-only node can build it). The coverage note was right in substance but imprecise: six tests did pass a scopePaths dictionary, but every one hand-wrote it in the test body with exactly the segments the walker expected, so nothing pinned a producer — and there was none to pin. That gap is closed by an artifact → parse → build → walk test. |
| 4 | No session lifecycle / AuthGenerationId |
Stamp the generation from the artifact RevisionHash at session activation; a new deploy invalidates cached decisions. UserAuthorizationState already has the freshness predicates. |
Plus, from the same audit: the node never parses ACLs out of the artifact, and OtOpcUaNodeManager
has no cluster identity (NodeScope.ClusterId is required and the evaluator fails closed on
mismatch).
Enforcement seams
| Operation | Seam today | Work |
|---|---|---|
| Write (Value) | EvaluateEquipmentWriteGate — pure, fail-closed, unit-testable |
Swap the role check for an ACL decision. Runs under the node-manager Lock, so the check must stay non-blocking — the evaluator is pure, which fits. |
| Alarm ack/confirm/shelve | two AlarmAck role gates |
Same swap. |
| HistoryRead | HistoryReadRawModified / Processed / AtTime / Events overrides already exist |
Add the check inside them. |
| Read / Browse / Subscribe | none | The real gap. Needs a Read override or per-variable OnSimpleReadValue, on the hot publish path — so the decision must be cached per (session, node), not evaluated per value. |
Smaller inconsistencies to settle en route
NodeAclScopeKind.FolderSegmentis offered in the authoring dropdown but the trie walker has no level to match it at — settled in stage 3, but not the way this line assumed. It is a retired v2 concept (it described theSystemPlatform/Galaxy hierarchy v3 deleted), so rather than being given a level it is now explicitly unresolvable, withRawFolderas its v3 replacement. It survives in the enum only because the artifact stores the ordinal, so removing it would renumber every later member in every deployed artifact. The dropdown labels it retired, and the deploy gate names it. ⚠️ The old claim that the walker could not match it was slightly overstated:Buildrouted it throughDescendlike any other kind, so given a path it would have matched and been reported under whatever level it happened to land on. It was unlabellable, not unplaceable.NodePermissions.AlarmReadhas noOpcUaOperationmapping — grantable but unreachable.Closed — stage 3 gave it a deploy-time existence check, and stage 7 an authoring-time one plus a picker that cannot produce a dangling id. There is still no FK:NodeAcl.ScopeIdhas no FK and no existence check, so scope ids dangle silently.ScopeIdnames a row in one of eight tables chosen byScopeKind, which is not a foreign key any schema can express.— stale; stage 1 added the reference. The line read as an open item long after it was closed.ZB.MOM.WW.OtOpcUa.OpcUaServer.csprojdoes not referenceCoreat allNodePermissions.HistoryUpdateis deliberately in no bundle until HistoryUpdate exists. Leave it.
Stages
Each stage is one commit, builds clean, and leaves the tree shippable.
-
✅ Foundation — principal model.
NodeAcl.LdapGroup→Principal(+ migration);ApiKeyentity- store + hashing;
OpcUaServer.csprojreferencesCore. No behaviour change.
- store + hashing;
-
✅ Identity cutover.
ApiKeyOpcUaUserAuthenticatorreplacesLdapOpcUaUserAuthenticatoron the OPC UA data plane; Anonymous resolves to theanonymousprincipal or is rejected per setting. AdminUI LDAP untouched. -
Raw scope. ✅ Done.
NodeHierarchyKind.Raw, a variable-depth scope model, thescopePathsproducer (AclScopePathResolver) fed from the artifact, a deploy-time gate for ACL scopes, and the first coverage of the production hierarchy path.It was substantially larger than this line originally implied, because the sketch had the shape of the raw tree wrong. The plan said the Raw path is the five-tuple
Folder/Driver/Device/TagGroup/Tag. It is not:RawFolder.ParentRawFolderIdandTagGroup.ParentTagGroupIdare both self-referencing, so folders and tag groups each nest without bound and a real RawPath isfolder… / Driver / Device / group… / Tag.NodeScopewas fixed-arity (five nullable level fields) andPermissionTrie.CollectMatcheswas five hardcoded steps, so the scope model could not address a real raw tree at all.NodeScopeis now{ClusterId, Kind, IReadOnlyList<string> Segments}and the walk is a segment loop. Good news found en route: the trie structure was already variable-depth (NodeAclPath.Segmentsis a list;PermissionTrieNode.Childrenis a recursive dictionary), so only the walker and the scope record had to change.Four decisions worth keeping:
- The namespace level became the realm. Level 1 of the walk gated on
NodeScope.NamespaceId, but v3 has noNamespaceentity — the snapshot was retired when the address space went dual-namespace, and the identifier appeared nowhere in production outsideNodeScope. So level 1 could only ever fail to match, which silently made every sub-cluster grant unreachable while cluster grants kept working. It is now the realm token (raw/uns), derived from the scope'sKindbyNodeAclRealmsrather than carried on it, so the walk and the path producer cannot disagree about its spelling.NodeAclScopeKind.Namespacebecomes meaningful again as "a whole realm". - Provenance comes from the granting row's
ScopeKind, recorded onTrieGrantat build time, not from the depth the grant sits at. Depth-derived provenance could not name a raw level at all (the depth of "Device" varies with folder nesting) and misreported any path whose length disagreed with its kind. - The trie comparer moved from
OrdinalIgnoreCasetoOrdinal. Raw sibling-name uniqueness is ordinal, soSpeedandspeedcan both exist under one device and a grant on either would have matched both — a silent widening. Case-sensitive is also the fail-closed direction. - The null-
scopePaths"deterministic test mode" is deleted. It planted unresolvable rows at a direct child of the trie root, a position the walker cannot reach — so the grant looked placed and could never match, and every test using that mode proved nothing about production placement. An unresolvable row is now dropped with a diagnostic, andscopePathsis required.
A scope resolves to a list of paths, not one: a raw tag projected into N equipments is reachable at N+1 NodeIds and a write through any of them lands on the same driver reference, so a Tag-scoped grant that covered only the raw node would be bypassable through a UNS one. The lookup is keyed on
(ScopeKind, ScopeId)rather thanScopeIdalone, becauseScopeIdhas no FK and no type discriminator — ids are prefixed only by convention, and an id collision across entity types would hand a grant the wrong node's paths.Also closed here:
NodeAclrows had no deploy-time validation of any kind and were not even reachable fromDraftValidator(DraftSnapshotcarried noNodeAclscollection). They are now gated for dangling ids, cross-cluster scopes, blank/contradictoryScopeId, the retiredFolderSegmentkind, and undefined ordinals. That gate substituted for the ACL scope picker until stage 7 shipped one; it remains the backstop for rows edited outside the AdminUI. - The namespace level became the realm. Level 1 of the walk gated on
-
Artifact + lifecycle. ✅ Done.
ConfigComposersnapshotsApiKeysalongside theNodeAclsit already carried;DeploymentArtifact.ParseDataPlaneAuthoritydecodes both node-side (grants scoped to the node's own cluster, keys not — a key is a fleet-wide credential and the ACL rows are what bind it);OpcUaPublishActorinstalls them onDataPlaneAuthorityStateon every apply;ArtifactApiKeyStoreserves them to session activation on a driver-only node, which until now denied every UserName token; andOtOpcUaNodeManagergains aDataPlaneAuthorityproperty, which is the cluster identity blocker #4 named.Two things went differently from the sketch, both deliberately:
- The generation is a counter, not the hash. "Stamp the generation from the
RevisionHash" reads as "derive a long from the hash bytes", and that is wrong:PermissionTrieCachetreats the highestGenerationIdas current (ClusterEntry.WithAdditional) andPrunekeeps the highest, while hash fragments are unordered — so roughly half of all deploys would install a lower generation and the cache would keep serving the previous deployment's ACLs, silently. The hash still drives the stamp (a new hash advances it, a re-applied one does not); it just does not become it. - Sessions are not stamped yet. Building a
UserAuthorizationStateper session belongs with the first seam that reads one, in stage 5. Constructing one now would be a per-session object nothing consults — the dormant-wiring failure mode this repo has been bitten by before.
Also settled here: the feed runs before
OpcUaPublishActor'splan.IsEmptyearly-return. An artifact that only revokes a key produces an empty address-space plan, so feeding after that return would leave the revoked key working until some unrelated structural change happened to redeploy. - The generation is a counter, not the hash. "Stamp the generation from the
-
Enforce write + alarms + HistoryRead. ✅ Done.
AclOptions(Security:Acl:Enabled, default false);DataPlaneAuthorizerprojects a NodeId + realm onto aNodeScopeand evaluates it through the existingTriePermissionEvaluator; the trie is built and installed insideDataPlaneAuthorityState.Apply; bothIApiKeyStoreimplementations widened to the full grant set.Five things went differently from this line, all deliberate:
- The role check is REPLACED, not supplemented. Running both in series would require a grant to
confer the server-wide role as well as pass the scope check — and only
Cluster-scoped grants confer a role — so every narrow grant would have stayed inert and the flag would have looked inert too. Narrow grants working is the point of the stage. Pinned by a test that writes successfully from a session carrying no roles at all; the deny-side test alone passes against a build where the ACL is never consulted, because the role gate denies too. UserAuthorizationStateis built per CALL, not per session. The per-session shape was LDAP-inherited: it exists so a long-lived session can re-resolve group membership that changed out of band. Nothing can change out of band under API keys, so the generation stamp is the whole invalidation signal and a per-call record picks it up for free — where a cached one would need explicit invalidation to do the same job.IsStale/NeedsRefreshtherefore cannot fire here and are documented as vestigial rather than left looking load-bearing. Stage 6 is where caching earns its keep, and it wants a per-(session, node)decision cache, not this type.- Revocation now reaches an OPEN session, which the sketch did not ask for. Every decision re-checks the authenticating key against the current snapshot. Without it a key revoked by a deployment kept its open session's authority until the client chose to reconnect — indefinitely, for a machine-to-machine consumer — which would have made the documented "revocation requires a deploy" sharp edge considerably sharper than documented.
- The trie is built inside
DataPlaneAuthorityState.Apply, not by a separate install step at the publish actor. As two wirings, the trie install is the one that silently does not run, and an absent trie denies everything — indistinguishable from working enforcement until someone checks that an ALLOW is possible. Falsified: deleting the install turns 13 of the 22 stage-5 tests red. - The deploy log now reports what the BUILD dropped instead of re-deriving it. The previous
unresolvedScopescount agreed with the builder on one of its four drop reasons; a rule with a blankScopeId, a retired scope kind, or a scope path not starting at a realm vanished with the log reading zero.
Also found and fixed en route: the principal never reached the session. The authenticator passed it as
OpcUaUserAuthResult.DisplayName, butRoleCarryingUserIdentityderivesDisplayNamefrom the presented token — which for every key-bearing client is the static usernameapikey. So the trie's matching key was being dropped at the identity boundary, and the auditClientUserIdon a rejected write recordedapikeyfor every consuming system.PrincipalandApiKeyIdare now carried explicitly. - The role check is REPLACED, not supplemented. Running both in series would require a grant to
confer the server-wide role as well as pass the scope check — and only
-
Enforce read/browse/subscribe. ✅ Done.
Read,Browse,TranslateBrowsePath,CreateMonitoredItems,SubscribeToEventsandSetMonitoringModeare gated at the node manager;SessionAclDecisionCachememoizes the grant walk per(session, node, operation, realm)and flushes on a generation change.Four things went differently from this line, all deliberate:
- Browse needed a new trie concept: ancestor visibility. The trie inherits downward, and a client
reaches a deep node by browsing the nodes above it — so without this, every grant narrower than the
cluster was authored, stored, shipped and unreachable. ⚠️ The obvious implementation, planting a
synthetic
Browsegrant on each ancestor, is a silent widening: the additive downward walk carries it fromPlantontoPlant/Siemens/dev9and every other sibling subtree. It is a separate, non-inheriting marker (BrowseTransitPrincipals) checked at the exact node only, set only by grants that actually carryBrowse, and conferring onlyBrowse. The consequence to accept: the names of the folders on the path to a granted node are visible to whoever holds that grant. - The cache holds the grant walk, not the decision. Grants and the trie are fixed for a
generation; a key's
ExpiresUtcpasses with the clock. Caching the whole decision would have let an expired key keep working until the next deploy — the stage-5 open-session hole in a subtler form. - Revocation had to reach LIVE subscriptions, which needed a push. Every other operation re-asks
per request; a subscription is authorized once and streams. Since subscriptions are how machine
consumers actually read this server, stopping only the re-asking operations would have made
revocation close to cosmetic.
IDataPlaneAuthoritySourcegained aChangedevent and the node manager re-authorizes every live item on it. ⚠️ A denied item is queuedBadUserAccessDeniedand then detached from its node's fan-out, not setMonitoringMode.Disabled— disabling discards queued notifications, silencing the client without telling it why. Caught by the over-the-wire test rather than by reasoning; the unit-level path reports the item as stood down either way. Detaching is also reversible, so a restored grant resumes the feed with no client action. TranslateBrowsePathis gated too, without which browse filtering is decorative: a client refused a browse would askTranslateBrowsePathsToNodeIdsfor the same path and get the NodeId.
Also found and fixed en route:
UserAuthorizationState's staleness ceiling couples two clocks.MembershipResolvedUtcis stamped fromDataPlaneAuthorizer'sTimeProviderwhileIsStaleis evaluated againstTriePermissionEvaluator's. They are the same instance in production and diverge the moment anything injects a test clock — at which point the evaluator denies everything with a verdict indistinguishable from "no grant matched". Stage 5's claim that these predicates "can never fire" was accidentally true, not true by construction; they are now disabled explicitly. - Browse needed a new trie concept: ancestor visibility. The trie inherits downward, and a client
reaches a deep node by browsing the nodes above it — so without this, every grant narrower than the
cluster was authored, stored, shipped and unreachable. ⚠️ The obvious implementation, planting a
synthetic
-
AdminUI. ✅ Done. Key management (create/revoke, show-once) landed early, in stage 4's predecessor. This stage added the scope picker (
AclScopePickerModalover the cluster's raw and UNS trees, backed byIAclScopeService), authoring-time scope and principal resolution, and reshaped the two banners.Four things went differently from this line, all deliberate:
- The banners were reshaped, not removed. "Remove the two warning banners" assumed the only thing
they said was not enforced, which stage 6 falsified. What remains true is that enforcement is a
per-node flag the AdminUI cannot see, so an
Equipment-scoped grant still confers nothing on a default-configured node — deleting the banner would have replaced one wrong impression with another. They are nowalert-infodescribing both modes, flag-on first. - The picker sets the kind AND the id, and there is no longer a hand-edit path. The plan asked
for "a
RawTreepicker forScopeId", which would have left the kind dropdown beside it and keptRawDevice+ a TagId authorable. Selecting a tree node supplies both, which makesNodeAclScopeCrossCluster,NodeAclScopeIdUnexpected,NodeAclScopeIdMissingandNodeAclScopeKindRetiredunauthorable by construction rather than merely discouraged. - It is a new component, not a third mode on
RawTree.RawTreealready carries the/rawediting tree and the equipment reference picker. A third mode would put the scope picker's bugs on the device-authoring page, and with no bUnit in this repo the only thing that would catch that is someone noticing.AclScopeTreerenders a unifiedAclScopeNodeinstead, which is also what lets ONE component serve both hierarchies —RawTreecovers only the raw side. - Saving is blocked on an unresolved scope, and the list page marks the rot. Not asked for; it is
what closes
ScopeIddangling silently (the open item at the top of this doc) rather than moving the discovery from deploy time to slightly-earlier deploy time. The check re-runs on submit rather than trusting the verdict from when the form loaded, because a row loaded unchanged is exactly the case where the operator never touches the picker.
Also fixed en route: the ACL editor had a
DataAnnotationsValidatorand noValidationSummary, so an invalidNodeAclIdmade the Create button a silent no-op — the project_adminui_no_bunit_live_verify failure mode, present in this page since it was written. - The banners were reshaped, not removed. "Remove the two warning banners" assumed the only thing
they said was not enforced, which stage 6 falsified. What remains true is that enforcement is a
per-node flag the AdminUI cannot see, so an
-
Rollout. Docs, the ScadaBridge key hand-off, and flipping
Security:Acl:Enabledon docker-dev as the live gate.
Stage 4 live gate — PASSED (docker-dev, 2026-07-31)
Run against the six-node docker-dev rig, with site-a-1 as the subject: Cluster:Roles = driver,cluster-SITE-A, no ConnectionStrings:ConfigDb, ConfigSource:Mode=FetchAndCache. Before this
stage that node registered NullOpcUaUserAuthenticator and denied every UserName token.
The key was minted with an independent Python HMAC rather than by the C# minter, so the verifier was checked against something other than itself.
| # | Leg | Result |
|---|---|---|
| 1 | Boot from LocalDb cache installs an authority generation | generation 1 installed (cluster=SITE-A, revision=3f4a908c, keys=0, grants=0) — cluster identity resolved with no ConfigDb |
| 2 | Deploy a key + a SITE-A cluster grant | generation 2 installed (revision=cb165478, keys=1, grants=1); revision matches the deploy response |
| 3 | The key authenticates on the driver-only node | ApiKeyAuth: granted key Stage4 Live Gate (a1b2c3d4e5f60011) as principal stage4-gate with roles [WriteOperate] |
| 4 | A wrong secret is denied there | denied — secret mismatch for key id a1b2c3d4e5f60011 |
| 5 | Revocation asymmetry, both halves | Enabled=0 in SQL, no deploy: central-1 denies on the next request (no enabled, unexpired key…) while site-a-1 still accepts. After deploying: generation 3, and site-a-1 denies. This is the documented sharp edge, observed rather than asserted. |
| 6 | Grants do not leak across clusters | site-b-1 installed keys=1, grants=0 from the same artifact — it received the key but not SITE-A's grant |
Rig left clean (0 keys, 0 ACL rows, redeployed, all six nodes healthy).
Live gate — PASSED (docker-dev, 2026-08-01)
Subject: site-a-1 (driver,cluster-SITE-A, no ConfigDb, ConfigSource:Mode=FetchAndCache) with
Security:Acl:Enabled=true. Control: central-1, same artifact, flag unset. All six nodes were
rebuilt onto the gate commit first and image parity asserted per container, because :9200
round-robins central-1/central-2 and a stale central-2 rejected the deploy outright (Cannot convert string value 'RawDevice' … enum — its image predated the stage-3 scope kinds).
The key was minted with an independent Python HMAC, so the verifier was checked against something
other than itself. One grant throughout: RawDevice → SITEA-dev, i.e. raw/sa-modbus2/plc.
| # | Leg | Result |
|---|---|---|
| 0 | The grant is placed, not dropped | generation 2 installed (keys=1, grants=1, resolvedScopes=1), no dropped-rule warning. This is the leg that distinguishes working enforcement from a trie that silently denies everything |
| 1 | Read INSIDE the grant | Value: 16100, Status: 0x00000000 |
| 2 | Read OUTSIDE it (sibling device plc2) |
BadUserAccessDenied |
| 3 | Browse the driver, gated | plc only — plc2 filtered out |
| 4 | Browse the same driver on the control | plc and plc2 — the A/B that shows the filtering is the ACL and not the address space |
| 5 | Browse the ungranted device directly | BadUserAccessDenied (the throw path) |
| 6 | Anonymous on the gated node | BadUserAccessDenied — principal anonymous holds no grant |
| 7 | Write with a grant lacking WriteOperate |
0x801F0000 |
| 8 | Same write after granting WriteOperate (+redeploy) |
Write successful |
| 9 | Wrong secret | BadIdentityTokenRejected at activation |
| 10 | Revoke mid-subscription | see below |
| 11 | Restore the grant | see below |
Legs 10–11 are the ones worth reading as a transcript, because they are what stage 6 added and what no offline test could settle:
13:49:39 SiteAOnly = 53 (0x00000000) ← streaming, ~1/s
13:49:40 SiteAOnly = (0x801F0000) ← the revoking deployment lands; the client is TOLD
… 25 seconds of complete silence — ~25 updates that did not arrive …
13:50:05 SiteAOnly = 157 (0x00000000) ← the restoring deployment lands; the feed resumes
Note 53 → 157: the device counter ran throughout, so it was delivery that stopped, not the
source. The 0x801F0000 frame is the half that the MonitoringMode.Disabled implementation would
have swallowed.
Not proven by this gate: a UNS-realm grant (the rig's SITE-A cluster has no UNS equipment), and
alarm-command scoping (no alarms configured on SITE-A). Both are covered offline; neither has been
seen live. Reads on the control node returned BadNodeIdUnknown rather than a value — central does
not materialise another cluster's tag variables — so leg 4's browse is the meaningful A/B, not a read.
Found en route, unrelated to ACLs: the rig's SITEA-modbus driver had an empty DriverConfig, so
its factory rejected the config and the driver ran as a stub serving nothing. Fixed on the rig
(Host/Port added) — without it every read returns BadWaitingForInitialData and the allow legs
cannot be told from a denial.
Rig left as found: key, ACL row, the second device and its tag all deleted and redeployed; site-a
returned to Security:Acl:Enabled unset, confirmed by the absence of the startup banner and an
anonymous read succeeding again.
Stage 7 live gate — PASSED (docker-dev, 2026-08-01)
Driven through the AdminUI at :9200 against cluster SITE-A. Both central containers were asserted
to be running the same freshly-built image by container id, not from one docker compose build — the
stale-central-2 trap from the stage-6 gate.
| # | Leg | Result |
|---|---|---|
| 1 | Picker lazily expands the real raw tree | ✅ cluster → sa-modbus2 → plc → SiteAOnly, one level per expand |
| 2 | Each level offers its own scope kind | ✅ CLUSTER / RAWDRIVER / RAWDEVICE / TAG chips |
| 3 | Selecting a node sets kind and id | ✅ → RAWDEVICE + SITEA-dev, "✓ Device 'plc' in cluster 'SITE-A'" |
| 4 | Principal reaching nobody warns | ✅ not-a-real-principal → ⚠ "No active API key uses this principal" |
| 5 | …and the positive control | ✅ anonymous → ✓ "The reserved anonymous principal" |
| 6 | Grant saves and lists with its resolved target | ✅ |
| 7 | A rotted row is marked on the list | ✅ id broken by SQL → UNRESOLVED chip + reason |
| 8 | Saving a rotted row is blocked | ✅ "Scope: No device in this configuration has the id …" |
| 9 | Re-picking a valid scope unblocks it | ✅ NAMESPACE raw → saved |
| 10 | The authored grant deploys | ✅ POST /api/deployments → Sealed |
| 11 | The deploy gate rejects what the page rejects | ✅ same row broken by SQL → HTTP 422 NodeAclScopeUnresolved |
| 12 | UNS column offers area / line / equipment | ✅ UNSAREA / UNSLINE / EQUIPMENT, eager, after a UNS was authored under SITE-A |
| 13 | Selecting an equipment resolves | ✅ EQUIPMENT + EQ-f3dd0e8e27a4, "✓ Equipment 'packer-1' in cluster 'SITE-A'" |
| 14 | A UNS-realm grant deploys | ✅ Sealed |
| 15 | A cross-cluster scope is caught at both ends | ✅ re-pointed at MAIN's equipment → list marks it UNRESOLVED naming MAIN; deploy → HTTP 422 NodeAclScopeCrossCluster |
Legs 8 and 11 are the pair that matters: the page and the deploy gate reject the same row for the same
reason, which is the claim "CheckScopeAsync mirrors DraftValidator" reduces to. Leg 11 also
confirmed the kind is honoured on both sides — the row was re-pointed at RawDriver, and the gate's
message named DriverInstance.
Legs 5 and 9 are the positive controls. Without them the gate passes against a page that refuses
everything, which is the same failure shape stage 6's resolvedScopes=N leg exists to catch.
Legs 12–15 were added in a second pass after authoring a UNS (area → line → equipment) under SITE-A, which had none when the first pass ran. Leg 15 is the one worth keeping: it is the only rejection whose id is real, so a check that asked only "does a row with this id exist?" would accept it and confer authority across a cluster boundary. Both ends name the owning cluster.
Found while authoring that UNS, unrelated to this stage: the /uns area and line modals do
not validate the UNS segment format, so "Packaging" and "Line 1" saved happily and then failed the
deploy with UnsSegmentInvalid — fleet-wide, since a deployment snapshots the whole database, so one
bad area name blocks every cluster's deploy. Only EquipmentPage.razor:570 carries the
^[a-z0-9-]{1,32}$ rule. Exactly the authoring-vs-gate gap this stage closed for ACLs. Filed
separately; not fixed here.
Rig state afterwards: the gate's ACL rows deleted and a further deployment sealed, leaving SITE-A at
zero ACL rows as found. The UNS structure was kept (area packaging → line line-1 → equipment
packer-1, renamed to valid segments), since SITE-A having a UNS is what made legs 12–15 possible and
keeps them re-runnable.
Known consequence to design around
Driver-only nodes have no ConfigDb (Phase 4). Keys and ACLs must therefore reach them inside the
deployment artifact, exactly as NodeAcl rows already do. That means key revocation requires a
deploy — there is no out-of-band revocation channel to a site node. This is a real operational
sharp edge and must be documented rather than discovered; if immediate revocation is needed later,
the Phase 5 gRPC telemetry stream is the existing node-reachable channel to carry it.