With Security:Acl:Enabled=true, a session holding any grant narrower than
Cluster could not browse the served address space at all — confirmed by the
live gate.
Mechanism. Both realm trees hang under ONE anchor folder minted in
CreateAddressSpace: NodeId("OtOpcUa", <raw ns>). ResolveParentFolder parents
every top-level raw folder AND every top-level UNS area under it, and the
Browse override throws BadUserAccessDenied for the WHOLE call when its source
node is denied. DataPlaneAuthorizer.BuildScope split that identifier like a
path, producing {Kind: Raw, Segments: ["OtOpcUa"]} — but trie grant paths are
["raw", <RawPath segments…>] and "OtOpcUa" is never a path segment, so
PermissionTrie.HasBrowseTransit failed at the realm-child lookup and
CollectMatches yielded only Cluster-scoped grants. Deny, at the front door.
The trie already handled a zero-segment scope as "the realm root, itself an
ancestor of everything granted in that realm" (PermissionTrie.cs:101-118) —
no production path had ever produced that scope. This wires it up.
- V3NodeIds.SharedRootFolderId names the anchor identity; CreateAddressSpace
now mints the NodeId/BrowseName/DisplayName from it (behaviour-identical,
makes the coupling explicit).
- BuildScope maps that identifier (realm-qualified to Raw — in the UNS
namespace s=OtOpcUa is an ordinary Area) onto the zero-segment Raw scope.
- Evaluate adds a second-realm OR-leg for Browse on the anchor only: it fronts
both trees, so a principal whose only Browse grant lives in the UNS tree
still enters through this raw node. Browse only; transit confers navigation,
never read/write/subscribe.
- No trie, evaluator or builder change. No reordering above the decision cache
— key revocation is still re-checked before the lookup — and the cache key
is unchanged (the combined verdict caches under the anchor's own
identifier/realm; a generation advance clears it).
Falsifications actually run (implement-then-revert-the-line, plus one
genuine red-first):
- DataPlaneAuthorizerTests: the 7 new facts were written BEFORE the fix and 4
went red (the 3 deny-only ones passed vacuously, as a deny-only test must).
- Deleting the UNS OR-leg reddens exactly one fact,
The_shared_root_folder_is_browsable_on_a_uns_only_browse_grant (26/27 pass).
- Dropping the realm qualification from IsSharedRootFolder reddens exactly
A_uns_area_named_like_the_anchor_gets_no_anchor_semantics. Its first draft
did NOT catch this — a raw-only grant is refused either way; the
discriminating grant had to be an unrelated UNS one, whose own transit marks
the zero-segment UNS scope the un-qualified mapping would land on.
- Disabling the BuildScope anchor branch reddens the new end-to-end fact
NodeManagerStage6EnforcementTests.The_shared_root_folder_is_browsable_on_a_narrow_grant_and_still_filters
(real booted server, real Browse dispatch: BadUserAccessDenied on the whole
call), which also pins that the reference filter still withholds the
non-granted sibling.
- Core PermissionTrieBrowseTransitTests gains 3 characterization facts pinning
the previously-dead realm-root branch (transit true for the granting realm's
root, false for the other realm's, false without the Browse bit). These were
green on arrival — the trie was always correct; nothing called it.
Tests: Core 351 -> 354 passed; OpcUaServer 482 -> 490 passed. Build clean.
Review follow-ups to 34829ea0.
IMPORTANT - the leak filter had its own flake vector. This test saturates the
thread pool by design, and TimedScriptEvaluator starts its 5s clock at WaitAsync
rather than when the inner Task.Run gets a thread, so a legitimately-enqueued
evaluation can blow the budget while still queued and surface "script timed out
after 5.0s". The reason != exhaustion filter classified that as a raw disposed
escape and hard-failed on it - handing the test back the load-fragility the
reshape removed. Timeouts are now their own bucket, tolerated and rate-bounded
exactly as exhaustions are; a raw disposed escape still fails on first occurrence.
Doc-comment names this second residual vector alongside the exhaustion calibration.
MINOR (scope extension authorised by review) - "evaluator disposed during
evaluation" existed as three unshared literals. Promoted to
ScriptEvalFailureReasons alongside VirtualTagEvalResult in Commons.Engines (the
assembly all three already reference) and referenced from both evaluators and the
test. The timeout prefix joins it, since the test now classifies on it too. String
values are byte-identical - no behaviour change.
Falsified: with a 12ms script budget the race produced real timeouts (3/2695).
Classified as leaks (the pre-review filter) they hard-failed the test 6/6 items;
bucketed as starvation they sit 0.11% under the 10% bound and the test is green.
TagConfigJson.ClassifyEnumText classified via Enum.TryParse(text, ignoreCase: true),
which accepts a numeric STRING for any enum: "1" returns the member at that ordinal
and an out-of-range "99" parses just as happily. Every driver contract's strict enum
read inherits that, so a Sql tag authored "type":"99" was ACCEPTED, carried
(DriverDataType)99, and SqlPollReader.TryCoerce fell through its switch to the String
arm — a stringified cell published Good under a type nothing can map.
Refuse a digit- or sign-leading spelling before the parse (judged on the TRIMMED text,
because Enum.TryParse trims), and require Enum.IsDefined — which covers what the
leading-character check cannot, since TryParse also ORs a comma-separated list for a
non-[Flags] enum. Mirrors RawBrowseCommitMapper.ParseNativeType and
MTConnectTagDefinitionFactory.TryParseDataType, which already guard their own parses.
Three AdminUI restatements of the same rule now ask ClassifyEnumText instead of
Enum.TryParse (TagConfigJson.GetEnum, SqlTagConfigModel.ReadEnumStrict,
MqttTagConfigModel.DescribeInvalidEnum/GetEnumNullable) — leaving them would have opened
the #549 direction the moment the runtime tightened: a blob the modal accepts and the
driver skips.
RED first: 10 failing in Core.Abstractions.Tests, 4 in Driver.Sql.Tests, and 10 of the
11 new fleet-wide parity cases (re-confirmed by reverting the Core fix).
MTConnectTagConfigModel read `dataType` with TagConfigJson.GetEnum, which falls
back to String on ANY unrecognised value. A misspelled name ("Strng") from a
hand-authored blob, a CSV import or the API was therefore accepted at authoring
and silently rewritten, while the driver's strict parse rejected it and the tag
died with BadNodeIdUnknown. Found by the G-7 parity tests' falsifiability
control, which had to use the digits-only shape to have a case that failed.
The rule is not restated in the AdminUI. MTConnectTagDefinitionFactory exposes
its parse as `IsAcceptedDataTypeName`, documented as the one authority the
driver, the deploy gate and the authoring model all consult; the model's
Validate() asks it about the raw value the blob carries. The old digits-only
message is NOT retained as a special case: the runtime draws no distinction
between a bare number and a typo — one predicate, one Inspect warning — so a
second message here would be a second statement of a rule this model does not
own. The raw value is captured as the JSON TOKEN, because the runtime
deserialises the key into a `string?` DTO field and an unquoted number throws
there too.
The survey found the same asymmetry at six more editors, contrary to the
assumption that those runtimes silently default: R2-11 Phase C moved every
driver's TagConfig enum read to TagConfigJson.TryReadEnumStrict, which REJECTS
the tag on a present-but-invalid string, while the editors kept loading them
with a silent fallback. Modbus (region/dataType/byteOrder), S7, AbCip, AbLegacy,
TwinCAT and FOCAS now check their enums in Validate() through the runtime's own
TagConfigJson.DescribeInvalidEnumText — added alongside a public
ClassifyEnumText that TryReadEnum now delegates to, so the JSON-token door and
the string door reach one implementation. Mqtt and Sql already carried
equivalent strict checks and are untouched.
Care taken NOT to become stricter than the runtime: a present-but-non-string
token stays accepted for those six (TryReadEnum calls it absent), as does an
absent key, each pinned by a test.
Also adds a direct ProjectReference to Core.Abstractions from the AdminUI — the
save gate now leans on a type it was consuming only transitively.
The connect path was the one capability call the resilience pipeline never
saw, so a driver that could not connect contributed zero breaker input and
the /hosts resilience panel read 'All targets healthy' against a dead
endpoint. InitializeAsync now routes through the invoker as
DriverCapability.Initialize, keyed on the driver-instance id.
- No pipeline retry: the actor's RetryConnect timer is the retry loop
(parser forces retryCount overrides to 0, same funnel as write-shaped).
- Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and
the reconnect timer fires every 10 s, so at most three attempts fit one
window — threshold 5 could never open. Pinned by its own test.
- An open breaker fast-fails into the same InitializeFailed path as a real
connect failure; the 15 s break window costs at most one extra tick.
The best-effort teardown ShutdownAsync stays unwrapped.
- Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the
instance's tracker snapshots with its pipelines — a respawn during an
outage discarded the only breaker whose OnClosed could clear the OPEN
flag, stranding a false 'Breaker OPEN' chip forever.
Live gate PASSED on docker-dev (image symbol-verified per container):
rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on
both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat
5021→5020 forwarder recovered it (both breakers closed within 1 s, panel
back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker
events. Full suite 50/50 green. Evidence:
docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
#542 filed FleetStatusBroadcaster as blind to site nodes since the Phase 6 per-cluster mesh
split: it derives "the fleet" from central's own Akka gossip ring, which after the split is
just central's own pair. Scoping the cross-mesh rewrite found the deeper truth — the whole
chain is dead end to end, so this deletes it rather than reviving it.
- Zero consumers. FleetStatusBroadcaster published FleetStatusChanged on the DPS
"fleet-status" topic; FleetStatusSignalRBridge forwarded it to FleetStatusHub
("/hubs/fleet-status"); nothing anywhere connected. The AdminUI bans self-connecting
HubConnections, no browser JS subscribes, and /fleet (Fleet.razor) reads
NodeDeploymentState from the ConfigDb plus IClusterRoleInfo on a 5 s poll instead.
- Its second input never existed. DriverHostStatusHeartbeat had no production sender —
only the broadcaster's own test told it — so CurrentRevision was always null and the
30 s staleness -> Degraded flip could never fire.
- Post-Phase-6 DPS is pair-local, so even the central-only view it did produce was an
accident of topology rather than a design.
The fleet visibility that actually works is unchanged and already cross-mesh-correct:
/fleet over NodeDeploymentState, and the four Phase 5 per-node telemetry dials.
Precedent for deleting inert machinery rather than activating it: #522 (driver stability
tiers), #507 (discovered-node injection), #525 (GetMemoryFootprint).
Deleted: FleetStatusBroadcaster (+ its nested DriverHostStatusHeartbeat), its admin
singleton registration and FleetStatusBroadcasterKey, FleetStatusSignalRBridge and its
key, FleetStatusHub and its route mapping, and the FleetStatusChanged / FleetNodeStatus /
FleetNodeHealth messages. Both Fleet/ directories survive — ControlPlane keeps
ClusterNodeAddressReconciler, Commons keeps GetDiagnostics. Docs swept where they claimed
a fleet-status channel exists (Telemetry.md's Phase 5 non-migrated channels,
Configuration.md, CLAUDE.md); completed-program plan docs under docs/plans and docs/v2 are
left as the historical record they are.
Build 0 errors, 868 warnings — byte-identical to master's rebuild baseline.
ControlPlane.Tests 231/231, AdminUI.Tests 1109/1109.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
MeshPaths.CentralCommunication / .NodeCommunication (the "/user/..." full-path constants) had
ZERO production consumers after the ClusterClient deletion: both gRPC services resolve their
comm actor through ActorRegistry by typed key, never by ActorSelection over a path. Only the
path-probe test still read them, while the class doc went on calling them "the wire contract
of the central↔node boundary" — a claim that stopped being true the moment nothing addressed
an actor across a process boundary by name.
- The two full-path constants are deleted; the two Name constants remain.
- MeshPaths' doc now says what it is (the local actor-naming convention the registry-keyed
services are spawned under) and records what it stopped being, so the next reader does not
re-derive a contract from the file's history.
- MeshCommActorPathTests composes "/user/" + Name itself — the absolute path is that test's
own addressing detail now, not something the product declares. Renamed
Both_mesh_comm_actors_exist_at_their_contract_paths ->
..._are_spawned_under_their_declared_names, because what it still pins is that the two
registration sites actually ran on a real booted node; a rename is now a compile error, so
"contract" was the wrong word for what would break.
- NodeCommunicationActor's summary no longer cross-references the deleted constant or calls
the name an address. CentralCommunicationActor was checked and makes no such claim.
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
MeshTransportOptions.ModeClusterClient is gone. MeshTransportOptionsValidator accepts Dps or
Grpc and nothing else, so a node left carrying the string "ClusterClient" is now REFUSED at
boot naming MeshTransport:Mode and both remaining values — deliberately fail-closed rather
than falling back to Dps, which on a split-topology node would deliver nothing at all. A new
fact pins exactly that.
SplitTopologyTransportValidator tightens from an allow-list of two to Grpc only, as its own
comment said it would once ClusterClient was deleted. Dps on a cluster-role node stays
refused; the rule is still an explicit allow-list, so a typo'd mode fails too.
Tests: the two ClusterClient-mode facts in MeshTransportOptionsValidatorTests are deleted
(their Dps siblings already assert the same rule for the one remaining non-Grpc mode), the
SplitTopology fixtures move to Grpc, and the three-mode Theories in
ControlPlaneStarterOrderTests / CentralCommunicationActorTests become two-mode.
The property, the four validator rules that graded it (empty / non-akka.tcp / actor-path
suffix / one-bad-among-good) and all twelve docker-compose entries (6 nodes x 2) are gone.
The compose comment blocks that explained "/system/receptionist is appended at construction
time" and the client's contact rotation go with them, and the surrounding prose now
describes the Grpc command plane it actually runs.
CentralGrpcEndpoints' doc no longer says it "replaces" the Akka contact points — there is
nothing left to replace — and ContactRefreshSeconds' failure message names the node
dispatch set rather than the ClusterClient contact points.
`rg CentralContactPoints src tests docker-dev` is now empty.
akka.extensions loses ClusterClientReceptionistExtensionProvider and the whole
akka.cluster.client block (buffer-size / reconnect-timeout / receptionist.role) goes with
it. The DistributedPubSub extension and the maximum-frame-size / log-frame-size-exceeding
canary both stay — DPS is the pair-local transport and the frame canary guards every
remoting message, neither of which the cross-mesh deletion touches.
MeshTransportHoconTests drops its three ClusterClientSettings/ClusterReceptionistSettings
asserts and keeps the frame-size pin and the DPS-extension pin, renamed since it no longer
has a receptionist to survive alongside.
Found on the docker-dev live gate, not in tests. Two drivers that faulted
during initialise (rtu-gate, opcua1) each rendered "0 tags" on /hosts against
two authored tags apiece, because a ledger that had never committed returned
SkippedTagReport.Empty.
A zero report is a positive claim -- "this driver skipped nothing" -- and
making it on behalf of a driver that never got as far as looking is exactly
the quiet inaccuracy #541 exists to remove. GetSkippedTags() and
SkippedTagLedger.Snapshot() now return null until the first committed build,
so the column renders the same em-dash it already used for a driver that
cannot report at all. Both are 'no answer'; neither is 'nothing skipped'.
Knock-on: a FOCAS driver whose only init threw now reports null rather than a
zero report, which is strictly more honest than the assertion the test
originally made.
A driver builds its RawPath -> definition table at init and tolerates a
per-tag miss: an unmappable TagConfig, an unparseable address, a
capability-matrix rejection. That tolerance is right -- one bad tag must not
take down the other few thousand. What was missing is any statement of the
fact afterwards. A skip is not an error and happens before any read, so the
driver stays Healthy, errors/5 min stays 0, Last error stays empty, and Last
read advances live off the tags that did map. On the wonder Makino in #540
that reading held while focas-z-34184 ignored 20 of its 22 tags.
New optional capability ISkippedTagReporter, backed by a shared
SkippedTagLedger so the reset/commit/cap rules are one implementation rather
than one per driver. The report -- authored count, exact skipped count, and a
capped sample of RawPath + reason -- rides DriverHealthChanged to a new Tags
column on /hosts, over the Phase 5 telemetry stream. No table: a driver-only
node has no ConfigDb to write to, the lesson from #521.
The issue named eight drivers. It is twelve -- every driver in
DriverTypeNames.All has an authored-tag skip path. Calculation (no scriptId),
MTConnect (no readable DataItem id), Galaxy (no attributeRef) and OpcUaClient
(no nodeId) were unlisted, and OpcUaClient's was the quietest of the twelve:
NamespaceMap drops such an entry with a bare `continue` and not even a log
line. SkippedTagReporterFleetTests pins the invariant reflectively over every
IDriver in src/Drivers/, so a thirteenth cannot re-introduce the silence.
Sql drops a tag at TWO points and its report is the union of both: a blob
that does not map, AND an identifier the 8.1 catalog gate rejects against the
live catalog. The second is what schema drift produces -- a renamed column, a
dropped view -- months after a deployment sealed green, and reporting only
the first would have that driver claim "0 skipped" while polling half its
tags, which is worse than reporting nothing because the column would then be
actively reassuring.
Deliberately additive to driver state, never a change of it. Latching
non-Healthy for a config problem would satisfy both halves of the #527
comms-loss conjunction -- 90 s without a successful read AND non-Healthy --
and a driver whose tags were ALL skipped has no successful reads by
construction, so it would reconnect-loop forever against a healthy device.
Three details that are load-bearing rather than incidental:
- The skip digest is part of the publish fingerprint. A skipped tag leaves
every other fingerprint component identical to a clean driver's, so without
it PublishHealthSnapshot's dedup swallows exactly the publish carrying the
signal -- the trap that already bit the rediscovery and host-status
channels. The digest covers the sample, not just the counts: a redeploy
that fixes one tag and breaks another leaves both counts unchanged.
- The count is exact; only the sample is capped, and the tooltip says so when
it is showing fewer than it counted. A truncated list that looked complete
would have an operator fix twenty tags and believe the job done.
- The ledger stages and commits, so an init that throws part-way leaves the
previous report standing. A partial snapshot would read as "we now skip
fewer tags", the opposite of the truth.
The proto field needs no companion presence flag, unlike host_statuses: a
singular message field carries real presence in proto3, and a has_ flag would
collide with protoc's generated HasSkippedTags.
Verified red-first three ways: dropping SkippedTagDigest from the fingerprint
fails the two dedup tests, removing the seam from one driver fails the fleet
guard, and dropping the catalog-gate leg fails the Sql union test. Full
solution builds with zero new warnings against 4e22f024.
Joseph Doherty