97854a074e8eb679c5373b596a8a33d268d6483e
735 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
cd7135570c |
refactor(mesh): collapse MeshPaths to local actor names — no longer a wire contract (review follow-up)
MeshPaths.CentralCommunication / .NodeCommunication (the "/user/..." full-path constants) had
ZERO production consumers after the ClusterClient deletion: both gRPC services resolve their
comm actor through ActorRegistry by typed key, never by ActorSelection over a path. Only the
path-probe test still read them, while the class doc went on calling them "the wire contract
of the central↔node boundary" — a claim that stopped being true the moment nothing addressed
an actor across a process boundary by name.
- The two full-path constants are deleted; the two Name constants remain.
- MeshPaths' doc now says what it is (the local actor-naming convention the registry-keyed
services are spawned under) and records what it stopped being, so the next reader does not
re-derive a contract from the file's history.
- MeshCommActorPathTests composes "/user/" + Name itself — the absolute path is that test's
own addressing detail now, not something the product declares. Renamed
Both_mesh_comm_actors_exist_at_their_contract_paths ->
..._are_spawned_under_their_declared_names, because what it still pins is that the two
registration sites actually ran on a real booted node; a rename is now a compile error, so
"contract" was the wrong word for what would break.
- NodeCommunicationActor's summary no longer cross-references the deleted constant or calls
the name an address. CentralCommunicationActor was checked and makes no such claim.
|
||
|
Joseph Doherty
|
4d2dfb5b50 |
test(mesh): step 7 — replace the boundary test with MeshGrpcBoundaryTests and sweep the comments
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
|
||
|
Joseph Doherty
|
af3b0c3146 |
refactor(mesh): step 6 — delete the ClusterClient mode value; split topology now requires Grpc
MeshTransportOptions.ModeClusterClient is gone. MeshTransportOptionsValidator accepts Dps or Grpc and nothing else, so a node left carrying the string "ClusterClient" is now REFUSED at boot naming MeshTransport:Mode and both remaining values — deliberately fail-closed rather than falling back to Dps, which on a split-topology node would deliver nothing at all. A new fact pins exactly that. SplitTopologyTransportValidator tightens from an allow-list of two to Grpc only, as its own comment said it would once ClusterClient was deleted. Dps on a cluster-role node stays refused; the rule is still an explicit allow-list, so a typo'd mode fails too. Tests: the two ClusterClient-mode facts in MeshTransportOptionsValidatorTests are deleted (their Dps siblings already assert the same rule for the one remaining non-Grpc mode), the SplitTopology fixtures move to Grpc, and the three-mode Theories in ControlPlaneStarterOrderTests / CentralCommunicationActorTests become two-mode. |
||
|
Joseph Doherty
|
ab5b958d47 |
refactor(mesh): step 2 — delete the node-side ClusterClient and its centralClient parameter
WithOtOpcUaRuntimeActors no longer builds a ClusterClient from MeshTransport:CentralContactPoints, and NodeCommunicationActor.Props takes only the gRPC IApplyAckSender. The ackRouter gate narrows from `useClusterClient || useGrpc` to `useGrpc`, and the CentralClusterClientName constant is gone. NodeCommunicationActor's no-transport warning now names MeshTransport:CentralGrpcEndpoints and MeshTransport:Mode only — the old text pointed an operator at a key that no longer exists. Ripples in this step: the two ClusterClient-shaped facts in NodeCommunicationActorTests (Send-not-SendToAll, and sender-wins-over-a-client) are deleted — the first has no subject left, the second guarded a both-transports-present state that is now unconstructible. The Runtime ServiceCollectionExtensions fact keeps its ackRouter pin and drops its "no ClusterClient actor exists" half for the same reason. |
||
|
Joseph Doherty
|
0db40a5e7f |
refactor(mesh): step 1 — delete the central ClusterClient factory and per-cluster client cache
IMeshClusterClientFactory/DefaultMeshClusterClientFactory are gone, along with
CentralCommunicationActor's _clients map, RebuildClient, the SendToAll fan and the
receptionist-address half of the contact load. LoadContactsFromDb SURVIVES — it feeds the
gRPC dialer's endpoint resolution and the known-cluster gate — but now projects only the
columns those two need (the AkkaPort/ActorPath.TryParse parse and its Malformed list went
with the receptionist addresses).
Ripples fixed in this step because the build must be green at every commit:
- ControlPlane/ServiceCollectionExtensions drops the factory argument.
- CentralCommunicationActorTests: the six ClusterClient-only facts are deleted (each has a
Grpc counterpart that already asserted the same property independently), the three
transport-agnostic facts move to Grpc mode, and the Dps fact keeps its own coverage.
- MeshClusterClientFactoryTests deleted (its subject is gone).
- MeshClusterClientBoundaryTests deleted here rather than in step 7: it drives
DefaultMeshClusterClientFactory directly, so it cannot compile past this commit. Its
replacement, MeshGrpcBoundaryTests, lands in step 7 and keeps the load-bearing assertion
(two separate meshes, no cross-mesh Akka association).
|
||
|
Joseph Doherty
|
027f0403c4 | test(mesh): role-scoping scan handles braceless if (hasAdmin) — closes the phantom-block false pass | ||
|
Joseph Doherty
|
d7c9ead64e |
test(mesh): close review findings on the PreStart-race logging fix — non-vacuous once-per-incarnation pin, broken-dialer symmetry, comment idiom
Follow-up to
|
||
|
Joseph Doherty
|
2e06dfe52d | test(mesh): pin the admin-only role scoping of IMeshPskProvider/IMeshCommandDialer registrations (security register item 3) | ||
|
Joseph Doherty
|
a94eb8a335 |
fix(mesh): PreStart resolve race logs honestly — quiet on heal, loud on permanent absence
PreStart's eager resolve of IMeshCommandDialer/IMeshPskProvider loses a
millisecond-wide race against the Akka builder's remaining starters on
essentially every admin boot (live gates: 1A 2/3 boots, 1B 4/4 boots) and
self-heals at the memoized Dialer/PskProvider properties' first use. Before
this change that produced two FALSE Errors per healthy boot, which made a
soak error-grep on a healthy rig impossible.
Resolve() now takes a `firstUse` flag and logs at Debug on BOTH call sites
(PreStart and the memoized properties' retry) — from inside Resolve a lost
race and a genuinely broken factory are indistinguishable, so severity is
decided by the CONSUMER of a still-null result after the retry, not by
Resolve itself:
- Dialer: DispatchOverGrpc's existing per-dispatch Error ("no command
dialer is wired") is unchanged and is now the ONLY loud signal for that
dependency.
- PskProvider: PublishKnownClusters gained a new Error carrying the old
PreStart guard's diagnostic content (gate never published, cached keys
never dropped, rotated key needs restart), guarded to fire ONCE per
actor incarnation via `_pskProviderAbsenceReported` — that handler runs
on the contact-refresh timer for the actor's whole life, so an
unguarded Error there would repeat every ContactRefreshSeconds forever.
Design choice for the "factory never supplied" case (Dps/ClusterClient test
constructions that pass no factories at all): kept silent at BOTH call
sites for both dependencies, same as before. For the dialer this was
already true (DispatchOverGrpc's check doesn't care why the dialer is
null). For the PSK provider the new once-guarded Error is gated on
`_pskProviderFactory is not null` specifically so the many existing
ClusterClient/Dps facts in CentralCommunicationActorTests that never wire a
PSK provider (using the bare Spawn helper) keep passing unmodified — a
never-supplied factory is a deliberate wiring/test choice, not a resolve
failure, and Resolve() already stayed silent about it for the same reason.
Two new tests pin the net behavior: a healthy boot (factories that throw
once then succeed) produces zero Warning/Error end-to-end through PreStart,
the first contact load and the first dispatch; a permanently-broken PSK
provider logs the absence exactly once across two contact loads, not once
per refresh.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
|
||
|
Joseph Doherty
|
07bc1878fc | docs(mesh): sweep the remaining doubled-quorum comment sites — the coordinator absorbs duplicates idempotently | ||
|
Joseph Doherty
|
667639e6ab |
fix(mesh): per-endpoint deadline + review fixes — failover survives a hanging central
CRITICAL. The ack deadline was one absolute DateTime computed once for the whole endpoint walk. Against the characteristic WAN failure — a central that HANGS (SYN black-hole, frozen host) rather than refusing — endpoint 1 burned the whole AckTimeoutSeconds and endpoint 2 was then dialled with a deadline already in the past: instant DeadlineExceeded, no packet leaving the node, the surviving central never really tried. Failover looked like it ran and did not. It also contradicted MeshTransportOptions' own documented "applied per endpoint, not per ack". The deadline is now computed fresh inside the loop; headers stay hoisted (read-only, sequential walk). Two test gaps that let that survive are closed: nothing pinned the ackRouter gate (narrowing it back to ClusterClient-only compiled and passed every test while dead-ending every Grpc ack to DPS), and the deadline fact's lower bound accepted any positive timeout at all. Both new pins are mutation-verified. Docs corrected where they overstated the case: duplicate acks CANNOT mis-seal a deployment — ConfigPublishCoordinator.HandleAck keys _acks by NodeId behind an expected-set guard, so a duplicate overwrites its own entry. The single-endpoint rule is defence in depth and a bound on wasted work, not a correctness dependency; and the walk is at-least-once on the deadline path, not at-most-once. |
||
|
Joseph Doherty
|
308c6aaa0f |
feat(mesh): gRPC ApplyAck path behind Mode=Grpc
Task 1B.2 — the node→central half of the ClusterClient→gRPC migration. GrpcApplyAckSender walks MeshTransport:CentralGrpcEndpoints in order and STOPS AT THE FIRST SUCCESS. That single `return` is the whole single-count guarantee: the transport it replaces was ClusterClient.Send, deliberately not SendToAll, because both central nodes forward to the SAME ConfigPublishCoordinator singleton proxy — an ack posted to both would have the coordinator count one node twice and could seal a deployment on a doubled count. The list is an availability list, not a fan-out list. No retry beyond that one ordered pass, no queue: the receptionist buffered nothing and retried nothing, and this is like-for-like, not an upgrade to delivery guarantees. All endpoints failing is still a DROP — but a loud one, naming the node, deployment, correlation id and the status each endpoint answered with, which is what separates "central is down" from "central rejected this node's key". The bearer never reaches a log; only endpoints and status codes do. Per-endpoint GrpcChannel cache and the endpoint loop clone GrpcDeploymentArtifactFetcher; the cluster id in x-otopcua-cluster comes from this node's own cluster-<id> role via RoleParser, off the same bound Cluster:Roles ClusterRoleInfo reads. SendAsync never throws: the caller discards the task, so an escaping exception would become an unobserved fault — a drop with no log line, the exact failure mode this migration removes. NodeCommunicationActor gains an OPTIONAL ackSender; every existing test compiles and passes unedited. It is checked before the ClusterClient so the two can never both fire. The registration arm creates no ClusterClient under Grpc. NOTE — one wiring change beyond the actor: DriverHostActor's ackRouter is now `useClusterClient || useGrpc ? nodeComm : null`. Left at ClusterClient-only, a Grpc node's ack would have gone out on the DPS deployment-acks topic and never reached the sender at all. |
||
|
Joseph Doherty
|
e1db90f9c9 |
fix(mesh): 1B.1 review follow-ups
Two review items from Task 1B.1, both diagnostic: 1. CentralCommandGrpcService's "actor not ready" Warning omitted the peer and the correlation id while the decode-failure Warning six lines above carried both. These are the only two ways an ack fails to reach the coordinator, and an operator has to correlate either against the node's own log. Decoding has already succeeded on this path, so the DECODED correlation id is free. 2. A fused driver,admin,cluster-MAIN node under Mode=Grpc owes BOTH validator surfaces at once (driver keys AND ConfigServe:GrpcListenPort). The two blocks are independent `if`s; chained as `else if` a fused central node would be graded on one surface only and boot with nowhere for any ack to land. Pinned from both sides plus the fully-configured pass. |
||
|
Joseph Doherty
|
7e7df337cc | feat(mesh): CentralCommandService on the ConfigServe listener | ||
|
Joseph Doherty
|
e54883a8e0 | fix(mesh): memoize dialer/provider on first use — a lost PreStart race self-heals at the first refresh (review-prescribed) | ||
|
Joseph Doherty
|
c0475f2df4 |
fix(mesh): resolve dialer + PSK provider lazily — central boots again
Phase 1A live gate defect 4. Neither central node booted on this branch,
in ANY transport mode:
Akka.Hosting.MissingActorRegistryEntryException: No actor registered
for key ...ControlPlane.AuditWriterActorKey
at AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
at ControlPlane.ServiceCollectionExtensions.<WithOtOpcUaControlPlaneSingletons>
WithOtOpcUaControlPlaneSingletons' first WithActors starter resolved
IMeshCommandDialer and IMeshPskProvider eagerly for the Props. Their
graph reaches back into the registry — dialer -> IMeshPskProvider ->
ISecretResolver -> AddZbSecrets -> IAuditWriter -> ActorAuditWriter,
whose ctor calls registry.Get<AuditWriterActorKey>() — and
WithSingleton<AuditWriterActorKey> is registered ~60 lines LATER in the
same builder. Boot died before the entry existed. Mode-independent: the
resolve happened before anything consulted MeshTransport:Mode, so Dps
(the default, and the rollback target) died exactly like Grpc.
Both dependencies now arrive as Func<> and are invoked in PreStart, by
which point the host is fully built — the same shape, for the same
class of reason, as the coordinator proxy Func one argument earlier.
PreStart re-resolves on restart rather than caching across one; both
are container singletons, so a restart gets the same instances. A
resolution throw is caught and logged rather than propagated: a throw
out of PreStart is an actor failure, the supervisor answers it with a
restart, and a broken DI graph is not transient — that is an unbounded
restart loop that would take the transport-independent ack path down
with it.
Regression pin: ControlPlaneStarterOrderTests boots a REAL admin host
through WithOtOpcUaClusterBootstrap + WithOtOpcUaControlPlaneSingletons
in production starter order, with the production graph from
IMeshCommandDialer down to a registry-bound IAuditWriter, and asserts
StartAsync itself — the failure occurs while the builder's starters
execute, so building a ServiceProvider and resolving by hand cannot
reproduce it. Verified RED against the eager resolve: 3/3 modes fail
with MissingActorRegistryEntryException. A second fact pins that the
deferral did not become 'never resolved' — the actor must resolve the
provider from the container and drive it through the contact refresh,
signalled by a TaskCompletionSource (PreStart runs on the actor thread
and the contact load is a piped DB read, so no synchronous assertion
after StartAsync is correct).
Why nothing caught this: the dialer's container-activation test resolves
from a collection with no secrets registration, so its graph stops one
hop in; the actor facts inject fakes and never touch DI; and the
existing whole-host boots register no mesh provider, so the eager
lookups returned null and walked nothing.
Two aggravators recorded by the gate, no code here: the failed process
did not exit (docker Up while /health/ready 503s — Phase 2 readiness
concern, tracked), and this is the third instance of the same
eager-resolve-during-AddAkka class after scadaproj#1 and mesh Phase 6's
IClusterRoleInfo boot crash.
ControlPlane.Tests 236 -> 240, Host.Tests 92, Cluster.Tests 184,
0 warnings in every touched project.
|
||
|
Joseph Doherty
|
86633f140e | test(mesh): pin the Dps no-ClusterClient guard; bound the channel-share test's blackhole worst case | ||
|
Joseph Doherty
|
48dc2120c6 |
fix(mesh): gate follows the endpoint not the mode + review fixes
The known-cluster gate published only under Mode=Grpc, but the interceptor it feeds is registered on every admin node and 1B maps CentralCommandService by LISTENER PORT — neither consults the mode. A dark-switch rollback to ClusterClient or Dps would therefore have left that endpoint serving with the gate permanently inactive, silently restoring both the timing oracle and the pre-auth secret-store/audit flood, as a side effect of a rollback nobody would connect to authentication. The gate is central's cluster inventory, not transport state, so it is now decoupled from the mode entirely: - SetKnownClusters + InvalidateAll run on EVERY successful contact load in EVERY mode, and the contact-refresh timer starts in all three. Under Dps this is a ConfigDb query every ContactRefreshSeconds that did not run before; central always has a ConfigDb, and one indexed read a minute is the cheaper half of the trade. - The GATE set is built from ALL ClusterNode rows. The eligibility filter moved out of the query's WHERE and into the loop, so the FAN set is byte-for-byte what it was (enabled && !maintenance) while the gate widens. This also dissolves the maintenance coupling flagged in 1A.2: a cluster whose nodes are all disabled or in maintenance can still ack, and refusing it would make maintenance mode an authentication switch. Central still refuses clusters it has never heard of. - HandleContactsLoaded keeps a documented order — targets adopted, gate published, cache dropped — now pinned by test. Guards: PreStart mirrors the null-dialer error for a null PSK provider (silent absence is the worst outcome there — nothing malfunctions, so nothing is investigated, while the gate never publishes and keys never rotate); a failed contact load now says out loud that the gate is stale and, before any load has succeeded, inactive; MeshGrpcCommandDialer clamps a non-positive DispatchTimeoutSeconds to 10 with a construction Warning, because the validator for that key is driver-role-scoped and a pure-admin central reads it unvalidated; the dialer's local PSK refusal was shortened so its remediation survives the actor's 200-char truncation of remote-authored status detail; the portless-node warn-once set is rebuilt each refresh, so a node that regains then loses a port warns again — the regression is the occurrence worth seeing. Docs/pins: dialer disposal comment no longer claims one-way; the two interceptor-test comments claiming Matches has a default implementation corrected (it deliberately has none); new pins for per-refresh invalidation, the SetKnownClusters-before-InvalidateAll order, and one channel per endpoint via a new CachedChannelCount. The_known_cluster_set_is_not_published_by_the_other_two_modes is replaced — its premise is obsolete — by three facts: every mode publishes, gate set is a strict superset of the fan set, and the handler ordering. ControlPlane.Tests 230 -> 236, Host.Tests 92, Cluster.Tests 184, 0 warnings. |
||
|
Joseph Doherty
|
6a86201105 |
feat(mesh): gRPC command fan-out behind Mode=Grpc
MeshGrpcCommandDialer places one NodeCommandService.Execute call on one node: a GrpcChannel cached per endpoint (the GrpcDeploymentArtifactFetcher shape), the TARGET node's cluster's bearer plus x-otopcua-cluster naming that cluster, and a per-call deadline linked to the caller's token. A cluster with no resolvable key is refused locally with Unauthenticated — there is no arm on which a command leaves central unauthenticated, even one the far side would have rejected. CentralCommunicationActor gains the third mode branch. The contact load now also projects GrpcPort and, from the SAME rows in the SAME pass, builds the dispatch set — which is what makes 'the gRPC fan reaches exactly the SendToAll recipient set' structural rather than two queries that have to keep agreeing. A row with a null GrpcPort is skipped warn-once (the TelemetryNodeSource precedent). Per-node results are piped back for LOGGING ONLY: named node, gRPC status, dropped. No retry, no queue. Every dial is scheduled on the thread pool rather than called from the receive. Found by review and pinned red-first: an async method runs synchronously to its first await, and the dialer's pre-await work is TryGetPsk, which blocks on the secret store — bounded but seconds, and repeated after every refresh since the refresh drops the cache. Dialing inline parked the actor's mailbox, so deployment acks queued behind a slow node; the regression test tells an ApplyAck while a dial is held and fails without the hop. The contact refresh also wires the Phase 1A register: it publishes the loaded cluster ids as the PSK provider's known-cluster set and drops the key cache, so a rotated key takes effect within one ContactRefreshSeconds. Neither happens under Dps or ClusterClient. Both existing mode branches are behaviourally untouched and every existing fact passes unmodified. The one edit to shared text: the Status.Failure warning drops the word 'ClusterClient', since that handler now serves all three transports. ControlPlane.Tests 205 -> 230, Host.Tests 92, Cluster.Tests 184, 0 warnings. |
||
|
Joseph Doherty
|
14d21496fe |
feat(mesh): known-cluster gate + Matches on the PSK provider
Phase 1A security register items 1 and 2, landed ahead of the command fan-out because they are independently testable and the fan-out is the first caller of TryGetPsk. Item 1 — known-cluster gate. MeshPskProvider gains SetKnownClusters, an atomic whole-set swap consulted by TryGetPsk BEFORE the config map and before the secret store. A cluster id outside the set is refused in O(1) with no store read and no audit write, which closes two things at once: the pre-auth store/audit flood an unauthenticated caller could drive with a forged x-otopcua-cluster header, and the timing oracle that defeated CentralCommandAuthInterceptor's uniform denial text (unknown clusters are never cached, so they paid a store read on every probe while known ones answered from memory). Empty or never-published leaves the gate inactive — central publishes the set from its DB contact refresh, and 'the DB told me nothing' must not become a fleet-wide authentication outage during admin boot. Item 2 — bool Matches(clusterId, presented) on IMeshPskProvider, with FixedTimeEquals inside the provider. CentralCommandAuthInterceptor now calls it and never holds key material; its two provider-side denial logs collapse into one (the provider answers both as false) and the operator hint is restated there. No production caller publishes a cluster set yet, so behaviour is unchanged until the fan-out commit wires the contact refresh. ControlPlane.Tests 190 -> 205, Host.Tests 91 -> 92. |
||
|
Joseph Doherty
|
64774652b7 | feat(mesh): NodeCommandService on the telemetry listener | ||
|
Joseph Doherty
|
a463141028 | test(mesh): stampede callers on dedicated threads — arrival gate must not depend on thread-pool injection | ||
|
Joseph Doherty
|
24e55d3395 | fix(mesh): Batch B review fixes — bounded store read, cache poison guard, test hardening | ||
|
Joseph Doherty
|
68a416e4fc | feat(mesh): per-cluster PSK provider + central interceptor | ||
|
Joseph Doherty
|
fbda015c6e | feat(mesh): node-side PSK interceptor | ||
|
Joseph Doherty
|
dd900f970a |
fix(hosts): a driver that never built a tag table reports NO ANSWER, not zero skips
Found on the docker-dev live gate, not in tests. Two drivers that faulted during initialise (rtu-gate, opcua1) each rendered "0 tags" on /hosts against two authored tags apiece, because a ledger that had never committed returned SkippedTagReport.Empty. A zero report is a positive claim -- "this driver skipped nothing" -- and making it on behalf of a driver that never got as far as looking is exactly the quiet inaccuracy #541 exists to remove. GetSkippedTags() and SkippedTagLedger.Snapshot() now return null until the first committed build, so the column renders the same em-dash it already used for a driver that cannot report at all. Both are 'no answer'; neither is 'nothing skipped'. Knock-on: a FOCAS driver whose only init threw now reports null rather than a zero report, which is strictly more honest than the assertion the test originally made. |
||
|
Joseph Doherty
|
39db75083d |
feat(drivers): every driver reports the authored tags it skipped (Gitea #541)
A driver builds its RawPath -> definition table at init and tolerates a
per-tag miss: an unmappable TagConfig, an unparseable address, a
capability-matrix rejection. That tolerance is right -- one bad tag must not
take down the other few thousand. What was missing is any statement of the
fact afterwards. A skip is not an error and happens before any read, so the
driver stays Healthy, errors/5 min stays 0, Last error stays empty, and Last
read advances live off the tags that did map. On the wonder Makino in #540
that reading held while focas-z-34184 ignored 20 of its 22 tags.
New optional capability ISkippedTagReporter, backed by a shared
SkippedTagLedger so the reset/commit/cap rules are one implementation rather
than one per driver. The report -- authored count, exact skipped count, and a
capped sample of RawPath + reason -- rides DriverHealthChanged to a new Tags
column on /hosts, over the Phase 5 telemetry stream. No table: a driver-only
node has no ConfigDb to write to, the lesson from #521.
The issue named eight drivers. It is twelve -- every driver in
DriverTypeNames.All has an authored-tag skip path. Calculation (no scriptId),
MTConnect (no readable DataItem id), Galaxy (no attributeRef) and OpcUaClient
(no nodeId) were unlisted, and OpcUaClient's was the quietest of the twelve:
NamespaceMap drops such an entry with a bare `continue` and not even a log
line. SkippedTagReporterFleetTests pins the invariant reflectively over every
IDriver in src/Drivers/, so a thirteenth cannot re-introduce the silence.
Sql drops a tag at TWO points and its report is the union of both: a blob
that does not map, AND an identifier the 8.1 catalog gate rejects against the
live catalog. The second is what schema drift produces -- a renamed column, a
dropped view -- months after a deployment sealed green, and reporting only
the first would have that driver claim "0 skipped" while polling half its
tags, which is worse than reporting nothing because the column would then be
actively reassuring.
Deliberately additive to driver state, never a change of it. Latching
non-Healthy for a config problem would satisfy both halves of the #527
comms-loss conjunction -- 90 s without a successful read AND non-Healthy --
and a driver whose tags were ALL skipped has no successful reads by
construction, so it would reconnect-loop forever against a healthy device.
Three details that are load-bearing rather than incidental:
- The skip digest is part of the publish fingerprint. A skipped tag leaves
every other fingerprint component identical to a clean driver's, so without
it PublishHealthSnapshot's dedup swallows exactly the publish carrying the
signal -- the trap that already bit the rediscovery and host-status
channels. The digest covers the sample, not just the counts: a redeploy
that fixes one tag and breaks another leaves both counts unchanged.
- The count is exact; only the sample is capped, and the tooltip says so when
it is showing fewer than it counted. A truncated list that looked complete
would have an operator fix twenty tags and believe the job done.
- The ledger stages and commits, so an init that throws part-way leaves the
previous report standing. A partial snapshot would read as "we now skip
fewer tags", the opposite of the truth.
The proto field needs no companion presence flag, unlike host_statuses: a
singular message field carries real presence in proto3, and a has_ flag would
collide with protoc's generated HasSkippedTags.
Verified red-first three ways: dropping SkippedTagDigest from the fingerprint
fails the two dedup tests, removing the seam from one driver fails the fleet
guard, and dropping the catalog-gate leg fails the Sql union test. Full
solution builds with zero new warnings against
|
||
|
Joseph Doherty
|
b61fe38823 |
fix(focas): browse-committed fixed-tree tags are readable (Gitea #540)
Authoring FOCAS tags through /raw -> Browse device... produced tags the driver
could never read. Every committed tag read BadNodeIdUnknown forever, and the
failure was silent end to end: the AdminUI accepted it, the deploy gate sealed
green, and the driver reported Healthy with 0 errors while ignoring the tags.
Net effect: the whole fixed-tree surface could not be published through any v3
authoring path. Found cutting wonder-app-vd03 over to v3 against a real Makino
Pro 5 / Fanuc 30i, where 0 of 20 committed tags read while two hand-authored
MACRO: tags on the same device read Good.
A captured browse leaf's id IS its DriverAttributeInfo.FullName, and the fixed
tree emitted "{deviceHost}/{path}" there -- an internal cache key, not an
address. Both authoring paths wrote it verbatim into TagConfig.address, and
FocasAddress.TryParse accepts no such form, so the driver skipped the tags at
init.
Fixed at the emitter rather than by stripping the host at commit, which fixes
the browse-commit flow and the manual picker's tree-select together and leaves
one canonical address form:
- FocasAddress gains a fourth kind, FIXED:<path>, and moves to Contracts so the
AdminUI and ControlPlane can parse without referencing the runtime driver
(#526). A FIXED: read is served from the fixed-tree snapshot the poll loop
already maintains -- no area read, no connection. Writes refuse: nothing in
that tree is a writable location on the CNC.
- The address carries no device. A tag's device comes from its Device row, and
a second routing channel for one fact is what created this defect.
- An authored FIXED: tag arms the fixed-tree poll loop by itself. Enabled
defaults false and the browser force-enables it only for a browse, so the
operator would otherwise browse, commit, deploy, and land on a driver whose
loop never starts -- every tag Bad with a correct address.
- Fixed-tree values are now dated from the last answered poll and go
BadCommunicationError once it is stale (8 cadences, floored at 30s). The poll
loop swallows a failed tick by design, so a dead CNC would otherwise serve its
last snapshot as Good indefinitely -- harmless while nothing could bind to the
tree, not harmless now.
Guardrails, so this class cannot recur silently:
- FocasTagConfigModel.Validate() runs the real parser instead of a non-empty
check, and FocasTagDefinitionFactory.Inspect() warns at the deploy gate, which
covers CSV/API config that never touches the modal.
- The manual picker offered axis/spindle/program/status "parameter groups" --
none of which is a FOCAS addressing space -- and its own default value,
axis:5, was unusable. It now composes PMC / PARAM: / MACRO: through
FocasAddress, so what it emits is parseable by construction.
Tests assert addresses ROUND-TRIP through the parser rather than matching an
expected string: the previous FocasAddressBuilderTests pinned "axis:5" exactly,
which is how a green suite coexisted with a picker nobody could use. Verified
red-first against the old emitter. FOCAS 308, AdminUI 1095, ControlPlane 169,
Core 581 -- all pass, 0 warnings.
The remaining half of #540 -- a driver reporting Healthy/0 errors while
skipping tags at init -- is fleet-wide (eight drivers do this) and is filed
separately rather than patched per-driver.
|
||
|
Joseph Doherty
|
08c1728a98 | feat(sql-adminui): Postgres + Oracle in the provider dropdown; docs | ||
|
Joseph Doherty
|
2c19a50bf9 | fix(raw): a superseded manual-entry open can no longer feed the pickers another device's config | ||
|
Joseph Doherty
|
4e3a6e8497 | fix(raw): a stale open can no longer feed the picker another device's config | ||
|
Joseph Doherty
|
c1c21f3ff8 | test(raw): pin the manual-entry accessor as live, not a captured snapshot | ||
|
Joseph Doherty
|
a2093b4400 | fix(raw): manual tag entry feeds each row's picker the merged device config (#468 follow-up) | ||
|
Joseph Doherty
|
abb9f8a1cb | fix(raw): tag edit modal feeds the picker the merged device config, not '{}' (#468 follow-up) | ||
|
Joseph Doherty
|
e698aee401 | test(raw): FakeRawTreeService for raw-modal component tests | ||
|
Joseph Doherty
|
e7fbe97759 |
fix(historian): gate historization capture on the Primary, quarantine a poison tag (#537, #538)
Both defects were found by #491's live gate against a real HistorianGateway and filed rather than fixed at the time, because each one's fix is a design decision rather than a patch. #537 — both nodes of a redundant pair captured every value, so the historian received duplicate live samples for every historized tag (measured: the gateway's write counter advanced twice per value and halved when one central was stopped). The recorder now reads a capture gate per value, fed from the new PrimaryGatePolicy.ShouldRecordHistorizedValues through IHistorizationRoleView — a third verdict on the same RedundancyRoleView object, registered unconditionally on driver nodes rather than alongside the alarm section's IRedundancyRoleView (which only exists when that section is enabled; piggybacking is the #511 registration accident). Two things are deliberate and easy to get backwards: * It gates the CAPTURE, not the drain — the opposite of the alarm sink. That queue is in LocalDb and replicates, so the Secondary already holds the Primary's rows. The historization outbox is a node-local FasterLog replicating to nobody, so a drain-only gate would leave the standby appending forever to a queue no one empties: unbounded disk, then a flood of stale duplicates the moment it is promoted. Entries a node already captured are still drained by that node. * The peer it stands down for must RECORD THE SAME REFS, not merely hold the same rows — a Primary in this node's own per-cluster mesh. A node carrying no cluster-scoped role cannot establish that and never stands down: under a fleet-wide election the elected Primary is usually running a different cluster's configuration, and deferring to it would end this cluster's historization silently, with the drain reporting success on an empty outbox forever. That is the scope mismatch that once suspended the alarm drain on every node of the rig. An unknown role captures, matching the alarm drain — a duplicate sample is recoverable, a hole in the trend is not. #538 — the drain writes every per-tag group and then acks all-or-nothing (deliberate: RemoveAsync truncates THROUGH an id, so partial acking could drop older unwritten entries). One permanently failing tag therefore pinned the FIFO head while every healthy tag in the batch was re-submitted to the historian on every pass, indefinitely — duplicate samples and an outbox that never truncated, observed live at 8.3 MB from a single unprovisionable tag. The drain now tracks consecutive failures per historian tag and abandons that tag's entries once it exceeds MaxWriteAttempts (new, default 5), so the head advances. The counter only advances on a pass where some OTHER tag wrote successfully: that is what separates "this tag is poison" from "the backend is down". A count-only quarantine looks identical on the poison-tag case and silently discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug into a lost-history one. Abandonment is also deferred until every failing tag is over budget, so nothing is discarded on a pass that could not have advanced the head anyway. Each abandonment logs a Warning naming the tag; the total rides RecorderStatus.AbandonedEntries. Capacity also gains a non-zero default (500,000 entries) as the disk backstop for the one case the quarantine deliberately cannot resolve — a historian refusing every write, where retaining is correct. Set 0 to restore unbounded retention. Every new test was falsified by reverting the production change it guards, and each one reddens alone: * drop the backend-reachable evidence -> only A_backend_that_fails_every_tag_abandons_nothing * never abandon -> only A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten * latch the capture gate at construction -> only The_capture_gate_is_re_read_per_value... * treat any Primary as a scope peer -> only the two scope-mismatch cases Full suite green. (Four OpcUaClient.Browser failures during the run were the opc-plc fixture being down on the shared Docker host; brought it back up and they pass.) Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
ad74473452 |
fix(historian): the recorder died on its first value — live gate for continuous historization (#491)
Ran #491's live gate against a real ZB.MOM.WW.HistorianGateway talking to a real AVEVA Historian, and it found a production-fatal defect on the first value it ever saw. ContinuousHistorizationRecorder.OnValueChangedAsync touched `Self` after awaiting IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false). ConfigureAwait(false) discards Akka's ActorTaskScheduler synchronization context, so `Self` — which reads ActorBase.Context — threw NotSupportedException. The throw escaped into supervision, which RESTARTED the actor; because production seeds an EMPTY ref set (the deployed set arrives later via UpdateHistorizedRefs), the restart silently discarded every historized ref. Net effect: one value captured, then nothing, with no further error to show for it. Fixed by capturing `IActorRef self = Self` before the await — the idiom the GetStatus handler already applied to Sender. Thirteen unit tests missed it because InMemoryOutbox.AppendAsync returned ValueTask.CompletedTask: a completed ValueTask never yields, so the continuation ran inline with the context still installed. The production FasterLog outbox fsyncs per entry and does yield. The first regression test was itself vacuous — `await Task.Yield()` posts back to the CAPTURED context, so the append still completed on the actor thread and reverting the fix left the test green. The fake must complete on a POOL thread (Task.Delay(1).ConfigureAwait(false)), as a real fsync does; with that, reverting the fix reddens exactly one test. The rig could not run this gate at all: docker-dev carried ServerHistorian__* but no ContinuousHistorization__* keys, so the recorder could never spawn. Both centrals now carry them (default OFF — the recorder needs ContinuousHistorization:Enabled AND ServerHistorian:Enabled), with the outbox on the per-node named volume so it survives a container recreate. ServerHistorian__UseTls was hard-coded true and is now parameterized; a plaintext h2c gateway was unreachable without it. Gate results: outbox append, drain, WriteLiveValues accepted by a real historian, and restart convergence (capture resumes after a restart with no new deployment) all pass. The read-back hop is NOT proven — the LAN historian accepts writes but serves no reads under a "License violation: Database tag count exceeds license tag count", and the licensed corporate one needs the VPN. EnsureTags provisioning is likewise unexercised there (RemoteGrpc-only, and that CAP is not listening). #491 stays open for the read-back hop alone. Also recorded, not fixed: the recorder has NO Primary gate, unlike the alarm store-and-forward sink, so both nodes of a redundant pair send duplicate live values. Gating it would introduce a failover window with no capture — that is a design decision, not a cleanup. Evidence: docs/plans/2026-08-01-continuous-historization-live-gate.md Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
6b3da93151 |
fix(browse): commit the browsed data type, and fix the probe classification the fixture was hiding (#536)
A browse-committed AB CIP / TwinCAT / FOCAS tag bound with its model's DEFAULT
data type, not the one the device reported. A browsed REAL committed as
`{"tagPath":"Pump_1_Speed","dataType":"DInt"}` — and those three drivers decode
the wire bytes by that field, not by the raw row's OPC UA DataType — so the tag
read four bytes of a float through an integer decoder and published a plausible
wrong number with no error raised anywhere. FOCAS was missed when #536 was filed.
The type is CARRIED, not derived. Reverse-mapping the leaf's driver-agnostic
DriverDataType looks like the smaller fix and is wrong: SInt/Int/DInt/USInt/UInt
all collapse into Int32 on the way out of the driver, so that version would bind
a 16-bit INT as a 32-bit DINT while appearing to fix the REAL. Drivers now state
their own type name on DriverAttributeInfo.NativeDataType, which reaches the
picker as AttributeInfo.AddressFields["dataType"] and the commit mapper from
there. A leaf that states nothing keeps the model default — the pre-#536
behaviour, kept deliberately as the fallback rather than a guess.
Numeric strings are refused before parsing: Enum.TryParse accepts "9" for any
enum and Enum.IsDefined then confirms it, so an unguarded parse would bind a tag
to a type nobody named. A test caught this in the first version of the guard.
--- and the other half, which the fixture was hiding ---
Test Connect on a device with NO authored tags — every AB CIP device at the
moment an operator first presses the button — reported "CIP handshake failed"
against a healthy controller. That was written down as a fixture limitation
("ab_server doesn't implement @raw_cpu_type"). Both halves of that were wrong:
* @raw_cpu_type is not a libplctag special tag at all (only @raw, @tags and
@udt/ are), so it goes out as an ordinary symbolic name that no controller
defines, and the controller's "tag does not exist" IS the successful answer.
* ab_server answered an unresolvable tag path with CIP 0x03 where a Logix
controller answers 0x04 — ErrorBadParam vs ErrorNotFound, which the probe
classifies oppositely. Fixed in vendored patch 0002.
* With the fixture then telling the truth, the real defect surfaced:
AbCipDriverProbe.IsReachableException matched the native C status spellings
("STATUS_NOT_FOUND"), but libplctag.NET throws with the MANAGED enum name
("ErrorNotFound"). Those substrings can never match, so the reachable
classification could not fire on the exception path — which is the path every
no-authored-tags probe takes. This was failing on real hardware too, not just
against the sim. It now recovers the Status and defers to IsReachableStatus,
so there is one classification rather than two that can drift.
Rejected: switching the fallback to @tags. libplctag builds a tag-listing tag
with NO network I/O, so it reports OK against any TCP listener — measured green
against a Mosquitto broker.
Verification: full solution suite green. Falsification runs confirm exactly the
intended tests go red — 9 for the type carry, 3 (the reachable statuses only) for
the probe classification. Live on docker-dev: browse-committed TestString/
Motor_Run/TestSINT now store String/Bool/SInt, alongside last session's
Pump_1_Speed still holding the wrong DInt. TestSINT is the falsifier — its row
DataType is Int32, identical to TestDINT's, so no reverse map could have produced
"SInt".
Known consequence: pre-existing browse-committed tags keep their wrong dataType;
the fix applies to new commits only.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
1a1dbc3f00 |
fix(uns): enforce the UNS segment rule on every authoring path (#535)
An area or line could be named "Packaging" or "Line 1", save cleanly, and then fail the NEXT deployment with UnsSegmentInvalid. Because a deployment snapshots the whole database, one badly-named area blocked EVERY cluster's deploy - and whoever hit that was usually not whoever typed the name. Only the equipment form carried the rule; the deploy gate held its own private copy. The rule now lives once, on Configuration.Validation.UnsSegment, and both ends call it. DraftValidator no longer restates it. Enforced in UnsTreeService, not just on the forms: - Create/Update Area, Line and Equipment reject an invalid name before touching the row. - The CSV import reports a bad name as an ERROR, not a skip - a skip means "you already have this row", which would tell the operator the equipment exists when it does not. Good rows in the same batch still insert. A form attribute could never have covered the import, and with no bUnit in this repo a rule only a form enforces is also a rule no test can reach. The forms keep an attribute for the immediate message, but it now references the shared pattern rather than restating it - a second copy is how these drifted apart. One behaviour change beyond the fix: UnsSegment.AttributePattern admits "_default", which the equipment form's hand-written literal rejected. The form was stricter than the gate it was mirroring, so a _default row was uneditable through the AdminUI. Tests: 41, falsified with three production mutations - dropping the CreateArea guard reds exactly the CreateArea rejections, turning the import error back into a skip reds only the import test, and narrowing AttributePattern reds only the attribute/IsValid agreement. Every rejection is paired with a name that must still be accepted, or a guard that refused everything would pass. Live-verified on docker-dev: the area modal now refuses "Packaging" in place with the rule spelled out, "packaging-b" saves, and a CSV import of two rows reports Inserted 1 / Skipped 0 / Errors 1 naming the offending row - the path no form attribute reaches. Rig cleaned up and a further deployment sealed. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
d542ab5b7c |
feat(security): author ACL grants from the real tree, not free text (#520 stage 7)
The last stage of #520. Enforcement landed in stages 5-6; this closes the authoring end, where a grant could be saved, deployed and sealed green while resolving to nothing. Scope kind and scope target are now chosen TOGETHER from a picker over the cluster's raw and UNS trees (AclScopePickerModal -> AclScopeTree, backed by IAclScopeService). They were previously an unrelated dropdown and a free-text box, so "RawDevice + a TagId" was a well-formed row. Because the trees are rooted at the grant's own cluster and each node supplies its own kind, NodeAclScopeCrossCluster, NodeAclScopeIdUnexpected, NodeAclScopeIdMissing and NodeAclScopeKindRetired all become unauthorable by construction. CheckScopeAsync mirrors DraftValidator.ValidateNodeAclScopes rule for rule and blocks the save when a scope resolves to nothing - re-checked on submit, not trusted from form load, because a row whose entity was deleted since is exactly where the operator never touches the picker. The list page marks such rows unresolved with the reason. The principal field suggests known principals and reports how many live keys reach the one typed; reaching nobody warns rather than blocks, since authoring ahead of minting is legitimate. Deliberate departures from the plan line, all recorded in the design doc: - The banners are reshaped, not removed. "Not enforced" stopped being true at stage 6, but enforcement is still a per-node flag the AdminUI cannot see, so a narrow grant really does confer nothing on a default-configured node. - A Cluster row produces a NULL ScopeId. The deploy gate rejects one that carries an id, so filling in the cluster id would author undeployable rows. - AclScopeTree is a new component rather than a third mode on RawTree, which already carries the /raw editing tree and the equipment reference picker. - UNS tags are not offered: a UNS signal is a UnsTagReference, and a Tag grant already covers the raw node and every UNS projection of it. Also fixed: the ACL editor had a DataAnnotationsValidator and no ValidationSummary, so an invalid NodeAclId made Create a silent no-op. Tests: 30 in AdminUI.Tests/Acls, falsified with six production mutations - each turned exactly the intended tests red and no others. Live gate PASSED on docker-dev, 11 legs; both central containers asserted on the same image by container id. The pair that matters is legs 8 and 11: the page and the deploy gate reject the same row with the same reason. Legs 5 and 9 are the positive controls - without them the gate passes against a page that refuses everything. Not covered: UNS-realm scope selection (SITE-A has no UNS areas). Rig restored to zero ACL rows. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
8cdbcb9823 |
fix(security): resolve the event-subscribe gate's source through IsHandleInNamespace
SubscribeToEvents' sourceId is the manager HANDLE, not a NodeState — the same object-typed trap as ContinuationPoint.NodeToBrowse, found the same way. The `sourceId as NodeState` cast compiled, always yielded null, and silently disabled the gate: an ungranted session could subscribe to an equipment's event notifier and receive every alarm it raises, including the condition fields, while every value on it was refused. Now uses the SDK's own IsHandleInNamespace, which cannot be wrong about the type. The new test passes a null monitored item deliberately: the gate must deny without touching the item, which also makes the test a discriminator rather than a restatement — against the broken cast the call falls through to the base implementation, which does dereference it, so the test fails loudly. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
00aa486677 |
feat(security): enforce node ACLs on read, browse and subscribe (#520 stage 6)
Completes data-plane enforcement behind Security:Acl:Enabled. Read (every attribute, not just Value), Browse, TranslateBrowsePath, CreateMonitoredItems, SubscribeToEvents and SetMonitoringMode now consult scoped NodeAcl grants. With the flag off nothing changes. Four things went beyond the stage as written, each because leaving it out would have made the feature look like it worked: - Browse needed ancestor visibility. The trie inherits downward, and a client reaches a deep node by browsing the nodes above it, so without this every grant narrower than the cluster was authored, shipped and unreachable. It is a separate non-inheriting marker, NOT a synthesized Browse grant on each ancestor: the additive downward walk would carry such a grant from Plant onto Plant/Siemens/dev9 and every other sibling subtree. Pinned by a test that goes through the evaluator — probing the trie helper directly passes against the broken design, because a sibling is absent from the trie either way. - Revocation had to reach LIVE subscriptions. Every other operation re-asks per request; a subscription is authorized once and then streams, and streaming is how machine consumers actually read this server — so stopping only the re-asking operations would have made revocation close to cosmetic. IDataPlaneAuthoritySource gained a Changed event. A denied item is queued BadUserAccessDenied and then DETACHED from its MonitoredNode2 fan-out, not set MonitoringMode.Disabled. Disabling discards the item's queued notifications, so the client is silenced without ever being told why. The over-the-wire test caught this; the unit-level path reports the item as stood down either way. Detaching is also reversible, so a restored grant resumes the feed with no client action. - The decision cache holds the grant WALK, not the decision. Grants and the trie are fixed for a generation; a key's ExpiresUtc passes with the clock, so caching the whole decision would let an expired key keep working until the next deploy — the stage-5 open-session hole in a subtler form. - TranslateBrowsePath is gated, without which browse filtering is decorative: a client refused a browse asks TranslateBrowsePathsToNodeIds for the same path. Found en route: UserAuthorizationState's staleness ceiling couples two clocks. MembershipResolvedUtc is stamped from DataPlaneAuthorizer's TimeProvider while IsStale is evaluated against TriePermissionEvaluator's; they are the same instance in production and diverge the moment anything injects a test clock, at which point the evaluator denies EVERYTHING with a verdict indistinguishable from "no grant matched". Stage 5's claim that these predicates cannot fire was accidentally true; they are now disabled explicitly. Also fixed: ContinuationPoint.NodeToBrowse is the manager HANDLE, not a NodeState. The declared type is object, so the wrong cast compiled cleanly and silently yielded null — skipping the browse source gate entirely. Falsified: disabling the transit marking reddens exactly the three tests that assert transit works; implementing transit as an ancestor grant reddens the sibling-leak test; skipping the fan-out detach reddens the revocation test. Full solution suite green. The stage 8 live gate has NOT run yet. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
f692f6afe4 |
feat(security): enforce node ACLs on write, alarms and HistoryRead (#520 stage 5)
`Security:Acl:Enabled` (default false) makes scoped `NodeAcl` grants decide data-plane authorization at the three seams that have one, and gives HistoryRead a check it never had. Read, Browse and Subscribe stay ungated — stage 6. The ACL decision REPLACES the server-wide role check rather than chaining with it. Chained, a grant would have to confer the role as well as pass the scope check, and only Cluster-scoped grants confer a role — so every narrow grant would have stayed inert and the flag would have looked inert too. The load-bearing test is an ALLOW: a session carrying no roles at all writing successfully on the strength of its grant. A deny-only test passes against a build where the ACL is never consulted, because the role gate denies too. Found en route: the principal never reached the session. The authenticator passed it as OpcUaUserAuthResult.DisplayName, but RoleCarryingUserIdentity derives DisplayName from the presented token — for a key-bearing client that is the static username `apikey`, identical fleet-wide. So the trie's matching key was dropped at the identity boundary, and the audit ClientUserId on a rejected write recorded `apikey` for every consuming system. Principal and ApiKeyId are now carried explicitly. Beyond the stage as sketched: - Revocation reaches an OPEN session. Every decision re-checks the authenticating key against the current snapshot; without it a key revoked by a deployment kept its authority until the client chose to reconnect, which for a machine-to-machine consumer can be indefinitely. - UserAuthorizationState is built per CALL, not per session. The per-session shape exists so an LDAP session can re-resolve membership that changed out of band; nothing can change out of band under API keys, so the generation stamp is the whole invalidation signal and a per-call record picks it up for free. IsStale/NeedsRefresh therefore cannot fire here and are documented as vestigial. - The trie is built inside DataPlaneAuthorityState.Apply, under the same generation as the snapshot, rather than by a second wiring at the publish actor. An uninstalled trie denies everything — indistinguishable from working enforcement. Deleting the install turns 13 of the 22 new tests red. - The deploy log reports what the BUILD dropped instead of re-deriving it. The old count agreed with the builder on one of its four drop reasons; a rule with a blank ScopeId, a retired scope kind, or a path not starting at a realm vanished with the log reading zero. Both IApiKeyStore implementations widened to the full grant set. The narrowing was redundant — ApiKeyInterimRoleMapper applies the same filter itself — and became a trap once scope was enforced. Pinned by asserting the roles are unchanged, with the two grants carrying different flags so it cannot pass by accident. Falsified by mutation: enforcement never engaging, the write seam ignoring the ACL, the realm dropped from the scope, the revocation check removed, and the trie never installed each turn the intended tests red and no others. Full solution suite green. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
df340b8bba |
fix(tests): three real failures the integration suite had stopped catching
None of these were fixture absence. Each is a guard that went quietly wrong and then reported the wrong thing. PrimaryGateFailoverTests — per-cluster mesh Phase 6 ( |
||
|
Joseph Doherty
|
3a6b2a1daf |
feat(acl): variable-depth scope model + the scopePaths producer (#520 stage 3)
Makes the /raw device tree addressable by an ACL rule, and builds the
scope-path producer PermissionTrieBuilder has accepted since it was written
but which had no caller anywhere.
The plan for this stage said "add NodeHierarchyKind.Raw, whose path is the
RawPath segments Folder/Driver/Device/TagGroup/Tag". That shape is wrong.
RawFolder.ParentRawFolderId and TagGroup.ParentTagGroupId are both
self-referencing, so folders and tag groups each nest without bound and a real
RawPath is folder…/Driver/Device/group…/Tag. NodeScope was fixed-arity and
PermissionTrie.CollectMatches was five hardcoded steps, so the scope model
could not address a real raw tree at all. NodeScope is now {ClusterId, Kind,
Segments} and the walk is a segment loop. The trie STRUCTURE was already
variable-depth (NodeAclPath.Segments is a list, Children is a recursive
dictionary), so only the walker and the scope record had to change.
Four decisions worth keeping:
- The namespace level became the realm. Level 1 gated on NodeScope.NamespaceId,
but v3 has no Namespace entity — the snapshot was retired when the address
space went dual-namespace, and the identifier appears nowhere in production
outside NodeScope. So the level could only ever fail to match, which silently
made every sub-cluster grant unreachable while cluster grants kept working.
It is now a realm token derived from the scope's Kind, not carried on it, so
the walk and the producer cannot disagree about its spelling.
- Provenance comes from the granting row's own ScopeKind, recorded on TrieGrant
at build time. Depth-derived provenance could not name a raw level at all,
since the depth of "Device" varies with folder nesting.
- The trie comparer moved from OrdinalIgnoreCase to Ordinal. Raw sibling-name
uniqueness is ordinal, so Speed and speed can both exist under one device and
a grant on either would have matched both.
- The null-scopePaths "deterministic test mode" is deleted. It planted
unresolvable rows at a direct child of the trie root, a position the walker
cannot reach, so the grant looked placed and could never match — and every
test using that mode proved nothing about production placement. scopePaths is
now required, and a row that does not resolve is dropped with a diagnostic.
A scope resolves to a LIST of paths: a raw tag projected into N equipments is
reachable at N+1 NodeIds and a write through any of them lands on the same
driver reference, so a Tag-scoped grant covering only the raw node would be
bypassable through a UNS one. The lookup is keyed on (ScopeKind, ScopeId), not
ScopeId alone — ScopeId has no FK and no type discriminator, so an id collision
across entity types would hand a grant the wrong node's paths.
Also closed here, all found by review rather than asked for:
- NodeAcl rows had no deploy-time validation of any kind and were not reachable
from DraftValidator at all (DraftSnapshot carried no NodeAcls). They are now
gated for dangling ids, cross-cluster scopes, blank/contradictory ScopeId, the
retired FolderSegment kind, and undefined ordinals.
- ConfigDbApiKeyStore conferred roles from Cluster-scoped grants authored in ANY
cluster, because it filtered on scope kind with no ClusterId predicate while
ArtifactApiKeyStore read an already-cluster-filtered snapshot. On a fused
admin+driver node that was a cross-cluster over-grant. Its doc comment claimed
the two stores matched.
- The two AdminUI ACL banners still said enforcement was LDAP-group -> role
mapping, which stages 1/2 made false in both directions.
- The /apikeys page was never classified in PageAuthorizationGuardTests, so that
test has been red since the key-minting commit merged — I did not run
AdminUI.Tests before merging stage 4.
Falsified rather than assumed: 8 mutations against the trie tests, 5 against the
deploy gate (including one proving a cross-cluster row is not silently
mis-diagnosed as unresolved), a staged red->green for the over-grant regression,
and the realm gate, the comparer and the malformed-path guard each broken
individually to confirm only their own tests go red.
Node ACLs are still not ENFORCED — nothing constructs a NodeScope or calls the
evaluator yet. That is stages 5 and 6; this stage makes the model able to
express what those will enforce.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
a438ac1640 |
feat(acl): keys + ACLs reach driver-only nodes via the artifact (#520 stage 4)
A driver-only node has had no ConfigDb since per-cluster mesh Phase 4, so it registered NullOpcUaUserAuthenticator and denied every UserName token. Its keys now ride the deployment artifact instead. - ConfigComposer snapshots ApiKeys alongside the NodeAcls it already carried. Only the peppered HMAC ships, never a secret, and the pepper is node config that is deliberately not in the artifact. - DeploymentArtifact.ParseDataPlaneAuthority decodes both node-side. Grants are scoped to the node's own cluster; keys are not, because ApiKey carries no ClusterId — a key is a fleet-wide credential and the ACL rows are what bind it. - OpcUaPublishActor installs them on DataPlaneAuthorityState on every apply, BEFORE its plan.IsEmpty early-return: an artifact that only revokes a key produces an empty address-space plan, so feeding after that return would leave the revoked key working until an unrelated structural change redeployed. - ArtifactApiKeyStore serves them to session activation; a node with a ConfigDb keeps ConfigDbApiKeyStore, whose per-request read is what makes revocation immediate there. - OtOpcUaNodeManager gains DataPlaneAuthority — the cluster identity blocker #4 named. NodeScope.ClusterId is required and the evaluator fails closed on a mismatch, so until now no scope it built could have evaluated. Nothing reads it until stages 5/6. The generation is a node-local COUNTER driven by the RevisionHash, not derived from its bits. PermissionTrieCache treats the highest GenerationId as current, so a generation taken from hash bytes would install a lower number about half the time and the cache would go on serving the previous deployment's ACLs, silently. Sessions are not stamped with it yet — that belongs with the first seam that reads a UserAuthorizationState, in stage 5, rather than being built now and consulted by nothing. Live-gated on docker-dev against site-a-1 (driver-only, no ConfigDb): key authenticates, wrong secret denied, both halves of the revocation asymmetry observed, and a SITE-A grant confirmed not to reach site-b-1. Key minted with an independent Python HMAC so the verifier was checked against something other than itself. Rig left clean. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
dd939d3c97 |
feat(acl): anonymous becomes a normal grantable principal (#520 D3)
Anonymous sessions used to short-circuit in HandleImpersonation — "let the SDK's
default validation stand" — so they always connected and always carried no roles.
Now admission and authority are separated:
admission -> OpcUa:AllowAnonymous, the HOST's decision (it owns token policies)
authority -> the authenticator, resolving the reserved principal "anonymous"
through the SAME grant path a key's principal goes through
So anonymous can be granted WRITE, not merely tolerated as read-only, and it
arrives at the downstream gates as the same RoleCarryingUserIdentity a key
produces — which is the real payoff: no gate carries an anonymous-shaped branch,
so there is no bypass path to get wrong.
AllowAnonymous defaults to TRUE on purpose. Defaulting it false would silently
deny every client without a key the moment this deploys, including ScadaBridge's
Data Connection Layer in another repo. Making keys mandatory is a rollout
decision, not an upgrade side effect. When false, the Anonymous token policy is
withheld so a client is refused at DISCOVERY rather than only at activation, and
activation is rejected independently in case a client attempts a token type the
endpoint never advertised.
Two fail-closed choices, both falsified:
- A failure to read anonymous's grants DENIES rather than admitting with no
roles. Admitting is the pre-#520 behaviour, and defaulting to it exactly when
the server cannot determine what anonymous may do stops being harmless once
reads are gated. Making it admit reds Anonymous_denies_when_grants_cannot_be_read.
- A key may not authenticate as the reserved "anonymous" principal, or a
credentialled client silently inherits whatever anonymous was granted.
AuthenticateAnonymousAsync is a DEFAULT interface implementation returning
admit-with-no-roles, i.e. exactly the pre-#520 behaviour, so
NullOpcUaUserAuthenticator and every test double keep working rather than being
forced to reimplement it — important because driver-only nodes run the Null one
and would otherwise black out. Note a default interface method is only reachable
through the interface-typed reference, which is how OpcUaApplicationHost holds it.
An existing test asserted the old fall-through and correctly went red; it is
rewritten to pin the new contract rather than deleted, with the reason recorded.
OpcUaServer 366, Host 42 green.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
65fda7237f |
refactor(acl): delete the LDAP data-plane authenticator and its tests (#520)
Stage 2b. LdapOpcUaUserAuthenticator was left registered nowhere by stage 2a; this removes it, its three test files, and the config surface that existed only to serve it. Deleted: LdapOpcUaUserAuthenticator, LdapOpcUaUserAuthenticatorTests, LdapAuthResilienceTests, LdapGlauthOutageLiveTests (19 tests). Also removed four Security:Ldap keys — AuthTimeoutMs, MaxConcurrentAuthentications, OutageFailureThreshold, OutageCooldownSeconds — plus their validator rules and binding/validator test coverage. Verified first that the ONLY consumers were the deleted class and the validator rules asserting them; none appear in any checked-in appsettings. Leaving them would have created exactly the dead-config surface #521/#524/#525 were filed to clear up. Worth stating why that resilience machinery does not need relocating rather than just deleting. It existed because the OPC UA SDK raises impersonation SYNCHRONOUSLY under a SessionManager-wide event lock, so a hung LDAP bind there serialized every session activation server-wide — Anonymous and X509 included. Key verification is one indexed lookup plus one HMAC with no network I/O, so the hazard the wall-clock bound, concurrency cap and outage circuit guarded against no longer exists on that path. LDAP now runs only on AdminUI login, an ordinary async request path where a slow directory affects the one request making it. ConnectionTimeoutMs is RETAINED: it is projected into the shared library and is not authenticator-specific. Docs corrected rather than left to rot: - CLAUDE.md's "LDAP Authentication" section described the deleted file as the data-plane authenticator. Rewritten, including a warning not to "restore" it on seeing the LDAP wiring that AdminUI legitimately still needs. - infra/README §5b (the GLAuth outage live gate) marked RETIRED with the reason, rather than silently dropped — the harness GLAuth is still used by OTOPCUA_HARNESS_USE_LDAP=1 and is unaffected. - docs/security.md's five-key resilience table cut to the one surviving key, and a new API-key identity section added. Historical archreview/ records are left as-is; they are dated accounts. Host.IntegrationTests 211 tests (was 230, i.e. exactly the 19 removed) with the same 3 pre-existing failures verified against clean master. Host 37, Security 89. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
0329fa4d4b |
feat(acl): API-key identity replaces LDAP on the OPC UA data plane (#520)
Stage 2a. Clients present the static username "apikey" with an
otua_<keyId>_<secret> credential as the password; the key resolves to a
principal. LDAP registrations are deliberately RETAINED for AdminUI login —
only IOpcUaUserAuthenticator moves, which is what D2 asked for.
The load-bearing decision here is what roles a key gets. The two live gates
(WriteOperate on Value writes, AlarmAck on Part 9 methods) check a SERVER-WIDE
role set, so the cutover has to keep producing one or it would lock every client
out of writes long before per-node evaluation lands in stage 5.
ApiKeyInterimRoleMapper therefore maps ACL grants to roles, but ONLY
cluster-scoped ones. The obvious version — union the permission flags across all
of a principal's grants — would let a grant authored narrowly ("WriteOperate on
Equipment X") confer server-wide write authority, because nothing in the pipeline
can enforce scope yet. That fails in the dangerous direction: the operator's
narrow intent silently becomes broad. Restricting to cluster scope under-grants
instead, which is no worse than today (where every ACL row does nothing) and can
never exceed what was authored. Falsified: deleting the scope check reds exactly
A_narrower_scoped_write_grant_confers_nothing_yet and nothing else.
Verification is one indexed lookup plus one HMAC, with no network I/O at all.
That matters because the SDK raises ImpersonateUser synchronously while holding a
SessionManager-wide event lock, so this path does not stall one request — it
serializes every session activation server-wide. It is the main practical gain
over the LDAP bind it replaces.
Every rejection returns one opaque message so the server cannot be used as an
oracle for whether a key id exists or which half of a credential was wrong; a
test asserts the failure modes really are indistinguishable AND that a valid key
still succeeds, so it cannot pass by denying everything.
Caught while wiring: OtOpcUaServerHostedService takes IOpcUaUserAuthenticator as
a REQUIRED ctor parameter, so leaving a driver-only node unregistered is not a
quiet fallback to deny-all — it fails DI and the node does not boot.
NullOpcUaUserAuthenticator is now registered explicitly there. A driver-only node
has no ConfigDb since Phase 4 and so has no key source until the artifact-backed
store lands in stage 4; it denies UserName tokens until then, which is the safe
direction, and anonymous sessions are unaffected.
LdapOpcUaUserAuthenticator is now registered nowhere and is dead code. Deleting
it plus LdapOpcUaUserAuthenticatorTests / LdapAuthResilienceTests /
LdapGlauthOutageLiveTests is stage 2b, kept separate because those cover a
live-gated resilience feature (hard wall-clock bound, bounded concurrency,
directory-outage circuit) that exists precisely BECAUSE the LDAP bind ran inline
on session activation — a constraint the key path does not have. That deserves
its own commit, not a footnote in this one.
Anonymous-as-a-grantable-principal (D3) is also stage 2b.
Host 37, Security 89, OpcUaServer 364 green.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
5a7a08deca |
fix(drivers): detect mid-run comms loss and recover from it (#527)
DriverInstanceActor.DisconnectObserved had no production sender. The only paths into Reconnecting were a failed *initial* connect and an operator's Force Reconnect, so a driver that connected and then lost its device stayed Connected indefinitely: publishing Bad for every tag, never re-initialising, and NOT recovering when the device returned. PublishHealthSnapshot read GetHealth() for reporting only and never acted on a driver reporting itself unhealthy. Both comms-loss quality paths -- #477's native-condition annotation and #481's scripted-alarm input fan-out -- hang off ConnectivityChanged, so both were correctly wired to a signal that never fired. The Connected health tick now evaluates a watchdog and self-sends DisconnectObserved when the driver reports a non-Healthy state AND its LastSuccessfulRead has been frozen for DefaultCommsLossTimeout (90s). The conjunction is the design, and the issue's own proposal -- "non-Healthy for N consecutive polls" -- is wrong. Degraded means *connected but some data is bad*; several drivers write health per tag inside the read loop (ModbusDriver.ReadAsync), so one unreadable address latches Degraded forever against a perfectly reachable device, and a state-only watchdog would reconnect that driver every 90s indefinitely -- worse than the bug. LastSuccessfulRead is the honest evidence: preserved across Degraded/Faulted writes, advanced by any success. The state check remains the second conjunct so an idle Healthy driver with nothing subscribed is not declared dead for not reading. Verified across all 12 drivers that a sustained failure does reach a non-Healthy state -- nine poll drivers via Degraded, MQTT via live connection state, Galaxy via its reconnect supervisor. CalculationDriver reports Healthy while running and so can never trip, which is correct: it has no transport. Also: the Reconnecting retry now calls ShutdownAsync before InitializeAsync. ModbusDriver (_transport) and MqttDriver (_connection) overwrite an IAsyncDisposable field inside InitializeAsync without disposing the previous one, so re-initialising without a teardown leaks a socket per attempt -- latent while only a manual Force Reconnect reached this path, but the watchdog makes it automatic and unbounded (~360 leaks for an hour-long outage at the 10s retry). Tests: 8 new in DriverInstanceActorCommsLossWatchdogTests, each absence assertion carrying a positive control (a proven count of processed health-poll ticks). Falsified five ways -- dropping either conjunct reds exactly the test written for it, and removing the teardown, the null baseline, or the watchdog call each reds its own guard. Live gate PASSED on docker-dev (Modbus + otopcua-pymodbus-standard), run twice: - fixture stopped -> watchdog fires at ~108s, "declaring comms lost" - fan-out observed at Debug: "pushed BadNoCommunication for 2 mux ref(s)" (#481 Layer 4) and the #477 native-condition annotation on a driver owning conditions - fixture restarted -> reconnected in ~8s with NO node restart; both the raw (ns=2) and UNS (ns=3) NodeIds read Good with fresh timestamps Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |