c9f182b2d728b42c5c60ff09d7c0e8ac2fb30dce
158 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
0125634400 |
docs(secrets): correct stale non-functional warning on the replication block — pair-local replication is live-gated
The Secrets:Replication _comment claimed the feature was KNOWN NON-FUNCTIONAL against ZB.MOM.WW.Secrets.Replicator.AkkaDotNet 0.2.0 and said "do not enable until the library is fixed". That has been false since 0.2.2 — 0.2.1 fixed the shadowed ISecretReplicator registration and 0.2.2 the hosted-process DI deadlock — and the 2026-08-07 three-mesh live gate proved replication works (PASS 4/4, docs/plans/2026-08-07-secrets-pair-local-live-gate.md). Replaced with the truthful operator-facing note: functional since 0.2.2, still default-off, PAIR-LOCAL in scope (mesh Phase 6 gives each application cluster its own 2-node mesh, so a secret converges within its pair and nowhere else), same KEK required per pair, and rewrap-all runs per node because re-wraps do not replicate. Comment-only; no behaviour change. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1 |
||
|
Joseph Doherty
|
4870785209 |
docs(auth): #543 add the one-public-ctor remarks paragraph to the two newly-pinned interceptors
ConfigServeAuthInterceptor and LocalDbSyncAuthInterceptor gained HasExactlyOnePublicConstructor pin tests in this branch but lacked the sibling-standard <remarks> paragraph naming them (TelemetryStream/MeshCommand/CentralCommand all carry it). Cosmetic consistency item from the #543 code review. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1 |
||
|
Joseph Doherty
|
f10f4c3754 |
fix(auth): #543 whitespace-only configured key fails closed in the three pre-existing gRPC auth interceptors
TelemetryStreamAuthInterceptor, ConfigServeAuthInterceptor and LocalDbSyncAuthInterceptor all decided "is a key configured?" with string.IsNullOrEmpty. A whitespace-only configured value therefore passed that guard and fell through to the constant-time comparison — where it is satisfied by a whitespace-only bearer token. The gate would be open to anyone who guessed a blank credential. This backports the check the two NEW mesh interceptors already use: MeshCommandAuthInterceptor (and its central-side clone) validate with string.IsNullOrWhiteSpace, so the four/five planes now answer the same way to the same misconfiguration. Nothing else changes — the denial status codes (PermissionDenied / Unauthenticated / PermissionDenied respectively) and every denial message are untouched, and no key material is logged. No deployment configures a whitespace key: the docker-dev rig and production both set real keys, so this is a fail-closed hardening of an unreachable-today misconfiguration rather than a live exposure. Tests: each of the three test files gains a ..._WithBlankKeyConfigured_IsDenied fact mirroring MeshCommandAuthInterceptorTests.ExecuteMethod_WithBlankPskConfigured_IsDenied (configured key " ", presented "Bearer "), verified red before the fix. ConfigServeAuthInterceptor and LocalDbSyncAuthInterceptor also gain the HasExactlyOnePublicConstructor pin their two siblings already had — Grpc.AspNetCore silently stops invoking an interceptor with more than one public constructor, which would disable auth without failing anything. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1 |
||
|
Joseph Doherty
|
4d2dfb5b50 |
test(mesh): step 7 — replace the boundary test with MeshGrpcBoundaryTests and sweep the comments
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
|
||
|
Joseph Doherty
|
7e7df337cc | feat(mesh): CentralCommandService on the ConfigServe listener | ||
|
Joseph Doherty
|
6a86201105 |
feat(mesh): gRPC command fan-out behind Mode=Grpc
MeshGrpcCommandDialer places one NodeCommandService.Execute call on one node: a GrpcChannel cached per endpoint (the GrpcDeploymentArtifactFetcher shape), the TARGET node's cluster's bearer plus x-otopcua-cluster naming that cluster, and a per-call deadline linked to the caller's token. A cluster with no resolvable key is refused locally with Unauthenticated — there is no arm on which a command leaves central unauthenticated, even one the far side would have rejected. CentralCommunicationActor gains the third mode branch. The contact load now also projects GrpcPort and, from the SAME rows in the SAME pass, builds the dispatch set — which is what makes 'the gRPC fan reaches exactly the SendToAll recipient set' structural rather than two queries that have to keep agreeing. A row with a null GrpcPort is skipped warn-once (the TelemetryNodeSource precedent). Per-node results are piped back for LOGGING ONLY: named node, gRPC status, dropped. No retry, no queue. Every dial is scheduled on the thread pool rather than called from the receive. Found by review and pinned red-first: an async method runs synchronously to its first await, and the dialer's pre-await work is TryGetPsk, which blocks on the secret store — bounded but seconds, and repeated after every refresh since the refresh drops the cache. Dialing inline parked the actor's mailbox, so deployment acks queued behind a slow node; the regression test tells an ApplyAck while a dial is held and fails without the hop. The contact refresh also wires the Phase 1A register: it publishes the loaded cluster ids as the PSK provider's known-cluster set and drops the key cache, so a rotated key takes effect within one ContactRefreshSeconds. Neither happens under Dps or ClusterClient. Both existing mode branches are behaviourally untouched and every existing fact passes unmodified. The one edit to shared text: the Status.Failure warning drops the word 'ClusterClient', since that handler now serves all three transports. ControlPlane.Tests 205 -> 230, Host.Tests 92, Cluster.Tests 184, 0 warnings. |
||
|
Joseph Doherty
|
14d21496fe |
feat(mesh): known-cluster gate + Matches on the PSK provider
Phase 1A security register items 1 and 2, landed ahead of the command fan-out because they are independently testable and the fan-out is the first caller of TryGetPsk. Item 1 — known-cluster gate. MeshPskProvider gains SetKnownClusters, an atomic whole-set swap consulted by TryGetPsk BEFORE the config map and before the secret store. A cluster id outside the set is refused in O(1) with no store read and no audit write, which closes two things at once: the pre-auth store/audit flood an unauthenticated caller could drive with a forged x-otopcua-cluster header, and the timing oracle that defeated CentralCommandAuthInterceptor's uniform denial text (unknown clusters are never cached, so they paid a store read on every probe while known ones answered from memory). Empty or never-published leaves the gate inactive — central publishes the set from its DB contact refresh, and 'the DB told me nothing' must not become a fleet-wide authentication outage during admin boot. Item 2 — bool Matches(clusterId, presented) on IMeshPskProvider, with FixedTimeEquals inside the provider. CentralCommandAuthInterceptor now calls it and never holds key material; its two provider-side denial logs collapse into one (the provider answers both as false) and the operator hint is restated there. No production caller publishes a cluster set yet, so behaviour is unchanged until the fan-out commit wires the contact refresh. ControlPlane.Tests 190 -> 205, Host.Tests 91 -> 92. |
||
|
Joseph Doherty
|
64774652b7 | feat(mesh): NodeCommandService on the telemetry listener | ||
|
Joseph Doherty
|
24e55d3395 | fix(mesh): Batch B review fixes — bounded store read, cache poison guard, test hardening | ||
|
Joseph Doherty
|
68a416e4fc | feat(mesh): per-cluster PSK provider + central interceptor | ||
|
Joseph Doherty
|
fbda015c6e | feat(mesh): node-side PSK interceptor | ||
|
Joseph Doherty
|
39db75083d |
feat(drivers): every driver reports the authored tags it skipped (Gitea #541)
A driver builds its RawPath -> definition table at init and tolerates a
per-tag miss: an unmappable TagConfig, an unparseable address, a
capability-matrix rejection. That tolerance is right -- one bad tag must not
take down the other few thousand. What was missing is any statement of the
fact afterwards. A skip is not an error and happens before any read, so the
driver stays Healthy, errors/5 min stays 0, Last error stays empty, and Last
read advances live off the tags that did map. On the wonder Makino in #540
that reading held while focas-z-34184 ignored 20 of its 22 tags.
New optional capability ISkippedTagReporter, backed by a shared
SkippedTagLedger so the reset/commit/cap rules are one implementation rather
than one per driver. The report -- authored count, exact skipped count, and a
capped sample of RawPath + reason -- rides DriverHealthChanged to a new Tags
column on /hosts, over the Phase 5 telemetry stream. No table: a driver-only
node has no ConfigDb to write to, the lesson from #521.
The issue named eight drivers. It is twelve -- every driver in
DriverTypeNames.All has an authored-tag skip path. Calculation (no scriptId),
MTConnect (no readable DataItem id), Galaxy (no attributeRef) and OpcUaClient
(no nodeId) were unlisted, and OpcUaClient's was the quietest of the twelve:
NamespaceMap drops such an entry with a bare `continue` and not even a log
line. SkippedTagReporterFleetTests pins the invariant reflectively over every
IDriver in src/Drivers/, so a thirteenth cannot re-introduce the silence.
Sql drops a tag at TWO points and its report is the union of both: a blob
that does not map, AND an identifier the 8.1 catalog gate rejects against the
live catalog. The second is what schema drift produces -- a renamed column, a
dropped view -- months after a deployment sealed green, and reporting only
the first would have that driver claim "0 skipped" while polling half its
tags, which is worse than reporting nothing because the column would then be
actively reassuring.
Deliberately additive to driver state, never a change of it. Latching
non-Healthy for a config problem would satisfy both halves of the #527
comms-loss conjunction -- 90 s without a successful read AND non-Healthy --
and a driver whose tags were ALL skipped has no successful reads by
construction, so it would reconnect-loop forever against a healthy device.
Three details that are load-bearing rather than incidental:
- The skip digest is part of the publish fingerprint. A skipped tag leaves
every other fingerprint component identical to a clean driver's, so without
it PublishHealthSnapshot's dedup swallows exactly the publish carrying the
signal -- the trap that already bit the rediscovery and host-status
channels. The digest covers the sample, not just the counts: a redeploy
that fixes one tag and breaks another leaves both counts unchanged.
- The count is exact; only the sample is capped, and the tooltip says so when
it is showing fewer than it counted. A truncated list that looked complete
would have an operator fix twenty tags and believe the job done.
- The ledger stages and commits, so an init that throws part-way leaves the
previous report standing. A partial snapshot would read as "we now skip
fewer tags", the opposite of the truth.
The proto field needs no companion presence flag, unlike host_statuses: a
singular message field carries real presence in proto3, and a has_ flag would
collide with protoc's generated HasSkippedTags.
Verified red-first three ways: dropping SkippedTagDigest from the fingerprint
fails the two dedup tests, removing the seam from one driver fails the fleet
guard, and dropping the catalog-gate leg fails the Sql union test. Full
solution builds with zero new warnings against
|
||
|
Joseph Doherty
|
2aa42f90eb |
fix(security): the ACL startup warning described stage 5's scope, not stage 6's
It told the operator "Read, Browse and Subscribe remain ungated" — which stage 6 is precisely the change to. A warning that misstates what a security control does is worse than no warning, because it is read as authoritative. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
f692f6afe4 |
feat(security): enforce node ACLs on write, alarms and HistoryRead (#520 stage 5)
`Security:Acl:Enabled` (default false) makes scoped `NodeAcl` grants decide data-plane authorization at the three seams that have one, and gives HistoryRead a check it never had. Read, Browse and Subscribe stay ungated — stage 6. The ACL decision REPLACES the server-wide role check rather than chaining with it. Chained, a grant would have to confer the role as well as pass the scope check, and only Cluster-scoped grants confer a role — so every narrow grant would have stayed inert and the flag would have looked inert too. The load-bearing test is an ALLOW: a session carrying no roles at all writing successfully on the strength of its grant. A deny-only test passes against a build where the ACL is never consulted, because the role gate denies too. Found en route: the principal never reached the session. The authenticator passed it as OpcUaUserAuthResult.DisplayName, but RoleCarryingUserIdentity derives DisplayName from the presented token — for a key-bearing client that is the static username `apikey`, identical fleet-wide. So the trie's matching key was dropped at the identity boundary, and the audit ClientUserId on a rejected write recorded `apikey` for every consuming system. Principal and ApiKeyId are now carried explicitly. Beyond the stage as sketched: - Revocation reaches an OPEN session. Every decision re-checks the authenticating key against the current snapshot; without it a key revoked by a deployment kept its authority until the client chose to reconnect, which for a machine-to-machine consumer can be indefinitely. - UserAuthorizationState is built per CALL, not per session. The per-session shape exists so an LDAP session can re-resolve membership that changed out of band; nothing can change out of band under API keys, so the generation stamp is the whole invalidation signal and a per-call record picks it up for free. IsStale/NeedsRefresh therefore cannot fire here and are documented as vestigial. - The trie is built inside DataPlaneAuthorityState.Apply, under the same generation as the snapshot, rather than by a second wiring at the publish actor. An uninstalled trie denies everything — indistinguishable from working enforcement. Deleting the install turns 13 of the 22 new tests red. - The deploy log reports what the BUILD dropped instead of re-deriving it. The old count agreed with the builder on one of its four drop reasons; a rule with a blank ScopeId, a retired scope kind, or a path not starting at a realm vanished with the log reading zero. Both IApiKeyStore implementations widened to the full grant set. The narrowing was redundant — ApiKeyInterimRoleMapper applies the same filter itself — and became a trap once scope was enforced. Pinned by asserting the roles are unchanged, with the two grants carrying different flags so it cannot pass by accident. Falsified by mutation: enforcement never engaging, the write seam ignoring the ACL, the realm dropped from the scope, the revocation check removed, and the trie never installed each turn the intended tests red and no others. Full solution suite green. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
06c56d7719 |
fix(cluster,acl): OTOPCUA_ROLES fallback + one string-identity rule (#532, #533)
Two independent defects surfaced in review of #520 stage 3.
#532 — Cluster:Roles had no OTOPCUA_ROLES fallback, despite two doc sites
promising one. A node configured only through the env var — which drives ALL of
Program.cs's conditional wiring, so it reads as the authoritative knob — bound an
empty Roles array and joined the cluster carrying no Akka roles at all. It ran,
served, and logged nothing wrong, while central reported "no driver-role nodes
are Up" and every deployment to it timed out. Not hypothetical: the field fix on
wonder-app-vd03 was to hand-add Cluster__Roles__0/1 to a Windows service's
registry environment. Anyone reading the code to diagnose that box would have
concluded from the XML doc that the fallback was already applied.
AddOtOpcUaCluster now post-configures AkkaClusterOptions through the shared
RoleParser.ResolveEffectiveRoles, so BuildClusterOptions (Akka's own member
roles), ClusterRoleInfo, the ControlPlane singleton scope and
ConfigDbApiKeyStore's ClusterId all see one answer. Two sibling validators
(ConfigSource, Telemetry) read Cluster:Roles directly and were exempting the
exact node they exist for — an env-only driver-only node has no ConfigDb and
could sit on ConfigSource:Mode=Direct unchallenged. They resolve the same way now.
Also collapsed the four open-coded `Roles.FirstOrDefault(IsClusterRole)` +
ClusterIdFromRole copies into RoleParser.ClusterIdFromRoles. They had drifted:
three classified the cluster role ordinally, so a verbatim-bound `Cluster-SITE-A`
yielded a null ClusterId — and a null ClusterId applies NO cluster filter, which
made #528's cross-cluster over-grant guard fail OPEN. IsClusterRole is now
case-insensitive; the ClusterId suffix keeps its authored spelling, because
normalizing it would silently rename the Akka role of a running deployment.
#533 — three string-identity rules in one authorization path. AclScopeRef used
default (ordinal) equality while the trie builder filtered ClusterId and hashed
principals OrdinalIgnoreCase; the deploy gate's cross-cluster check was ordinal
too, which was the worse half — it could reject a whole deploy over a ClusterId
case difference the runtime would have accepted. It only worked because resolver
and builder were handed the same NodeAcl instances.
The rule is now stated once, on AclScopeRef, and applied throughout:
identifiers (ScopeId, ClusterId, Principal) compare OrdinalIgnoreCase; path
segments compare Ordinal. AclScopePathResolver re-keys its own maps rather than
documenting a comparer contract for callers, so the two independent producers —
EF at the gate, artifact JSON on the node — cannot diverge from the outside.
PermissionTrieNode.Children stays Ordinal: those are browse names, and Speed /
speed are two nodes that can coexist under one device.
Split PermissionTrieBuildDiagnosticReason.RetiredScopeKind out of
UnresolvedScope. A legacy FolderSegment row and a typo'd id had the same message
and different remedies.
Falsified: five mutations, each reddening only its own guard — remove the
PostConfigure (3 tests), revert IsClusterRole to ordinal (3), revert AclScopeRef
equality (1), revert the resolver's map comparer (6), revert the gate's
comparisons (2), remove the FolderSegment short-circuit (1).
Test isolation: OTOPCUA_ROLES is process-global and xUnit runs classes in
parallel, so the three validator suites that now resolve effective roles share a
non-parallel collection. Without it, cases passing no roles would intermittently
observe another class's env var.
Full solution build clean; full suite green apart from two pre-existing failures
confirmed identical on
|
||
|
Joseph Doherty
|
3a6b2a1daf |
feat(acl): variable-depth scope model + the scopePaths producer (#520 stage 3)
Makes the /raw device tree addressable by an ACL rule, and builds the
scope-path producer PermissionTrieBuilder has accepted since it was written
but which had no caller anywhere.
The plan for this stage said "add NodeHierarchyKind.Raw, whose path is the
RawPath segments Folder/Driver/Device/TagGroup/Tag". That shape is wrong.
RawFolder.ParentRawFolderId and TagGroup.ParentTagGroupId are both
self-referencing, so folders and tag groups each nest without bound and a real
RawPath is folder…/Driver/Device/group…/Tag. NodeScope was fixed-arity and
PermissionTrie.CollectMatches was five hardcoded steps, so the scope model
could not address a real raw tree at all. NodeScope is now {ClusterId, Kind,
Segments} and the walk is a segment loop. The trie STRUCTURE was already
variable-depth (NodeAclPath.Segments is a list, Children is a recursive
dictionary), so only the walker and the scope record had to change.
Four decisions worth keeping:
- The namespace level became the realm. Level 1 gated on NodeScope.NamespaceId,
but v3 has no Namespace entity — the snapshot was retired when the address
space went dual-namespace, and the identifier appears nowhere in production
outside NodeScope. So the level could only ever fail to match, which silently
made every sub-cluster grant unreachable while cluster grants kept working.
It is now a realm token derived from the scope's Kind, not carried on it, so
the walk and the producer cannot disagree about its spelling.
- Provenance comes from the granting row's own ScopeKind, recorded on TrieGrant
at build time. Depth-derived provenance could not name a raw level at all,
since the depth of "Device" varies with folder nesting.
- The trie comparer moved from OrdinalIgnoreCase to Ordinal. Raw sibling-name
uniqueness is ordinal, so Speed and speed can both exist under one device and
a grant on either would have matched both.
- The null-scopePaths "deterministic test mode" is deleted. It planted
unresolvable rows at a direct child of the trie root, a position the walker
cannot reach, so the grant looked placed and could never match — and every
test using that mode proved nothing about production placement. scopePaths is
now required, and a row that does not resolve is dropped with a diagnostic.
A scope resolves to a LIST of paths: a raw tag projected into N equipments is
reachable at N+1 NodeIds and a write through any of them lands on the same
driver reference, so a Tag-scoped grant covering only the raw node would be
bypassable through a UNS one. The lookup is keyed on (ScopeKind, ScopeId), not
ScopeId alone — ScopeId has no FK and no type discriminator, so an id collision
across entity types would hand a grant the wrong node's paths.
Also closed here, all found by review rather than asked for:
- NodeAcl rows had no deploy-time validation of any kind and were not reachable
from DraftValidator at all (DraftSnapshot carried no NodeAcls). They are now
gated for dangling ids, cross-cluster scopes, blank/contradictory ScopeId, the
retired FolderSegment kind, and undefined ordinals.
- ConfigDbApiKeyStore conferred roles from Cluster-scoped grants authored in ANY
cluster, because it filtered on scope kind with no ClusterId predicate while
ArtifactApiKeyStore read an already-cluster-filtered snapshot. On a fused
admin+driver node that was a cross-cluster over-grant. Its doc comment claimed
the two stores matched.
- The two AdminUI ACL banners still said enforcement was LDAP-group -> role
mapping, which stages 1/2 made false in both directions.
- The /apikeys page was never classified in PageAuthorizationGuardTests, so that
test has been red since the key-minting commit merged — I did not run
AdminUI.Tests before merging stage 4.
Falsified rather than assumed: 8 mutations against the trie tests, 5 against the
deploy gate (including one proving a cross-cluster row is not silently
mis-diagnosed as unresolved), a staged red->green for the over-grant regression,
and the realm gate, the comparer and the malformed-path guard each broken
individually to confirm only their own tests go red.
Node ACLs are still not ENFORCED — nothing constructs a NodeScope or calls the
evaluator yet. That is stages 5 and 6; this stage makes the model able to
express what those will enforce.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
a438ac1640 |
feat(acl): keys + ACLs reach driver-only nodes via the artifact (#520 stage 4)
A driver-only node has had no ConfigDb since per-cluster mesh Phase 4, so it registered NullOpcUaUserAuthenticator and denied every UserName token. Its keys now ride the deployment artifact instead. - ConfigComposer snapshots ApiKeys alongside the NodeAcls it already carried. Only the peppered HMAC ships, never a secret, and the pepper is node config that is deliberately not in the artifact. - DeploymentArtifact.ParseDataPlaneAuthority decodes both node-side. Grants are scoped to the node's own cluster; keys are not, because ApiKey carries no ClusterId — a key is a fleet-wide credential and the ACL rows are what bind it. - OpcUaPublishActor installs them on DataPlaneAuthorityState on every apply, BEFORE its plan.IsEmpty early-return: an artifact that only revokes a key produces an empty address-space plan, so feeding after that return would leave the revoked key working until an unrelated structural change redeployed. - ArtifactApiKeyStore serves them to session activation; a node with a ConfigDb keeps ConfigDbApiKeyStore, whose per-request read is what makes revocation immediate there. - OtOpcUaNodeManager gains DataPlaneAuthority — the cluster identity blocker #4 named. NodeScope.ClusterId is required and the evaluator fails closed on a mismatch, so until now no scope it built could have evaluated. Nothing reads it until stages 5/6. The generation is a node-local COUNTER driven by the RevisionHash, not derived from its bits. PermissionTrieCache treats the highest GenerationId as current, so a generation taken from hash bytes would install a lower number about half the time and the cache would go on serving the previous deployment's ACLs, silently. Sessions are not stamped with it yet — that belongs with the first seam that reads a UserAuthorizationState, in stage 5, rather than being built now and consulted by nothing. Live-gated on docker-dev against site-a-1 (driver-only, no ConfigDb): key authenticates, wrong secret denied, both halves of the revocation asymmetry observed, and a SITE-A grant confirmed not to reach site-b-1. Key minted with an independent Python HMAC so the verifier was checked against something other than itself. Rig left clean. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
087a969ee6 |
feat(acl): API key minting — AdminUI page + admin service (#520)
Pulled forward from stage 7 because without it the feature is unusable and un-live-gateable: there was no way to create a key short of hand-inserting a row and computing an HMAC by hand. The minting logic is an IApiKeyAdminService in Core.Authorization, NOT logic in the Razor page — this repo has no bUnit, so anything only a page can reach is untestable, and minting credentials is not where that belongs. The page is a thin shell over it. Moved to Core.Authorization so both ends can reach them: ApiKeyOptions (was in Host) and OpcUaPrincipals (was in OpcUaServer.Security). The OPC UA server resolves principals; the AdminUI must refuse to mint a key claiming a reserved one. OpcUaServer already gained a Core reference in stage 1b, so no new edge — AdminUI gains one. ApiKeyOptions is now bound UNCONDITIONALLY rather than in the driver branch: a driver node needs the pepper to VERIFY keys and an admin node needs it to MINT them, and getting that wrong on either side is a silent authentication failure rather than a startup error. Validation rules, all covered: the reserved 'anonymous' principal is refused case-insensitively (so "ANONYMOUS" cannot slip past into a distinct principal that the evaluator would then match case-INsensitively against anonymous's grants); blank name/principal; a principal longer than the column, rejected rather than truncated, because silent truncation yields a key that authenticates and is granted nothing with no visible cause; a past expiry. Generation runs BEFORE the DbContext is opened so an unconfigured pepper surfaces as a clear error instead of a stored row whose hash nothing can verify. The page shows the plaintext ONCE and says so; it is held in component state and written nowhere else, because the server keeps only a hash. Revocation is flagged in the UI as taking effect only after the next deployment — driver-only nodes have no ConfigDb, so keys ride the artifact. FleetAdmin-only (Administrator), not ConfigEditor: this mints credentials. Core.Tests 68 green. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
dd939d3c97 |
feat(acl): anonymous becomes a normal grantable principal (#520 D3)
Anonymous sessions used to short-circuit in HandleImpersonation — "let the SDK's
default validation stand" — so they always connected and always carried no roles.
Now admission and authority are separated:
admission -> OpcUa:AllowAnonymous, the HOST's decision (it owns token policies)
authority -> the authenticator, resolving the reserved principal "anonymous"
through the SAME grant path a key's principal goes through
So anonymous can be granted WRITE, not merely tolerated as read-only, and it
arrives at the downstream gates as the same RoleCarryingUserIdentity a key
produces — which is the real payoff: no gate carries an anonymous-shaped branch,
so there is no bypass path to get wrong.
AllowAnonymous defaults to TRUE on purpose. Defaulting it false would silently
deny every client without a key the moment this deploys, including ScadaBridge's
Data Connection Layer in another repo. Making keys mandatory is a rollout
decision, not an upgrade side effect. When false, the Anonymous token policy is
withheld so a client is refused at DISCOVERY rather than only at activation, and
activation is rejected independently in case a client attempts a token type the
endpoint never advertised.
Two fail-closed choices, both falsified:
- A failure to read anonymous's grants DENIES rather than admitting with no
roles. Admitting is the pre-#520 behaviour, and defaulting to it exactly when
the server cannot determine what anonymous may do stops being harmless once
reads are gated. Making it admit reds Anonymous_denies_when_grants_cannot_be_read.
- A key may not authenticate as the reserved "anonymous" principal, or a
credentialled client silently inherits whatever anonymous was granted.
AuthenticateAnonymousAsync is a DEFAULT interface implementation returning
admit-with-no-roles, i.e. exactly the pre-#520 behaviour, so
NullOpcUaUserAuthenticator and every test double keep working rather than being
forced to reimplement it — important because driver-only nodes run the Null one
and would otherwise black out. Note a default interface method is only reachable
through the interface-typed reference, which is how OpcUaApplicationHost holds it.
An existing test asserted the old fall-through and correctly went red; it is
rewritten to pin the new contract rather than deleted, with the reason recorded.
OpcUaServer 366, Host 42 green.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
65fda7237f |
refactor(acl): delete the LDAP data-plane authenticator and its tests (#520)
Stage 2b. LdapOpcUaUserAuthenticator was left registered nowhere by stage 2a; this removes it, its three test files, and the config surface that existed only to serve it. Deleted: LdapOpcUaUserAuthenticator, LdapOpcUaUserAuthenticatorTests, LdapAuthResilienceTests, LdapGlauthOutageLiveTests (19 tests). Also removed four Security:Ldap keys — AuthTimeoutMs, MaxConcurrentAuthentications, OutageFailureThreshold, OutageCooldownSeconds — plus their validator rules and binding/validator test coverage. Verified first that the ONLY consumers were the deleted class and the validator rules asserting them; none appear in any checked-in appsettings. Leaving them would have created exactly the dead-config surface #521/#524/#525 were filed to clear up. Worth stating why that resilience machinery does not need relocating rather than just deleting. It existed because the OPC UA SDK raises impersonation SYNCHRONOUSLY under a SessionManager-wide event lock, so a hung LDAP bind there serialized every session activation server-wide — Anonymous and X509 included. Key verification is one indexed lookup plus one HMAC with no network I/O, so the hazard the wall-clock bound, concurrency cap and outage circuit guarded against no longer exists on that path. LDAP now runs only on AdminUI login, an ordinary async request path where a slow directory affects the one request making it. ConnectionTimeoutMs is RETAINED: it is projected into the shared library and is not authenticator-specific. Docs corrected rather than left to rot: - CLAUDE.md's "LDAP Authentication" section described the deleted file as the data-plane authenticator. Rewritten, including a warning not to "restore" it on seeing the LDAP wiring that AdminUI legitimately still needs. - infra/README §5b (the GLAuth outage live gate) marked RETIRED with the reason, rather than silently dropped — the harness GLAuth is still used by OTOPCUA_HARNESS_USE_LDAP=1 and is unaffected. - docs/security.md's five-key resilience table cut to the one surviving key, and a new API-key identity section added. Historical archreview/ records are left as-is; they are dated accounts. Host.IntegrationTests 211 tests (was 230, i.e. exactly the 19 removed) with the same 3 pre-existing failures verified against clean master. Host 37, Security 89. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
0329fa4d4b |
feat(acl): API-key identity replaces LDAP on the OPC UA data plane (#520)
Stage 2a. Clients present the static username "apikey" with an
otua_<keyId>_<secret> credential as the password; the key resolves to a
principal. LDAP registrations are deliberately RETAINED for AdminUI login —
only IOpcUaUserAuthenticator moves, which is what D2 asked for.
The load-bearing decision here is what roles a key gets. The two live gates
(WriteOperate on Value writes, AlarmAck on Part 9 methods) check a SERVER-WIDE
role set, so the cutover has to keep producing one or it would lock every client
out of writes long before per-node evaluation lands in stage 5.
ApiKeyInterimRoleMapper therefore maps ACL grants to roles, but ONLY
cluster-scoped ones. The obvious version — union the permission flags across all
of a principal's grants — would let a grant authored narrowly ("WriteOperate on
Equipment X") confer server-wide write authority, because nothing in the pipeline
can enforce scope yet. That fails in the dangerous direction: the operator's
narrow intent silently becomes broad. Restricting to cluster scope under-grants
instead, which is no worse than today (where every ACL row does nothing) and can
never exceed what was authored. Falsified: deleting the scope check reds exactly
A_narrower_scoped_write_grant_confers_nothing_yet and nothing else.
Verification is one indexed lookup plus one HMAC, with no network I/O at all.
That matters because the SDK raises ImpersonateUser synchronously while holding a
SessionManager-wide event lock, so this path does not stall one request — it
serializes every session activation server-wide. It is the main practical gain
over the LDAP bind it replaces.
Every rejection returns one opaque message so the server cannot be used as an
oracle for whether a key id exists or which half of a credential was wrong; a
test asserts the failure modes really are indistinguishable AND that a valid key
still succeeds, so it cannot pass by denying everything.
Caught while wiring: OtOpcUaServerHostedService takes IOpcUaUserAuthenticator as
a REQUIRED ctor parameter, so leaving a driver-only node unregistered is not a
quiet fallback to deny-all — it fails DI and the node does not boot.
NullOpcUaUserAuthenticator is now registered explicitly there. A driver-only node
has no ConfigDb since Phase 4 and so has no key source until the artifact-backed
store lands in stage 4; it denies UserName tokens until then, which is the safe
direction, and anonymous sessions are unaffected.
LdapOpcUaUserAuthenticator is now registered nowhere and is dead code. Deleting
it plus LdapOpcUaUserAuthenticatorTests / LdapAuthResilienceTests /
LdapGlauthOutageLiveTests is stage 2b, kept separate because those cover a
live-gated resilience feature (hard wall-clock bound, bounded concurrency,
directory-outage circuit) that exists precisely BECAUSE the LDAP bind ran inline
on session activation — a constraint the key path does not have. That deserves
its own commit, not a footnote in this one.
Anonymous-as-a-grantable-principal (D3) is also stage 2b.
Host 37, Security 89, OpcUaServer 364 green.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
c74ccc55b5 |
feat(mqtt): flatten Sparkplug Template/DataSet metrics to sibling tags (#515)
`SparkplugCodec` handled scalars; `DataSet`, `Template` and `PropertySet` were skipped
with a warning. They now flatten to addressable tags (owner decision, recorded on the
issue):
Template → sibling metrics, `/`-joined, recursively: Motor → Motor/Speed, Motor/Stator/Temp
DataSet → one metric per cell, {column}/{rowIndex}: Recipe → Recipe/Step/0, Recipe/Target/1
PropertySet → node metadata, never tags — it describes a metric rather than being one
Flattening was chosen over OPC UA Structures because flattened scalars are consumable by
any client with no type dictionary, and the result mirrors how the `/raw` tree works.
**The row index is always present, including on row 0.** Emitting a bare column name for
a single-row table and an indexed one otherwise would make a tag's name depend on how many
rows the plant published — a table growing from one row to two would rename every tag
already authored against it.
**A DATA metric carries the parent's alias and no name**, so member names cannot be
composed until the alias table has resolved the parent. Expansion therefore happens in
`SparkplugIngestor.PublishStructuredMembers` — after resolution, at the single funnel both
the DATA and birth-value paths already pass through — not at decode time. The alias table
installs a binding per member so browse enumerates them and a DDATA member that omits its
datatype falls back to the birth's (guessing from the CLR type would read a signed Int32 of
-42 out of its unsigned field as 4294967254).
**Skip-and-warn survives, and refusal is whole** — the explicit requirement on the issue.
A template definition, an unnamed member, an extension value, a column/type mismatch, a
short row, or a payload past MaxDepth(5)/MaxMembers(512) refuses the ENTIRE metric. A
silently half-flattened UDT looks exactly like a complete one in the picker.
The browse picker offers the members and deliberately NOT the structured parent, which has
no value and no servable datatype. The rediscovery signature likewise tracks members rather
than the parent, or a UDT gaining a member would leave the signature unmoved.
Member paths contain `/`, which #509 already solved: the slashed name travels intact as
`metricName` while the derived tag Name is coerced to a legal RawPath segment.
Live-gated 3/3 against the Mosquitto fixture with the project-owned simulator extended to
publish a nested Template and a 2x2 DataSet. Falsified at three points: dropping the member
bindings, not expanding the parent, and refusing to flatten templates each turn exactly the
expected tests red.
Also fixes a regression from the #511 commit in the same branch: `AddOtOpcUaDriverFactories`
resolved `IDriverPrimaryRoleSource` with `GetRequiredService` but only `AddOtOpcUaRuntime`
registered it, so a graph wiring driver factories without the full runtime failed at
resolution. Two Host integration suites do exactly that and caught it; both extensions now
TryAdd the same concrete `RedundancyRoleView`, so the registration is order-independent and
the fail-loud resolve is kept.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
7f2108dcf2 |
feat(mqtt): Sparkplug primary-host STATE publishing, Primary-gated (#511)
`ActAsPrimaryHost` was accepted by config and by the AdminUI form and did nothing but
log a warning. It now delivers the whole three-part contract or none of it: a retained
`ONLINE` on `spBv1.0/STATE/{hostId}`, a matching retained `OFFLINE` armed as the MQTT
Last Will, and an explicit `OFFLINE` on a clean stand-down.
**The claim is a dedicated connection, not a message.** A Will can only be registered in
the CONNECT packet — it cannot be added to a live session, changed, or withdrawn. So a
claim that is acquired and released at runtime cannot ride the long-lived ingest session.
`SparkplugPrimaryHost` opens its own connection (client id `{clientId}-spbhost`, forced
clean-session) when the node gains Primary and closes it when the node loses Primary; the
connection's lifetime *is* the claim, so every leg follows from its lifecycle. Ingest is
untouched by a role change — no dropped subscriptions, no lost birth cache, no data gap at
the Secondary->Primary transition.
**Primary-gated** (owner decision, recorded on the issue). Both pair nodes subscribe
independently, so an ungated claim means both claim the same host id — and the sharp
failure is not the two fighting over the retained STATE, it is that the SECONDARY's death
fires its Will and publishes OFFLINE while the Primary is healthy, stopping every edge node
in the group. Registering the Will only on whichever node is Primary at connect time fails
the other way: a node that later gains Primary would publish ONLINE with no Will behind it.
New seam: `IDriverPrimaryRoleSource` (`Core.Abstractions`) carrying the *data-plane* verdict
(`PrimaryGatePolicy.ShouldServiceAsPrimary`), not the alarm-drain verdict — an unknown role
with a driver peer present must deny. Registered unconditionally by `AddOtOpcUaRuntime`,
deliberately NOT alongside `IRedundancyRoleView`, which `AddAlarmHistorian` registers only
when that section is enabled: piggybacking would leave a pair with no alarm historian
falling back to "claim unconditionally". Polled on a 1s timer rather than evented — a
process singleton raising events at per-instance drivers outlives every driver it holds.
Accepted trade: a brief edge-visible OFFLINE window during failover. That is a genuine
state transition, not a glitch to hide.
Live-gated 5/5 against the Mosquitto fixture, including the Will leg — `AbandonForTests`
drops the socket with no DISCONNECT, because a clean disconnect makes the broker discard
the Will by protocol rule, so a test driven through `StopAsync` would observe the
stand-down OFFLINE and pass against an implementation that never armed a Will. Falsified:
removing the Will turns that leg and only that leg red.
Also corrects two stale claims found en route: the AdminUI form still said MQTT tags are
read-only (#508 shipped), and the driver doc still said nothing consumes `IRediscoverable`.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
|
||
|
Joseph Doherty
|
30d0697c28 |
feat(hosts): surface per-host connectivity on /hosts; drop the unwritable table (#521)
`IHostConnectivityProbe` was a dead surface: eleven drivers implement it, `GetHostStatuses()` had ZERO production call sites, and `OnHostStatusChanged` had no subscriber outside the Galaxy driver's own aggregator. Per-host connectivity was computed by every driver and read by nobody. The issue offered "build the publisher or delete it". The publisher as its entity doc described it — driver nodes upserting `DriverHostStatus` rows — is not buildable: per-cluster mesh Phase 4 gates `AddOtOpcUaConfigDb` on the `admin` role, so a driver-only node has no ConfigDb connection to write rows with. So the capability is kept and the transport changed. `DriverHealthChanged.HostStatuses` now carries the probe result to `/hosts` as a Hosts column. That channel already reached the page, already survives the mesh split via the Phase 5 gRPC telemetry stream, and already replays a last-value snapshot on re-subscribe — so per-host state re-primes after a reconnect without a durable store. Both halves of the interface finally do what they are for: the event triggers a prompt publish, the pull is the source of truth. The point of the column is the case the driver-level state chip structurally cannot express: a multi-device driver stays aggregate-Healthy while ONE of its devices is unreachable. Two traps, both pinned by tests that were falsified against the prod code: - The host digest MUST be in the publish fingerprint. On a single-host-down transition every other fingerprint component is unchanged, so the dedup would swallow exactly the publish carrying the news — the trap that already bit the rediscovery signal. Removing it turns the guard test red, verified. - null (no probe) must stay distinct from empty (probe with no hosts). proto3 cannot tell an absent repeated field from an empty one, hence the explicit `has_host_statuses` flag; collapsing them would render every probe-less driver as one whose devices are all fine. Dropped: the DriverHostStatus entity, enum, DbSet, model config and table (migration DropDriverHostStatusTable — empty on every deployment, so the scaffolder's data-loss warning is moot, and Down() recreates it exactly). Found en route, NOT fixed here: `DriverInstanceResilienceStatus` is the identical defect — no writer, no reader, only a DbSet declaration, while the live data rides the `driver-resilience-status` telemetry channel. Its doc-comment now states that rather than describing the sampler and AdminUI join that were never built. Filed as #524 rather than widening this schema change beyond what was asked. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo |
||
|
Joseph Doherty
|
09a401b881 |
feat(drivers): consume IRediscoverable as an operator re-browse prompt (§8.2, #518)
Nine drivers raise IRediscoverable.OnRediscoveryNeeded when they observe that a remote's tag set may have changed — a Galaxy redeploy, a TwinCAT symbol-version bump, an MTConnect agent restart, a Sparkplug rebirth. Nothing in src/ subscribed, so every raise went into the void. Drivers even wrap the invoke in try/catch for "subscriber threw"; there has never been a subscriber. DriverInstanceActor now attaches the event in PreStart and detaches in PostStop, mirroring AttachAlarmSource/DetachAlarmSource. Attach is per-actor, not per-connect: the raise has no connection affinity and can land while the driver is between connects. The detach is load-bearing rather than tidy — the IDriver instance outlives the actor when the host respawns a child around the same driver object, so a missing -= accumulates one handler per respawn, each holding a dead Self. The signal rides the existing driver-health snapshot to /hosts as a "re-browse" chip. It is ADVISORY: the served address space is deliberately not rebuilt, because v3 authors raw tags explicitly through the /raw browse-commit flow and a runtime graft would materialise nodes no operator approved and no deployment artifact records. Two things worth knowing: - PublishHealthSnapshot dedups on a health fingerprint, and a rediscovery raise on an otherwise-healthy driver changes none of the four fields it hashed. Without adding the timestamp to that tuple the dedup swallows the exact publish carrying the signal. Reverting that one line turns 3 of the 5 new tests red, which is how it was confirmed rather than assumed. - The new test stub implements IDriver from scratch instead of deriving from the shared StubDriver, whose GetHealth() returns DateTime.UtcNow — its fingerprint differs on every call, so the dedup never engages and the dedup test would have passed vacuously either way. The planned discovery-result drift detector was NOT built. Modbus, S7, MQTT, AbLegacy and Sql all echo authored config from DiscoverAsync rather than browsing the device, so a discovered-vs-authored diff is permanently empty for them — it would have been another plausible-looking inert seam, which is the class this audit exists to remove. IRediscoverable is the trustworthy signal because the driver asserts it deliberately. DriverHealthChanged, IDriverHealthPublisher.Publish and HostsDriverRow gain defaulted optional parameters, so no existing call site changed. telemetry.proto gains fields 8/9 and both Phase-5 mappers carry them. Runtime.Tests 512 passed / 31 skipped. |
||
|
Joseph Doherty
|
a107a1a8b9 |
feat(mtconnect): host registration + DriverTypeNames const + guard parity (Task 16)
Register the MTConnect factory + probe with the Host, add the DriverTypeNames.MTConnect constant, and keep DriverTypeNamesGuardTests green. - DriverTypeNames: new MTConnect const + appended to All. - DriverFactoryBootstrap: MTConnectDriverFactoryExtensions.Register in Register(...) at the default Tier A (managed HttpClient/XML, in-process; Tier C is the only tier arming process-recycle, wrong here), and TryAddEnumerable(IDriverProbe -> MTConnectDriverProbe) in AddOtOpcUaDriverProbes so the probe reaches admin-only nodes (the admin-pinned Test-Connect singleton) without double-registering on a fused admin,driver node. - Host.csproj: ProjectReference to Driver.MTConnect (required to compile the Register call). - Core.Abstractions.Tests.csproj: ProjectReference to Driver.MTConnect so the guard's bin scan discovers the factory. This and the constant MUST land together -- verified RED (2/4 facts) with the constant alone. - DriverProbeRegistrationTests: MTConnect added to AdminUiDriverTypeKeys; its idempotency fact asserts distinct-probe-count and goes RED without this (and RED if TryAddEnumerable is downgraded to AddSingleton). Reconciles all four "MTConnect" literals onto the one constant: the factory's DriverTypeName, MTConnectDriver.DriverType, and MTConnectDriverProbe.DriverType now all alias DriverTypeNames.MTConnect (no circular reference -- Core.Abstractions is a leaf both driver projects already reference). This is the repo's TwinCat/Focas anti-drift convention. |
||
|
Joseph Doherty
|
c3a2b0f773 |
Merge branch 'feat/mqtt-sparkplug-driver'
# Conflicts: # src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/DriverTypeNames.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Drivers/DriverConfigModal.razor # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/EndpointRouteBuilderExtensions.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/TagConfigEditorMap.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Uns/TagEditors/TagConfigValidator.cs # src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ZB.MOM.WW.OtOpcUa.AdminUI.csproj # src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs |
||
|
Joseph Doherty
|
f79d13e2d8 |
feat(mqtt): factory + DriverTypeNames.Mqtt + host factory/probe registration
Task 9. Adds MqttDriverFactoryExtensions (DriverTypeName = DriverTypeNames.Mqtt, direct deserialization of MqttDriverOptions — no intermediate DTO, mirroring OpcUaClientDriverFactoryExtensions), the DriverTypeNames.Mqtt constant, and both host wiring sites: the factory in DriverFactoryBootstrap.Register and the probe in AddOtOpcUaDriverProbes (the admin-node path Program.cs calls in its hasAdmin block — a probe wired only on driver nodes makes Test-connect silently dead). Carried-forward items: - Converges every MQTT config-parsing seam onto ONE JsonSerializerOptions — MqttJson.Options, in .Contracts alongside MqttDriverOptions. It replaces the probe's internal JsonOpts and the browser's separate private copy; the factory, the probe, MqttDriver.ParseOptions and MqttDriverBrowser now all parse through it. .Contracts is the only assembly all four consumers reference, and the browser's reference to the runtime .Driver project is a documented layering exception scheduled for removal — anchoring the options there would resurrect the duplicate the day it goes away. - Replaces the "Mqtt" literals in MqttDriverProbe, MqttDriverBrowser and MqttDriver with the constant (string value unchanged). - Tightens MqttDriverProbeTests.ProbeAsync_EnumAsName: it asserted only Ok == false + non-empty message, which is also exactly what a JSON-parse failure produces — so it stayed green under the very regression it names. It now asserts the probe got past the parse and reached the network. Falsifiability: deleting JsonStringEnumConverter from MqttJson.Options reddens 9 tests across 4 suites, including the tightened probe test (message becomes "Config JSON is invalid: The JSON value could not be converted to MqttProtocolVersion") — which the pre-fix assertions would have passed. Also references the MQTT driver from Core.Abstractions.Tests so DriverTypeNamesGuardTests' reflective bin scan discovers the new factory and the constant/factory parity check stays honest. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
9b5a96876e |
feat(sql): register Sql factory + probe in Host, add DriverTypeNames.Sql
Add DriverTypeNames.Sql (+ All) as the shared source of truth, and wire the driver into the Host: register the factory in DriverFactoryBootstrap.Register via Driver.Sql.SqlDriverFactoryExtensions.Register(registry, loggerFactory), add the SqlProbe (= Driver.Sql.SqlDriverProbe) probe via TryAddEnumerable, and add the Driver.Sql project reference to the Host. Default tier A (SQL client is managed + cross-platform, so ShouldStub needs no change). Repoint the interim SqlDriver.DriverTypeName / SqlDriverFactoryExtensions.DriverTypeName literals at DriverTypeNames.Sql; both still resolve to "Sql", so the AdminUI TagConfigEditorMap/TagConfigValidator keys and the DriverTypeName-parity test stay green. The Core guard test discovers factories from its own bin, so it also gains a Driver.Sql project reference — that is the "registered-factory side" that lets DriverTypeNamesGuardTests' bidirectional-parity check pass (4/4 green). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
88adec047b |
refactor(health): adopt the shared active-node check (Health 0.3.0)
ClusterPrimaryHealthCheck was written three days' worth of debugging ago because the shared ActiveNodeHealthCheck selected by RoleLeader and reported Healthy for any node lacking the role. Health 0.3.0 fixes the shared check — oldest Up member, role-preference scoping, Unhealthy when the node owns no active work — so the private copy is now duplication rather than a workaround, and it is deleted. RolePreference [admin, driver] reproduces exactly what it did: a fused central node answers for admin (where the singletons and the AdminUI are pinned), a driver-only site node answers for driver. SelectOldestUpMemberOfRole now delegates to the shared ClusterActiveNode instead of re-implementing the age ordering. This is the point of the change rather than a tidy-up: the redundancy snapshot drives the OPC UA ServiceLevel 250/240 split while the health tier drives Traefik's admin routing, so two copies of "oldest Up member of a role" would be two chances to advertise one node as authoritative while gating the data plane on another. ControlPlane takes a ZB.MOM.WW.Health.Akka reference for it; the layering trade-off is recorded at the PackageReference. Behaviour note: a node carrying neither admin nor driver now answers 503 on the active tier instead of 200. That case is reachable — RoleParser admits dev-only and cluster-role-only nodes — and 503 is correct, since such a node owns no active work and must not be in an active-tier pool. No docker-dev node is affected: every rig node is admin+driver or driver-only. Tests replaced in kind. The rule itself is now pinned in the library against a real two-node cluster; what stays OtOpcUa's own decision is the wiring, so ActiveTierRegistrationTests pins which check is on the active tag, that it is registered on driver-only nodes too, and that it is not on the ready tier. Verified: Host.Tests 25/25, ControlPlane redundancy 15/15. |
||
|
Joseph Doherty
|
4550486144 |
fix(health): active tier reports the cluster Primary, not the admin role leader
Closes the remaining half of #494, and corrects the half fixed in |
||
|
Joseph Doherty
|
279d1d0fb1 |
feat(cluster): simultaneous-cold-start split-brain guard (opt-in)
Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6 live gate reproduced this reliably on docker). The prior mitigation was operational (staggered start / compose depends_on), which does not exist on production hardware. The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split without giving up cold-start-alone: - The lower-address node is the preferred founder: self-first, forms immediately. - The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds: reachable => peer-first (join the founder, never race it); unreachable => self-first (partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes, from an explicit reachability signal — never re-formed mid-handshake (the retired SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions) and ClusterBootstrapCoordinator drives the join. Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would reopen the split); fail-fast validation of the timing knobs; the residual "founder dies in the probe->join window" hang is documented and made operator-visible (warning + a restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node cold-start-alone case; 147/147 Cluster tests pass. Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on; site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split; higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
3a4ed8ddb5 |
fix(mesh): break Phase 6 boot-crash — cluster-redundancy singleton scope must not resolve IClusterRoleInfo eagerly
Phase 6 wired the re-homed redundancy singleton in Program.cs as `WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>())` inside the AddAkka configurator lambda. That lambda runs WHILE the ActorSystem is being built, but ClusterRoleInfo depends on the ActorSystem (it is a live Cluster.State view) — so resolving it there recurses into ActorSystem construction and every node dies at boot with StackOverflowException. It compiles cleanly (a runtime DI cycle) and RedundancyStateSingletonRehomeTests passed a hand-built FakeClusterRoleInfo straight into the extension, mocking around the composition-root line that overflows. The docker-dev live gate caught it on first boot. Derive the singleton's cluster-role scope from AkkaClusterOptions (pure config, no ActorSystem) instead: BuildClusterRedundancySingletonOptions / the extension now take AkkaClusterOptions and compute Role = Roles.FirstOrDefault(IsClusterRole) ?? driver — identical to ClusterRoleInfo.ClusterRole, which derives from the same config. Program.cs passes IOptions<AkkaClusterOptions>.Value, which has no ActorSystem dependency. Live-verified: rebuilt image boots with zero stack-overflow lines, cluster singletons form per mesh. 5/5 rehome tests pass; Host builds clean. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
a4d3ab4005 |
docs(mesh): fix stale redundancy-singleton comments + prove the driver fallback end-to-end
Review follow-ups on the redundancy re-home (Phase 6, Task 2):
- Program.cs: the hasDriver comment claimed IClusterRoleInfo "throws at host
start if this driver node carries no cluster role" — false since the fallback
landed. Rewrite it to describe the cluster-{ClusterId} scope with the driver-role
fallback for legacy single-mesh / not-yet-migrated nodes.
- ClusterRoleInfo.cs: the SubscriberActor comment described RedundancyStateActor as
the "admin-role singleton" — stale. Note it is now a cluster-{ClusterId}-scoped
singleton spawned on every driver node (LeaderChanged stays a no-op here).
- Add a host/registry-level test for the driver-role FALLBACK path (roles ["driver"],
no cluster role) closing the coverage asymmetry — it was proven only in the pure
helper. Asserts the node boots and registers RedundancyStateActorKey, i.e. the boot
the earlier throw-based version would have aborted.
The boot helper now supplies an in-memory IDbContextFactory and a fake IClusterRoleInfo:
Akka.Hosting invokes singleton props factories eagerly at StartAsync, and the admin
ClusterNodeAddressReconciler factory reads both (the latter for its per-cluster reconcile
scope, added by a sibling Phase 6 task) — without them the admin boot NREs.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
a2c5d57fa0 |
feat(mesh): re-home the redundancy singleton to a per-cluster driver singleton
Phase 6 splits the single fleet-wide Akka mesh into one 2-node mesh per
application Cluster. RedundancyStateActor was an admin-scoped cluster singleton
that elected ONE fleet-wide driver Primary — but after the split a driver-only
site pair has no admin node in its mesh to host it, so its Primary would never
be elected.
Re-home it: remove the redundancy WithSingleton block from the admin
WithOtOpcUaControlPlaneSingletons set, and add WithOtOpcUaClusterRedundancySingleton,
a singleton spawned from Program.cs's hasDriver branch on EVERY driver node (the
fused central included). It scopes to the node's own cluster-{ClusterId} role when
present, and falls back to the fixed `driver` role otherwise. Each mesh then has
exactly one, electing its own pair-local Primary and publishing redundancy-state
on its own mesh's DistributedPubSub. The election logic (oldest Up driver member)
and the DPS publish are unchanged — they become pair-local after the split.
The driver-role fallback (Decision 2) deliberately does NOT throw when a node has
no cluster role: that is the pre-Phase-6 fleet-wide behavior on a legacy single
mesh, so legacy/harness (TwoNodeClusterHarness: admin,driver) and not-yet-migrated
deployments keep booting unchanged. On a genuinely split 2-node mesh the `driver`
role is already pair-local (the mesh IS the pair), so the fallback is correct in
both worlds. Cluster-scope, when present, additionally survives an accidental
two-mesh merge by keeping one singleton per cluster.
The role scope + driver-role fallback are pinned by unit tests against the
extracted BuildClusterRedundancySingletonOptions helper (the BuildDowningHocon
pattern); admin-removal + driver-registration are pinned behaviorally against a
booted node's ActorRegistry.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
|
||
|
Joseph Doherty
|
af9c9d78da |
test(mesh-phase5): 3-port listener reachability + comment fix for the telemetry endpoint
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
f131c1cca5 |
feat(mesh-phase5): host the telemetry gRPC server on driver nodes (dedicated h2c port)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
2e70ef88aa |
fix(mesh-phase5): telemetry service — graceful client-disconnect + null-safe required-string mapping
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
50f1620d79 |
feat(mesh-phase5): fail-closed bearer interceptor for the telemetry stream
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
e742fee452 |
feat(mesh-phase5): node-side TelemetryStreamService (hub -> server-streaming)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
8e5090edf3 |
feat(mesh-phase5): tap the 4 telemetry publish seams into the local hub (DPS intact)
Each of the four telemetry publish seams now also emits its DPS payload into the node-local ITelemetryLocalHub (Phase 5), leaving the existing DistributedPubSub publish untouched: - driver-health AkkaDriverHealthPublisher -> TelemetryItem.Health - resilience DriverResilienceStatusPublisherService -> TelemetryItem.Resilience - alerts ScriptedAlarmHostActor + DriverHostActor (native) -> TelemetryItem.Alarm - script-logs VirtualTagActor + DpsScriptLogPublisher -> TelemetryItem.Script DI services take the hub as a required ctor param; actors take an optional nullable hub threaded through their Props and the real spawn sites (DriverHostActor <- ServiceCollectionExtensions; VirtualTagActor/ScriptedAlarmHostActor <- DriverHostActor), so existing test Props constructions keep compiling and simply do not emit. The hub is a no-op until a gRPC client subscribes, so the emit is safe on every node. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
d907160747 |
feat(mesh-phase4): DriverHostActor runs ConfigDb-free (LocalDb alarm store, guarded acks)
Nullable ConfigDb factory; UpsertNodeDeploymentState no-ops when absent (central persists acks from the ApplyAck); scripted-alarm condition state served from the LocalDb store instead of the ConfigDb-backed Ef store. Combines Phase-4 Tasks 4 + 6. Removes the interim dbFactory! cast. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
4f4cdd05ec |
feat(mesh-phase4): LocalDb alarm-condition-state store (replicated, pair-local)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
833e45db9c |
fix(mesh-phase4): driver-only nodes map LDAP groups from appsettings (no ConfigDb grants)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
a41582ea6a |
feat(mesh-phase4): register ConfigDb only on admin-role nodes
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
648f173b18 |
feat(mesh): FetchAndCache apply path — fetch->cache->apply-from-bytes, null=apply-failure
Phase 3 Task 5. Under ConfigSource:Mode=FetchAndCache, a DispatchDeployment whose revision differs from _currentRevision kicks off a gRPC artifact fetch that PipeTo's its result back to Self as FetchedForApply (never a blocking await in a receive — that would freeze the mailbox for the fetch deadline). HandleFetchedForApply then applies synchronously on the actor thread, mirroring ApplyAndAck: null bytes = apply FAILURE (keep last-known-good, do not advance the revision, ack Failed — #485); non-null bytes reconcile drivers, rebuild the address space and push subscriptions FROM the bytes in hand (OpcUaPublishActor never reads central SQL), then cache them. A faulted fetch task maps to null bytes so it can't escape as an unhandled message. Extracted ReconcileDriversFromBlob from ReconcileDrivers so Direct-read, FetchAndCache and boot-from-cache share one reconcile body (the #485 empty guard moves into it); ApplyCachedArtifact now reuses it instead of duplicating the spawn-plan logic. DI: DriverHostActor.Props gains fetchAndCacheMode + artifactFetcher (LAST, per the positional-forwarding warning); Runtime resolves them from IOptions<ConfigSourceOptions> (fetcher only in FetchAndCache mode); Host registers GrpcDeploymentArtifactFetcher under hasDriver. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
4d88f67c0d |
feat(mesh): config-serve auth interceptor + dedicated h2c listener (fail-closed bearer)
Phase 3 Task 3. Central serves the deployment artifact over a dedicated h2c Kestrel listener (ConfigServe:GrpcListenPort), gated by the ConfigServeAuthInterceptor (shared bearer key, FixedTimeEquals, fail-closed, path-scoped to /deployment_artifact.v1.DeploymentArtifactService/). The listener block is merged with the LocalDb-sync listener so a fused admin+driver node re-applies its existing HTTP surface exactly ONCE and adds both dedicated h2c ports on top — double-binding would throw 'address already in use'; zero re-binding would silently unbind the AdminUI behind Traefik. AddGrpc now registers under hasDriver || hasAdmin with both path-scoped interceptors sharing one pipeline. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
69b697bc3e |
feat(redundancy): manual failover service — graceful Leave of the driver Primary
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
f9f1b8fcee |
fix(localdb): phase-2 live gate — 4 production defects found and fixed
Gate record: docs/plans/2026-07-20-localdb-phase2-live-gate.md. Checks 1, 2, 5, 6 pass. Checks 3 and 4 are NOT satisfied — the defect they were meant to confirm turned out to be the opposite of what the plan assumed. Three defects crash-looped every driver node before check 1 could even run: 1. An empty ServerHistorian:ApiKey kills the host. ServerHistorianOptions- Validator exists to turn exactly that class of failure into a named OptionsValidationException, but its documented fail tier explicitly excluded ApiKey on the reasoning that a keyless client "degrades — the gateway rejects calls". It does not: the client validates its own options at construction, so the process dies during Akka startup and never makes a call. 2. UseTls disagreeing with the endpoint scheme kills the host too, in both directions (both messages confirmed in the shipped client assembly). Moving an endpoint from https to http without clearing UseTls is an ordinary migration slip. 3. Plaintext h2c was UNREACHABLE. HistorianGatewayClientAdapter forwarded the TLS-only options unconditionally, and AllowUntrustedServerCertificate defaults to false, so it always sent RequireCertificateValidation=true — which the client rejects outright when UseTls=false. Every http:// deployment crashed, though the scheme is documented as the supported way to select h2c, and the only workaround was to assert a certificate posture for a connection that has no certificate. The fourth was the blocker, and it is Phase 2's own: 4. The drain gate deferred to a Primary that cannot deliver. Redundancy roles are elected CLUSTER-WIDE; the alarm queue is PAIR-LOCAL. On the rig the elected driver Primary is central-1 — it carries the driver Akka role, replicates nobody's LocalDb and does not even run the alarm historian — so every driver node logged "Historian drain suspended", including the two site-b nodes that have no peer at all. Nothing drained anywhere, where before Phase 2 it drained fine. The cost is not a duplicate; it is the buffer growing to the capacity wall and evicting the audit trail it exists to protect. Fixed in three layers: a separate ShouldDrainAlarmHistory policy (unknown role drains; the two gates now deliberately disagree, and a test pins that); peer- host matching in DriverHostActor so a node stands down only for a Primary holding its rows; and AddAlarmHistorian short-circuiting the gate when replication is unconfigured — testing BOTH Replication:PeerAddress and SyncListenPort, since only the dialing half sets the former while both halves share the queue. Every one of these follows from the asymmetry: a false allow costs a duplicate row, which at-least-once delivery already accepts and payload-hash ids collapse; a false deny loses data silently. A third vacuous test, caught by the same delete-the-guard discipline: the role-view tests stayed green with the guard removed, because AwaitAssert polls until an assertion passes and the assertion was "reads open" — which is the SEEDED value, satisfied at the first poll before the actor processed anything. They now assert the sequence of published values through a recording view; the control then goes red for exactly the cases that matter. Migration evidence: 11 legacy rows across two deliberately overlapping files converged to exactly 9 identical rows on both nodes, proving D-6's payload-hash identity on real nodes rather than in a fixture. Open design fork, recorded in the gate doc rather than decided here: a pair cannot currently identify its own Primary, so both halves drain. Safe in every topology — nothing loses data — but the gate's de-duplication benefit is unrealised until roles are scoped per pair. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
2e4ccf7fe9 |
chore(localdb): phase-2 DoD sweep
Build: 0 errors solution-wide, and 0 warnings from every project this branch
touches. The ~816 solution-wide warnings are pre-existing xUnit1051 /
OTOPCUA0001 / CS86xx in untouched driver + client test projects.
Tests: full solution run compared against a full run on a detached worktree at
the pre-branch baseline
|