`Security:Acl:Enabled` (default false) makes scoped `NodeAcl` grants decide
data-plane authorization at the three seams that have one, and gives HistoryRead
a check it never had. Read, Browse and Subscribe stay ungated — stage 6.
The ACL decision REPLACES the server-wide role check rather than chaining with
it. Chained, a grant would have to confer the role as well as pass the scope
check, and only Cluster-scoped grants confer a role — so every narrow grant
would have stayed inert and the flag would have looked inert too. The
load-bearing test is an ALLOW: a session carrying no roles at all writing
successfully on the strength of its grant. A deny-only test passes against a
build where the ACL is never consulted, because the role gate denies too.
Found en route: the principal never reached the session. The authenticator
passed it as OpcUaUserAuthResult.DisplayName, but RoleCarryingUserIdentity
derives DisplayName from the presented token — for a key-bearing client that is
the static username `apikey`, identical fleet-wide. So the trie's matching key
was dropped at the identity boundary, and the audit ClientUserId on a rejected
write recorded `apikey` for every consuming system. Principal and ApiKeyId are
now carried explicitly.
Beyond the stage as sketched:
- Revocation reaches an OPEN session. Every decision re-checks the
authenticating key against the current snapshot; without it a key revoked by a
deployment kept its authority until the client chose to reconnect, which for a
machine-to-machine consumer can be indefinitely.
- UserAuthorizationState is built per CALL, not per session. The per-session
shape exists so an LDAP session can re-resolve membership that changed out of
band; nothing can change out of band under API keys, so the generation stamp is
the whole invalidation signal and a per-call record picks it up for free.
IsStale/NeedsRefresh therefore cannot fire here and are documented as vestigial.
- The trie is built inside DataPlaneAuthorityState.Apply, under the same
generation as the snapshot, rather than by a second wiring at the publish
actor. An uninstalled trie denies everything — indistinguishable from working
enforcement. Deleting the install turns 13 of the 22 new tests red.
- The deploy log reports what the BUILD dropped instead of re-deriving it. The
old count agreed with the builder on one of its four drop reasons; a rule with
a blank ScopeId, a retired scope kind, or a path not starting at a realm
vanished with the log reading zero.
Both IApiKeyStore implementations widened to the full grant set. The narrowing
was redundant — ApiKeyInterimRoleMapper applies the same filter itself — and
became a trap once scope was enforced. Pinned by asserting the roles are
unchanged, with the two grants carrying different flags so it cannot pass by
accident.
Falsified by mutation: enforcement never engaging, the write seam ignoring the
ACL, the realm dropped from the scope, the revocation check removed, and the
trie never installed each turn the intended tests red and no others.
Full solution suite green.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Makes the /raw device tree addressable by an ACL rule, and builds the
scope-path producer PermissionTrieBuilder has accepted since it was written
but which had no caller anywhere.
The plan for this stage said "add NodeHierarchyKind.Raw, whose path is the
RawPath segments Folder/Driver/Device/TagGroup/Tag". That shape is wrong.
RawFolder.ParentRawFolderId and TagGroup.ParentTagGroupId are both
self-referencing, so folders and tag groups each nest without bound and a real
RawPath is folder…/Driver/Device/group…/Tag. NodeScope was fixed-arity and
PermissionTrie.CollectMatches was five hardcoded steps, so the scope model
could not address a real raw tree at all. NodeScope is now {ClusterId, Kind,
Segments} and the walk is a segment loop. The trie STRUCTURE was already
variable-depth (NodeAclPath.Segments is a list, Children is a recursive
dictionary), so only the walker and the scope record had to change.
Four decisions worth keeping:
- The namespace level became the realm. Level 1 gated on NodeScope.NamespaceId,
but v3 has no Namespace entity — the snapshot was retired when the address
space went dual-namespace, and the identifier appears nowhere in production
outside NodeScope. So the level could only ever fail to match, which silently
made every sub-cluster grant unreachable while cluster grants kept working.
It is now a realm token derived from the scope's Kind, not carried on it, so
the walk and the producer cannot disagree about its spelling.
- Provenance comes from the granting row's own ScopeKind, recorded on TrieGrant
at build time. Depth-derived provenance could not name a raw level at all,
since the depth of "Device" varies with folder nesting.
- The trie comparer moved from OrdinalIgnoreCase to Ordinal. Raw sibling-name
uniqueness is ordinal, so Speed and speed can both exist under one device and
a grant on either would have matched both.
- The null-scopePaths "deterministic test mode" is deleted. It planted
unresolvable rows at a direct child of the trie root, a position the walker
cannot reach, so the grant looked placed and could never match — and every
test using that mode proved nothing about production placement. scopePaths is
now required, and a row that does not resolve is dropped with a diagnostic.
A scope resolves to a LIST of paths: a raw tag projected into N equipments is
reachable at N+1 NodeIds and a write through any of them lands on the same
driver reference, so a Tag-scoped grant covering only the raw node would be
bypassable through a UNS one. The lookup is keyed on (ScopeKind, ScopeId), not
ScopeId alone — ScopeId has no FK and no type discriminator, so an id collision
across entity types would hand a grant the wrong node's paths.
Also closed here, all found by review rather than asked for:
- NodeAcl rows had no deploy-time validation of any kind and were not reachable
from DraftValidator at all (DraftSnapshot carried no NodeAcls). They are now
gated for dangling ids, cross-cluster scopes, blank/contradictory ScopeId, the
retired FolderSegment kind, and undefined ordinals.
- ConfigDbApiKeyStore conferred roles from Cluster-scoped grants authored in ANY
cluster, because it filtered on scope kind with no ClusterId predicate while
ArtifactApiKeyStore read an already-cluster-filtered snapshot. On a fused
admin+driver node that was a cross-cluster over-grant. Its doc comment claimed
the two stores matched.
- The two AdminUI ACL banners still said enforcement was LDAP-group -> role
mapping, which stages 1/2 made false in both directions.
- The /apikeys page was never classified in PageAuthorizationGuardTests, so that
test has been red since the key-minting commit merged — I did not run
AdminUI.Tests before merging stage 4.
Falsified rather than assumed: 8 mutations against the trie tests, 5 against the
deploy gate (including one proving a cross-cluster row is not silently
mis-diagnosed as unresolved), a staged red->green for the over-grant regression,
and the realm gate, the comparer and the malformed-path guard each broken
individually to confirm only their own tests go red.
Node ACLs are still not ENFORCED — nothing constructs a NodeScope or calls the
evaluator yet. That is stages 5 and 6; this stage makes the model able to
express what those will enforce.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
A driver-only node has had no ConfigDb since per-cluster mesh Phase 4, so it
registered NullOpcUaUserAuthenticator and denied every UserName token. Its keys
now ride the deployment artifact instead.
- ConfigComposer snapshots ApiKeys alongside the NodeAcls it already carried.
Only the peppered HMAC ships, never a secret, and the pepper is node config
that is deliberately not in the artifact.
- DeploymentArtifact.ParseDataPlaneAuthority decodes both node-side. Grants are
scoped to the node's own cluster; keys are not, because ApiKey carries no
ClusterId — a key is a fleet-wide credential and the ACL rows are what bind it.
- OpcUaPublishActor installs them on DataPlaneAuthorityState on every apply,
BEFORE its plan.IsEmpty early-return: an artifact that only revokes a key
produces an empty address-space plan, so feeding after that return would leave
the revoked key working until an unrelated structural change redeployed.
- ArtifactApiKeyStore serves them to session activation; a node with a ConfigDb
keeps ConfigDbApiKeyStore, whose per-request read is what makes revocation
immediate there.
- OtOpcUaNodeManager gains DataPlaneAuthority — the cluster identity blocker #4
named. NodeScope.ClusterId is required and the evaluator fails closed on a
mismatch, so until now no scope it built could have evaluated. Nothing reads it
until stages 5/6.
The generation is a node-local COUNTER driven by the RevisionHash, not derived
from its bits. PermissionTrieCache treats the highest GenerationId as current, so
a generation taken from hash bytes would install a lower number about half the
time and the cache would go on serving the previous deployment's ACLs, silently.
Sessions are not stamped with it yet — that belongs with the first seam that
reads a UserAuthorizationState, in stage 5, rather than being built now and
consulted by nothing.
Live-gated on docker-dev against site-a-1 (driver-only, no ConfigDb): key
authenticates, wrong secret denied, both halves of the revocation asymmetry
observed, and a SITE-A grant confirmed not to reach site-b-1. Key minted with an
independent Python HMAC so the verifier was checked against something other than
itself. Rig left clean.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Pulled forward from stage 7 because without it the feature is unusable and
un-live-gateable: there was no way to create a key short of hand-inserting a row
and computing an HMAC by hand.
The minting logic is an IApiKeyAdminService in Core.Authorization, NOT logic in
the Razor page — this repo has no bUnit, so anything only a page can reach is
untestable, and minting credentials is not where that belongs. The page is a thin
shell over it.
Moved to Core.Authorization so both ends can reach them: ApiKeyOptions (was in
Host) and OpcUaPrincipals (was in OpcUaServer.Security). The OPC UA server
resolves principals; the AdminUI must refuse to mint a key claiming a reserved
one. OpcUaServer already gained a Core reference in stage 1b, so no new edge —
AdminUI gains one.
ApiKeyOptions is now bound UNCONDITIONALLY rather than in the driver branch: a
driver node needs the pepper to VERIFY keys and an admin node needs it to MINT
them, and getting that wrong on either side is a silent authentication failure
rather than a startup error.
Validation rules, all covered: the reserved 'anonymous' principal is refused
case-insensitively (so "ANONYMOUS" cannot slip past into a distinct principal
that the evaluator would then match case-INsensitively against anonymous's
grants); blank name/principal; a principal longer than the column, rejected
rather than truncated, because silent truncation yields a key that authenticates
and is granted nothing with no visible cause; a past expiry. Generation runs
BEFORE the DbContext is opened so an unconfigured pepper surfaces as a clear
error instead of a stored row whose hash nothing can verify.
The page shows the plaintext ONCE and says so; it is held in component state and
written nowhere else, because the server keeps only a hash. Revocation is flagged
in the UI as taking effect only after the next deployment — driver-only nodes
have no ConfigDb, so keys ride the artifact.
FleetAdmin-only (Administrator), not ConfigEditor: this mints credentials.
Core.Tests 68 green.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Stage 2a. Clients present the static username "apikey" with an
otua_<keyId>_<secret> credential as the password; the key resolves to a
principal. LDAP registrations are deliberately RETAINED for AdminUI login —
only IOpcUaUserAuthenticator moves, which is what D2 asked for.
The load-bearing decision here is what roles a key gets. The two live gates
(WriteOperate on Value writes, AlarmAck on Part 9 methods) check a SERVER-WIDE
role set, so the cutover has to keep producing one or it would lock every client
out of writes long before per-node evaluation lands in stage 5.
ApiKeyInterimRoleMapper therefore maps ACL grants to roles, but ONLY
cluster-scoped ones. The obvious version — union the permission flags across all
of a principal's grants — would let a grant authored narrowly ("WriteOperate on
Equipment X") confer server-wide write authority, because nothing in the pipeline
can enforce scope yet. That fails in the dangerous direction: the operator's
narrow intent silently becomes broad. Restricting to cluster scope under-grants
instead, which is no worse than today (where every ACL row does nothing) and can
never exceed what was authored. Falsified: deleting the scope check reds exactly
A_narrower_scoped_write_grant_confers_nothing_yet and nothing else.
Verification is one indexed lookup plus one HMAC, with no network I/O at all.
That matters because the SDK raises ImpersonateUser synchronously while holding a
SessionManager-wide event lock, so this path does not stall one request — it
serializes every session activation server-wide. It is the main practical gain
over the LDAP bind it replaces.
Every rejection returns one opaque message so the server cannot be used as an
oracle for whether a key id exists or which half of a credential was wrong; a
test asserts the failure modes really are indistinguishable AND that a valid key
still succeeds, so it cannot pass by denying everything.
Caught while wiring: OtOpcUaServerHostedService takes IOpcUaUserAuthenticator as
a REQUIRED ctor parameter, so leaving a driver-only node unregistered is not a
quiet fallback to deny-all — it fails DI and the node does not boot.
NullOpcUaUserAuthenticator is now registered explicitly there. A driver-only node
has no ConfigDb since Phase 4 and so has no key source until the artifact-backed
store lands in stage 4; it denies UserName tokens until then, which is the safe
direction, and anonymous sessions are unaffected.
LdapOpcUaUserAuthenticator is now registered nowhere and is dead code. Deleting
it plus LdapOpcUaUserAuthenticatorTests / LdapAuthResilienceTests /
LdapGlauthOutageLiveTests is stage 2b, kept separate because those cover a
live-gated resilience feature (hard wall-clock bound, bounded concurrency,
directory-outage circuit) that exists precisely BECAUSE the LDAP bind ran inline
on session activation — a constraint the key path does not have. That deserves
its own commit, not a footnote in this one.
Anonymous-as-a-grantable-principal (D3) is also stage 2b.
Host 37, Security 89, OpcUaServer 364 green.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the
same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for
the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6
live gate reproduced this reliably on docker). The prior mitigation was operational
(staggered start / compose depends_on), which does not exist on production hardware.
The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split
without giving up cold-start-alone:
- The lower-address node is the preferred founder: self-first, forms immediately.
- The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds:
reachable => peer-first (join the founder, never race it); unreachable => self-first
(partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes,
from an explicit reachability signal — never re-formed mid-handshake (the retired
SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions)
and ClusterBootstrapCoordinator drives the join.
Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would
reopen the split); fail-fast validation of the timing knobs; the residual "founder dies
in the probe->join window" hang is documented and made operator-visible (warning + a
restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node
cold-start-alone case; 147/147 Cluster tests pass.
Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on;
site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous
start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split;
higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 wired the re-homed redundancy singleton in Program.cs as
`WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>())`
inside the AddAkka configurator lambda. That lambda runs WHILE the ActorSystem is
being built, but ClusterRoleInfo depends on the ActorSystem (it is a live
Cluster.State view) — so resolving it there recurses into ActorSystem construction
and every node dies at boot with StackOverflowException. It compiles cleanly (a
runtime DI cycle) and RedundancyStateSingletonRehomeTests passed a hand-built
FakeClusterRoleInfo straight into the extension, mocking around the composition-root
line that overflows. The docker-dev live gate caught it on first boot.
Derive the singleton's cluster-role scope from AkkaClusterOptions (pure config, no
ActorSystem) instead: BuildClusterRedundancySingletonOptions / the extension now take
AkkaClusterOptions and compute Role = Roles.FirstOrDefault(IsClusterRole) ?? driver —
identical to ClusterRoleInfo.ClusterRole, which derives from the same config. Program.cs
passes IOptions<AkkaClusterOptions>.Value, which has no ActorSystem dependency.
Live-verified: rebuilt image boots with zero stack-overflow lines, cluster singletons
form per mesh. 5/5 rehome tests pass; Host builds clean.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Review follow-ups on the redundancy re-home (Phase 6, Task 2):
- Program.cs: the hasDriver comment claimed IClusterRoleInfo "throws at host
start if this driver node carries no cluster role" — false since the fallback
landed. Rewrite it to describe the cluster-{ClusterId} scope with the driver-role
fallback for legacy single-mesh / not-yet-migrated nodes.
- ClusterRoleInfo.cs: the SubscriberActor comment described RedundancyStateActor as
the "admin-role singleton" — stale. Note it is now a cluster-{ClusterId}-scoped
singleton spawned on every driver node (LeaderChanged stays a no-op here).
- Add a host/registry-level test for the driver-role FALLBACK path (roles ["driver"],
no cluster role) closing the coverage asymmetry — it was proven only in the pure
helper. Asserts the node boots and registers RedundancyStateActorKey, i.e. the boot
the earlier throw-based version would have aborted.
The boot helper now supplies an in-memory IDbContextFactory and a fake IClusterRoleInfo:
Akka.Hosting invokes singleton props factories eagerly at StartAsync, and the admin
ClusterNodeAddressReconciler factory reads both (the latter for its per-cluster reconcile
scope, added by a sibling Phase 6 task) — without them the admin boot NREs.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 splits the single fleet-wide Akka mesh into one 2-node mesh per
application Cluster. RedundancyStateActor was an admin-scoped cluster singleton
that elected ONE fleet-wide driver Primary — but after the split a driver-only
site pair has no admin node in its mesh to host it, so its Primary would never
be elected.
Re-home it: remove the redundancy WithSingleton block from the admin
WithOtOpcUaControlPlaneSingletons set, and add WithOtOpcUaClusterRedundancySingleton,
a singleton spawned from Program.cs's hasDriver branch on EVERY driver node (the
fused central included). It scopes to the node's own cluster-{ClusterId} role when
present, and falls back to the fixed `driver` role otherwise. Each mesh then has
exactly one, electing its own pair-local Primary and publishing redundancy-state
on its own mesh's DistributedPubSub. The election logic (oldest Up driver member)
and the DPS publish are unchanged — they become pair-local after the split.
The driver-role fallback (Decision 2) deliberately does NOT throw when a node has
no cluster role: that is the pre-Phase-6 fleet-wide behavior on a legacy single
mesh, so legacy/harness (TwoNodeClusterHarness: admin,driver) and not-yet-migrated
deployments keep booting unchanged. On a genuinely split 2-node mesh the `driver`
role is already pair-local (the mesh IS the pair), so the fallback is correct in
both worlds. Cluster-scope, when present, additionally survives an accidental
two-mesh merge by keeping one singleton per cluster.
The role scope + driver-role fallback are pinned by unit tests against the
extracted BuildClusterRedundancySingletonOptions helper (the BuildDowningHocon
pattern); admin-removal + driver-registration are pinned behaviorally against a
booted node's ActorRegistry.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Each of the four telemetry publish seams now also emits its DPS payload into the
node-local ITelemetryLocalHub (Phase 5), leaving the existing DistributedPubSub
publish untouched:
- driver-health AkkaDriverHealthPublisher -> TelemetryItem.Health
- resilience DriverResilienceStatusPublisherService -> TelemetryItem.Resilience
- alerts ScriptedAlarmHostActor + DriverHostActor (native) -> TelemetryItem.Alarm
- script-logs VirtualTagActor + DpsScriptLogPublisher -> TelemetryItem.Script
DI services take the hub as a required ctor param; actors take an optional
nullable hub threaded through their Props and the real spawn sites
(DriverHostActor <- ServiceCollectionExtensions; VirtualTagActor/ScriptedAlarmHostActor
<- DriverHostActor), so existing test Props constructions keep compiling and simply
do not emit. The hub is a no-op until a gRPC client subscribes, so the emit is safe
on every node.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 5. Under ConfigSource:Mode=FetchAndCache, a DispatchDeployment whose
revision differs from _currentRevision kicks off a gRPC artifact fetch that PipeTo's
its result back to Self as FetchedForApply (never a blocking await in a receive — that
would freeze the mailbox for the fetch deadline). HandleFetchedForApply then applies
synchronously on the actor thread, mirroring ApplyAndAck: null bytes = apply FAILURE
(keep last-known-good, do not advance the revision, ack Failed — #485); non-null bytes
reconcile drivers, rebuild the address space and push subscriptions FROM the bytes in
hand (OpcUaPublishActor never reads central SQL), then cache them. A faulted fetch task
maps to null bytes so it can't escape as an unhandled message.
Extracted ReconcileDriversFromBlob from ReconcileDrivers so Direct-read, FetchAndCache
and boot-from-cache share one reconcile body (the #485 empty guard moves into it);
ApplyCachedArtifact now reuses it instead of duplicating the spawn-plan logic.
DI: DriverHostActor.Props gains fetchAndCacheMode + artifactFetcher (LAST, per the
positional-forwarding warning); Runtime resolves them from IOptions<ConfigSourceOptions>
(fetcher only in FetchAndCache mode); Host registers GrpcDeploymentArtifactFetcher under
hasDriver.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 3 Task 3. Central serves the deployment artifact over a dedicated h2c
Kestrel listener (ConfigServe:GrpcListenPort), gated by the ConfigServeAuthInterceptor
(shared bearer key, FixedTimeEquals, fail-closed, path-scoped to
/deployment_artifact.v1.DeploymentArtifactService/). The listener block is merged
with the LocalDb-sync listener so a fused admin+driver node re-applies its existing
HTTP surface exactly ONCE and adds both dedicated h2c ports on top — double-binding
would throw 'address already in use'; zero re-binding would silently unbind the
AdminUI behind Traefik. AddGrpc now registers under hasDriver || hasAdmin with both
path-scoped interceptors sharing one pipeline.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Build: 0 errors solution-wide, and 0 warnings from every project this branch
touches. The ~816 solution-wide warnings are pre-existing xUnit1051 /
OTOPCUA0001 / CS86xx in untouched driver + client test projects.
Tests: full solution run compared against a full run on a detached worktree at
the pre-branch baseline 2e46d054. The two failure SETS are identical -- all 13
tests, same names, zero new failures. Net +26 tests: +3 Core.AlarmHistorian
(drain gate), +6 Runtime (role view), +17 Host.IntegrationTests (migrator +
convergence). Set comparison rather than counts, because the suite carries
standing environment- and load-dependent failures a count would hide.
The greps found real drift Task 6 missed -- eight live sites still naming the
deleted SqliteStoreAndForwardSink, including the AdminUI /alarms/historian
panel text, which is user-visible, and a <see cref> in HistorianAdapterActor
that resolved to nothing without warning. All repointed at
LocalDbStoreAndForwardSink; CLAUDE.md's alarm-history paragraph now also
records the LocalDb buffer and the primary-gated drain, and drops DatabasePath
from the knob list. docs/AlarmTracking.md still promised an
AlarmHistorianOptions.Validate() startup warning for a relative DatabasePath
and an empty SharedSecret; both branches are gone, so it now says so.
Code references to AlarmHistorian:DatabasePath reduce to exactly two intentional
ones: AlarmSfLegacyMigrator.LegacyPathKey and its test. No `new SqliteConnection`
remains anywhere in Core.AlarmHistorian.
Recon doc gains the durable verification record: guard-deletion evidence for
both vacuous passes, the two exact-set replicated-table pins (both assert set
equality, so an added or a dropped registration fails), and the baseline test
comparison with a per-failure account of why each of the 13 is not this
branch's.
Stops here per the plan. Task 8's live gate needs explicit go-ahead; nothing on
this branch is to be merged.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
The Kestrel re-bind read only urls/ASPNETCORE_URLS. The aspnet:8.0+ base images
set ASPNETCORE_HTTP_PORTS=8080 (all interfaces) as the CONTAINER default in place
of ASPNETCORE_URLS, so a driver node that never sets URLS fell straight through to
Kestrel's localhost:5000 default — silently moving its health/metrics surface to
loopback when the sync listener took over Kestrel. Now falls through URLS →
HTTP_PORTS/HTTPS_PORTS (wildcard, all-interfaces) → localhost:5000, mirroring
Kestrel's own source precedence. Found on the docker-dev live gate (central nodes
were fine — they set ASPNETCORE_URLS=9000 explicitly).
Routes the host's secrets registration through a new AddOtOpcUaSecrets extension
that gates the ISecretStore implementation on Secrets:Replication:Enabled.
Opt-in gate (default FALSE)
This call decides which ISecretStore every node resolves — including driver-role
nodes with no auth/AdminUI, where a wrong store surfaces as drivers failing to
open sessions rather than as a failing test. With the flag false the wiring is
the pre-existing AddZbSecrets(config, "Secrets") call, unchanged, so current
behavior is byte-identical. With it true, AddZbSecretsAkkaReplication replaces
that call (it invokes AddZbSecrets internally; calling both would double-register).
Extracted to a named extension specifically so the registration is testable:
Program.cs is top-level statements and cannot be exercised by a container test,
which is how a "registered but never resolvable" defect ships unnoticed.
Serializer HOCON
AkkaSecretsReplication.SerializationConfig is merged into the ActorSystem config
inside the AddAkka configurator, conditionally on the same gate — a non-replicating
node carries no bindings for messages it will never see. Merged via
AddHocon(..., HoconAddMode.Append), Akka.Hosting's fallback merge and the same mode
the existing base-config merge uses; a raw Config.WithFallback would fight the
builder's own assembly.
Lazy-actor mitigation
The replication actor is created lazily on first ISecretStore resolution, so a node
that never touches a secret would never announce a manifest and would silently never
converge. SecretReplicationStarter (IHostedService) resolves the store once at
startup to make participation unconditional.
KNOWN BLOCKER — replication is currently NON-FUNCTIONAL; do not enable
ZB.MOM.WW.Secrets.Replicator.AkkaDotNet 0.2.0 never binds its own ISecretReplicator.
AddZbSecretsAkkaReplication calls AddZbSecrets FIRST, which does
TryAddSingleton<ISecretReplicator, NoOpSecretReplicator>(); the package's own
TryAddSingleton<ISecretReplicator>(AkkaSecretReplicator) that follows is therefore
a no-op. Verified empirically in a built container: with Enabled=true,
ISecretReplicator resolves to NoOpSecretReplicator, so ReplicatingSecretStore
publishes into a sink and no actor is ever spawned.
Consequence: the startup hook cannot create the actor, and the test asserting it
does is committed Skipped with the evidence. Not worked around here — the fix
belongs upstream (AddSingleton, or register before calling AddZbSecrets).
Because the flag defaults false, this commit is inert in production.
Tests: SecretsReplicationRegistrationTests (new) — disabled path resolves plain
SqliteSecretStore and needs no ActorSystem; enabled path resolves
ReplicatingSecretStore AND the undecorated concrete SqliteSecretStore the decorator
is built from (the exact registration gap that shipped once); startup hook registered
only when enabled. Red before wiring (4 assertion failures), green after: 6 pass,
1 skipped (blocker above).
Build: 861 warnings / 0 errors, unchanged from baseline (full --no-incremental A/B).
Host.IntegrationTests: 123 pass, 6 skip, 1 fail — AbCip_Green_AgainstSim, verified
pre-existing on the stashed tree (fixture-gated).
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Runtime ISecretResolver must be present on every clustered node regardless of
role: driver-only nodes resolve Layer-B DriverConfig secret: refs but have no
auth/DP/AdminUI (all admin-role-gated). Placed in the unconditional flow next to
AddOtOpcUaHealth(), outside both role blocks. Data Protection left untouched.
Adopt the ZB.MOM.WW.Secrets library (0.1.2, Gitea feed) into OtOpcUa under CPM:
- Directory.Packages.props + NuGet.config source mapping for the three packages
- Host/AdminUI/Driver.Galaxy.Contracts/Driver.OpcUaClient package references
- Layer-A pre-host ${secret:} config expander in Host/Program.cs, placed after all
config providers assemble and before the first config read (mechanism only,
behavior-neutral no-op until Task 4 introduces tokens)
- Secrets section in appsettings.json (env-var KEK, SQLite store, 30s cache TTL)
Mechanism mirrors the proven HistorianGateway adoption. KEK never committed.
Corrects the #459 finding. 2-node keep-oldest recovery works fine (the ScadaBridge
sister project proves it); OtOpcUa was missing the supervision pieces that make it
automatic, and docs/Redundancy.md wrongly claimed in-place oldest-crash failover.
Mechanism (confirmed on a 2-container rig + by decompiling Akka KeepOldest.OldestDecision):
on an OLDEST-node crash keep-oldest downs the LONE survivor (DownReachable including
myself) — down-if-alone can't rescue a lone survivor (its branch needs >=2 survivors).
Recovery is exit-and-rejoin: run-coordinated-shutdown-when-down terminates the node and
the service supervisor restarts it. My earlier 'total outage' was a docker-dev artifact
(no restart policy); production Install-Services.ps1 already has sc.exe failure restart.
Changes (ScadaBridge parity):
- ActorSystemTerminationWatchdog (Host, registered after AddAkka): watches
ActorSystem.WhenTerminated and on an unexpected self-down calls StopApplication so the
process exits (supervisor restarts it) instead of idling with a dead actor system.
Distinguishes graceful shutdown via _stopRequested + ApplicationStopping. 3 unit tests.
- docker-dev: restart: unless-stopped on the host anchor (models production supervision) +
both redundancy peers in SeedNodes so a restarted node re-forms via either peer.
- docs/Redundancy.md: rewrote the split-brain recovery section — younger-loss = in-place
fast failover; oldest-loss = exit-and-rejoin under supervision (not in-place); the three
requirements (supervisor + watchdog + both-node seeds); flagged HardKillFailoverTests as
non-representative (Transport.Shutdown, not a real crash). Instant in-place takeover on
ANY single loss needs 3+ members.
Cluster.Tests 29/29 (SBR guards), watchdog tests 3/3, full solution builds.
Live re-verify of the watchdog image pending (host docker disk full).
Four continuous-historization types (IHistorizationOutbox, HistorizationOutboxEntry,
IHistorianValueWriter/HistorizationValue, HistorizationCommitMode) drifted into a
ZB.MOM.WW.OtOpcUa.Core.Abstractions.Historian sub-namespace while the other six files
in the same folder correctly use the root namespace. This violated decision #59's
flat-public-surface rule and left InterfaceIndependenceTests.AllPublicTypes_LiveInRootNamespace
red on master (pre-existing, predates round-2 remediation).
Move all four to the root namespace and drop/retarget the now-redundant
'using ...Core.Abstractions.Historian;' in the ~11 consumers. No behavioral change.
Verified: Core.Abstractions.Tests 129/129 (was 128/129), Runtime.Tests Historian+DI 72/72,
Gateway driver unit 102/102, full solution build 0 errors.
Adds <summary>/<param>/<returns>/<inheritdoc> where missing and removes
project bookkeeping IDs (task/tracking refs) from shipped code comments,
so the docs read cleanly and CommentChecker is quiet except for known
false positives (PLC/protocol terms, event/IEqualityComparer inheritdoc).
Doc/comment-only; no logic changed; solution builds clean.
PR #423 shipped GatewayTagProvisioner + unit tests but never registered it in
DI nor passed it into the AddressSpaceApplier, so deploying historized tags used
the no-op NullHistorianProvisioning and never called the gateway's EnsureTags
(confirmed live on wonder-app-vd03: zero EnsureTags calls on a historized deploy).
Addresses HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md.
Issue 1 (wire provisioner):
- Runtime: AddHistorianProvisioning extension (gated on ServerHistorian:Enabled,
mirrors AddServerHistorian) + NullHistorianProvisioning TryAdd default in
AddOtOpcUaRuntime; WithOtOpcUaRuntimeActors resolves IHistorianProvisioning and
passes it into the applier.
- Gateway driver: GatewayHistorian.CreateProvisioner factory (mirrors CreateDataSource).
- Host: Program.cs calls AddHistorianProvisioning after AddServerHistorian.
- Tests: AddHistorianProvisioningTests (config-gated registration + the
register->resolve->applier->EnsureTags chain).
Issue 2 (observability): AddressSpaceApplier logs the provisioning tally on every
successful dispatch (was gated behind Failed/Skipped > 0), including dispatched=N
so a dispatched=N/requested=0 line flags the dormant no-op. +2 tests.
Issue 3 (30s HistoryRead on unprovisioned tags): root coupling fixed by Issue 1;
documented the CallTimeout knob + coupling. Default left at 30s pending the
multi-data-point investigation the issue requests (lowering risks truncating
legitimate large reads).
Issue 4 (docs): docs/Historian.md gains a "Tag auto-provisioning (EnsureTags)"
section and CLAUDE.md a wiring/gating note (both stress ServerHistorian:Enabled).
Sibling scadaproj/CLAUDE.md carries no false claim -> unchanged.
Pre-existing Serilog observation: anchor CWD to AppContext.BaseDirectory before
AddZbSerilog so the relative file sink stops landing in C:\Windows\System32 under
the Windows-service CWD.
Builds 0-error; Runtime.Tests 355, OpcUaServer.Tests 329, Gateway.Tests 99 (+4
live-skipped) all green.
The HistorianGateway driver is now the sole historian read/write+alarm backend, so the
Wonderware sidecar projects are dead code. Removes the 5 Wonderware projects (driver,
.Client, .Client.Contracts, + their 2 test projects) from the solution and tree, and fully
retires the vestigial 'Historian.Wonderware' driver type (UI/probe-only; it had no driver
factory): the Host probe registration, the AdminUI driver-config surface (driver page,
tag-config editor/model/validator entry, address picker/builder, driver-type catalog +
dropdown + edit-router entries), and their tests. Prunes the now-unused Wonderware
connection fields (Host/Port/UseTls/ServerCertThumbprint/SharedSecret) from
AlarmHistorianOptions (keeping Enabled + the SQLite store-and-forward knobs) and refreshes
the stale XML docs that named Wonderware as the production backend.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Bind ContinuousHistorizationOptions (Enabled/OutboxPath/CommitMode/
CommitIntervalMs/DrainBatchSize/DrainIntervalSeconds/Capacity/backoff) with a
warn-only Validate(); gated on Enabled AND the ServerHistorian gateway being
configured, the Host registers the durable FasterLogHistorizationOutbox (container
-disposed) + a gateway-backed GatewayHistorianValueWriter, and binds outbox
depth/dropped observable gauges on the central scraped meter. WithOtOpcUaRuntimeActors
spawns the recorder (over the same dependency-mux ref) when the options + writer +
outbox resolve, registering ContinuousHistorizationRecorderKey. Spawned with an EMPTY
historized-ref set: the deployed address space builds later, so ref population is a
documented follow-on (a later SetHistorizedRefs feed) — T18 wires the actor + outbox
+ writer + meters; the ref feed is the known remaining gap.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
DriverHostActor/DriverInstanceActor and cluster events log via Akka's ILoggingAdapter, which had no Serilog bridge — under the Windows service host the default StandardOutLogger output is discarded, making the driver-role actor graph invisible (this masked the data-plane diagnosis).
- Add Akka.Logger.Serilog 1.5.60 (deps satisfied by Akka 1.5.62 / Serilog 4.3.1).
- WithOtOpcUaClusterBootstrap: ConfigureLoggers(DebugLevel; ClearLoggers(); AddLogger<SerilogLogger>()) — Akka.Hosting owns logger setup, so HOCON akka.loggers alone is not honored.
- Program.cs: set static Serilog.Log.Logger from the DI root logger after Build() (AddZbSerilog registers the MEL provider but not the static logger, which the Akka SerilogLogger and the startup banner both need).
Code review at HEAD 7286d320. Host-001 (High): /metrics was auth-gated on admin
nodes (Prometheus 401) -> AllowAnonymous. Host-002: register LdapOptionsValidator
unconditionally for fail-fast startup validation on admin-only nodes. Host-004: fix
metrics XML doc. Host-003 (docs) left Open.
Wire the materialised AlarmConditionState method handlers so a client calling
Acknowledge/Confirm/Shelve/AddComment is gated on the AlarmAck data-plane role
and, when allowed, routed back to the scripted-alarm engine via a new
`alarm-commands` DistributedPubSub topic.
- Commons: new AlarmCommand DTO (AlarmId/Operation/User/Comment/UnshelveAtUtc).
- ScriptedAlarmHostActor: add AlarmCommandsTopic const.
- OtOpcUaNodeManager: settable AlarmCommandRouter + wire OnAcknowledge/OnConfirm/
OnAddComment/OnShelve/OnTimedUnshelve. Each resolves the principal off
ISessionOperationContext.UserIdentity as RoleCarryingUserIdentity, fails closed
(BadUserAccessDenied) when the AlarmAck role is absent or no identity, else maps
+ routes an AlarmCommand and returns Good. OnShelve discriminates OneShotShelve/
TimedShelve/Unshelve from the SDK flags; TimedShelve expiry = UtcNow + ms.
No Akka/IActorRef handle — only the Action<AlarmCommand> delegate. T20 de-dup
note left; WriteAlarmCondition untouched.
- OpcUaServer.Security: OpcUaDataPlaneRoles.AlarmAck shared const (the role was a
bare string everywhere; introduced one symbol for the gate + tests).
- OtOpcUaSdkServer: SetAlarmCommandRouter pass-through.
- Host: boot wiring publishes each command via mediator.Tell(Publish(...)) using a
lazy ActorSystem accessor (mirrors DpsScriptLogPublisher).
- Tests: 11 new gate + mapping tests (OpcUaServer.Tests 88->99, all green).
A thin gateway over the admin-operations cluster singleton so CI/scripts can trigger a
deployment without the Blazor button. Forwards to the same IAdminOperationsClient.
StartDeploymentAsync; mounted on admin-role nodes. Auth is a fixed-time X-Api-Key check
against Security:DeployApiKey (orthogonal to the cookie-only web auth); AllowAnonymous so the
auth fallback doesn't 401 it, self-disabling (503) until the key is set. Outcome->status:
202/200/409/422. Unit tests for the key check + outcome mapping; HTTP E2E (real auth + real
deploy via the 2-node harness). Documented in docs/security.md.
The per-role overlay (appsettings.{role}.json) was appended after WebApplicationBuilder's
default sources, so it outranked environment variables — a baked role file could not be
overridden by a deployment env var. In otopcua-dev this meant appsettings.admin.json's
Security:Ldap:DevStubMode=false beat the compose's DevStub override, so every AdminUI login
attempted a real LDAP bind against a non-existent server and failed with 'Unexpected
authentication error'.
- Program.cs: re-append AddEnvironmentVariables() + AddCommandLine(args) after the role
overlay so deployment overrides keep top precedence (overlay still beats base appsettings).
- docker-dev/docker-compose.yml: the DevStub env var targeted the stale 'Authentication:Ldap'
section; the code reads 'Security:Ldap'. Corrected the prefix on every host node (+ comment).
Dev AdminUI login now signs in as Administrator via the DevStub bypass.
Joseph Doherty