ConfigSource:Mode = Direct (default, read central SQL) | FetchAndCache (fetch
the artifact from central over gRPC, read LocalDb). The validator fails host
start on a FetchAndCache shape that cannot fetch — no endpoints, a non-http(s)
endpoint, no shared key, or a non-positive timeout — because each otherwise
surfaces as a silent absence (a deploy that never applies). ConfigServe (the
central serve surface) takes a plain Configure: a 0 port is a valid "disabled".
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 1. Three changes to the embedded base config, plus a test that
reads every one of them back off a RUNNING ActorSystem rather than off the
HOCON text -- the settled idiom here (SplitBrainResolverActivationTests),
because several fragments compete and the losing one still reads correctly in
the file.
- Registers the ClusterClientReceptionist extension. It is not auto-started, so
without this the boundary only listens if some code path happens to call
ClusterClientReceptionist.Get(system) first: "is the boundary up?" would
depend on startup ordering rather than on configuration.
- buffer-size = 0. This is the one genuine behaviour change and it is NOT what
the sister project runs -- they never wrote this section and inherit 1000.
Buffering would replay a DispatchDeployment on reconnect and apply a
deployment the coordinator already sealed as TimedOut, leaving the node on a
configuration the database says failed.
- log-frame-size-exceeding = 32000b. An oversized frame is dropped WITHOUT
tearing down the association: heartbeats keep flowing, the node reports
healthy, and the only symptom is a command that never arrived. Our deploy
notify is payload-free, so this canary should stay quiet.
reconnect-timeout and receptionist.role restate Akka defaults; they are stated
explicitly and pinned because the behaviour is load-bearing, and the comments
say which is which.
One test was VACUOUS on first write: Config.GetInt returns 0 for a missing key,
so the buffering assertion passed before the feature existed -- it could not
distinguish "explicitly 0" from "absent, default 1000 in force". Rewritten to
assert through ClusterClientSettings/ClusterReceptionistSettings, the objects
the client is actually built from. Sabotage-verified after the fix: setting
buffer-size back to 1000 reddens exactly that one test.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 0. Binds a validated `MeshTransport` section selecting which
transport carries central->node commands, defaulting to Dps so nothing changes
until the rig gate passes.
The validator exists because every fault it catches otherwise surfaces as an
ABSENCE -- a deployment that never arrives, an alarm ack that does nothing --
which has no stack trace and no failing node to point at. Two of the shapes it
rejects are ported from the sister project's shipped mistakes: a contact point
carrying an actor-path suffix, and (documented in the options XML) a template
listing the node's OWN remoting port as a central contact, which is a permanent
failure in the initial-contact rotation.
Contact points are required only under ClusterClient mode; requiring them under
the default would make the section mandatory on admin-only nodes that have no
reason to carry them.
Sabotage-verified: relaxing the empty-contacts guard and the actor-path-suffix
guard turns exactly those two tests red, and nothing else.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.
- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
self-first). The site nodes are untouched -- they seed off central-1 only and
are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
would refuse to boot every driver-only site node. Identity is compared on
PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
outside Akka's join handshake, so it could not distinguish "no seed answered"
from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
is not ignored mid-handshake, it wins. The live gate (ea45ace1) caught it
islanding a node that a manual failover had bounced: InitJoinAck received, no
Welcome inside the window because the peer's ring still held the old
incarnation, watchdog fired, second cluster. The TCP reachability guard patched
that one shape; the race stayed. It was also inert for every non-seed node, so
after this change it can never usefully fire.
- SelfFormBootstrapTests -> SelfFirstSeedBootstrapTests: real in-process clusters
through the production bootstrap at production failure-detection timings, incl.
a falsifiability control proving the OLD peer-first ordering never forms (that
test failed at 11s against the watchdog, which is what proved the retirement),
the restart-into-a-live-peer case that killed the watchdog, and a simultaneous
cold start converging on ONE cluster.
Mirrors ScadaBridge 4a6341d8; anticipates per-cluster-mesh-program Phase 6, which
had this retirement queued.
The live gate for the manual-failover control caught the self-form fallback
forming a SECOND cluster. A node bounced by a failover restarted, received
InitJoinAck from its live peer — the join was in flight and healthy — but did
not get the Welcome inside the 10s window, because the peer's ring still held
the node's previous incarnation (Exiting -> Down -> Removed). The fallback
fired on the timer and the node islanded itself until an operator restarted it.
The original design assumed Cluster.Join(SelfAddress) would be ignored
mid-handshake. It is not — it wins. And since manual failover deliberately
produces that restart, every failover could island the node it bounced.
Before self-forming, TCP-probe the other seed addresses; a reachable peer means
wait another window instead. That is also what the fallback claims to detect:
'no seed answered InitJoin (peer down at boot)'.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
RedundancyStateActor derived the driver Primary from ClusterState.RoleLeader("driver").
Akka offers two different notions of a "first" member and they are not the same
one: role leader is the lowest-ADDRESSED Up member (host, then port), while
ClusterSingletonManager places singletons on the OLDEST — lowest up-number.
They agree on a freshly-formed cluster, which is why every existing test passed.
They diverge after any restart: the restarted node re-joins as the youngest while
keeping its address, so if it holds the lower address it becomes role leader while
the singletons stay put. The snapshot would then name a Primary that is not
hosting the work, and every Primary-gated surface follows it — inbound device
writes, native-alarm acks, the fleet-wide alerts emit, and the alarm-history
drain would all enable on the wrong node while the node actually running the
singletons stayed gated off.
BuildSnapshot now selects the oldest Up member carrying the driver role, matching
singleton placement. Leaving members are excluded: a node handing its singletons
over must not be named Primary.
NodeRedundancyState.IsRoleLeaderForDriver is renamed IsDriverPrimary, and
NodeHealthInputs.IsDriverRoleLeader likewise. Keeping the old names would have
left the wire contract asserting a derivation the code no longer uses — the same
drift that made this defect invisible.
Proven by a real two-node cluster rather than a mock. RedundancyPrimaryElectionTests
binds the first-joining node to the HIGHER port, so oldest and lowest-address name
different nodes, and includes a fixture assertion that the divergence actually
occurred — without it the real assertion could pass for the wrong reason. Positive
control: restoring the RoleLeader derivation turns exactly the two election tests
red while the fixture check stays green.
Runtime.Tests 440 passed, ControlPlane.Tests 82, Cluster.Tests 36.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Ports the sister project's live-proven fix (ScadaBridge cf3bd52f). OtOpcUa ran
the identical configuration it indicts: keep-oldest with down-if-alone = on.
Akka.NET 1.5.62's KeepOldest.OldestDecision only lets the down-if-alone branch
rescue a side holding >= 2 members. A two-node cluster losing a peer is always a
1-vs-1 split, so the branch never fires, the lone survivor falls through to
DownReachable and downs ITSELF, and run-coordinated-shutdown-when-down then
terminates it. down-if-alone is a 3+-node feature and does not do what its name
suggests for a pair: a crash of the oldest node is a total outage, which is the
exact failure the redundancy pair exists to absorb.
Cluster:SplitBrainResolverStrategy now selects the provider, defaulting to
auto-down: Akka's AutoDowning with auto-down-unreachable-after = 15s, so the
leader among the reachable members downs the unreachable peer and a crash of
either node fails over in place. keep-oldest remains available for deployments
that would rather take an outage than ever run dual-active during a real
partition. An unrecognised value fails the host at startup rather than falling
through to the fatal default.
The tests assert the EFFECTIVE configuration — they start a real host through
WithOtOpcUaClusterBootstrap and read akka.cluster.downing-provider-class back off
the running ActorSystem. This is not incidental. The first draft inlined the
three calls the bootstrap makes instead of calling it, which pinned the test's
own wiring: sabotaging production's HOCON precedence left all 36 green. The
prior file had the same shape at a smaller scale, asserting only that
BuildClusterOptions returned a KeepOldestOption — true, and true of a
configuration that cannot fail over. Positive control: making BuildDowningHocon
emit nothing for auto-down turns exactly the two effective-config guards red.
Measured while verifying rather than assumed: HoconAddMode.Append also wins here,
purely because it is added last, so the mode name is not the guarantee. The
comment now says so instead of asserting a precedence rule that does not hold.
Not yet live-drilled on OtOpcUa. The docker-dev rig is a single six-node mesh
where the 1-vs-1 pathology cannot occur; the kill-the-oldest drill belongs with
the per-cluster mesh work that makes every mesh exactly two nodes (design doc
6.2 / Phase 0a). Recorded as an outstanding gate in docs/Redundancy.md.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Akka.Cluster.Hosting's WithClustering enables an SBR downing provider by default
(applies SplitBrainResolverOption.Default when ClusterOptions.SplitBrainResolver is
null), which reads the pre-existing akka.conf keep-oldest block. So the cluster was
NOT running NoDowning before Critical 1 and hard-crash failover already worked — the
typed KeepOldestOption is reinforcing/explicit-in-code, not the sole activator.
Corrects the inaccurate 'HOCON inert / NoDowning / never fails over' framing in:
- ServiceCollectionExtensions.BuildClusterOptions XML comment
- akka.conf split-brain-resolver comment
- docs/Redundancy.md Split-brain section
- SplitBrainResolverActivationTests summary + assertion message (+ method rename)
No code revert (the typed option is correct belt-and-suspenders). Cluster.Tests 29/29.
Surfaced by the #9 hard-kill failover negative control.
The split-brain-resolver HOCON block in akka.conf was inert — nothing
registered a downing provider, so the cluster ran Akka's default NoDowning:
a hard-crashed node was never downed, cluster singletons + the driver
role-leader never failed over, and a partition left both redundancy sides
at ServiceLevel 240 indefinitely.
Activate the resolver via the typed ClusterOptions.SplitBrainResolver
(KeepOldestOption { DownIfAlone = true }) in a new testable
BuildClusterOptions helper. keep-oldest + down-if-alone is correct for a
2-node warm-redundancy pair. HOCON keeps stable-after (typed option can't
express it) + a cross-reference comment. Deterministic unit guard added;
end-to-end verified the resolved ActorSystem config carries the provider
and stable-after survives the HOCON/Hosting merge. Redundancy.md corrected.
Hard-kill failover integration test tracked as a follow-up (effort-M harness).
Adds <summary>/<param>/<returns>/<inheritdoc> where missing and removes
project bookkeeping IDs (task/tracking refs) from shipped code comments,
so the docs read cleanly and CommentChecker is quiet except for known
false positives (PLC/protocol terms, event/IEqualityComparer inheritdoc).
Doc/comment-only; no logic changed; solution builds clean.
DriverHostActor/DriverInstanceActor and cluster events log via Akka's ILoggingAdapter, which had no Serilog bridge — under the Windows service host the default StandardOutLogger output is discarded, making the driver-role actor graph invisible (this masked the data-plane diagnosis).
- Add Akka.Logger.Serilog 1.5.60 (deps satisfied by Akka 1.5.62 / Serilog 4.3.1).
- WithOtOpcUaClusterBootstrap: ConfigureLoggers(DebugLevel; ClearLoggers(); AddLogger<SerilogLogger>()) — Akka.Hosting owns logger setup, so HOCON akka.loggers alone is not honored.
- Program.cs: set static Serilog.Log.Logger from the DI root logger after Build() (AddZbSerilog registers the MEL provider but not the static logger, which the Akka SerilogLogger and the startup banner both need).
Code review at HEAD 7286d320. Cluster-001 (SeedFromCurrentState reads from one
snapshot), Cluster-003 (HoconLoader double-dispose), Cluster-004 (stale akka.conf
header), Cluster-005 (ServiceLevelCalculator tests added to Cluster.Tests). Cluster-002
deferred (no production caller).
All five suppressed advisories are now resolved at baseline/resolved versions,
so every NuGetAuditSuppress is removed repo-wide:
- System.Security.Cryptography.Xml (GHSA-37gx-xxp4-5rgx / GHSA-w3x6-4m5h-cxqf)
-> fixed by the .NET 10 baseline (10.0.6)
- OPCFoundation Opc.Ua.Core (GHSA-h958-fxgg-g7w3) -> fixed at resolved 1.5.378.106
Two were still live and are now patched via direct security pins:
- OpenTelemetry.Api 1.9.0 -> 1.15.3 (GHSA-g94r-2vxg-569j) pinned in Cluster;
Runtime/ControlPlane/AdminUI + tests inherit via project reference
- Tmds.DBus.Protocol 0.20.0 -> 0.21.3 (GHSA-xrw6-gwf8-vvr9) pinned in Client.UI
Also correct the Historian sidecar runtime comments (x86 -> x64, matching the
csproj PlatformTarget). Solution audit: 0 vulnerable packages; full build clean.
Adds <summary>, <param>, <typeparam>, and <inheritdoc/> tags to public
members surfaced by commentchecker — resolves 5,847 of 5,869 issues
(99.6%) across three /fixdocs passes.
DeployHappyPathTests exercises the full deploy pipeline on the 2-node harness:
AdminOperationsActor → ConfigPublishCoordinator → DistributedPubSub →
DriverHostActor on both nodes → ApplyAck → coordinator seals. Verifies both
NodeDeploymentState rows reach Applied and Deployment.Status reaches Sealed.
Exposed + fixed two production bugs along the way:
1. Coordinator was publishing DispatchDeployment on the "deployments" topic but
never subscribed to anything — DriverHostActor ACKs published on the same
topic could not reach it. Added dedicated "deployment-acks" topic with
coordinator subscription in PreStart, and DriverHostActor publishes ACKs
there.
2. NodeId derivation used member.Address.Host only — two cluster members on a
shared loopback host (test harness, dev VMs) collided to one identity. The
coordinator's expected-ack set became {1} and the system sealed after only
half the nodes acked. Switched to host:port everywhere (ClusterRoleInfo +
coordinator) so loopback nodes stay distinct and production identities are
harmlessly more specific.
Tests: 95 v2 tests pass (was 93 + 2 deploy tests), 0 skipped.
Failover scenarios (design §8 cases 3-7: node-kill-mid-apply, split-brain,
restart-during-deploy) deferred — they need controlled node-down primitives
on the harness. Tracked as F22 (failover scenario test cases).
Builds TwoNodeClusterHarness: two in-process Host-equivalent nodes sharing
an in-memory ConfigDb. Forms a 2-member Akka cluster. ClusterFormationTests
proves both nodes see each other as admin+driver role members.
Fixes a real production bug uncovered while wiring the harness — Program.cs
ran two separate ActorSystems (one from AddOtOpcUaCluster.AkkaHostedService
with cluster HOCON, one from Akka.Hosting.AddAkka with bare HOCON). Cluster
singletons landed on the bare ActorSystem and could not actually form a
cluster ("Configuration does not contain `akka.cluster` node").
Consolidation:
- AddOtOpcUaCluster now only binds AkkaClusterOptions + registers IClusterRoleInfo
- New WithOtOpcUaClusterBootstrap pushes embedded HOCON + Remote/Cluster options
into Akka.Hosting's AkkaConfigurationBuilder
- AkkaHostedService.cs deleted — Akka.Hosting now owns the lifecycle
- Program.cs + harness call WithOtOpcUaClusterBootstrap inside AddAkka
Why not WebApplicationFactory<Program>? Program.cs reads OTOPCUA_ROLES from
process env (shared across in-process WAFs); the harness replays Program.cs's
DI graph from a clean WebApplicationBuilder per node with per-node config
overrides. Same production extensions, isolated config + Kestrel + Akka ports.
Tests: 93 v2 tests pass (was 91 + 2 new cluster formation), 0 skipped.
Adds the empty project skeletons that subsequent v2 tasks fill in:
src/Core/ZB.MOM.WW.OtOpcUa.Commons (types, interfaces, message contracts)
src/Core/ZB.MOM.WW.OtOpcUa.Cluster (Akka.Hosting + cluster wiring)
src/Server/ZB.MOM.WW.OtOpcUa.Security (cookie+JWT auth, LDAP)
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane (admin-role cluster singletons)
src/Server/ZB.MOM.WW.OtOpcUa.Runtime (per-node driver actors)
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer (OPC UA SDK application host)
src/Server/ZB.MOM.WW.OtOpcUa.AdminUI (Razor class library)
src/Server/ZB.MOM.WW.OtOpcUa.Host (single fused web binary)
Each project sets TreatWarningsAsErrors=true in its own csproj (per the
Directory.Build.props deviation note in the previous commit). NuGetAuditSuppress
entries cover transitive vulnerability advisories the new strictness surfaces:
- GHSA-g94r-2vxg-569j (OpenTelemetry.Api 1.9.0 via Akka.Cluster.Hosting/Tools)
- GHSA-h958-fxgg-g7w3 (Opc.Ua.Core 1.5.374.126 via OpcUaServer)
- GHSA-37gx-xxp4-5rgx + GHSA-w3x6-4m5h-cxqf (legacy advisories already accepted)
OpcUaServer pins OPCFoundation.NetStandard.Opc.Ua.Configuration to 1.5.374.126
via VersionOverride to match Opc.Ua.Server's transitive Opc.Ua.Core (same
constraint as the legacy Server project).
Runtime does NOT project-reference any concrete Driver.* assemblies; drivers
load reflectively at runtime (Phase 6). Runtime gets the IDriver contract
through Core.Abstractions instead.
Host's Microsoft.Extensions.Hosting.WindowsServices is conditional on the
Windows OS so the project builds on macOS dev machines.
Build verification: dotnet build -> 438 warnings (all pre-existing xUnit1051
in legacy Server.Tests/Admin.Tests), 0 errors. Closes Task 9 (build green
smoke check, no separate commit).
Joseph Doherty