ClusterPrimaryHealthCheck was written three days' worth of debugging ago because
the shared ActiveNodeHealthCheck selected by RoleLeader and reported Healthy for
any node lacking the role. Health 0.3.0 fixes the shared check — oldest Up
member, role-preference scoping, Unhealthy when the node owns no active work —
so the private copy is now duplication rather than a workaround, and it is
deleted.
RolePreference [admin, driver] reproduces exactly what it did: a fused central
node answers for admin (where the singletons and the AdminUI are pinned), a
driver-only site node answers for driver.
SelectOldestUpMemberOfRole now delegates to the shared ClusterActiveNode instead
of re-implementing the age ordering. This is the point of the change rather than
a tidy-up: the redundancy snapshot drives the OPC UA ServiceLevel 250/240 split
while the health tier drives Traefik's admin routing, so two copies of "oldest Up
member of a role" would be two chances to advertise one node as authoritative
while gating the data plane on another. ControlPlane takes a ZB.MOM.WW.Health.Akka
reference for it; the layering trade-off is recorded at the PackageReference.
Behaviour note: a node carrying neither admin nor driver now answers 503 on the
active tier instead of 200. That case is reachable — RoleParser admits dev-only
and cluster-role-only nodes — and 503 is correct, since such a node owns no
active work and must not be in an active-tier pool. No docker-dev node is
affected: every rig node is admin+driver or driver-only.
Tests replaced in kind. The rule itself is now pinned in the library against a
real two-node cluster; what stays OtOpcUa's own decision is the wiring, so
ActiveTierRegistrationTests pins which check is on the active tag, that it is
registered on driver-only nodes too, and that it is not on the ready tier.
Verified: Host.Tests 25/25, ControlPlane redundancy 15/15.
Closes the remaining half of #494, and corrects the half fixed in 8dd9da7d.
The shared ActiveNodeHealthCheck(role: "admin") answered the wrong question twice
over. It returns Healthy for any node LACKING the role, so all four driver-only
site nodes called themselves active and no consumer could find the Primary of a
site Cluster. And it selects by RoleLeader - the lowest-ADDRESSED member - which
is not where Akka places singletons.
Replaced with ClusterPrimaryHealthCheck ("cluster-primary"). One rule: this node
is active iff it is the OLDEST Up member carrying its own active role, where the
active role is admin when the node has it and driver otherwise. That serves both
consumers correctly - a fused admin node answers for admin, so Traefik pins the
AdminUI to the node hosting the singletons; a driver-only site node answers for
driver, which is exactly SelectDriverPrimary, the same election behind
IsDriverPrimary and the OPC UA ServiceLevel 250/240 split. Per-mesh scoping is
free after Phase 6: ClusterState.Members already contains only this node's own
Cluster.
SelectDriverPrimary is generalised to SelectOldestUpMemberOfRole so the
age-ordering rule has one implementation. Two copies of "oldest Up member of a
role" would be two chances to silently disagree about who is in charge, and the
tier and the redundancy snapshot must never disagree.
THE ROLE-LEADER HALF MATTERED IN PRACTICE, not just in theory. On the rebuilt rig
the two orderings diverge right now:
akka leader (lowest address) = central-1
oldest admin member = central-2 <- hosts the singletons
8dd9da7d made the tier a real 503 but still selected by RoleLeader, so it would
have pinned Traefik to central-1 - the node NOT hosting the work. With this change
Traefik correctly routes to central-2.
Live-verified, exactly one 200 per mesh:
MAIN central-2 200 central-1 503 traefik: central-2 UP, central-1 DOWN
SITE-A site-a-1 200 site-a-2 503
SITE-B site-b-1 200 site-b-2 503
Tests: role-selection matrix and the startup-safe Degraded path in Host.Tests
(26 pass); parity between the tier's selector and the redundancy snapshot's, plus
role scoping, added to RedundancyPrimaryElectionTests, which forms a real two-node
cluster deliberately built so the oldest member is not the lowest-addressed one
(5 pass).
Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the
same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for
the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6
live gate reproduced this reliably on docker). The prior mitigation was operational
(staggered start / compose depends_on), which does not exist on production hardware.
The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split
without giving up cold-start-alone:
- The lower-address node is the preferred founder: self-first, forms immediately.
- The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds:
reachable => peer-first (join the founder, never race it); unreachable => self-first
(partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes,
from an explicit reachability signal — never re-formed mid-handshake (the retired
SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions)
and ClusterBootstrapCoordinator drives the join.
Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would
reopen the split); fail-fast validation of the timing knobs; the residual "founder dies
in the probe->join window" hang is documented and made operator-visible (warning + a
restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node
cold-start-alone case; 147/147 Cluster tests pass.
Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on;
site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous
start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split;
higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 wired the re-homed redundancy singleton in Program.cs as
`WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>())`
inside the AddAkka configurator lambda. That lambda runs WHILE the ActorSystem is
being built, but ClusterRoleInfo depends on the ActorSystem (it is a live
Cluster.State view) — so resolving it there recurses into ActorSystem construction
and every node dies at boot with StackOverflowException. It compiles cleanly (a
runtime DI cycle) and RedundancyStateSingletonRehomeTests passed a hand-built
FakeClusterRoleInfo straight into the extension, mocking around the composition-root
line that overflows. The docker-dev live gate caught it on first boot.
Derive the singleton's cluster-role scope from AkkaClusterOptions (pure config, no
ActorSystem) instead: BuildClusterRedundancySingletonOptions / the extension now take
AkkaClusterOptions and compute Role = Roles.FirstOrDefault(IsClusterRole) ?? driver —
identical to ClusterRoleInfo.ClusterRole, which derives from the same config. Program.cs
passes IOptions<AkkaClusterOptions>.Value, which has no ActorSystem dependency.
Live-verified: rebuilt image boots with zero stack-overflow lines, cluster singletons
form per mesh. 5/5 rehome tests pass; Host builds clean.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Code review found SplitTopologyTransportValidator read the wrong role view,
defeating its fail-loud purpose two ways:
(a) Roles-source divergence. AkkaClusterOptions.Roles binds Cluster:Roles
but falls back to OTOPCUA_ROLES (RoleParser.Parse) when it is empty. A
node configured with only OTOPCUA_ROLES=driver,cluster-SITE-A genuinely
carries the cluster role in Akka, yet the validator saw no cluster role
and passed silently on MeshTransport:Mode=Dps (the documented incident).
(b) Case. The Cluster:Roles bind path is verbatim while RoleParser.Parse
lowercases the OTOPCUA_ROLES path; 'Cluster-SITE-A' / 'Driver' / 'Admin'
slipped past the ordinal checks.
Resolve effective roles the same way the node does — Cluster:Roles, else
RoleParser.Parse(env OTOPCUA_ROLES) exactly as Program.cs — then normalize
both (trim + ToLowerInvariant) before IsClusterRole/driver/admin. The message
still names the ClusterId in the operator's original spelling. Exemption and
all other behaviour unchanged. Contained to the validator.
Tests: OTOPCUA_ROLES-only cluster role on Dps fails; mixed-case roles on Dps
fail; cluster role with neither admin nor driver needs only ClusterClient;
Cluster:Roles wins over a stray OTOPCUA_ROLES. 16/16 green (124/124 project).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Per-cluster mesh Phase 6 (Task 5). A node carrying a cluster-{ClusterId}
role (RoleParser.IsClusterRole) is in the split topology, where the
Phase 2/3/5 DPS dark-switch branches deliver nothing across the mesh
boundary. SplitTopologyTransportValidator fails host start unless such a
node uses the mesh-crossing transports: MeshTransport:Mode=ClusterClient
always; Telemetry:Mode=Grpc if it has the driver role; TelemetryDial:Mode
=Grpc if it has the admin role. It is a no-op for any node with no
cluster-role (legacy / single-mesh / test). Roles + the three modes are
cross-read from IConfiguration, mirroring ConfigSourceOptionsValidator.
Registered via AddValidatedOptions on MeshTransportOptions with
ValidateOnStart, alongside the sibling validators.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Task 3 review: scoping only the rows/enabled DB queries by ClusterId left
the observed set (from Cluster.State.Members) filtered by the driver role
alone. In the mixed-mesh window — DB scoping live via a cluster role, but
the physical Akka mesh not yet split so central still sees foreign members
via gossip — a foreign-cluster driver member stays in observed while its row
is filtered out, hitting the RunningNodeHasNoRow branch, which logs at ERROR.
That traded the Warning-level false positive for a worse Error-level one.
Filter observed by the node's own cluster ROLE too. Extracted the projection
into a static FilterOwnClusterDriverMembers(members, ownClusterRole) so the
membership filter is unit-testable without seeding a live cluster (the actor
harness joins as admin only, so observed is always empty). Sourced at
registration from IClusterRoleInfo.ClusterRole alongside ClusterId. Null role
(legacy, no cluster role) keeps every driver member — reconcile-all, unchanged.
Also clarified the ownClusterId doc wording (non-null-and-non-empty).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Review follow-ups on the redundancy re-home (Phase 6, Task 2):
- Program.cs: the hasDriver comment claimed IClusterRoleInfo "throws at host
start if this driver node carries no cluster role" — false since the fallback
landed. Rewrite it to describe the cluster-{ClusterId} scope with the driver-role
fallback for legacy single-mesh / not-yet-migrated nodes.
- ClusterRoleInfo.cs: the SubscriberActor comment described RedundancyStateActor as
the "admin-role singleton" — stale. Note it is now a cluster-{ClusterId}-scoped
singleton spawned on every driver node (LeaderChanged stays a no-op here).
- Add a host/registry-level test for the driver-role FALLBACK path (roles ["driver"],
no cluster role) closing the coverage asymmetry — it was proven only in the pure
helper. Asserts the node boots and registers RedundancyStateActorKey, i.e. the boot
the earlier throw-based version would have aborted.
The boot helper now supplies an in-memory IDbContextFactory and a fake IClusterRoleInfo:
Akka.Hosting invokes singleton props factories eagerly at StartAsync, and the admin
ClusterNodeAddressReconciler factory reads both (the latter for its per-cluster reconcile
scope, added by a sibling Phase 6 task) — without them the admin boot NREs.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
After the fleet splits into one Akka mesh per application Cluster, a
ClusterClientReceptionist serves only its own mesh — so the single
fleet-wide client's SendToAll reached only the mesh whose receptionist
answered, leaving every other cluster silently on its old configuration.
Central now holds one ClusterClient per ClusterId and fans SendToAll
across all of them. LoadContactsFromDb selects ClusterId and groups the
receptionist contacts per cluster (keeping the per-row TryParse guard and
the enabled/non-maintenance filter); ContactsLoaded carries
ContactsByCluster; HandleContactsLoaded diffs per cluster (rebuild changed,
stop+drop vanished, warn-don't-create on empty); RebuildClient builds a
per-cluster client under a clusterId-derived actor name. IMeshClusterClientFactory.Create
gained a clusterId parameter so the per-cluster clients get distinct,
diagnosable names. ApplyAck handling and the DPS branch are unchanged; the
deploy path stays payload-free.
Tests: focused unit coverage of the grouping + fan-out (one-client-per-cluster,
fan-out SendToAll to every cluster client) via the recording factory double,
plus the real two-mesh boundary test extended to central + two separate site
meshes proving one dispatch reaches both and a client scoped to one cluster
never crosses into another. Red-before-green verified: crippling the fan-out
to a single client fails the reaches-both-meshes assertion.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 splits the fleet into one mesh per Cluster, so an admin (central)
node's Cluster.State.Members shows only its own pair — it no longer sees
site members via gossip. The ClusterNodeAddressReconciler singleton compared
live membership against EVERY ClusterNode row fleet-wide, so post-split every
foreign-cluster row would log EnabledRowNotInCluster forever.
Scope the actor's two DB queries to rows whose ClusterId matches the admin
node's own (IClusterRoleInfo.ClusterId, sourced at registration). A legacy
admin node with no cluster role (null ClusterId) still reconciles the whole
fleet — it genuinely sees every member via gossip. The pure Reconcile
function and AddressMismatchKind semantics are unchanged; scoping lives
entirely in the actor's queries.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 splits the single fleet-wide Akka mesh into one 2-node mesh per
application Cluster. RedundancyStateActor was an admin-scoped cluster singleton
that elected ONE fleet-wide driver Primary — but after the split a driver-only
site pair has no admin node in its mesh to host it, so its Primary would never
be elected.
Re-home it: remove the redundancy WithSingleton block from the admin
WithOtOpcUaControlPlaneSingletons set, and add WithOtOpcUaClusterRedundancySingleton,
a singleton spawned from Program.cs's hasDriver branch on EVERY driver node (the
fused central included). It scopes to the node's own cluster-{ClusterId} role when
present, and falls back to the fixed `driver` role otherwise. Each mesh then has
exactly one, electing its own pair-local Primary and publishing redundancy-state
on its own mesh's DistributedPubSub. The election logic (oldest Up driver member)
and the DPS publish are unchanged — they become pair-local after the split.
The driver-role fallback (Decision 2) deliberately does NOT throw when a node has
no cluster role: that is the pre-Phase-6 fleet-wide behavior on a legacy single
mesh, so legacy/harness (TwoNodeClusterHarness: admin,driver) and not-yet-migrated
deployments keep booting unchanged. On a genuinely split 2-node mesh the `driver`
role is already pair-local (the mesh IS the pair), so the fallback is correct in
both worlds. Cluster-scope, when present, additionally survives an accidental
two-mesh merge by keeping one singleton per cluster.
The role scope + driver-role fallback are pinned by unit tests against the
extracted BuildClusterRedundancySingletonOptions helper (the BuildDowningHocon
pattern); admin-removal + driver-registration are pinned behaviorally against a
booted node's ActorRegistry.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 Task 1. IClusterRoleInfo gains ClusterRole/ClusterId, derived from
the node's configured AkkaClusterOptions.Roles at construction time (not
live Cluster.State) so the identity is available before the cluster forms.
First cluster-scoped role wins when more than one is configured, logged as
a Warning. Updates the FakeClusterRoleInfo test double in
ServiceCollectionExtensionsTests to satisfy the new interface members.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 6 (per-cluster mesh split) needs nodes to carry a cluster-scoped
role like cluster-SITE-A; RoleParser previously rejected anything
outside the fixed admin/driver/dev set. Adds IsClusterRole/
ClusterIdFromRole plus well-known-role constants, and points the three
duplicate "driver" string literals (RedundancyStateActor,
ClusterNodeAddressReconcilerActor, ServiceCollectionExtensions) at the
new RoleParser.Driver so the value has one source, without renaming
any existing DriverRole symbol.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Closes two test-completeness gaps flagged in review of the seam-tap commit — the
ScriptedAlarmHostActor and VirtualTagActor hub emits had no test proving the hub
actually receives them.
- ScriptedAlarm_activation_emits_one_Alarm_item_with_matching_payload: spawns the
host with a capturing fake hub, drives a real engine Inactive->Active transition
through the existing ScriptedAlarms harness, and asserts exactly one
TelemetryItem.Alarm carrying the Activated transition.
- VirtualTag_script_log_emits_one_Script_item_with_same_payload: passes a fake hub
via Props + a publisherFactory, drives an evaluator failure, and asserts a
TelemetryItem.Script carrying the SAME ScriptLogEntry instance handed to the
publisher.
To make the VirtualTagActor tap observable, its hub emit moved to the TOP of
PublishLog, before the test-only publisherFactory early-return branch, so the emit
fires on BOTH the production DPS path and the factory path. Production behavior is
unchanged: production passes publisherFactory: null, so it still reaches the DPS
Tell; the emit just moved a couple lines earlier (both are fire-and-forget).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Each of the four telemetry publish seams now also emits its DPS payload into the
node-local ITelemetryLocalHub (Phase 5), leaving the existing DistributedPubSub
publish untouched:
- driver-health AkkaDriverHealthPublisher -> TelemetryItem.Health
- resilience DriverResilienceStatusPublisherService -> TelemetryItem.Resilience
- alerts ScriptedAlarmHostActor + DriverHostActor (native) -> TelemetryItem.Alarm
- script-logs VirtualTagActor + DpsScriptLogPublisher -> TelemetryItem.Script
DI services take the hub as a required ctor param; actors take an optional
nullable hub threaded through their Props and the real spawn sites
(DriverHostActor <- ServiceCollectionExtensions; VirtualTagActor/ScriptedAlarmHostActor
<- DriverHostActor), so existing test Props constructions keep compiling and simply
do not emit. The hub is a no-op until a gRPC client subscribes, so the emit is safe
on every node.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Scripted-alarm condition state lives in LocalDb (Phase 4); the ConfigDb-backed
Ef store + ScriptedAlarmState table are now dead. Removes the test-harness Ef
fallback (a DB-backed node with no store now skips the alarm host), deletes the
store + entity + model config, and drops the table via migration.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
A driver-only node has a null dbFactory but a wired applier; the old combined
guard routed it to the raw-sink fallback that wipes the address space and never
re-materialises. Split the guard so in-hand artifact bytes drive the real
diff-and-apply; a missing artifact abandons the rebuild (keep last-known-good,
#485) instead of wiping.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Nullable ConfigDb factory; UpsertNodeDeploymentState no-ops when absent
(central persists acks from the ApplyAck); scripted-alarm condition state
served from the LocalDb store instead of the ConfigDb-backed Ef store.
Combines Phase-4 Tasks 4 + 6. Removes the interim dbFactory! cast.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty