7 Commits

Author	SHA1	Message	Date
Joseph Doherty	06c56d7719	fix(cluster,acl): OTOPCUA_ROLES fallback + one string-identity rule (#532, #533) ... Two independent defects surfaced in review of #520 stage 3. #532 — Cluster:Roles had no OTOPCUA_ROLES fallback, despite two doc sites promising one. A node configured only through the env var — which drives ALL of Program.cs's conditional wiring, so it reads as the authoritative knob — bound an empty Roles array and joined the cluster carrying no Akka roles at all. It ran, served, and logged nothing wrong, while central reported "no driver-role nodes are Up" and every deployment to it timed out. Not hypothetical: the field fix on wonder-app-vd03 was to hand-add Cluster__Roles__0/1 to a Windows service's registry environment. Anyone reading the code to diagnose that box would have concluded from the XML doc that the fallback was already applied. AddOtOpcUaCluster now post-configures AkkaClusterOptions through the shared RoleParser.ResolveEffectiveRoles, so BuildClusterOptions (Akka's own member roles), ClusterRoleInfo, the ControlPlane singleton scope and ConfigDbApiKeyStore's ClusterId all see one answer. Two sibling validators (ConfigSource, Telemetry) read Cluster:Roles directly and were exempting the exact node they exist for — an env-only driver-only node has no ConfigDb and could sit on ConfigSource:Mode=Direct unchallenged. They resolve the same way now. Also collapsed the four open-coded `Roles.FirstOrDefault(IsClusterRole)` + ClusterIdFromRole copies into RoleParser.ClusterIdFromRoles. They had drifted: three classified the cluster role ordinally, so a verbatim-bound `Cluster-SITE-A` yielded a null ClusterId — and a null ClusterId applies NO cluster filter, which made #528's cross-cluster over-grant guard fail OPEN. IsClusterRole is now case-insensitive; the ClusterId suffix keeps its authored spelling, because normalizing it would silently rename the Akka role of a running deployment. #533 — three string-identity rules in one authorization path. AclScopeRef used default (ordinal) equality while the trie builder filtered ClusterId and hashed principals OrdinalIgnoreCase; the deploy gate's cross-cluster check was ordinal too, which was the worse half — it could reject a whole deploy over a ClusterId case difference the runtime would have accepted. It only worked because resolver and builder were handed the same NodeAcl instances. The rule is now stated once, on AclScopeRef, and applied throughout: identifiers (ScopeId, ClusterId, Principal) compare OrdinalIgnoreCase; path segments compare Ordinal. AclScopePathResolver re-keys its own maps rather than documenting a comparer contract for callers, so the two independent producers — EF at the gate, artifact JSON on the node — cannot diverge from the outside. PermissionTrieNode.Children stays Ordinal: those are browse names, and Speed / speed are two nodes that can coexist under one device. Split PermissionTrieBuildDiagnosticReason.RetiredScopeKind out of UnresolvedScope. A legacy FolderSegment row and a typo'd id had the same message and different remedies. Falsified: five mutations, each reddening only its own guard — remove the PostConfigure (3 tests), revert IsClusterRole to ordinal (3), revert AclScopeRef equality (1), revert the resolver's map comparer (6), revert the gate's comparisons (2), remove the FolderSegment short-circuit (1). Test isolation: OTOPCUA_ROLES is process-global and xUnit runs classes in parallel, so the three validator suites that now resolve effective roles share a non-parallel collection. Without it, cases passing no roles would intermittently observe another class's env var. Full solution build clean; full suite green apart from two pre-existing failures confirmed identical on 3a6b2a1d (PrimaryGateFailoverTests, and DriverProbeRegistrationTests expecting 10 probe types where 12 now register) plus the AbLegacy integration suite, which needs a docker fixture. Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo	2026-08-01 01:51:22 -04:00
Joseph Doherty	279d1d0fb1	feat(cluster): simultaneous-cold-start split-brain guard (opt-in) ... Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6 live gate reproduced this reliably on docker). The prior mitigation was operational (staggered start / compose depends_on), which does not exist on production hardware. The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split without giving up cold-start-alone: - The lower-address node is the preferred founder: self-first, forms immediately. - The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds: reachable => peer-first (join the founder, never race it); unreachable => self-first (partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes, from an explicit reachability signal — never re-formed mid-handshake (the retired SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions) and ClusterBootstrapCoordinator drives the join. Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would reopen the split); fail-fast validation of the timing knobs; the residual "founder dies in the probe->join window" hang is documented and made operator-visible (warning + a restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node cold-start-alone case; 147/147 Cluster tests pass. Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on; site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split; higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW	2026-07-24 08:34:29 -04:00
Joseph Doherty	3f24d4d6bf	feat(cluster)!: self-first seed ordering replaces the InitJoin self-form watchdog ... Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the node's own address. Every other node runs JoinSeedNodeProcess and retries InitJoin forever. That is why "both peers are seeds" never meant "either can cold-start alone", and it is fixed here the way Akka itself intends: each seed node lists ITSELF first. - docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already self-first). The site nodes are untouched -- they seed off central-1 only and are deliberately not seeds. - AkkaClusterOptionsValidator enforces the invariant at boot via the shared ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it would refuse to boot every driver-only site node. Identity is compared on PublicHostname (falling back to Hostname when blank) AND port, i.e. the address Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed anywhere and leave the rule silently inert on the whole docker-dev rig. - ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat outside Akka's join handshake, so it could not distinguish "no seed answered" from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress) is not ignored mid-handshake, it wins. The live gate (ea45ace1) caught it islanding a node that a manual failover had bounced: InitJoinAck received, no Welcome inside the window because the peer's ring still held the old incarnation, watchdog fired, second cluster. The TCP reachability guard patched that one shape; the race stayed. It was also inert for every non-seed node, so after this change it can never usefully fire. - SelfFormBootstrapTests -> SelfFirstSeedBootstrapTests: real in-process clusters through the production bootstrap at production failure-detection timings, incl. a falsifiability control proving the OLD peer-first ordering never forms (that test failed at 11s against the watchdog, which is what proved the retirement), the restart-into-a-live-peer case that killed the watchdog, and a simultaneous cold start converging on ONE cluster. Mirrors ScadaBridge 4a6341d8; anticipates per-cluster-mesh-program Phase 6, which had this retirement queued.	2026-07-22 07:42:06 -04:00
Joseph Doherty	979dc102dd	feat(cluster): SelfFormAfter option (bootstrap self-form fallback window) ... Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW	2026-07-22 06:05:34 -04:00
Joseph Doherty	2964361a6f	fix(cluster): auto-down downing strategy — a two-node pair could not survive an oldest-node crash ... Ports the sister project's live-proven fix (ScadaBridge cf3bd52f). OtOpcUa ran the identical configuration it indicts: keep-oldest with down-if-alone = on. Akka.NET 1.5.62's KeepOldest.OldestDecision only lets the down-if-alone branch rescue a side holding >= 2 members. A two-node cluster losing a peer is always a 1-vs-1 split, so the branch never fires, the lone survivor falls through to DownReachable and downs ITSELF, and run-coordinated-shutdown-when-down then terminates it. down-if-alone is a 3+-node feature and does not do what its name suggests for a pair: a crash of the oldest node is a total outage, which is the exact failure the redundancy pair exists to absorb. Cluster:SplitBrainResolverStrategy now selects the provider, defaulting to auto-down: Akka's AutoDowning with auto-down-unreachable-after = 15s, so the leader among the reachable members downs the unreachable peer and a crash of either node fails over in place. keep-oldest remains available for deployments that would rather take an outage than ever run dual-active during a real partition. An unrecognised value fails the host at startup rather than falling through to the fatal default. The tests assert the EFFECTIVE configuration — they start a real host through WithOtOpcUaClusterBootstrap and read akka.cluster.downing-provider-class back off the running ActorSystem. This is not incidental. The first draft inlined the three calls the bootstrap makes instead of calling it, which pinned the test's own wiring: sabotaging production's HOCON precedence left all 36 green. The prior file had the same shape at a smaller scale, asserting only that BuildClusterOptions returned a KeepOldestOption — true, and true of a configuration that cannot fail over. Positive control: making BuildDowningHocon emit nothing for auto-down turns exactly the two effective-config guards red. Measured while verifying rather than assumed: HoconAddMode.Append also wins here, purely because it is added last, so the mode name is not the guarantee. The comment now says so instead of asserting a precedence rule that does not hold. Not yet live-drilled on OtOpcUa. The docker-dev rig is a single six-node mesh where the 1-vs-1 pathology cannot occur; the kill-the-oldest drill belongs with the per-cluster mesh work that makes every mesh exactly two nodes (design doc 6.2 / Phase 0a). Recorded as an outstanding gate in docs/Redundancy.md. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW	2026-07-21 18:32:42 -04:00
Joseph Doherty	64e3fbe035	docs: backfill XML documentation across 756 files ... Adds <summary>, <param>, <typeparam>, and <inheritdoc/> tags to public members surfaced by commentchecker — resolves 5,847 of 5,869 issues (99.6%) across three /fixdocs passes.	2026-05-28 08:10:17 -04:00
Joseph Doherty	f184f8ed1b	feat(cluster): AkkaHostedService and DI extension	2026-05-26 04:31:05 -04:00