lmxopcua/docs/plans/2026-07-24-mesh-phase6-live-gate.md

Per-cluster mesh Phase 6 — live gate record

Exit gate for Phase 6 (mesh partition). Bring up the rewritten docker-dev rig (three independent 2-node Akka meshes) against merged master and prove every previously-live-gated behaviour per pair, plus the split itself.

Merge landed first (origin/master @ 2839deb5, "merge now, gate after"); this gate runs against the merged code and fixes forward on master if anything surfaces.

Rig topology (docker-dev, local OrbStack):

Mesh	Nodes	Roles	OPC UA (host)	Seeds (self-first, pair-local)
MAIN	central-1, central-2	admin,driver,cluster-MAIN	4840 / 4841	own pair only
SITE-A	site-a-1, site-a-2	driver,cluster-SITE-A	4842 / 4843	own pair only
SITE-B	site-b-1, site-b-2	driver,cluster-SITE-B	4844 / 4845	own pair only

• Akka port 4053 (per container); telemetry gRPC :4056; ConfigServe :4055 (central); LocalDb sync :9001 (site-a); AdminUI http://localhost:9200 (Traefik → central-1/2).
• Transports: MeshTransport__Mode=ClusterClient (all), Telemetry__Mode=Grpc (all six), TelemetryDial__Mode=Grpc (central pair). Secrets replication pair-local.
• Deploy: POST http://localhost:9200/api/deployments, header X-Api-Key: docker-dev-deploy-key.

---

Legs

#	Leg	Status	Evidence
1	Three meshes form — each node sees ONLY its own pair (2 members)	✅ PASS	each node handshakes ONLY its pair partner
2	Two+ Primaries, one per pair; ServiceLevel 250/240 within each pair	✅ PASS	MAIN central-2=250/central-1=240; SITE-A site-a-1=250/site-a-2=240; SITE-B site-b-1=250/site-b-2=240
3	Deploy reaches all clusters over per-cluster ClusterClient; seals green	✅ PASS	"Deployment 019dbd99… sealed (acks=6)"; all 6 NodeDeploymentState rows acked; one ClusterClient per site cluster
4	Telemetry per cluster over gRPC (:4056); pills live; kill/reconnect ~5s	✅ PASS	central-1 6 outbound :4056 dials, each site node 2 inbound (both admin dial in), 0 failures/30s; site-b restart failed→recovered the streams
5	Reconciler quiet — no EnabledRowNotInCluster for foreign rows	✅ PASS	central-2 reconciler: "reconcile with cluster membership (2 driver members)" — own MAIN pair only; 0 foreign-row errors
6	Secrets pair-local — no cross-mesh replication / no unreachable-peer errors	✅ PASS (by fallback criterion)	0 secret errors fleet-wide; DPS topic rides each pair's isolated mesh (Leg 1) ⇒ structurally pair-local
7	Split validator — a site node flipped to MeshTransport__Mode=Dps refuses to boot	✅ PASS	"Cluster:Roles declares a cluster-SITE-B role … MeshTransport:Mode is 'Dps' … set MeshTransport:Mode=ClusterClient"
8	Failover within a pair — kill a Primary, Secondary takes over (auto-down 1-vs-1)	✅ PASS	killed site-a-1 (Primary); site-a-2 auto-downed it, took the singleton, 240→250, survived alone (auto-down 1-vs-1 = deferred Phase 0a gate, now proven)

Phase 6 exit gate: MET. All 8 legs pass. Three findings fixed forward on master (3a4ed8dd product boot-crash, 792f28ec rig LDAP + split-brain serialization). One test-methodology note: the Leg 7 compose run … site-b-2 throwaway shares the site-b-2 network alias and briefly disrupted the SITE-B Akka association — restart the affected pair after that probe (done), or run the validator check via a container that does not join the compose network alias.

---

Findings

FINDING 1 (fix-forward on master) — boot-crash: StackOverflowException on EVERY node

Symptom. First up of the fresh Phase-6 images: all six nodes crash at startup with Stack overflow. — an infinite recursion entering through ZB.MOM.WW.OtOpcUa.Cluster.ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap.

Root cause (Phase 6, Task 2 wiring). Program.cs resolved IClusterRoleInfo eagerly, inside the AddAkka configurator lambda:

ab.WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>());  // line 385

That lambda runs while the ActorSystem is being constructed, but ClusterRoleInfo's ctor depends on the ActorSystem (it is a live Cluster.State view). Resolving it there forces a nested ActorSystem build → re-runs the same configurator → sp.GetRequiredService<IClusterRoleInfo>() again → ∞. The cycle: WithOtOpcUaClusterBootstrap → GetService<IClusterRoleInfo> → ActorSystemFactory → WithOtOpcUaClusterBootstrap → …

Why merge + build + tests + reviews all missed it. It compiles cleanly (a runtime DI cycle, not a compile error). RedundancyStateSingletonRehomeTests boots a real host but passes a hand-built FakeClusterRoleInfo directly into the extension — it never exercised the sp.GetRequiredService line in the composition root that overflows. The bug lived in Program.cs, which every test mocked around. Only a real boot of the shipped image surfaces it — precisely the live gate's job.

Fix. Derive the singleton's cluster-role scope from AkkaClusterOptions (pure config, no ActorSystem) instead of IClusterRoleInfo:

• BuildClusterRedundancySingletonOptions(AkkaClusterOptions) + WithOtOpcUaClusterRedundancySingleton(AkkaClusterOptions) — derive Role = clusterOptions.Roles.FirstOrDefault(RoleParser.IsClusterRole) ?? RoleParser.Driver (identical to ClusterRoleInfo.ClusterRole, which derives from the same config).
• Program.cs passes sp.GetRequiredService<IOptions<AkkaClusterOptions>>().Value — IOptions<> has no ActorSystem dependency, breaking the cycle.
• Tests updated to pass AkkaClusterOptions { Roles = … }. 5/5 rehome tests pass; Host builds clean.

Files: Program.cs, ControlPlane/ServiceCollectionExtensions.cs, RedundancyStateSingletonRehomeTests.cs. Status: FIXED + live-verified — rebuilt image boots with zero stack-overflow lines; all six nodes start; cluster singletons form (redundancy-state identified per mesh). 5/5 rehome tests pass.

FINDING 2 (fix-forward on master) — driver-only site nodes crash: LDAP options fail validation

Symptom. site-a-1/site-a-2/site-b-1/site-b-2 crash at Host.StartAsync with OptionsValidationException: LDAP transport is None (plaintext) but AllowInsecure is false.

Root cause (Task 7 rig). Only central-1/central-2 carried the Security__Ldap__* env block; the compose even asserted "Only the admin-role central nodes carry" it. But the site nodes serve OPC UA endpoints (4842-4845) and LDAP options are validated at boot on every node (appsettings default Enabled=true, Transport=None, AllowInsecure=false ⇒ throw). The site nodes overrode environment wholesale (YAML <<: does not deep-merge environment), so they inherited no LDAP config. docker compose config validates the file but never boots the app, so this passed the Task 7 acceptance check.

Fix (rig). Extracted the LDAP block to a shared &ldap-env anchor merged into every host node (<<: [*secrets-env, *ldap-env]); corrected the two misleading header comments. Site nodes now boot.

FINDING 3 (fix-forward on master) — site pairs split-brain at simultaneous startup (250/250)

Symptom. Both nodes of each site pair logged is JOINING itself ... and forming a new cluster and each elected itself Primary → ServiceLevel 250/250 instead of 250/240. Each pair was two independent 1-node clusters, not one 2-node mesh.

Root cause (Task 7 rig — a startup race, not a code defect). Both pair nodes are self-first seeds (seed[0]=self), so both run Akka's FirstSeedNodeProcess and can form alone. The partner (site-a-2/site-b-2) only depends_on sql/central/migrator — not its pair founder — so the pair started simultaneously and each formed its own cluster before discovering the other. The central pair avoids this only because central-2 depends_on central-1 (and even then won the InitJoin race by luck). depends_on: service_started is insufficient — it fires when the container launches, before Akka binds. This is an inherent property of symmetric self-first seeding (present for central since #459), now extended to sites; carry to Phase 7 as a production concern (simultaneous cold-start of both pair VMs). NOTE: two separate 1-node clusters do NOT merge via SBR — auto-down resolves partitions of one cluster, not two independent clusters.

Fix (rig). Added an &akka-founder-healthcheck (bash /dev/tcp probe of Akka port 4053; the image has no nc/curl) to site-a-1/site-b-1, and switched site-a-2/site-b-2 to depends_on: { site-a-1: service_healthy }. The founder now binds + forms its mesh before the partner starts, so the partner JOINS it. Verified: site-a-2/site-b-2 log Welcome from <founder> (joined), ServiceLevel 250/240.

---

Evidence log

• Leg 1 — join handshakes: central-1↔central-2, site-a-1↔site-a-2, site-b-1↔site-b-2; no node handshakes a node outside its pair. central dials site nodes over ClusterClient/telemetry (Akka association + gRPC), which is the split-topology transport, NOT gossip membership.
• Leg 2 — Client.CLI redundancy per endpoint: 4840 central-1=240, 4841 central-2=250, 4842 site-a-1=250, 4843 site-a-2=240, 4844 site-b-1=250, 4845 site-b-2=240. One Primary per pair.