fix(docker-dev): Phase 6 rig — LDAP on all nodes + serialize site-pair startup

Two docker-dev-only defects the Phase 6 live gate surfaced (both invisible to
`docker compose config`, which validates the file but never boots the app):

1. Driver-only site nodes crashed at boot with OptionsValidationException — LDAP
   transport None + AllowInsecure false. Only central carried the Security__Ldap__*
   block, but the site nodes serve OPC UA endpoints and validate LDAP options too, and
   they override `environment` wholesale (YAML `<<:` doesn't deep-merge it). Extracted
   the block to a shared &ldap-env anchor merged into every host node.

2. Site pairs split-brained at simultaneous startup (both nodes "JOINING itself",
   250/250 instead of 250/240). Both are self-first seeds, so both run Akka's
   FirstSeedNodeProcess; the partner didn't depend on its pair founder, so they raced
   and each formed a 1-node cluster. Added an &akka-founder-healthcheck (bash /dev/tcp
   probe of Akka 4053 — no nc/curl in the image) to site-a-1/site-b-1 and switched the
   partners to depends_on service_healthy, so the founder forms first and the partner
   JOINS it. (Carry the underlying simultaneous-cold-start property to Phase 7 — it is
   inherent to symmetric self-first seeding, present for central since #459.)

Live gate record: docs/plans/2026-07-24-mesh-phase6-live-gate.md (legs 1-2 PASS —
three isolated meshes, one Primary per pair at 250/240).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

docker-dev/docker-compose.yml

@@ -49,14 +49,19 @@
 #   site-a-1, site-a-2     OTOPCUA_ROLES=driver,cluster-SITE-A — driver-only
 #   site-b-1, site-b-2     OTOPCUA_ROLES=driver,cluster-SITE-B — members of their
 #                          OWN 2-node mesh (self-first seeded within the pair, NOT
-#                          off central). They serve no UI and authenticate no
-#                          users; central manages + deploys to them over the
+#                          off central). They serve no UI, but they DO expose OPC UA
+#                          data-plane endpoints (4842-4845) and authenticate those
+#                          users against the same shared GLAuth (via the *ldap-env
+#                          anchor); central manages + deploys to them over the
 #                          ClusterClient/gRPC/FetchAndCache transports above, not
 #                          gossip.
 #
 #   Auth is real LDAP against the shared GLAuth on the Linux Docker host
 #   (10.100.0.35:3893, dc=zb,dc=local) — there is no LDAP container here.
-#   Only the admin-role central nodes carry the Security__Ldap__* block.
+#   EVERY host node carries the Security__Ldap__* block (shared *ldap-env anchor):
+#   the driver-only site nodes serve OPC UA endpoints too, and LDAP options are
+#   validated at boot on all of them (a node with no LDAP config crashes with
+#   OptionsValidationException — transport None + AllowInsecure false).
 #   Sign in `multi-role` / `password`.
 #
 #   traefik    PathPrefix(`/`) → central-1 / central-2 (the single UI route).
@@ -126,6 +131,43 @@ x-secrets-env: &secrets-env
   Secrets__Replication__AnnounceInterval: "00:00:05"
   ZB_SECRETS_MASTER_KEY: "${OTOPCUA_SECRETS_KEK:-ZYGhIX0luS/XsevpCB2W18jYHMcqO6AjM9oXy+T6Zp4=}"
 
+# EVERY node that serves an OPC UA endpoint authenticates data-plane users against the shared GLAuth —
+# that includes the driver-only site nodes (ports 4842-4845), not just the admin/central pair. LDAP
+# options are validated at host start on ALL of them (Enabled=true by appsettings default, Transport=None,
+# AllowInsecure=false ⇒ OptionsValidationException), so a node that carries no LDAP config crashes at boot.
+# This anchor is merged into every host node's environment (via `<<: [*secrets-env, *ldap-env]`). Central
+# still also lists the block inline for readability; the explicit keys win over the merge, same values.
+x-ldap-env: &ldap-env
+  Security__Ldap__Enabled: "true"
+  Security__Ldap__DevStubMode: "false"
+  Security__Ldap__Server: "10.100.0.35"
+  Security__Ldap__Port: "3893"
+  Security__Ldap__Transport: "None"
+  Security__Ldap__AllowInsecure: "true"
+  Security__Ldap__SearchBase: "dc=zb,dc=local"
+  Security__Ldap__ServiceAccountDn: "cn=serviceaccount,dc=zb,dc=local"
+  Security__Ldap__ServiceAccountPassword: "serviceaccount123"
+  Security__Ldap__GroupToRole__ReadOnly: "ReadOnly"
+  Security__Ldap__GroupToRole__WriteOperate: "WriteOperate"
+  Security__Ldap__GroupToRole__WriteTune: "WriteTune"
+  Security__Ldap__GroupToRole__WriteConfigure: "WriteConfigure"
+  Security__Ldap__GroupToRole__AlarmAck: "AlarmAck"
+
+# Readiness gate for a pair FOUNDER node (site-a-1 / site-b-1): passes once its Akka port 4053 is bound.
+# The partner (site-a-2 / site-b-2) waits on this via `depends_on: { condition: service_healthy }` so the
+# founder forms its 2-node mesh FIRST and the partner JOINS it — instead of both self-first seeds racing
+# and each forming its own single-node cluster (the split brain the Phase 6 live gate observed: 250/250,
+# both Primary). `service_started` is NOT enough — it fires the instant the container launches, long
+# before Akka binds. The image ships bash but no nc/curl, so the probe uses bash's /dev/tcp. start_period
+# gives the founder a head start to not just bind but FORM (a 1-node cluster forms within the same second
+# as the bind), so the partner's InitJoin lands on an existing cluster and is ack'd.
+x-akka-founder-healthcheck: &akka-founder-healthcheck
+  test: ["CMD-SHELL", "bash -c '</dev/tcp/localhost/4053' || exit 1"]
+  interval: 3s
+  timeout: 2s
+  retries: 20
+  start_period: 8s
+
 services:
 
   sql:
@@ -226,7 +268,7 @@ services:
       sql: { condition: service_healthy }
       migrator: { condition: service_completed_successfully }
     environment:
-      <<: *secrets-env
+      <<: [*secrets-env, *ldap-env]
       OTOPCUA_ROLES: "admin,driver,cluster-MAIN"
       ASPNETCORE_URLS: "http://+:9000"
       ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
@@ -345,7 +387,7 @@ services:
       central-1: { condition: service_started }
       migrator: { condition: service_completed_successfully }
     environment:
-      <<: *secrets-env
+      <<: [*secrets-env, *ldap-env]
       OTOPCUA_ROLES: "admin,driver,cluster-MAIN"
       ASPNETCORE_URLS: "http://+:9000"
       ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
@@ -438,6 +480,7 @@ services:
 
   site-a-1:
     <<: *otopcua-host
+    healthcheck: *akka-founder-healthcheck
     depends_on:
       sql: { condition: service_healthy }
       central-1: { condition: service_started }
@@ -487,7 +530,7 @@ services:
       MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
       MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
       MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
-      <<: *secrets-env
+      <<: [*secrets-env, *ldap-env]
       # Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
       # mem_reservation are inherited from the *otopcua-host anchor.
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
@@ -545,6 +588,13 @@ services:
       sql: { condition: service_healthy }
       central-1: { condition: service_started }
       migrator: { condition: service_completed_successfully }
+      # Serialize the SITE-A pair's startup so site-a-1 forms the mesh first and site-a-2 JOINS it —
+      # exactly as central-2 waits on central-1. Both nodes are self-first seeds (seed[0]=self), so BOTH
+      # run Akka's FirstSeedNodeProcess and, if they start simultaneously, EACH forms its own single-node
+      # cluster (split brain — observed in the Phase 6 live gate: both logged "JOINING itself ... forming
+      # a new cluster", both elected themselves Primary → 250/250 instead of 250/240). Waiting on site-a-1
+      # gives it the head start to be Up before site-a-2's InitJoin, so site-a-2 joins the existing mesh.
+      site-a-1: { condition: service_healthy }
     environment:
       OTOPCUA_ROLES: "driver,cluster-SITE-A"
       # Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
@@ -580,7 +630,7 @@ services:
       MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
       MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
       MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
-      <<: *secrets-env
+      <<: [*secrets-env, *ldap-env]
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
@@ -628,6 +678,7 @@ services:
 
   site-b-1:
     <<: *otopcua-host
+    healthcheck: *akka-founder-healthcheck
     depends_on:
       sql: { condition: service_healthy }
       central-1: { condition: service_started }
@@ -667,7 +718,7 @@ services:
       MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
       MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
       MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
-      <<: *secrets-env
+      <<: [*secrets-env, *ldap-env]
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
@@ -700,6 +751,10 @@ services:
       sql: { condition: service_healthy }
       central-1: { condition: service_started }
       migrator: { condition: service_completed_successfully }
+      # Serialize the SITE-B pair's startup so site-b-1 forms the mesh first and site-b-2 JOINS it — see
+      # the identical comment on site-a-2. Without this both self-first seeds form their own single-node
+      # cluster on simultaneous start (split brain).
+      site-b-1: { condition: service_healthy }
     environment:
       OTOPCUA_ROLES: "driver,cluster-SITE-B"
       # Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
@@ -735,7 +790,7 @@ services:
       MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
       MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
       MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
-      <<: *secrets-env
+      <<: [*secrets-env, *ldap-env]
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"

docs/plans/2026-07-24-mesh-phase6-live-gate.md

@@ -0,0 +1,126 @@
+# Per-cluster mesh Phase 6 — live gate record
+
+> **Exit gate for Phase 6** (mesh partition). Bring up the rewritten docker-dev rig
+> (three independent 2-node Akka meshes) against **merged master** and prove every
+> previously-live-gated behaviour *per pair*, plus the split itself.
+>
+> Merge landed first (`origin/master` @ `2839deb5`, "merge now, gate after"); this gate
+> runs against the merged code and **fixes forward on master** if anything surfaces.
+
+**Rig topology** (docker-dev, local OrbStack):
+
+| Mesh | Nodes | Roles | OPC UA (host) | Seeds (self-first, pair-local) |
+|---|---|---|---|---|
+| MAIN | central-1, central-2 | `admin,driver,cluster-MAIN` | 4840 / 4841 | own pair only |
+| SITE-A | site-a-1, site-a-2 | `driver,cluster-SITE-A` | 4842 / 4843 | own pair only |
+| SITE-B | site-b-1, site-b-2 | `driver,cluster-SITE-B` | 4844 / 4845 | own pair only |
+
+- Akka port 4053 (per container); telemetry gRPC `:4056`; ConfigServe `:4055` (central);
+  LocalDb sync `:9001` (site-a); AdminUI `http://localhost:9200` (Traefik → central-1/2).
+- Transports: `MeshTransport__Mode=ClusterClient` (all), `Telemetry__Mode=Grpc` (all six),
+  `TelemetryDial__Mode=Grpc` (central pair). Secrets replication pair-local.
+- Deploy: `POST http://localhost:9200/api/deployments`, header `X-Api-Key: docker-dev-deploy-key`.
+
+---
+
+## Legs
+
+| # | Leg | Status | Evidence |
+|---|---|---|---|
+| 1 | Three meshes form — each node sees ONLY its own pair (2 members) | ✅ PASS | each node handshakes ONLY its pair partner |
+| 2 | Two+ Primaries, one per pair; ServiceLevel 250/240 within each pair | ✅ PASS | MAIN central-2=250/central-1=240; SITE-A site-a-1=250/site-a-2=240; SITE-B site-b-1=250/site-b-2=240 |
+| 3 | Deploy reaches all clusters over per-cluster ClusterClient; seals green | PENDING | |
+| 4 | Telemetry per cluster over gRPC (:4056); pills live; kill/reconnect ~5s | PENDING | |
+| 5 | Reconciler quiet — no `EnabledRowNotInCluster` for foreign rows | PENDING | |
+| 6 | Secrets pair-local — no cross-mesh replication / no unreachable-peer errors | PENDING | |
+| 7 | Split validator — a site node flipped to `MeshTransport__Mode=Dps` refuses to boot | PENDING | |
+| 8 | Failover within a pair — kill a Primary, Secondary takes over (auto-down 1-vs-1) | PENDING | |
+
+---
+
+## Findings
+
+### FINDING 1 (fix-forward on master) — boot-crash: `StackOverflowException` on EVERY node
+
+**Symptom.** First `up` of the fresh Phase-6 images: all six nodes crash at startup with
+`Stack overflow.` — an infinite recursion entering through
+`ZB.MOM.WW.OtOpcUa.Cluster.ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap`.
+
+**Root cause (Phase 6, Task 2 wiring).** `Program.cs` resolved `IClusterRoleInfo` **eagerly, inside
+the `AddAkka` configurator lambda**:
+
+```csharp
+ab.WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>());  // line 385
+```
+
+That lambda runs *while the `ActorSystem` is being constructed*, but `ClusterRoleInfo`'s ctor
+**depends on the `ActorSystem`** (it is a live `Cluster.State` view). Resolving it there forces a
+nested `ActorSystem` build → re-runs the same configurator → `sp.GetRequiredService<IClusterRoleInfo>()`
+again → ∞. The cycle:
+`WithOtOpcUaClusterBootstrap → GetService<IClusterRoleInfo> → ActorSystemFactory → WithOtOpcUaClusterBootstrap → …`
+
+**Why merge + build + tests + reviews all missed it.** It compiles cleanly (a *runtime* DI cycle,
+not a compile error). `RedundancyStateSingletonRehomeTests` boots a real host but passes a hand-built
+`FakeClusterRoleInfo` **directly** into the extension — it never exercised the `sp.GetRequiredService`
+line in the composition root that overflows. The bug lived in `Program.cs`, which every test mocked
+around. Only a real boot of the shipped image surfaces it — precisely the live gate's job.
+
+**Fix.** Derive the singleton's cluster-role scope from `AkkaClusterOptions` (pure config, no
+ActorSystem) instead of `IClusterRoleInfo`:
+- `BuildClusterRedundancySingletonOptions(AkkaClusterOptions)` + `WithOtOpcUaClusterRedundancySingleton(AkkaClusterOptions)`
+  — derive `Role = clusterOptions.Roles.FirstOrDefault(RoleParser.IsClusterRole) ?? RoleParser.Driver`
+  (identical to `ClusterRoleInfo.ClusterRole`, which derives from the same config).
+- `Program.cs` passes `sp.GetRequiredService<IOptions<AkkaClusterOptions>>().Value` — `IOptions<>` has
+  no ActorSystem dependency, breaking the cycle.
+- Tests updated to pass `AkkaClusterOptions { Roles = … }`. 5/5 rehome tests pass; Host builds clean.
+
+Files: `Program.cs`, `ControlPlane/ServiceCollectionExtensions.cs`, `RedundancyStateSingletonRehomeTests.cs`.
+Status: **FIXED + live-verified** — rebuilt image boots with zero stack-overflow lines; all six nodes
+start; cluster singletons form (`redundancy-state` identified per mesh). 5/5 rehome tests pass.
+
+### FINDING 2 (fix-forward on master) — driver-only site nodes crash: LDAP options fail validation
+
+**Symptom.** site-a-1/site-a-2/site-b-1/site-b-2 crash at `Host.StartAsync` with
+`OptionsValidationException: LDAP transport is None (plaintext) but AllowInsecure is false`.
+
+**Root cause (Task 7 rig).** Only central-1/central-2 carried the `Security__Ldap__*` env block; the
+compose even asserted "Only the admin-role central nodes carry" it. But the site nodes serve OPC UA
+endpoints (4842-4845) and LDAP options are validated at boot on every node (appsettings default
+`Enabled=true`, `Transport=None`, `AllowInsecure=false` ⇒ throw). The site nodes overrode `environment`
+wholesale (YAML `<<:` does not deep-merge `environment`), so they inherited no LDAP config. `docker
+compose config` validates the file but never boots the app, so this passed the Task 7 acceptance check.
+
+**Fix (rig).** Extracted the LDAP block to a shared `&ldap-env` anchor merged into **every** host node
+(`<<: [*secrets-env, *ldap-env]`); corrected the two misleading header comments. Site nodes now boot.
+
+### FINDING 3 (fix-forward on master) — site pairs split-brain at simultaneous startup (250/250)
+
+**Symptom.** Both nodes of each site pair logged `is JOINING itself ... and forming a new cluster` and
+each elected itself Primary → ServiceLevel **250/250** instead of 250/240. Each pair was two independent
+1-node clusters, not one 2-node mesh.
+
+**Root cause (Task 7 rig — a startup race, not a code defect).** Both pair nodes are self-first seeds
+(`seed[0]=self`), so both run Akka's `FirstSeedNodeProcess` and can form alone. The partner
+(site-a-2/site-b-2) only `depends_on` sql/central/migrator — **not its pair founder** — so the pair
+started simultaneously and each formed its own cluster before discovering the other. The central pair
+avoids this only because central-2 `depends_on central-1` (and even then won the InitJoin race by luck).
+`depends_on: service_started` is insufficient — it fires when the container launches, before Akka binds.
+This is an inherent property of symmetric self-first seeding (present for central since #459), now
+extended to sites; **carry to Phase 7** as a production concern (simultaneous cold-start of both pair
+VMs). NOTE: two separate 1-node clusters do NOT merge via SBR — auto-down resolves partitions of one
+cluster, not two independent clusters.
+
+**Fix (rig).** Added an `&akka-founder-healthcheck` (bash `/dev/tcp` probe of Akka port 4053; the image
+has no nc/curl) to site-a-1/site-b-1, and switched site-a-2/site-b-2 to `depends_on: { site-a-1:
+service_healthy }`. The founder now binds + forms its mesh before the partner starts, so the partner
+JOINS it. Verified: site-a-2/site-b-2 log `Welcome from <founder>` (joined), ServiceLevel 250/240.
+
+---
+
+## Evidence log
+
+- **Leg 1** — join handshakes: central-1↔central-2, site-a-1↔site-a-2, site-b-1↔site-b-2; no node
+  handshakes a node outside its pair. central *dials* site nodes over ClusterClient/telemetry (Akka
+  association + gRPC), which is the split-topology transport, NOT gossip membership.
+- **Leg 2** — `Client.CLI redundancy` per endpoint: 4840 central-1=240, 4841 central-2=250, 4842
+  site-a-1=250, 4843 site-a-2=240, 4844 site-b-1=250, 4845 site-b-2=240. One Primary per pair.