# Per-cluster mesh Phase 6 — live gate record

> **Exit gate for Phase 6** (mesh partition). Bring up the rewritten docker-dev rig
> (three independent 2-node Akka meshes) against **merged master** and prove every
> previously-live-gated behaviour *per pair*, plus the split itself.
>
> Merge landed first (`origin/master` @ `2839deb5`, "merge now, gate after"); this gate
> runs against the merged code and **fixes forward on master** if anything surfaces.

**Rig topology** (docker-dev, local OrbStack):

| Mesh | Nodes | Roles | OPC UA (host) | Seeds (self-first, pair-local) |
|---|---|---|---|---|
| MAIN | central-1, central-2 | `admin,driver,cluster-MAIN` | 4840 / 4841 | own pair only |
| SITE-A | site-a-1, site-a-2 | `driver,cluster-SITE-A` | 4842 / 4843 | own pair only |
| SITE-B | site-b-1, site-b-2 | `driver,cluster-SITE-B` | 4844 / 4845 | own pair only |

- Akka port 4053 (per container); telemetry gRPC `:4056`; ConfigServe `:4055` (central);
  LocalDb sync `:9001` (site-a); AdminUI `http://localhost:9200` (Traefik → central-1/2).
- Transports: `MeshTransport__Mode=ClusterClient` (all), `Telemetry__Mode=Grpc` (all six),
  `TelemetryDial__Mode=Grpc` (central pair). Secrets replication pair-local.
- Deploy: `POST http://localhost:9200/api/deployments`, header `X-Api-Key: docker-dev-deploy-key`.

---

## Legs

| # | Leg | Status | Evidence |
|---|---|---|---|
| 1 | Three meshes form — each node sees ONLY its own pair (2 members) | ✅ PASS | each node handshakes ONLY its pair partner |
| 2 | Two+ Primaries, one per pair; ServiceLevel 250/240 within each pair | ✅ PASS | MAIN central-2=250/central-1=240; SITE-A site-a-1=250/site-a-2=240; SITE-B site-b-1=250/site-b-2=240 |
| 3 | Deploy reaches all clusters over per-cluster ClusterClient; seals green | ✅ PASS | "Deployment 019dbd99… sealed (acks=6)"; all 6 NodeDeploymentState rows acked; one ClusterClient per site cluster |
| 4 | Telemetry per cluster over gRPC (:4056); pills live; kill/reconnect ~5s | ✅ PASS | central-1 6 outbound :4056 dials, each site node 2 inbound (both admin dial in), 0 failures/30s; site-b restart failed→recovered the streams |
| 5 | Reconciler quiet — no `EnabledRowNotInCluster` for foreign rows | ✅ PASS | central-2 reconciler: "reconcile with cluster membership (2 driver members)" — own MAIN pair only; 0 foreign-row errors |
| 6 | Secrets pair-local — no cross-mesh replication / no unreachable-peer errors | ✅ PASS (by fallback criterion) | 0 secret errors fleet-wide; DPS topic rides each pair's isolated mesh (Leg 1) ⇒ structurally pair-local |
| 7 | Split validator — a site node flipped to `MeshTransport__Mode=Dps` refuses to boot | ✅ PASS | "Cluster:Roles declares a cluster-SITE-B role … MeshTransport:Mode is 'Dps' … set MeshTransport:Mode=ClusterClient" |
| 8 | Failover within a pair — kill a Primary, Secondary takes over (auto-down 1-vs-1) | ✅ PASS | killed site-a-1 (Primary); site-a-2 auto-downed it, took the singleton, 240→250, survived alone (auto-down 1-vs-1 = deferred Phase 0a gate, now proven) |

**Phase 6 exit gate: MET.** All 8 legs pass. Three findings fixed forward on master (`3a4ed8dd` product
boot-crash, `792f28ec` rig LDAP + split-brain serialization). One test-methodology note: the Leg 7
`compose run … site-b-2` throwaway shares the `site-b-2` network alias and briefly disrupted the SITE-B
Akka association — restart the affected pair after that probe (done), or run the validator check via a
container that does not join the compose network alias.

---

## Findings

### FINDING 1 (fix-forward on master) — boot-crash: `StackOverflowException` on EVERY node

**Symptom.** First `up` of the fresh Phase-6 images: all six nodes crash at startup with
`Stack overflow.` — an infinite recursion entering through
`ZB.MOM.WW.OtOpcUa.Cluster.ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap`.

**Root cause (Phase 6, Task 2 wiring).** `Program.cs` resolved `IClusterRoleInfo` **eagerly, inside
the `AddAkka` configurator lambda**:

```csharp
ab.WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>());  // line 385
```

That lambda runs *while the `ActorSystem` is being constructed*, but `ClusterRoleInfo`'s ctor
**depends on the `ActorSystem`** (it is a live `Cluster.State` view). Resolving it there forces a
nested `ActorSystem` build → re-runs the same configurator → `sp.GetRequiredService<IClusterRoleInfo>()`
again → ∞. The cycle:
`WithOtOpcUaClusterBootstrap → GetService<IClusterRoleInfo> → ActorSystemFactory → WithOtOpcUaClusterBootstrap → …`

**Why merge + build + tests + reviews all missed it.** It compiles cleanly (a *runtime* DI cycle,
not a compile error). `RedundancyStateSingletonRehomeTests` boots a real host but passes a hand-built
`FakeClusterRoleInfo` **directly** into the extension — it never exercised the `sp.GetRequiredService`
line in the composition root that overflows. The bug lived in `Program.cs`, which every test mocked
around. Only a real boot of the shipped image surfaces it — precisely the live gate's job.

**Fix.** Derive the singleton's cluster-role scope from `AkkaClusterOptions` (pure config, no
ActorSystem) instead of `IClusterRoleInfo`:
- `BuildClusterRedundancySingletonOptions(AkkaClusterOptions)` + `WithOtOpcUaClusterRedundancySingleton(AkkaClusterOptions)`
  — derive `Role = clusterOptions.Roles.FirstOrDefault(RoleParser.IsClusterRole) ?? RoleParser.Driver`
  (identical to `ClusterRoleInfo.ClusterRole`, which derives from the same config).
- `Program.cs` passes `sp.GetRequiredService<IOptions<AkkaClusterOptions>>().Value` — `IOptions<>` has
  no ActorSystem dependency, breaking the cycle.
- Tests updated to pass `AkkaClusterOptions { Roles = … }`. 5/5 rehome tests pass; Host builds clean.

Files: `Program.cs`, `ControlPlane/ServiceCollectionExtensions.cs`, `RedundancyStateSingletonRehomeTests.cs`.
Status: **FIXED + live-verified** — rebuilt image boots with zero stack-overflow lines; all six nodes
start; cluster singletons form (`redundancy-state` identified per mesh). 5/5 rehome tests pass.

### FINDING 2 (fix-forward on master) — driver-only site nodes crash: LDAP options fail validation

**Symptom.** site-a-1/site-a-2/site-b-1/site-b-2 crash at `Host.StartAsync` with
`OptionsValidationException: LDAP transport is None (plaintext) but AllowInsecure is false`.

**Root cause (Task 7 rig).** Only central-1/central-2 carried the `Security__Ldap__*` env block; the
compose even asserted "Only the admin-role central nodes carry" it. But the site nodes serve OPC UA
endpoints (4842-4845) and LDAP options are validated at boot on every node (appsettings default
`Enabled=true`, `Transport=None`, `AllowInsecure=false` ⇒ throw). The site nodes overrode `environment`
wholesale (YAML `<<:` does not deep-merge `environment`), so they inherited no LDAP config. `docker
compose config` validates the file but never boots the app, so this passed the Task 7 acceptance check.

**Fix (rig).** Extracted the LDAP block to a shared `&ldap-env` anchor merged into **every** host node
(`<<: [*secrets-env, *ldap-env]`); corrected the two misleading header comments. Site nodes now boot.

### FINDING 3 (fix-forward on master) — site pairs split-brain at simultaneous startup (250/250)

**Symptom.** Both nodes of each site pair logged `is JOINING itself ... and forming a new cluster` and
each elected itself Primary → ServiceLevel **250/250** instead of 250/240. Each pair was two independent
1-node clusters, not one 2-node mesh.

**Root cause (Task 7 rig — a startup race, not a code defect).** Both pair nodes are self-first seeds
(`seed[0]=self`), so both run Akka's `FirstSeedNodeProcess` and can form alone. The partner
(site-a-2/site-b-2) only `depends_on` sql/central/migrator — **not its pair founder** — so the pair
started simultaneously and each formed its own cluster before discovering the other. The central pair
avoids this only because central-2 `depends_on central-1` (and even then won the InitJoin race by luck).
`depends_on: service_started` is insufficient — it fires when the container launches, before Akka binds.
This is an inherent property of symmetric self-first seeding (present for central since #459), now
extended to sites; **carry to Phase 7** as a production concern (simultaneous cold-start of both pair
VMs). NOTE: two separate 1-node clusters do NOT merge via SBR — auto-down resolves partitions of one
cluster, not two independent clusters.

**Fix (rig).** Added an `&akka-founder-healthcheck` (bash `/dev/tcp` probe of Akka port 4053; the image
has no nc/curl) to site-a-1/site-b-1, and switched site-a-2/site-b-2 to `depends_on: { site-a-1:
service_healthy }`. The founder now binds + forms its mesh before the partner starts, so the partner
JOINS it. Verified: site-a-2/site-b-2 log `Welcome from <founder>` (joined), ServiceLevel 250/240.

---

## Evidence log

- **Leg 1** — join handshakes: central-1↔central-2, site-a-1↔site-a-2, site-b-1↔site-b-2; no node
  handshakes a node outside its pair. central *dials* site nodes over ClusterClient/telemetry (Akka
  association + gRPC), which is the split-topology transport, NOT gossip membership.
- **Leg 2** — `Client.CLI redundancy` per endpoint: 4840 central-1=240, 4841 central-2=250, 4842
  site-a-1=250, 4843 site-a-2=240, 4844 site-b-1=250, 4845 site-b-2=240. One Primary per pair.