fix(docker-dev): Phase 6 rig — LDAP on all nodes + serialize site-pair startup
Two docker-dev-only defects the Phase 6 live gate surfaced (both invisible to `docker compose config`, which validates the file but never boots the app): 1. Driver-only site nodes crashed at boot with OptionsValidationException — LDAP transport None + AllowInsecure false. Only central carried the Security__Ldap__* block, but the site nodes serve OPC UA endpoints and validate LDAP options too, and they override `environment` wholesale (YAML `<<:` doesn't deep-merge it). Extracted the block to a shared &ldap-env anchor merged into every host node. 2. Site pairs split-brained at simultaneous startup (both nodes "JOINING itself", 250/250 instead of 250/240). Both are self-first seeds, so both run Akka's FirstSeedNodeProcess; the partner didn't depend on its pair founder, so they raced and each formed a 1-node cluster. Added an &akka-founder-healthcheck (bash /dev/tcp probe of Akka 4053 — no nc/curl in the image) to site-a-1/site-b-1 and switched the partners to depends_on service_healthy, so the founder forms first and the partner JOINS it. (Carry the underlying simultaneous-cold-start property to Phase 7 — it is inherent to symmetric self-first seeding, present for central since #459.) Live gate record: docs/plans/2026-07-24-mesh-phase6-live-gate.md (legs 1-2 PASS — three isolated meshes, one Primary per pair at 250/240). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,126 @@
|
||||
# Per-cluster mesh Phase 6 — live gate record
|
||||
|
||||
> **Exit gate for Phase 6** (mesh partition). Bring up the rewritten docker-dev rig
|
||||
> (three independent 2-node Akka meshes) against **merged master** and prove every
|
||||
> previously-live-gated behaviour *per pair*, plus the split itself.
|
||||
>
|
||||
> Merge landed first (`origin/master` @ `2839deb5`, "merge now, gate after"); this gate
|
||||
> runs against the merged code and **fixes forward on master** if anything surfaces.
|
||||
|
||||
**Rig topology** (docker-dev, local OrbStack):
|
||||
|
||||
| Mesh | Nodes | Roles | OPC UA (host) | Seeds (self-first, pair-local) |
|
||||
|---|---|---|---|---|
|
||||
| MAIN | central-1, central-2 | `admin,driver,cluster-MAIN` | 4840 / 4841 | own pair only |
|
||||
| SITE-A | site-a-1, site-a-2 | `driver,cluster-SITE-A` | 4842 / 4843 | own pair only |
|
||||
| SITE-B | site-b-1, site-b-2 | `driver,cluster-SITE-B` | 4844 / 4845 | own pair only |
|
||||
|
||||
- Akka port 4053 (per container); telemetry gRPC `:4056`; ConfigServe `:4055` (central);
|
||||
LocalDb sync `:9001` (site-a); AdminUI `http://localhost:9200` (Traefik → central-1/2).
|
||||
- Transports: `MeshTransport__Mode=ClusterClient` (all), `Telemetry__Mode=Grpc` (all six),
|
||||
`TelemetryDial__Mode=Grpc` (central pair). Secrets replication pair-local.
|
||||
- Deploy: `POST http://localhost:9200/api/deployments`, header `X-Api-Key: docker-dev-deploy-key`.
|
||||
|
||||
---
|
||||
|
||||
## Legs
|
||||
|
||||
| # | Leg | Status | Evidence |
|
||||
|---|---|---|---|
|
||||
| 1 | Three meshes form — each node sees ONLY its own pair (2 members) | ✅ PASS | each node handshakes ONLY its pair partner |
|
||||
| 2 | Two+ Primaries, one per pair; ServiceLevel 250/240 within each pair | ✅ PASS | MAIN central-2=250/central-1=240; SITE-A site-a-1=250/site-a-2=240; SITE-B site-b-1=250/site-b-2=240 |
|
||||
| 3 | Deploy reaches all clusters over per-cluster ClusterClient; seals green | PENDING | |
|
||||
| 4 | Telemetry per cluster over gRPC (:4056); pills live; kill/reconnect ~5s | PENDING | |
|
||||
| 5 | Reconciler quiet — no `EnabledRowNotInCluster` for foreign rows | PENDING | |
|
||||
| 6 | Secrets pair-local — no cross-mesh replication / no unreachable-peer errors | PENDING | |
|
||||
| 7 | Split validator — a site node flipped to `MeshTransport__Mode=Dps` refuses to boot | PENDING | |
|
||||
| 8 | Failover within a pair — kill a Primary, Secondary takes over (auto-down 1-vs-1) | PENDING | |
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
### FINDING 1 (fix-forward on master) — boot-crash: `StackOverflowException` on EVERY node
|
||||
|
||||
**Symptom.** First `up` of the fresh Phase-6 images: all six nodes crash at startup with
|
||||
`Stack overflow.` — an infinite recursion entering through
|
||||
`ZB.MOM.WW.OtOpcUa.Cluster.ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap`.
|
||||
|
||||
**Root cause (Phase 6, Task 2 wiring).** `Program.cs` resolved `IClusterRoleInfo` **eagerly, inside
|
||||
the `AddAkka` configurator lambda**:
|
||||
|
||||
```csharp
|
||||
ab.WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>()); // line 385
|
||||
```
|
||||
|
||||
That lambda runs *while the `ActorSystem` is being constructed*, but `ClusterRoleInfo`'s ctor
|
||||
**depends on the `ActorSystem`** (it is a live `Cluster.State` view). Resolving it there forces a
|
||||
nested `ActorSystem` build → re-runs the same configurator → `sp.GetRequiredService<IClusterRoleInfo>()`
|
||||
again → ∞. The cycle:
|
||||
`WithOtOpcUaClusterBootstrap → GetService<IClusterRoleInfo> → ActorSystemFactory → WithOtOpcUaClusterBootstrap → …`
|
||||
|
||||
**Why merge + build + tests + reviews all missed it.** It compiles cleanly (a *runtime* DI cycle,
|
||||
not a compile error). `RedundancyStateSingletonRehomeTests` boots a real host but passes a hand-built
|
||||
`FakeClusterRoleInfo` **directly** into the extension — it never exercised the `sp.GetRequiredService`
|
||||
line in the composition root that overflows. The bug lived in `Program.cs`, which every test mocked
|
||||
around. Only a real boot of the shipped image surfaces it — precisely the live gate's job.
|
||||
|
||||
**Fix.** Derive the singleton's cluster-role scope from `AkkaClusterOptions` (pure config, no
|
||||
ActorSystem) instead of `IClusterRoleInfo`:
|
||||
- `BuildClusterRedundancySingletonOptions(AkkaClusterOptions)` + `WithOtOpcUaClusterRedundancySingleton(AkkaClusterOptions)`
|
||||
— derive `Role = clusterOptions.Roles.FirstOrDefault(RoleParser.IsClusterRole) ?? RoleParser.Driver`
|
||||
(identical to `ClusterRoleInfo.ClusterRole`, which derives from the same config).
|
||||
- `Program.cs` passes `sp.GetRequiredService<IOptions<AkkaClusterOptions>>().Value` — `IOptions<>` has
|
||||
no ActorSystem dependency, breaking the cycle.
|
||||
- Tests updated to pass `AkkaClusterOptions { Roles = … }`. 5/5 rehome tests pass; Host builds clean.
|
||||
|
||||
Files: `Program.cs`, `ControlPlane/ServiceCollectionExtensions.cs`, `RedundancyStateSingletonRehomeTests.cs`.
|
||||
Status: **FIXED + live-verified** — rebuilt image boots with zero stack-overflow lines; all six nodes
|
||||
start; cluster singletons form (`redundancy-state` identified per mesh). 5/5 rehome tests pass.
|
||||
|
||||
### FINDING 2 (fix-forward on master) — driver-only site nodes crash: LDAP options fail validation
|
||||
|
||||
**Symptom.** site-a-1/site-a-2/site-b-1/site-b-2 crash at `Host.StartAsync` with
|
||||
`OptionsValidationException: LDAP transport is None (plaintext) but AllowInsecure is false`.
|
||||
|
||||
**Root cause (Task 7 rig).** Only central-1/central-2 carried the `Security__Ldap__*` env block; the
|
||||
compose even asserted "Only the admin-role central nodes carry" it. But the site nodes serve OPC UA
|
||||
endpoints (4842-4845) and LDAP options are validated at boot on every node (appsettings default
|
||||
`Enabled=true`, `Transport=None`, `AllowInsecure=false` ⇒ throw). The site nodes overrode `environment`
|
||||
wholesale (YAML `<<:` does not deep-merge `environment`), so they inherited no LDAP config. `docker
|
||||
compose config` validates the file but never boots the app, so this passed the Task 7 acceptance check.
|
||||
|
||||
**Fix (rig).** Extracted the LDAP block to a shared `&ldap-env` anchor merged into **every** host node
|
||||
(`<<: [*secrets-env, *ldap-env]`); corrected the two misleading header comments. Site nodes now boot.
|
||||
|
||||
### FINDING 3 (fix-forward on master) — site pairs split-brain at simultaneous startup (250/250)
|
||||
|
||||
**Symptom.** Both nodes of each site pair logged `is JOINING itself ... and forming a new cluster` and
|
||||
each elected itself Primary → ServiceLevel **250/250** instead of 250/240. Each pair was two independent
|
||||
1-node clusters, not one 2-node mesh.
|
||||
|
||||
**Root cause (Task 7 rig — a startup race, not a code defect).** Both pair nodes are self-first seeds
|
||||
(`seed[0]=self`), so both run Akka's `FirstSeedNodeProcess` and can form alone. The partner
|
||||
(site-a-2/site-b-2) only `depends_on` sql/central/migrator — **not its pair founder** — so the pair
|
||||
started simultaneously and each formed its own cluster before discovering the other. The central pair
|
||||
avoids this only because central-2 `depends_on central-1` (and even then won the InitJoin race by luck).
|
||||
`depends_on: service_started` is insufficient — it fires when the container launches, before Akka binds.
|
||||
This is an inherent property of symmetric self-first seeding (present for central since #459), now
|
||||
extended to sites; **carry to Phase 7** as a production concern (simultaneous cold-start of both pair
|
||||
VMs). NOTE: two separate 1-node clusters do NOT merge via SBR — auto-down resolves partitions of one
|
||||
cluster, not two independent clusters.
|
||||
|
||||
**Fix (rig).** Added an `&akka-founder-healthcheck` (bash `/dev/tcp` probe of Akka port 4053; the image
|
||||
has no nc/curl) to site-a-1/site-b-1, and switched site-a-2/site-b-2 to `depends_on: { site-a-1:
|
||||
service_healthy }`. The founder now binds + forms its mesh before the partner starts, so the partner
|
||||
JOINS it. Verified: site-a-2/site-b-2 log `Welcome from <founder>` (joined), ServiceLevel 250/240.
|
||||
|
||||
---
|
||||
|
||||
## Evidence log
|
||||
|
||||
- **Leg 1** — join handshakes: central-1↔central-2, site-a-1↔site-a-2, site-b-1↔site-b-2; no node
|
||||
handshakes a node outside its pair. central *dials* site nodes over ClusterClient/telemetry (Akka
|
||||
association + gRPC), which is the split-topology transport, NOT gossip membership.
|
||||
- **Leg 2** — `Client.CLI redundancy` per endpoint: 4840 central-1=240, 4841 central-2=250, 4842
|
||||
site-a-1=250, 4843 site-a-2=240, 4844 site-b-1=250, 4845 site-b-2=240. One Primary per pair.
|
||||
Reference in New Issue
Block a user