From 792f28ec7b9b5a1e7f621f399b488d77e9c71cb6 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 24 Jul 2026 06:54:27 -0400 Subject: [PATCH] =?UTF-8?q?fix(docker-dev):=20Phase=206=20rig=20=E2=80=94?= =?UTF-8?q?=20LDAP=20on=20all=20nodes=20+=20serialize=20site-pair=20startu?= =?UTF-8?q?p?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two docker-dev-only defects the Phase 6 live gate surfaced (both invisible to `docker compose config`, which validates the file but never boots the app): 1. Driver-only site nodes crashed at boot with OptionsValidationException — LDAP transport None + AllowInsecure false. Only central carried the Security__Ldap__* block, but the site nodes serve OPC UA endpoints and validate LDAP options too, and they override `environment` wholesale (YAML `<<:` doesn't deep-merge it). Extracted the block to a shared &ldap-env anchor merged into every host node. 2. Site pairs split-brained at simultaneous startup (both nodes "JOINING itself", 250/250 instead of 250/240). Both are self-first seeds, so both run Akka's FirstSeedNodeProcess; the partner didn't depend on its pair founder, so they raced and each formed a 1-node cluster. Added an &akka-founder-healthcheck (bash /dev/tcp probe of Akka 4053 — no nc/curl in the image) to site-a-1/site-b-1 and switched the partners to depends_on service_healthy, so the founder forms first and the partner JOINS it. (Carry the underlying simultaneous-cold-start property to Phase 7 — it is inherent to symmetric self-first seeding, present for central since #459.) Live gate record: docs/plans/2026-07-24-mesh-phase6-live-gate.md (legs 1-2 PASS — three isolated meshes, one Primary per pair at 250/240). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- docker-dev/docker-compose.yml | 73 ++++++++-- .../plans/2026-07-24-mesh-phase6-live-gate.md | 126 ++++++++++++++++++ 2 files changed, 190 insertions(+), 9 deletions(-) create mode 100644 docs/plans/2026-07-24-mesh-phase6-live-gate.md diff --git a/docker-dev/docker-compose.yml b/docker-dev/docker-compose.yml index 4d26abed..f0abd190 100644 --- a/docker-dev/docker-compose.yml +++ b/docker-dev/docker-compose.yml @@ -49,14 +49,19 @@ # site-a-1, site-a-2 OTOPCUA_ROLES=driver,cluster-SITE-A — driver-only # site-b-1, site-b-2 OTOPCUA_ROLES=driver,cluster-SITE-B — members of their # OWN 2-node mesh (self-first seeded within the pair, NOT -# off central). They serve no UI and authenticate no -# users; central manages + deploys to them over the +# off central). They serve no UI, but they DO expose OPC UA +# data-plane endpoints (4842-4845) and authenticate those +# users against the same shared GLAuth (via the *ldap-env +# anchor); central manages + deploys to them over the # ClusterClient/gRPC/FetchAndCache transports above, not # gossip. # # Auth is real LDAP against the shared GLAuth on the Linux Docker host # (10.100.0.35:3893, dc=zb,dc=local) — there is no LDAP container here. -# Only the admin-role central nodes carry the Security__Ldap__* block. +# EVERY host node carries the Security__Ldap__* block (shared *ldap-env anchor): +# the driver-only site nodes serve OPC UA endpoints too, and LDAP options are +# validated at boot on all of them (a node with no LDAP config crashes with +# OptionsValidationException — transport None + AllowInsecure false). # Sign in `multi-role` / `password`. # # traefik PathPrefix(`/`) → central-1 / central-2 (the single UI route). @@ -126,6 +131,43 @@ x-secrets-env: &secrets-env Secrets__Replication__AnnounceInterval: "00:00:05" ZB_SECRETS_MASTER_KEY: "${OTOPCUA_SECRETS_KEK:-ZYGhIX0luS/XsevpCB2W18jYHMcqO6AjM9oXy+T6Zp4=}" +# EVERY node that serves an OPC UA endpoint authenticates data-plane users against the shared GLAuth — +# that includes the driver-only site nodes (ports 4842-4845), not just the admin/central pair. LDAP +# options are validated at host start on ALL of them (Enabled=true by appsettings default, Transport=None, +# AllowInsecure=false ⇒ OptionsValidationException), so a node that carries no LDAP config crashes at boot. +# This anchor is merged into every host node's environment (via `<<: [*secrets-env, *ldap-env]`). Central +# still also lists the block inline for readability; the explicit keys win over the merge, same values. +x-ldap-env: &ldap-env + Security__Ldap__Enabled: "true" + Security__Ldap__DevStubMode: "false" + Security__Ldap__Server: "10.100.0.35" + Security__Ldap__Port: "3893" + Security__Ldap__Transport: "None" + Security__Ldap__AllowInsecure: "true" + Security__Ldap__SearchBase: "dc=zb,dc=local" + Security__Ldap__ServiceAccountDn: "cn=serviceaccount,dc=zb,dc=local" + Security__Ldap__ServiceAccountPassword: "serviceaccount123" + Security__Ldap__GroupToRole__ReadOnly: "ReadOnly" + Security__Ldap__GroupToRole__WriteOperate: "WriteOperate" + Security__Ldap__GroupToRole__WriteTune: "WriteTune" + Security__Ldap__GroupToRole__WriteConfigure: "WriteConfigure" + Security__Ldap__GroupToRole__AlarmAck: "AlarmAck" + +# Readiness gate for a pair FOUNDER node (site-a-1 / site-b-1): passes once its Akka port 4053 is bound. +# The partner (site-a-2 / site-b-2) waits on this via `depends_on: { condition: service_healthy }` so the +# founder forms its 2-node mesh FIRST and the partner JOINS it — instead of both self-first seeds racing +# and each forming its own single-node cluster (the split brain the Phase 6 live gate observed: 250/250, +# both Primary). `service_started` is NOT enough — it fires the instant the container launches, long +# before Akka binds. The image ships bash but no nc/curl, so the probe uses bash's /dev/tcp. start_period +# gives the founder a head start to not just bind but FORM (a 1-node cluster forms within the same second +# as the bind), so the partner's InitJoin lands on an existing cluster and is ack'd. +x-akka-founder-healthcheck: &akka-founder-healthcheck + test: ["CMD-SHELL", "bash -c ' **Exit gate for Phase 6** (mesh partition). Bring up the rewritten docker-dev rig +> (three independent 2-node Akka meshes) against **merged master** and prove every +> previously-live-gated behaviour *per pair*, plus the split itself. +> +> Merge landed first (`origin/master` @ `2839deb5`, "merge now, gate after"); this gate +> runs against the merged code and **fixes forward on master** if anything surfaces. + +**Rig topology** (docker-dev, local OrbStack): + +| Mesh | Nodes | Roles | OPC UA (host) | Seeds (self-first, pair-local) | +|---|---|---|---|---| +| MAIN | central-1, central-2 | `admin,driver,cluster-MAIN` | 4840 / 4841 | own pair only | +| SITE-A | site-a-1, site-a-2 | `driver,cluster-SITE-A` | 4842 / 4843 | own pair only | +| SITE-B | site-b-1, site-b-2 | `driver,cluster-SITE-B` | 4844 / 4845 | own pair only | + +- Akka port 4053 (per container); telemetry gRPC `:4056`; ConfigServe `:4055` (central); + LocalDb sync `:9001` (site-a); AdminUI `http://localhost:9200` (Traefik → central-1/2). +- Transports: `MeshTransport__Mode=ClusterClient` (all), `Telemetry__Mode=Grpc` (all six), + `TelemetryDial__Mode=Grpc` (central pair). Secrets replication pair-local. +- Deploy: `POST http://localhost:9200/api/deployments`, header `X-Api-Key: docker-dev-deploy-key`. + +--- + +## Legs + +| # | Leg | Status | Evidence | +|---|---|---|---| +| 1 | Three meshes form — each node sees ONLY its own pair (2 members) | ✅ PASS | each node handshakes ONLY its pair partner | +| 2 | Two+ Primaries, one per pair; ServiceLevel 250/240 within each pair | ✅ PASS | MAIN central-2=250/central-1=240; SITE-A site-a-1=250/site-a-2=240; SITE-B site-b-1=250/site-b-2=240 | +| 3 | Deploy reaches all clusters over per-cluster ClusterClient; seals green | PENDING | | +| 4 | Telemetry per cluster over gRPC (:4056); pills live; kill/reconnect ~5s | PENDING | | +| 5 | Reconciler quiet — no `EnabledRowNotInCluster` for foreign rows | PENDING | | +| 6 | Secrets pair-local — no cross-mesh replication / no unreachable-peer errors | PENDING | | +| 7 | Split validator — a site node flipped to `MeshTransport__Mode=Dps` refuses to boot | PENDING | | +| 8 | Failover within a pair — kill a Primary, Secondary takes over (auto-down 1-vs-1) | PENDING | | + +--- + +## Findings + +### FINDING 1 (fix-forward on master) — boot-crash: `StackOverflowException` on EVERY node + +**Symptom.** First `up` of the fresh Phase-6 images: all six nodes crash at startup with +`Stack overflow.` — an infinite recursion entering through +`ZB.MOM.WW.OtOpcUa.Cluster.ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap`. + +**Root cause (Phase 6, Task 2 wiring).** `Program.cs` resolved `IClusterRoleInfo` **eagerly, inside +the `AddAkka` configurator lambda**: + +```csharp +ab.WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService()); // line 385 +``` + +That lambda runs *while the `ActorSystem` is being constructed*, but `ClusterRoleInfo`'s ctor +**depends on the `ActorSystem`** (it is a live `Cluster.State` view). Resolving it there forces a +nested `ActorSystem` build → re-runs the same configurator → `sp.GetRequiredService()` +again → ∞. The cycle: +`WithOtOpcUaClusterBootstrap → GetService → ActorSystemFactory → WithOtOpcUaClusterBootstrap → …` + +**Why merge + build + tests + reviews all missed it.** It compiles cleanly (a *runtime* DI cycle, +not a compile error). `RedundancyStateSingletonRehomeTests` boots a real host but passes a hand-built +`FakeClusterRoleInfo` **directly** into the extension — it never exercised the `sp.GetRequiredService` +line in the composition root that overflows. The bug lived in `Program.cs`, which every test mocked +around. Only a real boot of the shipped image surfaces it — precisely the live gate's job. + +**Fix.** Derive the singleton's cluster-role scope from `AkkaClusterOptions` (pure config, no +ActorSystem) instead of `IClusterRoleInfo`: +- `BuildClusterRedundancySingletonOptions(AkkaClusterOptions)` + `WithOtOpcUaClusterRedundancySingleton(AkkaClusterOptions)` + — derive `Role = clusterOptions.Roles.FirstOrDefault(RoleParser.IsClusterRole) ?? RoleParser.Driver` + (identical to `ClusterRoleInfo.ClusterRole`, which derives from the same config). +- `Program.cs` passes `sp.GetRequiredService>().Value` — `IOptions<>` has + no ActorSystem dependency, breaking the cycle. +- Tests updated to pass `AkkaClusterOptions { Roles = … }`. 5/5 rehome tests pass; Host builds clean. + +Files: `Program.cs`, `ControlPlane/ServiceCollectionExtensions.cs`, `RedundancyStateSingletonRehomeTests.cs`. +Status: **FIXED + live-verified** — rebuilt image boots with zero stack-overflow lines; all six nodes +start; cluster singletons form (`redundancy-state` identified per mesh). 5/5 rehome tests pass. + +### FINDING 2 (fix-forward on master) — driver-only site nodes crash: LDAP options fail validation + +**Symptom.** site-a-1/site-a-2/site-b-1/site-b-2 crash at `Host.StartAsync` with +`OptionsValidationException: LDAP transport is None (plaintext) but AllowInsecure is false`. + +**Root cause (Task 7 rig).** Only central-1/central-2 carried the `Security__Ldap__*` env block; the +compose even asserted "Only the admin-role central nodes carry" it. But the site nodes serve OPC UA +endpoints (4842-4845) and LDAP options are validated at boot on every node (appsettings default +`Enabled=true`, `Transport=None`, `AllowInsecure=false` ⇒ throw). The site nodes overrode `environment` +wholesale (YAML `<<:` does not deep-merge `environment`), so they inherited no LDAP config. `docker +compose config` validates the file but never boots the app, so this passed the Task 7 acceptance check. + +**Fix (rig).** Extracted the LDAP block to a shared `&ldap-env` anchor merged into **every** host node +(`<<: [*secrets-env, *ldap-env]`); corrected the two misleading header comments. Site nodes now boot. + +### FINDING 3 (fix-forward on master) — site pairs split-brain at simultaneous startup (250/250) + +**Symptom.** Both nodes of each site pair logged `is JOINING itself ... and forming a new cluster` and +each elected itself Primary → ServiceLevel **250/250** instead of 250/240. Each pair was two independent +1-node clusters, not one 2-node mesh. + +**Root cause (Task 7 rig — a startup race, not a code defect).** Both pair nodes are self-first seeds +(`seed[0]=self`), so both run Akka's `FirstSeedNodeProcess` and can form alone. The partner +(site-a-2/site-b-2) only `depends_on` sql/central/migrator — **not its pair founder** — so the pair +started simultaneously and each formed its own cluster before discovering the other. The central pair +avoids this only because central-2 `depends_on central-1` (and even then won the InitJoin race by luck). +`depends_on: service_started` is insufficient — it fires when the container launches, before Akka binds. +This is an inherent property of symmetric self-first seeding (present for central since #459), now +extended to sites; **carry to Phase 7** as a production concern (simultaneous cold-start of both pair +VMs). NOTE: two separate 1-node clusters do NOT merge via SBR — auto-down resolves partitions of one +cluster, not two independent clusters. + +**Fix (rig).** Added an `&akka-founder-healthcheck` (bash `/dev/tcp` probe of Akka port 4053; the image +has no nc/curl) to site-a-1/site-b-1, and switched site-a-2/site-b-2 to `depends_on: { site-a-1: +service_healthy }`. The founder now binds + forms its mesh before the partner starts, so the partner +JOINS it. Verified: site-a-2/site-b-2 log `Welcome from ` (joined), ServiceLevel 250/240. + +--- + +## Evidence log + +- **Leg 1** — join handshakes: central-1↔central-2, site-a-1↔site-a-2, site-b-1↔site-b-2; no node + handshakes a node outside its pair. central *dials* site nodes over ClusterClient/telemetry (Akka + association + gRPC), which is the split-topology transport, NOT gossip membership. +- **Leg 2** — `Client.CLI redundancy` per endpoint: 4840 central-1=240, 4841 central-2=250, 4842 + site-a-1=250, 4843 site-a-2=240, 4844 site-b-1=250, 4845 site-b-2=240. One Primary per pair.