lmxopcua/archreview/plans/R3-04-one-role-authority-plan.md

# Design + Implementation Plan — R3-04 One Role Authority

- **Source reports:** `archreview/01-core-composition.md` §S-12 (`:198-210`) + §C-9 (`:264-278`) · `archreview/03-server-runtime.md` §C5 (`:131-133`) (2026-08-19 round-3 re-review at `12b0035a`)
- **Prior designs consumed:** none — both findings are **new in round 3**. The nearest ancestor is Gitea **#532** (`RoleParser.ResolveEffectiveRoles` + the `AddOtOpcUaCluster` PostConfigure), whose remediation created the second authority these findings are about; the field incident it closed is `docs/deployments/wonder-app-vd03-makino-z-34184.md`.
- **Plan verified against tree at:** master `7c77a8c7` (clean; code identical to `12b0035a`) — every anchor re-opened and re-checked 2026-08-20.
- **Scope:** OVERALL prioritized action item **#4** — "One role authority: `ResolveEffectiveRoles` normalizes + validates; `Program.cs` and all four validators consume it; boot guard asserting Akka and Host role views agree" (01 S-12/C-9, 03 C5). Effort Small / Impact Medium-High.
- **Cross-cutting requirement (OVERALL theme 4, "One rule stated twice diverges"):** the deliverable is not "fix two call sites" — it is that **the role list is derived in exactly one place and every consumer inherits it**, enforced by a guard test so the rule cannot be re-stated. The authoring-vs-deploy-gate arc (#535→#559) is the named precedent: one authority, enforced in the service, mirrored by the gate. Concrete pick here: a `EffectiveRoles` value type produced by one factory, consumed by `Program.cs`, the four `Cluster` validators, the Akka options PostConfigure and every gossip-side predicate; plus `RoleAuthorityGuardTests`, a source-scanning guard modelled on `CommandDisposalConventionsTests` / `StatusCodeParityTests`, that fails when any production file outside the authority reads `OTOPCUA_ROLES`, `Cluster:Roles`, or a bare `"admin"`/`"driver"` role literal.
- **Cross-cutting requirement (OVERALL theme 2, "verification that cannot see its own blind spot"):** the docker-dev rig **sets both role sources, consistently, on all six nodes** (`docker-compose.yml:298/352-354`, `:448/485-487`, `:564/606-607`, `:686/716-717`, `:782/803-804`, `:853/874-875`). Neither finding can reproduce on the rig as configured. The live gate below therefore **changes the rig** (site-b-2 becomes `Cluster:Roles`-only) so the divergence becomes observable — the same lesson 03/S14 taught about `DisableLogin=true`.

## Verification summary

Both findings **CONFIRMED at `7c77a8c7`** — neither stale, neither partially fixed. Every report anchor is **exact**; no drift. Anchor status:

| Finding | Report anchor | Status at `7c77a8c7` |
|---|---|---|
| 01/S-12 = 03/C5 | `Program.cs:61-63` (`RoleParser.Parse(Environment.GetEnvironmentVariable("OTOPCUA_ROLES"))` → `hasAdmin`/`hasDriver`) | **Exact.** `:61` resolve, `:62` `roles.Contains("admin")`, `:63` `roles.Contains("driver")` — **bare string literals, not `RoleParser.Admin`/`.Driver`**, a detail neither report names. The env-only array reaches **four** further decisions the reports do not enumerate: the per-role appsettings overlay filename (`:80`), the health-check registration (`:564` `AddOtOpcUaHealth(hasAdmin)`), the three dedicated gRPC listener ports (`:624-626`), and the boot banner (`:789-790`). |
| 01/S-12 = 03/C5 | `RoleParser.cs:115-125` / `:115-119` (`ResolveEffectiveRoles`) | **Exact.** Two-arg overload `:115-119` (`configured is { Count: > 0 }` ⇒ verbatim copy; else `Parse(envRoles)`), one-arg convenience `:124-125`, `RolesEnvVar` const `:128`, `Parse` `:133-152`. |
| 01/S-12 = 03/C5 | `Cluster/ServiceCollectionExtensions.cs` PostConfigure | **Exact**, now pinpointed: `:51` — `options => options.Roles = RoleParser.ResolveEffectiveRoles(options.Roles)`, applied inside `AddOtOpcUaCluster` (`:34-51`). The gossiped list is handed to Akka at `:267` (`Roles = options.Roles`). |
| 01/S-12 = 03/C5 | `RoleParser.cs:96-103` ("the exact knob the wonder-app-vd03 field fix hand-added") | **Exact** — the `<remarks>` paragraph is `:94-103`. |
| 01/C-9 | `SplitTopologyTransportValidator.cs:69-78` (the one validator that normalizes) | **Exact.** Comment `:69-72`; `configured` read `:73`; `ResolveEffectiveRoles` `:74`; **`normalized = Array.ConvertAll(effective, r => r.Trim().ToLowerInvariant())` `:76`**; classification `:78`, `:89-90`. |
| 01/C-9 | `MeshTransportOptionsValidator.cs:76-79` | **Exact** — `ResolveEffectiveRoles(GetSection("Cluster:Roles").Get<string[]>())` `:76-77`; case-sensitive `Array.IndexOf(roles, RoleParser.Driver)` `:78`, `.Admin` `:79`. |
| 01/C-9 | `TelemetryOptions.cs:124-126` (`TelemetryOptionsValidator`, class at `:88`) | **Exact** — resolve `:124-125`, case-sensitive `IndexOf` `:126`. Note the sibling `TelemetryDialOptionsValidator` (`:160`) reads **no** roles at all. |
| 01/C-9 | `ConfigSourceOptionsValidator.cs:101-104` | **Exact** — resolve `:101-102`, case-sensitive `IndexOf` `:103-104`. |
| 01/C-9 | "the four Cluster fail-fast validators" | **Correct as a count, but the brief's fourth candidate is wrong.** `AkkaClusterOptionsValidator` (`AkkaClusterOptionsValidator.cs:37`) reads **no roles** — it validates self-first seed ordering only. The four role-reading validators are `SplitTopologyTransportValidator`, `MeshTransportOptionsValidator`, `TelemetryOptionsValidator`, `ConfigSourceOptionsValidator`; all four are registered in `Cluster/ServiceCollectionExtensions.cs:56/63/70/80`. |
| 01/C-9 | "only the env fallback is lowercased/validated by `Parse`" | **Exact and load-bearing.** `Parse` `:137-141` lower-cases + trims + dedupes and `:143-149` throws `ArgumentException` on an unknown role. The configured path (`:117`) does **none** of that: `Cluster:Roles: ["Drivr"]` is accepted silently, gossiped verbatim, and matches nothing. |

**The blast radius is materially larger than either finding states, and the plan is sized to the larger one.** 01/C-9 describes the case hazard as "skips every driver-scoped boot-time rule". That is true and it is the *smaller* half. The same verbatim spelling is **gossiped to Akka** (`ServiceCollectionExtensions.cs:267`), and six production predicates match gossiped roles **ordinally** against the lowercase constant: `RedundancyStateActor.cs:146/193` (Primary election), `ManualFailoverService.cs:125`, `ClusterNodeAddressReconcilerActor.cs:185`, `DriverHostActor.cs:1612` (the R2-04 primary-gate member count), `ScriptedAlarmHostActor.cs:520`, and `Fleet.razor:149`. A pair configured `Cluster:Roles: ["Driver"]` therefore **elects no Primary at all** (`SelectOldestUpMemberOfRole(members, "driver")` matches nothing ⇒ every node Detached ⇒ the R2-04 data-plane gate denies every write, the alarm S&F queue never drains, ServiceLevel never rises) while `/hosts` cheerfully renders the role chip `Driver`. Akka's own `ClusterSingletonOptions.Role` matching (`ControlPlane/ServiceCollectionExtensions.cs:52` `AdminRole = "admin"`, `:247` `?? RoleParser.Driver`) is likewise ordinal and outside this repo's control — which is why the fix must normalize the **gossiped** plain-role strings, not only the local predicates.

Supporting facts verified: `RoleParser.IsClusterRole` is already case-**insensitive** (`:37-40`, Gitea #532) while `ClusterIdFromRole` (`:53`) deliberately preserves the operator's suffix spelling, with an XML `<remarks>` (`:44-52`) explicitly warning that normalizing one side "would silently rename the Akka role of a running deployment"; every in-repo ClusterId comparison is `OrdinalIgnoreCase` (`ConfigDbApiKeyStore.cs:144-152`, `DeploymentArtifact.ParseDataPlaneAuthority`, `AclScopeRef`); `ClusterRoleInfo` builds `_localRoles` as an `OrdinalIgnoreCase` HashSet (`ClusterRoleInfo.cs:45`) so `HasRole` is already case-insensitive but `LocalRoles` still exposes the verbatim strings; none of the four shipped `appsettings*.json` overlays carries a `Cluster` section (keys are `Serilog` + `Security` only), so the "the role overlay may not set `Cluster:Roles`" invariant this plan states costs zero migration; `Install-Services.ps1:177-182` writes **only** `OTOPCUA_ROLES` to the Windows service environment (`$normalisedRoles` is trimmed + sorted + deduped but **not** lower-cased) and `docs/deployments/wonder-app-vd03-makino-z-34184.md:46-60` records the field fix that hand-added `Cluster__Roles__0/1` beside it — so **prod runs both sources and `OTOPCUA_ROLES` must keep working**; the repo has an established source-scanning convention-test idiom (`CommandDisposalConventionsTests.cs:54-66`, walking up from `AppContext.BaseDirectory` to the `ZB.MOM.WW.OtOpcUa.slnx` root) and an established reflection-census idiom (`StatusCodeParityTests`, `PageAuthorizationGuardTests`, `DeferredSinkForwardingReflectionTests`); `EnvVarScope` + the `OTOPCUA_ROLES` xUnit collection (`tests/Core/…Cluster.Tests/EnvVarScope.cs:26-37`) already serialize every test class that touches the process env var — new tests must join that collection.

### Role-reading census (every production site, `7c77a8c7`)

"Source" is what the site actually reads. "Normalizes?" is whether **that site** trims/case-folds before deciding. `env` = `OTOPCUA_ROLES` via `RoleParser.Parse`; `cfg⇒env` = `ResolveEffectiveRoles` (configured-first, env fallback); `bound` = `AkkaClusterOptions.Roles` (post-`AddOtOpcUaCluster` PostConfigure, i.e. `cfg⇒env` verbatim); `gossip` = `Member.Roles` off live `Cluster.State`.

| # | Site | Source | Normalizes? | What it decides |
|---|---|---|---|---|
| 1 | `Host/Program.cs:61` | **env only** | lower+validate (via `Parse`) | the whole host role view |
| 2 | `Host/Program.cs:62` `roles.Contains("admin")` | env only | n/a (bare literal, ordinal) | `hasAdmin` |
| 3 | `Host/Program.cs:63` `roles.Contains("driver")` | env only | n/a (bare literal, ordinal) | `hasDriver` |
| 4 | `Host/Program.cs:80` `roleSuffix` | env only | no (ordinal sort of already-lowered) | which `appsettings.{role}.json` overlay loads |
| 5 | `Host/Program.cs:158/172/374/414/435/477/482/520/586/590/596/730/747/755/780` | env only (via 2/3) | n/a | **every** conditional registration + endpoint map |
| 6 | `Host/Program.cs:564` `AddOtOpcUaHealth(hasAdmin)` | env only | n/a | the ConfigDb health check |
| 7 | `Host/Program.cs:624-626` | env only | n/a | LocalDb-sync / ConfigServe / Telemetry listener ports |
| 8 | `Host/Program.cs:789-790` banner | env only | n/a | the operator-visible boot line |
| 9 | `Cluster/ServiceCollectionExtensions.cs:51` PostConfigure | **cfg⇒env** | **no** | `AkkaClusterOptions.Roles` — the authority for everything below |
| 10 | `Cluster/ServiceCollectionExtensions.cs:267` `Roles = options.Roles` | bound | no | **what Akka gossips** |
| 11 | `Cluster/SplitTopologyTransportValidator.cs:73-76,89-90` | cfg⇒env | **yes** (trim + lower, `:76`) | split-topology transport fail-fast |
| 12 | `Cluster/MeshTransportOptionsValidator.cs:76-79` | cfg⇒env | **no** | GrpcPsk / CentralGrpcEndpoints fail-fast |
| 13 | `Cluster/TelemetryOptions.cs:124-126` (`TelemetryOptionsValidator`) | cfg⇒env | **no** | Telemetry ApiKey / listen-port fail-fast |
| 14 | `Cluster/ConfigSourceOptionsValidator.cs:101-104` | cfg⇒env | **no** | driver-only ⇒ `FetchAndCache` fail-fast |
| 15 | `Cluster/ClusterRoleInfo.cs:45` `_localRoles` | bound | **yes** for lookups (`OrdinalIgnoreCase` set); **no** trim; `LocalRoles` exposes verbatim | `HasRole`, `/fleet` + `/hosts` role chips |
| 16 | `Cluster/ClusterRoleInfo.cs:52` `ClusterRolesIn` | bound | prefix case-insensitive; suffix verbatim | `ClusterId` / `ClusterRole` identity |
| 17 | `Cluster/ClusterRoleInfo.cs:116/123/142/150` | **gossip** | no | fleet-wide role→member index |
| 18 | `ControlPlane/ServiceCollectionExtensions.cs:24,52` `AdminRole = "admin"` | literal ⇒ Akka | n/a (Akka matches **ordinally** vs gossip) | which node hosts the 5 admin singletons |
| 19 | `ControlPlane/ServiceCollectionExtensions.cs:246-247` | bound | prefix case-insensitive; falls back to `RoleParser.Driver` | redundancy-singleton cluster scope |
| 20 | `ControlPlane/Redundancy/RedundancyStateActor.cs:146,193` | **gossip** | **no** (`Roles.Contains("driver")`) | **Primary election** |
| 21 | `ControlPlane/Redundancy/ManualFailoverService.cs:125` | **gossip** | **no** | manual failover target set |
| 22 | `ControlPlane/Fleet/ClusterNodeAddressReconcilerActor.cs:185` | **gossip** | **no** | `ClusterNode` row reconciliation |
| 23 | `Runtime/Drivers/DriverHostActor.cs:1612` | **gossip** | **no** | R2-04 primary-gate driver member count |
| 24 | `Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs:520` | **gossip** | **no** | scripted-alarm emit gate member count |
| 25 | `Runtime/ServiceCollectionExtensions.cs:323` `GetService<IClusterRoleInfo>()` | bound (via 15/16) | inherits 15 | runtime actor cluster identity |
| 26 | `ControlPlane/ServiceCollectionExtensions.cs:175` `roleInfo.ClusterId/.ClusterRole` | bound (via 16) | inherits 16 | reconciler own-cluster scoping |
| 27 | `Host/OpcUa/ConfigDbApiKeyStore.cs:83` `ResolveClusterId` | bound | prefix case-insensitive; compared `OrdinalIgnoreCase` (`:150`) | which cluster's ACL grants this node honours |
| 28 | `Host/Program.cs:427` / `:494` | bound | inherits 27 / 19 | ACL cluster filter; redundancy-singleton scope |
| 29 | `Host/Health/HealthEndpoints.cs:82` `RolePreference = [Admin, Driver]` | literals ⇒ shared Health lib vs gossip | no | `/health/active` node preference |
| 30 | `AdminUI/Components/Pages/Fleet.razor:149,159` `MembersWithRole("driver")` | **gossip** | no (bare literal) | the `/fleet` "Akka drivers" column |
| 31 | `AdminUI/Components/Pages/Hosts.razor:480,487` | **gossip** | display only (`OrdinalIgnoreCase` sort) | the `/hosts` role + leader-role chips |
| — | `scripts/install/Install-Services.ps1:177-182` | writes **env only** | trim + sort + dedupe, **not** lower | the prod Windows-service role knob |
| — | `docker-dev/docker-compose.yml` ×6 nodes | writes **both**, consistently | n/a | the rig — and the reason neither finding reproduces there |

**Census read:** 31 production sites. **8 read the env var only** (all in `Program.cs`, rows 1-8 — the whole of 01/S-12). **6 read `cfg⇒env`** and only **one of them normalizes** (rows 11-14 — the whole of 01/C-9, plus the PostConfigure at row 9 and the Akka handoff at row 10 which normalize nothing). **11 read the bound options**, mostly through `ClusterRoleInfo`, which is case-insensitive for lookups but verbatim for exposure. **6 match live gossip ordinally against a lowercase constant** (rows 17, 20-24, 30) — the half neither finding names, and the half that turns a miscased `Cluster:Roles` into a no-Primary outage. **2 deployment surfaces** set roles: prod (env only) and the rig (both).

**How the two knobs relate today, exactly.** There is **no** mapping of `OTOPCUA_ROLES` into `Cluster:Roles` in configuration — no `AddEnvironmentVariables` prefix trick, no alias. The two are read independently: `Program.cs:61` reads the env var directly, and `AddOtOpcUaCluster:51` post-configures the *bound* `Cluster:Roles` with `ResolveEffectiveRoles`, which falls back to the env var only when the bound array is empty. So the **documented precedence — `Cluster:Roles` wins outright when non-empty, else `OTOPCUA_ROLES`** (`RoleParser.cs:104-109`, `docs/Configuration.md:126`) — governs **only the Akka side**. The Host side has no precedence at all: it sees the env var and nothing else. The two views coincide on every node that sets both consistently (all six docker-dev nodes, and wonder-app-vd03 after its field fix) and diverge on exactly the shape #532 made first-class: `Cluster:Roles` set, `OTOPCUA_ROLES` unset. What operators are told: `docs/Configuration.md:29` ("the roles themselves come from the `OTOPCUA_ROLES` env var"), `:124` (`Roles` … "when empty, falls back to `OTOPCUA_ROLES`"), `:126` (the two-knobs note, which **already tells operators to prefer `Cluster:Roles`** — i.e. the docs currently recommend the broken configuration), `:449` (the env-var table), `docs/ServiceHosting.md:19-27` (role gating is "`Program.cs` reads `OTOPCUA_ROLES`"), `docs/Redundancy.md:211-231` (per-node identity example uses `OTOPCUA_ROLES=admin,driver`), `docs/v2/Cluster.md:68` ("these two should stay in sync" — stated as an operator obligation, which is the whole defect). `infra/` says nothing about roles.

### Open design choices — resolved in this plan

1. **Precedence** (keep two sources + reconcile · vs · deprecate `OTOPCUA_ROLES` · vs · deprecate `Cluster:Roles`): resolved to **keep both, one documented precedence — `Cluster:Roles` non-empty wins outright, else `OTOPCUA_ROLES`, else empty**, i.e. `ResolveEffectiveRoles`'s existing rule applied to *both* sides instead of one. Deprecating `OTOPCUA_ROLES` is rejected outright: `Install-Services.ps1:182` writes it to the Windows service key, it is the only role knob on every non-field-fixed production install (memory `reference_servecli_wonder_app_vd03`: 23 env vars on the service key), and it names the appsettings overlay — a deprecation would break every existing deployment for a cosmetic gain. Deprecating `Cluster:Roles` is rejected because it is what Akka binds and what #532's own remediation made first-class. A **Warning** (not an error) when both are set and their normalized sets differ is added instead (choice 5).
2. **Case normalization scope** (normalize the whole gossiped list · vs · normalize only local predicates · vs · a two-layer value type): resolved to **two layers in one type**. `EffectiveRoles` carries a **`Gossiped` list** (what Akka receives) in which *plain* roles are lower-cased but a **cluster role keeps its ClusterId suffix verbatim** (`Cluster-SITE-A` ⇒ `cluster-SITE-A`), and exposes **case-insensitive predicates** (`HasAdmin`/`HasDriver`/`Has`) for every decision. Rationale: lower-casing a plain role changes the gossiped string **only for a node that is already broken** (a node spelled `driver` today is byte-identical after the change), while lower-casing a ClusterId suffix would rename `cluster-MAIN`/`cluster-SITE-A` on a running fleet — precisely what `RoleParser.ClusterIdFromRole`'s `<remarks>` (`:44-52`) warns against, and a rolling upgrade of a pair would gossip two different role names mid-window and split it. Normalizing predicates *only* is insufficient because Akka's own `ClusterSingletonOptions.Role` match (rows 18/19) is ordinal and not ours to change.
3. **Structural identity vs a runtime assert** (the OVERALL action item's "boot guard asserting Akka and Host role views agree"): resolved to **make them the same object**, and the runtime assert is then **not written**. `Program.cs` resolves `EffectiveRoles` once and registers it as a singleton; `AddOtOpcUaCluster` sets `options.Roles = effectiveRoles.Gossiped` from that instance; the four validators take the instance by constructor injection. There is then no second derivation to disagree with, so an assert comparing two views would compare an object to itself — a test that can never fail is worse than no test, because it reads as coverage. What *is* kept is the one genuinely non-redundant residual check (choice 4). Note also that a literal "Akka gossip vs Host view" assert is not implementable where it would be wanted: gossiped roles do not exist until the `ActorSystem` is built, and resolving anything ActorSystem-shaped during composition is the Phase-6 boot-`StackOverflow` trap (CLAUDE.md; `ControlPlane/ServiceCollectionExtensions.cs:239-248`).
4. **The residual divergence surface — the config-layer ordering trap.** `Program.cs` must resolve roles **before** it adds the per-role overlay (`:80` needs the role names to pick the filename), so the authority reads the *pre-overlay* configuration (base `appsettings.json` + env + args). If a role overlay — or a `${secret:}` expansion — ever set `Cluster:Roles`, the post-overlay bind would differ from the instance every consumer holds. Resolved by **stating the invariant** ("`Cluster:Roles` may not be set in a role overlay") and enforcing it with a **narrow** fail-fast added to the existing `AkkaClusterOptionsValidator`: if the bound `Roles` (post-overlay) normalizes to a different set than the registered `EffectiveRoles`, fail host start naming both. This is 5 lines, is *not* the redundant assert of choice 3 (it compares two genuinely different config snapshots), and is a no-op today — none of the four shipped overlays carries a `Cluster` section.
5. **Unknown / miscased role handling** (silent · warn · fail): resolved to **fail host start on an unknown role from either source**, matching what `Parse` already does for the env path (`:143-149`) and closing the configured path's total absence of validation. The message follows the house validator style (`MeshTransportOptionsValidator.cs:49-55`): name the key, the offending value, the source it came from, and the allowed set. A *miscased but known* role does **not** fail — it normalizes silently, because failing it would brick every node an operator authored as `Driver` at exactly the moment they upgrade. Separately, a **Warning** fires when both sources are set and their normalized sets differ (the `Cluster:Roles`-wins case an operator almost certainly did not intend).
6. **Gossip-side predicates** (leave them ordinal · vs · one case-insensitive helper): resolved to a **shared `ClusterRoles.Has(IEnumerable<string> gossipedRoles, string role)` helper** (`OrdinalIgnoreCase`) adopted at all six ordinal gossip sites (rows 20-24, 30). With choice 2 in place a *local* node can no longer gossip a miscased plain role, so this is **defence in depth for a peer running an older build** — precisely the rolling-upgrade window choice 2 is designed to survive. Cheap, and it removes the last bare `"driver"` literals from production code so the guard test (choice 7) can be strict.
7. **Guard style** (reflection census · vs · source scan · vs · Roslyn analyzer): resolved to a **source scan**, because the thing being guarded is a *textual* read (`Environment.GetEnvironmentVariable("OTOPCUA_ROLES")`, `GetSection("Cluster:Roles")`, `"driver"`) that reflection cannot see — the same limitation `StatusCodeParityTests`' own XML doc calls out ("an inline literal is invisible to this guard"). The repo already has the idiom and the root-walk helper (`CommandDisposalConventionsTests.cs:54-66`). A Roslyn analyzer was considered and rejected as disproportionate for two forbidden strings.
8. **Rig fidelity** (leave docker-dev alone · vs · make one node `Cluster:Roles`-only): resolved to **change site-b-2 to `Cluster:Roles`-only** (drop its `OTOPCUA_ROLES`). site-b is already the repo's designated A/B *control* node for the bootstrap guard, so it is the established place to pin a configuration variant, and site-b-2 is a driver-only node whose failure mode (deploy times out) is exactly what 01/S-12 predicts. Before the fix that node must **fail** the gate; after it, pass. Without this the gate is theatre — see theme 2.

---

## Finding 1 — 01/S-12 = 03/C5: two role authorities can wire a node inconsistently (Medium, two independent reviewers)

### Restatement

`Program.cs:61-63` derives `hasAdmin`/`hasDriver` — the switches for the entire conditional host wiring, ~15 registration blocks, the health check, three gRPC listener ports and the appsettings overlay — **solely from `OTOPCUA_ROLES`**. The Akka cluster roles resolve configured-first through `RoleParser.ResolveEffectiveRoles` (`RoleParser.cs:115-119`), applied to the bound options by `AddOtOpcUaCluster`'s PostConfigure (`Cluster/ServiceCollectionExtensions.cs:51`). A node configured **only** through `Cluster:Roles` — the shape #532's own remediation made first-class, and the shape `docs/Configuration.md:126` currently *recommends* — joins the mesh advertising `driver` (and `cluster-*`), passes `SplitTopologyTransportValidator` and `ConfigSourceOptionsValidator` (which read effective roles), becomes an expected-ack member of every deployment via `ConfigPublishCoordinator`'s enabled-`ClusterNode`-rows set, and **times out every deployment forever** because no driver stack was registered: no `WithOtOpcUaRuntimeActors`, no LocalDb, no OPC UA endpoint, no ConfigDb, and the deny-all `NullOpcUaUserAuthenticator`. It can even be the oldest role-carrying member for singleton placement without running the singleton manager. This is the exact inverse of the #532 incident, and nothing cross-checks the two sources.

### Verification

Confirmed exact (see summary table). Precisely, at `7c77a8c7`:

- `Program.cs:61` — `var roles = RoleParser.Parse(Environment.GetEnvironmentVariable("OTOPCUA_ROLES"));`, with the comment "Roles drive the entire conditional wiring below" one line above. Nothing else in `Program.cs` reads `Cluster:Roles`; the file's only other role reads are the *bound* `AkkaClusterOptions` at `:427` and `:494`, both for cluster **identity** (ClusterId / singleton scope), never for wiring.
- `:62-63` — `roles.Contains("admin")` / `roles.Contains("driver")` against **bare literals**, not `RoleParser.Admin`/`.Driver`. (A second, independent brittleness: these two literals are invisible to any constant-based census.)
- `:80` — the overlay filename is `string.Join('-', roles.OrderBy(...))`, so a `Cluster:Roles`-only node also loads **no role overlay** — a quiet second consequence neither report names.
- `Cluster/ServiceCollectionExtensions.cs:34-51` — the PostConfigure. `:267` hands `options.Roles` to Akka's `ClusterOptions`.
- Deploy-path consequence re-verified against CLAUDE.md's Phase-1 note: `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, and `Enabled = 0` is rejected by `DraftValidator.ValidateClusterTopology`; so a mis-wired node cannot even be excluded without the `MaintenanceMode` hatch. "Times out every deploy" is fleet-scoped, not node-scoped.
- Rig blindness re-verified: all six docker-dev nodes set `OTOPCUA_ROLES` **and** `Cluster__Roles__*` to the same values (`:298`/`:352-354`, `:448`/`:485-487`, `:564`/`:606-607`, `:686`/`:716-717`, `:782`/`:803-804`, `:853`/`:874-875`).

### Root cause

`Program.cs`'s role read predates the cluster options entirely and was never revisited when #532 introduced the fallback. #532 closed the env-only→**no** Akka roles direction by teaching the *Akka* side about the env var; it did not teach the *Host* side about `Cluster:Roles`, so the fix created a second authority rather than collapsing to one. The structural reason it stayed hidden is ordering: `Program.cs` needs roles at line 61, before the host is built and before any options bind, so reaching for `Environment.GetEnvironmentVariable` is the path of least resistance — and the configuration object that could answer the question (`builder.Configuration`, available from line 66) is created five lines later.

### Proposed design

One authority, one object, consumed everywhere.

**The type (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/EffectiveRoles.cs`, new):**

```csharp
/// <summary>The node's effective cluster roles — the SINGLE authority for "what is this node".
/// Resolved once, from one documented precedence, and consumed by BOTH the Host's conditional
/// wiring and the Akka role list, so the two cannot disagree (archreview 01/S-12 = 03/C5).</summary>
/// <remarks>
///     <para><b>Precedence:</b> <c>Cluster:Roles</c> when non-empty, else <c>OTOPCUA_ROLES</c>,
///     else empty. Unchanged from <see cref="RoleParser.ResolveEffectiveRoles"/> — what changed is
///     that <c>Program.cs</c> now obeys it too, instead of reading the environment directly.</para>
///     <para><b>Normalization is two-layer, on purpose.</b> <see cref="Gossiped"/> lower-cases plain
///     roles (Akka matches gossiped role names ORDINALLY, and every consumer compares against a
///     lowercase constant) but keeps a cluster role's ClusterId suffix VERBATIM — lower-casing that
///     would rename <c>cluster-SITE-A</c> on a running fleet, and a rolling pair upgrade would gossip
///     two different role names mid-window. Lower-casing a plain role changes the wire only for a
///     node that is already broken.</para>
/// </remarks>
public sealed class EffectiveRoles
{
    public static EffectiveRoles Resolve(IConfiguration configuration);      // the one factory
    public static EffectiveRoles Resolve(IReadOnlyList<string>? configured, string? envRoles);

    public IReadOnlyList<string> Gossiped { get; }   // what AkkaClusterOptions.Roles becomes
    public RoleSource Source { get; }                // Configured | Environment | None
    public bool HasAdmin { get; }
    public bool HasDriver { get; }
    public bool HasDev { get; }
    public bool Has(string role);                    // OrdinalIgnoreCase
    public string? ClusterRole { get; }              // verbatim suffix
    public string? ClusterId { get; }
}
```

- `Resolve` **validates both sources**: every entry must be `admin`/`driver`/`dev` (case-insensitive after trim) or a well-formed `cluster-<non-empty>`; otherwise it throws with the house message shape — `"Cluster:Roles[1] = 'Drivr' is not a known role. Allowed: admin, driver, dev, or 'cluster-<ClusterId>'."` — naming the key that supplied it.
- `RoleParser.ResolveEffectiveRoles` is **retained** as a thin delegation (`=> EffectiveRoles.Resolve(configured, env).Gossiped.ToArray()`) so nothing outside this plan's touched files breaks, and its XML doc points at the new authority. `RoleParser.Parse` stays as the env-string parser `Resolve` uses.

**`Program.cs` (`:61-63`):**

```csharp
// The SINGLE role authority: Cluster:Roles first, OTOPCUA_ROLES as the fallback (archreview 01/S-12).
// Resolved off builder.Configuration BEFORE the per-role overlay is added, because the overlay's
// FILENAME depends on the answer — which is why a role overlay may not itself set Cluster:Roles
// (AkkaClusterOptionsValidator fails start if one ever does).
var builder = WebApplication.CreateBuilder(args);
var effectiveRoles = EffectiveRoles.Resolve(builder.Configuration);
var hasAdmin = effectiveRoles.HasAdmin;
var hasDriver = effectiveRoles.HasDriver;
builder.Services.AddSingleton(effectiveRoles);
```

`WebApplication.CreateBuilder(args)` moves **above** the role read (it is currently at `:66`). `:80`'s `roleSuffix` uses `effectiveRoles.Gossiped`; `:789-790`'s banner gains `source={Source}` so the boot line states *which knob won* — the single cheapest operator-visible artifact of this change.

**`AddOtOpcUaCluster` (`Cluster/ServiceCollectionExtensions.cs:34-51`):** the PostConfigure consumes the registered instance rather than re-deriving —

```csharp
services.TryAddSingleton(sp => EffectiveRoles.Resolve(sp.GetRequiredService<IConfiguration>()));
services.AddOptions<AkkaClusterOptions>().PostConfigure<EffectiveRoles>(
    (options, roles) => options.Roles = [.. roles.Gossiped]);
```

`TryAddSingleton` keeps `AddOtOpcUaCluster` usable from harnesses that never ran `Program.cs`. **This is the structural identity**: `AkkaClusterOptions.Roles` is now *by construction* the same list `hasDriver` was computed from — there is no second derivation, hence no assert (resolved choice 3).

**Residual guard (`AkkaClusterOptionsValidator`, existing, `ValidateOnStart`):** if the bound `Roles` (post-overlay) normalizes to a different **set** than the injected `EffectiveRoles.Gossiped`, fail with `"Cluster:Roles was changed by a configuration layer applied after role resolution (host view: [driver], bound: [admin,driver]). A per-role appsettings overlay may not set Cluster:Roles."` No-op today (choice 4).

**Alternatives considered:**
- *One-line fix: `Program.cs:61` calls `RoleParser.ResolveEffectiveRoles(builder.Configuration.GetSection("Cluster:Roles").Get<string[]>())`* — the fix both reports propose. Rejected as the whole answer: it closes S-12 and leaves C-9, leaves the two bare `"admin"`/`"driver"` literals, leaves the configured path unvalidated, and — critically — leaves **two derivations of the same rule** (Program's and the PostConfigure's), which is the exact disease theme 4 names. It is, however, the correct *first commit* (T3 below) and is sequenced as such.
- *Deprecate `OTOPCUA_ROLES` with a warning* — rejected (choice 1).
- *Map `OTOPCUA_ROLES` into configuration as `Cluster:Roles` via a custom `IConfigurationSource`* — attractive (it would make one key, not two), but rejected: the env var is comma-separated while `Cluster:Roles` is an indexed array, so the mapping needs a bespoke provider whose ordering against `AddEnvironmentVariables`/`AddCommandLine` is subtle, and it would silently change what `Cluster:Roles` reads as in every diagnostic dump. A value type is simpler and states the precedence in code where a reader will find it.
- *A runtime boot guard comparing Akka gossip to the Host view* — rejected (choice 3), replaced by the narrower config-layer guard.

### Implementation steps

1. `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/EffectiveRoles.cs` (new) — the type, `Resolve` ×2, normalization, validation, `RoleSource` enum.
2. `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs` — `ResolveEffectiveRoles` ×2 delegate to `EffectiveRoles`; XML docs re-point at the authority; `Parse` unchanged.
3. `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` — `TryAddSingleton<EffectiveRoles>`; PostConfigure at `:51` consumes it.
4. `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` — `:61-63` rewrite (builder first), `:80` overlay, `:789-790` banner + `source=`; register the singleton.
5. `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptionsValidator.cs` — the post-overlay set-equality guard.

### Tests

- `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/EffectiveRolesTests.cs` (new; joins the `OTOPCUA_ROLES` collection via `EnvVarScope`): precedence matrix (configured-only / env-only / both-agree / both-differ / neither); `Gossiped` normalization (`" Driver "` ⇒ `driver`; `Cluster-SITE-A` ⇒ `cluster-SITE-A` — **suffix case preserved**, the falsification control for choice 2); `HasDriver` true for every casing; unknown role from **either** source throws naming the key; `Source` reports the winner.
- `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/AkkaClusterOptionsRoleFallbackTests.cs` (existing, 5 cases) — must stay green unchanged; it is the #532 regression net.
- `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ClusterRolesOnlyBootTests.cs` (new, on the `DriverOnlyNoConfigDbBootTests` direct-host-build pattern — **not** `WebApplicationFactory`, per `TwoNodeClusterHarness.cs:48`): build the host with `Cluster:Roles = ["driver"]` and **`OTOPCUA_ROLES` unset**, assert the driver stack is present (`ILocalDb` singleton resolvable, `IOpcUaUserAuthenticator` is `ApiKeyOpcUaUserAuthenticator` not `NullOpcUaUserAuthenticator`, the runtime actors registered) and that `IOptions<AkkaClusterOptions>.Value.Roles` equals `["driver"]`. **Negative control:** the same test against the pre-fix build must FAIL — record that it does.
- `AkkaClusterOptionsValidatorTests` — one case: a post-role-resolution config layer setting `Cluster:Roles` fails start; the no-overlay case passes.

### Effort + Risk

**Effort:** S. **Risk:** **medium-high — this changes what every node wires and what it gossips.** The change is behaviour-preserving for every node that sets both sources consistently (all six rig nodes, every current prod install) and behaviour-*correcting* for the two broken shapes. The residual risks are (a) the `WebApplication.CreateBuilder` move ahead of the role read — mechanical but it reorders the first ten lines of the composition root, and (b) a node whose `Cluster:Roles` and `OTOPCUA_ROLES` genuinely differ today will now wire from `Cluster:Roles`, which is a real behaviour change the Warning in choice 5 is there to announce. Land T1-T3 behind the full `Cluster.Tests` + `Host.Tests` suites, and do not merge without the live gate (T13) — a role-wiring regression is a total node outage and the rig is the only place it shows.

---

## Finding 2 — 01/C-9: role normalization is stated in one validator of four (Medium)

### Restatement

`SplitTopologyTransportValidator` normalizes roles (trim + `ToLowerInvariant`, `:76`) and its comment names the exact hazard: "the `Cluster:Roles` bind path is verbatim (no lowercasing)… so 'Driver' / 'Admin' cannot slip past the ordinal checks" (`:69-72`). Its three siblings do the opposite: `MeshTransportOptionsValidator.cs:78-79`, `TelemetryOptions.cs:126`, and `ConfigSourceOptionsValidator.cs:103-104` all run case-sensitive `Array.IndexOf(roles, RoleParser.Driver)` over `ResolveEffectiveRoles`, which returns configured roles **verbatim** (`RoleParser.cs:117` — only the env fallback is lowercased/validated by `Parse`). A node configured `Cluster:Roles: ["Driver"]` silently skips every driver-scoped boot-time rule (GrpcPsk-required, CentralGrpcEndpoints-required, FetchAndCache-required) and fails later at runtime with precisely the absence-class failures these validators exist to fail fast on. Nothing validates unknown or miscased plain role names on the configured path at all.

### Verification

Confirmed exact (see summary table), and **the report under-states the consequence**. The same verbatim string is gossiped (`Cluster/ServiceCollectionExtensions.cs:267`), and six production predicates match gossip ordinally against the lowercase constant — census rows 17, 20-24, 30. Concretely, a pair configured `Cluster:Roles: ["Driver", "cluster-SITE-A"]`:

- `RedundancyStateActor.SelectOldestUpMemberOfRole(members, "driver")` (`:146`) matches **no member** ⇒ no Primary is ever elected ⇒ `IsDriverPrimary` false on both nodes ⇒ (R2-04) every operator write is denied `"not primary"`, alarm acks are dropped, the alerts fan-out never publishes, and the LocalDb alarm store-and-forward queue **never drains** (`HistorianDrainState.NotPrimary` on both). `ServiceLevel` never rises.
- `DriverHostActor.cs:1612` / `ScriptedAlarmHostActor.cs:520` count **0** driver members, so the R2-04 primary gate takes its single-node "allow" branch on a genuine 2-node pair — the exact dual-primary window R2-04 exists to close, silently re-opened by a spelling.
- `ClusterNodeAddressReconcilerActor.cs:185` sees no driver members ⇒ Error-logs `EnabledRowNotInCluster` for every row, forever.
- `/hosts` (`Hosts.razor:487`) renders the chip `Driver` and `/fleet` (`Fleet.razor:149`) shows zero Akka drivers — an operator sees a healthy-looking node in a fleet that reports none.

Also verified: `AkkaClusterOptionsValidator` — the fourth validator the task brief guessed at — reads no roles at all; the four are as listed above. And `TelemetryDialOptionsValidator` (`TelemetryOptions.cs:160`) reads none, so the central/dial side is unaffected.

### Root cause

Two derivations of one rule, at two different times. `SplitTopologyTransportValidator` landed with Phase 6 (`344b0d33`) and normalized because its author reasoned about the two role sources explicitly; the other three copied the `ResolveEffectiveRoles` + `Array.IndexOf` shape without the normalization, which was the *newer* and less obvious half. Nothing structural connects the four — each re-derives "is this a driver node" from a raw array — so the correct spelling in one file is invisible to the next author. The deeper cause is the same as Finding 1's: normalization lives at the consumers instead of at the authority.

### Proposed design

**Delete the derivation from all four validators; inject the authority.** Each validator's ctor takes `EffectiveRoles` instead of (or alongside) `IConfiguration`, and the four role lines collapse to `roles.HasDriver` / `roles.HasAdmin`. `SplitTopologyTransportValidator` additionally uses `roles.ClusterRole` / `roles.ClusterId` in place of its private `FindIndex`/`ClusterIdFromRole` pair — its message keeps the operator's original spelling because `ClusterId` preserves the suffix verbatim by design (choice 2). Its `:59-72` comment block, which is now the *history* of a rule the authority states, shrinks to a one-line pointer.

**Gossip-side defence in depth (choice 6).** Add to `RoleParser`:

```csharp
/// <summary>True when a GOSSIPED role list carries <paramref name="role"/>, compared
/// case-insensitively. Akka compares role names ordinally and a PEER may be running an older
/// build that gossips a verbatim 'Driver' (archreview 01/C-9), so every predicate over
/// Member.Roles goes through here rather than List.Contains.</summary>
public static bool GossipedRolesHave(IEnumerable<string>? gossipedRoles, string role);
```

Adopted at `RedundancyStateActor.cs:193`, `ManualFailoverService.cs:125`, `ClusterNodeAddressReconcilerActor.cs:185`, `DriverHostActor.cs:1612`, `ScriptedAlarmHostActor.cs:520`, `Fleet.razor:149`, and `SelectOldestUpMemberOfRole` (`RedundancyStateActor.cs:146`). `ClusterRoleInfo`'s gossip index (`:116/123/142/150`) becomes an `OrdinalIgnoreCase`-keyed dictionary.

**The guard that stops it recurring (choice 7).** New `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/RoleAuthorityGuardTests.cs`, scanning `src/**/*.cs` + `src/**/*.razor` from the solution root (the `CommandDisposalConventionsTests.cs:54-66` root-walk):

- **No file except `EffectiveRoles.cs` and `RoleParser.cs` contains `OTOPCUA_ROLES`** (outside a comment or XML doc).
- **No file except `EffectiveRoles.cs` contains `"Cluster:Roles"`** as a configuration key (outside a comment / an error-message string literal — the scan strips `//` and `///` lines and allows occurrences inside a string that also contains a space, which is how the validators' operator messages read).
- **No file except `RoleParser.cs` declares or compares a bare `"admin"` / `"driver"` / `"dev"` role literal** — the constants exist for this. This is the check that catches `Program.cs:62-63` and `Fleet.razor:149` today.
- Each rule reports **every** offending `file:line`, not the first, and the XML doc states the escape hatch (add the file to the small, named allow-list *with a reason*) so the guard is maintainable rather than deleted at the first inconvenience.

**Alternatives considered:**
- *Normalize inside `ResolveEffectiveRoles` and leave the four validators alone* — the fix 01/C-9 itself proposes. Rejected as insufficient on its own: it fixes the four validators (they inherit) but does nothing for the six ordinal **gossip** predicates, which are the half with the outage-grade consequence, and it leaves four re-derivations in place for the next rule to diverge on. It is, however, subsumed — the authority *does* normalize; the validators are converted anyway so the derivation stops existing.
- *Make every gossip predicate case-insensitive and skip normalizing the gossiped list* — rejected: Akka's own singleton `Role` matching (rows 18/19) is ordinal and not ours to intercept, so a miscased node would still fail to host its singletons.
- *A Roslyn analyzer forbidding the literals* — rejected as disproportionate (choice 7); the repo's analyzer project exists but the payload here is two strings.
- *`StringComparer.OrdinalIgnoreCase` on the raw arrays at each of the four validators* — rejected: that is the rule stated four more times, which is the finding.

### Implementation steps

1. `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/SplitTopologyTransportValidator.cs` — ctor takes `EffectiveRoles`; `:59-90` collapses; comment shrinks to a pointer.
2. `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptionsValidator.cs` — same, `:71-79`.
3. `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/TelemetryOptions.cs` (`TelemetryOptionsValidator`) — same, `:119-126`.
4. `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptionsValidator.cs` — same, `:95-104`.
5. `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs` — `GossipedRolesHave`.
6. `RedundancyStateActor.cs:146,193` · `ManualFailoverService.cs:125` · `ClusterNodeAddressReconcilerActor.cs:185` · `DriverHostActor.cs:1612` · `ScriptedAlarmHostActor.cs:520` · `AdminUI/Components/Pages/Fleet.razor:149,159` · `ClusterRoleInfo.cs:116-150` — adopt it.
7. `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs:56/63/70/80` — the four `AddValidatedOptions` registrations resolve the new ctor dependency (no signature change at the extension level; the validators are DI-constructed).

### Tests

- `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/RoleCaseVariantValidatorTests.cs` (new): a `[Theory]` over `"Driver"`, `" driver "`, `"DRIVER"`, `"Admin"`, `" cluster-SITE-A "` × **all four validators**, asserting each fires the *same* failure it fires for the canonical spelling. **The load-bearing case is the failure**, not the pass: each variant must produce a `ValidateOptionsResult.Fail` naming the same key, because a validator that never runs its driver branch also "passes". Add the mirrored canonical-spelling case as the control.
- `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/RoleAuthorityGuardTests.cs` (new, choice 7) — and it must be **shown to fail on the pre-fix tree** (`Program.cs:62-63` and `Fleet.razor:149` are live violations today); record the pre-fix offender list in the commit body.
- `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/GossipedRoleMatchTests.cs` (new): `GossipedRolesHave(["Driver"], "driver")` true; `(["cluster-SITE-A"], "cluster-site-a")` true; null/empty false.
- `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests` — one `RedundancyStateActor` case where the member gossips `"Driver"` and a Primary is still elected (the outage this finding actually causes; a pure-unit test over the selection helper is sufficient and deterministic).
- Existing suites stay green unchanged: `SplitTopologyTransportValidatorTests` (16 cases incl. the two `OTOPCUA_ROLES`-source cases at `:238`/`:258`), `MeshTransportOptionsValidatorTests` (incl. `:308`/`:329`), `ConfigSourceOptionsValidatorTests`, `RoleParserTests`, `AkkaClusterOptionsRoleFallbackTests`. Their construction changes from `new XValidator(config)` to `new XValidator(EffectiveRoles.Resolve(config))` — mechanical, and it keeps them testing the validator rather than the resolution.

### Effort + Risk

**Effort:** S-M (the four validators are ~4 lines each; the gossip-predicate sweep is 8 files; the guard test is the bulk). **Risk:** low-medium. The validators' behaviour is strictly widened (they now fire on inputs they used to skip), which can only turn a silent misconfiguration into a loud one — but that means **a node an operator authored as `Cluster:Roles: ["Driver"]` will now refuse to start** if it lacks its driver-scoped keys, where before it booted and misbehaved. That is the intent, and it is exactly the kind of change the live gate must exercise before merge. The gossip-side helper is behaviour-preserving for every correctly-spelled fleet.

---

## Docs to update

| File | Change |
|---|---|
| `docs/Configuration.md:29` | "The roles themselves come from the `OTOPCUA_ROLES` env var" ⇒ "…from `Cluster:Roles`, falling back to `OTOPCUA_ROLES`" (the overlay is now chosen from the effective roles). |
| `docs/Configuration.md:124` (`Cluster:Roles` row) | Add: values are trimmed and (for plain roles) lower-cased; a cluster role's ClusterId keeps its spelling; an unknown role **fails host start**. |
| `docs/Configuration.md:126` (the two-knobs note) | **Rewrite.** It currently says the two knobs drive *different* things ("`OTOPCUA_ROLES` drives all of `Program.cs`'s conditional wiring; `Cluster:Roles` becomes the gossiped Akka roles") — that description is the defect. Replace with: one authority, one precedence, both consumers; the boot banner names the winning source; setting both to different values logs a Warning; a role overlay may not set `Cluster:Roles`. Keep the #532 history as a short parenthetical. |
| `docs/Configuration.md:449` (env-var table) | `OTOPCUA_ROLES` row: read by `EffectiveRoles.Resolve` only, as the fallback when `Cluster:Roles` is empty. |
| `docs/ServiceHosting.md:19-27` | "`Program.cs` reads `OTOPCUA_ROLES`" ⇒ resolves `EffectiveRoles`; the role table and the "Single-node dev / Production" line get the precedence sentence. |
| `docs/Redundancy.md:211-231` | The per-node identity example shows `OTOPCUA_ROLES=admin,driver`; add the `Cluster__Roles__0/1` equivalent and state that either works, both together must agree. |
| `docs/v2/Cluster.md:68` | Delete "these two should stay in sync" — the operator obligation this plan removes. |
| `scripts/install/Install-Services.ps1` | Doc-comment only: `-Roles` writes `OTOPCUA_ROLES`, which is now the documented fallback rather than the primary knob; no behaviour change (choice 1 — do not break it). |
| `CLAUDE.md` | One paragraph under the cluster/redundancy material: the role authority, the precedence, the guard test, and the "a role overlay may not set `Cluster:Roles`" invariant. |

---

## Task breakdown

Branch: one branch `fix/archreview-r3-04-one-role-authority` off master. Ordering is genuinely sequential for T1→T4 (each consumes the previous type), then the sweep and guards fan out. Verification bar: every task has a deterministic unit guard; T3, T4 and T9 additionally carry falsification controls (a test that must FAIL on the pre-fix tree); T13 is the merge-blocking live gate. Follow TDD: write the failing test, see it FAIL, implement, see it PASS.

---

### T1 — `EffectiveRoles` value type: one factory, one precedence, two-layer normalization, both-source validation

**Classification:** HIGH-RISK (defines the node's identity; every downstream decision inherits it)
**Estimated implement time:** 10 min
**Parallelizable with:** T8 (guard-test scaffolding only); NOT T2-T7
**Files:** `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/EffectiveRoles.cs` (new), `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs`, `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/EffectiveRolesTests.cs` (new)

1. Write `EffectiveRolesTests` per Finding 1's Tests list — precedence matrix, `Gossiped` normalization **including the `cluster-SITE-A` suffix-preservation control**, `Has*` case-insensitivity, unknown-role-throws from **both** sources, `Source`. Join the `OTOPCUA_ROLES` xUnit collection (`EnvVarScope`).
2. `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~EffectiveRolesTests"` → **FAIL** (no such type).
3. Implement `EffectiveRoles` + `RoleSource`; make `RoleParser.ResolveEffectiveRoles` (both overloads) delegate to it; leave `Parse` as the env-string parser.
4. Filter → **PASS**; then the full project: `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests` → **PASS** (`RoleParserTests` + `AkkaClusterOptionsRoleFallbackTests` are the #532 regression net and must not need edits).
5. Commit: `feat(cluster): EffectiveRoles — one role authority with one precedence + two-layer normalization (archreview 01/S-12, 01/C-9)`.

### T2 — `AddOtOpcUaCluster` post-configures from the registered instance (structural identity)

**Classification:** high-risk (changes what the node gossips)
**Estimated implement time:** 5 min
**Parallelizable with:** T8; after T1
**Files:** `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs`, `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/AkkaClusterOptionsRoleFallbackTests.cs`

1. Add a case: with `Cluster:Roles = ["Driver", "Cluster-SITE-A"]`, the bound `AkkaClusterOptions.Roles` is `["driver", "cluster-SITE-A"]` (plain lowered, suffix preserved) and is **reference-equal in content** to the registered `EffectiveRoles.Gossiped`. Run → **FAIL**.
2. Implement: `TryAddSingleton<EffectiveRoles>` + `PostConfigure<EffectiveRoles>` replacing `:51`.
3. Filter + full `Cluster.Tests` → **PASS**. Commit: `fix(cluster): AkkaClusterOptions.Roles is the EffectiveRoles instance, not a second derivation`.

### T3 — `Program.cs` consumes the authority (closes 01/S-12 = 03/C5)

**Classification:** HIGH-RISK (every conditional registration in the composition root)
**Estimated implement time:** 10 min
**Parallelizable with:** nothing; after T2
**Files:** `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`

1. Regression gate BEFORE editing: `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests` → record green.
2. Implement: move `WebApplication.CreateBuilder(args)` above the role read; `EffectiveRoles.Resolve(builder.Configuration)`; `hasAdmin`/`hasDriver` from it; register the singleton; `:80` overlay from `Gossiped`; `:789-790` banner gains `source=`.
3. `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → **PASS**; `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests` → **PASS unchanged**.
4. Commit: `fix(host): Program.cs derives roles from the one authority, not OTOPCUA_ROLES alone (archreview 01/S-12 = 03/C5)`.

### T4 — The four role-reading validators consume the authority (closes 01/C-9)

**Classification:** medium-risk (fail-fast rules now fire on inputs they used to skip)
**Estimated implement time:** 10 min
**Parallelizable with:** T5, T6, T7; after T2
**Files:** `SplitTopologyTransportValidator.cs`, `MeshTransportOptionsValidator.cs`, `TelemetryOptions.cs`, `ConfigSourceOptionsValidator.cs`, `Cluster/ServiceCollectionExtensions.cs`, `tests/Core/…Cluster.Tests/RoleCaseVariantValidatorTests.cs` (new) + the four existing validator suites

1. Write `RoleCaseVariantValidatorTests` (Theory: 5 spellings × 4 validators, each asserting the **same Fail** as the canonical spelling, plus the canonical control). Run → **FAIL for three of the four** (SplitTopology already normalizes — that asymmetry *is* the finding, and it makes the first run a built-in falsification control).
2. Implement: each validator's ctor takes `EffectiveRoles`; the `ResolveEffectiveRoles` + `Array.IndexOf` pairs collapse to `roles.HasDriver` / `roles.HasAdmin`; `SplitTopologyTransportValidator` uses `roles.ClusterRole`/`.ClusterId`; update the four existing test suites' construction.
3. Filter → **PASS**; full `Cluster.Tests` → **PASS**. Commit: `fix(cluster): all four role-scoped validators read the authority — 'Driver' no longer skips every driver rule (archreview 01/C-9)`.

### T5 — Gossip-side predicates: `GossipedRolesHave` at the six ordinal sites

**Classification:** medium-risk (Primary election + the R2-04 gate's member count)
**Estimated implement time:** 10 min
**Parallelizable with:** T4, T6, T7; after T1
**Files:** `RoleParser.cs`, `RedundancyStateActor.cs`, `ManualFailoverService.cs`, `ClusterNodeAddressReconcilerActor.cs`, `DriverHostActor.cs`, `ScriptedAlarmHostActor.cs`, `ClusterRoleInfo.cs`, `AdminUI/Components/Pages/Fleet.razor`, `tests/Core/…Cluster.Tests/GossipedRoleMatchTests.cs` (new), a `RedundancyStateActor` selection test

1. Tests: `GossipedRolesHave` truth table; and a `RedundancyStateActor` selection case where the member gossips `"Driver"` and a Primary **is** elected. Run → **FAIL**.
2. Implement the helper and adopt it at all seven sites (incl. `SelectOldestUpMemberOfRole`); `ClusterRoleInfo`'s gossip index becomes `OrdinalIgnoreCase`-keyed.
3. Filters → **PASS**; run `Cluster.Tests`, `ControlPlane.Tests`, `Runtime.Tests`. Commit: `fix(cluster): gossiped-role predicates compare case-insensitively — a miscased peer no longer un-elects the Primary`.

### T6 — Both-sources-disagree Warning + `source=` in the boot banner

**Classification:** standard (observability)
**Estimated implement time:** 5 min
**Parallelizable with:** T4, T5, T7; after T3
**Files:** `EffectiveRoles.cs`, `Program.cs`, `EffectiveRolesTests.cs`

1. Test: both sources set to *different* normalized sets ⇒ the resolve reports a diagnostic the caller logs (return it on the type — do **not** log from a static factory; `Program.cs` logs it after Serilog is configured). Run → **FAIL**.
2. Implement + emit at boot. Filter → **PASS**. Commit: `feat(host): warn when Cluster:Roles and OTOPCUA_ROLES disagree; boot banner names the winning source`.

### T7 — Post-overlay config-layer guard in `AkkaClusterOptionsValidator`

**Classification:** standard (narrow fail-fast; no-op today)
**Estimated implement time:** 5 min
**Parallelizable with:** T4, T5, T6; after T2
**Files:** `AkkaClusterOptionsValidator.cs`, `tests/Core/…Cluster.Tests/AkkaClusterOptionsValidatorTests.cs`

1. Test: a configuration layer applied after role resolution changes `Cluster:Roles` ⇒ `Fail` naming both views; the ordinary case ⇒ `Success`. Run → **FAIL**.
2. Implement (set-equality against the injected `EffectiveRoles.Gossiped`). Filter → **PASS**. Commit: `fix(cluster): fail start when a late config layer changes Cluster:Roles (role overlays may not set it)`.

### T8 — `RoleAuthorityGuardTests` — the rule cannot be stated twice again

**Classification:** high-risk verification (this is the deliverable of theme 4, not a nicety)
**Estimated implement time:** 15 min (the one deliberately-larger task — source scan + allow-list + comment stripping)
**Parallelizable with:** T1-T7 for scaffolding; the assertions only go green after T3+T4+T5
**Files:** `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/RoleAuthorityGuardTests.cs` (new)

1. Write the three rules per Finding 2 (no `OTOPCUA_ROLES` outside `EffectiveRoles.cs`/`RoleParser.cs`; no `"Cluster:Roles"` key outside `EffectiveRoles.cs`; no bare `"admin"`/`"driver"`/`"dev"` role literal outside `RoleParser.cs`), scanning `src/**/*.cs` + `src/**/*.razor` from the `.slnx` root, stripping `//`/`///` lines, reporting **every** offender as `file:line`, with a named allow-list each entry of which carries a reason.
2. Run on the pre-T3 tree → **FAIL**, listing at minimum `Program.cs:61,62,63` and `Fleet.razor:149,159`. **Paste that list into the commit body** — it is the proof the guard can fail.
3. After T3-T5, run → **PASS**. Commit: `test(cluster): guard that role reads live only in the authority (archreview theme 4)`.

### T9 — `Cluster:Roles`-only node gets the driver stack (integration)

**Classification:** high-risk verification (the exact shape 01/S-12 describes)
**Estimated implement time:** 15 min (harness plumbing)
**Parallelizable with:** T5-T8; after T3
**Files:** `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ClusterRolesOnlyBootTests.cs` (new)

1. Build the host on the `DriverOnlyNoConfigDbBootTests` direct-host-build pattern with `Cluster:Roles = ["driver"]` and `OTOPCUA_ROLES` **unset** (via `EnvVarScope(null)`); assert the driver stack is registered (`ILocalDb`, `ApiKeyOpcUaUserAuthenticator` not the Null one, runtime actors) and `AkkaClusterOptions.Roles == ["driver"]`.
2. **Negative control:** `git stash` T3's change (or run against master) → the test must **FAIL** with the Null authenticator / missing `ILocalDb`. Record it.
3. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~ClusterRolesOnlyBootTests"` → **PASS**. Commit: `test(integration): a Cluster:Roles-only node wires the driver stack`.

### T10 — Docs sweep

**Classification:** standard (docs)
**Estimated implement time:** 15 min
**Parallelizable with:** all
**Files:** per the "Docs to update" table

Rewrite `docs/Configuration.md:126` (the note that currently documents the defect as design), plus the seven other rows. Commit: `docs: one role authority — precedence, normalization, and the overlay invariant (R3-04)`.

### T11 — docker-dev rig: make the divergence observable

**Classification:** standard (rig config) — **but it is what makes T13 non-vacuous**
**Estimated implement time:** 5 min
**Parallelizable with:** T10; before T13
**Files:** `docker-dev/docker-compose.yml`, `docker-dev/README.md`

1. Drop `OTOPCUA_ROLES` from **site-b-2** (keep its `Cluster__Roles__0/1`), and add a comment naming this as the `Cluster:Roles`-only pin for archreview 01/S-12 — site-b is already the rig's A/B control node. Leave the other five nodes as they are (they are the "both sources agree" control).
2. Add a `docker-dev/README.md` row saying so, and that the node is **expected to fail deploys on a pre-R3-04 build**.
3. Commit: `chore(docker-dev): site-b-2 is Cluster:Roles-only — the rig can now see the two-authority split`.

### T12 — Optional: a miscased pin

**Classification:** standard (rig config)
**Estimated implement time:** 5 min
**Parallelizable with:** T11; before T13
**Files:** `docker-dev/docker-compose.yml`

Spell **site-a-2**'s `Cluster__Roles__0` as `Driver` (capital D), keeping `cluster-SITE-A` verbatim, as the standing 01/C-9 pin. Expected post-fix: site-a still elects exactly one Primary and `/hosts` shows a lowercase `driver` chip on both nodes. Pre-fix: site-a elects none. Commit: `chore(docker-dev): site-a-2 pins a miscased Cluster:Roles entry (archreview 01/C-9)`.

### T13 — LIVE GATE: docker-dev six-node rig — operator/rig task

**Classification:** HIGH-RISK verification gate (merge-blocking)
**Estimated implement time:** ~40 min rig session
**Parallelizable with:** nothing (needs the built branch + T11/T12)
**Files:** none (rig session; findings recorded in `archreview/plans/STATUS.md`)

**How roles are set on the rig today:** every node sets `OTOPCUA_ROLES` **and** `Cluster__Roles__*` to the same values — central-1/2 `admin,driver,cluster-MAIN` (`:298`/`:352-354`, `:448`/`:485-487`), site-a-1/2 `driver,cluster-SITE-A` (`:564`/`:606-607`, `:686`/`:716-717`), site-b-1/2 `driver,cluster-SITE-B` (`:782`/`:803-804`, `:853`/`:874-875`). T11/T12 make two of them divergent on purpose. Login is disabled on the rig, so drive the AdminUI directly at `http://localhost:9200` (round-robins central-1/central-2 — **rebuild both**).

1. **Pre-fix baseline (on master, with T11/T12's compose applied).** `docker compose -f docker-dev/docker-compose.yml up -d --build` → confirm the failure is real: site-b-2's log shows the boot banner with **no** roles (`roles=[]`), `/hosts` shows it gossiping `driver` while `/fleet` counts it, and a fleet deploy **seals `TimedOut`** naming site-b-2 as silent. site-a shows **no Primary** (both nodes' `Server_ServiceLevel` low; `/hosts` shows no `driver` leader role). Record both — without this step the gate proves nothing.
2. **Build the branch:** `docker compose -f docker-dev/docker-compose.yml up -d --build` on all six.
3. **Role parity, per node:** `docker logs otopcua-dev-<node>-1 | grep "starting with roles"` → each node's banner names the same role set its compose entry declares, with `source=Configured`, and the two divergent nodes report the *config* roles (site-b-2 `driver,cluster-SITE-B`; site-a-2 `driver,cluster-SITE-A` — **lower-cased**). Cross-check against `/hosts`, which renders the **gossiped** `m.Roles` per member (`Hosts.razor:487`): the chips must match the banners exactly, on all six. That comparison — banner (Host view) vs `/hosts` chips (Akka view) — is the live form of the "two views agree" guard.
4. **site-a Primary election (01/C-9):** `/hosts` shows exactly one `driver` leader role in the SITE-A pair; Client.CLI `read` of `Server_ServiceLevel` on `opc.tcp://localhost:4842` vs `:4843` shows the 250/240 split. Pre-fix this leg showed neither.
5. **Deploy seals Applied (01/S-12):** deploy a config from `/fleet`; it must seal **`Applied`** with an ack from **all six** nodes including site-b-2 (pre-fix: `TimedOut`). Confirm site-b-2 actually serves — Client.CLI `browse -u opc.tcp://localhost:4845` returns the deployed tree.
6. **Fail-fast still fails (T4 regression):** temporarily set site-b-1's `Cluster__Roles__0` to `Drivr`, `docker compose up -d site-b-1` → the container must **exit** with the unknown-role message naming `Cluster:Roles`. Revert.
7. Record every leg in `archreview/plans/STATUS.md` with the pre-fix baseline beside the post-fix result.

---

### Sequencing summary

```
T1 → T2 → T3 → T9              (the authority, then the Host, then its integration proof)
        ↘ T4, T5, T6, T7        (validators / gossip predicates / warning / overlay guard — parallel after T2)
T8 scaffolded early; green only after T3+T4+T5
T10 anytime
T11, T12 (rig config) before T13; T13 last, after everything is merge-ready
```

Overall effort: **~1 dev-day code+unit, plus the rig session.** The three HIGH-RISK slices (T1 identity type, T3 composition root, T8 the guard) each land behind their full project suites before anything stacks on them; T13 is merge-blocking and is only meaningful because T11/T12 gave the rig a node that can actually exhibit the defect.

---

## Execution notes (recorded during T1–T12 implementation, 2026-08-20)

**T5 — `SelectOldestUpMemberOfRole`/`SelectDriverPrimary` cannot be reached from this repo; the plan
was wrong about this one call site.** `RedundancyStateActor.SelectOldestUpMemberOfRole`
(`RedundancyStateActor.cs:172-178`) is a thin wrapper around `ClusterActiveNode.OldestUpMember`
(`ZB.MOM.WW.Health.Akka`, a separately-versioned NuGet package out of `scadaproj`), which itself
matches via **Akka's own `Member.HasRole`** — an ordinal comparison this repo does not own. This is
structurally identical to the "not ours to change" carve-out the plan's own choice 2 already makes
for `ClusterSingletonOptions.Role` (rows 18/19); the plan simply did not notice the same constraint
applies to `SelectOldestUpMemberOfRole` before naming it as a `GossipedRolesHave` adoption site.

Consequence: the plan's stated T5 test — "a `RedundancyStateActor` selection case where the member
gossips 'Driver' and a Primary IS elected" — is **not achievable**. What shipped instead
(`tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/GossipedRoleCaseInsensitivityTests.cs`):

- `GossipedRolesHave` was implemented and adopted at every site that genuinely reads gossiped roles
  directly in this repo: `RedundancyStateActor.cs` (`BuildSnapshot`'s own role classification, NOT
  the Primary-selection call), `ManualFailoverService.cs` (`OrderedDrivers`),
  `ClusterNodeAddressReconcilerActor.cs` (`FilterOwnClusterDriverMembers`), `DriverHostActor.cs` and
  `ScriptedAlarmHostActor.cs` (`DriverMemberCount`), and `Fleet.razor` (the bare `"driver"` literals,
  now `RoleParser.Driver` — the actual case-insensitivity there comes from `ClusterRoleInfo`'s
  `_roleLeaders`/`_membersByRole` dictionaries moving to `OrdinalIgnoreCase`, which
  `IClusterRoleInfo.MembersWithRole`/`RoleLeader` read through).
- `A_member_gossiping_a_miscased_driver_role_classifies_Secondary_not_Detached` pins the fix that
  **is** achievable: a single member gossiping `"Driver"` now classifies `Secondary` in the
  redundancy snapshot (was `Detached` pre-fix) — real value, because `/hosts` and `/fleet` render
  this list, even though that member can still never be *named* Primary by itself.
- `Primary_election_itself_remains_ordinal_and_still_elects_no_one_for_an_all_miscased_pair` pins the
  **true** (negative) outcome instead of asserting a false positive: a node (or pair) gossiping only
  `"Driver"` still elects no Primary, because that decision runs through the vendored ordinal match.
  This is a genuine residual — **not closed by R3-04** — worth a follow-up issue against
  `ZB.MOM.WW.Health.Akka`'s `ClusterActiveNode.OldestUpMember` (e.g. an optional case-insensitive
  role-match parameter) if the rolling-upgrade window this defends against is judged worth closing
  fully. In practice T1/T2 already close the *headline* outage the census describes for a
  FIXED-build node (a node can no longer gossip a miscased plain role itself — normalization happens
  before the gossip handoff), so this residual only bites a peer still running a pre-R3-04 build.
- `ManualFailoverService_recognises_a_miscased_driver_node_as_a_candidate` uses
  `GetSnapshot()`/`PrimaryAddress` rather than `FailOverCore(dryRun: true)` as originally sketched —
  `FailOverCore` refuses below 2 driver members ("nothing to fail over TO"), which a single-node
  fixture can never satisfy; `GetSnapshot()` shares the same `OrderedDrivers` candidate selection
  this test is actually about, without needing a second live node.

No other T1–T5 deviations from the plan.

**T6-T9 — no deviations.** Implemented as designed; `AkkaClusterOptionsValidator` gained an optional
(default-null) `EffectiveRoles?` constructor parameter rather than a required one specifically so the
pre-existing zero-arg test constructions in `AkkaClusterOptionsValidatorTests` did not all need
touching — the guard is simply inert when nothing is injected, which is also the correct behaviour
for any harness that builds this validator without the rest of the role authority.

**T8 — two additional bare-role-literal offenders found and fixed, beyond the two the plan named.**
The census's row-by-row list did not claim to be exhaustive of every bare literal in the tree, only
of the *production predicate* sites; running the guard for real (rather than reasoning about it)
found two more before it could go green:

- `DriverInstanceActor.ShouldStub`'s `roles.Contains("dev")` — a genuine Akka-cluster-role read
  (`_localRoles`, threaded from `IClusterRoleInfo.LocalRoles`) gating dev-stub mode for
  Windows-only drivers. Fixed to `RoleParser.GossipedRolesHave(roles, RoleParser.Dev)`.
- `ControlPlane/ServiceCollectionExtensions.AdminRole = "admin"` — a second constant duplicating the
  literal `RoleParser.Admin` names, feeding Akka's own `ClusterSingletonOptions.Role` matching (the
  same "not ours to change" ordinal match as `SelectOldestUpMemberOfRole`). The VALUE was already
  correct (Akka's own matching is unaffected either way); fixed to alias `RoleParser.Admin` rather
  than duplicate it, matching the sibling `Runtime/ServiceCollectionExtensions.DriverRole` pattern.

The guard also required a small allow-list (six files) for bare `"admin"`/`"driver"`/`"dev"` literals
that name something OTHER than an Akka `Cluster:Roles` role — a raw-tree node-KIND discriminator, an
ACL-scope entity-type name, the Sql driver's own TagConfig JSON marker (×3 files), and an LDAP-derived
AdminUI RBAC role. Each carries its reason in `RoleAuthorityGuardTests.RoleLiteralAllowListWithReasons`,
per the plan's "add to the allow-list WITH A REASON" instruction; a companion test
(`Every_allow_list_entry_has_a_reason_and_still_applies`) fails if an entry goes stale.

The falsification control (`The_scan_actually_detects_the_pre_R3_04_offenders`) could not literally
re-run the guard against the pre-T3/pre-T5 GIT COMMITS (T1–T8 landed sequentially in this same
worktree, so "the tree" is always post-fix by the time T8's test runs) — it instead feeds the scan
logic the exact pre-fix `Program.cs`/`Fleet.razor` snippets as string literals, captured verbatim
from this session's own earlier reads of those files before they were edited. Functionally
equivalent proof-of-falsifiability; recorded as a deviation because it is a different MECHANISM than
"run on the pre-T3 tree," not because the result differs.

**T9 — scope note, not a deviation.** "The driver stack" is proven via two representative slices
(`ILocalDb` resolution + the `IOpcUaUserAuthenticator` selection block), not the ENTIRE
`WithOtOpcUaRuntimeActors` Akka actor graph — replicating that too would need a real `ActorSystem`
(the exact thing `DriverOnlyNoConfigDbBootTests`'s own doc comment explains this test family avoids).
The negative control uses an inline `PreFixDerivation` helper (reading `OTOPCUA_ROLES` only, matching
pre-T3 `Program.cs:61-63` verbatim) rather than a `git stash`, for the same "already sequential
commits in one worktree" reason as T8's falsification control — and it is run in the SAME test
process against the SAME configuration, which is a stronger control than a separate stash-and-revert
cycle would have been (both derivations are proven against literally identical input).

**T11/T12 — combined into one commit.** The docker-compose.yml header comment documenting all six
nodes' role sources interleaves site-a-2 and site-b-2 on adjacent lines, so a clean git-level split
into the plan's two separate commits was not practical; both tasks' edits landed in one commit
(`985de0b6`). The rig was NOT brought up as part of either task, per the brief.

## Review outcome

A code review after T1–T12 returned verdict **FIX-FIRST**, two items. Both are fixed, each in its
own TDD commit, on top of the T1–T12 work above.

### 1. [Blocking] T7's post-overlay guard was structurally unreachable

**The finding.** `AkkaClusterOptionsValidator.ValidateAgainstEffectiveRoles` compared
`options.Roles` against `_effectiveRoles.Gossiped` — but `ServiceCollectionExtensions.cs`'s
PostConfigure (T2's OWN structural-identity fix) unconditionally sets
`options.Roles = [.. effectiveRoles.Gossiped]` first. The two values are therefore **equal by
construction**, on every real host, always — the guard could never observe a divergence no matter
what a role overlay did to `Cluster:Roles`. My original two tests (now deleted) hand-built an
`AkkaClusterOptions` instance directly and set `.Roles` to a "mismatched" value bypassing
PostConfigure entirely — which exercised the comparison logic in isolation but never proved the
guard reachable through the real pipeline. **Concretely, before this fix: an overlay that set
`Cluster:Roles` was silently ignored (PostConfigure overwrote it back to the pre-overlay value), not
failed at boot** — the exact opposite of the guard's purpose (choice 4: "a role overlay may not set
Cluster:Roles").

**The fix.** `AkkaClusterOptionsValidator` now also takes an optional `IConfiguration?` (default
null, same pattern as `EffectiveRoles?` — so every pre-existing zero/one-arg test construction keeps
compiling and behaving as before). The guard re-reads the RAW `Cluster:Roles` section directly off
the LIVE configuration (`_configuration.GetSection(EffectiveRoles.ConfiguredRolesKey).Get<string[]>()`)
rather than the bound options, normalizes it by reusing `EffectiveRoles.Resolve(raw, envRoles: null)`
(one rule, stated once — archreview theme 4, rather than re-implementing the configured-path
normalization a second time), and compares that against `_effectiveRoles.Gossiped`. Skipped entirely
when the raw post-overlay value is empty — required for correctness, not merely tidiness: an
`OTOPCUA_ROLES`-only node (most of production, per CLAUDE.md) has an empty `Cluster:Roles` both
before AND after any overlay, and comparing "empty" against a non-empty env-derived `Gossiped` list
would be a permanent false positive on every one of those nodes. `AddOtOpcUaCluster` now also
`TryAddSingleton(configuration)` so a bare `ServiceCollection` test harness (not just
`WebApplicationBuilder`, which auto-registers `IConfiguration`) can supply it.

**The tests.** Both guard tests are rewritten to go through the REAL `AddOtOpcUaCluster` +
PostConfigure pipeline, over a `ConfigurationManager` (not a one-shot `ConfigurationBuilder`) — the
same "add a provider to the SAME live object after the fact" shape `WebApplicationBuilder.Configuration`
has, which is what lets a role overlay be simulated faithfully: `EffectiveRoles.Resolve(configuration)`
is called and its result registered BEFORE a second `AddInMemoryCollection` call mutates the same
`configuration` object with different `Cluster:Roles` values, mirroring `Program.cs`'s own
resolve-then-`AddJsonFile(overlay)` sequencing exactly. Confirmed red first: the "changes" test
failed with "should throw OptionsValidationException but did not" against the pre-fix (options.Roles
-comparing) validator — direct proof of the reviewer's structural-unreachability claim. Two
supporting tests were added beyond the two named in the review comment: `No_overlay_at_all_passes`
(the ordinary case) and `EffectiveRoles_injected_without_IConfiguration_skips_the_guard_entirely`
(the two-optional-dependency ctor's other skip leg). One self-inflicted false-positive surfaced and
was fixed along the way: the new catch-block's error message split `"Cluster:Roles."` onto its own
string-literal token, which `RoleAuthorityGuardTests`' own space-inside-the-literal heuristic (T8)
correctly flagged as NOT looking like an operator sentence — merging it back into one literal
containing surrounding words fixed it without weakening the guard.

### 2. [Important] `ClusterRoleInfo._roleLeaders` must stay `Ordinal`, not `OrdinalIgnoreCase`

**The finding.** T5 moved BOTH `_roleLeaders` and `_membersByRole` to `OrdinalIgnoreCase` under one
shared rationale (defence in depth against a miscased rolling-upgrade peer). That rationale is right
for `_membersByRole` ("every member carrying a spelling of this role" — merging casings is the
point) but wrong for `_roleLeaders`: Akka computes — and raises `RoleLeaderChanged` for — role
leadership **per exact gossiped role string**. "driver" and "Driver" are two entirely distinct roles
to Akka's own role-leader machinery, each independently electing its own leader. Keying
`_roleLeaders` case-insensitively collapses both onto ONE dictionary slot, so whichever
`RoleLeaderChanged` event landed last silently overwrites the other's leader — turning two real,
independent facts Akka tracks separately into one, wrong, merged fact.

**The fix.** `_roleLeaders` reverted to `StringComparer.Ordinal`; `_membersByRole` stays
`OrdinalIgnoreCase` (the split itself, not merely one value, is now documented in the field
comment). No other production call site touches `_roleLeaders`'s comparer, so this is a one-line
revert plus doc.

**The test.** `ClusterRoleInfoRoleLeaderCaseTests.RoleLeaderChanged_for_driver_then_Driver_keeps_distinct_entries`
— a real two-node cluster (the only faithful option: `HandleRoleLeaderChanged` is `internal` with no
`InternalsVisibleTo`, so it cannot be unit-tested directly, and `ClusterRoleInfo`'s constructor needs
a genuinely clustered `ActorSystem`). Node A gossips only `"driver"`, node B (joining the same
cluster) gossips only `"Driver"` — each is therefore the sole, and so leader, member of its own
exact-string role. Confirmed red first: against the pre-fix (`OrdinalIgnoreCase`) comparer, the test
failed with `driverLeader` and `capitalDriverLeader` resolving to the SAME address — the exact
collision the review predicted, reproduced live rather than merely reasoned about.

### Optional item — done

Added one sentence to `docs/ServiceHosting.md`'s "Per-role configuration overlays" section: a plain
role's overlay-filename fragment is always lower-case regardless of how it was spelled in
`Cluster:Roles`, but a `cluster-{ClusterId}` role's suffix carries into the filename verbatim (since
`EffectiveRoles.Gossiped` preserves that spelling by design) — e.g.
`Cluster:Roles: ["driver", "cluster-SITE-A"]` loads `appsettings.cluster-SITE-A-driver.json`.

### Verification

`dotnet build ZB.MOM.WW.OtOpcUa.slnx` → 0 errors throughout.

- After item 2 (`ClusterRoleInfo`): `Cluster.Tests` 241 → **242/242** (+1,
  `ClusterRoleInfoRoleLeaderCaseTests`).
- After item 1 (T7 guard): `Cluster.Tests` 242 → **244/244** (net +2: the two old hand-built T7
  tests deleted, four new ones added —
  `An_overlay_that_changes_Cluster_Roles_after_resolution_fails_naming_both_views`,
  `An_overlay_that_repeats_the_same_Cluster_Roles_passes`, `No_overlay_at_all_passes`,
  `EffectiveRoles_injected_without_IConfiguration_skips_the_guard_entirely` — the pre-existing
  `No_EffectiveRoles_injected_skips_the_guard_entirely` was kept unchanged).
- `Host.Tests` **106/106**, unchanged by either fix (`Program.cs` itself was not touched).
- `ControlPlane.Tests` **267/267** — an extra safety check beyond what was asked, since
  `ClusterRoleInfo` is a shared dependency of `RedundancyStateActor` et al.

No pushes, no merges.

## Live gate result (2026-08-20) — **PASS** on docker-dev (pre-fix control at `7c77a8c7` failed first)

Rig: `docker-dev/docker-compose.yml` on OrbStack (Mac), fresh volumes (`down -v` before the control), the
six host containers + SQL + Traefik, AdminUI on `Security__Auth__DisableLogin=true`. The compose carried the
T11/T12 pins in BOTH runs (site-b-2 `Cluster__Roles__*` only, no `OTOPCUA_ROLES`; site-a-2
`Cluster__Roles__0="Driver"`). Deploys via `./docker-dev/deploy.sh` (`POST /api/deployments`,
`X-Api-Key: docker-dev-deploy-key`, `202` both times). `Server_ServiceLevel` (`i=2267`) read with Client.CLI on
`:4840`–`:4845`. Image parity asserted ×6 via `docker inspect --format '{{.Image}}'` in both runs. The Modbus
fixture on `10.100.0.35:5020` was left running throughout. Note: master advanced `9242630c → f665e8b8` (R3-05 S18
follow-up) *while* the post-fix run was in progress; the post-fix image had already been built from the
`9242630c` working tree (image created 13:29:46 local, merge landed 13:29:59) — the only later delta is
`BoundedApiKeyStore.cs` + a compose env wire, neither of which touches roles.

### Leg 1 — pre-fix control, image `otopcua-host:prefix-r304` = `32a2dcee0285` (scratch worktree at `7c77a8c7`) ×6: **FAILED as required**

| what the plan predicted | observed |
|---|---|
| site-b-2 boot banner shows **no** roles | `[17:25:22 INF] OtOpcUa.Host starting with roles=[] (admin=False, driver=False)` — every other node `roles=[admin,driver,cluster-main]` / `[driver,cluster-site-a]` / `[driver,cluster-site-b]` (pre-fix banner lower-cases the cluster suffix; no `source=`) |
| site-b-2 gossips `driver` anyway | site-b-1: `Node [akka.tcp://otopcua@site-b-2:4053] is JOINING, roles [cluster-SITE-B,driver]`; site-b-2's own `cluster-primary` health check: `Standby: active node for role 'driver' is akka.tcp://otopcua@site-b-1:4053` — Akka counted it as a driver member while the Host wired nothing |
| fleet deploy seals **TimedOut** naming site-b-2 | Deployment `6c653c61…` (rev `942ad3cc…`): `Status = 4`, `SealedAtUtc = NULL`; `NodeDeploymentState` five rows `Status = 1` (`AppliedAtUtc 17:25:41.077–.107`), **site-b-2 `Status = 0`, `AppliedAtUtc = NULL`**. central-2: `[17:27:41 WRN] Deployment 6c653c61… timed out after 00:02:00 (5/6 acks landed). No ack from: site-b-2:4053. Each is an enabled, non-maintenance ClusterNode row …` |
| site-b-2 does not serve | Client.CLI `read` on `opc.tcp://localhost:4845` → `Remote side closed connection` (no OPC UA server started; no DriverHost / Kestrel lines in its log at all, only Akka remoting + cluster join) |
| site-a (01/C-9): "elects **no** Primary" | **Partly wrong, and the execution notes already said why.** site-a-1 gossips ordinal `driver`, so `ClusterActiveNode.OldestUpMember` still elected it: `:4842` = **250**. What failed was site-a-2: gossiped `roles [cluster-SITE-A,Driver]` (site-a-1's JOINING line), its own `cluster-primary` check `Node carries none of the active roles: admin, driver.`, and `:4843` = **0** — the pair ran with a Primary and **no eligible standby**. A pair whose nodes were BOTH miscased would elect none (pinned by `Primary_election_itself_remains_ordinal_and_still_elects_no_one_for_an_all_miscased_pair`); the rig pins only one |
| control reference | `:4840`=240, `:4841`=250 (MAIN), `:4844`=250 (site-b-1 sole eligible driver) |

### Leg 2 — post-fix, image `otopcua-host:dev` = `b7cfae1be510` (master `9242630c`) ×6: **PASS**

1. **Role banners, all six `source=Configured`, config roles win, normalised** — central-1/2
   `starting with roles=[admin,driver,cluster-MAIN] (admin=True, driver=True) source=Configured`; site-a-1/**site-a-2**
   `roles=[driver,cluster-SITE-A]` (the `Driver` pin lower-cased, cluster suffix verbatim); site-b-1/**site-b-2**
   `roles=[driver,cluster-SITE-B] (admin=False, driver=True)` — the `Cluster:Roles`-only node now wires the driver
   stack. Gossip view agrees with the Host view on every node: site-a-1 `Node [site-a-2] is JOINING, roles
   [cluster-SITE-A,driver]` (was `Driver`), site-b-1 `Node [site-b-2] is JOINING, roles [cluster-SITE-B,driver]`,
   central-2 `Node [central-1] is JOINING, roles [cluster-MAIN,driver,admin]`. (`/hosts` chips were not used as the
   cross-check — the page is a Blazor circuit and a `curl` of it renders no member rows; the Akka JOINING lines are the
   same gossiped `m.Roles` the page reads.)
2. **Deploy seals Applied 6/6 (01/S-12):** Deployment `a5a7018d…` (same revision `942ad3cc…`): `Status = 2`,
   `CreatedAtUtc 17:31:21.914`, `SealedAtUtc 17:31:22.138` (224 ms); six `NodeDeploymentState` rows all `Status = 1`,
   `AppliedAtUtc 17:31:22.090–.132` including **site-b-2**. central-1: `Deployment a5a7018d… sealed (acks=6)`.
   site-b-2: `Fetched + verified artifact … from http://central-1:4055 (1468 bytes)` → `DriverHost site-b-2:4053:
   applied fetched deployment a5a7018d… (rev 942ad3cc…, children=0)` → `cached … as this node's local fallback`.
   Client.CLI `browse -u opc.tcp://localhost:4845 -d 2` returns `Server`, `Aliases`, `Locations`, **`OtOpcUa
   (ns=2;s=OtOpcUa)`** — site-b-2 serves (was `Remote side closed connection`).
3. **One Primary per pair, with an eligible standby (01/C-9):** `:4840`=240/`:4841`=250 (MAIN), **`:4842`=250 /
   `:4843`=240** (SITE-A — site-a-2 was 0 pre-fix; its health check now reads `Standby: active node for role 'driver'
   is akka.tcp://otopcua@site-a-1:4053`), `:4844`=250/`:4845`=240 (SITE-B — site-b-2 was unreachable). After the leg-5
   restart of site-b-1 the SITE-B pair re-elected with **site-b-2 as Primary** (`:4845`=250, `:4844`=240) — the
   formerly role-less node now holds the Primary-gated data plane.
4. **T6 disagreement Warning fires:** scratch compose override on site-b-1 setting `OTOPCUA_ROLES: "driver"` against its
   `Cluster__Roles__0/1 = driver,cluster-SITE-B` → `[17:32:35 INF] … starting with roles=[driver,cluster-SITE-B] …
   source=Configured` followed by `[17:32:35 WRN] Cluster:Roles ([driver,cluster-SITE-B]) and OTOPCUA_ROLES ([driver])
   disagree; Cluster:Roles wins outright.` — and the node still joined and served with the config roles. (The rig's
   standing pins do NOT fire this Warning by themselves: site-a-2's `Driver` normalises equal to its env value, and
   site-b-2 has only one source — choice 5 is explicitly "both set AND differ".)
5. **Unknown role fails host start:** override `Cluster__Roles__0: "operator"` on site-b-1 → the log is exactly six
   lines: `Unhandled exception. System.ArgumentException: Cluster:Roles[0] = 'operator' is not a known role. Allowed:
   admin, driver, dev, or 'cluster-<ClusterId>'.` at `EffectiveRoles.NormalizeAndValidateConfigured` ←
   `EffectiveRoles.Resolve(IConfiguration)` ← `Program.cs:67` — no boot banner, no Akka remoting, no Kestrel, no
   cluster join. Override removed; site-b-1 recreated from the plain compose → `roles=[driver,cluster-SITE-B] …
   source=Configured`, no Warning, `(healthy)`.
   ⚠️ **Observation, pre-existing and NOT R3-04's:** the `dotnet OtOpcUa.Host.dll` process did **not exit** after the
   unhandled exception — `docker inspect` kept reporting `running exit=0 restarts=0` for > 2 min, the main thread in
   state `R` burning ~50 % of a core with 14 Server-GC threads parked. Reproduced identically on a bare `docker run
   --rm -e OTOPCUA_ROLES=operator` against **both** `otopcua-host:prefix-r304` (7c77a8c7, pre-fix `RoleParser.Parse`
   throw at `Program.cs:61`) and the master image: neither exits within 90 s (`timeout` → 124). So "fail-start" holds
   (nothing is wired, nothing listens) but the container's `unless-stopped` restart policy never fires and an orchestrator
   watching the exit code would not see a crash; only the compose healthcheck (never `healthy`) shows it. Arm64 OrbStack
   `aspnet:10.0` runtime image; root cause not chased here — worth a follow-up (is it the runtime's unhandled-exception
   abort path on this host, or something in our `Program.cs` top-level statements?) before relying on a process-exit
   signal for any of the boot-time validators (`SplitTopologyTransportValidator`, `AkkaClusterOptionsValidator`,
   `ConfigSourceOptionsValidator` fail the same way).

### Leg 3 — restore

Six hosts on `b7cfae1be510` ×6 (parity asserted), `GET /health/ready` → 200 via `:9200`, SQL + Traefik up, all six
ServiceLevels one 250 + one 240 per pair; `otopcua-pymodbus-standard` on `10.100.0.35` untouched (`Up 2 weeks`). Scratch
worktree at `7c77a8c7` removed; the control image tag `otopcua-host:prefix-r304` left in the local image cache (same layer
id as R3-03's `prefix-r303` — both are `7c77a8c7`).

**Verdict: T13 PASS.** Both findings reproduce on the pre-fix build with the rig as pinned (01/S-12 as a real fleet
deploy outage; 01/C-9 as a standby-less pair rather than the plan's "no Primary" — the execution notes' ordinal-election
residual, confirmed live), and neither reproduces on master. One pre-existing process-does-not-exit-after-fail-start
observation recorded above for follow-up.