lmxopcua/archreview/plans/R3-04-one-role-authority-plan.md

Design + Implementation Plan — R3-04 One Role Authority

• Source reports: archreview/01-core-composition.md §S-12 (:198-210) + §C-9 (:264-278) · archreview/03-server-runtime.md §C5 (:131-133) (2026-08-19 round-3 re-review at 12b0035a)
• Prior designs consumed: none — both findings are new in round 3. The nearest ancestor is Gitea #532 (RoleParser.ResolveEffectiveRoles + the AddOtOpcUaCluster PostConfigure), whose remediation created the second authority these findings are about; the field incident it closed is docs/deployments/wonder-app-vd03-makino-z-34184.md.
• Plan verified against tree at: master 7c77a8c7 (clean; code identical to 12b0035a) — every anchor re-opened and re-checked 2026-08-20.
• Scope: OVERALL prioritized action item #4 — "One role authority: ResolveEffectiveRoles normalizes + validates; Program.cs and all four validators consume it; boot guard asserting Akka and Host role views agree" (01 S-12/C-9, 03 C5). Effort Small / Impact Medium-High.
• Cross-cutting requirement (OVERALL theme 4, "One rule stated twice diverges"): the deliverable is not "fix two call sites" — it is that the role list is derived in exactly one place and every consumer inherits it, enforced by a guard test so the rule cannot be re-stated. The authoring-vs-deploy-gate arc (#535→#559) is the named precedent: one authority, enforced in the service, mirrored by the gate. Concrete pick here: a EffectiveRoles value type produced by one factory, consumed by Program.cs, the four Cluster validators, the Akka options PostConfigure and every gossip-side predicate; plus RoleAuthorityGuardTests, a source-scanning guard modelled on CommandDisposalConventionsTests / StatusCodeParityTests, that fails when any production file outside the authority reads OTOPCUA_ROLES, Cluster:Roles, or a bare "admin"/"driver" role literal.
• Cross-cutting requirement (OVERALL theme 2, "verification that cannot see its own blind spot"): the docker-dev rig sets both role sources, consistently, on all six nodes (docker-compose.yml:298/352-354, :448/485-487, :564/606-607, :686/716-717, :782/803-804, :853/874-875). Neither finding can reproduce on the rig as configured. The live gate below therefore changes the rig (site-b-2 becomes Cluster:Roles-only) so the divergence becomes observable — the same lesson 03/S14 taught about DisableLogin=true.

Verification summary

Both findings CONFIRMED at 7c77a8c7 — neither stale, neither partially fixed. Every report anchor is exact; no drift. Anchor status:

Finding	Report anchor	Status at 7c77a8c7
01/S-12 = 03/C5	Program.cs:61-63 (RoleParser.Parse(Environment.GetEnvironmentVariable("OTOPCUA_ROLES")) → hasAdmin/hasDriver)	Exact. :61 resolve, :62 roles.Contains("admin"), :63 roles.Contains("driver") — bare string literals, not RoleParser.Admin/.Driver, a detail neither report names. The env-only array reaches four further decisions the reports do not enumerate: the per-role appsettings overlay filename (:80), the health-check registration (:564 AddOtOpcUaHealth(hasAdmin)), the three dedicated gRPC listener ports (:624-626), and the boot banner (:789-790).
01/S-12 = 03/C5	RoleParser.cs:115-125 / :115-119 (ResolveEffectiveRoles)	Exact. Two-arg overload :115-119 (configured is { Count: > 0 } ⇒ verbatim copy; else Parse(envRoles)), one-arg convenience :124-125, RolesEnvVar const :128, Parse :133-152.
01/S-12 = 03/C5	Cluster/ServiceCollectionExtensions.cs PostConfigure	Exact, now pinpointed: :51 — options => options.Roles = RoleParser.ResolveEffectiveRoles(options.Roles), applied inside AddOtOpcUaCluster (:34-51). The gossiped list is handed to Akka at :267 (Roles = options.Roles).
01/S-12 = 03/C5	RoleParser.cs:96-103 ("the exact knob the wonder-app-vd03 field fix hand-added")	Exact — the <remarks> paragraph is :94-103.
01/C-9	SplitTopologyTransportValidator.cs:69-78 (the one validator that normalizes)	Exact. Comment :69-72; configured read :73; ResolveEffectiveRoles :74; normalized = Array.ConvertAll(effective, r => r.Trim().ToLowerInvariant()) :76; classification :78, :89-90.
01/C-9	MeshTransportOptionsValidator.cs:76-79	Exact — ResolveEffectiveRoles(GetSection("Cluster:Roles").Get<string[]>()) :76-77; case-sensitive Array.IndexOf(roles, RoleParser.Driver) :78, .Admin :79.
01/C-9	TelemetryOptions.cs:124-126 (TelemetryOptionsValidator, class at :88)	Exact — resolve :124-125, case-sensitive IndexOf :126. Note the sibling TelemetryDialOptionsValidator (:160) reads no roles at all.
01/C-9	ConfigSourceOptionsValidator.cs:101-104	Exact — resolve :101-102, case-sensitive IndexOf :103-104.
01/C-9	"the four Cluster fail-fast validators"	Correct as a count, but the brief's fourth candidate is wrong. AkkaClusterOptionsValidator (AkkaClusterOptionsValidator.cs:37) reads no roles — it validates self-first seed ordering only. The four role-reading validators are SplitTopologyTransportValidator, MeshTransportOptionsValidator, TelemetryOptionsValidator, ConfigSourceOptionsValidator; all four are registered in Cluster/ServiceCollectionExtensions.cs:56/63/70/80.
01/C-9	"only the env fallback is lowercased/validated by Parse"	Exact and load-bearing. Parse :137-141 lower-cases + trims + dedupes and :143-149 throws ArgumentException on an unknown role. The configured path (:117) does none of that: Cluster:Roles: ["Drivr"] is accepted silently, gossiped verbatim, and matches nothing.

The blast radius is materially larger than either finding states, and the plan is sized to the larger one. 01/C-9 describes the case hazard as "skips every driver-scoped boot-time rule". That is true and it is the smaller half. The same verbatim spelling is gossiped to Akka (ServiceCollectionExtensions.cs:267), and six production predicates match gossiped roles ordinally against the lowercase constant: RedundancyStateActor.cs:146/193 (Primary election), ManualFailoverService.cs:125, ClusterNodeAddressReconcilerActor.cs:185, DriverHostActor.cs:1612 (the R2-04 primary-gate member count), ScriptedAlarmHostActor.cs:520, and Fleet.razor:149. A pair configured Cluster:Roles: ["Driver"] therefore elects no Primary at all (SelectOldestUpMemberOfRole(members, "driver") matches nothing ⇒ every node Detached ⇒ the R2-04 data-plane gate denies every write, the alarm S&F queue never drains, ServiceLevel never rises) while /hosts cheerfully renders the role chip Driver. Akka's own ClusterSingletonOptions.Role matching (ControlPlane/ServiceCollectionExtensions.cs:52 AdminRole = "admin", :247 ?? RoleParser.Driver) is likewise ordinal and outside this repo's control — which is why the fix must normalize the gossiped plain-role strings, not only the local predicates.

Supporting facts verified: RoleParser.IsClusterRole is already case-insensitive (:37-40, Gitea #532) while ClusterIdFromRole (:53) deliberately preserves the operator's suffix spelling, with an XML <remarks> (:44-52) explicitly warning that normalizing one side "would silently rename the Akka role of a running deployment"; every in-repo ClusterId comparison is OrdinalIgnoreCase (ConfigDbApiKeyStore.cs:144-152, DeploymentArtifact.ParseDataPlaneAuthority, AclScopeRef); ClusterRoleInfo builds _localRoles as an OrdinalIgnoreCase HashSet (ClusterRoleInfo.cs:45) so HasRole is already case-insensitive but LocalRoles still exposes the verbatim strings; none of the four shipped appsettings*.json overlays carries a Cluster section (keys are Serilog + Security only), so the "the role overlay may not set Cluster:Roles" invariant this plan states costs zero migration; Install-Services.ps1:177-182 writes only OTOPCUA_ROLES to the Windows service environment ($normalisedRoles is trimmed + sorted + deduped but not lower-cased) and docs/deployments/wonder-app-vd03-makino-z-34184.md:46-60 records the field fix that hand-added Cluster__Roles__0/1 beside it — so prod runs both sources and OTOPCUA_ROLES must keep working; the repo has an established source-scanning convention-test idiom (CommandDisposalConventionsTests.cs:54-66, walking up from AppContext.BaseDirectory to the ZB.MOM.WW.OtOpcUa.slnx root) and an established reflection-census idiom (StatusCodeParityTests, PageAuthorizationGuardTests, DeferredSinkForwardingReflectionTests); EnvVarScope + the OTOPCUA_ROLES xUnit collection (tests/Core/…Cluster.Tests/EnvVarScope.cs:26-37) already serialize every test class that touches the process env var — new tests must join that collection.

Role-reading census (every production site, 7c77a8c7)

"Source" is what the site actually reads. "Normalizes?" is whether that site trims/case-folds before deciding. env = OTOPCUA_ROLES via RoleParser.Parse; cfg⇒env = ResolveEffectiveRoles (configured-first, env fallback); bound = AkkaClusterOptions.Roles (post-AddOtOpcUaCluster PostConfigure, i.e. cfg⇒env verbatim); gossip = Member.Roles off live Cluster.State.

#	Site	Source	Normalizes?	What it decides
1	Host/Program.cs:61	env only	lower+validate (via Parse)	the whole host role view
2	Host/Program.cs:62 roles.Contains("admin")	env only	n/a (bare literal, ordinal)	hasAdmin
3	Host/Program.cs:63 roles.Contains("driver")	env only	n/a (bare literal, ordinal)	hasDriver
4	Host/Program.cs:80 roleSuffix	env only	no (ordinal sort of already-lowered)	which appsettings.{role}.json overlay loads
5	Host/Program.cs:158/172/374/414/435/477/482/520/586/590/596/730/747/755/780	env only (via 2/3)	n/a	every conditional registration + endpoint map
6	Host/Program.cs:564 AddOtOpcUaHealth(hasAdmin)	env only	n/a	the ConfigDb health check
7	Host/Program.cs:624-626	env only	n/a	LocalDb-sync / ConfigServe / Telemetry listener ports
8	Host/Program.cs:789-790 banner	env only	n/a	the operator-visible boot line
9	Cluster/ServiceCollectionExtensions.cs:51 PostConfigure	cfg⇒env	no	AkkaClusterOptions.Roles — the authority for everything below
10	Cluster/ServiceCollectionExtensions.cs:267 Roles = options.Roles	bound	no	what Akka gossips
11	Cluster/SplitTopologyTransportValidator.cs:73-76,89-90	cfg⇒env	yes (trim + lower, :76)	split-topology transport fail-fast
12	Cluster/MeshTransportOptionsValidator.cs:76-79	cfg⇒env	no	GrpcPsk / CentralGrpcEndpoints fail-fast
13	Cluster/TelemetryOptions.cs:124-126 (TelemetryOptionsValidator)	cfg⇒env	no	Telemetry ApiKey / listen-port fail-fast
14	Cluster/ConfigSourceOptionsValidator.cs:101-104	cfg⇒env	no	driver-only ⇒ FetchAndCache fail-fast
15	Cluster/ClusterRoleInfo.cs:45 _localRoles	bound	yes for lookups (OrdinalIgnoreCase set); no trim; LocalRoles exposes verbatim	HasRole, /fleet + /hosts role chips
16	Cluster/ClusterRoleInfo.cs:52 ClusterRolesIn	bound	prefix case-insensitive; suffix verbatim	ClusterId / ClusterRole identity
17	Cluster/ClusterRoleInfo.cs:116/123/142/150	gossip	no	fleet-wide role→member index
18	ControlPlane/ServiceCollectionExtensions.cs:24,52 AdminRole = "admin"	literal ⇒ Akka	n/a (Akka matches ordinally vs gossip)	which node hosts the 5 admin singletons
19	ControlPlane/ServiceCollectionExtensions.cs:246-247	bound	prefix case-insensitive; falls back to RoleParser.Driver	redundancy-singleton cluster scope
20	ControlPlane/Redundancy/RedundancyStateActor.cs:146,193	gossip	no (Roles.Contains("driver"))	Primary election
21	ControlPlane/Redundancy/ManualFailoverService.cs:125	gossip	no	manual failover target set
22	ControlPlane/Fleet/ClusterNodeAddressReconcilerActor.cs:185	gossip	no	ClusterNode row reconciliation
23	Runtime/Drivers/DriverHostActor.cs:1612	gossip	no	R2-04 primary-gate driver member count
24	Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs:520	gossip	no	scripted-alarm emit gate member count
25	Runtime/ServiceCollectionExtensions.cs:323 GetService<IClusterRoleInfo>()	bound (via 15/16)	inherits 15	runtime actor cluster identity
26	ControlPlane/ServiceCollectionExtensions.cs:175 roleInfo.ClusterId/.ClusterRole	bound (via 16)	inherits 16	reconciler own-cluster scoping
27	Host/OpcUa/ConfigDbApiKeyStore.cs:83 ResolveClusterId	bound	prefix case-insensitive; compared OrdinalIgnoreCase (:150)	which cluster's ACL grants this node honours
28	Host/Program.cs:427 / :494	bound	inherits 27 / 19	ACL cluster filter; redundancy-singleton scope
29	Host/Health/HealthEndpoints.cs:82 RolePreference = [Admin, Driver]	literals ⇒ shared Health lib vs gossip	no	/health/active node preference
30	AdminUI/Components/Pages/Fleet.razor:149,159 MembersWithRole("driver")	gossip	no (bare literal)	the /fleet "Akka drivers" column
31	AdminUI/Components/Pages/Hosts.razor:480,487	gossip	display only (OrdinalIgnoreCase sort)	the /hosts role + leader-role chips
—	scripts/install/Install-Services.ps1:177-182	writes env only	trim + sort + dedupe, not lower	the prod Windows-service role knob
—	docker-dev/docker-compose.yml ×6 nodes	writes both, consistently	n/a	the rig — and the reason neither finding reproduces there

Census read: 31 production sites. 8 read the env var only (all in Program.cs, rows 1-8 — the whole of 01/S-12). 6 read cfg⇒env and only one of them normalizes (rows 11-14 — the whole of 01/C-9, plus the PostConfigure at row 9 and the Akka handoff at row 10 which normalize nothing). 11 read the bound options, mostly through ClusterRoleInfo, which is case-insensitive for lookups but verbatim for exposure. 6 match live gossip ordinally against a lowercase constant (rows 17, 20-24, 30) — the half neither finding names, and the half that turns a miscased Cluster:Roles into a no-Primary outage. 2 deployment surfaces set roles: prod (env only) and the rig (both).

How the two knobs relate today, exactly. There is no mapping of OTOPCUA_ROLES into Cluster:Roles in configuration — no AddEnvironmentVariables prefix trick, no alias. The two are read independently: Program.cs:61 reads the env var directly, and AddOtOpcUaCluster:51 post-configures the bound Cluster:Roles with ResolveEffectiveRoles, which falls back to the env var only when the bound array is empty. So the documented precedence — Cluster:Roles wins outright when non-empty, else OTOPCUA_ROLES (RoleParser.cs:104-109, docs/Configuration.md:126) — governs only the Akka side. The Host side has no precedence at all: it sees the env var and nothing else. The two views coincide on every node that sets both consistently (all six docker-dev nodes, and wonder-app-vd03 after its field fix) and diverge on exactly the shape #532 made first-class: Cluster:Roles set, OTOPCUA_ROLES unset. What operators are told: docs/Configuration.md:29 ("the roles themselves come from the OTOPCUA_ROLES env var"), :124 (Roles … "when empty, falls back to OTOPCUA_ROLES"), :126 (the two-knobs note, which already tells operators to prefer Cluster:Roles — i.e. the docs currently recommend the broken configuration), :449 (the env-var table), docs/ServiceHosting.md:19-27 (role gating is "Program.cs reads OTOPCUA_ROLES"), docs/Redundancy.md:211-231 (per-node identity example uses OTOPCUA_ROLES=admin,driver), docs/v2/Cluster.md:68 ("these two should stay in sync" — stated as an operator obligation, which is the whole defect). infra/ says nothing about roles.

Open design choices — resolved in this plan

1. Precedence (keep two sources + reconcile · vs · deprecate OTOPCUA_ROLES · vs · deprecate Cluster:Roles): resolved to keep both, one documented precedence — Cluster:Roles non-empty wins outright, else OTOPCUA_ROLES, else empty, i.e. ResolveEffectiveRoles's existing rule applied to both sides instead of one. Deprecating OTOPCUA_ROLES is rejected outright: Install-Services.ps1:182 writes it to the Windows service key, it is the only role knob on every non-field-fixed production install (memory reference_servecli_wonder_app_vd03: 23 env vars on the service key), and it names the appsettings overlay — a deprecation would break every existing deployment for a cosmetic gain. Deprecating Cluster:Roles is rejected because it is what Akka binds and what #532's own remediation made first-class. A Warning (not an error) when both are set and their normalized sets differ is added instead (choice 5).
2. Case normalization scope (normalize the whole gossiped list · vs · normalize only local predicates · vs · a two-layer value type): resolved to two layers in one type. EffectiveRoles carries a Gossiped list (what Akka receives) in which plain roles are lower-cased but a cluster role keeps its ClusterId suffix verbatim (Cluster-SITE-A ⇒ cluster-SITE-A), and exposes case-insensitive predicates (HasAdmin/HasDriver/Has) for every decision. Rationale: lower-casing a plain role changes the gossiped string only for a node that is already broken (a node spelled driver today is byte-identical after the change), while lower-casing a ClusterId suffix would rename cluster-MAIN/cluster-SITE-A on a running fleet — precisely what RoleParser.ClusterIdFromRole's <remarks> (:44-52) warns against, and a rolling upgrade of a pair would gossip two different role names mid-window and split it. Normalizing predicates only is insufficient because Akka's own ClusterSingletonOptions.Role match (rows 18/19) is ordinal and not ours to change.
3. Structural identity vs a runtime assert (the OVERALL action item's "boot guard asserting Akka and Host role views agree"): resolved to make them the same object, and the runtime assert is then not written. Program.cs resolves EffectiveRoles once and registers it as a singleton; AddOtOpcUaCluster sets options.Roles = effectiveRoles.Gossiped from that instance; the four validators take the instance by constructor injection. There is then no second derivation to disagree with, so an assert comparing two views would compare an object to itself — a test that can never fail is worse than no test, because it reads as coverage. What is kept is the one genuinely non-redundant residual check (choice 4). Note also that a literal "Akka gossip vs Host view" assert is not implementable where it would be wanted: gossiped roles do not exist until the ActorSystem is built, and resolving anything ActorSystem-shaped during composition is the Phase-6 boot-StackOverflow trap (CLAUDE.md; ControlPlane/ServiceCollectionExtensions.cs:239-248).
4. The residual divergence surface — the config-layer ordering trap. Program.cs must resolve roles before it adds the per-role overlay (:80 needs the role names to pick the filename), so the authority reads the pre-overlay configuration (base appsettings.json + env + args). If a role overlay — or a ${secret:} expansion — ever set Cluster:Roles, the post-overlay bind would differ from the instance every consumer holds. Resolved by stating the invariant ("Cluster:Roles may not be set in a role overlay") and enforcing it with a narrow fail-fast added to the existing AkkaClusterOptionsValidator: if the bound Roles (post-overlay) normalizes to a different set than the registered EffectiveRoles, fail host start naming both. This is 5 lines, is not the redundant assert of choice 3 (it compares two genuinely different config snapshots), and is a no-op today — none of the four shipped overlays carries a Cluster section.
5. Unknown / miscased role handling (silent · warn · fail): resolved to fail host start on an unknown role from either source, matching what Parse already does for the env path (:143-149) and closing the configured path's total absence of validation. The message follows the house validator style (MeshTransportOptionsValidator.cs:49-55): name the key, the offending value, the source it came from, and the allowed set. A miscased but known role does not fail — it normalizes silently, because failing it would brick every node an operator authored as Driver at exactly the moment they upgrade. Separately, a Warning fires when both sources are set and their normalized sets differ (the Cluster:Roles-wins case an operator almost certainly did not intend).
6. Gossip-side predicates (leave them ordinal · vs · one case-insensitive helper): resolved to a shared ClusterRoles.Has(IEnumerable<string> gossipedRoles, string role) helper (OrdinalIgnoreCase) adopted at all six ordinal gossip sites (rows 20-24, 30). With choice 2 in place a local node can no longer gossip a miscased plain role, so this is defence in depth for a peer running an older build — precisely the rolling-upgrade window choice 2 is designed to survive. Cheap, and it removes the last bare "driver" literals from production code so the guard test (choice 7) can be strict.
7. Guard style (reflection census · vs · source scan · vs · Roslyn analyzer): resolved to a source scan, because the thing being guarded is a textual read (Environment.GetEnvironmentVariable("OTOPCUA_ROLES"), GetSection("Cluster:Roles"), "driver") that reflection cannot see — the same limitation StatusCodeParityTests' own XML doc calls out ("an inline literal is invisible to this guard"). The repo already has the idiom and the root-walk helper (CommandDisposalConventionsTests.cs:54-66). A Roslyn analyzer was considered and rejected as disproportionate for two forbidden strings.
8. Rig fidelity (leave docker-dev alone · vs · make one node Cluster:Roles-only): resolved to change site-b-2 to Cluster:Roles-only (drop its OTOPCUA_ROLES). site-b is already the repo's designated A/B control node for the bootstrap guard, so it is the established place to pin a configuration variant, and site-b-2 is a driver-only node whose failure mode (deploy times out) is exactly what 01/S-12 predicts. Before the fix that node must fail the gate; after it, pass. Without this the gate is theatre — see theme 2.

---

Finding 1 — 01/S-12 = 03/C5: two role authorities can wire a node inconsistently (Medium, two independent reviewers)

Restatement

Program.cs:61-63 derives hasAdmin/hasDriver — the switches for the entire conditional host wiring, ~15 registration blocks, the health check, three gRPC listener ports and the appsettings overlay — solely from OTOPCUA_ROLES. The Akka cluster roles resolve configured-first through RoleParser.ResolveEffectiveRoles (RoleParser.cs:115-119), applied to the bound options by AddOtOpcUaCluster's PostConfigure (Cluster/ServiceCollectionExtensions.cs:51). A node configured only through Cluster:Roles — the shape #532's own remediation made first-class, and the shape docs/Configuration.md:126 currently recommends — joins the mesh advertising driver (and cluster-*), passes SplitTopologyTransportValidator and ConfigSourceOptionsValidator (which read effective roles), becomes an expected-ack member of every deployment via ConfigPublishCoordinator's enabled-ClusterNode-rows set, and times out every deployment forever because no driver stack was registered: no WithOtOpcUaRuntimeActors, no LocalDb, no OPC UA endpoint, no ConfigDb, and the deny-all NullOpcUaUserAuthenticator. It can even be the oldest role-carrying member for singleton placement without running the singleton manager. This is the exact inverse of the #532 incident, and nothing cross-checks the two sources.

Verification

Confirmed exact (see summary table). Precisely, at 7c77a8c7:

• Program.cs:61 — var roles = RoleParser.Parse(Environment.GetEnvironmentVariable("OTOPCUA_ROLES"));, with the comment "Roles drive the entire conditional wiring below" one line above. Nothing else in Program.cs reads Cluster:Roles; the file's only other role reads are the bound AkkaClusterOptions at :427 and :494, both for cluster identity (ClusterId / singleton scope), never for wiring.
• :62-63 — roles.Contains("admin") / roles.Contains("driver") against bare literals, not RoleParser.Admin/.Driver. (A second, independent brittleness: these two literals are invisible to any constant-based census.)
• :80 — the overlay filename is string.Join('-', roles.OrderBy(...)), so a Cluster:Roles-only node also loads no role overlay — a quiet second consequence neither report names.
• Cluster/ServiceCollectionExtensions.cs:34-51 — the PostConfigure. :267 hands options.Roles to Akka's ClusterOptions.
• Deploy-path consequence re-verified against CLAUDE.md's Phase-1 note: ConfigPublishCoordinator sources its expected-ack set from enabled ClusterNode rows, and Enabled = 0 is rejected by DraftValidator.ValidateClusterTopology; so a mis-wired node cannot even be excluded without the MaintenanceMode hatch. "Times out every deploy" is fleet-scoped, not node-scoped.
• Rig blindness re-verified: all six docker-dev nodes set OTOPCUA_ROLES and Cluster__Roles__* to the same values (:298/:352-354, :448/:485-487, :564/:606-607, :686/:716-717, :782/:803-804, :853/:874-875).

Root cause

Program.cs's role read predates the cluster options entirely and was never revisited when #532 introduced the fallback. #532 closed the env-only→no Akka roles direction by teaching the Akka side about the env var; it did not teach the Host side about Cluster:Roles, so the fix created a second authority rather than collapsing to one. The structural reason it stayed hidden is ordering: Program.cs needs roles at line 61, before the host is built and before any options bind, so reaching for Environment.GetEnvironmentVariable is the path of least resistance — and the configuration object that could answer the question (builder.Configuration, available from line 66) is created five lines later.

Proposed design

One authority, one object, consumed everywhere.

The type (src/Core/ZB.MOM.WW.OtOpcUa.Cluster/EffectiveRoles.cs, new):

/// <summary>The node's effective cluster roles — the SINGLE authority for "what is this node".
/// Resolved once, from one documented precedence, and consumed by BOTH the Host's conditional
/// wiring and the Akka role list, so the two cannot disagree (archreview 01/S-12 = 03/C5).</summary>
/// <remarks>
///     <para><b>Precedence:</b> <c>Cluster:Roles</c> when non-empty, else <c>OTOPCUA_ROLES</c>,
///     else empty. Unchanged from <see cref="RoleParser.ResolveEffectiveRoles"/> — what changed is
///     that <c>Program.cs</c> now obeys it too, instead of reading the environment directly.</para>
///     <para><b>Normalization is two-layer, on purpose.</b> <see cref="Gossiped"/> lower-cases plain
///     roles (Akka matches gossiped role names ORDINALLY, and every consumer compares against a
///     lowercase constant) but keeps a cluster role's ClusterId suffix VERBATIM — lower-casing that
///     would rename <c>cluster-SITE-A</c> on a running fleet, and a rolling pair upgrade would gossip
///     two different role names mid-window. Lower-casing a plain role changes the wire only for a
///     node that is already broken.</para>
/// </remarks>
public sealed class EffectiveRoles
{
    public static EffectiveRoles Resolve(IConfiguration configuration);      // the one factory
    public static EffectiveRoles Resolve(IReadOnlyList<string>? configured, string? envRoles);

    public IReadOnlyList<string> Gossiped { get; }   // what AkkaClusterOptions.Roles becomes
    public RoleSource Source { get; }                // Configured | Environment | None
    public bool HasAdmin { get; }
    public bool HasDriver { get; }
    public bool HasDev { get; }
    public bool Has(string role);                    // OrdinalIgnoreCase
    public string? ClusterRole { get; }              // verbatim suffix
    public string? ClusterId { get; }
}

• Resolve validates both sources: every entry must be admin/driver/dev (case-insensitive after trim) or a well-formed cluster-<non-empty>; otherwise it throws with the house message shape — "Cluster:Roles[1] = 'Drivr' is not a known role. Allowed: admin, driver, dev, or 'cluster-<ClusterId>'." — naming the key that supplied it.
• RoleParser.ResolveEffectiveRoles is retained as a thin delegation (=> EffectiveRoles.Resolve(configured, env).Gossiped.ToArray()) so nothing outside this plan's touched files breaks, and its XML doc points at the new authority. RoleParser.Parse stays as the env-string parser Resolve uses.

Program.cs (:61-63):

// The SINGLE role authority: Cluster:Roles first, OTOPCUA_ROLES as the fallback (archreview 01/S-12).
// Resolved off builder.Configuration BEFORE the per-role overlay is added, because the overlay's
// FILENAME depends on the answer — which is why a role overlay may not itself set Cluster:Roles
// (AkkaClusterOptionsValidator fails start if one ever does).
var builder = WebApplication.CreateBuilder(args);
var effectiveRoles = EffectiveRoles.Resolve(builder.Configuration);
var hasAdmin = effectiveRoles.HasAdmin;
var hasDriver = effectiveRoles.HasDriver;
builder.Services.AddSingleton(effectiveRoles);

WebApplication.CreateBuilder(args) moves above the role read (it is currently at :66). :80's roleSuffix uses effectiveRoles.Gossiped; :789-790's banner gains source={Source} so the boot line states which knob won — the single cheapest operator-visible artifact of this change.

AddOtOpcUaCluster (Cluster/ServiceCollectionExtensions.cs:34-51): the PostConfigure consumes the registered instance rather than re-deriving —

services.TryAddSingleton(sp => EffectiveRoles.Resolve(sp.GetRequiredService<IConfiguration>()));
services.AddOptions<AkkaClusterOptions>().PostConfigure<EffectiveRoles>(
    (options, roles) => options.Roles = [.. roles.Gossiped]);

TryAddSingleton keeps AddOtOpcUaCluster usable from harnesses that never ran Program.cs. This is the structural identity: AkkaClusterOptions.Roles is now by construction the same list hasDriver was computed from — there is no second derivation, hence no assert (resolved choice 3).

Residual guard (AkkaClusterOptionsValidator, existing, ValidateOnStart): if the bound Roles (post-overlay) normalizes to a different set than the injected EffectiveRoles.Gossiped, fail with "Cluster:Roles was changed by a configuration layer applied after role resolution (host view: [driver], bound: [admin,driver]). A per-role appsettings overlay may not set Cluster:Roles." No-op today (choice 4).

Alternatives considered:

• One-line fix: Program.cs:61 calls RoleParser.ResolveEffectiveRoles(builder.Configuration.GetSection("Cluster:Roles").Get<string[]>()) — the fix both reports propose. Rejected as the whole answer: it closes S-12 and leaves C-9, leaves the two bare "admin"/"driver" literals, leaves the configured path unvalidated, and — critically — leaves two derivations of the same rule (Program's and the PostConfigure's), which is the exact disease theme 4 names. It is, however, the correct first commit (T3 below) and is sequenced as such.
• Deprecate OTOPCUA_ROLES with a warning — rejected (choice 1).
• Map OTOPCUA_ROLES into configuration as Cluster:Roles via a custom IConfigurationSource — attractive (it would make one key, not two), but rejected: the env var is comma-separated while Cluster:Roles is an indexed array, so the mapping needs a bespoke provider whose ordering against AddEnvironmentVariables/AddCommandLine is subtle, and it would silently change what Cluster:Roles reads as in every diagnostic dump. A value type is simpler and states the precedence in code where a reader will find it.
• A runtime boot guard comparing Akka gossip to the Host view — rejected (choice 3), replaced by the narrower config-layer guard.

Implementation steps

1. src/Core/ZB.MOM.WW.OtOpcUa.Cluster/EffectiveRoles.cs (new) — the type, Resolve ×2, normalization, validation, RoleSource enum.
2. src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs — ResolveEffectiveRoles ×2 delegate to EffectiveRoles; XML docs re-point at the authority; Parse unchanged.
3. src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs — TryAddSingleton<EffectiveRoles>; PostConfigure at :51 consumes it.
4. src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs — :61-63 rewrite (builder first), :80 overlay, :789-790 banner + source=; register the singleton.
5. src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptionsValidator.cs — the post-overlay set-equality guard.

Tests

• tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/EffectiveRolesTests.cs (new; joins the OTOPCUA_ROLES collection via EnvVarScope): precedence matrix (configured-only / env-only / both-agree / both-differ / neither); Gossiped normalization (" Driver " ⇒ driver; Cluster-SITE-A ⇒ cluster-SITE-A — suffix case preserved, the falsification control for choice 2); HasDriver true for every casing; unknown role from either source throws naming the key; Source reports the winner.
• tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/AkkaClusterOptionsRoleFallbackTests.cs (existing, 5 cases) — must stay green unchanged; it is the #532 regression net.
• tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ClusterRolesOnlyBootTests.cs (new, on the DriverOnlyNoConfigDbBootTests direct-host-build pattern — not WebApplicationFactory, per TwoNodeClusterHarness.cs:48): build the host with Cluster:Roles = ["driver"] and OTOPCUA_ROLES unset, assert the driver stack is present (ILocalDb singleton resolvable, IOpcUaUserAuthenticator is ApiKeyOpcUaUserAuthenticator not NullOpcUaUserAuthenticator, the runtime actors registered) and that IOptions<AkkaClusterOptions>.Value.Roles equals ["driver"]. Negative control: the same test against the pre-fix build must FAIL — record that it does.
• AkkaClusterOptionsValidatorTests — one case: a post-role-resolution config layer setting Cluster:Roles fails start; the no-overlay case passes.

Effort + Risk

Effort: S. Risk: medium-high — this changes what every node wires and what it gossips. The change is behaviour-preserving for every node that sets both sources consistently (all six rig nodes, every current prod install) and behaviour-correcting for the two broken shapes. The residual risks are (a) the WebApplication.CreateBuilder move ahead of the role read — mechanical but it reorders the first ten lines of the composition root, and (b) a node whose Cluster:Roles and OTOPCUA_ROLES genuinely differ today will now wire from Cluster:Roles, which is a real behaviour change the Warning in choice 5 is there to announce. Land T1-T3 behind the full Cluster.Tests + Host.Tests suites, and do not merge without the live gate (T13) — a role-wiring regression is a total node outage and the rig is the only place it shows.

---

Finding 2 — 01/C-9: role normalization is stated in one validator of four (Medium)

Restatement

SplitTopologyTransportValidator normalizes roles (trim + ToLowerInvariant, :76) and its comment names the exact hazard: "the Cluster:Roles bind path is verbatim (no lowercasing)… so 'Driver' / 'Admin' cannot slip past the ordinal checks" (:69-72). Its three siblings do the opposite: MeshTransportOptionsValidator.cs:78-79, TelemetryOptions.cs:126, and ConfigSourceOptionsValidator.cs:103-104 all run case-sensitive Array.IndexOf(roles, RoleParser.Driver) over ResolveEffectiveRoles, which returns configured roles verbatim (RoleParser.cs:117 — only the env fallback is lowercased/validated by Parse). A node configured Cluster:Roles: ["Driver"] silently skips every driver-scoped boot-time rule (GrpcPsk-required, CentralGrpcEndpoints-required, FetchAndCache-required) and fails later at runtime with precisely the absence-class failures these validators exist to fail fast on. Nothing validates unknown or miscased plain role names on the configured path at all.

Verification

Confirmed exact (see summary table), and the report under-states the consequence. The same verbatim string is gossiped (Cluster/ServiceCollectionExtensions.cs:267), and six production predicates match gossip ordinally against the lowercase constant — census rows 17, 20-24, 30. Concretely, a pair configured Cluster:Roles: ["Driver", "cluster-SITE-A"]:

• RedundancyStateActor.SelectOldestUpMemberOfRole(members, "driver") (:146) matches no member ⇒ no Primary is ever elected ⇒ IsDriverPrimary false on both nodes ⇒ (R2-04) every operator write is denied "not primary", alarm acks are dropped, the alerts fan-out never publishes, and the LocalDb alarm store-and-forward queue never drains (HistorianDrainState.NotPrimary on both). ServiceLevel never rises.
• DriverHostActor.cs:1612 / ScriptedAlarmHostActor.cs:520 count 0 driver members, so the R2-04 primary gate takes its single-node "allow" branch on a genuine 2-node pair — the exact dual-primary window R2-04 exists to close, silently re-opened by a spelling.
• ClusterNodeAddressReconcilerActor.cs:185 sees no driver members ⇒ Error-logs EnabledRowNotInCluster for every row, forever.
• /hosts (Hosts.razor:487) renders the chip Driver and /fleet (Fleet.razor:149) shows zero Akka drivers — an operator sees a healthy-looking node in a fleet that reports none.

Also verified: AkkaClusterOptionsValidator — the fourth validator the task brief guessed at — reads no roles at all; the four are as listed above. And TelemetryDialOptionsValidator (TelemetryOptions.cs:160) reads none, so the central/dial side is unaffected.

Root cause

Two derivations of one rule, at two different times. SplitTopologyTransportValidator landed with Phase 6 (344b0d33) and normalized because its author reasoned about the two role sources explicitly; the other three copied the ResolveEffectiveRoles + Array.IndexOf shape without the normalization, which was the newer and less obvious half. Nothing structural connects the four — each re-derives "is this a driver node" from a raw array — so the correct spelling in one file is invisible to the next author. The deeper cause is the same as Finding 1's: normalization lives at the consumers instead of at the authority.

Proposed design

Delete the derivation from all four validators; inject the authority. Each validator's ctor takes EffectiveRoles instead of (or alongside) IConfiguration, and the four role lines collapse to roles.HasDriver / roles.HasAdmin. SplitTopologyTransportValidator additionally uses roles.ClusterRole / roles.ClusterId in place of its private FindIndex/ClusterIdFromRole pair — its message keeps the operator's original spelling because ClusterId preserves the suffix verbatim by design (choice 2). Its :59-72 comment block, which is now the history of a rule the authority states, shrinks to a one-line pointer.

Gossip-side defence in depth (choice 6). Add to RoleParser:

/// <summary>True when a GOSSIPED role list carries <paramref name="role"/>, compared
/// case-insensitively. Akka compares role names ordinally and a PEER may be running an older
/// build that gossips a verbatim 'Driver' (archreview 01/C-9), so every predicate over
/// Member.Roles goes through here rather than List.Contains.</summary>
public static bool GossipedRolesHave(IEnumerable<string>? gossipedRoles, string role);

Adopted at RedundancyStateActor.cs:193, ManualFailoverService.cs:125, ClusterNodeAddressReconcilerActor.cs:185, DriverHostActor.cs:1612, ScriptedAlarmHostActor.cs:520, Fleet.razor:149, and SelectOldestUpMemberOfRole (RedundancyStateActor.cs:146). ClusterRoleInfo's gossip index (:116/123/142/150) becomes an OrdinalIgnoreCase-keyed dictionary.

The guard that stops it recurring (choice 7). New tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/RoleAuthorityGuardTests.cs, scanning src/**/*.cs + src/**/*.razor from the solution root (the CommandDisposalConventionsTests.cs:54-66 root-walk):

• No file except EffectiveRoles.cs and RoleParser.cs contains OTOPCUA_ROLES (outside a comment or XML doc).
• No file except EffectiveRoles.cs contains "Cluster:Roles" as a configuration key (outside a comment / an error-message string literal — the scan strips // and /// lines and allows occurrences inside a string that also contains a space, which is how the validators' operator messages read).
• No file except RoleParser.cs declares or compares a bare "admin" / "driver" / "dev" role literal — the constants exist for this. This is the check that catches Program.cs:62-63 and Fleet.razor:149 today.
• Each rule reports every offending file:line, not the first, and the XML doc states the escape hatch (add the file to the small, named allow-list with a reason) so the guard is maintainable rather than deleted at the first inconvenience.

Alternatives considered:

• Normalize inside ResolveEffectiveRoles and leave the four validators alone — the fix 01/C-9 itself proposes. Rejected as insufficient on its own: it fixes the four validators (they inherit) but does nothing for the six ordinal gossip predicates, which are the half with the outage-grade consequence, and it leaves four re-derivations in place for the next rule to diverge on. It is, however, subsumed — the authority does normalize; the validators are converted anyway so the derivation stops existing.
• Make every gossip predicate case-insensitive and skip normalizing the gossiped list — rejected: Akka's own singleton Role matching (rows 18/19) is ordinal and not ours to intercept, so a miscased node would still fail to host its singletons.
• A Roslyn analyzer forbidding the literals — rejected as disproportionate (choice 7); the repo's analyzer project exists but the payload here is two strings.
• StringComparer.OrdinalIgnoreCase on the raw arrays at each of the four validators — rejected: that is the rule stated four more times, which is the finding.

Implementation steps

1. src/Core/ZB.MOM.WW.OtOpcUa.Cluster/SplitTopologyTransportValidator.cs — ctor takes EffectiveRoles; :59-90 collapses; comment shrinks to a pointer.
2. src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptionsValidator.cs — same, :71-79.
3. src/Core/ZB.MOM.WW.OtOpcUa.Cluster/TelemetryOptions.cs (TelemetryOptionsValidator) — same, :119-126.
4. src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptionsValidator.cs — same, :95-104.
5. src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs — GossipedRolesHave.
6. RedundancyStateActor.cs:146,193 · ManualFailoverService.cs:125 · ClusterNodeAddressReconcilerActor.cs:185 · DriverHostActor.cs:1612 · ScriptedAlarmHostActor.cs:520 · AdminUI/Components/Pages/Fleet.razor:149,159 · ClusterRoleInfo.cs:116-150 — adopt it.
7. src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs:56/63/70/80 — the four AddValidatedOptions registrations resolve the new ctor dependency (no signature change at the extension level; the validators are DI-constructed).

Tests

• tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/RoleCaseVariantValidatorTests.cs (new): a [Theory] over "Driver", " driver ", "DRIVER", "Admin", " cluster-SITE-A " × all four validators, asserting each fires the same failure it fires for the canonical spelling. The load-bearing case is the failure, not the pass: each variant must produce a ValidateOptionsResult.Fail naming the same key, because a validator that never runs its driver branch also "passes". Add the mirrored canonical-spelling case as the control.
• tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/RoleAuthorityGuardTests.cs (new, choice 7) — and it must be shown to fail on the pre-fix tree (Program.cs:62-63 and Fleet.razor:149 are live violations today); record the pre-fix offender list in the commit body.
• tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/GossipedRoleMatchTests.cs (new): GossipedRolesHave(["Driver"], "driver") true; (["cluster-SITE-A"], "cluster-site-a") true; null/empty false.
• tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests — one RedundancyStateActor case where the member gossips "Driver" and a Primary is still elected (the outage this finding actually causes; a pure-unit test over the selection helper is sufficient and deterministic).
• Existing suites stay green unchanged: SplitTopologyTransportValidatorTests (16 cases incl. the two OTOPCUA_ROLES-source cases at :238/:258), MeshTransportOptionsValidatorTests (incl. :308/:329), ConfigSourceOptionsValidatorTests, RoleParserTests, AkkaClusterOptionsRoleFallbackTests. Their construction changes from new XValidator(config) to new XValidator(EffectiveRoles.Resolve(config)) — mechanical, and it keeps them testing the validator rather than the resolution.

Effort + Risk

Effort: S-M (the four validators are ~4 lines each; the gossip-predicate sweep is 8 files; the guard test is the bulk). Risk: low-medium. The validators' behaviour is strictly widened (they now fire on inputs they used to skip), which can only turn a silent misconfiguration into a loud one — but that means a node an operator authored as Cluster:Roles: ["Driver"] will now refuse to start if it lacks its driver-scoped keys, where before it booted and misbehaved. That is the intent, and it is exactly the kind of change the live gate must exercise before merge. The gossip-side helper is behaviour-preserving for every correctly-spelled fleet.

---

Docs to update

File	Change
docs/Configuration.md:29	"The roles themselves come from the OTOPCUA_ROLES env var" ⇒ "…from Cluster:Roles, falling back to OTOPCUA_ROLES" (the overlay is now chosen from the effective roles).
docs/Configuration.md:124 (Cluster:Roles row)	Add: values are trimmed and (for plain roles) lower-cased; a cluster role's ClusterId keeps its spelling; an unknown role fails host start.
docs/Configuration.md:126 (the two-knobs note)	Rewrite. It currently says the two knobs drive different things ("OTOPCUA_ROLES drives all of Program.cs's conditional wiring; Cluster:Roles becomes the gossiped Akka roles") — that description is the defect. Replace with: one authority, one precedence, both consumers; the boot banner names the winning source; setting both to different values logs a Warning; a role overlay may not set Cluster:Roles. Keep the #532 history as a short parenthetical.
docs/Configuration.md:449 (env-var table)	OTOPCUA_ROLES row: read by EffectiveRoles.Resolve only, as the fallback when Cluster:Roles is empty.
docs/ServiceHosting.md:19-27	"Program.cs reads OTOPCUA_ROLES" ⇒ resolves EffectiveRoles; the role table and the "Single-node dev / Production" line get the precedence sentence.
docs/Redundancy.md:211-231	The per-node identity example shows OTOPCUA_ROLES=admin,driver; add the Cluster__Roles__0/1 equivalent and state that either works, both together must agree.
docs/v2/Cluster.md:68	Delete "these two should stay in sync" — the operator obligation this plan removes.
scripts/install/Install-Services.ps1	Doc-comment only: -Roles writes OTOPCUA_ROLES, which is now the documented fallback rather than the primary knob; no behaviour change (choice 1 — do not break it).
CLAUDE.md	One paragraph under the cluster/redundancy material: the role authority, the precedence, the guard test, and the "a role overlay may not set Cluster:Roles" invariant.

---

Task breakdown

Branch: one branch fix/archreview-r3-04-one-role-authority off master. Ordering is genuinely sequential for T1→T4 (each consumes the previous type), then the sweep and guards fan out. Verification bar: every task has a deterministic unit guard; T3, T4 and T9 additionally carry falsification controls (a test that must FAIL on the pre-fix tree); T13 is the merge-blocking live gate. Follow TDD: write the failing test, see it FAIL, implement, see it PASS.

---

T1 — EffectiveRoles value type: one factory, one precedence, two-layer normalization, both-source validation

Classification: HIGH-RISK (defines the node's identity; every downstream decision inherits it) Estimated implement time: 10 min Parallelizable with: T8 (guard-test scaffolding only); NOT T2-T7 Files: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/EffectiveRoles.cs (new), src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs, tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/EffectiveRolesTests.cs (new)

1. Write EffectiveRolesTests per Finding 1's Tests list — precedence matrix, Gossiped normalization including the cluster-SITE-A suffix-preservation control, Has* case-insensitivity, unknown-role-throws from both sources, Source. Join the OTOPCUA_ROLES xUnit collection (EnvVarScope).
2. dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~EffectiveRolesTests" → FAIL (no such type).
3. Implement EffectiveRoles + RoleSource; make RoleParser.ResolveEffectiveRoles (both overloads) delegate to it; leave Parse as the env-string parser.
4. Filter → PASS; then the full project: dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests → PASS (RoleParserTests + AkkaClusterOptionsRoleFallbackTests are the #532 regression net and must not need edits).
5. Commit: feat(cluster): EffectiveRoles — one role authority with one precedence + two-layer normalization (archreview 01/S-12, 01/C-9).

T2 — AddOtOpcUaCluster post-configures from the registered instance (structural identity)

Classification: high-risk (changes what the node gossips) Estimated implement time: 5 min Parallelizable with: T8; after T1 Files: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs, tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/AkkaClusterOptionsRoleFallbackTests.cs

1. Add a case: with Cluster:Roles = ["Driver", "Cluster-SITE-A"], the bound AkkaClusterOptions.Roles is ["driver", "cluster-SITE-A"] (plain lowered, suffix preserved) and is reference-equal in content to the registered EffectiveRoles.Gossiped. Run → FAIL.
2. Implement: TryAddSingleton<EffectiveRoles> + PostConfigure<EffectiveRoles> replacing :51.
3. Filter + full Cluster.Tests → PASS. Commit: fix(cluster): AkkaClusterOptions.Roles is the EffectiveRoles instance, not a second derivation.

T3 — Program.cs consumes the authority (closes 01/S-12 = 03/C5)

Classification: HIGH-RISK (every conditional registration in the composition root) Estimated implement time: 10 min Parallelizable with: nothing; after T2 Files: src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs

1. Regression gate BEFORE editing: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests → record green.
2. Implement: move WebApplication.CreateBuilder(args) above the role read; EffectiveRoles.Resolve(builder.Configuration); hasAdmin/hasDriver from it; register the singleton; :80 overlay from Gossiped; :789-790 banner gains source=.
3. dotnet build ZB.MOM.WW.OtOpcUa.slnx → PASS; dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests → PASS unchanged.
4. Commit: fix(host): Program.cs derives roles from the one authority, not OTOPCUA_ROLES alone (archreview 01/S-12 = 03/C5).

T4 — The four role-reading validators consume the authority (closes 01/C-9)

Classification: medium-risk (fail-fast rules now fire on inputs they used to skip) Estimated implement time: 10 min Parallelizable with: T5, T6, T7; after T2 Files: SplitTopologyTransportValidator.cs, MeshTransportOptionsValidator.cs, TelemetryOptions.cs, ConfigSourceOptionsValidator.cs, Cluster/ServiceCollectionExtensions.cs, tests/Core/…Cluster.Tests/RoleCaseVariantValidatorTests.cs (new) + the four existing validator suites

1. Write RoleCaseVariantValidatorTests (Theory: 5 spellings × 4 validators, each asserting the same Fail as the canonical spelling, plus the canonical control). Run → FAIL for three of the four (SplitTopology already normalizes — that asymmetry is the finding, and it makes the first run a built-in falsification control).
2. Implement: each validator's ctor takes EffectiveRoles; the ResolveEffectiveRoles + Array.IndexOf pairs collapse to roles.HasDriver / roles.HasAdmin; SplitTopologyTransportValidator uses roles.ClusterRole/.ClusterId; update the four existing test suites' construction.
3. Filter → PASS; full Cluster.Tests → PASS. Commit: fix(cluster): all four role-scoped validators read the authority — 'Driver' no longer skips every driver rule (archreview 01/C-9).

T5 — Gossip-side predicates: GossipedRolesHave at the six ordinal sites

Classification: medium-risk (Primary election + the R2-04 gate's member count) Estimated implement time: 10 min Parallelizable with: T4, T6, T7; after T1 Files: RoleParser.cs, RedundancyStateActor.cs, ManualFailoverService.cs, ClusterNodeAddressReconcilerActor.cs, DriverHostActor.cs, ScriptedAlarmHostActor.cs, ClusterRoleInfo.cs, AdminUI/Components/Pages/Fleet.razor, tests/Core/…Cluster.Tests/GossipedRoleMatchTests.cs (new), a RedundancyStateActor selection test

1. Tests: GossipedRolesHave truth table; and a RedundancyStateActor selection case where the member gossips "Driver" and a Primary is elected. Run → FAIL.
2. Implement the helper and adopt it at all seven sites (incl. SelectOldestUpMemberOfRole); ClusterRoleInfo's gossip index becomes OrdinalIgnoreCase-keyed.
3. Filters → PASS; run Cluster.Tests, ControlPlane.Tests, Runtime.Tests. Commit: fix(cluster): gossiped-role predicates compare case-insensitively — a miscased peer no longer un-elects the Primary.

T6 — Both-sources-disagree Warning + source= in the boot banner

Classification: standard (observability) Estimated implement time: 5 min Parallelizable with: T4, T5, T7; after T3 Files: EffectiveRoles.cs, Program.cs, EffectiveRolesTests.cs

1. Test: both sources set to different normalized sets ⇒ the resolve reports a diagnostic the caller logs (return it on the type — do not log from a static factory; Program.cs logs it after Serilog is configured). Run → FAIL.
2. Implement + emit at boot. Filter → PASS. Commit: feat(host): warn when Cluster:Roles and OTOPCUA_ROLES disagree; boot banner names the winning source.

T7 — Post-overlay config-layer guard in AkkaClusterOptionsValidator

Classification: standard (narrow fail-fast; no-op today) Estimated implement time: 5 min Parallelizable with: T4, T5, T6; after T2 Files: AkkaClusterOptionsValidator.cs, tests/Core/…Cluster.Tests/AkkaClusterOptionsValidatorTests.cs

1. Test: a configuration layer applied after role resolution changes Cluster:Roles ⇒ Fail naming both views; the ordinary case ⇒ Success. Run → FAIL.
2. Implement (set-equality against the injected EffectiveRoles.Gossiped). Filter → PASS. Commit: fix(cluster): fail start when a late config layer changes Cluster:Roles (role overlays may not set it).

T8 — RoleAuthorityGuardTests — the rule cannot be stated twice again

Classification: high-risk verification (this is the deliverable of theme 4, not a nicety) Estimated implement time: 15 min (the one deliberately-larger task — source scan + allow-list + comment stripping) Parallelizable with: T1-T7 for scaffolding; the assertions only go green after T3+T4+T5 Files: tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/RoleAuthorityGuardTests.cs (new)

1. Write the three rules per Finding 2 (no OTOPCUA_ROLES outside EffectiveRoles.cs/RoleParser.cs; no "Cluster:Roles" key outside EffectiveRoles.cs; no bare "admin"/"driver"/"dev" role literal outside RoleParser.cs), scanning src/**/*.cs + src/**/*.razor from the .slnx root, stripping ////// lines, reporting every offender as file:line, with a named allow-list each entry of which carries a reason.
2. Run on the pre-T3 tree → FAIL, listing at minimum Program.cs:61,62,63 and Fleet.razor:149,159. Paste that list into the commit body — it is the proof the guard can fail.
3. After T3-T5, run → PASS. Commit: test(cluster): guard that role reads live only in the authority (archreview theme 4).

T9 — Cluster:Roles-only node gets the driver stack (integration)

Classification: high-risk verification (the exact shape 01/S-12 describes) Estimated implement time: 15 min (harness plumbing) Parallelizable with: T5-T8; after T3 Files: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ClusterRolesOnlyBootTests.cs (new)

1. Build the host on the DriverOnlyNoConfigDbBootTests direct-host-build pattern with Cluster:Roles = ["driver"] and OTOPCUA_ROLES unset (via EnvVarScope(null)); assert the driver stack is registered (ILocalDb, ApiKeyOpcUaUserAuthenticator not the Null one, runtime actors) and AkkaClusterOptions.Roles == ["driver"].
2. Negative control: git stash T3's change (or run against master) → the test must FAIL with the Null authenticator / missing ILocalDb. Record it.
3. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~ClusterRolesOnlyBootTests" → PASS. Commit: test(integration): a Cluster:Roles-only node wires the driver stack.

T10 — Docs sweep

Classification: standard (docs) Estimated implement time: 15 min Parallelizable with: all Files: per the "Docs to update" table

Rewrite docs/Configuration.md:126 (the note that currently documents the defect as design), plus the seven other rows. Commit: docs: one role authority — precedence, normalization, and the overlay invariant (R3-04).

T11 — docker-dev rig: make the divergence observable

Classification: standard (rig config) — but it is what makes T13 non-vacuous Estimated implement time: 5 min Parallelizable with: T10; before T13 Files: docker-dev/docker-compose.yml, docker-dev/README.md

1. Drop OTOPCUA_ROLES from site-b-2 (keep its Cluster__Roles__0/1), and add a comment naming this as the Cluster:Roles-only pin for archreview 01/S-12 — site-b is already the rig's A/B control node. Leave the other five nodes as they are (they are the "both sources agree" control).
2. Add a docker-dev/README.md row saying so, and that the node is expected to fail deploys on a pre-R3-04 build.
3. Commit: chore(docker-dev): site-b-2 is Cluster:Roles-only — the rig can now see the two-authority split.

T12 — Optional: a miscased pin

Classification: standard (rig config) Estimated implement time: 5 min Parallelizable with: T11; before T13 Files: docker-dev/docker-compose.yml

Spell site-a-2's Cluster__Roles__0 as Driver (capital D), keeping cluster-SITE-A verbatim, as the standing 01/C-9 pin. Expected post-fix: site-a still elects exactly one Primary and /hosts shows a lowercase driver chip on both nodes. Pre-fix: site-a elects none. Commit: chore(docker-dev): site-a-2 pins a miscased Cluster:Roles entry (archreview 01/C-9).

T13 — LIVE GATE: docker-dev six-node rig — operator/rig task

Classification: HIGH-RISK verification gate (merge-blocking) Estimated implement time: ~40 min rig session Parallelizable with: nothing (needs the built branch + T11/T12) Files: none (rig session; findings recorded in archreview/plans/STATUS.md)

How roles are set on the rig today: every node sets OTOPCUA_ROLES and Cluster__Roles__* to the same values — central-1/2 admin,driver,cluster-MAIN (:298/:352-354, :448/:485-487), site-a-1/2 driver,cluster-SITE-A (:564/:606-607, :686/:716-717), site-b-1/2 driver,cluster-SITE-B (:782/:803-804, :853/:874-875). T11/T12 make two of them divergent on purpose. Login is disabled on the rig, so drive the AdminUI directly at http://localhost:9200 (round-robins central-1/central-2 — rebuild both).

1. Pre-fix baseline (on master, with T11/T12's compose applied). docker compose -f docker-dev/docker-compose.yml up -d --build → confirm the failure is real: site-b-2's log shows the boot banner with no roles (roles=[]), /hosts shows it gossiping driver while /fleet counts it, and a fleet deploy seals TimedOut naming site-b-2 as silent. site-a shows no Primary (both nodes' Server_ServiceLevel low; /hosts shows no driver leader role). Record both — without this step the gate proves nothing.
2. Build the branch: docker compose -f docker-dev/docker-compose.yml up -d --build on all six.
3. Role parity, per node: docker logs otopcua-dev-<node>-1 | grep "starting with roles" → each node's banner names the same role set its compose entry declares, with source=Configured, and the two divergent nodes report the config roles (site-b-2 driver,cluster-SITE-B; site-a-2 driver,cluster-SITE-A — lower-cased). Cross-check against /hosts, which renders the gossiped m.Roles per member (Hosts.razor:487): the chips must match the banners exactly, on all six. That comparison — banner (Host view) vs /hosts chips (Akka view) — is the live form of the "two views agree" guard.
4. site-a Primary election (01/C-9): /hosts shows exactly one driver leader role in the SITE-A pair; Client.CLI read of Server_ServiceLevel on opc.tcp://localhost:4842 vs :4843 shows the 250/240 split. Pre-fix this leg showed neither.
5. Deploy seals Applied (01/S-12): deploy a config from /fleet; it must seal Applied with an ack from all six nodes including site-b-2 (pre-fix: TimedOut). Confirm site-b-2 actually serves — Client.CLI browse -u opc.tcp://localhost:4845 returns the deployed tree.
6. Fail-fast still fails (T4 regression): temporarily set site-b-1's Cluster__Roles__0 to Drivr, docker compose up -d site-b-1 → the container must exit with the unknown-role message naming Cluster:Roles. Revert.
7. Record every leg in archreview/plans/STATUS.md with the pre-fix baseline beside the post-fix result.

---

Sequencing summary

T1 → T2 → T3 → T9              (the authority, then the Host, then its integration proof)
        ↘ T4, T5, T6, T7        (validators / gossip predicates / warning / overlay guard — parallel after T2)
T8 scaffolded early; green only after T3+T4+T5
T10 anytime
T11, T12 (rig config) before T13; T13 last, after everything is merge-ready

Overall effort: ~1 dev-day code+unit, plus the rig session. The three HIGH-RISK slices (T1 identity type, T3 composition root, T8 the guard) each land behind their full project suites before anything stacks on them; T13 is merge-blocking and is only meaningful because T11/T12 gave the rig a node that can actually exhibit the defect.

---

Execution notes (recorded during T1–T12 implementation, 2026-08-20)

T5 — SelectOldestUpMemberOfRole/SelectDriverPrimary cannot be reached from this repo; the plan was wrong about this one call site. RedundancyStateActor.SelectOldestUpMemberOfRole (RedundancyStateActor.cs:172-178) is a thin wrapper around ClusterActiveNode.OldestUpMember (ZB.MOM.WW.Health.Akka, a separately-versioned NuGet package out of scadaproj), which itself matches via Akka's own Member.HasRole — an ordinal comparison this repo does not own. This is structurally identical to the "not ours to change" carve-out the plan's own choice 2 already makes for ClusterSingletonOptions.Role (rows 18/19); the plan simply did not notice the same constraint applies to SelectOldestUpMemberOfRole before naming it as a GossipedRolesHave adoption site.

Consequence: the plan's stated T5 test — "a RedundancyStateActor selection case where the member gossips 'Driver' and a Primary IS elected" — is not achievable. What shipped instead (tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/GossipedRoleCaseInsensitivityTests.cs):

• GossipedRolesHave was implemented and adopted at every site that genuinely reads gossiped roles directly in this repo: RedundancyStateActor.cs (BuildSnapshot's own role classification, NOT the Primary-selection call), ManualFailoverService.cs (OrderedDrivers), ClusterNodeAddressReconcilerActor.cs (FilterOwnClusterDriverMembers), DriverHostActor.cs and ScriptedAlarmHostActor.cs (DriverMemberCount), and Fleet.razor (the bare "driver" literals, now RoleParser.Driver — the actual case-insensitivity there comes from ClusterRoleInfo's _roleLeaders/_membersByRole dictionaries moving to OrdinalIgnoreCase, which IClusterRoleInfo.MembersWithRole/RoleLeader read through).
• A_member_gossiping_a_miscased_driver_role_classifies_Secondary_not_Detached pins the fix that is achievable: a single member gossiping "Driver" now classifies Secondary in the redundancy snapshot (was Detached pre-fix) — real value, because /hosts and /fleet render this list, even though that member can still never be named Primary by itself.
• Primary_election_itself_remains_ordinal_and_still_elects_no_one_for_an_all_miscased_pair pins the true (negative) outcome instead of asserting a false positive: a node (or pair) gossiping only "Driver" still elects no Primary, because that decision runs through the vendored ordinal match. This is a genuine residual — not closed by R3-04 — worth a follow-up issue against ZB.MOM.WW.Health.Akka's ClusterActiveNode.OldestUpMember (e.g. an optional case-insensitive role-match parameter) if the rolling-upgrade window this defends against is judged worth closing fully. In practice T1/T2 already close the headline outage the census describes for a FIXED-build node (a node can no longer gossip a miscased plain role itself — normalization happens before the gossip handoff), so this residual only bites a peer still running a pre-R3-04 build.
• ManualFailoverService_recognises_a_miscased_driver_node_as_a_candidate uses GetSnapshot()/PrimaryAddress rather than FailOverCore(dryRun: true) as originally sketched — FailOverCore refuses below 2 driver members ("nothing to fail over TO"), which a single-node fixture can never satisfy; GetSnapshot() shares the same OrderedDrivers candidate selection this test is actually about, without needing a second live node.

No other T1–T5 deviations from the plan.

T6-T9 — no deviations. Implemented as designed; AkkaClusterOptionsValidator gained an optional (default-null) EffectiveRoles? constructor parameter rather than a required one specifically so the pre-existing zero-arg test constructions in AkkaClusterOptionsValidatorTests did not all need touching — the guard is simply inert when nothing is injected, which is also the correct behaviour for any harness that builds this validator without the rest of the role authority.

T8 — two additional bare-role-literal offenders found and fixed, beyond the two the plan named. The census's row-by-row list did not claim to be exhaustive of every bare literal in the tree, only of the production predicate sites; running the guard for real (rather than reasoning about it) found two more before it could go green:

• DriverInstanceActor.ShouldStub's roles.Contains("dev") — a genuine Akka-cluster-role read (_localRoles, threaded from IClusterRoleInfo.LocalRoles) gating dev-stub mode for Windows-only drivers. Fixed to RoleParser.GossipedRolesHave(roles, RoleParser.Dev).
• ControlPlane/ServiceCollectionExtensions.AdminRole = "admin" — a second constant duplicating the literal RoleParser.Admin names, feeding Akka's own ClusterSingletonOptions.Role matching (the same "not ours to change" ordinal match as SelectOldestUpMemberOfRole). The VALUE was already correct (Akka's own matching is unaffected either way); fixed to alias RoleParser.Admin rather than duplicate it, matching the sibling Runtime/ServiceCollectionExtensions.DriverRole pattern.

The guard also required a small allow-list (six files) for bare "admin"/"driver"/"dev" literals that name something OTHER than an Akka Cluster:Roles role — a raw-tree node-KIND discriminator, an ACL-scope entity-type name, the Sql driver's own TagConfig JSON marker (×3 files), and an LDAP-derived AdminUI RBAC role. Each carries its reason in RoleAuthorityGuardTests.RoleLiteralAllowListWithReasons, per the plan's "add to the allow-list WITH A REASON" instruction; a companion test (Every_allow_list_entry_has_a_reason_and_still_applies) fails if an entry goes stale.

The falsification control (The_scan_actually_detects_the_pre_R3_04_offenders) could not literally re-run the guard against the pre-T3/pre-T5 GIT COMMITS (T1–T8 landed sequentially in this same worktree, so "the tree" is always post-fix by the time T8's test runs) — it instead feeds the scan logic the exact pre-fix Program.cs/Fleet.razor snippets as string literals, captured verbatim from this session's own earlier reads of those files before they were edited. Functionally equivalent proof-of-falsifiability; recorded as a deviation because it is a different MECHANISM than "run on the pre-T3 tree," not because the result differs.

T9 — scope note, not a deviation. "The driver stack" is proven via two representative slices (ILocalDb resolution + the IOpcUaUserAuthenticator selection block), not the ENTIRE WithOtOpcUaRuntimeActors Akka actor graph — replicating that too would need a real ActorSystem (the exact thing DriverOnlyNoConfigDbBootTests's own doc comment explains this test family avoids). The negative control uses an inline PreFixDerivation helper (reading OTOPCUA_ROLES only, matching pre-T3 Program.cs:61-63 verbatim) rather than a git stash, for the same "already sequential commits in one worktree" reason as T8's falsification control — and it is run in the SAME test process against the SAME configuration, which is a stronger control than a separate stash-and-revert cycle would have been (both derivations are proven against literally identical input).

T11/T12 — combined into one commit. The docker-compose.yml header comment documenting all six nodes' role sources interleaves site-a-2 and site-b-2 on adjacent lines, so a clean git-level split into the plan's two separate commits was not practical; both tasks' edits landed in one commit (985de0b6). The rig was NOT brought up as part of either task, per the brief.

Review outcome

A code review after T1–T12 returned verdict FIX-FIRST, two items. Both are fixed, each in its own TDD commit, on top of the T1–T12 work above.

1. [Blocking] T7's post-overlay guard was structurally unreachable

The finding. AkkaClusterOptionsValidator.ValidateAgainstEffectiveRoles compared options.Roles against _effectiveRoles.Gossiped — but ServiceCollectionExtensions.cs's PostConfigure (T2's OWN structural-identity fix) unconditionally sets options.Roles = [.. effectiveRoles.Gossiped] first. The two values are therefore equal by construction, on every real host, always — the guard could never observe a divergence no matter what a role overlay did to Cluster:Roles. My original two tests (now deleted) hand-built an AkkaClusterOptions instance directly and set .Roles to a "mismatched" value bypassing PostConfigure entirely — which exercised the comparison logic in isolation but never proved the guard reachable through the real pipeline. Concretely, before this fix: an overlay that set Cluster:Roles was silently ignored (PostConfigure overwrote it back to the pre-overlay value), not failed at boot — the exact opposite of the guard's purpose (choice 4: "a role overlay may not set Cluster:Roles").

The fix. AkkaClusterOptionsValidator now also takes an optional IConfiguration? (default null, same pattern as EffectiveRoles? — so every pre-existing zero/one-arg test construction keeps compiling and behaving as before). The guard re-reads the RAW Cluster:Roles section directly off the LIVE configuration (_configuration.GetSection(EffectiveRoles.ConfiguredRolesKey).Get<string[]>()) rather than the bound options, normalizes it by reusing EffectiveRoles.Resolve(raw, envRoles: null) (one rule, stated once — archreview theme 4, rather than re-implementing the configured-path normalization a second time), and compares that against _effectiveRoles.Gossiped. Skipped entirely when the raw post-overlay value is empty — required for correctness, not merely tidiness: an OTOPCUA_ROLES-only node (most of production, per CLAUDE.md) has an empty Cluster:Roles both before AND after any overlay, and comparing "empty" against a non-empty env-derived Gossiped list would be a permanent false positive on every one of those nodes. AddOtOpcUaCluster now also TryAddSingleton(configuration) so a bare ServiceCollection test harness (not just WebApplicationBuilder, which auto-registers IConfiguration) can supply it.

The tests. Both guard tests are rewritten to go through the REAL AddOtOpcUaCluster + PostConfigure pipeline, over a ConfigurationManager (not a one-shot ConfigurationBuilder) — the same "add a provider to the SAME live object after the fact" shape WebApplicationBuilder.Configuration has, which is what lets a role overlay be simulated faithfully: EffectiveRoles.Resolve(configuration) is called and its result registered BEFORE a second AddInMemoryCollection call mutates the same configuration object with different Cluster:Roles values, mirroring Program.cs's own resolve-then-AddJsonFile(overlay) sequencing exactly. Confirmed red first: the "changes" test failed with "should throw OptionsValidationException but did not" against the pre-fix (options.Roles -comparing) validator — direct proof of the reviewer's structural-unreachability claim. Two supporting tests were added beyond the two named in the review comment: No_overlay_at_all_passes (the ordinary case) and EffectiveRoles_injected_without_IConfiguration_skips_the_guard_entirely (the two-optional-dependency ctor's other skip leg). One self-inflicted false-positive surfaced and was fixed along the way: the new catch-block's error message split "Cluster:Roles." onto its own string-literal token, which RoleAuthorityGuardTests' own space-inside-the-literal heuristic (T8) correctly flagged as NOT looking like an operator sentence — merging it back into one literal containing surrounding words fixed it without weakening the guard.

2. [Important] ClusterRoleInfo._roleLeaders must stay Ordinal, not OrdinalIgnoreCase

The finding. T5 moved BOTH _roleLeaders and _membersByRole to OrdinalIgnoreCase under one shared rationale (defence in depth against a miscased rolling-upgrade peer). That rationale is right for _membersByRole ("every member carrying a spelling of this role" — merging casings is the point) but wrong for _roleLeaders: Akka computes — and raises RoleLeaderChanged for — role leadership per exact gossiped role string. "driver" and "Driver" are two entirely distinct roles to Akka's own role-leader machinery, each independently electing its own leader. Keying _roleLeaders case-insensitively collapses both onto ONE dictionary slot, so whichever RoleLeaderChanged event landed last silently overwrites the other's leader — turning two real, independent facts Akka tracks separately into one, wrong, merged fact.

The fix. _roleLeaders reverted to StringComparer.Ordinal; _membersByRole stays OrdinalIgnoreCase (the split itself, not merely one value, is now documented in the field comment). No other production call site touches _roleLeaders's comparer, so this is a one-line revert plus doc.

The test. ClusterRoleInfoRoleLeaderCaseTests.RoleLeaderChanged_for_driver_then_Driver_keeps_distinct_entries — a real two-node cluster (the only faithful option: HandleRoleLeaderChanged is internal with no InternalsVisibleTo, so it cannot be unit-tested directly, and ClusterRoleInfo's constructor needs a genuinely clustered ActorSystem). Node A gossips only "driver", node B (joining the same cluster) gossips only "Driver" — each is therefore the sole, and so leader, member of its own exact-string role. Confirmed red first: against the pre-fix (OrdinalIgnoreCase) comparer, the test failed with driverLeader and capitalDriverLeader resolving to the SAME address — the exact collision the review predicted, reproduced live rather than merely reasoned about.

Optional item — done

Added one sentence to docs/ServiceHosting.md's "Per-role configuration overlays" section: a plain role's overlay-filename fragment is always lower-case regardless of how it was spelled in Cluster:Roles, but a cluster-{ClusterId} role's suffix carries into the filename verbatim (since EffectiveRoles.Gossiped preserves that spelling by design) — e.g. Cluster:Roles: ["driver", "cluster-SITE-A"] loads appsettings.cluster-SITE-A-driver.json.

Verification

dotnet build ZB.MOM.WW.OtOpcUa.slnx → 0 errors throughout.

• After item 2 (ClusterRoleInfo): Cluster.Tests 241 → 242/242 (+1, ClusterRoleInfoRoleLeaderCaseTests).
• After item 1 (T7 guard): Cluster.Tests 242 → 244/244 (net +2: the two old hand-built T7 tests deleted, four new ones added — An_overlay_that_changes_Cluster_Roles_after_resolution_fails_naming_both_views, An_overlay_that_repeats_the_same_Cluster_Roles_passes, No_overlay_at_all_passes, EffectiveRoles_injected_without_IConfiguration_skips_the_guard_entirely — the pre-existing No_EffectiveRoles_injected_skips_the_guard_entirely was kept unchanged).
• Host.Tests 106/106, unchanged by either fix (Program.cs itself was not touched).
• ControlPlane.Tests 267/267 — an extra safety check beyond what was asked, since ClusterRoleInfo is a shared dependency of RedundancyStateActor et al.

No pushes, no merges.

Live gate result (2026-08-20) — PASS on docker-dev (pre-fix control at 7c77a8c7 failed first)

Rig: docker-dev/docker-compose.yml on OrbStack (Mac), fresh volumes (down -v before the control), the six host containers + SQL + Traefik, AdminUI on Security__Auth__DisableLogin=true. The compose carried the T11/T12 pins in BOTH runs (site-b-2 Cluster__Roles__* only, no OTOPCUA_ROLES; site-a-2 Cluster__Roles__0="Driver"). Deploys via ./docker-dev/deploy.sh (POST /api/deployments, X-Api-Key: docker-dev-deploy-key, 202 both times). Server_ServiceLevel (i=2267) read with Client.CLI on :4840–:4845. Image parity asserted ×6 via docker inspect --format '{{.Image}}' in both runs. The Modbus fixture on 10.100.0.35:5020 was left running throughout. Note: master advanced 9242630c → f665e8b8 (R3-05 S18 follow-up) while the post-fix run was in progress; the post-fix image had already been built from the 9242630c working tree (image created 13:29:46 local, merge landed 13:29:59) — the only later delta is BoundedApiKeyStore.cs + a compose env wire, neither of which touches roles.

Leg 1 — pre-fix control, image otopcua-host:prefix-r304 = 32a2dcee0285 (scratch worktree at 7c77a8c7) ×6: FAILED as required

what the plan predicted	observed
site-b-2 boot banner shows no roles	[17:25:22 INF] OtOpcUa.Host starting with roles=[] (admin=False, driver=False) — every other node roles=[admin,driver,cluster-main] / [driver,cluster-site-a] / [driver,cluster-site-b] (pre-fix banner lower-cases the cluster suffix; no source=)
site-b-2 gossips driver anyway	site-b-1: Node [akka.tcp://otopcua@site-b-2:4053] is JOINING, roles [cluster-SITE-B,driver]; site-b-2's own cluster-primary health check: Standby: active node for role 'driver' is akka.tcp://otopcua@site-b-1:4053 — Akka counted it as a driver member while the Host wired nothing
fleet deploy seals TimedOut naming site-b-2	Deployment 6c653c61… (rev 942ad3cc…): Status = 4, SealedAtUtc = NULL; NodeDeploymentState five rows Status = 1 (AppliedAtUtc 17:25:41.077–.107), site-b-2 Status = 0, AppliedAtUtc = NULL. central-2: [17:27:41 WRN] Deployment 6c653c61… timed out after 00:02:00 (5/6 acks landed). No ack from: site-b-2:4053. Each is an enabled, non-maintenance ClusterNode row …
site-b-2 does not serve	Client.CLI read on opc.tcp://localhost:4845 → Remote side closed connection (no OPC UA server started; no DriverHost / Kestrel lines in its log at all, only Akka remoting + cluster join)
site-a (01/C-9): "elects no Primary"	Partly wrong, and the execution notes already said why. site-a-1 gossips ordinal driver, so ClusterActiveNode.OldestUpMember still elected it: :4842 = 250. What failed was site-a-2: gossiped roles [cluster-SITE-A,Driver] (site-a-1's JOINING line), its own cluster-primary check Node carries none of the active roles: admin, driver., and :4843 = 0 — the pair ran with a Primary and no eligible standby. A pair whose nodes were BOTH miscased would elect none (pinned by Primary_election_itself_remains_ordinal_and_still_elects_no_one_for_an_all_miscased_pair); the rig pins only one
control reference	:4840=240, :4841=250 (MAIN), :4844=250 (site-b-1 sole eligible driver)

Leg 2 — post-fix, image otopcua-host:dev = b7cfae1be510 (master 9242630c) ×6: PASS

1. Role banners, all six source=Configured, config roles win, normalised — central-1/2 starting with roles=[admin,driver,cluster-MAIN] (admin=True, driver=True) source=Configured; site-a-1/site-a-2 roles=[driver,cluster-SITE-A] (the Driver pin lower-cased, cluster suffix verbatim); site-b-1/site-b-2 roles=[driver,cluster-SITE-B] (admin=False, driver=True) — the Cluster:Roles-only node now wires the driver stack. Gossip view agrees with the Host view on every node: site-a-1 Node [site-a-2] is JOINING, roles [cluster-SITE-A,driver] (was Driver), site-b-1 Node [site-b-2] is JOINING, roles [cluster-SITE-B,driver], central-2 Node [central-1] is JOINING, roles [cluster-MAIN,driver,admin]. (/hosts chips were not used as the cross-check — the page is a Blazor circuit and a curl of it renders no member rows; the Akka JOINING lines are the same gossiped m.Roles the page reads.)
2. Deploy seals Applied 6/6 (01/S-12): Deployment a5a7018d… (same revision 942ad3cc…): Status = 2, CreatedAtUtc 17:31:21.914, SealedAtUtc 17:31:22.138 (224 ms); six NodeDeploymentState rows all Status = 1, AppliedAtUtc 17:31:22.090–.132 including site-b-2. central-1: Deployment a5a7018d… sealed (acks=6). site-b-2: Fetched + verified artifact … from http://central-1:4055 (1468 bytes) → DriverHost site-b-2:4053: applied fetched deployment a5a7018d… (rev 942ad3cc…, children=0) → cached … as this node's local fallback. Client.CLI browse -u opc.tcp://localhost:4845 -d 2 returns Server, Aliases, Locations, OtOpcUa (ns=2;s=OtOpcUa) — site-b-2 serves (was Remote side closed connection).
3. One Primary per pair, with an eligible standby (01/C-9): :4840=240/:4841=250 (MAIN), :4842=250 / :4843=240 (SITE-A — site-a-2 was 0 pre-fix; its health check now reads Standby: active node for role 'driver' is akka.tcp://otopcua@site-a-1:4053), :4844=250/:4845=240 (SITE-B — site-b-2 was unreachable). After the leg-5 restart of site-b-1 the SITE-B pair re-elected with site-b-2 as Primary (:4845=250, :4844=240) — the formerly role-less node now holds the Primary-gated data plane.
4. T6 disagreement Warning fires: scratch compose override on site-b-1 setting OTOPCUA_ROLES: "driver" against its Cluster__Roles__0/1 = driver,cluster-SITE-B → [17:32:35 INF] … starting with roles=[driver,cluster-SITE-B] … source=Configured followed by [17:32:35 WRN] Cluster:Roles ([driver,cluster-SITE-B]) and OTOPCUA_ROLES ([driver]) disagree; Cluster:Roles wins outright. — and the node still joined and served with the config roles. (The rig's standing pins do NOT fire this Warning by themselves: site-a-2's Driver normalises equal to its env value, and site-b-2 has only one source — choice 5 is explicitly "both set AND differ".)
5. Unknown role fails host start: override Cluster__Roles__0: "operator" on site-b-1 → the log is exactly six lines: Unhandled exception. System.ArgumentException: Cluster:Roles[0] = 'operator' is not a known role. Allowed: admin, driver, dev, or 'cluster-<ClusterId>'. at EffectiveRoles.NormalizeAndValidateConfigured ← EffectiveRoles.Resolve(IConfiguration) ← Program.cs:67 — no boot banner, no Akka remoting, no Kestrel, no cluster join. Override removed; site-b-1 recreated from the plain compose → roles=[driver,cluster-SITE-B] … source=Configured, no Warning, (healthy). ⚠️ Observation, pre-existing and NOT R3-04's: the dotnet OtOpcUa.Host.dll process did not exit after the unhandled exception — docker inspect kept reporting running exit=0 restarts=0 for > 2 min, the main thread in state R burning ~50 % of a core with 14 Server-GC threads parked. Reproduced identically on a bare docker run --rm -e OTOPCUA_ROLES=operator against both otopcua-host:prefix-r304 (7c77a8c7, pre-fix RoleParser.Parse throw at Program.cs:61) and the master image: neither exits within 90 s (timeout → 124). So "fail-start" holds (nothing is wired, nothing listens) but the container's unless-stopped restart policy never fires and an orchestrator watching the exit code would not see a crash; only the compose healthcheck (never healthy) shows it. Arm64 OrbStack aspnet:10.0 runtime image; root cause not chased here — worth a follow-up (is it the runtime's unhandled-exception abort path on this host, or something in our Program.cs top-level statements?) before relying on a process-exit signal for any of the boot-time validators (SplitTopologyTransportValidator, AkkaClusterOptionsValidator, ConfigSourceOptionsValidator fail the same way).

Leg 3 — restore

Six hosts on b7cfae1be510 ×6 (parity asserted), GET /health/ready → 200 via :9200, SQL + Traefik up, all six ServiceLevels one 250 + one 240 per pair; otopcua-pymodbus-standard on 10.100.0.35 untouched (Up 2 weeks). Scratch worktree at 7c77a8c7 removed; the control image tag otopcua-host:prefix-r304 left in the local image cache (same layer id as R3-03's prefix-r303 — both are 7c77a8c7).

Verdict: T13 PASS. Both findings reproduce on the pre-fix build with the rig as pinned (01/S-12 as a real fleet deploy outage; 01/C-9 as a standby-less pair rather than the plan's "no Primary" — the execution notes' ordinal-election residual, confirmed live), and neither reproduces on master. One pre-existing process-does-not-exit-after-fail-start observation recorded above for follow-up.