docs: one role authority — precedence, normalization, and the overlay invariant (R3-04)
This commit is contained in:
Joseph Doherty
@@ -435,6 +435,50 @@ Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBr
|
||||
|
||||
See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)", and `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
|
||||
|
||||
## Role authority (`EffectiveRoles`)
|
||||
|
||||
**A node's roles are resolved in exactly one place** (archreview R3-04, closing 01/S-12 = 03/C5 and
|
||||
01/C-9): `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`), with one precedence —
|
||||
`Cluster:Roles` when non-empty, else `OTOPCUA_ROLES`, else empty — and two-layer normalization: a
|
||||
plain role (`admin`/`driver`/`dev`) is trimmed and lower-cased, but a cluster role's ClusterId suffix
|
||||
(`cluster-SITE-A`) is kept verbatim, because lower-casing it would rename the Akka role of a running
|
||||
deployment. An unknown role from **either** source fails host start. `Program.cs` resolves one
|
||||
`EffectiveRoles` instance at the top of the composition root and registers it as a singleton; that
|
||||
SAME instance drives both the conditional host wiring (`hasAdmin`/`hasDriver`) and
|
||||
`AkkaClusterOptions.Roles` (what Akka gossips) — there is no second derivation for the two to
|
||||
disagree about, which is why no runtime "do the views agree" assert exists. Before this, `Program.cs`
|
||||
read `OTOPCUA_ROLES` only, so a node configured solely through `Cluster:Roles` gossiped the right
|
||||
Akka roles while `Program.cs`'s own wiring mounted none of them — the inverse of the pre-#532 field
|
||||
incident (`docs/deployments/wonder-app-vd03-makino-z-34184.md`).
|
||||
|
||||
- **A role overlay may not itself set `Cluster:Roles`.** Roles are resolved *before* the per-role
|
||||
appsettings overlay loads (the overlay's filename depends on the answer), so a later layer changing
|
||||
`Cluster:Roles` would leave the bound options disagreeing with the registered instance.
|
||||
`AkkaClusterOptionsValidator` fails host start if this ever happens, naming both views — a no-op
|
||||
today since none of the four shipped overlays carries a `Cluster` section.
|
||||
- **Both sources set to different normalized role sets logs a Warning** at boot (`Cluster:Roles`
|
||||
still wins outright) — the boot banner also names which source won (`source=Configured` /
|
||||
`Environment` / `None`).
|
||||
- **`RoleParser.GossipedRolesHave`** (case-insensitive) is defence in depth at the handful of
|
||||
production sites that read LIVE GOSSIP directly (`Member.Roles`) rather than through
|
||||
`EffectiveRoles` — a peer running an older, pre-R3-04 build can still gossip a miscased plain role
|
||||
during a rolling upgrade. ⚠️ **Known residual, not closed by R3-04:**
|
||||
`RedundancyStateActor.SelectOldestUpMemberOfRole`/`SelectDriverPrimary` delegate to
|
||||
`ZB.MOM.WW.Health.Akka`'s `ClusterActiveNode.OldestUpMember`, which matches via Akka's OWN ordinal
|
||||
`Member.HasRole` — not this repo's to patch, the same class of "not ours to change" carve-out as
|
||||
`ClusterSingletonOptions.Role`. A pair where every member is on a fixed build can no longer gossip
|
||||
a miscased role at all (normalization happens before gossip), so this only bites a genuinely mixed
|
||||
fixed/pre-fix rolling-upgrade window.
|
||||
- **`RoleAuthorityGuardTests`** (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`) is the enforcement:
|
||||
a source scan failing on any production file outside `EffectiveRoles.cs`/`RoleParser.cs` that reads
|
||||
`OTOPCUA_ROLES`, `Cluster:Roles` as code, or declares a bare `"admin"`/`"driver"`/`"dev"` literal —
|
||||
with a small, reasoned allow-list for the handful of genuinely unrelated domain uses of those same
|
||||
English words (a raw-tree node-kind label, a Sql driver's own JSON marker, an LDAP-derived AdminUI
|
||||
RBAC role).
|
||||
|
||||
See `docs/Configuration.md` §`Roles`, `docs/ServiceHosting.md` §"Role gating", and
|
||||
`archreview/plans/R3-04-one-role-authority-plan.md`.
|
||||
|
||||
## Mesh command transport (`MeshTransport`)
|
||||
|
||||
**The Akka `ClusterClient` transport described in this section's earlier history is DELETED**
|
||||
|
||||
@@ -77,7 +77,7 @@
|
||||
{
|
||||
"id": "T10",
|
||||
"subject": "Docs sweep: Configuration.md roles section (rewrite the two-knobs note), ServiceHosting.md, Redundancy.md, v2/Cluster.md, install script comment, CLAUDE.md",
|
||||
"status": "pending",
|
||||
"status": "completed",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
|
||||
@@ -26,7 +26,7 @@ The Host (`Program.cs`) loads `appsettings.json`, then overlays a **per-role** f
|
||||
- Both roles → `appsettings.admin-driver.json` (roles joined with `-`, ordinal-sorted).
|
||||
- `appsettings.{ASPNETCORE_ENVIRONMENT}.json` (e.g. `appsettings.Development.json`) is layered on by the host builder.
|
||||
|
||||
All role overlays are **optional** — the base `appsettings.json` plus the Options-class C# defaults are enough to boot. The roles themselves come from the `OTOPCUA_ROLES` env var (see [`ServiceHosting.md`](ServiceHosting.md) and the table below).
|
||||
All role overlays are **optional** — the base `appsettings.json` plus the Options-class C# defaults are enough to boot. The roles themselves come from `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`) — `Cluster:Roles`, falling back to `OTOPCUA_ROLES` when that is empty (see [`ServiceHosting.md`](ServiceHosting.md) and the table below).
|
||||
|
||||
The checked-in `appsettings*.json` files are deliberately thin: they carry only `Serilog` and the `Security:Ldap` overlay. Everything else (`OpcUa`, `Cluster`, `ConnectionStrings`/`ConfigDb`) binds from the Options-class defaults documented below unless an operator adds the section explicitly or supplies the corresponding environment variable.
|
||||
|
||||
@@ -121,11 +121,13 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
|
||||
| `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. |
|
||||
| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. |
|
||||
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). **Per-pair since Phase 6:** each application `Cluster`'s pair is its own Akka mesh, so a node's `SeedNodes` lists only itself (first) and its own pair partner (second) — never a node from another cluster's pair. |
|
||||
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES` (see the note below). Allowed values: `admin`, `driver`, `dev`, plus a per-cluster `cluster-{ClusterId}` role (e.g. `cluster-MAIN`, `cluster-SITE-A`) — see [Per-cluster mesh split (Phase 6)](#per-cluster-mesh-split-phase-6-cluster-clusterid-roles--splittopologytransportvalidator) below. |
|
||||
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES` (see the note below). Values are trimmed and (for a plain role) lower-cased; a cluster role's ClusterId suffix keeps the operator's spelling. An unknown role from either source **fails host start**. Allowed values: `admin`, `driver`, `dev`, plus a per-cluster `cluster-{ClusterId}` role (e.g. `cluster-MAIN`, `cluster-SITE-A`) — see [Per-cluster mesh split (Phase 6)](#per-cluster-mesh-split-phase-6-cluster-clusterid-roles--splittopologytransportvalidator) below. |
|
||||
|
||||
> **`Cluster:Roles` and `OTOPCUA_ROLES` are two knobs for one thing, and only one of them is the Akka role list.** `OTOPCUA_ROLES` drives all of `Program.cs`'s conditional wiring (which subsystems mount); `Cluster:Roles` becomes the node's **gossiped Akka member roles**. Since Gitea #532 `AddOtOpcUaCluster` applies the documented fallback — an empty `Cluster:Roles` takes the env var — so setting only `OTOPCUA_ROLES` now works. Before that fix it did not, silently: the node joined the mesh carrying **no roles at all**, which reads as healthy locally and as *"no driver-role nodes are Up"* at central, and every deployment to it timed out ([`deployments/wonder-app-vd03-makino-z-34184.md`](deployments/wonder-app-vd03-makino-z-34184.md)).
|
||||
> **One role authority, one precedence, both consumers (archreview R3-04).** `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`) is the SINGLE place a node's roles are decided: `Cluster:Roles` wins outright when non-empty, else `OTOPCUA_ROLES`, else empty. Both `Program.cs`'s conditional host wiring (which subsystems mount) AND the Akka `AkkaClusterOptions.Roles` gossiped to the cluster are the SAME `EffectiveRoles` instance's `Gossiped` list — not two separate derivations that happen to usually agree. The boot banner (`OtOpcUa.Host starting with roles=[...] source=...`) names which source won. Setting both sources to *different* normalized role sets logs a Warning at boot (still resolves from `Cluster:Roles`) — almost certainly not what an operator intended.
|
||||
>
|
||||
> Setting `Cluster:Roles` explicitly is still the recommendation, because the two sources normalize differently: `Cluster:Roles` binds **verbatim** while the env path is **lower-cased** by `RoleParser.Parse`. Akka compares role names ordinally, so a redundant pair configured through *different* sources with a mixed-case `cluster-{ClusterId}` (e.g. `cluster-SITE-A` vs `cluster-site-a`) would gossip two different role names and elect two singletons. Configure both nodes of a pair the same way.
|
||||
> Before R3-04, `Program.cs` read `OTOPCUA_ROLES` only and never looked at `Cluster:Roles` at all — a node configured solely through `Cluster:Roles` (the shape Gitea #532's fix made viable on the Akka side) joined the mesh gossiping the right roles while `Program.cs`'s own conditional wiring mounted NONE of them, silently. See [`deployments/wonder-app-vd03-makino-z-34184.md`](deployments/wonder-app-vd03-makino-z-34184.md) for the inverse (env-only, pre-#532) incident this family of defect belongs to.
|
||||
>
|
||||
> **A role overlay may not itself set `Cluster:Roles`.** `Program.cs` resolves `EffectiveRoles` off the *pre-overlay* configuration — it needs the answer to pick the overlay's filename — so a role overlay (or a `${secret:}` expansion) that changed `Cluster:Roles` would leave the bound options disagreeing with every consumer holding the registered instance. `AkkaClusterOptionsValidator` fails host start if this ever happens, naming both views.
|
||||
|
||||
> The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.
|
||||
|
||||
@@ -446,7 +448,7 @@ All names are read in this repo's source via `Environment.GetEnvironmentVariable
|
||||
|
||||
| Variable | Read by | Effect / default |
|
||||
|---|---|---|
|
||||
| `OTOPCUA_ROLES` | `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (`RoleParser.Parse`) and `AddOtOpcUaCluster`'s `AkkaClusterOptions` post-configure (`RoleParser.ResolveEffectiveRoles`) | Comma-separated cluster roles for the node (`admin`, `driver`, `dev`, `cluster-{ClusterId}`). Drives the conditional wiring and the per-role appsettings overlay, and — when `Cluster:Roles` is empty — the node's Akka member roles too. See the note under `Cluster:Roles` for why the two sources are not interchangeable. |
|
||||
| `OTOPCUA_ROLES` | `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`) — the single role authority both `Program.cs` and `AddOtOpcUaCluster` consume | Comma-separated cluster roles for the node (`admin`, `driver`, `dev`, `cluster-{ClusterId}`), read as the **fallback** only when `Cluster:Roles` is empty. Drives the conditional wiring, the per-role appsettings overlay, and the node's Akka member roles — all from the same resolved instance. See the note under `Cluster:Roles` above. |
|
||||
| `OTOPCUA_CONFIG_CONNECTION` | `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/DesignTimeDbContextFactory.cs` (design-time / `dotnet ef` only) | Read at **design time** by `DesignTimeDbContextFactory.cs` for `dotnet ef` migrations. At **runtime** the server resolves the connection string from `ConnectionStrings:ConfigDb` (env form: `ConnectionStrings__ConfigDb`) via `configuration.GetConnectionString("ConfigDb")` in `ServiceCollectionExtensions.cs` — `OTOPCUA_CONFIG_CONNECTION` appears there only as a hint in an error message, not via `GetEnvironmentVariable`. No credential is embedded in source. |
|
||||
| `ASPNETCORE_ENVIRONMENT` | ASP.NET host builder (framework) | Selects `appsettings.{Environment}.json` (e.g. `Development`). |
|
||||
|
||||
|
||||
+6
-1
@@ -208,7 +208,9 @@ The admin singleton is the cluster's only `RedundancyStateActor`. If the admin l
|
||||
|
||||
## Configuration
|
||||
|
||||
Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var:
|
||||
Per-node identity comes from `appsettings.json` + one role source — `Cluster:Roles` (below) or the
|
||||
`OTOPCUA_ROLES` env var; either works, and if BOTH are set on a node they must agree (see
|
||||
[`Configuration.md`](Configuration.md) → `EffectiveRoles`, archreview R3-04):
|
||||
|
||||
```jsonc
|
||||
{
|
||||
@@ -227,6 +229,9 @@ Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var:
|
||||
}
|
||||
```
|
||||
|
||||
the `Cluster__Roles__0`/`Cluster__Roles__1` env-var equivalent, or (the fallback, only when
|
||||
`Cluster:Roles` is absent/empty):
|
||||
|
||||
```
|
||||
OTOPCUA_ROLES=admin,driver
|
||||
```
|
||||
|
||||
@@ -7,7 +7,7 @@ A production OtOpcUa deployment runs **one binary per node**. The historian back
|
||||
|
||||
| Process | Project | Runtime | Platform | Responsibility |
|
||||
|---|---|---|---|---|
|
||||
| **OtOpcUa Host** | `src/Server/ZB.MOM.WW.OtOpcUa.Host` | .NET 10 | AnyCPU | Single fused binary. `OTOPCUA_ROLES` env decides what to mount: `admin` (Blazor + auth + control-plane singletons), `driver` (OPC UA endpoint + per-driver actors), or both. |
|
||||
| **OtOpcUa Host** | `src/Server/ZB.MOM.WW.OtOpcUa.Host` | .NET 10 | AnyCPU | Single fused binary. The resolved [`EffectiveRoles`](#role-gating) decides what to mount: `admin` (Blazor + auth + control-plane singletons), `driver` (OPC UA endpoint + per-driver actors), or both. |
|
||||
| **ZB.MOM.WW.HistorianGateway** *(external — separate deployment)* | not in this repo | .NET 10 | — | The sole historian backend. OtOpcUa talks gRPC to it (via the `ZB.MOM.WW.HistorianGateway.Client` package) for HistoryRead, alarm `SendEvent`, and continuous `WriteLiveValues`. Must run `RuntimeDb:Enabled=true` + `RuntimeDb:EventReadsEnabled=true`; the API key must carry `historian:read` + `historian:write` + `historian:tags:write`. |
|
||||
|
||||
Galaxy access still uses the separately-installed **mxaccessgw** sidecar (see `docs/v2/Galaxy.ParityRig.md`); the gateway owns the MXAccess COM bitness constraint (its worker is x86 net48). Nothing in the OtOpcUa repo carries that constraint anymore. (The bespoke Wonderware historian sidecar this deployment used to ship was retired — see [drivers/Historian.Wonderware.md](drivers/Historian.Wonderware.md).)
|
||||
@@ -16,7 +16,11 @@ Galaxy access still uses the separately-installed **mxaccessgw** sidecar (see `d
|
||||
|
||||
## Role gating
|
||||
|
||||
`Program.cs` reads `OTOPCUA_ROLES`, parses it with `RoleParser`, and conditionally registers services:
|
||||
`Program.cs` resolves `EffectiveRoles.Resolve(builder.Configuration)` (`ZB.MOM.WW.OtOpcUa.Cluster`) —
|
||||
the single role authority (archreview R3-04): `Cluster:Roles` when non-empty, else `OTOPCUA_ROLES`,
|
||||
else neither role. The SAME resolved instance drives both the conditional wiring below AND the Akka
|
||||
gossiped roles (`AkkaClusterOptions.Roles`), so the two cannot disagree — and conditionally registers
|
||||
services:
|
||||
|
||||
| Role present | Wires |
|
||||
|---|---|
|
||||
@@ -24,11 +28,11 @@ Galaxy access still uses the separately-installed **mxaccessgw** sidecar (see `d
|
||||
| `driver` | `WithOtOpcUaRuntimeActors` (DriverHostActor + DbHealthProbeActor) — and the OPC UA endpoint on port 4840 |
|
||||
| Either / both | `AddOtOpcUaConfigDb`, `AddOtOpcUaCluster`, `AddOtOpcUaHealth` (`/health/ready`, `/health/active`, `/healthz`) |
|
||||
|
||||
Single-node dev: `OTOPCUA_ROLES=admin,driver`. Production: typically two admin nodes (HA pair) + N driver nodes.
|
||||
Single-node dev: `OTOPCUA_ROLES=admin,driver` (or the `Cluster:Roles` equivalent). Production: typically two admin nodes (HA pair) + N driver nodes.
|
||||
|
||||
### Per-role configuration overlays
|
||||
|
||||
`Program.cs:33-35` builds a role suffix by joining the parsed roles **alphabetically** with `-` and loads `appsettings.{roleSuffix}.json` as an optional overlay on top of base `appsettings.json`. Three overlays ship in `src/Server/ZB.MOM.WW.OtOpcUa.Host/`:
|
||||
`Program.cs` builds a role suffix by joining the resolved `EffectiveRoles.Gossiped` roles **alphabetically** with `-` and loads `appsettings.{roleSuffix}.json` as an optional overlay on top of base `appsettings.json`. Three overlays ship in `src/Server/ZB.MOM.WW.OtOpcUa.Host/`:
|
||||
|
||||
- `appsettings.admin.json` — admin-only nodes
|
||||
- `appsettings.driver.json` — driver-only nodes
|
||||
|
||||
+2
-2
@@ -65,9 +65,9 @@ The Cluster.Tests project verifies these key values stay correct (`HoconLoaderTe
|
||||
- `Port`: TCP port for cluster gossip. Default 4053.
|
||||
- `PublicHostname`: address advertised in cluster gossip. Must be reachable by every other node.
|
||||
- `SeedNodes`: where new nodes go to join. List one (or two) stable nodes. **Order is load-bearing:** Akka only lets `seed-nodes[0]` form a *new* cluster (`FirstSeedNodeProcess`); every other node retries `InitJoin` forever. A node that is one of its own seeds must therefore list ITSELF first, or it can never cold-start while its peer is down — enforced at boot by `AkkaClusterOptionsValidator`, explained in [Redundancy.md § Bootstrap: self-first seed ordering](../Redundancy.md#bootstrap-self-first-seed-ordering). Nodes seeded only by someone else (driver-only site nodes) are exempt.
|
||||
- `Roles`: free-form tags Akka gossip propagates. v2 uses `admin` + `driver`; per-role wiring in `Program.cs` reads `OTOPCUA_ROLES` env var, not this list — these two should stay in sync.
|
||||
- `Roles`: free-form tags Akka gossip propagates. v2 uses `admin` + `driver`. Since archreview R3-04, this list and `Program.cs`'s per-role wiring are the SAME resolved `EffectiveRoles` instance (`Cluster:Roles` here, falling back to `OTOPCUA_ROLES` when empty) — not two independent reads that need to be kept in sync by hand. See [Configuration.md § `Roles`](../Configuration.md#appsettings-sections).
|
||||
|
||||
Per-role overlay files (`appsettings.admin.json`, `appsettings.driver.json`, `appsettings.admin-driver.json`) layer on top of base `appsettings.json` based on the parsed `OTOPCUA_ROLES` (alphabetical, joined by `-`). See [ServiceHosting.md § Per-role configuration overlays](../ServiceHosting.md#per-role-configuration-overlays).
|
||||
Per-role overlay files (`appsettings.admin.json`, `appsettings.driver.json`, `appsettings.admin-driver.json`) layer on top of base `appsettings.json` based on the resolved `EffectiveRoles` (alphabetical, joined by `-`). See [ServiceHosting.md § Per-role configuration overlays](../ServiceHosting.md#per-role-configuration-overlays).
|
||||
|
||||
## IClusterRoleInfo
|
||||
|
||||
|
||||
@@ -6,7 +6,10 @@
|
||||
|
||||
.DESCRIPTION
|
||||
v2 consolidates the legacy OtOpcUa + OtOpcUaAdmin services into a single role-gated
|
||||
OtOpcUaHost binary. The -Roles parameter sets the OTOPCUA_ROLES service env so
|
||||
OtOpcUaHost binary. The -Roles parameter sets the OTOPCUA_ROLES service env — the
|
||||
documented FALLBACK role source (archreview R3-04's EffectiveRoles.Resolve prefers
|
||||
Cluster:Roles when it is set, else OTOPCUA_ROLES; this script only ever writes the
|
||||
latter, which is still every non-hand-patched production node's only role knob) — so
|
||||
Program.cs decides what to mount (admin / driver / both). The Wonderware historian
|
||||
sidecar is installed with -InstallWonderwareHistorian; it listens on a TCP port
|
||||
(default 32569, configurable via -HistorianTcpPort) protected by the shared secret.
|
||||
@@ -28,7 +31,9 @@
|
||||
- "admin,driver" — single-node dev or all-in-one production node
|
||||
- "admin" — admin-only HA pair member (Blazor + control-plane singletons)
|
||||
- "driver" — driver-only node (OPC UA endpoint + per-node actors)
|
||||
Written to the service env as OTOPCUA_ROLES.
|
||||
Written to the service env as OTOPCUA_ROLES — the fallback role source. If this node
|
||||
also carries a Cluster:Roles registry value (e.g. a hand-applied field fix), that
|
||||
wins outright; see docs/Configuration.md's EffectiveRoles note (archreview R3-04).
|
||||
|
||||
.PARAMETER HttpPort
|
||||
HTTP port for the AdminUI + auth endpoints. Default 9000. Written as ASPNETCORE_URLS.
|
||||
|
||||
Reference in New Issue
Block a user