docs(mesh-phase4): ServiceLevel semantics on DB-less nodes + driver ConfigDb cut
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
@@ -355,6 +355,40 @@ See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-loca
|
||||
cache), `docs/AlarmHistorian.md`, and the runbook
|
||||
`docs/operations/2026-07-20-localdb-pair-replication.md`.
|
||||
|
||||
## Phase 4 — cut the driver-side ConfigDb connection
|
||||
|
||||
Per-cluster mesh **Phase 4** finished the job Phase 3 started: a **driver-only** node (`Cluster:Roles`
|
||||
has `driver`, not `admin`) now boots with **no `ConfigDb` connection string at all**. `Program.cs`
|
||||
gates `AddOtOpcUaConfigDb` (and its health check) on `hasAdmin`; a fused `admin,driver` node is
|
||||
unaffected (it keeps ConfigDb via its admin role). Consequences:
|
||||
|
||||
- **`ConfigSource:Mode` must be `FetchAndCache` on a driver-only node** — `ConfigSourceOptionsValidator`
|
||||
fails host start if such a node is left on `Direct` (there is nothing to read Direct from). A fused
|
||||
node may stay `Direct`.
|
||||
- **ServiceLevel semantics change — client-visible.** With no ConfigDb, `DbHealthProbeActor` is not
|
||||
spawned (`ServiceCollectionExtensions.WithOtOpcUaRuntimeActors` skips it when the resolved db
|
||||
factory is null), and `OpcUaPublishActor` runs `dbLess`: `DbReachable` is fed constant `true` and
|
||||
`Stale` comes from the redundancy-snapshot age alone. A healthy driver-only node therefore publishes
|
||||
**240** (or **250** as driver Primary) even with central SQL unreachable — the survive-alone posture
|
||||
— where a `Direct`/DB-backed node in the same state drops to 100/0. `ServiceLevelCalculator` itself
|
||||
is unchanged; only the inputs a DB-less node feeds it changed. See `docs/Redundancy.md` §"DB-less
|
||||
(driver-only) nodes".
|
||||
- **Scripted-alarm Part 9 condition state moved to LocalDb.** `LocalDbAlarmConditionStateStore`
|
||||
(replicated `alarm_condition_state` table, pair-local like the Phase-2 alarm S&F buffer) replaces
|
||||
`EfAlarmConditionStateStore` on every driver-role node, fused central included. The DB-backed
|
||||
`ScriptedAlarmState` table is now unused (dropping it is a deferred follow-up, tracked as Task 9).
|
||||
- **Central persists deploy acks.** `DriverHostActor` no longer writes `NodeDeploymentState` to SQL on
|
||||
a driver-only node — `ConfigPublishCoordinator.PersistNodeAck` (already fed by every ApplyAck) is
|
||||
the system of record.
|
||||
- **LDAP group→role DB grants don't reach driver-only nodes.** They register the empty
|
||||
`NullLdapGroupRoleMappingService` (no ConfigDb to read `LdapGroupRoleMapping` from), so
|
||||
`OtOpcUaGroupRoleMapper` falls back to the `Security:Ldap:GroupToRole` appsettings baseline only —
|
||||
the same rows a driver node never received via config either, so this is not a regression.
|
||||
|
||||
Code-complete on `feat/mesh-phase4`; the table-drop (Task 9) and the live gate (Task 10) are still
|
||||
open. See `docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md` and
|
||||
`docs/plans/2026-07-22-per-cluster-mesh-program.md` §Phase 4.
|
||||
|
||||
## LDAP Authentication
|
||||
|
||||
The server uses LDAP-based user authentication via the `Security:Ldap` section in `appsettings.json`. When enabled, credentials are validated by LDAP bind against a GLAuth server, and LDAP group membership maps to OPC UA permissions: `ReadOnly` (browse/read), `WriteOperate` (write FreeAccess/Operate attributes), `WriteTune` (write Tune attributes), `WriteConfigure` (write Configure attributes), `AlarmAck` (alarm acknowledgment). `LdapOpcUaUserAuthenticator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`) implements `IOpcUaUserAuthenticator`, delegating the LDAP bind + group lookup to `OtOpcUaLdapAuthService` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs`, an `ILdapAuthService`). See `docs/security.md` for the full guide.
|
||||
|
||||
+16
-1
@@ -162,9 +162,24 @@ On the docker-dev rig every node carries these keys already; flip the whole rig
|
||||
|
||||
On the docker-dev rig the central pair carries `ConfigServe` (port `4055`, dev key) and stays `Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`. Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up` — central keeps SQL and models the target topology.
|
||||
|
||||
**`FetchAndCache` is mandatory on a driver-only node — per-cluster mesh Phase 4.** `Program.cs` now
|
||||
registers `AddOtOpcUaConfigDb` only when the node carries the `admin` role, so a **driver-only**
|
||||
node (`Cluster:Roles` has `driver`, not `admin`) holds **no `ConfigDb` connection string at all** —
|
||||
there is nothing for `Direct` to read. `ConfigSourceOptionsValidator` enforces this at
|
||||
`ValidateOnStart`: a driver-only node whose `ConfigSource:Mode` is `Direct` **fails host start**
|
||||
with an error naming the fix (`ConfigSource:Mode=FetchAndCache`). A **fused** `admin,driver` node
|
||||
still owns its ConfigDb via the admin role and may stay `Direct` — the validator only fires on
|
||||
`driver` without `admin`. On a driver-only node the deployed configuration, the scripted-alarm Part
|
||||
9 condition state (`LocalDbAlarmConditionStateStore`, replacing the ConfigDb-backed
|
||||
`EfAlarmConditionStateStore`), and the alarm store-and-forward buffer all live in the node's
|
||||
LocalDb; central remains the system of record for deploy acks (`ConfigPublishCoordinator.
|
||||
PersistNodeAck`) and LDAP group→role DB grants (a driver-only node maps LDAP groups to roles from
|
||||
the `Security:Ldap:GroupToRole` appsettings baseline only — see `docs/Redundancy.md` and
|
||||
`CLAUDE.md` §"Phase 4" for the full picture, including the client-visible ServiceLevel change).
|
||||
|
||||
### `ConnectionStrings` → `ConfigDb`
|
||||
|
||||
- **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally.
|
||||
- **Purpose:** the central Config DB connection string. **Required on admin-role nodes only** — `Program.cs` calls `AddOtOpcUaConfigDb` when `hasAdmin` (per-cluster mesh Phase 4). A driver-only node holds no `ConfigDb` connection string; it fetches its configuration via `ConfigSource:Mode=FetchAndCache` instead (see above). A fused `admin,driver` node still requires it.
|
||||
- **Bound by:** `AddOtOpcUaConfigDb(config)` (`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/ServiceCollectionExtensions.cs`). The connection-string name constant is `ConnectionStringName = "ConfigDb"`, read via `configuration.GetConnectionString("ConfigDb")`. If absent, startup throws with a message pointing to either `appsettings.json` or the `OTOPCUA_CONFIG_CONNECTION` env var.
|
||||
- **Shape:** standard `ConnectionStrings:ConfigDb` SQL Server connection string. There is no checked-in default in the thin `appsettings*.json` — supply it per environment.
|
||||
|
||||
|
||||
+28
-1
@@ -16,7 +16,7 @@ The runtime pieces live in:
|
||||
| `OpcUaPublishActor` | `OtOpcUa.Runtime.OpcUa` | Per-driver-node; subscribes to the `redundancy-state` topic, computes a health-aware ServiceLevel byte via `ServiceLevelCalculator` (see below), and forwards it to `IServiceLevelPublisher`. |
|
||||
| `IServiceLevelPublisher` / `SdkServiceLevelPublisher` | `OtOpcUa.Commons.OpcUa` / `OtOpcUa.OpcUaServer` | Writes the byte into the SDK's `Server.ServiceLevel` Variable. Production binds `DeferredServiceLevelPublisher`, which swaps in the real `SdkServiceLevelPublisher` once the SDK is up (it needs `IServerInternal`, available only after `StandardServer.Start`); until then writes route through `NullServiceLevelPublisher`. |
|
||||
| `ServiceLevelCalculator` | `OtOpcUa.Cluster.Redundancy` (`Core.Cluster`) | Pure function `(NodeHealthInputs) → byte` — the DB/probe-aware tiering (see truth table below). Covered by `ServiceLevelCalculatorTests`. **Now the live publish path** — `OpcUaPublishActor` calls it on every `HealthTick` and `RedundancyStateChanged` event. Moved to `Core.Cluster` so Runtime can reach it without a Runtime→ControlPlane reference. |
|
||||
| `DbHealthProbeActor` | `OtOpcUa.Runtime.Health` | Per-node; runs `SELECT 1` against ConfigDb every 5s. Read by the health endpoint AND by `OpcUaPublishActor` (the `DbReachable` ServiceLevel input). |
|
||||
| `DbHealthProbeActor` | `OtOpcUa.Runtime.Health` | Runs `SELECT 1` against ConfigDb every 5s. Read by the health endpoint AND by `OpcUaPublishActor` (the `DbReachable` ServiceLevel input). **Spawned only when a ConfigDb is present** (admin-role or fused admin+driver nodes) — a driver-only node has no `IDbContextFactory<OtOpcUaConfigDbContext>` to probe, so this actor is not spawned there at all; see "DB-less nodes" below. |
|
||||
| `PeerProbeSupervisor` | `OtOpcUa.Runtime.Health` | Per-node; subscribes to the `redundancy-state` topic and maintains one `PeerOpcUaProbeActor` child per OTHER driver-role peer (spawn on join, stop on departure), so every node is continuously probed by its peers. |
|
||||
| `PeerOpcUaProbeActor` | `OtOpcUa.Runtime.Health` | Spawned by `PeerProbeSupervisor`; pings a peer `opc.tcp://peer:4840` with a TCP connect (2s timeout) and publishes `OpcUaProbeResult` on the `redundancy-state` topic. A full secure-channel Hello handshake is a possible future upgrade; the TCP connect is the current real probe. |
|
||||
| `ClusterRoleInfo` | `OtOpcUa.Cluster` | Live view of cluster membership + role-leader; exposes `IClusterRoleInfo` to the rest of the host. (Akka's role-leader is *not* what elects the redundancy Primary — see below.) |
|
||||
@@ -56,6 +56,33 @@ The resulting truth table (all tiers are now reachable at runtime):
|
||||
> by the +10 bonus. Clients with the standard "pick highest ServiceLevel"
|
||||
> heuristic continue to prefer the primary.
|
||||
|
||||
#### DB-less (driver-only) nodes — per-cluster mesh Phase 4
|
||||
|
||||
**Client-visible change.** Per-cluster mesh Phase 4 cut the ConfigDb connection off driver-only
|
||||
nodes (Akka role `driver`, not `admin`) — `Program.cs` registers `AddOtOpcUaConfigDb` only when
|
||||
`hasAdmin`. With no ConfigDb to probe, `DbHealthProbeActor` is **not spawned** on such a node
|
||||
(`ServiceCollectionExtensions.WithOtOpcUaRuntimeActors` skips it whenever the resolved
|
||||
`IDbContextFactory<OtOpcUaConfigDbContext>` is null), and `OpcUaPublishActor` runs in a `dbLess`
|
||||
mode instead of the health-aware path above:
|
||||
|
||||
- `DbReachable` is fed **constant `true`** — LocalDb is the config store and is in-process, so it
|
||||
can never be "unreachable" the way a remote SQL Server can.
|
||||
- `Stale` is derived from the `redundancy-state` snapshot age alone (`(now − snapshotEntry.AsOfUtc)
|
||||
> 30 s`) — there is no DB-health term to fold in.
|
||||
- `OpcUaProbeOk` and `IsDriverPrimary` are unchanged (peer OPC UA probe, oldest-Up-member election).
|
||||
|
||||
**Consequence:** a healthy driver-only node publishes **240** as a follower or **250** as the
|
||||
driver Primary **even when central SQL is unreachable** — the survive-alone posture the fetch-and-
|
||||
cache design (Phase 3) exists to support. Contrast this with a `Direct`/DB-backed node (fused
|
||||
admin+driver, or any node still reading ConfigDb directly): an unreachable ConfigDb there still
|
||||
drives `DbReachable=false`, which forces `Stale=true` in the derivation above and the node drops to
|
||||
**100** (or **0** once the sample itself goes stale) via the health-aware tiering, not 240/250.
|
||||
Same byte, opposite meaning depending on which kind of node is publishing it — a client or operator
|
||||
reading ServiceLevel in isolation cannot tell "central is down and I don't care" (driver-only, 240)
|
||||
from "central is down and I'm degraded" (DB-backed, 100/0) without also knowing the node's role
|
||||
shape. `ServiceLevelCalculator.Compute` itself is unchanged; only the inputs a DB-less node feeds
|
||||
it changed — see `OpcUaPublishActor.RecomputeServiceLevel`'s `_dbLess` branch.
|
||||
|
||||
#### Backward-compatible fallback (legacy seam)
|
||||
|
||||
A node with no `DbHealthStatus` wired (e.g. early bootstrap window before the
|
||||
|
||||
@@ -141,7 +141,7 @@ admin-change refresh), clusters know central from appsettings (static, restart t
|
||||
flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly
|
||||
against a stopped node.
|
||||
|
||||
### Phase 3 — Config fetch-and-cache from central — **CODE-COMPLETE 2026-07-23 (Tasks 0–8; live gate Task 10 pending)**
|
||||
### Phase 3 — Config fetch-and-cache from central — **DONE 2026-07-23 (merged `d01b0695`, live gate PASSED, pushed to origin)**
|
||||
**Decisions taken:** the deploy artifact is served **by central over a gRPC fetch RPC**
|
||||
(`DeploymentArtifactService.Fetch`, streamed chunks — chosen over token-gated HTTP), authenticated by
|
||||
a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`,
|
||||
@@ -160,13 +160,14 @@ config SQL read); #485 coverage; and a **real two-Host gRPC boundary test** (rig
|
||||
wrong-key control returns null, failover). Chunking, SHA-256 verify, newest-2 retention, pair
|
||||
replication carry over unchanged. **Phase boundary held:** config **reads** only — the
|
||||
`NodeDeploymentState` ack-row write, `DbHealthProbe`, `EfAlarmConditionStateStore` stay for Phase 4.
|
||||
**Remaining (Tasks 9–10):** rig config + docs (done) and the live gate — deploy on a site pair flipped
|
||||
to `FetchAndCache` with central SQL stopped mid-fetch → retry lands; #485 last-known-good re-proven on
|
||||
the new path.
|
||||
**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable central
|
||||
boots into the current config; with central down it boots last-known-good.
|
||||
**Live gate PASSED:** deployed on a site pair flipped to `FetchAndCache` with central SQL stopped
|
||||
mid-fetch — retry landed; #485 last-known-good re-proven on the new path. Merged to master
|
||||
`d01b0695` and pushed to origin; the scadaproj umbrella index was updated + pushed (`b5e7bc8`) in the
|
||||
same pass.
|
||||
**Exit gate — MET:** live gate green on the rig; a driver node with an empty LocalDb and reachable
|
||||
central boots into the current config; with central down it boots last-known-good.
|
||||
|
||||
### Phase 4 — Cut the driver-side ConfigDb connection
|
||||
### Phase 4 — Cut the driver-side ConfigDb connection — **CODE-COMPLETE 2026-07-23 (Tasks 0–7 + 1b; Task 9 table-drop deferred; Task 10 rig + Task 11 live gate remain)**
|
||||
**Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local
|
||||
state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor`
|
||||
question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering,
|
||||
@@ -176,6 +177,27 @@ transports) — this is a client-visible ServiceLevel semantics change and must
|
||||
design) and re-source it; registration cleanup in `ServiceCollectionExtensions`; driver-role
|
||||
`Program.cs` branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only
|
||||
`AddConfigurationDatabase`).
|
||||
**Shipped (branch `feat/mesh-phase4`, Tasks 0–7 + 1b):** `Program.cs` gates `AddOtOpcUaConfigDb` +
|
||||
the ConfigDb health check on `hasAdmin`, so a driver-only node registers no ConfigDb connection
|
||||
string at all; `ConfigSourceOptionsValidator` fails host start if a driver-only node's
|
||||
`ConfigSource:Mode` is left `Direct` (fused `admin,driver` may stay `Direct`); the DB-health
|
||||
question resolved as **retire, don't replace** — `DbHealthProbeActor` is simply not spawned when no
|
||||
ConfigDb is present, and `OpcUaPublishActor` runs a `dbLess` mode feeding `DbReachable=true`
|
||||
constant with staleness from the redundancy-snapshot age alone (documented as the client-visible
|
||||
change in `docs/Redundancy.md`); `LocalDbAlarmConditionStateStore` (replicated
|
||||
`alarm_condition_state` table) replaces `EfAlarmConditionStateStore` on every driver-role node;
|
||||
`DriverHostActor` no longer writes `NodeDeploymentState` on a driver-only node (central's
|
||||
`ConfigPublishCoordinator.PersistNodeAck` was already the ack system of record); driver-only nodes
|
||||
register `NullLdapGroupRoleMappingService` so LDAP group→role falls back to the
|
||||
`Security:Ldap:GroupToRole` appsettings baseline (DB grants were never delivered to driver nodes via
|
||||
config anyway). Docs updated: `docs/Redundancy.md`, `docs/Configuration.md`, `CLAUDE.md`.
|
||||
**Deferred:** Task 9 — dropping the now-unused `ScriptedAlarmState` ConfigDb table (a follow-up, not
|
||||
a blocker; the table is simply unread).
|
||||
**Remaining (Tasks 10–11):** rig config for a driver-only node with no `ConfigDb` connection string,
|
||||
and the live gate itself — a driver pair runs a full deploy + alarm + historian cycle with no
|
||||
ConfigDb configured at all, confirming ServiceLevel 240/250 while central SQL is down and
|
||||
grep-level proof no driver-branch service can resolve the ConfigDb context. **Not yet run — treat
|
||||
Phase 4 as code-complete, not verified, until this lands.**
|
||||
**Exit gate:** its own live gate — a driver pair runs a full deploy + alarm + historian cycle with
|
||||
**no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can
|
||||
resolve the ConfigDb context.
|
||||
@@ -243,8 +265,8 @@ resource sizing on the site VMs should be checked once both products run the ful
|
||||
| Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. |
|
||||
| 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
|
||||
| 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
|
||||
| 3 fetch-and-cache | **CODE-COMPLETE 2026-07-23** (Tasks 0–8 merged to the phase branch, all green + sabotage-checked; live gate Task 10 pending). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
|
||||
| 4 cut driver ConfigDb | not started |
|
||||
| 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
|
||||
| 4 cut driver ConfigDb | **CODE-COMPLETE 2026-07-23** on `feat/mesh-phase4` (Tasks 0–7 + 1b done; Task 9 table-drop deferred; Task 10 rig + Task 11 live gate remain — not yet run). ConfigDb admin-only; `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central down); alarm condition state in LocalDb; central persists acks; driver-only LDAP maps from appsettings only. See `2026-07-23-mesh-phase4-cut-driver-configdb.md`. |
|
||||
| 5 gRPC telemetry | not started |
|
||||
| 6 mesh partition + co-location | not started |
|
||||
| 7 drill + live gates | not started |
|
||||
|
||||
@@ -76,14 +76,16 @@
|
||||
{
|
||||
"id": 7,
|
||||
"subject": "Task 7: Registration cleanup + full-graph driver-only DI sweep",
|
||||
"status": "pending",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"1b",
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
6
|
||||
]
|
||||
],
|
||||
"commit": "6f21213f",
|
||||
"result": "12/12; sweep verified clean; ordering+lazy-resolve pins APPROVED (red-check confirmed load-bearing)"
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
@@ -96,10 +98,11 @@
|
||||
{
|
||||
"id": 9,
|
||||
"subject": "Task 9: Drop dead ConfigDb ScriptedAlarmState table + retire EfAlarmConditionStateStore",
|
||||
"status": "pending",
|
||||
"status": "deferred",
|
||||
"blockedBy": [
|
||||
6
|
||||
]
|
||||
],
|
||||
"result": "DEFERRED to follow-up (plan-sanctioned): EF store is inert on all real nodes (test-harness fallback only); table-drop migration is irreversible. Not needed for the ConfigDb-cut goal. Tracked in program doc."
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
|
||||
Reference in New Issue
Block a user