diff --git a/CLAUDE.md b/CLAUDE.md index 307053f1..3a2f7ece 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -355,6 +355,40 @@ See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-loca cache), `docs/AlarmHistorian.md`, and the runbook `docs/operations/2026-07-20-localdb-pair-replication.md`. +## Phase 4 — cut the driver-side ConfigDb connection + +Per-cluster mesh **Phase 4** finished the job Phase 3 started: a **driver-only** node (`Cluster:Roles` +has `driver`, not `admin`) now boots with **no `ConfigDb` connection string at all**. `Program.cs` +gates `AddOtOpcUaConfigDb` (and its health check) on `hasAdmin`; a fused `admin,driver` node is +unaffected (it keeps ConfigDb via its admin role). Consequences: + +- **`ConfigSource:Mode` must be `FetchAndCache` on a driver-only node** — `ConfigSourceOptionsValidator` + fails host start if such a node is left on `Direct` (there is nothing to read Direct from). A fused + node may stay `Direct`. +- **ServiceLevel semantics change — client-visible.** With no ConfigDb, `DbHealthProbeActor` is not + spawned (`ServiceCollectionExtensions.WithOtOpcUaRuntimeActors` skips it when the resolved db + factory is null), and `OpcUaPublishActor` runs `dbLess`: `DbReachable` is fed constant `true` and + `Stale` comes from the redundancy-snapshot age alone. A healthy driver-only node therefore publishes + **240** (or **250** as driver Primary) even with central SQL unreachable — the survive-alone posture + — where a `Direct`/DB-backed node in the same state drops to 100/0. `ServiceLevelCalculator` itself + is unchanged; only the inputs a DB-less node feeds it changed. See `docs/Redundancy.md` §"DB-less + (driver-only) nodes". +- **Scripted-alarm Part 9 condition state moved to LocalDb.** `LocalDbAlarmConditionStateStore` + (replicated `alarm_condition_state` table, pair-local like the Phase-2 alarm S&F buffer) replaces + `EfAlarmConditionStateStore` on every driver-role node, fused central included. The DB-backed + `ScriptedAlarmState` table is now unused (dropping it is a deferred follow-up, tracked as Task 9). +- **Central persists deploy acks.** `DriverHostActor` no longer writes `NodeDeploymentState` to SQL on + a driver-only node — `ConfigPublishCoordinator.PersistNodeAck` (already fed by every ApplyAck) is + the system of record. +- **LDAP group→role DB grants don't reach driver-only nodes.** They register the empty + `NullLdapGroupRoleMappingService` (no ConfigDb to read `LdapGroupRoleMapping` from), so + `OtOpcUaGroupRoleMapper` falls back to the `Security:Ldap:GroupToRole` appsettings baseline only — + the same rows a driver node never received via config either, so this is not a regression. + +Code-complete on `feat/mesh-phase4`; the table-drop (Task 9) and the live gate (Task 10) are still +open. See `docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md` and +`docs/plans/2026-07-22-per-cluster-mesh-program.md` §Phase 4. + ## LDAP Authentication The server uses LDAP-based user authentication via the `Security:Ldap` section in `appsettings.json`. When enabled, credentials are validated by LDAP bind against a GLAuth server, and LDAP group membership maps to OPC UA permissions: `ReadOnly` (browse/read), `WriteOperate` (write FreeAccess/Operate attributes), `WriteTune` (write Tune attributes), `WriteConfigure` (write Configure attributes), `AlarmAck` (alarm acknowledgment). `LdapOpcUaUserAuthenticator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`) implements `IOpcUaUserAuthenticator`, delegating the LDAP bind + group lookup to `OtOpcUaLdapAuthService` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs`, an `ILdapAuthService`). See `docs/security.md` for the full guide. diff --git a/docs/Configuration.md b/docs/Configuration.md index 4fd14d89..734867d8 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -162,9 +162,24 @@ On the docker-dev rig every node carries these keys already; flip the whole rig On the docker-dev rig the central pair carries `ConfigServe` (port `4055`, dev key) and stays `Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`. Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up` — central keeps SQL and models the target topology. +**`FetchAndCache` is mandatory on a driver-only node — per-cluster mesh Phase 4.** `Program.cs` now +registers `AddOtOpcUaConfigDb` only when the node carries the `admin` role, so a **driver-only** +node (`Cluster:Roles` has `driver`, not `admin`) holds **no `ConfigDb` connection string at all** — +there is nothing for `Direct` to read. `ConfigSourceOptionsValidator` enforces this at +`ValidateOnStart`: a driver-only node whose `ConfigSource:Mode` is `Direct` **fails host start** +with an error naming the fix (`ConfigSource:Mode=FetchAndCache`). A **fused** `admin,driver` node +still owns its ConfigDb via the admin role and may stay `Direct` — the validator only fires on +`driver` without `admin`. On a driver-only node the deployed configuration, the scripted-alarm Part +9 condition state (`LocalDbAlarmConditionStateStore`, replacing the ConfigDb-backed +`EfAlarmConditionStateStore`), and the alarm store-and-forward buffer all live in the node's +LocalDb; central remains the system of record for deploy acks (`ConfigPublishCoordinator. +PersistNodeAck`) and LDAP group→role DB grants (a driver-only node maps LDAP groups to roles from +the `Security:Ldap:GroupToRole` appsettings baseline only — see `docs/Redundancy.md` and +`CLAUDE.md` §"Phase 4" for the full picture, including the client-visible ServiceLevel change). + ### `ConnectionStrings` → `ConfigDb` -- **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally. +- **Purpose:** the central Config DB connection string. **Required on admin-role nodes only** — `Program.cs` calls `AddOtOpcUaConfigDb` when `hasAdmin` (per-cluster mesh Phase 4). A driver-only node holds no `ConfigDb` connection string; it fetches its configuration via `ConfigSource:Mode=FetchAndCache` instead (see above). A fused `admin,driver` node still requires it. - **Bound by:** `AddOtOpcUaConfigDb(config)` (`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/ServiceCollectionExtensions.cs`). The connection-string name constant is `ConnectionStringName = "ConfigDb"`, read via `configuration.GetConnectionString("ConfigDb")`. If absent, startup throws with a message pointing to either `appsettings.json` or the `OTOPCUA_CONFIG_CONNECTION` env var. - **Shape:** standard `ConnectionStrings:ConfigDb` SQL Server connection string. There is no checked-in default in the thin `appsettings*.json` — supply it per environment. diff --git a/docs/Redundancy.md b/docs/Redundancy.md index cacc4bd7..2733b966 100644 --- a/docs/Redundancy.md +++ b/docs/Redundancy.md @@ -16,7 +16,7 @@ The runtime pieces live in: | `OpcUaPublishActor` | `OtOpcUa.Runtime.OpcUa` | Per-driver-node; subscribes to the `redundancy-state` topic, computes a health-aware ServiceLevel byte via `ServiceLevelCalculator` (see below), and forwards it to `IServiceLevelPublisher`. | | `IServiceLevelPublisher` / `SdkServiceLevelPublisher` | `OtOpcUa.Commons.OpcUa` / `OtOpcUa.OpcUaServer` | Writes the byte into the SDK's `Server.ServiceLevel` Variable. Production binds `DeferredServiceLevelPublisher`, which swaps in the real `SdkServiceLevelPublisher` once the SDK is up (it needs `IServerInternal`, available only after `StandardServer.Start`); until then writes route through `NullServiceLevelPublisher`. | | `ServiceLevelCalculator` | `OtOpcUa.Cluster.Redundancy` (`Core.Cluster`) | Pure function `(NodeHealthInputs) → byte` — the DB/probe-aware tiering (see truth table below). Covered by `ServiceLevelCalculatorTests`. **Now the live publish path** — `OpcUaPublishActor` calls it on every `HealthTick` and `RedundancyStateChanged` event. Moved to `Core.Cluster` so Runtime can reach it without a Runtime→ControlPlane reference. | -| `DbHealthProbeActor` | `OtOpcUa.Runtime.Health` | Per-node; runs `SELECT 1` against ConfigDb every 5s. Read by the health endpoint AND by `OpcUaPublishActor` (the `DbReachable` ServiceLevel input). | +| `DbHealthProbeActor` | `OtOpcUa.Runtime.Health` | Runs `SELECT 1` against ConfigDb every 5s. Read by the health endpoint AND by `OpcUaPublishActor` (the `DbReachable` ServiceLevel input). **Spawned only when a ConfigDb is present** (admin-role or fused admin+driver nodes) — a driver-only node has no `IDbContextFactory` to probe, so this actor is not spawned there at all; see "DB-less nodes" below. | | `PeerProbeSupervisor` | `OtOpcUa.Runtime.Health` | Per-node; subscribes to the `redundancy-state` topic and maintains one `PeerOpcUaProbeActor` child per OTHER driver-role peer (spawn on join, stop on departure), so every node is continuously probed by its peers. | | `PeerOpcUaProbeActor` | `OtOpcUa.Runtime.Health` | Spawned by `PeerProbeSupervisor`; pings a peer `opc.tcp://peer:4840` with a TCP connect (2s timeout) and publishes `OpcUaProbeResult` on the `redundancy-state` topic. A full secure-channel Hello handshake is a possible future upgrade; the TCP connect is the current real probe. | | `ClusterRoleInfo` | `OtOpcUa.Cluster` | Live view of cluster membership + role-leader; exposes `IClusterRoleInfo` to the rest of the host. (Akka's role-leader is *not* what elects the redundancy Primary — see below.) | @@ -56,6 +56,33 @@ The resulting truth table (all tiers are now reachable at runtime): > by the +10 bonus. Clients with the standard "pick highest ServiceLevel" > heuristic continue to prefer the primary. +#### DB-less (driver-only) nodes — per-cluster mesh Phase 4 + +**Client-visible change.** Per-cluster mesh Phase 4 cut the ConfigDb connection off driver-only +nodes (Akka role `driver`, not `admin`) — `Program.cs` registers `AddOtOpcUaConfigDb` only when +`hasAdmin`. With no ConfigDb to probe, `DbHealthProbeActor` is **not spawned** on such a node +(`ServiceCollectionExtensions.WithOtOpcUaRuntimeActors` skips it whenever the resolved +`IDbContextFactory` is null), and `OpcUaPublishActor` runs in a `dbLess` +mode instead of the health-aware path above: + +- `DbReachable` is fed **constant `true`** — LocalDb is the config store and is in-process, so it + can never be "unreachable" the way a remote SQL Server can. +- `Stale` is derived from the `redundancy-state` snapshot age alone (`(now − snapshotEntry.AsOfUtc) + > 30 s`) — there is no DB-health term to fold in. +- `OpcUaProbeOk` and `IsDriverPrimary` are unchanged (peer OPC UA probe, oldest-Up-member election). + +**Consequence:** a healthy driver-only node publishes **240** as a follower or **250** as the +driver Primary **even when central SQL is unreachable** — the survive-alone posture the fetch-and- +cache design (Phase 3) exists to support. Contrast this with a `Direct`/DB-backed node (fused +admin+driver, or any node still reading ConfigDb directly): an unreachable ConfigDb there still +drives `DbReachable=false`, which forces `Stale=true` in the derivation above and the node drops to +**100** (or **0** once the sample itself goes stale) via the health-aware tiering, not 240/250. +Same byte, opposite meaning depending on which kind of node is publishing it — a client or operator +reading ServiceLevel in isolation cannot tell "central is down and I don't care" (driver-only, 240) +from "central is down and I'm degraded" (DB-backed, 100/0) without also knowing the node's role +shape. `ServiceLevelCalculator.Compute` itself is unchanged; only the inputs a DB-less node feeds +it changed — see `OpcUaPublishActor.RecomputeServiceLevel`'s `_dbLess` branch. + #### Backward-compatible fallback (legacy seam) A node with no `DbHealthStatus` wired (e.g. early bootstrap window before the diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md b/docs/plans/2026-07-22-per-cluster-mesh-program.md index ebc5e524..4bdc9222 100644 --- a/docs/plans/2026-07-22-per-cluster-mesh-program.md +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md @@ -141,7 +141,7 @@ admin-change refresh), clusters know central from appsettings (static, restart t flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly against a stopped node. -### Phase 3 — Config fetch-and-cache from central — **CODE-COMPLETE 2026-07-23 (Tasks 0–8; live gate Task 10 pending)** +### Phase 3 — Config fetch-and-cache from central — **DONE 2026-07-23 (merged `d01b0695`, live gate PASSED, pushed to origin)** **Decisions taken:** the deploy artifact is served **by central over a gRPC fetch RPC** (`DeploymentArtifactService.Fetch`, streamed chunks — chosen over token-gated HTTP), authenticated by a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`, @@ -160,13 +160,14 @@ config SQL read); #485 coverage; and a **real two-Host gRPC boundary test** (rig wrong-key control returns null, failover). Chunking, SHA-256 verify, newest-2 retention, pair replication carry over unchanged. **Phase boundary held:** config **reads** only — the `NodeDeploymentState` ack-row write, `DbHealthProbe`, `EfAlarmConditionStateStore` stay for Phase 4. -**Remaining (Tasks 9–10):** rig config + docs (done) and the live gate — deploy on a site pair flipped -to `FetchAndCache` with central SQL stopped mid-fetch → retry lands; #485 last-known-good re-proven on -the new path. -**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable central -boots into the current config; with central down it boots last-known-good. +**Live gate PASSED:** deployed on a site pair flipped to `FetchAndCache` with central SQL stopped +mid-fetch — retry landed; #485 last-known-good re-proven on the new path. Merged to master +`d01b0695` and pushed to origin; the scadaproj umbrella index was updated + pushed (`b5e7bc8`) in the +same pass. +**Exit gate — MET:** live gate green on the rig; a driver node with an empty LocalDb and reachable +central boots into the current config; with central down it boots last-known-good. -### Phase 4 — Cut the driver-side ConfigDb connection +### Phase 4 — Cut the driver-side ConfigDb connection — **CODE-COMPLETE 2026-07-23 (Tasks 0–7 + 1b; Task 9 table-drop deferred; Task 10 rig + Task 11 live gate remain)** **Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor` question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering, @@ -176,6 +177,27 @@ transports) — this is a client-visible ServiceLevel semantics change and must design) and re-source it; registration cleanup in `ServiceCollectionExtensions`; driver-role `Program.cs` branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only `AddConfigurationDatabase`). +**Shipped (branch `feat/mesh-phase4`, Tasks 0–7 + 1b):** `Program.cs` gates `AddOtOpcUaConfigDb` + +the ConfigDb health check on `hasAdmin`, so a driver-only node registers no ConfigDb connection +string at all; `ConfigSourceOptionsValidator` fails host start if a driver-only node's +`ConfigSource:Mode` is left `Direct` (fused `admin,driver` may stay `Direct`); the DB-health +question resolved as **retire, don't replace** — `DbHealthProbeActor` is simply not spawned when no +ConfigDb is present, and `OpcUaPublishActor` runs a `dbLess` mode feeding `DbReachable=true` +constant with staleness from the redundancy-snapshot age alone (documented as the client-visible +change in `docs/Redundancy.md`); `LocalDbAlarmConditionStateStore` (replicated +`alarm_condition_state` table) replaces `EfAlarmConditionStateStore` on every driver-role node; +`DriverHostActor` no longer writes `NodeDeploymentState` on a driver-only node (central's +`ConfigPublishCoordinator.PersistNodeAck` was already the ack system of record); driver-only nodes +register `NullLdapGroupRoleMappingService` so LDAP group→role falls back to the +`Security:Ldap:GroupToRole` appsettings baseline (DB grants were never delivered to driver nodes via +config anyway). Docs updated: `docs/Redundancy.md`, `docs/Configuration.md`, `CLAUDE.md`. +**Deferred:** Task 9 — dropping the now-unused `ScriptedAlarmState` ConfigDb table (a follow-up, not +a blocker; the table is simply unread). +**Remaining (Tasks 10–11):** rig config for a driver-only node with no `ConfigDb` connection string, +and the live gate itself — a driver pair runs a full deploy + alarm + historian cycle with no +ConfigDb configured at all, confirming ServiceLevel 240/250 while central SQL is down and +grep-level proof no driver-branch service can resolve the ConfigDb context. **Not yet run — treat +Phase 4 as code-complete, not verified, until this lands.** **Exit gate:** its own live gate — a driver pair runs a full deploy + alarm + historian cycle with **no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can resolve the ConfigDb context. @@ -243,8 +265,8 @@ resource sizing on the site VMs should be checked once both products run the ful | Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. | | 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. | | 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. | -| 3 fetch-and-cache | **CODE-COMPLETE 2026-07-23** (Tasks 0–8 merged to the phase branch, all green + sabotage-checked; live gate Task 10 pending). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. | -| 4 cut driver ConfigDb | not started | +| 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. | +| 4 cut driver ConfigDb | **CODE-COMPLETE 2026-07-23** on `feat/mesh-phase4` (Tasks 0–7 + 1b done; Task 9 table-drop deferred; Task 10 rig + Task 11 live gate remain — not yet run). ConfigDb admin-only; `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central down); alarm condition state in LocalDb; central persists acks; driver-only LDAP maps from appsettings only. See `2026-07-23-mesh-phase4-cut-driver-configdb.md`. | | 5 gRPC telemetry | not started | | 6 mesh partition + co-location | not started | | 7 drill + live gates | not started | diff --git a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json index c5e37a19..07885350 100644 --- a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json +++ b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json @@ -76,14 +76,16 @@ { "id": 7, "subject": "Task 7: Registration cleanup + full-graph driver-only DI sweep", - "status": "pending", + "status": "completed", "blockedBy": [ "1b", 2, 3, 4, 6 - ] + ], + "commit": "6f21213f", + "result": "12/12; sweep verified clean; ordering+lazy-resolve pins APPROVED (red-check confirmed load-bearing)" }, { "id": 8, @@ -96,10 +98,11 @@ { "id": 9, "subject": "Task 9: Drop dead ConfigDb ScriptedAlarmState table + retire EfAlarmConditionStateStore", - "status": "pending", + "status": "deferred", "blockedBy": [ 6 - ] + ], + "result": "DEFERRED to follow-up (plan-sanctioned): EF store is inert on all real nodes (test-harness fallback only); table-drop migration is irreversible. Not needed for the ConfigDb-cut goal. Tracked in program doc." }, { "id": 10,