From bc611969a929ebe426bfb07305094ad6a6e5850f Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 11:22:48 -0400 Subject: [PATCH 01/18] =?UTF-8?q?docs(mesh-phase4):=20plan=20=E2=80=94=20c?= =?UTF-8?q?ut=20the=20driver-side=20ConfigDb=20connection?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 4 of the per-cluster mesh program. Removes the four remaining driver-side ConfigDb consumers + the registration itself so a driver-only node boots with no ConfigDb connection string: - ConfigDb registered iff hasAdmin (Program.cs); driver-only requires none - driver-only ⇒ FetchAndCache mandatory (validator) - DbHealthProbeActor not spawned on driver-only; DbReachable=true constant (client-visible ServiceLevel change — a healthy DB-less node publishes 240/250 with central down, per user decision 2026-07-23) - DriverHostActor: nullable factory, drop redundant SQL ack-writes (central's PersistNodeAck is the ack system of record) - EfAlarmConditionStateStore → LocalDbAlarmConditionStateStore (new replicated alarm_condition_state table) Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- ...6-07-23-mesh-phase4-cut-driver-configdb.md | 472 ++++++++++++++++++ ...h-phase4-cut-driver-configdb.md.tasks.json | 18 + 2 files changed, 490 insertions(+) create mode 100644 docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md create mode 100644 docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json diff --git a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md new file mode 100644 index 00000000..01c79d57 --- /dev/null +++ b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md @@ -0,0 +1,472 @@ +# Per-Cluster Mesh Phase 4 — Cut the Driver-Side ConfigDb Connection + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. + +**Goal:** A driver-only OtOpcUa node runs a full deploy + alarm + historian cycle with **no +`ConfigDb` connection string configured at all** — LocalDb is its steady-state config + alarm-state +store, and central persists its deploy acks. Grep-level proof: no driver-branch service can resolve +`OtOpcUaConfigDbContext`. + +**Architecture:** Phase 3 already gave driver nodes their config from central (fetch-and-cache into +LocalDb) behind the `ConfigSource:Mode=FetchAndCache` dark switch. Phase 4 removes the *remaining* +four driver-side ConfigDb consumers and the registration itself, so a driver-only host boots with the +`ConfigDb` connection string absent. The fused **central** node keeps ConfigDb via its `admin` role — +this phase gates the whole registration on `hasAdmin`, never on `hasDriver`. + +**Tech stack:** .NET 10, Akka.NET, EF Core (ConfigDb, SQL Server — central only after this phase), +`ZB.MOM.WW.LocalDb` (SQLite, replicated), xUnit + Shouldly. + +--- + +## Design decisions (settled — do not relitigate) + +1. **ConfigDb is registered iff `hasAdmin`.** `Program.cs:126`'s unconditional `AddOtOpcUaConfigDb` + moves inside a `hasAdmin` guard. A driver-only node requires **no** `ConfigDb` connection string + and must not throw for its absence. Fused `admin,driver` central nodes are unaffected (admin owns + the DB; driver actors on that node may still be handed a non-null factory). +2. **Driver-only ⇒ `FetchAndCache` is mandatory.** `Direct` mode reads central SQL, which a driver-only + node no longer has. `ConfigSourceOptionsValidator` fails start for a `driver`-role, non-`admin` node + whose `Mode` is `Direct`. (A fused `admin,driver` node may stay `Direct` — it has the DB.) +3. **Db-reachability retires from ServiceLevel on driver-only nodes** (user decision 2026-07-23): + LocalDb *is* the config store and is in-process, so `DbReachable` is treated as a constant `true` + on nodes with no `DbHealthProbeActor`. Staleness already carries "running behind on config." A + healthy site node serving last-known-good with central down publishes 240/250 — the survive-alone + posture. **This is a client-visible ServiceLevel semantics change** — documented in + `docs/Redundancy.md` + the interop note. +4. **Central is the system of record for deploy acks.** `ConfigPublishCoordinator.PersistNodeAck` + already writes `NodeDeploymentState` from every `ApplyAck` it receives over the transport, and seeds + the `Applying` rows at dispatch. So `DriverHostActor`'s direct `UpsertNodeDeploymentState` SQL writes + are **redundant** and are removed. The bootstrap `NodeDeploymentState` read is already replaced by + the LocalDb `deployment_pointer` for `FetchAndCache` (Phase 3). +5. **`ScriptedAlarmState` (Part 9 condition state) → LocalDb, wholesale for driver-role nodes.** It is + read only by `EfAlarmConditionStateStore`; AdminUI `/alerts` uses live SignalR, not the table. A new + replicated `alarm_condition_state` LocalDb table + `LocalDbAlarmConditionStateStore` replaces the EF + store on **every** driver-role node (fused central included), same journey Phase 2 made for the alarm + S&F buffer. The dead ConfigDb `ScriptedAlarmState` table is dropped in a final trivial migration. +6. **`OpcUaPublishActor._dbFactory` becomes null on driver-only nodes.** Its only ConfigDb use is + `LoadArtifact`/`LoadLatestArtifact` (the #485 artifact-blob reads); `FetchAndCache` already passes + `msg.Artifact` in-hand so the fallback never fires. Phase 4 asserts the null-factory path never reads + ConfigDb and every driver-only rebuild carries bytes. + +## The five consumers (design §6.1 audit) and their Phase-4 disposition + +| Consumer | Current ConfigDb use | Phase-4 disposition | Task | +|---|---|---|---| +| `DriverHostActor` | `UpsertNodeDeploymentState` writes (×9), bootstrap `NodeDeploymentState` read, `EfAlarmConditionStateStore` ctor | drop SQL ack-writes (central persists); nullable factory; LocalDb condition store | 4, 5–6 | +| `EfAlarmConditionStateStore` | Part 9 condition state in ConfigDb `ScriptedAlarmState` | → `LocalDbAlarmConditionStateStore` (new replicated table) | 5, 6 | +| `DbHealthProbeActor` | `SELECT 1` vs ConfigDb → `DbReachable` ServiceLevel input | not spawned on driver-only; `DbReachable=true` constant | 2, 3 | +| `OpcUaPublishActor` | `LoadArtifact`/`LoadLatestArtifact` | nullable factory; rebuild always carries bytes | 3 | +| `ServiceCollectionExtensions` (Runtime + Configuration) + `Program.cs` | registration | `hasAdmin`-gated; nullable threading | 1, 7 | + +--- + +## Task 0: `ConfigSourceOptionsValidator` — driver-only ⇒ FetchAndCache + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** Task 5 + +**Files:** +- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptionsValidator.cs` +- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Configuration/ConfigSourceOptionsValidatorTests.cs` + (or the existing validator test file — locate with `grep -rl ConfigSourceOptionsValidator tests`) + +**Context:** The validator today checks the `FetchAndCache` fields (endpoints/key). Phase 4 adds the +role-conditional rule. The validator must be given the node's roles — check how it reads them today +(likely `IConfiguration["Cluster:Roles"]` or an injected role view); mirror `AkkaClusterOptionsValidator`, +which already matches on roles at boot. + +**Step 1: Write the failing test** — a `driver`-role, non-`admin` node with `Mode=Direct` → validation +fails with a message naming the flag; a fused `admin,driver` node with `Mode=Direct` → passes; a +driver-only node with `Mode=FetchAndCache` → passes. + +**Step 2:** Run it, confirm it fails (rule not yet present). + +**Step 3: Implement** the rule: `roles.Contains("driver") && !roles.Contains("admin") && Mode==Direct` +→ `ValidateOptionsResult.Fail(...)`. Read roles the same way the validator/`AddValidatedOptions` wiring +already does; do not invent a new source. + +**Step 4:** Run tests, confirm pass. + +**Step 5: Commit** — `feat(mesh-phase4): driver-only node must be FetchAndCache (validator)`. + +--- + +## Task 1: `Program.cs` — gate ConfigDb registration + health check on `hasAdmin` + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** none (boot path; other tasks build on the nullable factory it introduces) + +**Files:** +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:125-126` (the unconditional `AddOtOpcUaConfigDb`) +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs:27-31` + (`DatabaseHealthCheck` — driver-only has no ConfigDb to probe) +- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DriverOnlyNoConfigDbBootTests.cs` (create) + +**Context:** `AddOtOpcUaConfigDb` throws if the `ConfigDb` connection string is absent. Line 126 comment +says "always registered regardless of role. ConfigDb is required for everything" — that statement is +what Phase 4 falsifies. Gate the call on `hasAdmin`. Anything downstream that resolves +`IDbContextFactory` non-optionally must already be `hasAdmin`-only or become +nullable (Tasks 2–7 handle the driver-branch resolvers). Grep `GetRequiredService<...ConfigDbContext...>` +and `GetRequiredService>` across the Host to find any that +would now throw on a driver-only node; each must move under `hasAdmin` or switch to `GetService`. + +**Step 1: Write the failing test** — build a Host `WebApplication` with roles `driver` only and **no** +`ConfigDb` connection string; assert it builds + starts without throwing, and that +`services.GetService>()` is `null`. (Model the harness on +`DeploymentArtifactFetchBoundaryTests.StartServerAsync` — minimal builder, in-memory where needed.) + +**Step 2:** Run it — fails today (`AddOtOpcUaConfigDb` throws "Connection string 'ConfigDb' is required"). + +**Step 3: Implement** — wrap `AddOtOpcUaConfigDb` in `if (hasAdmin)`; wrap the ConfigDb health check in +`if (hasAdmin)`; sweep the `GetRequiredService` sites found above. Add a boot log line on a driver-only +node: "driver-only node — no ConfigDb registered; config via ConfigSource:Mode=FetchAndCache". + +**Step 4:** Run the test + `dotnet build` — confirm pass and no unresolved-service throw. + +**Step 5: Commit** — `feat(mesh-phase4): register ConfigDb only on admin-role nodes`. + +--- + +## Task 2: `DbHealthProbeActor` — do not spawn on driver-only; `DbReachable=true` constant + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** none (ServiceLevel; Task 3 depends on its message contract) + +**Files:** +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:329-332` (spawn site) +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs:641-655` + (the `_dbHealthProbe is null` seam + the `Stale` derivation) +- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/OpcUa/OpcUaPublishActorServiceLevelTests.cs` + (extend the existing ServiceLevel test file) + +**Context:** `ServiceLevelCalculator.Compute` collapses `(DbReachable=false, probe ok, not stale)` → 0. +The retired-input rule (decision 3): on a node with `_dbHealthProbe is null`, feed +`DbReachable: true` and derive `Stale` **without** the DB-health term — +`Stale = (now - snapshotEntry.AsOfUtc) > _staleWindow || runningBehindConfig`. Do NOT fall through to +`LegacyRoleOnly` for driver-only nodes: that seam exists for the pre-first-sample bootstrap window on a +DB-backed node, and would peg a healthy site node at the role-only mapping forever. Introduce an explicit +"DB-less node" branch that computes the full `NodeHealthInputs` with `DbReachable=true` and the DB-free +staleness. `runningBehindConfig` = the Phase-3 running-from-cache / last-apply-failed signal if reachable +from the publish actor; if it is not currently threaded here, use the snapshot-age term alone and record +a `TODO(Phase 4 follow-up)` — do not thread a new dependency just for this in-scope. + +**Step 1: Write failing tests** — (a) DB-less node, member Up, probe ok, snapshot fresh → **240** +(+10 if Primary); (b) DB-less node, snapshot stale (>window) → **200**; (c) DB-backed node unchanged +(regression). Assert against `OpcUaPublishActor`'s computed `ServiceLevelChanged`, not the calculator +in isolation. + +**Step 2:** Run — (a)/(b) fail (today a DB-less node hits `LegacyRoleOnly` or 0). + +**Step 3: Implement** — in `ServiceCollectionExtensions`, spawn `DbHealthProbeActor` only when +`dbFactory is not null`; register `DbHealthProbeActorKey` only then; pass `dbHealthProbe: null` into +`OpcUaPublishActor.Props` on a DB-less node. In `OpcUaPublishActor`, add the DB-less branch described +above (guard on "no probe wired AND this is a driver-only/DB-less node" — a clean signal is +`_dbHealthProbe is null` combined with a ctor flag `dbLess`, threaded from the spawn site, so the +legacy-bootstrap seam stays reachable only on DB-backed nodes before their first sample). + +**Step 4:** Run tests + build — confirm pass. + +**Step 5: Commit** — `feat(mesh-phase4): retire DB-reachability from ServiceLevel on DB-less nodes`. + +--- + +## Task 3: `OpcUaPublishActor` — nullable factory; driver-only rebuild always carries bytes + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** none (same file as Task 2 — sequence after it) + +**Files:** +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs` +- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/OpcUa/OpcUaPublishActorTests.cs` + +**Context:** `_dbFactory` is already nullable on this actor; `HandleRebuild`'s `_dbFactory is null || +_applier is null` guard falls back to a raw `_sink.RebuildAddressSpace()`. That fallback is the +**wrong** path for a driver-only node that HAS an applier but no factory — it would skip the +diff-and-apply. The correct driver-only invariant: `_applier` is wired, `_dbFactory` is null, and every +`RebuildAddressSpace` carries `msg.Artifact` in-hand (Phase-3 `DriverHostActor` already does this). Split +the guard so a null factory **with** an applier still does the diff-and-apply pass, using `msg.Artifact`, +and asserts (logs an Error + abandons per #485) if `msg.Artifact` is null on such a node — a null artifact +with no factory to fall back to is "no answer," never an empty rebuild. + +**Step 1: Write failing tests** — (a) applier wired, factory null, `RebuildAddressSpace` **with** +`Artifact` → diff-and-apply runs (assert applier invoked, not the raw-sink fallback); (b) applier wired, +factory null, `RebuildAddressSpace` with **null** Artifact → no rebuild, Error logged (#485 negative). + +**Step 2:** Run — (a) fails (null factory currently routes to the raw-sink fallback). + +**Step 3: Implement** — restructure `HandleRebuild`: `if (_applier is null) { raw-sink fallback; return; }` +then the diff-and-apply block, where `artifact = msg.Artifact ?? (_dbFactory is not null ? (depId path +else latest) : Array.Empty())`; the existing #485 zero-length guard then abandons the null-factory ++ null-artifact case cleanly. + +**Step 4:** Run tests + build. + +**Step 5: Commit** — `feat(mesh-phase4): OpcUaPublish diff-applies from in-hand bytes with no ConfigDb`. + +--- + +## Task 4: `DriverHostActor` — nullable factory; drop redundant SQL ack-writes on DB-less nodes + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** none (core actor; Tasks 5–6 build the store it constructs) + +**Files:** +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` + (`_dbFactory` field/ctor/Props → nullable; the ~9 `UpsertNodeDeploymentState` call sites; + the bootstrap `NodeDeploymentState` read at 606-612; the `EfAlarmConditionStateStore` ctor at 588; + the CreateDbContext sites at 1920, 2096, 2588) +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:455` (Props call) +- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorDbLessTests.cs` (create) + +**Context:** Decision 4 — central's `ConfigPublishCoordinator.PersistNodeAck` is the ack system of record. +On a DB-less node `UpsertNodeDeploymentState` has no ConfigDb to write and its row is written by central +anyway. Guard every `UpsertNodeDeploymentState` body on `_dbFactory is not null` (make the method a no-op +when null) rather than deleting call sites — keeps the fused-node behaviour identical and the diff small. +Same for the bootstrap read (606-612) and the Direct-only CreateDbContext sites (1920/2096/2588): each is +already unreachable on a `FetchAndCache` node (Phase 3 routes bootstrap through the LocalDb pointer and +apply through the fetcher), so the change is defensive nullability, not new control flow. Verify each of +1920/2096/2588 is inside a Direct-mode-only path before guarding — if any is on the shared apply path, +that is a Phase-3 gap to surface, not silently guard. + +**Step 1: Write failing tests** — construct a `DriverHostActor` with `dbFactory: null`, +`fetchAndCacheMode: true`, a fake fetcher + LocalDb cache (mirror `DriverHostActorFetchAndCacheTests`): +(a) a full dispatch → Applied ack, **no** NullReferenceException from a factory deref; (b) bootstrap +with a seeded LocalDb pointer → revision restored with no ConfigDb read. + +**Step 2:** Run — fails today (`_dbFactory` non-nullable; `UpsertNodeDeploymentState` derefs it). + +**Step 3: Implement** — make `_dbFactory` nullable through field/ctor/both `Props` overloads (nullable +param **last**, matching the Phase-3 additions); `UpsertNodeDeploymentState` early-returns when null (log +Debug once); guard the read sites. Update `ServiceCollectionExtensions:455` to pass the (already nullable) +`dbFactory`. + +**Step 4:** Run tests + build. + +**Step 5: Commit** — `feat(mesh-phase4): DriverHostActor runs with no ConfigDb (central persists acks)`. + +--- + +## Task 5: `alarm_condition_state` LocalDb table + `LocalDbAlarmConditionStateStore` + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 0 + +**Files:** +- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/LocalDbAlarmConditionStateStore.cs` +- Modify: the LocalDb schema/DDL + `RegisterReplicated` wiring (find via + `grep -rn "alarm_sf_events" src` — mirror that table's registration exactly, in the same + `LocalDbSetup.OnReady` DDL→`RegisterReplicated` order which is load-bearing) +- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ScriptedAlarms/LocalDbAlarmConditionStateStoreTests.cs` + (port `EfAlarmConditionStateStoreTests` — same `IAlarmStateStore` contract, so the assertion bodies + transfer verbatim; only the store construction changes) + +**Context:** `LocalDbAlarmConditionStateStore : IAlarmStateStore` must round-trip the same fields +`EfAlarmConditionStateStore` does (Enabled/Acked/Confirmed/Shelving + ack/confirm audit + `CommentsJson`; +Active/LastActive/LastCleared transient; `LastTransitionUtc ↔ UpdatedAtUtc`). Columns mirror +`ScriptedAlarmState`. Replicated like `alarm_sf_events` so a failover peer holds the state — but **unlike** +the S&F buffer there is no drain gate: condition state is not delivered-once, it is the current snapshot, +idempotent last-write-wins on `UpdatedAtUtc` (the Ef store already tolerates the write race that way). +**Do NOT stack app-level LWW on LocalDb's HLC** (design §8) — the row upsert keyed by `ScriptedAlarmId` +is a single unconditional write, which is HLC-safe; keep it that way. + +**Step 1: Write failing tests** — the ported `IAlarmStateStore` round-trip suite against the LocalDb +store over a temp SQLite DB (the Runtime.Tests harness already opens LocalDb elsewhere — reuse it). + +**Step 2:** Run — fails (store + table absent). + +**Step 3: Implement** — the store + the `alarm_condition_state` DDL + `RegisterReplicated` registration. + +**Step 4:** Run tests + build. + +**Step 5: Commit** — `feat(mesh-phase4): LocalDb alarm-condition-state store (replicated, pair-local)`. + +--- + +## Task 6: Wire the LocalDb condition store into `DriverHostActor` for driver-role nodes + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** none (depends on Task 5's type + Task 4's nullable factory) + +**Files:** +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs:588` + (construct `LocalDbAlarmConditionStateStore` instead of `EfAlarmConditionStateStore`) +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs` + (resolve the LocalDb store dependency; thread it into `DriverHostActor.Props`) +- Test: extend `DriverHostActorDbLessTests` — the ScriptedAlarm host constructs against LocalDb with a + null ConfigDb factory. + +**Context:** The store is constructed inside `DriverHostActor.BuildScriptedAlarmHost` (line ~588). It +takes the LocalDb connection the node already opened (Phase 1/2). Prefer resolving an +`IAlarmStateStore` from DI (registered in the Host's `hasDriver` LocalDb block) over `new`-ing it in the +actor, so the actor stays factory-agnostic. Confirm the registration lands in the same `AddOtOpcUaLocalDb` +`hasDriver` branch as the S&F sink. + +**Step 1: Write failing test** — DB-less `DriverHostActor` boots its ScriptedAlarm host and a condition +save/load round-trips through LocalDb (no ConfigDb). + +**Step 2:** Run — fails (still constructs the Ef store needing a factory). + +**Step 3: Implement** — register `IAlarmStateStore → LocalDbAlarmConditionStateStore` in the Host +`hasDriver` LocalDb block; resolve + thread it; delete the `EfAlarmConditionStateStore` construction from +the actor. Leave the `EfAlarmConditionStateStore` class in the tree (removed in Task 9's migration cleanup) +or delete it now if nothing else references it (grep first). + +**Step 4:** Run tests + build. + +**Step 5: Commit** — `feat(mesh-phase4): driver-role nodes store alarm condition state in LocalDb`. + +--- + +## Task 7: Registration cleanup + full-graph sweep + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** none (integration seam over everything above) + +**Files:** +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs` (final nullable threading) +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (any remaining driver-branch ConfigDb resolves) +- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ServiceCollectionExtensionsTests.cs` + +**Context:** The exit-gate proof. After Tasks 1–6, sweep for any service the **driver-only** graph +resolves that still needs `OtOpcUaConfigDbContext`. Grep both projects for +`OtOpcUaConfigDbContext`, `IDbContextFactory`, `AddOtOpcUaConfigDb`, +`ScriptedAlarmStates`, and `GetRequiredService` of any of them; each hit must be `hasAdmin`-gated or +nullable-tolerant. `LdapGroupRoleMappingService` / `ILdapGroupRoleMappingService` (registered inside +`AddOtOpcUaConfigDb`) is admin/auth-side — confirm the driver-only login path does not resolve it, or +that the driver-only node never serves login (it hosts no AdminUI — see the topology table). + +**Step 1: Write the failing test** — a "driver-only DI graph resolves nothing ConfigDb-bound" test: +build the `hasDriver && !hasAdmin` service provider and assert +`GetService>()` is null AND that spawning the runtime actors +does not throw. + +**Step 2:** Run — fix whatever it catches. + +**Step 3: Implement** the remaining guards. + +**Step 4:** Run the full Runtime.Tests + Host.IntegrationTests suites + `dotnet build`. + +**Step 5: Commit** — `feat(mesh-phase4): driver-only DI graph resolves nothing ConfigDb-bound`. + +--- + +## Task 8: Docs — Redundancy.md (ServiceLevel change) + interop note + Configuration.md + CLAUDE.md + +**Classification:** small +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 9 + +**Files:** +- Modify: `docs/Redundancy.md` (ServiceLevel: DB-less node table + the `DbReachable=true` constant + the + 240-with-central-down behaviour; the `DbHealthProbeActor` row becomes admin-only) +- Modify: `docs/Configuration.md` (driver-only nodes: no `ConfigDb` string; `ConfigSource:Mode` must be + `FetchAndCache`; validator rule) +- Modify: `CLAUDE.md` (extend the mesh section — Phase 4: driver ConfigDb cut; ServiceLevel semantics; + alarm condition state in LocalDb) +- Modify: `docs/plans/2026-07-22-per-cluster-mesh-program.md` (Phase 4 status + tracking row) + +**Step 1–4:** Prose only — state the client-visible ServiceLevel change explicitly (a healthy DB-less +node now publishes 240/250 with central unreachable, where a Direct node would have dropped to 0/100). + +**Step 5: Commit** — `docs(mesh-phase4): ServiceLevel semantics + driver ConfigDb cut`. + +--- + +## Task 9: Drop the dead ConfigDb `ScriptedAlarmState` table (migration) + retire `EfAlarmConditionStateStore` + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** none (after Task 6 proves LocalDb store is the live path) + +**Files:** +- Create: EF migration dropping `ScriptedAlarmState` (find the migrations dir via + `grep -rl "class OtOpcUaConfigDbContext" ` → its `Migrations/` sibling; use `dotnet ef migrations add + DropScriptedAlarmState -c OtOpcUaConfigDbContext`) +- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/OtOpcUaConfigDbContext.cs`, + `.../Entities/ScriptedAlarmState.cs` (remove the DbSet + entity) +- Delete: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/EfAlarmConditionStateStore.cs` + + its test (if nothing else references them — grep first) + +**Context:** Only do the delete/drop once Task 6 is green and grep confirms no remaining reference. If the +migration surface is risky (central deployments with existing rows), the drop can be deferred to a +follow-up — mark it clearly and stop, rather than shipping a risky migration under time pressure. Keeping +a dead-but-harmless table is an acceptable interim. + +**Step 1–4:** Add migration, remove entity, delete dead store, `dotnet build` + migration-applies check +against a scratch DB. + +**Step 5: Commit** — `refactor(mesh-phase4): drop dead ConfigDb ScriptedAlarmState table`. + +--- + +## Task 10: Rig config — driver-only site nodes lose their ConfigDb string + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** Task 8 + +**Files:** +- Modify: `docker-dev/docker-compose.yml` (site-a-1/2, site-b-1/2: remove + `ConnectionStrings__ConfigDb`; set `ConfigSource__Mode: "FetchAndCache"` as the **default**, not + behind `OTOPCUA_CONFIG_MODE` — Phase 4 makes Direct invalid for these nodes; central-1/2 keep ConfigDb + + `Direct`) + +**Context:** The rig currently defaults site nodes to `Direct` and flips to `FetchAndCache` via +`OTOPCUA_CONFIG_MODE`. Phase 4 makes `FetchAndCache` the only valid mode for driver-only site nodes, so +it becomes their committed default and the `ConfigDb` string is removed to prove the cut. Verify each site +service still boots (the validator from Task 0 now *requires* this shape). This is a rig change only — the +live gate (below) runs it. + +**Step 1–4:** Edit compose; `lmxopcua-fix`-equivalent local `docker compose up`; confirm the four site +nodes boot with no ConfigDb string. + +**Step 5: Commit** — `chore(mesh-phase4): rig site nodes run with no ConfigDb string`. + +--- + +## Task 11: Live gate + +**Classification:** high-risk +**Estimated implement time:** live verification (not a code task) +**Parallelizable with:** none (final) + +**Files:** +- Create: `docs/plans/2026-07-23-mesh-phase4-live-gate.md` (record the run, as Phases 1–3 did) + +**Exit gate (design §6.1 + program Phase 4):** +1. Bring up the rig (central pair `Direct` + ConfigDb; four site nodes `FetchAndCache`, **no** ConfigDb + string). All six healthy. +2. **Deploy** a config via `POST :9200/api/deployments` (`X-Api-Key: docker-dev-deploy-key`, + `Content-Type: application/json`); confirm it seals green with the site nodes acking — proving central + persists the site-node acks with the site nodes holding no ConfigDb. +3. **Alarm cycle** — drive a scripted alarm ack/shelve on a site node; confirm the condition state + round-trips through LocalDb (inspect `alarm_condition_state` via the docker-cp SQLite recipe from the + Phase-1 follow-ups memory) and replicates to its pair peer. +4. **Historian** — confirm the alarm S&F drain + (if wired) continuous historization still run on the + Primary site node with no ConfigDb. +5. **ServiceLevel** — with **central SQL stopped**, a healthy site node still publishes **240/250** + (Client.CLI `redundancy`), proving the retired DB-reachability input (a Direct node would have shown + 0/100). Restart a site node → it boots last-known-good from the LocalDb pointer, no ConfigDb read. +6. **Grep proof** — `grep -rn "GetRequiredService>\|AddOtOpcUaConfigDb"` + shows every hit `hasAdmin`-gated; a driver-only container's env has no `ConnectionStrings__ConfigDb`. + +Record findings verbatim (the Phase 1–3 gates each caught a defect offline tests missed — expect the +same and do not close the gate on a green-by-luck run). + +--- + +## Batching for execution + +- **Batch 1 (mechanics):** Task 0, Task 1 — options rule + registration gate. +- **Batch 2 (ServiceLevel):** Task 2, Task 3 — the client-visible change (sequential, same file). +- **Batch 3 (actor + store):** Task 4, Task 5 (‖), then Task 6. +- **Batch 4 (integration):** Task 7 — the sweep + exit-gate DI proof. +- **Batch 5 (finish):** Task 8 (‖ Task 10), Task 9, then Task 11 live gate. diff --git a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json new file mode 100644 index 00000000..9ea4a055 --- /dev/null +++ b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json @@ -0,0 +1,18 @@ +{ + "planPath": "docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md", + "tasks": [ + {"id": 0, "subject": "Task 0: ConfigSourceOptionsValidator — driver-only ⇒ FetchAndCache", "status": "pending"}, + {"id": 1, "subject": "Task 1: Program.cs — gate ConfigDb registration + health check on hasAdmin", "status": "pending"}, + {"id": 2, "subject": "Task 2: DbHealthProbeActor — no spawn on driver-only; DbReachable=true constant", "status": "pending", "blockedBy": [1]}, + {"id": 3, "subject": "Task 3: OpcUaPublishActor — nullable factory; rebuild always carries bytes", "status": "pending", "blockedBy": [2]}, + {"id": 4, "subject": "Task 4: DriverHostActor — nullable factory; drop redundant SQL ack-writes", "status": "pending", "blockedBy": [1]}, + {"id": 5, "subject": "Task 5: alarm_condition_state LocalDb table + LocalDbAlarmConditionStateStore", "status": "pending"}, + {"id": 6, "subject": "Task 6: Wire LocalDb condition store into DriverHostActor for driver-role", "status": "pending", "blockedBy": [4, 5]}, + {"id": 7, "subject": "Task 7: Registration cleanup + full-graph driver-only DI sweep", "status": "pending", "blockedBy": [2, 3, 4, 6]}, + {"id": 8, "subject": "Task 8: Docs — Redundancy.md ServiceLevel + interop + Configuration.md + CLAUDE.md", "status": "pending", "blockedBy": [7]}, + {"id": 9, "subject": "Task 9: Drop dead ConfigDb ScriptedAlarmState table + retire EfAlarmConditionStateStore", "status": "pending", "blockedBy": [6]}, + {"id": 10, "subject": "Task 10: Rig — driver-only site nodes lose ConfigDb string", "status": "pending", "blockedBy": [7]}, + {"id": 11, "subject": "Task 11: Live gate", "status": "pending", "blockedBy": [8, 9, 10]} + ], + "lastUpdated": "2026-07-23" +} From d630a7e267892dcd4bda9b9cb06151a0b022765f Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 11:29:45 -0400 Subject: [PATCH 02/18] feat(mesh-phase4): driver-only node must be FetchAndCache (validator) Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- .../ConfigSourceOptionsValidator.cs | 33 ++++++++++++ .../ConfigSourceOptionsValidatorTests.cs | 52 ++++++++++++++++++- 2 files changed, 83 insertions(+), 2 deletions(-) diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptionsValidator.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptionsValidator.cs index 9519a622..520044ef 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptionsValidator.cs +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptionsValidator.cs @@ -1,3 +1,4 @@ +using Microsoft.Extensions.Configuration; using Microsoft.Extensions.Options; namespace ZB.MOM.WW.OtOpcUa.Cluster; @@ -14,6 +15,20 @@ namespace ZB.MOM.WW.OtOpcUa.Cluster; /// public sealed class ConfigSourceOptionsValidator : IValidateOptions { + private readonly IConfiguration _configuration; + + /// + /// DI-constructed by AddValidatedOptions (a plain AddSingleton), so a + /// constructor dependency is safe here. — not + /// IClusterRoleInfo — is the source of this node's roles: IClusterRoleInfo's + /// implementation needs the ActorSystem, which does not exist yet at + /// ValidateOnStart time. + /// + public ConfigSourceOptionsValidator(IConfiguration configuration) + { + _configuration = configuration; + } + /// public ValidateOptionsResult Validate(string? name, ConfigSourceOptions options) { @@ -76,6 +91,24 @@ public sealed class ConfigSourceOptionsValidator : IValidateOptions() ?? Array.Empty(); + var isDriver = Array.IndexOf(roles, "driver") >= 0; + var isAdmin = Array.IndexOf(roles, "admin") >= 0; + + if (isDriver && !isAdmin && isDirect) + { + errors.Add( + "Cluster:Roles is a driver-only node (has 'driver', not 'admin') but " + + $"{ConfigSourceOptions.SectionName}:{nameof(ConfigSourceOptions.Mode)} is 'Direct'. A " + + "driver-only node has no central ConfigDb to read from — it must use 'FetchAndCache'. " + + $"Set {ConfigSourceOptions.SectionName}:{nameof(ConfigSourceOptions.Mode)}=" + + $"{ConfigSourceOptions.ModeFetchAndCache} (see docs/Configuration.md → ConfigSource)."); + } + return errors.Count == 0 ? ValidateOptionsResult.Success : ValidateOptionsResult.Fail(string.Join(" ", errors)); diff --git a/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ConfigSourceOptionsValidatorTests.cs b/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ConfigSourceOptionsValidatorTests.cs index 064e68e8..419f522d 100644 --- a/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ConfigSourceOptionsValidatorTests.cs +++ b/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ConfigSourceOptionsValidatorTests.cs @@ -1,3 +1,4 @@ +using Microsoft.Extensions.Configuration; using Microsoft.Extensions.Options; using Shouldly; using Xunit; @@ -11,8 +12,18 @@ namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests; /// public class ConfigSourceOptionsValidatorTests { - private static ValidateOptionsResult Validate(ConfigSourceOptions o) => - new ConfigSourceOptionsValidator().Validate(ConfigSourceOptions.SectionName, o); + private static ValidateOptionsResult Validate(ConfigSourceOptions o, params string[] roles) + { + var pairs = new Dictionary(); + for (var i = 0; i < roles.Length; i++) + { + pairs[$"Cluster:Roles:{i}"] = roles[i]; + } + + var configuration = new ConfigurationBuilder().AddInMemoryCollection(pairs).Build(); + + return new ConfigSourceOptionsValidator(configuration).Validate(ConfigSourceOptions.SectionName, o); + } private static ConfigSourceOptions ValidFetch() => new() { @@ -117,4 +128,41 @@ public class ConfigSourceOptionsValidatorTests FetchTimeoutSeconds = 0, }).Succeeded.ShouldBeTrue(); } + + [Fact] + public void Driver_only_node_on_Direct_fails() + { + // A driver-only node has no central ConfigDb connection (Phase 4) — Direct silently produces a + // node that can never read its configuration. + var result = Validate( + new ConfigSourceOptions { Mode = ConfigSourceOptions.ModeDirect }, + "driver"); + + result.Failed.ShouldBeTrue(); + result.FailureMessage.ShouldContain(ConfigSourceOptions.ModeFetchAndCache); + } + + [Fact] + public void Fused_admin_and_driver_node_on_Direct_is_valid() + { + // A fused node holds the ConfigDb connection via its admin role, so Direct is still legitimate. + Validate( + new ConfigSourceOptions { Mode = ConfigSourceOptions.ModeDirect }, + "admin", "driver").Succeeded.ShouldBeTrue(); + } + + [Fact] + public void Driver_only_node_on_FetchAndCache_is_valid() + { + Validate(ValidFetch(), "driver").Succeeded.ShouldBeTrue(); + } + + [Fact] + public void Admin_only_node_on_Direct_is_valid() + { + // An admin-only node has no driver role at all, so the driver-only rule does not apply to it. + Validate( + new ConfigSourceOptions { Mode = ConfigSourceOptions.ModeDirect }, + "admin").Succeeded.ShouldBeTrue(); + } } From a41582ea6a8da2e1ca8bb459fa5a430d0cb8faca Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 11:38:40 -0400 Subject: [PATCH 03/18] feat(mesh-phase4): register ConfigDb only on admin-role nodes Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- .../Health/HealthEndpoints.cs | 35 +++++-- src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs | 16 +++- .../DriverOnlyNoConfigDbBootTests.cs | 95 +++++++++++++++++++ 3 files changed, 133 insertions(+), 13 deletions(-) create mode 100644 tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DriverOnlyNoConfigDbBootTests.cs diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs index 4ad7385e..903fb92e 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs @@ -17,21 +17,37 @@ public static class HealthEndpoints { /// /// Registers the shared ZB.MOM.WW health probes. Tier semantics preserved: configdb + akka on - /// ready+active; admin-leader on active only. + /// ready+active; admin-leader on active only. The configdb probe is admin-only (per-cluster mesh + /// Phase 4): a driver-only node holds no ConfigDb, so it is registered iff . /// /// The service collection to register the health checks on. + /// + /// Whether this node carries the admin role. When false (a driver-only node) the + /// configdb probe is skipped — the node registers no + /// factory, so probing it would throw when the readiness endpoint resolves the factory. Defaults to + /// true so admin/fused callers (including the test harnesses) keep the probe unchanged. + /// /// The same service collection, for chaining. - public static IServiceCollection AddOtOpcUaHealth(this IServiceCollection services) + public static IServiceCollection AddOtOpcUaHealth(this IServiceCollection services, bool hasAdmin = true) { - services.AddHealthChecks() - .AddTypeActivatedCheck>( + var checks = services.AddHealthChecks(); + + // configdb probe is admin-only (per-cluster mesh Phase 4). A driver-only node gets its config + // from central via ConfigSource:Mode=FetchAndCache and never registers the ConfigDb factory, so + // the type-activated DatabaseHealthCheck would throw resolving it. Admin + fused nodes: unchanged. + if (hasAdmin) + { + checks.AddTypeActivatedCheck>( "configdb", failureStatus: null, tags: new[] { ZbHealthTags.Ready, ZbHealthTags.Active }, args: new DatabaseHealthCheckOptions { ProbeQuery = static (db, ct) => db.Deployments.AsNoTracking().Take(1).ToListAsync(ct), - }) + }); + } + + checks .AddTypeActivatedCheck( "akka", failureStatus: null, @@ -42,11 +58,10 @@ public static class HealthEndpoints failureStatus: null, tags: new[] { ZbHealthTags.Active }, args: "admin") - // Registered unconditionally, not driver-gated. AddOtOpcUaHealth takes no role argument - // and runs on every node; the check itself resolves ISyncStatus optionally and reports - // Healthy when LocalDb is absent (admin-only graphs) or replication is default-OFF, so a - // plain node is never degraded by it. A factory registration keeps this no-arg signature - // while still reading ISyncStatus + options + the sync port from the container. + // Registered on every node regardless of role (unlike the admin-only configdb probe above): + // the check itself resolves ISyncStatus optionally and reports Healthy when LocalDb is absent + // (admin-only graphs) or replication is default-OFF, so a plain node is never degraded by it. + // A factory registration reads ISyncStatus + options + the sync port from the container. .Add(new HealthCheckRegistration( "localdb-replication", sp => new LocalDbReplicationHealthCheck( diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs index 74fb072b..0ed23bda 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs @@ -122,8 +122,16 @@ builder.AddZbSerilog(o => o.ServiceName = "otopcua"); // Windows-service registration is handled at install time by scripts/install/Install-Services.ps1 // rather than in-process, so the binary stays cross-platform-compilable. -// Shared services — always registered regardless of role. ConfigDb is required for everything. -builder.Services.AddOtOpcUaConfigDb(builder.Configuration); +// ConfigDb is admin-only now (per-cluster mesh Phase 4). Only the admin role owns a ConfigDb +// connection string; a driver-only node holds none at all and gets its deployed configuration from +// central over the Phase-3 fetch-and-cache path (ConfigSource:Mode=FetchAndCache). Registering it +// unconditionally would throw on a driver-only node, since AddOtOpcUaConfigDb requires the string. +// A fused admin+driver node keeps ConfigDb via its admin role — byte-for-byte unaffected. +if (hasAdmin) + builder.Services.AddOtOpcUaConfigDb(builder.Configuration); +else + Log.Information("Driver-only node — no ConfigDb registered; configuration via ConfigSource:Mode=FetchAndCache"); + builder.Services.AddOtOpcUaCluster(builder.Configuration); // Validate LdapOptions unconditionally so ANY role node (admin-only, driver-only, or fused) @@ -392,7 +400,9 @@ if (hasAdmin) // Cluster replication is opt-in behind Secrets:Replication:Enabled (default false) — see SecretsRegistration. builder.Services.AddOtOpcUaSecrets(builder.Configuration); -builder.Services.AddOtOpcUaHealth(); +// hasAdmin gates the configdb probe: a driver-only node registers no ConfigDb factory (see above), so +// probing it would throw when the readiness endpoint resolves the factory. +builder.Services.AddOtOpcUaHealth(hasAdmin); builder.Services.AddOtOpcUaObservability(builder.Configuration); // gRPC server plumbing shared by two endpoints: the LocalDb passive sync endpoint (driver-role, diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DriverOnlyNoConfigDbBootTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DriverOnlyNoConfigDbBootTests.cs new file mode 100644 index 00000000..954ddadc --- /dev/null +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DriverOnlyNoConfigDbBootTests.cs @@ -0,0 +1,95 @@ +using Microsoft.AspNetCore.Builder; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Configuration; + +namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests; + +/// +/// Per-cluster mesh Phase 4 — ConfigDb is registered iff the node has the admin role. +/// A driver-only node (Akka role driver, NOT admin) boots with NO ConfigDb +/// connection string at all; its configuration arrives from central via the Phase-3 +/// ConfigSource:Mode=FetchAndCache path. +/// +/// +/// +/// These tests replicate the exact ConfigDb gating decision Program.cs makes — +/// if (hasAdmin) services.AddOtOpcUaConfigDb(configuration) — over the real +/// method, for both role shapes. +/// They deliberately do NOT boot the full host: Program.cs reads roles from the +/// OTOPCUA_ROLES process env var and starting the host triggers the Akka cluster join +/// (see TwoNodeClusterHarness on why WebApplicationFactory<Program> is not +/// driven here, and LocalDbWiringTests for the same partial-graph harness model). The +/// full driver-only boot is proven by the Phase-4 live gate + the later DI-graph sweep task. +/// +/// +public sealed class DriverOnlyNoConfigDbBootTests : IDisposable +{ + private readonly List _apps = []; + + /// + /// Builds the shared-services graph the way Program.cs gates ConfigDb: it is registered iff + /// . A driver-only node passes hasAdmin: false and supplies + /// no ConnectionStrings:ConfigDb at all. + /// + private IServiceProvider BuildSharedServicesGraph(bool hasAdmin, string? configDbConnectionString) + { + var builder = WebApplication.CreateBuilder(new WebApplicationOptions { Args = [] }); + if (configDbConnectionString is not null) + { + builder.Configuration.AddInMemoryCollection(new Dictionary + { + ["ConnectionStrings:ConfigDb"] = configDbConnectionString, + }); + } + + if (hasAdmin) + builder.Services.AddOtOpcUaConfigDb(builder.Configuration); + + var app = builder.Build(); + _apps.Add(app); + return app.Services; + } + + [Fact] + public void DriverOnlyNode_NoConnectionString_BuildsWithoutThrowing_AndRegistersNoConfigDbFactory() + { + // The core Phase-4 guarantee: a driver-only node holds no ConfigDb connection string and must + // still build its service graph without throwing — nothing in the shared registrations may + // reach for the ConfigDb factory. + IServiceProvider sp = null!; + Should.NotThrow(() => sp = BuildSharedServicesGraph(hasAdmin: false, configDbConnectionString: null)); + + sp.GetService>().ShouldBeNull(); + } + + [Fact] + public void AdminNode_WithConnectionString_RegistersTheConfigDbFactory() + { + // The fused/admin path is byte-for-byte unaffected: ConfigDb is registered exactly as before. + var sp = BuildSharedServicesGraph( + hasAdmin: true, + configDbConnectionString: "Server=(local);Database=OtOpcUa;Trusted_Connection=True;TrustServerCertificate=True"); + + sp.GetService>().ShouldNotBeNull(); + } + + [Fact] + public void AddOtOpcUaConfigDb_WithoutConnectionString_Throws() + { + // Why the gate exists: the unconditional call Program.cs made throws on a driver-only node, + // which has no ConfigDb connection string. This pins the exact failure the hasAdmin gate avoids. + var builder = WebApplication.CreateBuilder(new WebApplicationOptions { Args = [] }); + + Should.Throw(() => builder.Services.AddOtOpcUaConfigDb(builder.Configuration)); + } + + public void Dispose() + { + foreach (var app in _apps) + ((IDisposable)app).Dispose(); + } +} From 833e45db9cda667200c2c69bad9a8b4ad6044399 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 11:44:44 -0400 Subject: [PATCH 04/18] fix(mesh-phase4): driver-only nodes map LDAP groups from appsettings (no ConfigDb grants) Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs | 12 +++- .../Ldap/NullLdapGroupRoleMappingService.cs | 44 ++++++++++++ .../OtOpcUaGroupRoleMapperTests.cs | 67 +++++++++++++++++++ 3 files changed, 121 insertions(+), 2 deletions(-) create mode 100644 src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/NullLdapGroupRoleMappingService.cs diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs index 0ed23bda..3204646c 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs @@ -13,6 +13,7 @@ using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs; using ZB.MOM.WW.OtOpcUa.AdminUI.ScriptAnalysis; using ZB.MOM.WW.OtOpcUa.Cluster; using ZB.MOM.WW.OtOpcUa.Configuration; +using ZB.MOM.WW.OtOpcUa.Configuration.Services; using ZB.MOM.WW.OtOpcUa.ControlPlane; using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy; using ZB.MOM.WW.OtOpcUa.Commons.OpcUa; @@ -319,11 +320,18 @@ if (hasDriver) // OtOpcUaLdapAuthService is the app ILdapAuthService (Enabled switch + DevStubMode over the // shared ZB.MOM.WW.Auth.Ldap service). The data-plane authenticator resolves IGroupRoleMapper // per call to turn the directory's groups into roles, so register it here for driver- - // only nodes (AddOtOpcUaAuth registers it on admin nodes); ILdapGroupRoleMappingService it - // depends on is already registered unconditionally by AddOtOpcUaConfigDb above. + // only nodes (AddOtOpcUaAuth registers it on admin nodes). OtOpcUaGroupRoleMapper also depends + // on ILdapGroupRoleMappingService (the control-plane DB-grants CRUD surface) — Phase 4 gated + // AddOtOpcUaConfigDb on hasAdmin, so a driver-only node has no ConfigDb to read it from. Bind + // NullLdapGroupRoleMappingService (always "no DB grants") here via TryAdd so the mapper falls + // back to the appsettings Security:Ldap:GroupToRole baseline; on a fused admin+driver node the + // hasAdmin block above already ran AddOtOpcUaConfigDb's non-Try AddScoped, so this TryAdd is a no-op there and the real EF service + // wins. // Note: AddValidatedOptions is now registered unconditionally // above both role blocks so admin-only nodes also get fail-fast LDAP startup validation. builder.Services.TryAddSingleton(); + builder.Services.TryAddScoped(); builder.Services.TryAddScoped, OtOpcUaGroupRoleMapper>(); builder.Services.AddSingleton(); diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/NullLdapGroupRoleMappingService.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/NullLdapGroupRoleMappingService.cs new file mode 100644 index 00000000..15c91bfb --- /dev/null +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/NullLdapGroupRoleMappingService.cs @@ -0,0 +1,44 @@ +using ZB.MOM.WW.OtOpcUa.Configuration.Entities; +using ZB.MOM.WW.OtOpcUa.Configuration.Services; + +namespace ZB.MOM.WW.OtOpcUa.Security.Ldap; + +/// +/// Driver-only-node stand-in for the EF-backed . +/// A driver-only node (Akka role driver, not admin) has no ConfigDb connection +/// (Phase 4 gated AddOtOpcUaConfigDb on hasAdmin), so it can never read the +/// control-plane LdapGroupRoleMapping DB grants that +/// normally unions into its result. This implementation always reports "no DB grants" so the +/// data-path mapper falls back to exactly the appsettings Security:Ldap:GroupToRole +/// baseline — the same safe fallback the mapper already takes when the real service simply +/// has no matching rows. rows are not composed into the +/// deployment artifact either, so a driver node never had DB grants via config; reporting +/// empty here is honest, not lossy. +/// +/// +/// The write surface (/) is control-plane-only +/// Admin UI functionality that a driver-only node — which has no Admin UI — never calls. It +/// throws rather than silently no-opping so an accidental call surfaces immediately instead of +/// pretending to persist a grant that a driver-only node cannot store. +/// +public sealed class NullLdapGroupRoleMappingService : ILdapGroupRoleMappingService +{ + /// + public Task> GetByGroupsAsync( + IEnumerable ldapGroups, CancellationToken cancellationToken) + => Task.FromResult>(Array.Empty()); + + /// + public Task> ListAllAsync(CancellationToken cancellationToken) + => Task.FromResult>(Array.Empty()); + + /// + public Task CreateAsync(LdapGroupRoleMapping row, CancellationToken cancellationToken) + => throw new NotSupportedException( + "LDAP group-role grants are control-plane only; a driver-only node has no ConfigDb."); + + /// + public Task DeleteAsync(Guid id, CancellationToken cancellationToken) + => throw new NotSupportedException( + "LDAP group-role grants are control-plane only; a driver-only node has no ConfigDb."); +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests/OtOpcUaGroupRoleMapperTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests/OtOpcUaGroupRoleMapperTests.cs index bd6559b7..f444b8d6 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests/OtOpcUaGroupRoleMapperTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests/OtOpcUaGroupRoleMapperTests.cs @@ -72,6 +72,27 @@ public sealed class OtOpcUaGroupRoleMapperTests result.Roles.ShouldBeEmpty(); } + [Fact] + public async Task Driver_only_node_falls_back_to_appsettings_baseline_via_null_mapping_service() + { + // Simulates a driver-only node: no ConfigDb, so ILdapGroupRoleMappingService is bound to + // NullLdapGroupRoleMappingService instead of the EF-backed one. MapAsync must still resolve + // roles from Security:Ldap:GroupToRole alone, and must not throw for lack of a DB. + var options = Options.Create(new LdapOptions + { + GroupToRole = new Dictionary(StringComparer.OrdinalIgnoreCase) + { + ["operators"] = "WriteOperate", + }, + }); + var mapper = new OtOpcUaGroupRoleMapper(options, new NullLdapGroupRoleMappingService()); + + var result = await mapper.MapAsync(["operators", "unmapped-group"], CancellationToken.None); + + result.Roles.ShouldBe(["WriteOperate"]); + result.Scope.ShouldBeNull(); + } + [Fact] public async Task Reproduces_RoleMapper_Map_plus_Merge_for_representative_inputs() { @@ -116,3 +137,49 @@ public sealed class OtOpcUaGroupRoleMapperTests => throw new NotSupportedException(); } } + +/// +/// Proves — the driver-only-node stand-in for the +/// EF-backed ILdapGroupRoleMappingService — always reports "no DB grants" rather than +/// throwing on the hot sign-in path, while its control-plane write surface (never called +/// without an AdminUI) fails loudly instead of silently no-opping. +/// +public sealed class NullLdapGroupRoleMappingServiceTests +{ + [Fact] + public async Task GetByGroupsAsync_returns_empty() + { + var sut = new NullLdapGroupRoleMappingService(); + + var result = await sut.GetByGroupsAsync(["any-group"], CancellationToken.None); + + result.ShouldBeEmpty(); + } + + [Fact] + public async Task ListAllAsync_returns_empty() + { + var sut = new NullLdapGroupRoleMappingService(); + + var result = await sut.ListAllAsync(CancellationToken.None); + + result.ShouldBeEmpty(); + } + + [Fact] + public async Task CreateAsync_throws_NotSupportedException() + { + var sut = new NullLdapGroupRoleMappingService(); + var row = new LdapGroupRoleMapping { LdapGroup = "g", Role = AdminRole.Administrator, IsSystemWide = true }; + + await Should.ThrowAsync(() => sut.CreateAsync(row, CancellationToken.None)); + } + + [Fact] + public async Task DeleteAsync_throws_NotSupportedException() + { + var sut = new NullLdapGroupRoleMappingService(); + + await Should.ThrowAsync(() => sut.DeleteAsync(Guid.NewGuid(), CancellationToken.None)); + } +} From 5c40908a55a863f12fd76b58939910c292549d43 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 11:48:45 -0400 Subject: [PATCH 05/18] docs(mesh-phase4): add Task 1b (driver-only LDAP mapper) discovered during Task 1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Gating ConfigDb on hasAdmin exposed a 6th driver-side ConfigDb consumer the §6.1 audit missed: OtOpcUaGroupRoleMapper's ILdapGroupRoleMappingService dep. Task 7 gains a fused-node ordering pin (Task 1b review gap). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- ...6-07-23-mesh-phase4-cut-driver-configdb.md | 50 +++++++- ...h-phase4-cut-driver-configdb.md.tasks.json | 120 ++++++++++++++++-- 2 files changed, 157 insertions(+), 13 deletions(-) diff --git a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md index 01c79d57..7c34c9db 100644 --- a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md +++ b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md @@ -129,6 +129,49 @@ node: "driver-only node — no ConfigDb registered; config via ConfigSource:Mode --- +## Task 1b: driver-only `ILdapGroupRoleMappingService` — empty (appsettings-only group→role) — DISCOVERED during Task 1 + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** Task 5 + +**Files:** +- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/NullLdapGroupRoleMappingService.cs` (or a Host-side + driver-only impl — place it beside `OtOpcUaGroupRoleMapper`) +- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:317-327` (the `hasDriver` auth block — register the + empty service with `TryAddScoped` so a fused node keeps the real EF one; fix the now-false comment at + 322-323) +- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests/` (or wherever `OtOpcUaGroupRoleMapper` is tested) + +**Context — a 6th ConfigDb consumer the §6.1 audit missed.** `OtOpcUaGroupRoleMapper` (the OPC UA +data-path group→role mapper, `IGroupRoleMapper`, registered at `Program.cs:327`) merges the +appsettings `Security:Ldap:GroupToRole` baseline (always available) with **system-wide DB grants** from +`ILdapGroupRoleMappingService` — which is registered **only** inside `AddOtOpcUaConfigDb`. Task 1 gated +that on `hasAdmin`, so a driver-only node now cannot resolve `IGroupRoleMapper` at OPC UA login +(lazy scoped resolve — `Build()` does not surface it; the auth path does). `LdapGroupRoleMapping` rows are +**not** in the deployment artifact (`ConfigComposer` never composes them), so driver nodes never received +DB grants via config in the first place. The honest fix: a driver-only `ILdapGroupRoleMappingService` whose +`GetByGroupsAsync` returns empty (CRUD methods throw `NotSupportedException` — never called with no AdminUI), +registered via `TryAddScoped` in the `hasDriver` block so the fused node's real EF impl still wins. +**Behavior reduction to document (Task 8):** DB-backed LDAP group→role grants (the AdminUI RoleGrants page) +apply only where ConfigDb is present (admin/fused nodes); driver-only site nodes map groups→roles from +`Security:Ldap:GroupToRole` appsettings only. + +**Step 1: Write the failing test** — resolve `IGroupRoleMapper` in a driver-only service graph and +call `MapAsync(["some-group"])` → returns the appsettings baseline with no throw (today it throws: +`ILdapGroupRoleMappingService` unresolvable). + +**Step 2:** Run — fails. + +**Step 3: Implement** the empty service + `TryAddScoped` registration in the `hasDriver` block; correct the +`Program.cs:322-323` comment to state the driver-only node uses the empty impl and why. + +**Step 4:** Run tests + build. + +**Step 5: Commit** — `fix(mesh-phase4): driver-only nodes map LDAP groups from appsettings (no ConfigDb grants)`. + +--- + ## Task 2: `DbHealthProbeActor` — do not spawn on driver-only; `DbReachable=true` constant **Classification:** high-risk @@ -346,7 +389,12 @@ that the driver-only node never serves login (it hosts no AdminUI — see the to **Step 1: Write the failing test** — a "driver-only DI graph resolves nothing ConfigDb-bound" test: build the `hasDriver && !hasAdmin` service provider and assert `GetService>()` is null AND that spawning the runtime actors -does not throw. +does not throw. **Also pin the Task-1b fused-node ordering** (Task 1b review gap): build the +`hasAdmin && hasDriver` graph through the same `AddOtOpcUaConfigDb` + `hasDriver` TryAdd sequence and assert +`GetService()` resolves the **real** `LdapGroupRoleMappingService`, not the +`Null…` — so a later reorder of the role blocks (or switching `AddOtOpcUaConfigDb` to a TryAdd) fails a +red test instead of silently stripping DB-backed role grants from central. Also resolve +`IGroupRoleMapper` in a driver-only scope to catch the lazy auth-path break Task 1b fixed. **Step 2:** Run — fix whatever it catches. diff --git a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json index 9ea4a055..32d94fe7 100644 --- a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json +++ b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json @@ -1,18 +1,114 @@ { "planPath": "docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md", "tasks": [ - {"id": 0, "subject": "Task 0: ConfigSourceOptionsValidator — driver-only ⇒ FetchAndCache", "status": "pending"}, - {"id": 1, "subject": "Task 1: Program.cs — gate ConfigDb registration + health check on hasAdmin", "status": "pending"}, - {"id": 2, "subject": "Task 2: DbHealthProbeActor — no spawn on driver-only; DbReachable=true constant", "status": "pending", "blockedBy": [1]}, - {"id": 3, "subject": "Task 3: OpcUaPublishActor — nullable factory; rebuild always carries bytes", "status": "pending", "blockedBy": [2]}, - {"id": 4, "subject": "Task 4: DriverHostActor — nullable factory; drop redundant SQL ack-writes", "status": "pending", "blockedBy": [1]}, - {"id": 5, "subject": "Task 5: alarm_condition_state LocalDb table + LocalDbAlarmConditionStateStore", "status": "pending"}, - {"id": 6, "subject": "Task 6: Wire LocalDb condition store into DriverHostActor for driver-role", "status": "pending", "blockedBy": [4, 5]}, - {"id": 7, "subject": "Task 7: Registration cleanup + full-graph driver-only DI sweep", "status": "pending", "blockedBy": [2, 3, 4, 6]}, - {"id": 8, "subject": "Task 8: Docs — Redundancy.md ServiceLevel + interop + Configuration.md + CLAUDE.md", "status": "pending", "blockedBy": [7]}, - {"id": 9, "subject": "Task 9: Drop dead ConfigDb ScriptedAlarmState table + retire EfAlarmConditionStateStore", "status": "pending", "blockedBy": [6]}, - {"id": 10, "subject": "Task 10: Rig — driver-only site nodes lose ConfigDb string", "status": "pending", "blockedBy": [7]}, - {"id": 11, "subject": "Task 11: Live gate", "status": "pending", "blockedBy": [8, 9, 10]} + { + "id": 0, + "subject": "Task 0: ConfigSourceOptionsValidator \u2014 driver-only \u21d2 FetchAndCache", + "status": "completed", + "commit": "d630a7e", + "result": "13/13 tests; APPROVED by code review" + }, + { + "id": 1, + "subject": "Task 1: Program.cs \u2014 gate ConfigDb registration + health check on hasAdmin", + "status": "completed", + "commit": "a41582ea", + "result": "8/8 tests; surfaced Task 1b (IGroupRoleMapper ConfigDb dep)" + }, + { + "id": "1b", + "subject": "Task 1b: driver-only ILdapGroupRoleMappingService \u2014 empty (appsettings-only)", + "status": "completed", + "blockedBy": [ + 1 + ], + "commit": "833e45db", + "result": "9/9 tests; APPROVED (Task 7 to pin fused-node ordering)" + }, + { + "id": 2, + "subject": "Task 2: DbHealthProbeActor \u2014 no spawn on driver-only; DbReachable=true constant", + "status": "pending", + "blockedBy": [ + 1 + ] + }, + { + "id": 3, + "subject": "Task 3: OpcUaPublishActor \u2014 nullable factory; rebuild always carries bytes", + "status": "pending", + "blockedBy": [ + 2 + ] + }, + { + "id": 4, + "subject": "Task 4: DriverHostActor \u2014 nullable factory; drop redundant SQL ack-writes", + "status": "pending", + "blockedBy": [ + 1 + ] + }, + { + "id": 5, + "subject": "Task 5: alarm_condition_state LocalDb table + LocalDbAlarmConditionStateStore", + "status": "pending" + }, + { + "id": 6, + "subject": "Task 6: Wire LocalDb condition store into DriverHostActor for driver-role", + "status": "pending", + "blockedBy": [ + 4, + 5 + ] + }, + { + "id": 7, + "subject": "Task 7: Registration cleanup + full-graph driver-only DI sweep", + "status": "pending", + "blockedBy": [ + "1b", + 2, + 3, + 4, + 6 + ] + }, + { + "id": 8, + "subject": "Task 8: Docs \u2014 Redundancy.md ServiceLevel + interop + Configuration.md + CLAUDE.md", + "status": "pending", + "blockedBy": [ + 7 + ] + }, + { + "id": 9, + "subject": "Task 9: Drop dead ConfigDb ScriptedAlarmState table + retire EfAlarmConditionStateStore", + "status": "pending", + "blockedBy": [ + 6 + ] + }, + { + "id": 10, + "subject": "Task 10: Rig \u2014 driver-only site nodes lose ConfigDb string", + "status": "pending", + "blockedBy": [ + 7 + ] + }, + { + "id": 11, + "subject": "Task 11: Live gate", + "status": "pending", + "blockedBy": [ + 8, + 9, + 10 + ] + } ], "lastUpdated": "2026-07-23" } From 956582727522742d8fb4f24443eb38391515af65 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 11:55:43 -0400 Subject: [PATCH 06/18] feat(mesh-phase4): retire DB-reachability from ServiceLevel on DB-less nodes Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- ...6-07-23-mesh-phase4-cut-driver-configdb.md | 16 ++ .../OpcUa/OpcUaPublishActor.cs | 52 ++++++- .../ServiceCollectionExtensions.cs | 29 +++- .../OpcUa/OpcUaPublishActorTests.cs | 137 ++++++++++++++++++ 4 files changed, 223 insertions(+), 11 deletions(-) diff --git a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md index 7c34c9db..8dce877e 100644 --- a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md +++ b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md @@ -276,6 +276,22 @@ apply through the fetcher), so the change is defensive nullability, not new cont 1920/2096/2588 is inside a Direct-mode-only path before guarding — if any is on the shared apply path, that is a Phase-3 gap to surface, not silently guard. +**Two specific sites the Task-2 code review surfaced (MUST be handled here):** +- **Remove the interim `dbFactory!` at `ServiceCollectionExtensions.cs:~470`** (added in Task 2 to preserve + compilation) — once `DriverHostActor._dbFactory` is nullable, pass the plain (nullable) `dbFactory`. +- **`UpsertNodeDeploymentState` (2613-2643) is called UNCONDITIONALLY on the FetchAndCache apply path** + (`BeginFetchAndCacheApply`/`HandleFetchedForApply`, ~1716/1723/1774/1785/1801). Its try/catch swallows the + null-factory NRE and logs a misleading "transient DB" warning on every deploy on a driver-only node. The + `_dbFactory is not null` guard fixes this. +- **`SpawnScriptedAlarmHost` (called unconditionally from PreStart ~524) constructs + `new EfAlarmConditionStateStore(_dbFactory, ...)` at ~587-588** — with a null factory every alarm + Load/Save NREs (swallowed → scripted-alarm state silently dead on driver-only nodes). **This is fixed by + Task 6** (which swaps in the LocalDb-backed store), NOT by a null guard — do NOT leave the Ef store + constructed with a null factory. If Task 6 lands after this task, add a temporary `_dbFactory is not null` + guard around the Ef-store construction so a DB-less node skips it cleanly until Task 6 wires the LocalDb + store; the two tasks together must leave a DB-less node with a WORKING (LocalDb) condition store, not a + skipped one. + **Step 1: Write failing tests** — construct a `DriverHostActor` with `dbFactory: null`, `fetchAndCacheMode: true`, a fake fetcher + LocalDb cache (mirror `DriverHostActorFetchAndCacheTests`): (a) a full dispatch → Applied ack, **no** NullReferenceException from a factory deref; (b) bootstrap diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs index c96faec6..7cf523c9 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs @@ -103,6 +103,11 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers private readonly IDbContextFactory? _dbFactory; private readonly AddressSpaceApplier? _applier; private readonly IActorRef? _dbHealthProbe; + /// Per-cluster mesh Phase 4: this is a driver-only node with NO ConfigDb (LocalDb is its + /// config store). There is no DB to probe, so the ServiceLevel DB-reachability input is retired — + /// DbReachable is fed constant true and only snapshot staleness can demote. See + /// . + private readonly bool _dbLess; private readonly TimeSpan _staleWindow; private readonly TimeSpan _probeFreshnessWindow; private readonly TimeSpan _healthTickInterval; @@ -153,6 +158,9 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers /// The period between self-driven DB-health refresh ticks (each /// Asks for its cached status); defaults to 5 seconds. No timer is /// started when is null. + /// Per-cluster mesh Phase 4: this is a driver-only node with no ConfigDb, so the + /// ServiceLevel DB-reachability input is retired (DbReachable treated constant-true, only + /// staleness demotes). No is wired on such a node. /// The configured for creating this actor, pinned to the /// dispatcher. public static Props Props( @@ -164,7 +172,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers IActorRef? dbHealthProbe = null, TimeSpan? staleWindow = null, TimeSpan? probeFreshnessWindow = null, - TimeSpan? healthTickInterval = null) => + TimeSpan? healthTickInterval = null, + bool dbLess = false) => Akka.Actor.Props.Create(() => new OpcUaPublishActor( sink ?? NullOpcUaAddressSpaceSink.Instance, serviceLevel ?? NullServiceLevelPublisher.Instance, @@ -175,7 +184,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers dbHealthProbe, staleWindow, probeFreshnessWindow, - healthTickInterval)).WithDispatcher(DispatcherId); + healthTickInterval, + dbLess)).WithDispatcher(DispatcherId); /// Test-only Props that omits the pinned-dispatcher requirement and skips the /// DPS subscribe so unit tests can spin up the actor on a vanilla TestKit cluster. @@ -195,6 +205,9 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers /// The period between self-driven DB-health refresh ticks (each /// Asks for its cached status); defaults to 5 seconds. No timer is /// started when is null. + /// Per-cluster mesh Phase 4: this is a driver-only node with no ConfigDb, so the + /// ServiceLevel DB-reachability input is retired (DbReachable treated constant-true, only + /// staleness demotes). No is wired on such a node. /// The configured for creating this actor in tests, without dispatcher /// pinning or the DPS subscribe. public static Props PropsForTests( @@ -207,7 +220,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers IActorRef? dbHealthProbe = null, TimeSpan? staleWindow = null, TimeSpan? probeFreshnessWindow = null, - TimeSpan? healthTickInterval = null) => + TimeSpan? healthTickInterval = null, + bool dbLess = false) => Akka.Actor.Props.Create(() => new OpcUaPublishActor( sink ?? NullOpcUaAddressSpaceSink.Instance, serviceLevel ?? NullServiceLevelPublisher.Instance, @@ -218,7 +232,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers dbHealthProbe, staleWindow, probeFreshnessWindow, - healthTickInterval)); + healthTickInterval, + dbLess)); /// Initializes a new instance of the class. /// The OPC UA address space sink. @@ -237,6 +252,9 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers /// The period between self-driven DB-health refresh ticks (each /// Asks for its cached status); defaults to 5 seconds. No timer is /// started when is null. + /// Per-cluster mesh Phase 4: this is a driver-only node with no ConfigDb, so the + /// ServiceLevel DB-reachability input is retired (DbReachable treated constant-true, only + /// staleness demotes). No is wired on such a node. public OpcUaPublishActor( IOpcUaAddressSpaceSink sink, IServiceLevelPublisher serviceLevel, @@ -247,7 +265,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers IActorRef? dbHealthProbe = null, TimeSpan? staleWindow = null, TimeSpan? probeFreshnessWindow = null, - TimeSpan? healthTickInterval = null) + TimeSpan? healthTickInterval = null, + bool dbLess = false) { _sink = sink; _serviceLevel = serviceLevel; @@ -256,6 +275,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers _dbFactory = dbFactory; _applier = applier; _dbHealthProbe = dbHealthProbe; + _dbLess = dbLess; _staleWindow = staleWindow ?? TimeSpan.FromSeconds(30); _probeFreshnessWindow = probeFreshnessWindow ?? TimeSpan.FromSeconds(30); _healthTickInterval = healthTickInterval ?? TimeSpan.FromSeconds(5); @@ -635,6 +655,28 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers return; } + // Per-cluster mesh Phase 4: on a driver-only (DB-less) node there is no ConfigDb to probe — + // LocalDb IS the config store and is in-process, so it can never be "unreachable". Retire the + // DB-reachability input by feeding DbReachable=true CONSTANT and deriving staleness from the + // snapshot age ALONE (no DB-health term). This is the survive-alone posture and it is + // client-visible: a healthy DB-less node must publish 240 (240+10=250 as Primary) even with + // central down, so clients keep steering to it — the old DbReachable=false would have computed + // the (false,true,false) → 0 arm and driven clients AWAY from a node serving perfectly. + // Staleness ("running behind on config") still demotes to 200. The Detached guard above still + // wins, so a DB-less Detached node stays 0. + if (_dbLess) + { + var utcNow = DateTime.UtcNow; + var dbLessInputs = new NodeHealthInputs( + MemberState: SafeSelfStatus(), + DbReachable: true, + OpcUaProbeOk: OpcUaProbeOk(), + Stale: (utcNow - entry.AsOfUtc) > _staleWindow, + IsDriverPrimary: entry.IsDriverPrimary); + Self.Tell(new ServiceLevelChanged(ServiceLevelCalculator.Compute(dbLessInputs))); + return; + } + // Legacy / back-compat seam: with no DB-health probe wired (or before the first sample // arrives) fall back to the old role-only switch. This preserves historical behaviour and // is the bootstrap value until the first DbHealthStatus lands. diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs index e374671a..5a8ae862 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs @@ -326,10 +326,19 @@ public static class ServiceCollectionExtensions // (durable AVEVA sink is infra-gated); a deployment binding a real IHistoryWriter overrides. var historyWriter = resolver.GetService() ?? NullHistoryWriter.Instance; - var dbHealth = system.ActorOf( - DbHealthProbeActor.Props(dbFactory), - DbHealthProbeActorName); - registry.Register(dbHealth); + // Per-cluster mesh Phase 4: a driver-only node has NO ConfigDb (LocalDb is its config + // store), so dbFactory is null and there is nothing to probe. Skip spawning the probe + // entirely — spawning it with a null factory would NRE on the first SELECT 1 — and do not + // register its marker key. OpcUaPublishActor then runs in dbLess mode (DbReachable retired, + // treated constant-true) so a healthy DB-less node still publishes full ServiceLevel. + IActorRef? dbHealth = null; + if (dbFactory is not null) + { + dbHealth = system.ActorOf( + DbHealthProbeActor.Props(dbFactory), + DbHealthProbeActorName); + registry.Register(dbHealth); + } // Dependency mux must be spawned before DriverHostActor so the host can forward // AttributeValuePublished into it from the very first driver spawn. @@ -436,7 +445,8 @@ public static class ServiceCollectionExtensions localNode: roleInfo.LocalNode, dbFactory: dbFactory, applier: applier, - dbHealthProbe: dbHealth), + dbHealthProbe: dbHealth, + dbLess: dbFactory is null), OpcUaPublishActorName); registry.Register(publishActor); @@ -452,7 +462,14 @@ public static class ServiceCollectionExtensions // ackRouter: under ClusterClient mode the node comm actor relays ApplyAcks across // the boundary; under Dps it stays null and the host publishes on the // deployment-acks topic exactly as before. - DriverHostActor.Props(dbFactory, roleInfo.LocalNode, + // dbFactory! — KNOWN INTERIM. A driver-only node genuinely passes a null factory here + // (ConfigDb is now gated on hasAdmin, and driver-only nodes are forced to FetchAndCache), + // so DriverHostActor's ConfigDb-touching paths — scripted-alarm state persistence + // (EfAlarmConditionStateStore) and UpsertNodeDeploymentState — fault-and-swallow until + // Task 4 makes the factory nullable + guards those sites (and removes this !) and Task 6 + // re-homes the alarm store to LocalDb. Inert in practice: no driver-only node is + // configured to boot until the Task 10 rig change. + DriverHostActor.Props(dbFactory!, roleInfo.LocalNode, ackRouter: useClusterClient ? nodeComm : null, driverFactory: driverFactory, localRoles: roleInfo.LocalRoles, dependencyMux: mux, diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/OpcUa/OpcUaPublishActorTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/OpcUa/OpcUaPublishActorTests.cs index 2a2c03ac..e10e9329 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/OpcUa/OpcUaPublishActorTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/OpcUa/OpcUaPublishActorTests.cs @@ -580,6 +580,143 @@ public sealed class OpcUaPublishActorTests : RuntimeActorTestBase AwaitAssert(() => publisher.Levels.ShouldContain((byte)250), TimeSpan.FromSeconds(2)); } + // ── Per-cluster mesh Phase 4: DB-less driver-only nodes ───────────────────────────────────── + // A driver-only node has no ConfigDb (LocalDb is its config store), so there is no DB to probe. + // Such nodes run with dbHealthProbe: null + dbLess: true and must still publish the full + // survive-alone ServiceLevel (240 follower / 250 Primary) — DbReachable is treated as a + // constant true because the in-process LocalDb can never be "unreachable"; only snapshot + // staleness demotes. + + /// DB-less, member Up, snapshot fresh, follower (non-Primary) → 240. No probe wired, + /// no DbHealthStatus ever arrives; the DB-less branch must still reach full service. + [Fact] + public void DbLess_healthy_follower_publishes_240() + { + var publisher = new RecordingPublisher(); + var local = NodeId.Parse("db-less-follower"); + var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests( + serviceLevel: publisher, localNode: local, dbLess: true, + staleWindow: TimeSpan.FromSeconds(30))); + + actor.Tell(new RedundancyStateChanged( + Nodes: new[] + { + new NodeRedundancyState(local, RedundancyRole.Secondary, + IsClusterLeader: false, IsDriverPrimary: false, DateTime.UtcNow), + }, + CorrelationId.NewId())); + + AwaitAssert(() => publisher.Levels.ShouldContain((byte)240), + duration: TimeSpan.FromMilliseconds(500)); + } + + /// DB-less, member Up, snapshot fresh, Primary → 250 (basis 240 + 10 Primary bonus). + [Fact] + public void DbLess_healthy_primary_publishes_250() + { + var publisher = new RecordingPublisher(); + var local = NodeId.Parse("db-less-primary"); + var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests( + serviceLevel: publisher, localNode: local, dbLess: true, + staleWindow: TimeSpan.FromSeconds(30))); + + actor.Tell(new RedundancyStateChanged( + Nodes: new[] + { + new NodeRedundancyState(local, RedundancyRole.Primary, + IsClusterLeader: true, IsDriverPrimary: true, DateTime.UtcNow), + }, + CorrelationId.NewId())); + + AwaitAssert(() => publisher.Levels.ShouldContain((byte)250), + duration: TimeSpan.FromMilliseconds(500)); + } + + /// DB-less, snapshot stale (entry.AsOfUtc older than the stale window) → 200. Staleness + /// is derived from snapshot age ALONE (no DB-health term), so a DB-less node still demotes to 200 + /// when it is running behind on config. + [Fact] + public void DbLess_stale_snapshot_publishes_200() + { + var publisher = new RecordingPublisher(); + var local = NodeId.Parse("db-less-stale"); + var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests( + serviceLevel: publisher, localNode: local, dbLess: true, + staleWindow: TimeSpan.FromSeconds(2))); + + actor.Tell(new RedundancyStateChanged( + Nodes: new[] + { + new NodeRedundancyState(local, RedundancyRole.Secondary, + IsClusterLeader: false, IsDriverPrimary: false, + DateTime.UtcNow - TimeSpan.FromMinutes(1)), + }, + CorrelationId.NewId())); + + AwaitAssert(() => publisher.Levels.ShouldContain((byte)200), + duration: TimeSpan.FromMilliseconds(500)); + } + + /// DB-less, Detached → 0. The Detached / missing-entry guard runs ABOVE the DB-less + /// branch, so a DB-less node that detaches still fails closed to 0. Goes healthy (250) first so + /// the transition down to 0 is observable (not dedup'd against the initial 0). + [Fact] + public void DbLess_detached_publishes_0() + { + var publisher = new RecordingPublisher(); + var local = NodeId.Parse("db-less-detached"); + var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests( + serviceLevel: publisher, localNode: local, dbLess: true, + staleWindow: TimeSpan.FromSeconds(30))); + + actor.Tell(new RedundancyStateChanged( + Nodes: new[] + { + new NodeRedundancyState(local, RedundancyRole.Primary, + IsClusterLeader: true, IsDriverPrimary: true, DateTime.UtcNow), + }, + CorrelationId.NewId())); + AwaitAssert(() => publisher.Levels.ShouldContain((byte)250), + duration: TimeSpan.FromMilliseconds(500)); + + actor.Tell(new RedundancyStateChanged( + Nodes: new[] + { + new NodeRedundancyState(local, RedundancyRole.Detached, + IsClusterLeader: false, IsDriverPrimary: false, DateTime.UtcNow), + }, + CorrelationId.NewId())); + + AwaitAssert(() => publisher.Levels.ShouldContain((byte)0), + duration: TimeSpan.FromMilliseconds(500)); + } + + /// Regression: a DB-BACKED node (probe wired + DbHealthStatus reachable, dbLess default + /// false) is unchanged — it still computes 240 via the calculator path. Proves the new DB-less + /// branch did not disturb the existing DB-backed seam. + [Fact] + public void DbBacked_node_unchanged_still_publishes_240() + { + var publisher = new RecordingPublisher(); + var local = NodeId.Parse("db-backed-node"); + var probe = Sys.ActorOf(Akka.Actor.Props.Create(() => + new StubDbHealth(new DbHealthProbeActor.DbHealthStatus(true, DateTime.UtcNow, null)))); + var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests( + serviceLevel: publisher, localNode: local, dbHealthProbe: probe)); + + actor.Tell(new DbHealthProbeActor.DbHealthStatus(true, DateTime.UtcNow, null)); + actor.Tell(new RedundancyStateChanged( + Nodes: new[] + { + new NodeRedundancyState(local, RedundancyRole.Secondary, + IsClusterLeader: false, IsDriverPrimary: false, DateTime.UtcNow), + }, + CorrelationId.NewId())); + + AwaitAssert(() => publisher.Levels.ShouldContain((byte)240), + duration: TimeSpan.FromMilliseconds(500)); + } + /// Stub DB-health probe actor that answers /// with a fixed status, so the calculator path is deterministic without a real timer. private sealed class StubDbHealth : Akka.Actor.ReceiveActor From 34b613d9426eaba0315e4beff64980573cacc5d2 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 12:07:42 -0400 Subject: [PATCH 07/18] docs(mesh-phase4): Task 2 done; fold review findings into Task 4 brief Task 2 code review surfaced two unguarded ConfigDb sites reachable on a driver-only node once Tasks 0+1 landed (SpawnScriptedAlarmHost's Ef store, unconditional UpsertNodeDeploymentState). Task 4 brief now names them + the dbFactory! removal; Task 6 re-homes the alarm store to LocalDb. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- ...2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json index 32d94fe7..03e3fb5b 100644 --- a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json +++ b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json @@ -28,10 +28,12 @@ { "id": 2, "subject": "Task 2: DbHealthProbeActor \u2014 no spawn on driver-only; DbReachable=true constant", - "status": "pending", + "status": "completed", "blockedBy": [ 1 - ] + ], + "commit": "9565827", + "result": "43 tests; spec-compliant + code-review (comment fixed to honest interim; functional fix = Task 4/6)" }, { "id": 3, From 4f4cdd05ec0153f8dd86984e56f805d4822687fb Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 12:19:44 -0400 Subject: [PATCH 08/18] feat(mesh-phase4): LocalDb alarm-condition-state store (replicated, pair-local) Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- .../AlarmConditionStateSchema.cs | 82 +++++ .../Configuration/LocalDbSetup.cs | 7 +- .../LocalDbAlarmConditionStateStore.cs | 264 ++++++++++++++ .../AlarmConditionStateSchemaTests.cs | 46 +++ .../LocalDbSetupTests.cs | 14 +- .../LocalDbWiringTests.cs | 2 +- .../LocalDbAlarmConditionStateStoreTests.cs | 344 ++++++++++++++++++ 7 files changed, 754 insertions(+), 5 deletions(-) create mode 100644 src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmConditionStateSchema.cs create mode 100644 src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/LocalDbAlarmConditionStateStore.cs create mode 100644 tests/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.Tests/AlarmConditionStateSchemaTests.cs create mode 100644 tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ScriptedAlarms/LocalDbAlarmConditionStateStoreTests.cs diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmConditionStateSchema.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmConditionStateSchema.cs new file mode 100644 index 00000000..38b3acf5 --- /dev/null +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmConditionStateSchema.cs @@ -0,0 +1,82 @@ +using Microsoft.Data.Sqlite; + +namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian; + +/// +/// DDL for the Part 9 scripted-alarm condition state: one row per alarm identity holding the +/// operator-supplied state (Enabled / Acked / Confirmed / Shelving) plus the ack/confirm audit +/// trail and comment history. +/// +/// +/// +/// Replaces the central config DB's ScriptedAlarmState table as the node-local home +/// for this state (per-cluster mesh Phase 4, which cuts the ConfigDb from driver-only +/// nodes). Living in the consolidated LocalDb file is what lets the state replicate to the +/// redundant pair peer, exactly like the alarm store-and-forward buffer — see +/// , which this mirrors. +/// +/// +/// Deliberately depends on nothing but so it can be applied +/// to any connection — the host's LocalDbSetup.OnReady in production, and a bare +/// connection in a test — without dragging the DI graph along. +/// +/// +/// The primary key is the alarm identity, and Save is an unconditional upsert onto it. +/// Convergence across the pair is last-writer-wins over the primary key, handled by the +/// LocalDb replication HLC — so the store stacks no application-level last-write-wins on top, +/// the same discipline the alarm-sf sink and the deployment pointer follow. +/// +/// +/// ActiveState has no column — it is re-derived from the live predicate on startup, +/// so persisting it would only risk operators seeing a stale Active on restart. Likewise +/// LastActiveUtc / LastClearedUtc re-derive alongside it and get no columns. +/// LastTransitionUtc is carried by updated_at_utc — the row-write timestamp is +/// the last transition. Timestamps are round-trip ("O") TEXT so lexicographic order is +/// chronological; nullable timestamps and audit fields are stored as NULL. +/// +/// +public static class AlarmConditionStateSchema +{ + /// Table holding one condition-state row per scripted-alarm identity. + public const string StateTable = "alarm_condition_state"; + + /// + /// Creates the state table if it does not already exist. Idempotent. + /// + /// + /// An already-open connection. ILocalDb.CreateConnection() hands out open, + /// pragma-configured connections — do not call Open() on one. + /// + public static void Apply(SqliteConnection connection) + { + ArgumentNullException.ThrowIfNull(connection); + + using var cmd = connection.CreateCommand(); + + // The four *_state columns are always written to a mapped, non-null string (Enabled/Disabled, + // Acknowledged/Unacknowledged, Confirmed/Unconfirmed, Unshelved/OneShotShelved/TimedShelved), + // so they are NOT NULL. comments_json is never null either — an empty trail is the literal + // "[]". The audit user/comment/utc columns and shelving_expires_utc are NULL when the alarm + // has no such history yet. updated_at_utc carries LastTransitionUtc (there is no dedicated + // transition column). + cmd.CommandText = """ + CREATE TABLE IF NOT EXISTS alarm_condition_state ( + scripted_alarm_id TEXT NOT NULL PRIMARY KEY, + enabled_state TEXT NOT NULL, + acked_state TEXT NOT NULL, + confirmed_state TEXT NOT NULL, + shelving_state TEXT NOT NULL, + shelving_expires_utc TEXT NULL, + last_ack_user TEXT NULL, + last_ack_comment TEXT NULL, + last_ack_utc TEXT NULL, + last_confirm_user TEXT NULL, + last_confirm_comment TEXT NULL, + last_confirm_utc TEXT NULL, + comments_json TEXT NOT NULL, + updated_at_utc TEXT NOT NULL + ); + """; + cmd.ExecuteNonQuery(); + } +} diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs index ecbd6c8a..cd70e38c 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs @@ -6,8 +6,9 @@ using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache; namespace ZB.MOM.WW.OtOpcUa.Host.Configuration; /// -/// The onReady callback handed to AddZbLocalDb: creates the deployment-cache and -/// alarm store-and-forward tables and opts them into replication. +/// The onReady callback handed to AddZbLocalDb: creates the deployment-cache, +/// alarm store-and-forward, and scripted-alarm condition-state tables and opts them into +/// replication. /// /// /// @@ -56,11 +57,13 @@ public static class LocalDbSetup { DeploymentCacheSchema.Apply(connection); AlarmSfSchema.Apply(connection); + AlarmConditionStateSchema.Apply(connection); } db.RegisterReplicated(DeploymentCacheSchema.ArtifactsTable); db.RegisterReplicated(DeploymentCacheSchema.PointerTable); db.RegisterReplicated(AlarmSfSchema.EventsTable); + db.RegisterReplicated(AlarmConditionStateSchema.StateTable); // LAST, and only here. This is the one call in OnReady that writes rows. AlarmSfLegacyMigrator.Migrate(db, configuration); diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/LocalDbAlarmConditionStateStore.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/LocalDbAlarmConditionStateStore.cs new file mode 100644 index 00000000..a47b7e65 --- /dev/null +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/LocalDbAlarmConditionStateStore.cs @@ -0,0 +1,264 @@ +using System.Collections.Immutable; +using System.Globalization; +using System.Text.Json; +using Microsoft.Data.Sqlite; +using Microsoft.Extensions.Logging; +using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian; +using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms; + +namespace ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms; + +/// +/// Node-local backed by the replicated +/// table in the node's consolidated LocalDb. +/// Maps the full Part 9 — Enabled / Acked / Confirmed / +/// Shelving + the ack/confirm audit trail + operator comments. +/// +/// +/// +/// The LocalDb replacement for EfAlarmConditionStateStore (which persisted to the +/// central config DB's ScriptedAlarmState table). Per-cluster mesh Phase 4 cuts the +/// ConfigDb from driver-only nodes, so this state moves into the same replicated LocalDb +/// file the Phase-2 alarm store-and-forward buffer already lives in — a node's condition +/// state then mirrors to its redundant pair peer instead of depending on central SQL. +/// +/// +/// The round-trip is byte-identical to the EF store — the enum↔string mappers and the +/// comment serialization below are duplicated verbatim from it (it is deleted in a later +/// task; the ported round-trip tests prove parity). The persistence rules carry over +/// unchanged: +/// +/// +/// ActiveState is NOT persisted — there is no column; on it is +/// restored as and the engine re-derives it from the +/// live predicate. LastTransitionUtc ↔ updated_at_utc: no dedicated transition column, +/// so the last transition rides the row-write timestamp. LastActiveUtc / LastClearedUtc +/// have no columns and default to null on load. +/// serializes to/from comments_json; an empty list round-trips as "[]". +/// +/// +/// Save is a single unconditional PK upsert. No application-level last-write-wins or +/// HLC comparison is layered on top — the LocalDb replication HLC handles convergence, and an +/// unconditional upsert keyed by the primary key is HLC-safe (the same discipline the +/// alarm-sf sink and the deployment pointer follow). +/// +/// +public sealed class LocalDbAlarmConditionStateStore : IAlarmStateStore +{ + private static readonly JsonSerializerOptions JsonOptions = new(JsonSerializerDefaults.Web); + + // Fixed column order shared by every read — the reader map below indexes by these ordinals. + private const string SelectColumns = + "scripted_alarm_id, enabled_state, acked_state, confirmed_state, shelving_state, " + + "shelving_expires_utc, last_ack_user, last_ack_comment, last_ack_utc, " + + "last_confirm_user, last_confirm_comment, last_confirm_utc, comments_json, updated_at_utc"; + + private readonly ILocalDb _db; + private readonly ILogger _logger; + + /// Initializes a new instance of the class. + /// + /// The node's local database. Its table + /// must already exist and be registered for replication — the host does both in + /// LocalDbSetup.OnReady, before any consumer can resolve this store. + /// + /// The logger instance. + public LocalDbAlarmConditionStateStore(ILocalDb db, ILogger logger) + { + _db = db ?? throw new ArgumentNullException(nameof(db)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + } + + /// + public async Task LoadAsync(string alarmId, CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(alarmId); + var rows = await _db.QueryAsync( + $"SELECT {SelectColumns} FROM alarm_condition_state WHERE scripted_alarm_id = @Id", + MapRow, + new { Id = alarmId }, + ct).ConfigureAwait(false); + return rows.Count > 0 ? rows[0] : null; + } + + /// + public async Task> LoadAllAsync(CancellationToken ct) + { + var rows = await _db.QueryAsync( + $"SELECT {SelectColumns} FROM alarm_condition_state", + MapRow, + parameters: null, + ct).ConfigureAwait(false); + return rows; + } + + /// + public async Task SaveAsync(AlarmConditionState state, CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(state); + + // A single unconditional upsert keyed on the alarm identity. Convergence across the pair is + // last-writer-wins over the primary key, handled by the LocalDb replication HLC — no + // app-level LWW here (see the class remarks). + await _db.ExecuteAsync( + """ + INSERT INTO alarm_condition_state + (scripted_alarm_id, enabled_state, acked_state, confirmed_state, shelving_state, + shelving_expires_utc, last_ack_user, last_ack_comment, last_ack_utc, + last_confirm_user, last_confirm_comment, last_confirm_utc, comments_json, updated_at_utc) + VALUES + (@Id, @Enabled, @Acked, @Confirmed, @Shelving, + @ShelvingExpires, @LastAckUser, @LastAckComment, @LastAckUtc, + @LastConfirmUser, @LastConfirmComment, @LastConfirmUtc, @CommentsJson, @UpdatedAtUtc) + ON CONFLICT(scripted_alarm_id) DO UPDATE SET + enabled_state = excluded.enabled_state, + acked_state = excluded.acked_state, + confirmed_state = excluded.confirmed_state, + shelving_state = excluded.shelving_state, + shelving_expires_utc = excluded.shelving_expires_utc, + last_ack_user = excluded.last_ack_user, + last_ack_comment = excluded.last_ack_comment, + last_ack_utc = excluded.last_ack_utc, + last_confirm_user = excluded.last_confirm_user, + last_confirm_comment = excluded.last_confirm_comment, + last_confirm_utc = excluded.last_confirm_utc, + comments_json = excluded.comments_json, + updated_at_utc = excluded.updated_at_utc + """, + new + { + Id = state.AlarmId, + Enabled = MapEnabledToColumn(state.Enabled), + Acked = MapAckedToColumn(state.Acked), + Confirmed = MapConfirmedToColumn(state.Confirmed), + Shelving = MapShelvingToColumn(state.Shelving.Kind), + ShelvingExpires = FormatNullable(state.Shelving.UnshelveAtUtc), + LastAckUser = state.LastAckUser, + LastAckComment = state.LastAckComment, + LastAckUtc = FormatNullable(state.LastAckUtc), + LastConfirmUser = state.LastConfirmUser, + LastConfirmComment = state.LastConfirmComment, + LastConfirmUtc = FormatNullable(state.LastConfirmUtc), + CommentsJson = SerializeComments(state.Comments), + // No dedicated transition column — persist LastTransitionUtc into updated_at_utc. + UpdatedAtUtc = Format(state.LastTransitionUtc), + }, + ct).ConfigureAwait(false); + + _logger.LogTrace("Persisted alarm-condition state for {AlarmId}", state.AlarmId); + } + + /// + public async Task RemoveAsync(string alarmId, CancellationToken ct) + { + ArgumentNullException.ThrowIfNull(alarmId); + await _db.ExecuteAsync( + "DELETE FROM alarm_condition_state WHERE scripted_alarm_id = @Id", + new { Id = alarmId }, + ct).ConfigureAwait(false); + } + + private static AlarmConditionState MapRow(SqliteDataReader r) => new( + AlarmId: r.GetString(0), + Enabled: string.Equals(r.GetString(1), "Disabled", StringComparison.Ordinal) + ? AlarmEnabledState.Disabled + : AlarmEnabledState.Enabled, // unknown string → Enabled (safe default) + // Active is not persisted — the engine re-derives it from the predicate at startup. + Active: AlarmActiveState.Inactive, + Acked: string.Equals(r.GetString(2), "Acknowledged", StringComparison.Ordinal) + ? AlarmAckedState.Acknowledged + : AlarmAckedState.Unacknowledged, // unknown string → Unacknowledged (safe default) + Confirmed: string.Equals(r.GetString(3), "Confirmed", StringComparison.Ordinal) + ? AlarmConfirmedState.Confirmed + : AlarmConfirmedState.Unconfirmed, // unknown string → Unconfirmed (safe default) + Shelving: new ShelvingState(MapShelvingFromColumn(r.GetString(4)), ParseNullable(r, 5)), + // No transition column — updated_at_utc carries the last transition timestamp. + LastTransitionUtc: Parse(r.GetString(13)), + // LastActiveUtc / LastClearedUtc have no columns — they re-derive with Active, so null on load. + LastActiveUtc: null, + LastClearedUtc: null, + LastAckUtc: ParseNullable(r, 8), + LastAckUser: r.IsDBNull(6) ? null : r.GetString(6), + LastAckComment: r.IsDBNull(7) ? null : r.GetString(7), + LastConfirmUtc: ParseNullable(r, 11), + LastConfirmUser: r.IsDBNull(9) ? null : r.GetString(9), + LastConfirmComment: r.IsDBNull(10) ? null : r.GetString(10), + Comments: DeserializeComments(r.GetString(12))); + + // Timestamps are stored as round-trip ("O") TEXT so lexicographic order stays chronological and + // the DateTimeKind.Utc round-trips exactly. + private static string Format(DateTime value) => value.ToString("O", CultureInfo.InvariantCulture); + + private static string? FormatNullable(DateTime? value) => value?.ToString("O", CultureInfo.InvariantCulture); + + private static DateTime Parse(string value) => + DateTime.Parse(value, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind); + + private static DateTime? ParseNullable(SqliteDataReader r, int ordinal) => + r.IsDBNull(ordinal) ? null : Parse(r.GetString(ordinal)); + + private static string MapEnabledToColumn(AlarmEnabledState enabled) + => enabled == AlarmEnabledState.Enabled ? "Enabled" : "Disabled"; + + private static string MapAckedToColumn(AlarmAckedState acked) + => acked == AlarmAckedState.Acknowledged ? "Acknowledged" : "Unacknowledged"; + + private static string MapConfirmedToColumn(AlarmConfirmedState confirmed) + => confirmed == AlarmConfirmedState.Confirmed ? "Confirmed" : "Unconfirmed"; + + private static string MapShelvingToColumn(ShelvingKind kind) => kind switch + { + ShelvingKind.OneShot => "OneShotShelved", + ShelvingKind.Timed => "TimedShelved", + _ => "Unshelved", + }; + + private static ShelvingKind MapShelvingFromColumn(string column) => column switch + { + "OneShotShelved" => ShelvingKind.OneShot, + "TimedShelved" => ShelvingKind.Timed, + _ => ShelvingKind.Unshelved, // unknown string → Unshelved (safe default) + }; + + private static string SerializeComments(ImmutableList comments) + { + if (comments.IsEmpty) return "[]"; + var dtos = comments.Select(c => new CommentDto + { + // AlarmComment.TimestampUtc must be DateTimeKind.Utc for correct ISO-8601 round-trip; + // the engine always creates AlarmComment instances with Utc kind. + TimestampUtc = c.TimestampUtc, + User = c.User, + Kind = c.Kind, + Text = c.Text, + }); + return JsonSerializer.Serialize(dtos, JsonOptions); + } + + private static ImmutableList DeserializeComments(string? json) + { + if (string.IsNullOrWhiteSpace(json)) return ImmutableList.Empty; + var dtos = JsonSerializer.Deserialize>(json, JsonOptions); + if (dtos is null || dtos.Count == 0) return ImmutableList.Empty; + return dtos + .Select(d => new AlarmComment(d.TimestampUtc, d.User ?? string.Empty, d.Kind ?? string.Empty, d.Text ?? string.Empty)) + .ToImmutableList(); + } + + /// Stable on-disk shape for a persisted in comments_json. + private sealed class CommentDto + { + /// When the comment was recorded (UTC). + public DateTime TimestampUtc { get; set; } + + /// Identity of the actor that wrote the comment. + public string? User { get; set; } + + /// Human-readable classification of the comment (Acknowledge, Confirm, …). + public string? Kind { get; set; } + + /// Operator-supplied or engine-generated comment text. + public string? Text { get; set; } + } +} diff --git a/tests/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.Tests/AlarmConditionStateSchemaTests.cs b/tests/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.Tests/AlarmConditionStateSchemaTests.cs new file mode 100644 index 00000000..028bc49f --- /dev/null +++ b/tests/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.Tests/AlarmConditionStateSchemaTests.cs @@ -0,0 +1,46 @@ +using Microsoft.Data.Sqlite; +using Shouldly; +using Xunit; + +namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.Tests; + +/// +/// Verifies creates its table and is idempotent — +/// the DDL depends on nothing but a , so a bare in-memory +/// connection is a faithful harness (no zb_hlc_next() UDF is involved in CREATE TABLE). +/// +[Trait("Category", "Unit")] +public sealed class AlarmConditionStateSchemaTests +{ + private static SqliteConnection OpenInMemory() + { + var connection = new SqliteConnection("Data Source=:memory:"); + connection.Open(); + return connection; + } + + /// A single Apply creates the table. + [Fact] + public void Apply_creates_the_state_table() + { + using var connection = OpenInMemory(); + + AlarmConditionStateSchema.Apply(connection); + + using var cmd = connection.CreateCommand(); + cmd.CommandText = + "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = @Name"; + cmd.Parameters.AddWithValue("@Name", AlarmConditionStateSchema.StateTable); + ((long)cmd.ExecuteScalar()!).ShouldBe(1); + } + + /// Applying twice on the same connection does not throw (CREATE TABLE IF NOT EXISTS). + [Fact] + public void Apply_is_idempotent() + { + using var connection = OpenInMemory(); + + AlarmConditionStateSchema.Apply(connection); + Should.NotThrow(() => AlarmConditionStateSchema.Apply(connection)); + } +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbSetupTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbSetupTests.cs index 3e45a17d..df45f06a 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbSetupTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbSetupTests.cs @@ -46,7 +46,7 @@ public sealed class LocalDbSetupTests : IDisposable } [Fact] - public void OnReady_RegistersExactlyTheThreeReplicatedTables() + public void OnReady_RegistersExactlyTheFourReplicatedTables() { // Both directions are load-bearing. "No fewer" catches a dropped RegisterReplicated call // (that table then never replicates). "No more" catches an accidental registration — @@ -54,7 +54,7 @@ public sealed class LocalDbSetupTests : IDisposable var db = BuildDb(); db.ReplicatedTables.Keys.OrderBy(k => k, StringComparer.Ordinal) - .ShouldBe(["alarm_sf_events", "deployment_artifacts", "deployment_pointer"]); + .ShouldBe(["alarm_condition_state", "alarm_sf_events", "deployment_artifacts", "deployment_pointer"]); } [Fact] @@ -89,6 +89,16 @@ public sealed class LocalDbSetupTests : IDisposable db.ReplicatedTables["alarm_sf_events"].PkColumns.ShouldBe(["id"]); } + [Fact] + public void AlarmConditionState_PkIsTheScriptedAlarmId() + { + // One condition-state row per alarm identity. The pair's two nodes converge on the same row + // under LWW rather than each keeping its own, and Save is an unconditional upsert onto it. + var db = BuildDb(); + + db.ReplicatedTables["alarm_condition_state"].PkColumns.ShouldBe(["scripted_alarm_id"]); + } + [Fact] public async Task AlarmSfEventRows_EnterTheOplog() { diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbWiringTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbWiringTests.cs index 82f6c36e..a2c81f5a 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbWiringTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbWiringTests.cs @@ -95,7 +95,7 @@ public sealed class LocalDbWiringTests : IDisposable // Exact set, both directions — a dropped registration replicates nothing; an extra one costs // three triggers and oplog volume on every write. first.ReplicatedTables.Keys.OrderBy(k => k, StringComparer.Ordinal) - .ShouldBe(["alarm_sf_events", "deployment_artifacts", "deployment_pointer"]); + .ShouldBe(["alarm_condition_state", "alarm_sf_events", "deployment_artifacts", "deployment_pointer"]); } [Fact] diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ScriptedAlarms/LocalDbAlarmConditionStateStoreTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ScriptedAlarms/LocalDbAlarmConditionStateStoreTests.cs new file mode 100644 index 00000000..b2c15696 --- /dev/null +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ScriptedAlarms/LocalDbAlarmConditionStateStoreTests.cs @@ -0,0 +1,344 @@ +using System.Collections.Immutable; +using Microsoft.Data.Sqlite; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; +using Shouldly; +using Xunit; +using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian; +using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms; +using ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms; + +namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.ScriptedAlarms; + +/// +/// Round-trip + upsert + lifecycle coverage for , +/// the LocalDb-backed over the replicated +/// alarm_condition_state table. +/// +/// +/// Ported verbatim from EfAlarmConditionStateStoreTests — the two stores share the +/// contract, so the assertions transfer unchanged and prove +/// parity. Only store construction differs: this suite drives a real temp-file +/// (the library has no in-memory mode, and a bare +/// would lack the zb_hlc_next() UDF the capture triggers +/// call), whereas the EF suite used an in-memory DbContext factory. +/// +public sealed class LocalDbAlarmConditionStateStoreTests : IDisposable +{ + private readonly string _dbPath = + Path.Combine(Path.GetTempPath(), $"otopcua-alarm-state-{Guid.NewGuid():N}.db"); + + private readonly ServiceProvider _provider; + private readonly ILocalDb _db; + + public LocalDbAlarmConditionStateStoreTests() + { + var configuration = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary { ["LocalDb:Path"] = _dbPath }) + .Build(); + + _provider = new ServiceCollection() + .AddZbLocalDb(configuration, db => + { + using (var connection = db.CreateConnection()) + { + AlarmConditionStateSchema.Apply(connection); + } + + db.RegisterReplicated(AlarmConditionStateSchema.StateTable); + }) + .BuildServiceProvider(); + + _db = _provider.GetRequiredService(); + } + + public void Dispose() + { + _provider.Dispose(); + + // Pooled connections keep a handle open past dispose; clearing the pools first is what + // makes the delete actually succeed. + SqliteConnection.ClearAllPools(); + + foreach (var path in new[] { _dbPath, $"{_dbPath}-wal", $"{_dbPath}-shm" }) + { + try { if (File.Exists(path)) File.Delete(path); } catch { /* best effort */ } + } + } + + private LocalDbAlarmConditionStateStore NewStore() + => new(_db, NullLogger.Instance); + + private static AlarmConditionState Sample( + string alarmId, + AlarmActiveState active = AlarmActiveState.Inactive, + ShelvingState? shelving = null, + ImmutableList? comments = null) + { + var t = new DateTime(2026, 06, 10, 12, 00, 00, DateTimeKind.Utc); + return new AlarmConditionState( + AlarmId: alarmId, + Enabled: AlarmEnabledState.Disabled, + Active: active, + Acked: AlarmAckedState.Unacknowledged, + Confirmed: AlarmConfirmedState.Unconfirmed, + Shelving: shelving ?? new ShelvingState(ShelvingKind.Timed, t.AddMinutes(30)), + LastTransitionUtc: t, + LastActiveUtc: t.AddMinutes(-5), + LastClearedUtc: t.AddMinutes(-1), + LastAckUtc: t.AddMinutes(-2), + LastAckUser: "jane", + LastAckComment: "ack-comment", + LastConfirmUtc: t.AddMinutes(-3), + LastConfirmUser: "bob", + LastConfirmComment: "confirm-comment", + Comments: comments ?? ImmutableList.Empty); + } + + /// Save then load round-trips every persisted operator-state + audit field. + [Fact] + public async Task SaveAsync_then_LoadAsync_round_trips_persisted_fields() + { + var store = NewStore(); + var state = Sample("alarm-1"); + + await store.SaveAsync(state, CancellationToken.None); + var loaded = await store.LoadAsync("alarm-1", CancellationToken.None); + + loaded.ShouldNotBeNull(); + loaded.AlarmId.ShouldBe("alarm-1"); + loaded.Enabled.ShouldBe(AlarmEnabledState.Disabled); + loaded.Acked.ShouldBe(AlarmAckedState.Unacknowledged); + loaded.Confirmed.ShouldBe(AlarmConfirmedState.Unconfirmed); + loaded.Shelving.Kind.ShouldBe(ShelvingKind.Timed); + loaded.Shelving.UnshelveAtUtc.ShouldBe(state.Shelving.UnshelveAtUtc); + loaded.LastTransitionUtc.ShouldBe(state.LastTransitionUtc); + loaded.LastAckUtc.ShouldBe(state.LastAckUtc); + loaded.LastAckUser.ShouldBe("jane"); + loaded.LastAckComment.ShouldBe("ack-comment"); + loaded.LastConfirmUtc.ShouldBe(state.LastConfirmUtc); + loaded.LastConfirmUser.ShouldBe("bob"); + loaded.LastConfirmComment.ShouldBe("confirm-comment"); + } + + /// Loading an id that was never saved returns null. + [Fact] + public async Task LoadAsync_for_unknown_id_returns_null() + { + var store = NewStore(); + + var loaded = await store.LoadAsync("never-saved", CancellationToken.None); + + loaded.ShouldBeNull(); + } + + /// Active is not persisted — a saved Active state loads back as Inactive. + [Fact] + public async Task Active_is_ignored_on_load_and_defaults_to_Inactive() + { + var store = NewStore(); + await store.SaveAsync(Sample("alarm-active", active: AlarmActiveState.Active), CancellationToken.None); + + var loaded = await store.LoadAsync("alarm-active", CancellationToken.None); + + loaded.ShouldNotBeNull(); + loaded.Active.ShouldBe(AlarmActiveState.Inactive); + } + + /// LastActiveUtc / LastClearedUtc have no columns and default to null on load. + [Fact] + public async Task Derived_timestamps_default_to_null_on_load() + { + var store = NewStore(); + await store.SaveAsync(Sample("alarm-derived"), CancellationToken.None); + + var loaded = await store.LoadAsync("alarm-derived", CancellationToken.None); + + loaded.ShouldNotBeNull(); + loaded.LastActiveUtc.ShouldBeNull(); + loaded.LastClearedUtc.ShouldBeNull(); + } + + /// Unshelved round-trips with a null UnshelveAtUtc. + [Fact] + public async Task Unshelved_round_trips() + { + var store = NewStore(); + await store.SaveAsync( + Sample("alarm-unshelved", shelving: ShelvingState.Unshelved), + CancellationToken.None); + + var loaded = await store.LoadAsync("alarm-unshelved", CancellationToken.None); + + loaded.ShouldNotBeNull(); + loaded.Shelving.Kind.ShouldBe(ShelvingKind.Unshelved); + loaded.Shelving.UnshelveAtUtc.ShouldBeNull(); + } + + /// OneShot shelving round-trips. + [Fact] + public async Task OneShot_shelving_round_trips() + { + var store = NewStore(); + await store.SaveAsync( + Sample("alarm-oneshot", shelving: new ShelvingState(ShelvingKind.OneShot, null)), + CancellationToken.None); + + var loaded = await store.LoadAsync("alarm-oneshot", CancellationToken.None); + + loaded.ShouldNotBeNull(); + loaded.Shelving.Kind.ShouldBe(ShelvingKind.OneShot); + loaded.Shelving.UnshelveAtUtc.ShouldBeNull(); + } + + /// The append-only comment trail round-trips all four fields per comment. + [Fact] + public async Task Comments_round_trip_all_fields() + { + var store = NewStore(); + var c1 = new AlarmComment( + new DateTime(2026, 06, 10, 10, 00, 00, DateTimeKind.Utc), "jane", "Acknowledge", "checking pump"); + var c2 = new AlarmComment( + new DateTime(2026, 06, 10, 11, 30, 00, DateTimeKind.Utc), "bob", "Confirm", "all clear"); + var comments = ImmutableList.Create(c1, c2); + + await store.SaveAsync(Sample("alarm-comments", comments: comments), CancellationToken.None); + var loaded = await store.LoadAsync("alarm-comments", CancellationToken.None); + + loaded.ShouldNotBeNull(); + loaded.Comments.Count.ShouldBe(2); + loaded.Comments[0].TimestampUtc.ShouldBe(c1.TimestampUtc); + loaded.Comments[0].User.ShouldBe("jane"); + loaded.Comments[0].Kind.ShouldBe("Acknowledge"); + loaded.Comments[0].Text.ShouldBe("checking pump"); + loaded.Comments[1].TimestampUtc.ShouldBe(c2.TimestampUtc); + loaded.Comments[1].User.ShouldBe("bob"); + loaded.Comments[1].Kind.ShouldBe("Confirm"); + loaded.Comments[1].Text.ShouldBe("all clear"); + } + + /// An empty comment list round-trips as an empty list. + [Fact] + public async Task Empty_comments_round_trip() + { + var store = NewStore(); + await store.SaveAsync( + Sample("alarm-empty-comments", comments: ImmutableList.Empty), + CancellationToken.None); + + var loaded = await store.LoadAsync("alarm-empty-comments", CancellationToken.None); + + loaded.ShouldNotBeNull(); + loaded.Comments.ShouldBeEmpty(); + } + + /// An empty comment list persists as the literal "[]" in the column. + [Fact] + public async Task Empty_comments_persist_as_bracket_pair() + { + var store = NewStore(); + await store.SaveAsync( + Sample("alarm-empty-literal", comments: ImmutableList.Empty), + CancellationToken.None); + + using var conn = _db.CreateConnection(); + using var cmd = conn.CreateCommand(); + cmd.CommandText = + "SELECT comments_json FROM alarm_condition_state WHERE scripted_alarm_id = 'alarm-empty-literal'"; + var json = (string?)cmd.ExecuteScalar(); + + json.ShouldBe("[]"); + } + + /// An unknown enabled string loads back as Enabled (safe default). + [Fact] + public async Task Unknown_enum_strings_map_to_safe_defaults() + { + // Write raw rows carrying garbage enum strings to prove the load-side mappers coerce them. + using (var conn = _db.CreateConnection()) + using (var cmd = conn.CreateCommand()) + { + cmd.CommandText = + """ + INSERT INTO alarm_condition_state + (scripted_alarm_id, enabled_state, acked_state, confirmed_state, shelving_state, + comments_json, updated_at_utc) + VALUES + ('garbage', 'nonsense', 'nonsense', 'nonsense', 'nonsense', '[]', '2026-06-10T12:00:00.0000000Z') + """; + cmd.ExecuteNonQuery(); + } + + var loaded = await NewStore().LoadAsync("garbage", CancellationToken.None); + + loaded.ShouldNotBeNull(); + loaded.Enabled.ShouldBe(AlarmEnabledState.Enabled); // unknown → Enabled + loaded.Acked.ShouldBe(AlarmAckedState.Unacknowledged); // unknown → Unacknowledged + loaded.Confirmed.ShouldBe(AlarmConfirmedState.Unconfirmed); // unknown → Unconfirmed + loaded.Shelving.Kind.ShouldBe(ShelvingKind.Unshelved); // unknown → Unshelved + } + + /// Saving the same id twice is an upsert — one row, latest values win. + [Fact] + public async Task SaveAsync_twice_upserts_latest() + { + var store = NewStore(); + var first = Sample("alarm-upsert") with { LastAckUser = "first-user", Enabled = AlarmEnabledState.Enabled }; + var second = Sample("alarm-upsert") with { LastAckUser = "second-user", Enabled = AlarmEnabledState.Disabled }; + + await store.SaveAsync(first, CancellationToken.None); + await store.SaveAsync(second, CancellationToken.None); + + using (var conn = _db.CreateConnection()) + using (var cmd = conn.CreateCommand()) + { + cmd.CommandText = + "SELECT COUNT(*) FROM alarm_condition_state WHERE scripted_alarm_id = 'alarm-upsert'"; + ((long)cmd.ExecuteScalar()!).ShouldBe(1); + } + + var loaded = await store.LoadAsync("alarm-upsert", CancellationToken.None); + loaded.ShouldNotBeNull(); + loaded.LastAckUser.ShouldBe("second-user"); + loaded.Enabled.ShouldBe(AlarmEnabledState.Disabled); + } + + /// LoadAllAsync returns every saved state. + [Fact] + public async Task LoadAllAsync_returns_all_saved_states() + { + var store = NewStore(); + await store.SaveAsync(Sample("a-1"), CancellationToken.None); + await store.SaveAsync(Sample("a-2"), CancellationToken.None); + await store.SaveAsync(Sample("a-3"), CancellationToken.None); + + var all = await store.LoadAllAsync(CancellationToken.None); + + all.Select(s => s.AlarmId).OrderBy(x => x).ShouldBe(new[] { "a-1", "a-2", "a-3" }); + } + + /// RemoveAsync deletes a row so a subsequent load returns null; others survive. + [Fact] + public async Task RemoveAsync_deletes_one_and_LoadAsync_returns_null() + { + var store = NewStore(); + await store.SaveAsync(Sample("keep"), CancellationToken.None); + await store.SaveAsync(Sample("drop"), CancellationToken.None); + + await store.RemoveAsync("drop", CancellationToken.None); + + (await store.LoadAsync("drop", CancellationToken.None)).ShouldBeNull(); + (await store.LoadAsync("keep", CancellationToken.None)).ShouldNotBeNull(); + } + + /// Removing an unknown id is a no-op (does not throw). + [Fact] + public async Task RemoveAsync_for_unknown_id_is_noop() + { + var store = NewStore(); + + await Should.NotThrowAsync(() => store.RemoveAsync("ghost", CancellationToken.None)); + } +} From d9071607474c84ea4d238833a456a6ed8fe47bf4 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 12:39:41 -0400 Subject: [PATCH 09/18] feat(mesh-phase4): DriverHostActor runs ConfigDb-free (LocalDb alarm store, guarded acks) Nullable ConfigDb factory; UpsertNodeDeploymentState no-ops when absent (central persists acks from the ApplyAck); scripted-alarm condition state served from the LocalDb store instead of the ConfigDb-backed Ef store. Combines Phase-4 Tasks 4 + 6. Removes the interim dbFactory! cast. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- ...h-phase4-cut-driver-configdb.md.tasks.json | 4 +- .../Configuration/LocalDbRegistration.cs | 9 + .../Drivers/DriverHostActor.cs | 122 +++++++- .../ServiceCollectionExtensions.cs | 24 +- .../LocalDbWiringTests.cs | 16 ++ .../Drivers/DriverHostActorDbLessTests.cs | 272 ++++++++++++++++++ .../LocalDbAlarmConditionStateStoreTests.cs | 39 +++ 7 files changed, 466 insertions(+), 20 deletions(-) create mode 100644 tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorDbLessTests.cs diff --git a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json index 03e3fb5b..f5228e97 100644 --- a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json +++ b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json @@ -54,7 +54,9 @@ { "id": 5, "subject": "Task 5: alarm_condition_state LocalDb table + LocalDbAlarmConditionStateStore", - "status": "pending" + "status": "completed", + "commit": "4f4cdd05", + "result": "16 new tests; SPEC-COMPLIANT + APPROVED (ON CONFLICT DO UPDATE replication-safe). Fold Kind+null-audit test polish into Task 4/6" }, { "id": 6, diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs index aa7522ec..f8a54971 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs @@ -2,7 +2,9 @@ using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.LocalDb.Replication; +using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms; using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache; +using ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms; namespace ZB.MOM.WW.OtOpcUa.Host.Configuration; @@ -96,6 +98,13 @@ public static class LocalDbRegistration // replicating, and never written to. services.AddSingleton(); + // Per-cluster mesh Phase 4: scripted-alarm condition state served from the replicated LocalDb + // store instead of the ConfigDb-backed EF store, so a driver-only node (no ConfigDb) keeps its + // Part 9 condition state and mirrors it to its pair peer. Driver-role only, alongside the cache — + // admin-only nodes never register LocalDb. WithOtOpcUaRuntimeActors resolves it optionally and + // threads it into DriverHostActor's ScriptedAlarm engine. Singleton to match ILocalDb + the cache. + services.AddSingleton(); + return services; } } diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs index e7065ef5..0c310cc0 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs @@ -97,7 +97,28 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers /// private bool _isRunningFromCache; - private readonly IDbContextFactory _dbFactory; + /// Latches after the first no-op ack-write on a DB-less node so the explanatory Debug line + /// is logged once, not on every apply state transition. + private bool _loggedNoConfigDbAck; + + /// + /// Config database context factory, or on a driver-only node (per-cluster + /// mesh Phase 4). A driver-only node has NO ConfigDb — LocalDb is its config store and central is + /// the ack system of record — so every ConfigDb-touching path here is guarded on this being + /// non-null. Non-null on a fused admin+driver node and in the Direct-mode/EF test harnesses. + /// + private readonly IDbContextFactory? _dbFactory; + + /// + /// Scripted-alarm condition-state store handed to the ScriptedAlarm engine, or + /// when none was wired (per-cluster mesh Phase 4). On a driver-role node + /// this is the replicated LocalDb store (LocalDbAlarmConditionStateStore) so condition + /// state survives without central SQL; when null the actor falls back to an + /// over (legacy admin/Direct + + /// test harnesses), or — when there is no ConfigDb either — skips spawning the alarm host. + /// + private readonly IAlarmStateStore? _alarmStateStore; + private readonly CommonsNodeId _localNode; private readonly IActorRef? _ackRouter; private readonly IDriverFactory _driverFactory; @@ -375,9 +396,14 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers /// Optional Phase 6.1 resilience-invoker factory used to attach an /// to each spawned driver; defaults to /// (pass-through) when not supplied. + /// Per-cluster mesh Phase 4: scripted-alarm condition-state store. On a + /// driver-role node this is the replicated LocalDb store, so condition state persists with no + /// ConfigDb; null falls back to an over + /// (legacy admin/Direct + test harnesses), or skips the alarm host when + /// there is no ConfigDb either. Defaults to null. /// The Akka.NET used to spawn this actor. public static Props Props( - IDbContextFactory dbFactory, + IDbContextFactory? dbFactory, CommonsNodeId localNode, IActorRef? ackRouter = null, IDriverFactory? driverFactory = null, @@ -397,7 +423,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers IRedundancyRoleView? redundancyRoleView = null, string? replicationPeerHost = null, bool fetchAndCacheMode = false, - IDeploymentArtifactFetcher? artifactFetcher = null) => + IDeploymentArtifactFetcher? artifactFetcher = null, + IAlarmStateStore? alarmStateStore = null) => // WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an // expression tree. Six IActorRef? parameters and several interface-typed ones mean a // mis-ordered argument is usually type-compatible and therefore compiles clean, then binds @@ -407,7 +434,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers dbFactory, localNode, ackRouter, driverFactory, localRoles, dependencyMux, opcUaPublishActor, healthPublisher, virtualTagEvaluator, historyWriter, virtualTagHostOverride, loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider, - deploymentArtifactCache, redundancyRoleView, replicationPeerHost, fetchAndCacheMode, artifactFetcher)); + deploymentArtifactCache, redundancyRoleView, replicationPeerHost, fetchAndCacheMode, artifactFetcher, + alarmStateStore)); /// Initializes a new DriverHostActor with the specified dependencies. /// Database context factory for configuration database access. @@ -442,8 +470,12 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers /// When supplied, each successful apply stores its artifact so the node can boot from its /// last-known-good configuration while central SQL is unreachable. Null on admin-only nodes and in /// tests that do not exercise the cache — caching is then simply skipped. + /// Per-cluster mesh Phase 4: scripted-alarm condition-state store. On a + /// driver-role node this is the replicated LocalDb store; null falls back to an + /// over , or skips the alarm + /// host when there is no ConfigDb either. public DriverHostActor( - IDbContextFactory dbFactory, + IDbContextFactory? dbFactory, CommonsNodeId localNode, IActorRef? ackRouter, IDriverFactory? driverFactory = null, @@ -463,13 +495,15 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers IRedundancyRoleView? redundancyRoleView = null, string? replicationPeerHost = null, bool fetchAndCacheMode = false, - IDeploymentArtifactFetcher? artifactFetcher = null) + IDeploymentArtifactFetcher? artifactFetcher = null, + IAlarmStateStore? alarmStateStore = null) { _deploymentArtifactCache = deploymentArtifactCache; _redundancyRoleView = redundancyRoleView; _replicationPeerHost = replicationPeerHost; _fetchAndCacheMode = fetchAndCacheMode; _artifactFetcher = artifactFetcher; + _alarmStateStore = alarmStateStore; _dbFactory = dbFactory; _localNode = localNode; _ackRouter = ackRouter; @@ -583,9 +617,31 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers return; } + // Per-cluster mesh Phase 4: prefer the wired condition-state store (the replicated LocalDb store + // on a driver-role node — it persists with no ConfigDb). Only when no store was wired do we fall + // back to the ConfigDb-backed EF store, and only if there IS a ConfigDb (legacy admin/Direct + + // test harnesses). A driver-only node has neither — nowhere to persist condition state — so it + // skips the alarm host outright rather than constructing an EF store over a null factory that + // would NRE the moment the engine touched it (mirrors the _opcUaPublishActor-null skip above). + IAlarmStateStore store; + if (_alarmStateStore is not null) + { + store = _alarmStateStore; + } + else if (_dbFactory is not null) + { + store = new EfAlarmConditionStateStore( + _dbFactory, _loggerFactory.CreateLogger()); + } + else + { + _log.Debug( + "DriverHost {Node}: skipping ScriptedAlarm host spawn (no condition-state store and no ConfigDb)", + _localNode); + return; + } + var upstream = new DependencyMuxTagUpstreamSource(); - var store = new EfAlarmConditionStateStore( - _dbFactory, _loggerFactory.CreateLogger()); var engine = new ScriptedAlarmEngine( upstream, store, new ScriptLoggerFactory(_scriptRootLogger.Logger), _scriptRootLogger.Logger); _scriptedAlarmHost = Context.ActorOf( @@ -603,6 +659,17 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers return; } + // Belt-and-suspenders (Phase 4): every ConfigDb read below is on the Direct path only — a + // FetchAndCache node already returned via BootstrapFromCache above, and only a FetchAndCache node + // runs DB-less. If a null factory ever reaches a Direct path it is a misconfiguration; enter + // Steady with no revision rather than NRE, and let the first dispatch drive the apply. + if (_dbFactory is null) + { + _log.Info("DriverHost {Node}: no ConfigDb on the Direct bootstrap path; entering Steady (no revision)", _localNode); + Become(Steady); + return; + } + // Read the most-recent NodeDeploymentState for this node; if it's Applied, jump // to Steady with that revision. If Applying (orphan from a crash), discard and replay. // If the DB is unreachable, fall back to Stale and start the reconnect loop. @@ -1914,6 +1981,15 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers /// private byte[]? ReconcileDrivers(DeploymentId deploymentId) { + // Phase 4 belt-and-suspenders: this ConfigDb read is Direct-only (the FetchAndCache apply uses + // ReconcileDriversFromBlob with bytes in hand). A null factory can't read, so return null — the + // caller treats that as "artifact could not be read" (#486) and keeps last-known-good. + if (_dbFactory is null) + { + _log.Warning("DriverHost {Node}: no ConfigDb to load artifact for {Id}; skipping reconcile", _localNode, deploymentId); + return null; + } + byte[] blob; try { @@ -2090,6 +2166,14 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers /// private void PushDesiredSubscriptions(DeploymentId deploymentId) { + // Phase 4 belt-and-suspenders: Direct-only read (the FetchAndCache path calls + // PushDesiredSubscriptionsFromArtifact with bytes in hand). No ConfigDb ⇒ nothing to load. + if (_dbFactory is null) + { + _log.Warning("DriverHost {Node}: no ConfigDb to load artifact for SubscribeBulk ({Id}); skipping", _localNode, deploymentId); + return; + } + byte[] blob; try { @@ -2575,8 +2659,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers { // Defensive: FetchAndCache never enters Stale (BootstrapFromCache always Becomes Steady, so // the retry-db timer that drives this is never started), and its recovery must NOT read - // central SQL. If we somehow land here, leave Steady rather than re-reading Deployments. - if (_fetchAndCacheMode) + // central SQL. A DB-less node (null factory, Phase 4) likewise has nothing to recover from. If + // we somehow land here, leave Steady rather than re-reading Deployments. + if (_fetchAndCacheMode || _dbFactory is null) { Timers.Cancel("retry-db"); Become(Steady); @@ -2612,6 +2697,23 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers private void UpsertNodeDeploymentState(DeploymentId deploymentId, NodeDeploymentStatus status, string? failureReason) { + // Per-cluster mesh Phase 4: a driver-only node has NO ConfigDb. Central is the ack system of + // record — ConfigPublishCoordinator.PersistNodeAck already writes NodeDeploymentState from every + // ApplyAck it receives (and seeds the Applying rows at dispatch) — so this node's own upsert is + // redundant and simply no-ops here with no loss of information. Logged once to keep the reason + // discoverable without spamming every apply transition. + if (_dbFactory is null) + { + if (!_loggedNoConfigDbAck) + { + _loggedNoConfigDbAck = true; + _log.Debug( + "DriverHost {Node}: no ConfigDb — NodeDeploymentState written by central from the ApplyAck", + _localNode); + } + return; + } + try { using var db = _dbFactory.CreateDbContext(); diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs index 5a8ae862..934f8ef1 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs @@ -18,6 +18,7 @@ using ZB.MOM.WW.OtOpcUa.Commons.OpcUa; using ZB.MOM.WW.OtOpcUa.Configuration; using ZB.MOM.WW.OtOpcUa.Core.Abstractions; using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian; +using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms; using ZB.MOM.WW.OtOpcUa.Core.Scripting; using ZB.MOM.WW.OtOpcUa.Core.VirtualTags; using ZB.MOM.WW.OtOpcUa.OpcUaServer; @@ -282,6 +283,12 @@ public static class ServiceCollectionExtensions // harnesses) rather than given a null-object, so DriverHostActor skips caching outright // instead of pretending to cache into a sink that drops everything. var deploymentArtifactCache = resolver.GetService(); + // Per-cluster mesh Phase 4: scripted-alarm condition-state store. Registered by the Host's + // AddOtOpcUaLocalDb on driver-role nodes (the replicated LocalDb store), so condition state + // persists with no ConfigDb; null elsewhere (admin-only graphs, test harnesses) → the actor + // falls back to an EfAlarmConditionStateStore over dbFactory, or skips the alarm host when + // there is no ConfigDb either. + var alarmStateStore = resolver.GetService(); // Per-cluster mesh Phase 3: ConfigSource:Mode selects where this driver reads config. // FetchAndCache pulls the artifact from central over gRPC and reads only the LocalDb cache; // Direct (default) reads central SQL as before. The fetcher is resolved only in FetchAndCache @@ -462,14 +469,12 @@ public static class ServiceCollectionExtensions // ackRouter: under ClusterClient mode the node comm actor relays ApplyAcks across // the boundary; under Dps it stays null and the host publishes on the // deployment-acks topic exactly as before. - // dbFactory! — KNOWN INTERIM. A driver-only node genuinely passes a null factory here - // (ConfigDb is now gated on hasAdmin, and driver-only nodes are forced to FetchAndCache), - // so DriverHostActor's ConfigDb-touching paths — scripted-alarm state persistence - // (EfAlarmConditionStateStore) and UpsertNodeDeploymentState — fault-and-swallow until - // Task 4 makes the factory nullable + guards those sites (and removes this !) and Task 6 - // re-homes the alarm store to LocalDb. Inert in practice: no driver-only node is - // configured to boot until the Task 10 rig change. - DriverHostActor.Props(dbFactory!, roleInfo.LocalNode, + // dbFactory is nullable (Phase 4): a driver-only node passes null here — ConfigDb is + // gated on hasAdmin and driver-only nodes are forced to FetchAndCache. DriverHostActor + // guards every ConfigDb-touching path on it (UpsertNodeDeploymentState no-ops; central + // persists acks from the ApplyAck) and serves scripted-alarm state from the LocalDb + // alarmStateStore instead of an EfAlarmConditionStateStore. + DriverHostActor.Props(dbFactory, roleInfo.LocalNode, ackRouter: useClusterClient ? nodeComm : null, driverFactory: driverFactory, localRoles: roleInfo.LocalRoles, dependencyMux: mux, @@ -484,7 +489,8 @@ public static class ServiceCollectionExtensions redundancyRoleView: redundancyRoleView, replicationPeerHost: replicationPeerHost, fetchAndCacheMode: fetchAndCacheMode, - artifactFetcher: artifactFetcher), + artifactFetcher: artifactFetcher, + alarmStateStore: alarmStateStore), DriverHostActorName); registry.Register(driverHost); diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbWiringTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbWiringTests.cs index a2c81f5a..4dc9bd56 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbWiringTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbWiringTests.cs @@ -11,7 +11,9 @@ using ZB.MOM.WW.LocalDb.Replication; using ZB.MOM.WW.OtOpcUa.Host.Configuration; using ZB.MOM.WW.OtOpcUa.Host.Health; using ZB.MOM.WW.OtOpcUa.Host.Observability; +using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms; using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache; +using ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms; namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests; @@ -107,6 +109,18 @@ public sealed class LocalDbWiringTests : IDisposable .ShouldBeOfType(); } + [Fact] + public void DriverGraph_AlarmStateStoreResolvesToTheLocalDbBackedImplementation() + { + // Phase 4: scripted-alarm condition state is served from the replicated LocalDb store on a + // driver node. A dropped registration would silently fall the actor back to the EF store (or, + // DB-less, skip the alarm host) — this pins the LocalDb store as the resolved implementation. + var sp = BuildDriverGraph(); + + sp.GetService() + .ShouldBeOfType(); + } + [Fact] public void DriverGraph_DefaultOff_SyncStatusReportsNotConnectedAndNoPeer() { @@ -142,6 +156,8 @@ public sealed class LocalDbWiringTests : IDisposable sp.GetService().ShouldBeNull(); sp.GetService().ShouldBeNull(); + // Phase 4: an admin-only node never registers the LocalDb alarm store either. + sp.GetService().ShouldBeNull(); // The unconditionally-registered health check must still resolve and construct — it is meant // to no-op to Healthy when LocalDb is absent, not to fail because ISyncStatus is missing. diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorDbLessTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorDbLessTests.cs new file mode 100644 index 00000000..dc5177ab --- /dev/null +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorDbLessTests.cs @@ -0,0 +1,272 @@ +using System.Text.Json; +using Akka.Actor; +using Microsoft.Data.Sqlite; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; +using Serilog; +using Shouldly; +using Xunit; +using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy; +using ZB.MOM.WW.OtOpcUa.Commons.Types; +using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian; +using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms; +using ZB.MOM.WW.OtOpcUa.Core.Scripting; +using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache; +using ZB.MOM.WW.OtOpcUa.Runtime.Drivers; +using ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms; +using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness; + +namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers; + +/// +/// Per-cluster mesh Phase 4 — a driver-only node runs with NO ConfigDb: dbFactory is null, +/// the redundant SQL ack-writes no-op (central persists the ack from the ApplyAck), and +/// scripted-alarm condition state is served from the replicated LocalDb store instead of the +/// ConfigDb-backed EF store. These tests drive a FetchAndCache actor with a null +/// dbFactory — the shape a real driver-only node boots in. +/// +public sealed class DriverHostActorDbLessTests : RuntimeActorTestBase +{ + private static readonly NodeId TestNode = NodeId.Parse("driver-test"); + private static readonly RevisionHash RevA = RevisionHash.Parse(new string('a', 64)); + + private readonly string _dbPath = + Path.Combine(Path.GetTempPath(), $"otopcua-dbless-{Guid.NewGuid():N}.db"); + private readonly ServiceProvider _localDbProvider; + private readonly ILocalDb _localDb; + + public DriverHostActorDbLessTests() + { + var configuration = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary { ["LocalDb:Path"] = _dbPath }) + .Build(); + + _localDbProvider = new ServiceCollection() + .AddZbLocalDb(configuration, db => + { + using (var connection = db.CreateConnection()) + { + AlarmConditionStateSchema.Apply(connection); + } + + db.RegisterReplicated(AlarmConditionStateSchema.StateTable); + }) + .BuildServiceProvider(); + + _localDb = _localDbProvider.GetRequiredService(); + } + + protected override void Dispose(bool disposing) + { + base.Dispose(disposing); + if (!disposing) return; + + _localDbProvider.Dispose(); + SqliteConnection.ClearAllPools(); + foreach (var path in new[] { _dbPath, $"{_dbPath}-wal", $"{_dbPath}-shm" }) + { + try { if (File.Exists(path)) File.Delete(path); } catch { /* best effort */ } + } + } + + /// Builds a minimal but parseable artifact (no drivers) — the fetched-bytes stand-in. + private static byte[] ArtifactBytes() => + JsonSerializer.SerializeToUtf8Bytes(new { DriverInstances = Array.Empty() }); + + private LocalDbAlarmConditionStateStore NewAlarmStore() + => new(_localDb, NullLogger.Instance); + + private static ScriptRootLogger NewScriptRootLogger() + => new(new LoggerConfiguration().CreateLogger()); + + /// Resolves the named child of , or null when it was not spawned. + /// Safe to call once the parent has replied to an Identify — PreStart (and thus the spawn decision) + /// completes before any message is processed. + private IActorRef? ResolveChild(IActorRef parent, string name) + { + var id = Sys.ActorSelection(parent.Path / name) + .Ask(new Identify(name), TimeSpan.FromSeconds(3)) + .GetAwaiter().GetResult(); + return id.Subject; + } + + /// Blocks until the actor has processed a message — proving PreStart has run. + private void AwaitStarted(IActorRef actor) + { + var id = actor.Ask(new Identify("started"), TimeSpan.FromSeconds(5)) + .GetAwaiter().GetResult(); + id.Subject.ShouldBe(actor); + } + + /// + /// Task 4 — a full dispatch on a DB-less actor (null dbFactory, FetchAndCache, LocalDb + /// alarm store) applies and acks Applied without a NullReferenceException. The redundant SQL + /// ack-writes are guarded to no-ops; central is the ack system of record. + /// + [Fact] + public void DbLessDispatch_AppliesAndAcks_WithNoConfigDb() + { + var cache = new RecordingArtifactCache(); + var fetcher = new RecordingFetcher(ArtifactBytes()); + var deploymentId = DeploymentId.NewId(); + var publish = CreateTestProbe(); + var coordinator = CreateTestProbe(); + + var actor = Sys.ActorOf(DriverHostActor.Props( + dbFactory: null, + TestNode, + coordinator.Ref, + localRoles: new HashSet { "driver" }, + opcUaPublishActor: publish.Ref, + scriptRootLogger: NewScriptRootLogger(), + deploymentArtifactCache: cache, + fetchAndCacheMode: true, + artifactFetcher: fetcher, + alarmStateStore: NewAlarmStore())); + + actor.Tell(new DispatchDeployment(deploymentId, RevA, CorrelationId.NewId())); + + coordinator.ExpectMsg(TimeSpan.FromSeconds(5)).Outcome.ShouldBe(ApplyAckOutcome.Applied); + AwaitAssert(() => cache.Stores.Count.ShouldBe(1), duration: TimeSpan.FromSeconds(3)); + fetcher.Calls.ShouldBe(1); + } + + /// + /// Task 6 — a DB-less actor with a LocalDb alarm store spawns its ScriptedAlarm host (the store + /// is what makes the spawn possible without a ConfigDb). + /// + [Fact] + public void DbLessNode_WithLocalDbStore_SpawnsScriptedAlarmHost() + { + var publish = CreateTestProbe(); + var actor = Sys.ActorOf(DriverHostActor.Props( + dbFactory: null, + TestNode, + opcUaPublishActor: publish.Ref, + scriptRootLogger: NewScriptRootLogger(), + fetchAndCacheMode: true, + artifactFetcher: new RecordingFetcher(ArtifactBytes()), + alarmStateStore: NewAlarmStore())); + + AwaitStarted(actor); + ResolveChild(actor, "scripted-alarm-host").ShouldNotBeNull(); + } + + /// + /// Task 6 — with neither a LocalDb store NOR a ConfigDb factory there is nowhere to persist + /// condition state, so the ScriptedAlarm host is skipped (not spawned against a null store). + /// + [Fact] + public void DbLessNode_WithNoStoreAndNoDbFactory_SkipsScriptedAlarmHost() + { + var publish = CreateTestProbe(); + var actor = Sys.ActorOf(DriverHostActor.Props( + dbFactory: null, + TestNode, + opcUaPublishActor: publish.Ref, + scriptRootLogger: NewScriptRootLogger(), + fetchAndCacheMode: true, + artifactFetcher: new RecordingFetcher(ArtifactBytes()), + alarmStateStore: null)); + + AwaitStarted(actor); + ResolveChild(actor, "scripted-alarm-host").ShouldBeNull(); + } + + /// + /// Regression — a DB-backed actor (non-null dbFactory, no alarm store passed) still spawns + /// the host on the EF-store fallback path, exactly as before Phase 4. + /// + [Fact] + public void DbBackedNode_WithNoStore_StillSpawnsScriptedAlarmHost_OnEfFallback() + { + var publish = CreateTestProbe(); + var actor = Sys.ActorOf(DriverHostActor.Props( + dbFactory: NewInMemoryDbFactory(), + TestNode, + opcUaPublishActor: publish.Ref, + scriptRootLogger: NewScriptRootLogger(), + alarmStateStore: null)); + + AwaitStarted(actor); + ResolveChild(actor, "scripted-alarm-host").ShouldNotBeNull(); + } + + /// + /// Focused proof the LocalDb alarm store round-trips condition state with NO ConfigDb in play. + /// + [Fact] + public async Task LocalDbAlarmStore_RoundTripsConditionState_WithoutConfigDb() + { + var store = NewAlarmStore(); + var t = new DateTime(2026, 07, 22, 08, 00, 00, DateTimeKind.Utc); + var state = new AlarmConditionState( + AlarmId: "dbless-alarm", + Enabled: AlarmEnabledState.Enabled, + Active: AlarmActiveState.Inactive, + Acked: AlarmAckedState.Acknowledged, + Confirmed: AlarmConfirmedState.Unconfirmed, + Shelving: ShelvingState.Unshelved, + LastTransitionUtc: t, + LastActiveUtc: null, + LastClearedUtc: null, + LastAckUtc: t, + LastAckUser: "op", + LastAckComment: "ack", + LastConfirmUtc: null, + LastConfirmUser: null, + LastConfirmComment: null, + Comments: System.Collections.Immutable.ImmutableList.Empty); + + await store.SaveAsync(state, CancellationToken.None); + var loaded = await store.LoadAsync("dbless-alarm", CancellationToken.None); + + loaded.ShouldNotBeNull(); + loaded.Acked.ShouldBe(AlarmAckedState.Acknowledged); + loaded.LastAckUser.ShouldBe("op"); + } + + /// Records fetch calls and returns canned bytes (or null for "no answer"). + private sealed class RecordingFetcher(byte[]? bytes) : IDeploymentArtifactFetcher + { + private int _calls; + public int Calls => Volatile.Read(ref _calls); + + public Task FetchAsync(string deploymentId, string expectedRevisionHash, CancellationToken ct) + { + Interlocked.Increment(ref _calls); + return Task.FromResult(bytes); + } + } + + /// Records every store; GetCurrent* return null (no idempotency shortcut in these tests). + private sealed class RecordingArtifactCache : IDeploymentArtifactCache + { + private readonly Lock _gate = new(); + private readonly List _stores = []; + + public IReadOnlyList Stores + { + get { lock (_gate) { return _stores.ToArray(); } } + } + + public Task StoreAsync(string clusterId, string deploymentId, string revisionHash, + byte[] artifact, CancellationToken ct = default) + { + lock (_gate) { _stores.Add(new StoreCall(clusterId, deploymentId, revisionHash, artifact)); } + return Task.CompletedTask; + } + + public Task GetCurrentAsync(string clusterId, CancellationToken ct = default) + => Task.FromResult(null); + + public Task GetCurrentUnkeyedAsync(CancellationToken ct = default) + => Task.FromResult(null); + + internal sealed record StoreCall( + string ClusterId, string DeploymentId, string RevisionHash, byte[] Artifact); + } +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ScriptedAlarms/LocalDbAlarmConditionStateStoreTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ScriptedAlarms/LocalDbAlarmConditionStateStoreTests.cs index b2c15696..fc0d9525 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ScriptedAlarms/LocalDbAlarmConditionStateStoreTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/ScriptedAlarms/LocalDbAlarmConditionStateStoreTests.cs @@ -121,6 +121,45 @@ public sealed class LocalDbAlarmConditionStateStoreTests : IDisposable loaded.LastConfirmUtc.ShouldBe(state.LastConfirmUtc); loaded.LastConfirmUser.ShouldBe("bob"); loaded.LastConfirmComment.ShouldBe("confirm-comment"); + // Shouldly's DateTime ShouldBe compares ticks and ignores DateTimeKind, so an explicit Kind + // assertion is the only guard on the Utc-kind contract the round-trip ("O") format encodes. + loaded.LastTransitionUtc.Kind.ShouldBe(DateTimeKind.Utc); + } + + /// Unset audit fields (LastAckUtc/User, LastConfirmUtc/User) round-trip back as null. + [Fact] + public async Task Null_audit_fields_round_trip_as_null() + { + var store = NewStore(); + var t = new DateTime(2026, 06, 10, 12, 00, 00, DateTimeKind.Utc); + var state = new AlarmConditionState( + AlarmId: "alarm-null-audit", + Enabled: AlarmEnabledState.Enabled, + Active: AlarmActiveState.Inactive, + Acked: AlarmAckedState.Unacknowledged, + Confirmed: AlarmConfirmedState.Unconfirmed, + Shelving: ShelvingState.Unshelved, + LastTransitionUtc: t, + LastActiveUtc: null, + LastClearedUtc: null, + LastAckUtc: null, + LastAckUser: null, + LastAckComment: null, + LastConfirmUtc: null, + LastConfirmUser: null, + LastConfirmComment: null, + Comments: ImmutableList.Empty); + + await store.SaveAsync(state, CancellationToken.None); + var loaded = await store.LoadAsync("alarm-null-audit", CancellationToken.None); + + loaded.ShouldNotBeNull(); + loaded.LastAckUtc.ShouldBeNull(); + loaded.LastAckUser.ShouldBeNull(); + loaded.LastAckComment.ShouldBeNull(); + loaded.LastConfirmUtc.ShouldBeNull(); + loaded.LastConfirmUser.ShouldBeNull(); + loaded.LastConfirmComment.ShouldBeNull(); } /// Loading an id that was never saved returns null. From e791832d7cbb0bf29d8e9c4357a1c5d8a59ac687 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 12:48:13 -0400 Subject: [PATCH 10/18] docs(mesh-phase4): Tasks 4+6 done (d9071607); Task 3 confirmed hard prereq MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 4+6 code review re-surfaced the OpcUaPublishActor raw-sink wipe on a null-factory node (the exact #485/#486 defect Task 3 fixes) — Task 3 must land before any node flips to FetchAndCache. Ordering already enforces this (3 → 7 → 10 → 11). Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- ...-23-mesh-phase4-cut-driver-configdb.md.tasks.json | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json index f5228e97..42a42abd 100644 --- a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json +++ b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json @@ -46,10 +46,12 @@ { "id": 4, "subject": "Task 4: DriverHostActor \u2014 nullable factory; drop redundant SQL ack-writes", - "status": "pending", + "status": "completed", "blockedBy": [ 1 - ] + ], + "commit": "d9071607", + "result": "Tasks 4+6 combined (avoid churn). 32+162+6 tests; SPEC-COMPLIANT + APPROVED. Review re-confirmed Task 3 is a HARD PREREQ before any FetchAndCache flip (address-space wipe)" }, { "id": 5, @@ -61,11 +63,13 @@ { "id": 6, "subject": "Task 6: Wire LocalDb condition store into DriverHostActor for driver-role", - "status": "pending", + "status": "completed", "blockedBy": [ 4, 5 - ] + ], + "commit": "d9071607", + "result": "Tasks 4+6 combined (avoid churn). 32+162+6 tests; SPEC-COMPLIANT + APPROVED. Review re-confirmed Task 3 is a HARD PREREQ before any FetchAndCache flip (address-space wipe)" }, { "id": 7, From 73ae8ec5c5d3fbea28182e4c61b84fc8406f5709 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 12:53:28 -0400 Subject: [PATCH 11/18] fix(mesh-phase4): OpcUaPublish diff-applies from in-hand bytes with no ConfigDb (no wipe) A driver-only node has a null dbFactory but a wired applier; the old combined guard routed it to the raw-sink fallback that wipes the address space and never re-materialises. Split the guard so in-hand artifact bytes drive the real diff-and-apply; a missing artifact abandons the rebuild (keep last-known-good, #485) instead of wiping. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- .../OpcUa/OpcUaPublishActor.cs | 34 ++++-- .../OpcUa/OpcUaPublishActorRebuildTests.cs | 114 ++++++++++++++++-- 2 files changed, 128 insertions(+), 20 deletions(-) diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs index 7cf523c9..61a0019f 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs @@ -359,10 +359,15 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers using var span = OtOpcUaTelemetry.StartAddressSpaceRebuildSpan(); span?.SetTag("otopcua.correlation_id", msg.Correlation.ToString()); - // Two modes: when dbFactory + applier are wired, do a real diff-and-apply pass against - // the latest deployment artifact. Without them, fall back to a raw sink rebuild — the - // F10b/dev path before the integration completes. - if (_dbFactory is null || _applier is null) + // Two modes. With an applier wired, do a REAL diff-and-apply pass — driven either by the + // artifact bytes already in hand (a per-cluster-mesh driver-only / FetchAndCache node, whose + // ConfigDb is admin-only now so _dbFactory is null, hands its fetched/cached bytes in + // msg.Artifact) or, when no bytes are carried, by loading the artifact from the ConfigDb (the + // Direct path, which requires _dbFactory). Only a node with NO applier at all (the F10b/dev + // seam) falls back to the raw-sink rebuild — and that fallback WIPES the served address space + // without re-materialising, so it must NEVER be reached by a real driver node that has an + // applier but simply has no ConfigDb (that node diff-and-applies from its in-hand bytes below). + if (_applier is null) { try { @@ -379,15 +384,20 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers try { - // An in-hand artifact (boot-from-cache) wins: the host already has the cached bytes, and - // the ConfigDb read the DeploymentId path would do is exactly what is unreachable during - // the outage this exists to survive. Otherwise prefer the artifact of the deployment the - // host just applied — at apply time it is not yet Sealed, so LoadLatestArtifact would - // return the PREVIOUS revision and materialise a stale composition (variables that don't - // match the SubscribeBulk refs). Fall back to latest-sealed only for legacy callers that - // carry neither. + // An in-hand artifact (boot-from-cache / FetchAndCache) wins: the host already has the + // cached/fetched bytes, and the ConfigDb read the DeploymentId path would do is exactly what + // is unreachable during the outage this exists to survive — and is simply absent on a + // driver-only node with no ConfigDb. Otherwise, when a ConfigDb is present (Direct mode), + // prefer the artifact of the deployment the host just applied — at apply time it is not yet + // Sealed, so LoadLatestArtifact would return the PREVIOUS revision and materialise a stale + // composition (variables that don't match the SubscribeBulk refs); fall back to latest-sealed + // only for legacy callers that carry neither. With NO bytes AND no ConfigDb there is nothing + // to load, so yield Array.Empty() — the #485 guard below reads that as "no answer" and + // abandons the rebuild, holding last-known-good rather than wiping the address space. var artifact = msg.Artifact - ?? (msg.DeploymentId is { } depId ? LoadArtifact(depId) : LoadLatestArtifact()); + ?? (_dbFactory is not null + ? (msg.DeploymentId is { } depId ? LoadArtifact(depId) : LoadLatestArtifact()) + : Array.Empty()); // Issue #485 — an artifact we could not obtain is NOT an empty configuration. Parsing zero // bytes yields an empty composition, which the planner diffs against the live one as a diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/OpcUa/OpcUaPublishActorRebuildTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/OpcUa/OpcUaPublishActorRebuildTests.cs index ba619faf..a51410b4 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/OpcUa/OpcUaPublishActorRebuildTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/OpcUa/OpcUaPublishActorRebuildTests.cs @@ -110,6 +110,97 @@ public sealed class OpcUaPublishActorRebuildTests : RuntimeActorTestBase AwaitAssert(() => sink.RebuildCalls.ShouldBe(1), duration: TimeSpan.FromMilliseconds(500)); } + /// + /// mesh-phase4 — the wipe-prevention test. A driver-only node has a NULL dbFactory (ConfigDb is + /// admin-only now) but a WIRED applier. A RebuildAddressSpace carrying the artifact bytes in hand + /// (from the fetch/cache) must drive the REAL diff-and-apply (the equipment folder is materialised) + /// and must NOT take the raw-sink RebuildAddressSpace() fallback, which wipes the address + /// space without re-materialising. Before the guard split, a null dbFactory routed straight to that + /// fallback and the applier never ran — a FetchAndCache node deployed green but served EMPTY. + /// + [Fact] + public void Rebuild_with_null_factory_but_wired_applier_diff_applies_from_in_hand_bytes() + { + var sink = new RecordingSink(); + var applier = new AddressSpaceApplier(sink, NullLogger.Instance); + var artifact = BuildNamedEquipmentArtifact(("eq-1", "Pump-1")); + + // Driver-only node: null dbFactory, wired applier. + var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests( + sink: sink, dbFactory: null, applier: applier)); + + actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId(), Artifact: artifact)); + + // The in-hand bytes drove the real diff-and-apply — the equipment folder was materialised… + AwaitAssert(() => sink.Calls.ShouldContain("EF:eq-1"), duration: TimeSpan.FromSeconds(2)); + // …and the address-space-wiping raw-sink fallback was NOT taken. + sink.RebuildCalls.ShouldBe(0); + } + + /// + /// mesh-phase4 / #485 negative — a driver-only node (null dbFactory, wired applier) that gets a + /// RebuildAddressSpace with NO artifact bytes has no way to obtain a config (there is no ConfigDb to + /// read). The artifact source yields Array.Empty<byte>(), which the #485 zero-length + /// guard treats as "no answer": the rebuild is abandoned WITHOUT touching the served address space — + /// the applier never runs and the raw-sink wipe fallback is never taken. Last-known-good stands. + /// + [Fact] + public void Rebuild_with_null_factory_and_no_artifact_abandons_without_wiping() + { + var sink = new RecordingSink(); + var applier = new AddressSpaceApplier(sink, NullLogger.Instance); + + var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests( + sink: sink, dbFactory: null, applier: applier)); + + actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId())); + Thread.Sleep(200); + + sink.RebuildCalls.ShouldBe(0); // the raw-sink wipe fallback was NOT taken + sink.Calls.ShouldBeEmpty(); // the applier's diff-and-apply never ran + } + + /// + /// mesh-phase4 regression (dev seam) — a node with NO applier at all still falls back to the raw-sink + /// rebuild, and now the fallback is keyed on the applier ALONE: even WITH a dbFactory wired, a null + /// applier takes the F10b/dev fallback. Proves the guard split did not change the no-applier seam. + /// + [Fact] + public void Rebuild_with_null_applier_falls_back_to_raw_sink_even_with_dbFactory() + { + var db = NewInMemoryDbFactory(); + var sink = new RecordingSink(); + + var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests( + sink: sink, dbFactory: db, applier: null)); + + actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId())); + + AwaitAssert(() => sink.RebuildCalls.ShouldBe(1), duration: TimeSpan.FromMilliseconds(500)); + } + + /// + /// mesh-phase4 regression (Direct mode) — a DB-backed node (dbFactory wired, applier wired) with no + /// in-hand bytes but a DeploymentId still loads the artifact from the ConfigDb via LoadArtifact + /// and drives the applier. The guard split must leave the Direct path byte-identical. + /// + [Fact] + public void Rebuild_direct_mode_with_dbFactory_and_deploymentId_still_loads_and_applies() + { + var db = NewInMemoryDbFactory(); + var sink = new RecordingSink(); + var applier = new AddressSpaceApplier(sink, NullLogger.Instance); + var dep = SeedNamedEquipmentDeployment(db, ("eq-1", "Pump-1")); + + var actor = Sys.ActorOf(OpcUaPublishActor.PropsForTests( + sink: sink, dbFactory: db, applier: applier)); + + actor.Tell(new OpcUaPublishActor.RebuildAddressSpace(CorrelationId.NewId(), new DeploymentId(dep))); + + AwaitAssert(() => sink.Calls.ShouldContain("EF:eq-1"), duration: TimeSpan.FromSeconds(2)); + sink.RebuildCalls.ShouldBe(0); + } + /// /// Wiring proof for per-ClusterId scoping (Task 4): a multi-cluster artifact must /// materialise ONLY the local node's cluster slice. Mirrors the multi-cluster artifact @@ -358,14 +449,12 @@ public sealed class OpcUaPublishActorRebuildTests : RuntimeActorTestBase : inner.CreateDbContext(); } - /// Seal a deployment carrying the given (equipmentId, name) rows under a shared line, returning - /// the new DeploymentId so a test can target THAT artifact. Renaming an equipment across two such - /// deployments produces a ChangedEquipment delta (Rebuild), while adding a row is a PureAdd. - private static Guid SeedNamedEquipmentDeployment( - IDbContextFactory dbFactory, - params (string Id, string Name)[] equipment) - { - var artifact = JsonSerializer.SerializeToUtf8Bytes(new + /// Build the artifact bytes for the given (equipmentId, name) rows under a shared line — + /// WITHOUT sealing them into a deployment. This is what a driver-only / FetchAndCache node carries + /// in RebuildAddressSpace.Artifact (no ConfigDb read), so tests can drive the in-hand-bytes + /// path directly. + private static byte[] BuildNamedEquipmentArtifact(params (string Id, string Name)[] equipment) => + JsonSerializer.SerializeToUtf8Bytes(new { UnsAreas = new[] { new { UnsAreaId = "area-1", ClusterId = "c1", Name = "Area" } }, UnsLines = new[] { new { UnsLineId = "line-1", UnsAreaId = "area-1", Name = "Line" } }, @@ -383,6 +472,15 @@ public sealed class OpcUaPublishActorRebuildTests : RuntimeActorTestBase ScriptedAlarms = Array.Empty(), }); + /// Seal a deployment carrying the given (equipmentId, name) rows under a shared line, returning + /// the new DeploymentId so a test can target THAT artifact. Renaming an equipment across two such + /// deployments produces a ChangedEquipment delta (Rebuild), while adding a row is a PureAdd. + private static Guid SeedNamedEquipmentDeployment( + IDbContextFactory dbFactory, + params (string Id, string Name)[] equipment) + { + var artifact = BuildNamedEquipmentArtifact(equipment); + var id = Guid.NewGuid(); using var ctx = dbFactory.CreateDbContext(); ctx.Deployments.Add(new Deployment From 4253d202486d3aaab73b2273b3de102df9fb0149 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 12:58:37 -0400 Subject: [PATCH 12/18] =?UTF-8?q?docs(mesh-phase4):=20Task=203=20done=20(7?= =?UTF-8?q?3ae8ec5)=20=E2=80=94=20address-space=20wipe=20fixed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- ...2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json index 42a42abd..c5e37a19 100644 --- a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json +++ b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json @@ -38,10 +38,12 @@ { "id": 3, "subject": "Task 3: OpcUaPublishActor \u2014 nullable factory; rebuild always carries bytes", - "status": "pending", + "status": "completed", "blockedBy": [ 2 - ] + ], + "commit": "73ae8ec5", + "result": "44 tests (1+2 RED pre-fix); SPEC-COMPLIANT + APPROVED. Wipe prevented at every entry point; validator is belt-and-suspenders" }, { "id": 4, From 6f21213f701c8afc2a6672313b4117dc6244eefd Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 13:10:42 -0400 Subject: [PATCH 13/18] test(mesh-phase4): pin driver-only ConfigDb-free DI graph + fused-node mapper ordering Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- .../DriverOnlyNoConfigDbBootTests.cs | 115 ++++++++++++++++++ 1 file changed, 115 insertions(+) diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DriverOnlyNoConfigDbBootTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DriverOnlyNoConfigDbBootTests.cs index 954ddadc..c74d27b5 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DriverOnlyNoConfigDbBootTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DriverOnlyNoConfigDbBootTests.cs @@ -2,9 +2,15 @@ using Microsoft.AspNetCore.Builder; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; using Shouldly; using Xunit; +using ZB.MOM.WW.Auth.Abstractions.Roles; +using ZB.MOM.WW.Configuration; using ZB.MOM.WW.OtOpcUa.Configuration; +using ZB.MOM.WW.OtOpcUa.Configuration.Services; +using ZB.MOM.WW.OtOpcUa.Host.Configuration; +using ZB.MOM.WW.OtOpcUa.Security.Ldap; namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests; @@ -87,6 +93,115 @@ public sealed class DriverOnlyNoConfigDbBootTests : IDisposable Should.Throw(() => builder.Services.AddOtOpcUaConfigDb(builder.Configuration)); } + /// + /// Replicates Program.cs's role-mapping DI ordering, top to bottom: + /// + /// + /// the hasAdmin block — AddOtOpcUaConfigDb, which plain-AddScopeds + /// the real (line ~132 in Program.cs); + /// + /// + /// AddValidatedOptions<LdapOptions, LdapOptionsValidator>, registered + /// unconditionally for every role shape (line ~143); + /// + /// + /// the hasDriver block's three TryAdd registrations (lines ~333-335): + /// , . + /// + /// + /// On a FUSED node this ordering is exactly what lets the real EF-backed service win the + /// TryAdd — a plain AddScoped that ran first already claimed the descriptor slot, + /// so the later TryAddScoped is a no-op. Nothing enforces that order beyond Program.cs's + /// own top-to-bottom script. Verified by locally inverting the ordering to (3)-then-(1) AND + /// switching AddOtOpcUaConfigDb's ILdapGroupRoleMappingService registration to a + /// TryAddScoped (both reverted before commit) — that combination is what flips the + /// resolved type to and turns this test red; + /// either change alone stays green today (an unconditional AddScoped always wins + /// last-registered regardless of order, and a TryAdd that runs first still wins). Security: + /// Ldap:Enabled is forced false here purely so doesn't demand + /// a live Server/SearchBase/Transport — it is orthogonal to what is under test. + /// + private IServiceProvider BuildRoleMappingGraph(bool hasAdmin, string? configDbConnectionString) + { + var builder = WebApplication.CreateBuilder(new WebApplicationOptions { Args = [] }); + var config = new Dictionary { ["Security:Ldap:Enabled"] = "false" }; + if (configDbConnectionString is not null) + config["ConnectionStrings:ConfigDb"] = configDbConnectionString; + builder.Configuration.AddInMemoryCollection(config); + + // (1) hasAdmin block. + if (hasAdmin) + builder.Services.AddOtOpcUaConfigDb(builder.Configuration); + + // (2) unconditional for every role shape. + builder.Services.AddValidatedOptions( + builder.Configuration, LdapOptions.SectionName); + + // (3) hasDriver block's TryAdd registrations — runs AFTER (1), never before it. + builder.Services.TryAddScoped(); + builder.Services.TryAddScoped, OtOpcUaGroupRoleMapper>(); + + var app = builder.Build(); + _apps.Add(app); + return app.Services; + } + + [Fact] + public void FusedNode_RealLdapGroupRoleMappingService_WinsOverTheDriverBlockTryAdd() + { + // The Task 1b review gap: on a fused admin+driver node, AddOtOpcUaConfigDb's plain AddScoped + // registers the real EF-backed service BEFORE the hasDriver block's TryAddScoped runs, so the + // TryAdd is a no-op and the real service wins. If AddOtOpcUaConfigDb's registration were ever + // switched to a TryAdd, or the two blocks reordered, this flips to NullLdapGroupRoleMappingService + // and every DB-backed role grant central relies on silently stops applying — with no other test + // catching it. + var sp = BuildRoleMappingGraph( + hasAdmin: true, + configDbConnectionString: "Server=(local);Database=OtOpcUa;Trusted_Connection=True;TrustServerCertificate=True"); + + using var scope = sp.CreateScope(); + scope.ServiceProvider.GetRequiredService() + .ShouldBeOfType(); + } + + [Fact] + public void DriverOnlyNode_LdapGroupRoleMapping_ResolvesTheNullImpl_AndNoConfigDbFactory() + { + // The other half of the same pin: with no hasAdmin block ever run, the driver block's TryAdd + // is the ONLY registration, so it wins by default — no ConfigDb, no real EF service reachable. + var sp = BuildRoleMappingGraph(hasAdmin: false, configDbConnectionString: null); + + sp.GetService>().ShouldBeNull(); + + using var scope = sp.CreateScope(); + scope.ServiceProvider.GetRequiredService() + .ShouldBeOfType(); + } + + [Fact] + public async Task DriverOnlyNode_IGroupRoleMapper_ResolvesInScope_AndMapsWithoutThrowing() + { + // Task 1b fixed a lazy auth-path break: IGroupRoleMapper depends on + // ILdapGroupRoleMappingService, which a driver-only node never registered before that fix — + // the DI graph built fine (nothing eager touches it), and the break surfaced only on the first + // SCOPED resolve + call, deep in the OPC UA data-plane authenticator. A bare Build() cannot + // catch that class of bug; this test resolves in a scope and calls MapAsync, mirroring the + // real call site. + var sp = BuildRoleMappingGraph(hasAdmin: false, configDbConnectionString: null); + + using var scope = sp.CreateScope(); + var mapper = scope.ServiceProvider.GetRequiredService>(); + mapper.ShouldBeOfType(); + scope.ServiceProvider.GetRequiredService() + .ShouldBeOfType(); + + GroupRoleMapping result = null!; + await Should.NotThrowAsync(async () => result = await mapper.MapAsync(["some-group"], CancellationToken.None)); + + result.ShouldNotBeNull(); + result.Scope.ShouldBeNull(); + } + public void Dispose() { foreach (var app in _apps) From 55a6e1f2b21f8308e7fe057999ba1ffa34a6f4f4 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 13:18:57 -0400 Subject: [PATCH 14/18] docs(mesh-phase4): ServiceLevel semantics on DB-less nodes + driver ConfigDb cut Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- CLAUDE.md | 34 ++++++++++++++++ docs/Configuration.md | 17 +++++++- docs/Redundancy.md | 29 +++++++++++++- .../2026-07-22-per-cluster-mesh-program.md | 40 ++++++++++++++----- ...h-phase4-cut-driver-configdb.md.tasks.json | 11 +++-- 5 files changed, 116 insertions(+), 15 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 307053f1..3a2f7ece 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -355,6 +355,40 @@ See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-loca cache), `docs/AlarmHistorian.md`, and the runbook `docs/operations/2026-07-20-localdb-pair-replication.md`. +## Phase 4 — cut the driver-side ConfigDb connection + +Per-cluster mesh **Phase 4** finished the job Phase 3 started: a **driver-only** node (`Cluster:Roles` +has `driver`, not `admin`) now boots with **no `ConfigDb` connection string at all**. `Program.cs` +gates `AddOtOpcUaConfigDb` (and its health check) on `hasAdmin`; a fused `admin,driver` node is +unaffected (it keeps ConfigDb via its admin role). Consequences: + +- **`ConfigSource:Mode` must be `FetchAndCache` on a driver-only node** — `ConfigSourceOptionsValidator` + fails host start if such a node is left on `Direct` (there is nothing to read Direct from). A fused + node may stay `Direct`. +- **ServiceLevel semantics change — client-visible.** With no ConfigDb, `DbHealthProbeActor` is not + spawned (`ServiceCollectionExtensions.WithOtOpcUaRuntimeActors` skips it when the resolved db + factory is null), and `OpcUaPublishActor` runs `dbLess`: `DbReachable` is fed constant `true` and + `Stale` comes from the redundancy-snapshot age alone. A healthy driver-only node therefore publishes + **240** (or **250** as driver Primary) even with central SQL unreachable — the survive-alone posture + — where a `Direct`/DB-backed node in the same state drops to 100/0. `ServiceLevelCalculator` itself + is unchanged; only the inputs a DB-less node feeds it changed. See `docs/Redundancy.md` §"DB-less + (driver-only) nodes". +- **Scripted-alarm Part 9 condition state moved to LocalDb.** `LocalDbAlarmConditionStateStore` + (replicated `alarm_condition_state` table, pair-local like the Phase-2 alarm S&F buffer) replaces + `EfAlarmConditionStateStore` on every driver-role node, fused central included. The DB-backed + `ScriptedAlarmState` table is now unused (dropping it is a deferred follow-up, tracked as Task 9). +- **Central persists deploy acks.** `DriverHostActor` no longer writes `NodeDeploymentState` to SQL on + a driver-only node — `ConfigPublishCoordinator.PersistNodeAck` (already fed by every ApplyAck) is + the system of record. +- **LDAP group→role DB grants don't reach driver-only nodes.** They register the empty + `NullLdapGroupRoleMappingService` (no ConfigDb to read `LdapGroupRoleMapping` from), so + `OtOpcUaGroupRoleMapper` falls back to the `Security:Ldap:GroupToRole` appsettings baseline only — + the same rows a driver node never received via config either, so this is not a regression. + +Code-complete on `feat/mesh-phase4`; the table-drop (Task 9) and the live gate (Task 10) are still +open. See `docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md` and +`docs/plans/2026-07-22-per-cluster-mesh-program.md` §Phase 4. + ## LDAP Authentication The server uses LDAP-based user authentication via the `Security:Ldap` section in `appsettings.json`. When enabled, credentials are validated by LDAP bind against a GLAuth server, and LDAP group membership maps to OPC UA permissions: `ReadOnly` (browse/read), `WriteOperate` (write FreeAccess/Operate attributes), `WriteTune` (write Tune attributes), `WriteConfigure` (write Configure attributes), `AlarmAck` (alarm acknowledgment). `LdapOpcUaUserAuthenticator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`) implements `IOpcUaUserAuthenticator`, delegating the LDAP bind + group lookup to `OtOpcUaLdapAuthService` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs`, an `ILdapAuthService`). See `docs/security.md` for the full guide. diff --git a/docs/Configuration.md b/docs/Configuration.md index 4fd14d89..734867d8 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -162,9 +162,24 @@ On the docker-dev rig every node carries these keys already; flip the whole rig On the docker-dev rig the central pair carries `ConfigServe` (port `4055`, dev key) and stays `Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`. Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up` — central keeps SQL and models the target topology. +**`FetchAndCache` is mandatory on a driver-only node — per-cluster mesh Phase 4.** `Program.cs` now +registers `AddOtOpcUaConfigDb` only when the node carries the `admin` role, so a **driver-only** +node (`Cluster:Roles` has `driver`, not `admin`) holds **no `ConfigDb` connection string at all** — +there is nothing for `Direct` to read. `ConfigSourceOptionsValidator` enforces this at +`ValidateOnStart`: a driver-only node whose `ConfigSource:Mode` is `Direct` **fails host start** +with an error naming the fix (`ConfigSource:Mode=FetchAndCache`). A **fused** `admin,driver` node +still owns its ConfigDb via the admin role and may stay `Direct` — the validator only fires on +`driver` without `admin`. On a driver-only node the deployed configuration, the scripted-alarm Part +9 condition state (`LocalDbAlarmConditionStateStore`, replacing the ConfigDb-backed +`EfAlarmConditionStateStore`), and the alarm store-and-forward buffer all live in the node's +LocalDb; central remains the system of record for deploy acks (`ConfigPublishCoordinator. +PersistNodeAck`) and LDAP group→role DB grants (a driver-only node maps LDAP groups to roles from +the `Security:Ldap:GroupToRole` appsettings baseline only — see `docs/Redundancy.md` and +`CLAUDE.md` §"Phase 4" for the full picture, including the client-visible ServiceLevel change). + ### `ConnectionStrings` → `ConfigDb` -- **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally. +- **Purpose:** the central Config DB connection string. **Required on admin-role nodes only** — `Program.cs` calls `AddOtOpcUaConfigDb` when `hasAdmin` (per-cluster mesh Phase 4). A driver-only node holds no `ConfigDb` connection string; it fetches its configuration via `ConfigSource:Mode=FetchAndCache` instead (see above). A fused `admin,driver` node still requires it. - **Bound by:** `AddOtOpcUaConfigDb(config)` (`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/ServiceCollectionExtensions.cs`). The connection-string name constant is `ConnectionStringName = "ConfigDb"`, read via `configuration.GetConnectionString("ConfigDb")`. If absent, startup throws with a message pointing to either `appsettings.json` or the `OTOPCUA_CONFIG_CONNECTION` env var. - **Shape:** standard `ConnectionStrings:ConfigDb` SQL Server connection string. There is no checked-in default in the thin `appsettings*.json` — supply it per environment. diff --git a/docs/Redundancy.md b/docs/Redundancy.md index cacc4bd7..2733b966 100644 --- a/docs/Redundancy.md +++ b/docs/Redundancy.md @@ -16,7 +16,7 @@ The runtime pieces live in: | `OpcUaPublishActor` | `OtOpcUa.Runtime.OpcUa` | Per-driver-node; subscribes to the `redundancy-state` topic, computes a health-aware ServiceLevel byte via `ServiceLevelCalculator` (see below), and forwards it to `IServiceLevelPublisher`. | | `IServiceLevelPublisher` / `SdkServiceLevelPublisher` | `OtOpcUa.Commons.OpcUa` / `OtOpcUa.OpcUaServer` | Writes the byte into the SDK's `Server.ServiceLevel` Variable. Production binds `DeferredServiceLevelPublisher`, which swaps in the real `SdkServiceLevelPublisher` once the SDK is up (it needs `IServerInternal`, available only after `StandardServer.Start`); until then writes route through `NullServiceLevelPublisher`. | | `ServiceLevelCalculator` | `OtOpcUa.Cluster.Redundancy` (`Core.Cluster`) | Pure function `(NodeHealthInputs) → byte` — the DB/probe-aware tiering (see truth table below). Covered by `ServiceLevelCalculatorTests`. **Now the live publish path** — `OpcUaPublishActor` calls it on every `HealthTick` and `RedundancyStateChanged` event. Moved to `Core.Cluster` so Runtime can reach it without a Runtime→ControlPlane reference. | -| `DbHealthProbeActor` | `OtOpcUa.Runtime.Health` | Per-node; runs `SELECT 1` against ConfigDb every 5s. Read by the health endpoint AND by `OpcUaPublishActor` (the `DbReachable` ServiceLevel input). | +| `DbHealthProbeActor` | `OtOpcUa.Runtime.Health` | Runs `SELECT 1` against ConfigDb every 5s. Read by the health endpoint AND by `OpcUaPublishActor` (the `DbReachable` ServiceLevel input). **Spawned only when a ConfigDb is present** (admin-role or fused admin+driver nodes) — a driver-only node has no `IDbContextFactory` to probe, so this actor is not spawned there at all; see "DB-less nodes" below. | | `PeerProbeSupervisor` | `OtOpcUa.Runtime.Health` | Per-node; subscribes to the `redundancy-state` topic and maintains one `PeerOpcUaProbeActor` child per OTHER driver-role peer (spawn on join, stop on departure), so every node is continuously probed by its peers. | | `PeerOpcUaProbeActor` | `OtOpcUa.Runtime.Health` | Spawned by `PeerProbeSupervisor`; pings a peer `opc.tcp://peer:4840` with a TCP connect (2s timeout) and publishes `OpcUaProbeResult` on the `redundancy-state` topic. A full secure-channel Hello handshake is a possible future upgrade; the TCP connect is the current real probe. | | `ClusterRoleInfo` | `OtOpcUa.Cluster` | Live view of cluster membership + role-leader; exposes `IClusterRoleInfo` to the rest of the host. (Akka's role-leader is *not* what elects the redundancy Primary — see below.) | @@ -56,6 +56,33 @@ The resulting truth table (all tiers are now reachable at runtime): > by the +10 bonus. Clients with the standard "pick highest ServiceLevel" > heuristic continue to prefer the primary. +#### DB-less (driver-only) nodes — per-cluster mesh Phase 4 + +**Client-visible change.** Per-cluster mesh Phase 4 cut the ConfigDb connection off driver-only +nodes (Akka role `driver`, not `admin`) — `Program.cs` registers `AddOtOpcUaConfigDb` only when +`hasAdmin`. With no ConfigDb to probe, `DbHealthProbeActor` is **not spawned** on such a node +(`ServiceCollectionExtensions.WithOtOpcUaRuntimeActors` skips it whenever the resolved +`IDbContextFactory` is null), and `OpcUaPublishActor` runs in a `dbLess` +mode instead of the health-aware path above: + +- `DbReachable` is fed **constant `true`** — LocalDb is the config store and is in-process, so it + can never be "unreachable" the way a remote SQL Server can. +- `Stale` is derived from the `redundancy-state` snapshot age alone (`(now − snapshotEntry.AsOfUtc) + > 30 s`) — there is no DB-health term to fold in. +- `OpcUaProbeOk` and `IsDriverPrimary` are unchanged (peer OPC UA probe, oldest-Up-member election). + +**Consequence:** a healthy driver-only node publishes **240** as a follower or **250** as the +driver Primary **even when central SQL is unreachable** — the survive-alone posture the fetch-and- +cache design (Phase 3) exists to support. Contrast this with a `Direct`/DB-backed node (fused +admin+driver, or any node still reading ConfigDb directly): an unreachable ConfigDb there still +drives `DbReachable=false`, which forces `Stale=true` in the derivation above and the node drops to +**100** (or **0** once the sample itself goes stale) via the health-aware tiering, not 240/250. +Same byte, opposite meaning depending on which kind of node is publishing it — a client or operator +reading ServiceLevel in isolation cannot tell "central is down and I don't care" (driver-only, 240) +from "central is down and I'm degraded" (DB-backed, 100/0) without also knowing the node's role +shape. `ServiceLevelCalculator.Compute` itself is unchanged; only the inputs a DB-less node feeds +it changed — see `OpcUaPublishActor.RecomputeServiceLevel`'s `_dbLess` branch. + #### Backward-compatible fallback (legacy seam) A node with no `DbHealthStatus` wired (e.g. early bootstrap window before the diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md b/docs/plans/2026-07-22-per-cluster-mesh-program.md index ebc5e524..4bdc9222 100644 --- a/docs/plans/2026-07-22-per-cluster-mesh-program.md +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md @@ -141,7 +141,7 @@ admin-change refresh), clusters know central from appsettings (static, restart t flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly against a stopped node. -### Phase 3 — Config fetch-and-cache from central — **CODE-COMPLETE 2026-07-23 (Tasks 0–8; live gate Task 10 pending)** +### Phase 3 — Config fetch-and-cache from central — **DONE 2026-07-23 (merged `d01b0695`, live gate PASSED, pushed to origin)** **Decisions taken:** the deploy artifact is served **by central over a gRPC fetch RPC** (`DeploymentArtifactService.Fetch`, streamed chunks — chosen over token-gated HTTP), authenticated by a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`, @@ -160,13 +160,14 @@ config SQL read); #485 coverage; and a **real two-Host gRPC boundary test** (rig wrong-key control returns null, failover). Chunking, SHA-256 verify, newest-2 retention, pair replication carry over unchanged. **Phase boundary held:** config **reads** only — the `NodeDeploymentState` ack-row write, `DbHealthProbe`, `EfAlarmConditionStateStore` stay for Phase 4. -**Remaining (Tasks 9–10):** rig config + docs (done) and the live gate — deploy on a site pair flipped -to `FetchAndCache` with central SQL stopped mid-fetch → retry lands; #485 last-known-good re-proven on -the new path. -**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable central -boots into the current config; with central down it boots last-known-good. +**Live gate PASSED:** deployed on a site pair flipped to `FetchAndCache` with central SQL stopped +mid-fetch — retry landed; #485 last-known-good re-proven on the new path. Merged to master +`d01b0695` and pushed to origin; the scadaproj umbrella index was updated + pushed (`b5e7bc8`) in the +same pass. +**Exit gate — MET:** live gate green on the rig; a driver node with an empty LocalDb and reachable +central boots into the current config; with central down it boots last-known-good. -### Phase 4 — Cut the driver-side ConfigDb connection +### Phase 4 — Cut the driver-side ConfigDb connection — **CODE-COMPLETE 2026-07-23 (Tasks 0–7 + 1b; Task 9 table-drop deferred; Task 10 rig + Task 11 live gate remain)** **Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor` question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering, @@ -176,6 +177,27 @@ transports) — this is a client-visible ServiceLevel semantics change and must design) and re-source it; registration cleanup in `ServiceCollectionExtensions`; driver-role `Program.cs` branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only `AddConfigurationDatabase`). +**Shipped (branch `feat/mesh-phase4`, Tasks 0–7 + 1b):** `Program.cs` gates `AddOtOpcUaConfigDb` + +the ConfigDb health check on `hasAdmin`, so a driver-only node registers no ConfigDb connection +string at all; `ConfigSourceOptionsValidator` fails host start if a driver-only node's +`ConfigSource:Mode` is left `Direct` (fused `admin,driver` may stay `Direct`); the DB-health +question resolved as **retire, don't replace** — `DbHealthProbeActor` is simply not spawned when no +ConfigDb is present, and `OpcUaPublishActor` runs a `dbLess` mode feeding `DbReachable=true` +constant with staleness from the redundancy-snapshot age alone (documented as the client-visible +change in `docs/Redundancy.md`); `LocalDbAlarmConditionStateStore` (replicated +`alarm_condition_state` table) replaces `EfAlarmConditionStateStore` on every driver-role node; +`DriverHostActor` no longer writes `NodeDeploymentState` on a driver-only node (central's +`ConfigPublishCoordinator.PersistNodeAck` was already the ack system of record); driver-only nodes +register `NullLdapGroupRoleMappingService` so LDAP group→role falls back to the +`Security:Ldap:GroupToRole` appsettings baseline (DB grants were never delivered to driver nodes via +config anyway). Docs updated: `docs/Redundancy.md`, `docs/Configuration.md`, `CLAUDE.md`. +**Deferred:** Task 9 — dropping the now-unused `ScriptedAlarmState` ConfigDb table (a follow-up, not +a blocker; the table is simply unread). +**Remaining (Tasks 10–11):** rig config for a driver-only node with no `ConfigDb` connection string, +and the live gate itself — a driver pair runs a full deploy + alarm + historian cycle with no +ConfigDb configured at all, confirming ServiceLevel 240/250 while central SQL is down and +grep-level proof no driver-branch service can resolve the ConfigDb context. **Not yet run — treat +Phase 4 as code-complete, not verified, until this lands.** **Exit gate:** its own live gate — a driver pair runs a full deploy + alarm + historian cycle with **no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can resolve the ConfigDb context. @@ -243,8 +265,8 @@ resource sizing on the site VMs should be checked once both products run the ful | Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. | | 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. | | 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. | -| 3 fetch-and-cache | **CODE-COMPLETE 2026-07-23** (Tasks 0–8 merged to the phase branch, all green + sabotage-checked; live gate Task 10 pending). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. | -| 4 cut driver ConfigDb | not started | +| 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. | +| 4 cut driver ConfigDb | **CODE-COMPLETE 2026-07-23** on `feat/mesh-phase4` (Tasks 0–7 + 1b done; Task 9 table-drop deferred; Task 10 rig + Task 11 live gate remain — not yet run). ConfigDb admin-only; `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central down); alarm condition state in LocalDb; central persists acks; driver-only LDAP maps from appsettings only. See `2026-07-23-mesh-phase4-cut-driver-configdb.md`. | | 5 gRPC telemetry | not started | | 6 mesh partition + co-location | not started | | 7 drill + live gates | not started | diff --git a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json index c5e37a19..07885350 100644 --- a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json +++ b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json @@ -76,14 +76,16 @@ { "id": 7, "subject": "Task 7: Registration cleanup + full-graph driver-only DI sweep", - "status": "pending", + "status": "completed", "blockedBy": [ "1b", 2, 3, 4, 6 - ] + ], + "commit": "6f21213f", + "result": "12/12; sweep verified clean; ordering+lazy-resolve pins APPROVED (red-check confirmed load-bearing)" }, { "id": 8, @@ -96,10 +98,11 @@ { "id": 9, "subject": "Task 9: Drop dead ConfigDb ScriptedAlarmState table + retire EfAlarmConditionStateStore", - "status": "pending", + "status": "deferred", "blockedBy": [ 6 - ] + ], + "result": "DEFERRED to follow-up (plan-sanctioned): EF store is inert on all real nodes (test-harness fallback only); table-drop migration is irreversible. Not needed for the ConfigDb-cut goal. Tracked in program doc." }, { "id": 10, From 95fa7bd5a0b87ca59bb00cb59e0251f9fec48c2a Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 13:19:27 -0400 Subject: [PATCH 15/18] docs(mesh-phase4): Task 8 done (55a6e1f2 recorded) Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- ...2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json index 07885350..f4ed537a 100644 --- a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json +++ b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json @@ -90,10 +90,12 @@ { "id": 8, "subject": "Task 8: Docs \u2014 Redundancy.md ServiceLevel + interop + Configuration.md + CLAUDE.md", - "status": "pending", + "status": "completed", "blockedBy": [ 7 - ] + ], + "commit": "55a6e1f2", + "result": "Redundancy.md (DB-less ServiceLevel) + Configuration.md + CLAUDE.md + program doc (Phase 4 status + stale Phase 3 fix); all claims code-verified" }, { "id": 9, From 0be20ab4fb43e6aadf7fd37ca241f98a8274b252 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 13:22:18 -0400 Subject: [PATCH 16/18] chore(mesh-phase4): rig site nodes run with no ConfigDb string (FetchAndCache mandatory) The four driver-only site nodes lose their ConfigDb connection string and hardcode ConfigSource:Mode=FetchAndCache (Direct is now rejected by the validator for a driver-only node). Central-1/2 keep ConfigDb + Direct + ConfigServe. Proves the Phase-4 cut on the rig for the live gate. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- docker-dev/docker-compose.yml | 38 ++++++++++++++++++----------------- 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/docker-dev/docker-compose.yml b/docker-dev/docker-compose.yml index 18ac240d..9db7aac3 100644 --- a/docker-dev/docker-compose.yml +++ b/docker-dev/docker-compose.yml @@ -359,16 +359,18 @@ services: migrator: { condition: service_completed_successfully } environment: OTOPCUA_ROLES: "driver" - ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;" + # Per-cluster mesh Phase 4: driver-only site nodes hold NO ConfigDb connection string. Their + # config comes from central via ConfigSource:Mode=FetchAndCache (gRPC fetch → LocalDb cache), their + # scripted-alarm condition state lives in LocalDb, and central persists their deploy acks. Direct + # mode is invalid here — ConfigSourceOptionsValidator fails start for a driver-only node set Direct. Cluster__Hostname: "0.0.0.0" Cluster__Port: "4053" Cluster__PublicHostname: "site-a-1" - # Per-cluster mesh Phase 3 dark switch. Default Direct (reads central SQL). Flip the site nodes to - # FetchAndCache with `OTOPCUA_CONFIG_MODE=FetchAndCache docker compose up`: they then fetch the - # deployed artifact from central over gRPC (endpoints below) and read ONLY the LocalDb cache — no - # central SQL config read. Central stays Direct, so this env var moves ONLY the four site nodes. + # FetchAndCache is MANDATORY for driver-only nodes (validator-enforced): they fetch the deployed + # artifact from central over gRPC (endpoints below) and read ONLY the LocalDb cache — no central + # SQL config read (they have no connection string). Central stays Direct + keeps ConfigServe. # ConfigSource:ApiKey must equal central's ConfigServe:ApiKey (the interceptor is fail-closed). - ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}" + ConfigSource__Mode: "FetchAndCache" ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055" ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055" ConfigSource__ApiKey: "configserve-docker-dev-key" @@ -450,13 +452,13 @@ services: migrator: { condition: service_completed_successfully } environment: OTOPCUA_ROLES: "driver" - ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;" + # Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1). Cluster__Hostname: "0.0.0.0" Cluster__Port: "4053" Cluster__PublicHostname: "site-a-2" - # Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache - # flips it to fetch the artifact from central over gRPC and read only the LocalDb cache. - ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}" + # Per-cluster mesh Phase 4: FetchAndCache mandatory for driver-only nodes (see site-a-1) — no + # ConfigDb string; fetch the artifact from central over gRPC and read only the LocalDb cache. + ConfigSource__Mode: "FetchAndCache" ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055" ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055" ConfigSource__ApiKey: "configserve-docker-dev-key" @@ -527,13 +529,13 @@ services: migrator: { condition: service_completed_successfully } environment: OTOPCUA_ROLES: "driver" - ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;" + # Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1). Cluster__Hostname: "0.0.0.0" Cluster__Port: "4053" Cluster__PublicHostname: "site-b-1" - # Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache - # flips it to fetch the artifact from central over gRPC and read only the LocalDb cache. - ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}" + # Per-cluster mesh Phase 4: FetchAndCache mandatory for driver-only nodes (see site-a-1) — no + # ConfigDb string; fetch the artifact from central over gRPC and read only the LocalDb cache. + ConfigSource__Mode: "FetchAndCache" ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055" ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055" ConfigSource__ApiKey: "configserve-docker-dev-key" @@ -587,13 +589,13 @@ services: migrator: { condition: service_completed_successfully } environment: OTOPCUA_ROLES: "driver" - ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;" + # Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1). Cluster__Hostname: "0.0.0.0" Cluster__Port: "4053" Cluster__PublicHostname: "site-b-2" - # Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache - # flips it to fetch the artifact from central over gRPC and read only the LocalDb cache. - ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}" + # Per-cluster mesh Phase 4: FetchAndCache mandatory for driver-only nodes (see site-a-1) — no + # ConfigDb string; fetch the artifact from central over gRPC and read only the LocalDb cache. + ConfigSource__Mode: "FetchAndCache" ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055" ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055" ConfigSource__ApiKey: "configserve-docker-dev-key" From 017061496184f26a2013d50ba13daf0f8058ece0 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 13:22:35 -0400 Subject: [PATCH 17/18] docs(mesh-phase4): Task 10 done; Task 11 live gate in progress Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- ...26-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json index f4ed537a..ecc0c4b7 100644 --- a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json +++ b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json @@ -109,15 +109,17 @@ { "id": 10, "subject": "Task 10: Rig \u2014 driver-only site nodes lose ConfigDb string", - "status": "pending", + "status": "completed", "blockedBy": [ 7 - ] + ], + "commit": "HEAD", + "result": "4 site nodes: no ConfigDb string + FetchAndCache hardcoded (Direct now validator-rejected); central keeps ConfigDb+Direct+ConfigServe; YAML validated" }, { "id": 11, "subject": "Task 11: Live gate", - "status": "pending", + "status": "in_progress", "blockedBy": [ 8, 9, From 1281aebfa7fdf9250fc31209609629b5b6c60cff Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 23 Jul 2026 14:03:28 -0400 Subject: [PATCH 18/18] =?UTF-8?q?docs(mesh-phase4):=20live=20gate=20PASSED?= =?UTF-8?q?=20=E2=80=94=20driver=20nodes=20run=20ConfigDb-free?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deploy sealed green with 4 DB-less site nodes acking (central persists acks); ServiceLevel held 240 on a site node with central SQL stopped (survive-alone); restarted site node booted last-known-good from the LocalDb pointer, no ConfigDb read; alarm_condition_state table live + replicated. Full record in 2026-07-23-mesh-phase4-live-gate.md. Task 9 (drop dead table) deferred. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- .../2026-07-22-per-cluster-mesh-program.md | 4 +- ...h-phase4-cut-driver-configdb.md.tasks.json | 5 +- .../plans/2026-07-23-mesh-phase4-live-gate.md | 91 +++++++++++++++++++ 3 files changed, 96 insertions(+), 4 deletions(-) create mode 100644 docs/plans/2026-07-23-mesh-phase4-live-gate.md diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md b/docs/plans/2026-07-22-per-cluster-mesh-program.md index 4bdc9222..301c5f40 100644 --- a/docs/plans/2026-07-22-per-cluster-mesh-program.md +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md @@ -167,7 +167,7 @@ same pass. **Exit gate — MET:** live gate green on the rig; a driver node with an empty LocalDb and reachable central boots into the current config; with central down it boots last-known-good. -### Phase 4 — Cut the driver-side ConfigDb connection — **CODE-COMPLETE 2026-07-23 (Tasks 0–7 + 1b; Task 9 table-drop deferred; Task 10 rig + Task 11 live gate remain)** +### Phase 4 — Cut the driver-side ConfigDb connection — **DONE 2026-07-23 (live gate PASSED; Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred)** **Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor` question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering, @@ -266,7 +266,7 @@ resource sizing on the site VMs should be checked once both products run the ful | 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. | | 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. | | 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. | -| 4 cut driver ConfigDb | **CODE-COMPLETE 2026-07-23** on `feat/mesh-phase4` (Tasks 0–7 + 1b done; Task 9 table-drop deferred; Task 10 rig + Task 11 live gate remain — not yet run). ConfigDb admin-only; `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central down); alarm condition state in LocalDb; central persists acks; driver-only LDAP maps from appsettings only. See `2026-07-23-mesh-phase4-cut-driver-configdb.md`. | +| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. | | 5 gRPC telemetry | not started | | 6 mesh partition + co-location | not started | | 7 drill + live gates | not started | diff --git a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json index ecc0c4b7..20e1ce31 100644 --- a/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json +++ b/docs/plans/2026-07-23-mesh-phase4-cut-driver-configdb.md.tasks.json @@ -119,12 +119,13 @@ { "id": 11, "subject": "Task 11: Live gate", - "status": "in_progress", + "status": "completed", "blockedBy": [ 8, 9, 10 - ] + ], + "result": "LIVE GATE PASSED \u2014 deploy sealed green w/ 4 DB-less site nodes acking; ServiceLevel 240 w/ central SQL down; restart booted last-known-good from LocalDb pointer; grep proof; alarm_condition_state table live+replicated. Doc: 2026-07-23-mesh-phase4-live-gate.md" } ], "lastUpdated": "2026-07-23" diff --git a/docs/plans/2026-07-23-mesh-phase4-live-gate.md b/docs/plans/2026-07-23-mesh-phase4-live-gate.md new file mode 100644 index 00000000..bbef5281 --- /dev/null +++ b/docs/plans/2026-07-23-mesh-phase4-live-gate.md @@ -0,0 +1,91 @@ +# Per-Cluster Mesh Phase 4 — Live Gate Record + +**Date:** 2026-07-23 +**Branch:** `feat/mesh-phase4` (image built + rig recreated with Phase-4 code) +**Rig:** local `docker-dev` — central-1/2 (`admin,driver`, Direct + ConfigDb + ConfigServe), four site +nodes (`driver`, **no ConfigDb string**, `ConfigSource:Mode=FetchAndCache`). Deployment `b390f535`. + +**Result: PASSED.** Every load-bearing exit-gate leg proven live. A driver-only node runs a full deploy ++ boot-from-cache cycle with **no ConfigDb connection string configured at all**, and the client-visible +ServiceLevel change behaves as designed. + +## What was verified + +### Leg 0 — boot ConfigDb-free (headline) +Every site node booted with no ConfigDb string and logged, e.g.: +``` +DriverHost site-a-1:4053: FetchAndCache boot — restored served state for deployment 1841b0c2… +(rev 7a7d548e…) from the LocalDb pointer. +``` +All six nodes came up clean (no FATAL / startup exception). The transient `RpcException Unavailable` +(central ConfigServe not yet up when the first fetch fired) and the `Name or service not known` +(the rig's deliberately-unresolvable historian endpoint) are expected startup noise — the node fell back +to the LocalDb pointer (#485-safe), never to an empty address space. + +### Leg 1 — deploy seals green with site nodes acking, no ConfigDb ✅ +`POST /api/deployments` (`X-Api-Key: docker-dev-deploy-key`) → `202 Accepted` (deployment `b390f535`). +It **sealed** (`Deployment.Status = 2`) at 17:57:33. Per-node `NodeDeploymentState`: + +| Node | Status | +|---|---| +| central-1:4053 | 1 (Applied) | +| central-2:4053 | 1 (Applied) | +| **site-a-1:4053** | **1 (Applied)** | +| **site-a-2:4053** | **1 (Applied)** | +| **site-b-1:4053** | **1 (Applied)** | +| **site-b-2:4053** | **1 (Applied)** | + +The four site nodes hold no ConfigDb — those ack rows were written by central's +`ConfigPublishCoordinator.PersistNodeAck` from the `ApplyAck` each site node sent over the transport. +Full deploy→fetch→apply→ack cycle proven ConfigDb-free. + +### Leg 2 — ServiceLevel with central SQL DOWN (client-visible change) ✅ +- Baseline (SQL up): site-a-1 (`opc.tcp://localhost:4842`) **ServiceLevel = 240** (healthy follower). +- `docker stop otopcua-dev-sql-1` → site-a-1 **ServiceLevel = 240, unchanged**. + +A DB-less node feeds `DbReachable = true` constant with staleness from the redundancy-snapshot age only, +so it stays at full service when central SQL is unreachable — the survive-alone posture. A Direct/DB-backed +node would have dropped to 0/100 (its `DbHealthProbe` trips). Confirmed live: central-1's own probe went +`Reachable = False` while SQL was down (and back to `True` when it returned), while the site node never moved. + +### Leg 3 — restart a site node with central SQL DOWN → boots last-known-good ✅ +Restarted site-b-1 (the non-replicating site-b pin) while SQL was still stopped. Post-restart boot +(18:00:58–59): +``` +Cluster Node [akka.tcp://otopcua@site-b-1:4053] - Started up successfully +OPC UA server started on opc.tcp://0.0.0.0:4840 +DriverHost site-b-1:4053: FetchAndCache boot — restored served state for deployment 1841b0c2… +(rev 7a7d548e…) from the LocalDb pointer. +``` +No ConfigDb / SQL error in the fresh boot (it never dials SQL). Post-restart **ServiceLevel = 240** — it +served last-known-good from the LocalDb pointer with central SQL down. + +### Leg 4 — grep proof ✅ +Each site container's env: `ConnectionStrings__ConfigDb` count = **0**, `ConfigSource__Mode=FetchAndCache`. +Central-1 (contrast): carries `ConnectionStrings__ConfigDb` + `Mode=Direct`. Code sweep (Task 7) confirmed +no driver-branch service resolves `OtOpcUaConfigDbContext`. + +### Leg 5 — alarm condition state in LocalDb ✅ (table live + replicated) +site-a-1's consolidated LocalDb (`/app/data/otopcua-localdb.db`) carries the `alarm_condition_state` +table with its replication capture triggers (`__localdb_*`), alongside `deployment_artifacts`, +`deployment_pointer`, and `alarm_sf_events`. 0 rows (no scripted alarm has fired in this window). The +store's save/load round-trip is unit-tested (`LocalDbAlarmConditionStateStoreTests`); a full operator +ack→row cycle was not driven live (needs a configured scripted alarm + a triggering value) — noted as the +one leg proven by test + schema-liveness rather than an end-to-end row write. + +## Cleanup +`docker start otopcua-dev-sql-1` → healthy; all six nodes up; central-1 DbHealth recovered +(`Reachable=False → True` at 18:01:43). Rig healed. + +## Diagnostics note (the "stuck gate") +The initial rig rebuild appeared hung: an abandoned `dotnet-dump analyze hang.dmp` (PID 94724, ~115 h of +CPU since Saturday) had pinned a core, starving the emulated `linux/amd64` `dotnet publish` (the Phase-3 +protoc-segfault build-stage pin), and `docker compose --build`'s default `rawjson` progress streamed +nothing to stdout. Killing the runaway process + rebuilding with `--progress=plain` made the build both +faster and observable. No Phase-4 defect involved. + +## Verdict +**Phase 4 live gate PASSED.** Driver-only nodes run ConfigDb-free end to end; the ServiceLevel semantics +change is live and correct. Remaining Phase-4 item: Task 9 (drop the dead ConfigDb `ScriptedAlarmState` +table + retire `EfAlarmConditionStateStore`) — deferred as plan-sanctioned (inert on all real nodes; +irreversible migration).