Merge docs/track-deferred-items: track deferred work as issues; fix stale limitation

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

CLAUDE.md

@@ -219,6 +219,24 @@ The server supports configurable OPC UA transport security via the `OpcUa:Enable
 
 The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide.
 
+## LocalDb pair-local config cache (Phase 1)
+
+Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
+LiteDB `LocalCache` subsystem was deleted as superseded). Phase-1 scope: it caches the
+**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a
+driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes
+the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`,
+in `Runtime/Deployment/`) and reads it as a boot fallback, flagging running-from-cache. Wired by
+`AddOtOpcUaLocalDb` in the `hasDriver` branch only (`Host/Configuration/LocalDbRegistration.cs` +
+`LocalDbSetup.OnReady`, whose DDL→`RegisterReplicated` order is load-bearing); **admin-only nodes
+never register it.** The cache can optionally replicate to the node's redundant pair peer over the
+library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAuthInterceptor`), gated
+on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the
+docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is
+the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret
+exception). See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
+cache), and the runbook `docs/operations/2026-07-20-localdb-pair-replication.md`.
+
 ## LDAP Authentication
 
 The server uses LDAP-based user authentication via the `Security:Ldap` section in `appsettings.json`. When enabled, credentials are validated by LDAP bind against a GLAuth server, and LDAP group membership maps to OPC UA permissions: `ReadOnly` (browse/read), `WriteOperate` (write FreeAccess/Operate attributes), `WriteTune` (write Tune attributes), `WriteConfigure` (write Configure attributes), `AlarmAck` (alarm acknowledgment). `LdapOpcUaUserAuthenticator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`) implements `IOpcUaUserAuthenticator`, delegating the LDAP bind + group lookup to `OtOpcUaLdapAuthService` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs`, an `ILdapAuthService`). See `docs/security.md` for the full guide.
@@ -397,11 +415,16 @@ The `AlarmHistorian` section's old Wonderware connection keys (`Host`/`Port`/`Us
 were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from
 `ServerHistorian`. See `docs/Historian.md` for the full guide.
 
-### KNOWN LIMITATION 1 — live-validation gate (do before merging/trusting the cutover)
+### Live-validation gate — PASSED (was "KNOWN LIMITATION 1")
 
-The cutover is code-complete but **must be live-validated against a real gateway** (VPN to
-`wonder-sql-vd03`, gateway running the prerequisites above) before it is merged or trusted. Run the
-env-gated suite:
+**No longer a limitation.** The cutover was live-validated against the real gateway on
+`wonder-sql-vd03`: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a *confirmed
+protocol limitation* (the captured `CM_EVENT` wire never carries `SourceName`, so the gateway cannot
+populate `Source_Object` — not a fixable bug). Evidence:
+`docs/plans/2026-06-27-otopcua-historian-followups.md`.
+
+Kept here because it is the recipe to **re-run** the gate after any gateway or backend change (VPN to
+`wonder-sql-vd03`, gateway running the prerequisites above):
 
 ```bash
 export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222   # absolute gateway URI; absent ⇒ all live tests skip
@@ -416,7 +439,7 @@ dotnet test --filter "Category=LiveIntegration"
 The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the
 gate the operator runs on the VPN before trusting the cutover.
 
-### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending
+### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending (issue #491)
 
 The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer +
 meters). It is spawned with an **empty *initial* ref set** (`Array.Empty<string>()` in
@@ -430,4 +453,4 @@ deployed set of historized value tags from the first deploy onward (the feed is
 path is code-complete.** What remains is the **live gate**: an end-to-end verification against a real gateway
 (deploy → dependency-mux value delta → outbox append → `WriteLiveValues`) plus a restart-convergence test
 proving the recorder re-registers the deployed set after a process restart. Until that live verification runs,
-treat continuous value-capture as unproven-in-production rather than absent.
+treat continuous value-capture as unproven-in-production rather than absent. Tracked as **issue #491**.

Directory.Packages.props

@@ -29,10 +29,10 @@
     <PackageVersion Include="coverlet.collector" Version="6.0.4" />
     <PackageVersion Include="FluentAssertions" Version="8.3.0" />
     <PackageVersion Include="Google.Protobuf" Version="3.34.1" />
+    <PackageVersion Include="Grpc.AspNetCore" Version="2.76.0" />
     <PackageVersion Include="Grpc.Core.Api" Version="2.76.0" />
     <PackageVersion Include="Grpc.Net.Client" Version="2.76.0" />
     <PackageVersion Include="libplctag" Version="1.5.2" />
-    <PackageVersion Include="LiteDB" Version="5.0.21" />
     <PackageVersion Include="MessagePack" Version="2.5.301" />
     <PackageVersion Include="Microsoft.AspNetCore.Authorization" Version="10.0.7" />
     <PackageVersion Include="Microsoft.AspNetCore.DataProtection" Version="10.0.7" />
@@ -122,6 +122,13 @@
     <PackageVersion Include="xunit.runner.visualstudio" Version="3.0.2" />
     <PackageVersion Include="xunit.v3" Version="1.1.0" />
     <PackageVersion Include="ZB.MOM.WW.Health" Version="0.1.0" />
+    <!--
+      LocalDb: embedded SQLite node-local cache + optional 2-node gRPC replication. The core
+      package floors the native SQLitePCLRaw.lib.e_sqlite3 at 2.1.12, so consumers inherit the
+      CVE-2025-6965-fixed native binary without needing the surgical pin below.
+    -->
+    <PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.3" />
+    <PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.3" />
     <PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.1.0" />
     <PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.1.0" />
     <PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
@@ -129,9 +136,13 @@
     <PackageVersion Include="ZB.MOM.WW.MxGateway.Client" Version="0.1.1" />
     <PackageVersion Include="ZB.MOM.WW.MxGateway.Contracts" Version="0.1.1" />
     <PackageVersion Include="ZB.MOM.WW.Configuration" Version="0.1.0" />
-    <PackageVersion Include="ZB.MOM.WW.Auth.Abstractions" Version="0.1.1" />
-    <PackageVersion Include="ZB.MOM.WW.Auth.Ldap" Version="0.1.1" />
-    <PackageVersion Include="ZB.MOM.WW.Auth.AspNetCore" Version="0.1.1" />
+    <PackageVersion Include="ZB.MOM.WW.Auth.Abstractions" Version="0.1.5" />
+    <PackageVersion Include="ZB.MOM.WW.Auth.Ldap" Version="0.1.5" />
+    <PackageVersion Include="ZB.MOM.WW.Auth.AspNetCore" Version="0.1.5" />
+    <PackageVersion Include="ZB.MOM.WW.Secrets" Version="0.2.3" />
+    <PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.3" />
+    <PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.3" />
+    <PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" Version="0.2.3" />
     <PackageVersion Include="ZB.MOM.WW.Audit" Version="0.1.0" />
     <PackageVersion Include="ZB.MOM.WW.Theme" Version="0.3.1" />
     <PackageVersion Include="ZB.MOM.WW.HistorianGateway.Client" Version="0.3.0" />

NuGet.config

@@ -25,6 +25,10 @@
       <package pattern="ZB.MOM.WW.Theme" />
       <package pattern="ZB.MOM.WW.HistorianGateway.Contracts" />
       <package pattern="ZB.MOM.WW.HistorianGateway.Client" />
+      <package pattern="ZB.MOM.WW.Secrets" />
+      <package pattern="ZB.MOM.WW.Secrets.*" />
+      <package pattern="ZB.MOM.WW.LocalDb" />
+      <package pattern="ZB.MOM.WW.LocalDb.*" />
     </packageSource>
   </packageSourceMapping>
 </configuration>

docker-dev/docker-compose.yml

@@ -47,6 +47,29 @@
 
 name: otopcua-dev
 
+# ── Shared secret-replication env (all six host nodes) ──────────────────────────
+# Akka peer-to-peer clustered secret replication rides the existing `otopcua` Akka
+# mesh (DistributedPubSub topic `zb-mom-ww-secrets`) across all six host nodes. Two
+# hard requirements are wired here, identically, on every node:
+#   1. Secrets__Replication__Enabled=true  → SecretsRegistration.AddOtOpcUaSecrets
+#      switches from plain AddZbSecrets to AddZbSecretsAkkaReplication + the eager
+#      SecretReplicationStarter hosted service, so every node joins anti-entropy.
+#   2. ZB_SECRETS_MASTER_KEY  → the ONE shared 32-byte base64 KEK. Every node MUST
+#      carry the SAME key or peers fail closed decrypting each other's rows
+#      (kek_id mismatch). appsettings.json binds Secrets:MasterKey:Source=Environment
+#      / EnvVarName=ZB_SECRETS_MASTER_KEY, so this env var IS the KEK on every node.
+# The KEK below is a committed DEV-ONLY default (matches the rig's zero-operator-step
+# philosophy, like the committed dev SQL password). Override for a fresh key with:
+#   OTOPCUA_SECRETS_KEK=$(openssl rand -base64 32) docker compose ... up -d
+# NEVER reuse this key outside this local dev rig.
+# AnnounceInterval is shortened to 5s (default 30s) so anti-entropy convergence is
+# observable quickly during dev exercise. Each node keeps its own local SQLite store
+# (Secrets:SqlitePath=otopcua-secrets.db, per-container); replication syncs them.
+x-secrets-env: &secrets-env
+  Secrets__Replication__Enabled: "true"
+  Secrets__Replication__AnnounceInterval: "00:00:05"
+  ZB_SECRETS_MASTER_KEY: "${OTOPCUA_SECRETS_KEK:-ZYGhIX0luS/XsevpCB2W18jYHMcqO6AjM9oXy+T6Zp4=}"
+
 services:
 
   sql:
@@ -145,6 +168,7 @@ services:
       sql: { condition: service_healthy }
       migrator: { condition: service_completed_successfully }
     environment:
+      <<: *secrets-env
       OTOPCUA_ROLES: "admin,driver"
       ASPNETCORE_URLS: "http://+:9000"
       ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
@@ -200,8 +224,15 @@ services:
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # Consolidated pair-local config cache (ZB.MOM.WW.LocalDb). Every driver-role node
+      # (central is admin,driver) gets one, on a per-node named volume so the cache survives
+      # container recreates. The central pair leaves replication OFF (default) — only the
+      # site-a pair replicates, as the enablement demo; site-b is the default-OFF pin.
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4840:4840"
+    volumes:
+      - otopcua-localdb-central-1:/app/data
 
   central-2:
     <<: *otopcua-host
@@ -210,6 +241,7 @@ services:
       central-1: { condition: service_started }
       migrator: { condition: service_completed_successfully }
     environment:
+      <<: *secrets-env
       OTOPCUA_ROLES: "admin,driver"
       ASPNETCORE_URLS: "http://+:9000"
       ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
@@ -258,8 +290,12 @@ services:
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # Pair-local config cache; central pair leaves replication OFF (see central-1).
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4841:4840"
+    volumes:
+      - otopcua-localdb-central-2:/app/data
 
   # ── Site A cluster (2-node driver-only) ─────────────────────────────────────
   # Driver-only members of the single mesh, scoped to SITE-A by ClusterId. No UI,
@@ -280,6 +316,7 @@ services:
       Cluster__PublicHostname: "site-a-1"
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
+      <<: *secrets-env
       # Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
       # mem_reservation are inherited from the *otopcua-host anchor.
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
@@ -287,8 +324,24 @@ services:
       # Resolved at runtime by GalaxyDriver.ResolveApiKey when a DriverInstance's
       # Gateway.ApiKeySecretRef = "env:GALAXY_MXGW_API_KEY".
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # Pair-local config cache + REPLICATION ON — the site-a pair is the enablement demo.
+      # site-a-1 is the initiator: it binds the h2c sync listener on 9001 AND dials the peer.
+      # Setting SyncListenPort makes Program.cs add a dedicated Http2-only Kestrel listener and
+      # re-bind the primary HTTP port (an explicit Listen* otherwise discards ASPNETCORE_URLS).
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
+      LocalDb__SyncListenPort: "9001"
+      LocalDb__Replication__PeerAddress: "http://site-a-2:9001"
+      # DEV-ONLY committed key (rig philosophy — like the SQL password + secrets KEK above). It
+      # MUST be byte-identical on both nodes: the interceptor is fail-closed, so any mismatch
+      # silently stops the pair converging. NEVER reuse this key outside this local dev rig.
+      LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
+      # Row-count batching against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so
+      # 16 × 171 KB ≈ 2.7 MB stays under the cap with headroom.
+      LocalDb__Replication__MaxBatchSize: "16"
     ports:
       - "4842:4840"
+    volumes:
+      - otopcua-localdb-site-a-1:/app/data
 
   site-a-2:
     <<: *otopcua-host
@@ -304,11 +357,22 @@ services:
       Cluster__PublicHostname: "site-a-2"
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
+      <<: *secrets-env
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # site-a-2 is the passive half of the replicating pair: it binds the sync listener on 9001
+      # but does NOT dial (no PeerAddress). The replication stream is bidirectional, so a-1's
+      # single dial carries both directions. Same key + MaxBatchSize as a-1 (byte-identical key
+      # is mandatory — the interceptor fail-closes on a mismatch).
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
+      LocalDb__SyncListenPort: "9001"
+      LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
+      LocalDb__Replication__MaxBatchSize: "16"
     ports:
       - "4843:4840"
+    volumes:
+      - otopcua-localdb-site-a-2:/app/data
 
   # ── Site B cluster (2-node driver-only) ─────────────────────────────────────
 
@@ -326,11 +390,18 @@ services:
       Cluster__PublicHostname: "site-b-1"
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
+      <<: *secrets-env
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # site-b is the default-OFF pin: it gets the pair-local cache but NO SyncListenPort and NO
+      # replication config, so no sync listener binds and ISyncStatus stays disconnected/Healthy.
+      # This is what the live gate's check 8 verifies — the cache works locally with replication off.
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4844:4840"
+    volumes:
+      - otopcua-localdb-site-b-1:/app/data
 
   site-b-2:
     <<: *otopcua-host
@@ -346,11 +417,16 @@ services:
       Cluster__PublicHostname: "site-b-2"
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
+      <<: *secrets-env
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # site-b default-OFF pin (see site-b-1): cache on, replication off.
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4845:4840"
+    volumes:
+      - otopcua-localdb-site-b-2:/app/data
 
   traefik:
     image: traefik:v3.1
@@ -371,3 +447,13 @@ services:
 volumes:
   # SQL Server data dir — persists the OtOpcUa ConfigDb across container recreates.
   otopcua-mssql-data:
+  # Per-node pair-local config-cache (ZB.MOM.WW.LocalDb) files. One volume per node — the
+  # cache is node-local, not shared; it persists the cached deployment artifact (and, on the
+  # site-a pair, the replicated oplog state) across container recreates so a node can boot
+  # from cache after a restart even with central SQL down.
+  otopcua-localdb-central-1:
+  otopcua-localdb-central-2:
+  otopcua-localdb-site-a-1:
+  otopcua-localdb-site-a-2:
+  otopcua-localdb-site-b-1:
+  otopcua-localdb-site-b-2:

docs/AlarmTracking.md

@@ -51,6 +51,136 @@ same condition is wired as an **event notifier of every referencing equipment fo
   equipment paths**, and the AdminUI `/alerts` page shows **one row per condition** (primary
   identity RawPath + condition NodeId) with the equipment list as display metadata.
 
+## Condition event identity fields (what a client reads on the wire)
+
+Every condition event — native and scripted — carries the mandatory `BaseEventType` identity
+fields, assigned at materialize time in `OtOpcUaNodeManager.MaterialiseAlarmCondition`. The SDK
+does **not** synthesize them on this path (`Create` builds the children from the type definition
+but leaves them unset; `ReportEvent` / `InstanceStateSnapshot` copy children verbatim), so they
+are set explicitly. Leaving them unset shipped them as **null** on every event — see issues #473
+(the `BaseEventType` trio) and #475 (the `ConditionType` classification pair).
+
+| Field | Value | Notes |
+|---|---|---|
+| `EventType` | the **concrete** materialized type (`TypeDefinitionId`) | e.g. `OffNormalAlarmType`; falls back to `AlarmConditionType` for an unknown authored type. Readable as a *field*, not only via an `OfType` where-clause |
+| `SourceNode` | the condition's **own NodeId** — equal to `ConditionId` | The condition **is** the source: an alarm-bearing raw tag materializes only the condition, with no sibling value variable, so there is no other node to point at |
+| `SourceName` | the same identifying id string: **RawPath** (native) / **ScriptedAlarmId** (scripted) | Deliberately the *unique* id, **not** the leaf name |
+| `ConditionName` | the leaf / display name (e.g. `HR200`) | Where the short human-readable name lives |
+| `ConditionClassId` | always **`BaseConditionClassType`** | Part 9's "no condition class modelled" value. Unset shipped `NodeId.Null` (#475) |
+| `ConditionClassName` | always **`"BaseConditionClass"`** | Matches `ConditionClassId`. Unset shipped empty text (#475) |
+| `Quality` | the condition's **source-data quality** — native tracks the source's connectivity (`Good` / `Bad`); scripted takes the worst of its input tags' qualities (#478) | A pure annotation; never alters Active/Acked/Retain. Unset shipped the accidentally-Good default (#477) — see below |
+
+**Why `BaseConditionClassType` and not `ProcessConditionClassType`.** We hold no per-alarm
+classification at the materialize seam, and `ConditionClassId` is a wire contract clients bucket on.
+`BaseConditionClassType` is the honest, spec-conformant report of *"this server does not model
+condition classes"* — it fixes the real defect (a null that breaks conformant clients) without
+asserting a classification we cannot back. `ProcessConditionClassType` — the SDK sample's pick —
+was rejected deliberately: it would be *actively wrong* for a Galaxy alarm whose upstream category is
+Safety or Diagnostics, trading a detectable null for an undetectable lie. Real per-alarm
+classification is a separate future feature: it needs the driver's alarm category, which today lives
+only on the runtime `AlarmEventArgs` transition, carried to the deploy-time authored composition that
+`MaterialiseAlarmCondition` sees. Until then the `IAlarmSource` doc comment claiming the category
+"maps to `ConditionClassName` downstream" describes an intent, not the implementation.
+
+**Why `SourceName` is the id, not the leaf name.** The leaf is ambiguous across devices (`HR200` on
+two PLCs collides) and is already carried by `ConditionName`, so the leaf-name option would add no
+identity while costing uniqueness. Carrying the id makes the `SourceNode`/`SourceName`/`ConditionId`
+triple mutually consistent and unique. This diverges from the loose OPC UA convention that
+`SourceName` mirrors the source node's BrowseName; the divergence is intentional.
+
+**Client guidance:** key on **`ConditionId`** — the condition node's own NodeId, which equals
+`SourceNode`, and whose identifier is the RawPath for native alarms and the ScriptedAlarmId for
+scripted ones. It is the identity ack/confirm/shelve route on. `SourceName` carries the same
+identifier and is unique, so it is safe to key on *by itself* — but do **not** compose it with
+`ConditionName` (`$"{SourceName}.{ConditionName}"`), because `SourceName` already ends in the
+condition's leaf name and the result stutters (`pymodbus/plc/HR200.HR200`).
+
+Wire-level guard: `NativeAlarmEventIdentityFieldDeliveryTests` asserts the three `BaseEventType`
+fields arrive populated on a real subscription using the standard `[EventType, SourceNode,
+SourceName, Time, Message, Severity]` select clause, and — in a second test with its own clause —
+that `ConditionClassId` / `ConditionClassName` do too. The two class fields are declared on
+`ConditionType`, **not** `BaseEventType`, so a client must select them against that type.
+`NodeManagerAlarmSourceFieldsTests` guards the node itself across both realms.
+
+> **Do not correlate live events to HistoryRead on `SourceName` — the two paths disagree.**
+> The HistoryRead *events* projection (`OtOpcUaNodeManager.ProjectEventField`) returns
+> `Variant.Null` for `EventType` / `SourceNode` **by design**: it projects from the historian's
+> `HistoricalEvent` rows, which do not carry them. It **does** project `SourceName` — but the
+> alarm-history writer stamps that field with the **EquipmentPath**
+> (`AlarmEventMapper`: `SourceName = alarm.EquipmentPath`), not the RawPath a live event carries.
+> So `SourceName` is the one field populated on both paths **with different values**, and it is not
+> a live↔history join key. Correlate on `ConditionId` / the RawPath instead. (Pre-existing; the
+> live-path fix above does not change the history path.)
+
+### Condition source-data Quality (#477)
+
+`ConditionType.Quality` reports the quality of the condition's source data. It was never assigned, and
+because `StatusCodes.Good == 0x00000000` an unassigned `StatusCode` **is** `Good` — so every condition
+reported `Good` unconditionally (a wrong *value*, not a null like #473/#475). A native alarm whose device
+went offline still read `Good`, so an operator could not tell *"genuinely inactive"* from *"we have lost
+contact and do not know"*.
+
+**How it is driven now (native alarms).** An alarm-bearing raw tag materializes a condition with **no
+sibling value variable**, so the value/quality path (`WriteValue`) never touches it, and a comms-lost
+driver emits **no alarm transitions** (the feed goes silent). The quality therefore comes from the
+**driver's connectivity**, out of band from alarm transitions:
+
+- `DriverInstanceActor` Tells its host `ConnectivityChanged(driverInstanceId, connected)` on every
+  transition into `Connected` (`true`) / `Reconnecting` (`false`).
+- `DriverHostActor.OnDriverConnectivityChanged` fans that out to **every** native condition the driver
+  owns as an `OpcUaPublishActor.AlarmQualityUpdate` (`Good` on connect, `Bad` on disconnect).
+- `OtOpcUaNodeManager.WriteAlarmQuality` sets **only** the condition's `Quality` and fires a Part 9
+  event **only on a quality-bucket change** — it never touches Active/Acked/Severity/Retain (an active
+  alarm that loses comms stays active). This is a dedicated path, *not* a full-snapshot re-projection, so
+  it cannot clobber a condition's severity/message and works for a condition that never fired a transition.
+- A freshly materialized native condition starts `BadWaitingForInitialData` (the "no driver data yet"
+  convention value variables use); the first `Connected` confirms it `Good`.
+- The connectivity annotation is **ungated by redundancy role** (a Secondary keeps its condition quality
+  warm for failover) and publishes **no `/alerts` row** — driver comms health already has its own status
+  surface (`IDriverHealthPublisher`); a row per condition would be alarm-fatigue.
+
+**Scripted alarms (Layer 3, #478).** A scripted condition's state is computed from one or more input tags,
+so its `Quality` is the **worst** quality across those inputs at evaluation time ("can I trust this
+condition's state?") — mirroring the native OT semantic:
+
+- The mux now forwards each input's source quality (`DependencyValueChanged.Quality`), and the scripted host
+  pushes it into the engine's read cache (previously every mux value was treated as `Good`).
+- The `ScriptedAlarmEngine` computes the worst input quality each evaluation. A **real transition** carries
+  it on the emitted event → `ScriptedAlarmHostActor.ToSnapshot` projects it (so a transition fired while an
+  input is `Uncertain` does not clobber quality back to `Good`).
+- A **Bad input freezes the condition** (`AreInputsReady` holds its state — no transition), exactly like a
+  comms-lost native driver. So a quality-bucket change with no transition is emitted as
+  `EmissionKind.QualityChanged` and routed to the **same** dedicated `AlarmQualityUpdate → WriteAlarmQuality`
+  node path native uses (quality only, one Part 9 event on a bucket change, **no `/alerts` row**, no
+  historian write). `ScriptedAlarmSource` skips `QualityChanged` so it never fabricates a phantom
+  `IAlarmSource` event.
+- An input that has **not been published yet** (cold start) is *not* a quality signal (that is the readiness
+  guard's job) — it contributes `Good`, so scripted conditions don't flash `Bad` at every deploy. The first
+  actually-`Bad` published value flips the bucket and annotates.
+
+**Coverage boundary (#478 as shipped).** Scripted quality tracks input tags whose driver **publishes a
+data change carrying a Bad/Uncertain `StatusCode`** (e.g. an OpcUaClient input forwarding a server's
+per-item Bad). It does **not** yet cover a driver **comms loss**: a poll driver (Modbus/S7) whose device
+goes unreachable emits only `ConnectivityChanged` and goes *silent* on the value feed (see
+`DriverInstanceActor.Reconnecting`), so the scripted engine keeps the last-known Good value and the
+condition stays `Good`. Bridging driver connectivity into scripted inputs — the symmetric of the native
+`OnDriverConnectivityChanged` path above, plus resolving the null-value/cold-start asymmetry (a runtime
+`Bad` with a null value is currently indistinguishable from cold start and contributes `Good`) — is tracked
+as the Layer-4 follow-up (#481).
+
+Guards: `ScriptedAlarmEngineTests` (transition carries `Uncertain`; `Bad` input with no transition emits
+`QualityChanged(Bad)`; restore emits `QualityChanged(Good)`; unchanged bucket emits nothing),
+`ScriptedAlarmSourceTests.QualityChanged_emission_raises_no_alarm_event`,
+`DependencyMuxActorTests.Publish_quality_is_forwarded_on_DependencyValueChanged`, and
+`ScriptedAlarmHostActorTests` (`Bad_quality_dependency_publishes_AlarmQualityUpdate_and_no_alerts`,
+`Transition_snapshot_carries_worst_input_quality`).
+
+Wire-level guard: `NativeAlarmEventIdentityFieldDeliveryTests.Condition_event_Quality_tracks_source_connectivity_on_the_wire`
+subscribes with a `[Quality, Message]` clause (Quality is declared on `ConditionType`) and asserts a
+healthy source reports `Good`, a comms-lost source reports non-`Good`, and recovery returns to `Good` — on
+a real client subscription. `NodeManagerAlarmSourceFieldsTests` guards the node itself + the no-clobber /
+unknown-node-no-op invariants.
+
 ## Galaxy driver path (driver-native)
 
 Restored in PR B.2 of the epic. `GalaxyDriver` implements

docs/Configuration.md

@@ -146,6 +146,28 @@ key must carry the scopes `historian:read`, `historian:write`, `historian:tags:w
 [`docs/Historian.md`](Historian.md) for the full key reference, the migration note (old Wonderware keys →
 gateway keys), and the deployment prerequisites.
 
+### `LocalDb` (pair-local config cache — driver role only)
+
+Driver-role nodes keep a consolidated **`ZB.MOM.WW.LocalDb`** SQLite database that caches the deployed
+configuration artifact (chunked) so the node can **boot from cache when central SQL is unreachable**. The
+section is bound by `AddOtOpcUaLocalDb` (`Host/Configuration/LocalDbRegistration.cs`) inside the `hasDriver`
+branch — **admin-only nodes never register it, and never require `LocalDb:Path`**. Storage is unconditional;
+replication stays inert until the sync port + peer are set.
+
+| Key | Default | Meaning |
+|---|---|---|
+| `LocalDb:Path` | `./data/otopcua-localdb.db` | SQLite file path. `ValidateOnStart`-required once `AddZbLocalDb` runs (driver nodes only). In containers put it on a durable volume. |
+| `LocalDb:SyncListenPort` | `0` | `0` = replication off (no sync listener). A non-zero port binds a dedicated **h2c** sync listener; set the **same** port on both pair nodes. Setting it makes the Host add an explicit Kestrel `Listen*` and re-bind the primary HTTP port (an explicit `Listen*` otherwise discards `ASPNETCORE_URLS`). |
+| `LocalDb:Replication:PeerAddress` | `""` | The peer this node **dials** (`http://<peer>:<port>`). Set on the initiator only; leave empty on the passive node. The stream is bidirectional. |
+| `LocalDb:Replication:ApiKey` | `""` | Bearer token for the sync stream. **Fail-closed:** no key ⇒ the passive endpoint refuses everything; a mismatch ⇒ the pair silently stops converging. **Must be byte-identical on both nodes.** Supply via `${secret:...}` / env — never a cleartext literal in production. |
+| `LocalDb:Replication:MaxBatchSize` | (library default) | Rows per replication batch — **row-count-only** against gRPC's 4 MB cap. `16` for the artifact cache (chunk rows ≈171 KB, so 16 × 171 KB ≈ 2.7 MB). |
+
+Replication is **default-OFF** across the fleet; it is enabled per-pair as an opt-in. See
+[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
+for the enablement runbook (ApiKey handling, stop/start-together rule, tombstone-retention window, and the
+never-`sqlite3`-a-live-WAL-DB inspection rules) and the "pair-local config cache" section of
+[`docs/Redundancy.md`](Redundancy.md) for what boot-from-cache does and does not cover.
+
 ---
 
 ## Environment variables

docs/Redundancy.md

@@ -226,6 +226,35 @@ Net effect: each alarm transition appears **once** on `/alerts` and would histor
 
 See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals.
 
+## Pair-local config cache (LocalDb — Phase 1)
+
+Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated
+[`ZB.MOM.WW.LocalDb`](../CLAUDE.md) SQLite database that caches the **deployed-configuration
+artifact** (chunked). Its job is a single failure mode redundancy does not otherwise cover: a driver
+node restarting into a **central-SQL-Server outage** would come up with no configuration at all. With
+the cache, it **boots from the last artifact it applied** instead, and logs a running-from-cache
+signal.
+
+- **What replicates.** The cache can *optionally* replicate to the node's redundant pair peer over
+  the LocalDb library's gRPC sync (HLC-stamped, last-writer-wins). Two tables replicate:
+  `deployment_artifacts` and `deployment_pointer`. Replication is **default-OFF and fail-closed** —
+  it stays inert until `LocalDb:SyncListenPort` + a peer are configured, and a missing/mismatched
+  `LocalDb:Replication:ApiKey` refuses or silently halts sync. It is enabled per-pair as an opt-in.
+- **The replicated-cache payoff.** In a pair with replication on, *either* node can be the one that
+  applied the latest deploy; the peer holds a byte-identical copy. So a node that restarts during a
+  central outage can boot the current config **even if it never applied that deploy itself** — its
+  peer did, and the row replicated.
+- **What it does NOT cover.** The cache is a **boot fallback for central-SQL outages only**, not a
+  standby data path and not a replacement for central. A **new deployment still requires central
+  SQL**; the cache is read only when the central fetch fails at startup, and the node resumes
+  fresh-config behavior once central returns. It does not replicate live tag values, alarms, or
+  historian data — only the config artifact. It is orthogonal to the ServiceLevel/primary-gate
+  logic: a node running from cache still participates in redundancy normally.
+
+Full detail: the enablement runbook at
+[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
+and the `LocalDb` section of [`docs/Configuration.md`](Configuration.md).
+
 ## Client-side failover
 
 The OtOpcUa Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` supports `-F` / `--failover-urls` for automatic client-side failover; for long-running subscriptions the CLI monitors session KeepAlive and reconnects to the next available server, recreating the subscription on the new endpoint. See [`Client.CLI.md`](Client.CLI.md).

docs/operations/2026-07-16-secrets-clustered-master-key.md

@@ -0,0 +1,177 @@
+# Secrets: Clustered Master-Key Posture (All Roles)
+
+## Purpose
+
+`ZB.MOM.WW.Secrets` resolves `${secret:...}` tokens in `appsettings.*.json` via a
+pre-host expander (`SecretReferenceExpander.ExpandConfigurationAsync`, wired in
+`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`) that runs at **every** OtOpcUa node
+boot, before the host is built. It reads rows from an envelope-encrypted SQLite
+store (`Secrets:SqlitePath`) unwrapped with a key-encryption key (KEK) sourced per
+`Secrets:MasterKey:Source`. The runtime `ISecretResolver` (`AddZbSecrets`) is also
+registered unconditionally, independent of node role.
+
+OtOpcUa is Akka.NET-clustered (`builder.Services.AddAkka("otopcua", (ab, sp) => { ... })`
+in `Program.cs`), with roles parsed by `RoleParser` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs`):
+`admin`, `driver`, `dev`. A node can carry any combination of these roles (e.g. a
+fused admin+driver node, or a driver-only node). This runbook covers what a
+production deployment needs so that secret resolution behaves identically on
+**every node regardless of role** — not just admin nodes. That matters here more
+than it might elsewhere: the pre-host expander and the runtime resolver both run
+unconditionally on driver-only nodes too, and driver-only nodes are the ones that
+will resolve Layer-B `DriverConfig` secret references (coming in Slice 2) at
+runtime, not just at boot. It does **not** change any code; it is an
+operations/deployment posture, delivered out-of-band from the committed config.
+
+## The two hard requirements
+
+For the pre-host expander (and the runtime resolver) to resolve the same
+plaintext secret on every node, no matter its role:
+
+1. **Identical KEK on every node.** All nodes — admin, driver, and any fused
+   combination — must unwrap the store with the exact same master key. A
+   per-node KEK (e.g. a per-box DPAPI-protected key) would make each node
+   decrypt every *other* node's ciphertext rows to garbage.
+2. **Identical store rows on every node.** All nodes must read the same SQLite
+   database (same file, or a replicated/shared copy with the same rows) — not
+   independently-seeded stores that happen to share a KEK.
+
+`ZB.MOM.WW.Secrets` ships a SQLite-only `ISecretStore` with a `NoOpSecretReplicator`
+— there is no built-in cross-node replication today. Meeting both requirements in
+production is a deployment concern, covered below.
+
+## Recommended interim posture (G-5)
+
+Until real replication exists (G-7, below), the recommended production posture is:
+
+- **`Secrets:MasterKey:Source = File`**, with `FilePath` pointing at a **read-only
+  key file that is identical on every node of every role** — a base64-encoded
+  32-byte key, generated out-of-band (e.g. `openssl rand -base64 32`), distributed
+  to each node's filesystem/secret-mount by the deployment tooling, and **never
+  committed** to the repo. Treat it with the same discipline as any other
+  production secret (restrictive file ACLs, no logging, rotated via a future
+  KEK-rotation runbook — not yet built).
+- **`Secrets:SqlitePath`** pointing at a **single shared or replicated volume**
+  that every node mounts (admin, driver, and dev/fused alike), so every node's
+  migrator opens and reads the same rows at boot.
+
+Writes to the store are rare and human-driven — an operator using the
+`/admin/secrets` UI (admin nodes only) or the `ZB.MOM.WW.Secrets` CLI — while
+reads happen on every node's boot and on the `ResolveCacheTtl` refresh cycle,
+regardless of role. The access pattern is read-mostly / effectively
+single-writer, which is what makes a shared SQLite volume viable as an interim
+posture (see caveat below).
+
+## How it's delivered (do NOT commit these values)
+
+The File-KEK + shared-store posture is supplied per-node at deployment time —
+**never** by editing the committed `appsettings.json` or the role-overlay files
+(`appsettings.admin.json`, `appsettings.driver.json`, `appsettings.admin-driver.json`).
+Those committed overlays are also consumed by local dev and the
+`TwoNodeClusterHarness` integration tests, so hardcoding a `Source=File` path
+into them would break every dev/test/CI boot. Two acceptable delivery
+mechanisms instead:
+
+**Option A — environment variable overrides** (Windows Service / NSSM env block,
+container `env_file`, etc.), applied identically on every node regardless of role:
+
+```
+# production deployment — do not commit to the dev appsettings
+Secrets__MasterKey__Source=File
+Secrets__MasterKey__FilePath=/run/secrets/otopcua-master.key
+Secrets__SqlitePath=/shared/secrets/otopcua-secrets.db
+```
+
+**Option B — a production-only config layer** that is *not* the committed dev
+base (e.g. an untracked `appsettings.Production.json` deployed alongside the
+binaries, or an orchestrator-injected config mount):
+
+```jsonc
+// production deployment — do not commit to the dev appsettings
+{
+  "Secrets": {
+    "MasterKey": {
+      "Source": "File",
+      "FilePath": "/run/secrets/otopcua-master.key"
+    },
+    "SqlitePath": "/shared/secrets/otopcua-secrets.db"
+  }
+}
+```
+
+Either way, the file/path referenced must exist and be identical on every node
+**before** that node boots — the pre-host expander runs unconditionally on every
+role and will throw (`SecretNotFoundException` / migration failure) if the store
+or key is missing.
+
+## Caveat: SQLite over a shared volume is not real replication
+
+SQLite's file-locking model does not tolerate concurrent multi-writer access well
+over network filesystems (SMB/NFS locking is unreliable, and even on a clustered
+block volume only one writer should be active at a time). The interim posture
+above is acceptable because:
+
+- Reads dominate (every node's boot + cache-refresh cycle, across every role).
+- Writes are rare, human-initiated, and effectively single-writer in practice
+  (an operator runs the CLI/UI against one admin node at a time).
+
+It is **not** a substitute for real replication, and it is not safe if multiple
+nodes attempt concurrent writes. Do not build automation that writes secrets
+from more than one node simultaneously.
+
+## Data Protection is independent — do not touch it here
+
+OtOpcUa's cookie/session protection already has its own clustered-key story:
+`AddDataProtection().PersistKeysToDbContext<OtOpcUaConfigDbContext>()`
+(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`),
+which shares the Data Protection key ring across every node via the existing
+ConfigDb. That mechanism is unrelated to `ZB.MOM.WW.Secrets`' envelope encryption
+(KEK + SQLite store) and must **not** be reconfigured as part of secrets-adoption
+work — doing so risks invalidating active sessions for an unrelated reason.
+
+It is, however, the model for where the *next* iteration of secret storage
+should go — see the G-7 hand-off below.
+
+## The G-7 hand-off
+
+The posture above is an interim, ops-only workaround. The long-term shape,
+tracked as **G-7** in `scadaproj/components/secrets/GAPS.md`, is a
+ConfigDb-backed `ISecretStore` that mirrors the pattern OtOpcUa already uses for
+the Data Protection key ring:
+
+```csharp
+services.AddDataProtection()
+    .PersistKeysToDbContext<OtOpcUaConfigDbContext>()
+    .SetApplicationName("OtOpcUa");
+```
+
+(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`).
+`OtOpcUaConfigDbContext` already gives every node — admin, driver, and dev/fused
+alike — a single MS SQL-backed source of truth for the Data Protection key ring;
+the secret store is the natural next tenant of that same shared database instead
+of a shared SQLite file. Building this requires new `ZB.MOM.WW.Secrets` library
+code (a ConfigDb-backed `ISecretStore` implementation) that does not exist yet,
+overlaps the G-7 tracking item, and is explicitly **deferred there** — it is not
+built as part of this cut. This runbook's shared-SQLite-volume posture is the
+bridge until G-7 lands.
+
+## Dev/test/default posture (unchanged)
+
+The committed default in `appsettings.json` is:
+
+```json
+"Secrets": {
+  "SqlitePath": "otopcua-secrets.db",
+  "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
+  "RunMigrationsOnStartup": true
+}
+```
+
+This is dev-safe: `Source=Environment` needs no filesystem key, and the SQLite
+path is relative to the working directory, so local dev, the role-overlay
+appsettings (`appsettings.admin.json`, `appsettings.driver.json`,
+`appsettings.admin-driver.json`), and the `TwoNodeClusterHarness` integration
+tests all boot cleanly with no external mount. The File-KEK + shared-volume
+posture in this runbook applies only to real clustered production deployments —
+it must never be baked into the committed dev/role-overlay base, because the
+expander runs unconditionally at every node boot (any role) and would break
+dev/CI if pointed at a nonexistent `/shared` mount.

docs/operations/2026-07-20-localdb-pair-replication.md

@@ -0,0 +1,136 @@
+# LocalDb pair replication — operations runbook
+
+> **Phase 1 scope.** Every driver-role OtOpcUa node keeps a consolidated
+> [`ZB.MOM.WW.LocalDb`](../../CLAUDE.md) SQLite database. Today it caches exactly one thing: the
+> **deployed-configuration artifact**, chunked, so a node can **boot from cache when central SQL
+> Server is unreachable**. That cache can *optionally* replicate to the node's redundant pair peer
+> over the library's gRPC sync. **Replication is default-OFF and fail-closed.** This runbook covers
+> enabling it, the operational rules that keep a pair converging, and the DB-inspection safety
+> rules.
+
+## What replicates, and what it buys you
+
+- **Replicated tables:** `deployment_artifacts` (the artifact, split into base64 chunks) and
+  `deployment_pointer` (one current-deployment pointer per cluster). Registered — in this order,
+  after the DDL — by `LocalDbSetup.OnReady`.
+- **The payoff:** in a redundant driver pair, *either* node can be the one that applied the latest
+  deploy and wrote the cache. With replication on, the peer holds a byte-identical copy. So a node
+  that restarts into a central-SQL outage can boot the current config **even if it never applied
+  that deploy itself** — its peer did, and the row replicated.
+- **Convergence model:** last-writer-wins per primary key, HLC-stamped. Both nodes also store
+  locally after every apply, so the pair converges to identical content regardless of which node
+  deployed. Retention (newest-2 deployments per cluster) prunes on one node and the deletes
+  replicate as tombstones.
+
+## Enabling replication on a pair
+
+Replication has two knobs, both under `LocalDb`:
+
+| Key | Role | Notes |
+|---|---|---|
+| `LocalDb:SyncListenPort` | binds the dedicated **h2c** sync listener | `0` (default) = no listener, replication off. Set the **same** non-zero port on **both** nodes of the pair. |
+| `LocalDb:Replication:PeerAddress` | the address this node **dials** | Set on **one** node (the initiator); leave empty on the other (passive). The stream is bidirectional — one dial carries both directions. |
+| `LocalDb:Replication:ApiKey` | bearer token for the sync stream | **Must be byte-identical on both nodes.** See fail-closed rule below. Supply via `${secret:...}` / env — never a cleartext literal in production config. |
+| `LocalDb:Replication:MaxBatchSize` | rows per replication batch | `16` for the artifact cache. Batching is **row-count-only** against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so 16 × 171 KB ≈ 2.7 MB stays under the cap. Raising it risks tripping the cap. |
+
+Example (site-a-1 initiator, site-a-2 passive):
+
+```
+site-a-1:  LocalDb__SyncListenPort=9001
+           LocalDb__Replication__PeerAddress=http://site-a-2:9001
+           LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key}
+           LocalDb__Replication__MaxBatchSize=16
+site-a-2:  LocalDb__SyncListenPort=9001
+           LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key}   # no PeerAddress
+           LocalDb__Replication__MaxBatchSize=16
+```
+
+> **Kestrel note.** Setting `SyncListenPort` makes the Host add an explicit `Listen*` for the h2c
+> sync listener. An explicit `Listen*` makes Kestrel **ignore `ASPNETCORE_URLS` entirely**, so the
+> Host also re-binds the primary HTTP port in the same block. If you change the primary port,
+> verify both `:<httpPort>` and `:<syncPort>` appear in the startup "Now listening on" lines.
+
+### The ApiKey rule (fail-closed)
+
+The replication library's passive endpoint verifies **no** authentication — the host
+`LocalDbSyncAuthInterceptor` is the only gate, and it is **fail-closed**:
+
+- **No key configured ⇒ every sync call is refused** (`PermissionDenied`). "No key" is never
+  "no auth required".
+- **A key mismatch ⇒ the pair silently stops converging.** The initiator's stream is rejected at
+  the peer; nothing errors loudly. A typo in one node's key looks exactly like "replication is
+  broken". If a pair is not converging, **check the keys match first.**
+
+## Operational rules
+
+- **Stop / start the pair together where you can.** Each node keeps working (and caching locally)
+  while its peer is down; the outage is not a data-loss event — the surviving node accumulates
+  writes and the peer catches up on rejoin. But a long-lived solo node drifts further from its
+  peer, so avoid leaving a pair split for extended periods.
+- **Tombstone-retention resurrection window.** Retention prunes to the newest 2 deployments and
+  replicates the prune as tombstones. Tombstones are themselves retained only for a bounded window.
+  If a node is offline **longer than the tombstone-retention window**, a delete that happened during
+  its outage may no longer be expressible as a tombstone on rejoin — a pruned deployment could
+  briefly reappear until the next deploy re-prunes it. Keep pair outages well inside that window.
+- **A rebuilt node is back-filled by its peer.** If a node loses its LocalDb file — a wiped volume,
+  a re-imaged host, a fresh container — it rejoins with an empty database and its peer snapshots the
+  cached configuration back to it, with no new deploy and no central SQL. Two things are worth
+  knowing: this needs **`ZB.MOM.WW.LocalDb` ≥ 0.1.3**, and it heals the *cache*, so the node regains
+  boot-from-cache for future outages. (On `0.1.1` a converged pair has pruned every oplog row on ack
+  and the wiped node was never snapshotted — it stayed empty until the next deploy. On `0.1.2` it is
+  back-filled but its OWN writes are silently dropped until its restarted seq counter climbs past the
+  peer's stale watermark, so the pair looks converged right up until the moment only the rebuilt node
+  applies a deploy.) A node
+  wiped **during** a central outage is still repopulated by its peer under 0.1.2; a node with **no**
+  peer replication configured self-heals only from central on its next successful apply.
+
+- **What boot-from-cache does NOT cover.** The cache is a *fallback for central-SQL outages at
+  boot*, not a replacement for central. A **new deployment still requires central SQL** — the cache
+  is only read when the central fetch fails at startup. When central returns, the node resumes
+  fresh-config behavior. Boot-from-cache logs a running-from-cache signal; treat a node that stays
+  on it as a central-connectivity incident, not steady state.
+
+## Inspecting the LocalDb file — safety rules
+
+The database runs in **WAL mode**. These rules exist because violating them corrupted a live DB in
+the 2026-07-20 ScadaBridge incident:
+
+- **Never run `sqlite3` on the live file** (host-side, against a bind-mounted or container path).
+  Opening a live WAL DB from a second process across virtiofs poisons the WAL. **Always copy the
+  triplet first** and query the copy:
+
+  ```bash
+  docker cp <node>:/app/data/otopcua-localdb.db      /tmp/localdb.db
+  docker cp <node>:/app/data/otopcua-localdb.db-wal  /tmp/localdb.db-wal
+  docker cp <node>:/app/data/otopcua-localdb.db-shm  /tmp/localdb.db-shm
+  sqlite3 /tmp/localdb.db 'SELECT cluster_id, deployment_id FROM deployment_pointer;'
+  ```
+
+- **Metrics** come from the container, not a host curl (`aspnet:10.0` has no `curl`):
+
+  ```bash
+  docker run --rm --network container:<node> curlimages/curl:latest -s localhost:<httpPort>/metrics | grep localdb_
+  ```
+
+- **If an anomaly appears within seconds of your own measurement, suspect the measurement.**
+  Restart both nodes and re-observe untouched before blaming the library.
+
+## Useful queries (on a copied triplet)
+
+```sql
+-- What each node thinks the current deployment is
+SELECT cluster_id, deployment_id, revision_hash, applied_at_utc FROM deployment_pointer;
+
+-- Convergence check: the pointer's origin stamp should match on both nodes
+SELECT pk_json, hlc, node_id, is_tombstone
+FROM __localdb_row_version WHERE table_name = 'deployment_pointer' ORDER BY pk_json;
+
+-- Replication backlog (should drain to 0 when a pair is caught up)
+SELECT COUNT(*) FROM __localdb_oplog;
+```
+
+## See also
+
+- [`docs/Redundancy.md`](../Redundancy.md) — the pair-local config cache section.
+- [`docs/Configuration.md`](../Configuration.md) — the `LocalDb` appsettings section.
+- The two-node convergence harness: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/`.

docs/plans/2026-07-16-v3-batch4-PR.md

@@ -94,7 +94,7 @@ dual-namespace + event-delivery), Runtime 377 (+ VT reassert regression), AdminU
 
 ## Documented follow-ups (non-blocking)
 
-- **Scripted-alarm redeploy recovery (CONFIRMED pre-existing bug, deferred — deserves its own careful fix).**
+- **Scripted-alarm redeploy recovery (CONFIRMED pre-existing bug, deferred — deserves its own careful fix). Filed as issue #487.**
   The scripted-alarm Part 9 condition node has the same redeploy-reset race the VT `ReassertValue` fix closed:
   `RebuildAddressSpace` clears `_alarmConditions` and `MaterialiseScriptedAlarms` recreates each condition
   fresh/normal, but `ScriptedAlarmEngine.LoadAsync` reloads from the persisted state and yields
@@ -106,10 +106,11 @@ dual-namespace + event-delivery), Runtime 377 (+ VT reassert regression), AdminU
   historian issue). Proposed shape: a host-driven, node-only `AlarmStateUpdate` re-assert in `OnAlarmsLoaded`
   (same safe post-materialise ordering as the VT fix), reusing `_engine.GetState(alarmId)` + `LoadedAlarmIds`,
   **never touching the `alerts` topic**.
-- Raw-tag rename hot-rebind of the renamed tag's live value without a recreate (today: deploy-THEN-recreate).
-- Absolute-path Monaco completion → RawPath (from Batch 3; the `{{equip}}/` completion works).
+- Raw-tag rename hot-rebind of the renamed tag's live value without a recreate (today: deploy-THEN-recreate). Filed as issue #489.
+- Absolute-path Monaco completion → RawPath (from Batch 3; the `{{equip}}/` completion works). Filed as issue #490.
 - Cross-repo (post-merge): ScadaBridge Data-Connection-Layer NodeId/namespace cutover (raw `s=<RawPath>` /
-  UNS `s=<Area>/<Line>/<Equipment>/<EffectiveName>`, retired `EquipmentNodeIds`) + the umbrella index
-  `../scadaproj/CLAUDE.md` OtOpcUa entry.
+  UNS `s=<Area>/<Line>/<Equipment>/<EffectiveName>`, retired `EquipmentNodeIds`) — tracked in the
+  ScadaBridge repo as its issue #14 (its #20 landed the `nsu=` binding + native-alarm routing). The umbrella
+  index `../scadaproj/CLAUDE.md` OtOpcUa entry is **DONE** (scadaproj `ede5275`).
 
 https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox

docs/plans/2026-07-17-alarm-condition-quality-477-design.md

@@ -0,0 +1,187 @@
+# Alarm condition Quality (issue #477) — design
+
+**Status:** implemented (L1+L2) · **Date:** 2026-07-17 · **Issue:** #477 (follow-up chain #473 → #475 → #477)
+**Scope decision:** Layer 1 + Layer 2, Bad-direct, annotate-only. Layer 3 (scripted worst-of-input) deferred → **#478**.
+
+## Problem
+
+`AlarmConditionState.Quality` is never assigned anywhere in `src/` — neither by
+`OtOpcUaNodeManager.MaterialiseAlarmCondition` nor by the `WriteAlarmCondition` transition path.
+Because `StatusCodes.Good == 0x00000000`, `default(StatusCode)` **is** `Good`, so the field is
+*accidentally valid* — clients parse it, but it reports **`Good` unconditionally regardless of the
+backing tag's real quality**.
+
+This is a wrong-*value* bug, not the null-value bug class of #473/#475. Part 9 defines
+`ConditionType.Quality` as "the quality of the Condition's source data". OT impact: when a native
+alarm's device goes offline (comms lost) the condition still reports `Quality = Good`, so an operator
+(or an HMI bucketing on `IsGood`) cannot distinguish *"genuinely not active"* from *"we have lost
+contact and do not know"*.
+
+## Why it isn't a 2-line default (confirmed by code)
+
+1. **Alarm-bearing raw tags have no value variable.** `AddressSpaceApplier` materialises a raw tag as
+   *either* a condition node (`tag.Alarm is not null`) *or* a value variable (`else`) — never both,
+   since they'd share the same `s=<RawPath>` NodeId. So `WriteValue` (the only path carrying
+   `OpcUaQuality`) is never invoked for an alarm node. Quality has nowhere to land today.
+2. **The alarm channel is quality-blind.** `AlarmEventArgs` (driver → host) and `AlarmConditionSnapshot`
+   (host → SDK sink) both carry no quality field.
+3. **On comms-loss the alarm feed goes silent.** `DriverInstanceActor` on `DisconnectObserved` detaches
+   the alarm subscription and re-enters `Reconnecting` — no transition event ever arrives to carry Bad.
+   So the "device offline" signal must come from **driver connectivity**, independently of alarm
+   transitions.
+
+## Decisions (the issue's open questions)
+
+| # | Question | Decision | Rationale |
+|---|----------|----------|-----------|
+| 1 | Does an alarm tag get quality today? | No | Confirmed above — new plumbing required. |
+| 2 | Direct status code vs. policy map | **Direct Bad** on comms-loss; Good on reconnect | Matches how a value variable would read; unambiguous for `IsGood` bucketing. |
+| 3 | Does Bad also suppress transitions / touch Retain? | **No — annotate only** | A comms-lost *active* condition must stay active + retained. Silently clearing an active alarm on comms-loss is the unsafe direction. Quality is a pure annotation; the Active/Ack/Retain state machine is untouched. |
+| 4 | Scripted alarms: worst-of-inputs quality? | **Deferred (Layer 3)** | Scripted conditions stay `Good`. Filed as a follow-up issue. |
+
+## Architecture — reuse the existing publish path, add no sink method
+
+The key move: **do not add a new `IOpcUaAddressSpaceSink` method.** A new sink-interface surface would
+have to be forwarded through `DeferredAddressSpaceSink` or it is inert on driver hosts (the F10b
+prod-inertness trap). Instead the `NativeAlarmProjector` becomes the single owner of per-condition
+state *and* quality, and a connectivity change re-projects the *last* snapshot with a swapped quality
+through the **existing** `AlarmStateUpdate → OpcUaPublishActor → WriteAlarmCondition` path.
+
+### Layer 1 — make Quality a real, plumbed field
+
+- `AlarmConditionSnapshot` (Commons) gains `OpcUaQuality Quality` (last positional param, default
+  `OpcUaQuality.Good` so scripted callers and existing tests keep compiling; Commons already knows
+  `OpcUaQuality` via `IOpcUaAddressSpaceSink`).
+- `MaterialiseAlarmCondition` sets `alarm.Quality.Value` at build time:
+  **native → `BadWaitingForInitialData`** (honest until connectivity confirms Good, matching the
+  value-variable "waiting for initial data" convention), **scripted → `Good`** (script-computed, always
+  live in this scope).
+- `WriteAlarmCondition` projects `StatusFromQuality(state.Quality)` onto `condition.Quality.Value`
+  (+ `SourceTimestamp`).
+- The delta-gate (`AlarmConditionDelta` / `ReadConditionDelta` / `ToConditionDelta`) gains a `Quality`
+  member, so a Good→Bad bucket change is a genuine delta and **fires a Part 9 condition event**.
+
+### Layer 2 — drive native quality from driver connectivity
+
+- `DriverInstanceActor`: new `public sealed record ConnectivityChanged(string DriverInstanceId, bool Connected)`.
+  `Context.Parent.Tell` it on `Become(Connected)` entry (`true`) and on the transitions into
+  `Reconnecting` (`DisconnectObserved` / `ForceReconnect`) (`false`). Fire-and-forget, mirrors
+  `DeltaApplied`.
+- `NativeAlarmProjector`: per-node state becomes `(bool Active, bool Acked, OpcUaQuality Quality)`.
+  `Project(transition)` preserves the current quality; new `ProjectQuality(nodeId, quality)` preserves
+  Active/Acked and swaps only the quality, returning a full snapshot.
+- `DriverHostActor`: `Receive<ConnectivityChanged>` iterates `_alarmNodeIdByDriverRef` for that driver
+  instance and Tells one `AlarmStateUpdate` per condition with the re-projected snapshot
+  (`connected ? Good : Bad`). **Ungated** — both redundancy nodes track their own driver's comms, matching
+  the existing "condition write stays ungated (Secondary keeps its address space warm)" rule.
+  **No `/alerts` row** for a quality-only change — driver health already has its own status/alerts surface
+  via `IDriverHealthPublisher`; a row here would be alarm-fatigue.
+
+Scripted alarms are unaffected: they are not driver instances, receive no `ConnectivityChanged`, and
+their snapshot quality stays `Good`.
+
+## Files
+
+**Layer 1**
+- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/AlarmConditionSnapshot.cs`
+- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs` (`MaterialiseAlarmCondition`,
+  `WriteAlarmCondition`, `AlarmConditionDelta`/`ReadConditionDelta`/`ToConditionDelta`)
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs` (`ToSnapshot` — Quality=Good, or rely on default)
+
+**Layer 2**
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs`
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/NativeAlarmProjector.cs`
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
+
+## Tests (TDD, RED-first)
+
+1. **Wire-level (the issue's suggested guard)** — extend `NativeAlarmEventIdentityFieldDeliveryTests`
+   (OpcUaServer.IntegrationTests): active alarm → event `Quality.IsGood`; driver disconnect → condition
+   event `Quality.IsGood == false`; reconnect → Good. Verify RED against pre-fix.
+2. **Node-level** — `NodeManagerAlarmSourceFieldsTests`: materialise sets Quality (native
+   `BadWaitingForInitialData`, scripted `Good`); `WriteAlarmCondition` projects snapshot quality and
+   fires on a quality-bucket change only.
+3. **`NativeAlarmProjector`** unit: `ProjectQuality` keeps Active/Acked + swaps quality; `Project`
+   preserves quality.
+4. **`DriverInstanceActor`**: `Connected` entry Tells `ConnectivityChanged(true)`; `DisconnectObserved`
+   Tells `ConnectivityChanged(false)`.
+5. **`DriverHostActor`**: `ConnectivityChanged(false)` pushes a Bad-quality `AlarmStateUpdate` to every
+   condition of that driver instance.
+
+## Deferred / notes
+
+- **Layer 3** (scripted worst-of-input quality) → **Gitea #478**.
+- **Implementation note:** L2 uses a **dedicated `IOpcUaAddressSpaceSink.WriteAlarmQuality`** path (not a
+  full-snapshot re-projection). Rationale: a connectivity change must set *only* Quality; re-projecting a full
+  snapshot would clobber a cold condition's severity/message and can't annotate a condition that never fired a
+  transition. The new sink method is forwarded through `DeferredAddressSpaceSink` (the F10b inertness trap) —
+  auto-verified by `DeferredSinkForwardingReflectionTests` (reflection guard) + its realm-discriminator guard.
+- **Test-harness note:** the new `DriverInstanceActor → parent` `ConnectivityChanged` Tell polluted existing
+  parent-`TestProbe` assertions in 3 `DriverInstanceActor*Tests` files; those tests now
+  `parent.IgnoreMessages(m => m is ConnectivityChanged)` since they assert on data/alarm/discovery forwards,
+  not connectivity.
+- `Bad_NoCommunication` vs generic `Bad`: v1 maps `OpcUaQuality.Bad → StatusCodes.Bad`; refining
+  `StatusFromQuality` to emit `BadNoCommunication` for the comms-loss case is a one-line nicety, noted in
+  the issue.
+- `docs/AlarmTracking.md` §"Condition event identity fields" gains a Quality subsection (Good/Bad
+  semantics, annotation-not-state-change, quality-bucket change fires an event).
+
+## Layer 3 — scripted worst-of-input quality (Gitea #478, implemented 2026-07-17)
+
+**Problem.** A scripted alarm is computed from one or more input tags. Its condition should report the
+**worst** quality of those inputs ("can I trust this condition's state?"), not the hardcoded `Good` Layer 1
+left at `ScriptedAlarmHostActor.ToSnapshot`.
+
+**Two blockers discovered in the live path (both silently discard quality):**
+1. `DependencyMuxActor.OnAttributeValuePublished` builds `VirtualTagActor.DependencyValueChanged` **without**
+   the `AttributeValuePublished.Quality` it already carries.
+2. `ScriptedAlarmHostActor.OnDependencyChanged` pushes each mux value into the engine's upstream with a
+   **hardcoded `0u` (Good)** StatusCode.
+So even a `Bad` driver value reached the scripted engine as Good — Layer 3 has to plumb quality first.
+
+**Design (mirrors Layer 2's native OT semantic through the scripted channel):**
+- **Plumb quality end-to-end.** `DependencyValueChanged` gains `OpcUaQuality Quality` (defaulted `Good`, so the
+  virtual-tag engine's calls are unchanged); the mux forwards `msg.Quality`; `OnDependencyChanged` maps it to a
+  StatusCode (`Good→0`, `Uncertain→0x40000000`, `Bad→0x80000000`) on the pushed `DataValueSnapshot`.
+- **Engine computes the worst input quality** each evaluation (over the refilled read cache, **before** the
+  `AreInputsReady` short-circuit so a `Bad` input is still observed) and carries it as
+  `ScriptedAlarmEvent.WorstInputStatusCode` (a raw `uint` StatusCode — `Core.ScriptedAlarms` doesn't reference
+  Commons, so it stays in the engine's existing StatusCode vocabulary; the host maps it to `OpcUaQuality`).
+- **Transitions carry the current worst quality** → `ToSnapshot` projects it (no clobber-back-to-Good when a
+  transition fires while an input is `Uncertain`).
+- **Quality-only changes emit out of band.** A `Bad` input freezes the condition (`AreInputsReady` returns
+  false → no transition), exactly like a comms-lost native driver — so quality can't ride a transition. The
+  engine tracks the last worst-quality **bucket** per alarm and, when the bucket changes with **no** transition
+  emission, emits a new `EmissionKind.QualityChanged` event. The host routes that to the **existing** Layer 2
+  `OpcUaPublishActor.AlarmQualityUpdate → IOpcUaAddressSpaceSink.WriteAlarmQuality` path (sets ONLY Quality,
+  one Part 9 event on a bucket change, **no `/alerts` row**, no historian write). No new sink surface.
+- **`ScriptedAlarmSource` (the `IAlarmSource` fan-out adapter) skips `QualityChanged`** — quality is delivered
+  through the dedicated node path, never as a phantom `AlarmEventArgs` (which would materialize/historize a
+  native condition).
+
+**Files (Layer 3):**
+- `src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/Part9StateMachine.cs` — `EmissionKind.QualityChanged`.
+- `src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmEngine.cs` — worst-of-input, bucket tracking,
+  `WorstInputStatusCode` on the event, quality-only emission.
+- `src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmSource.cs` — skip `QualityChanged`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs` — `DependencyValueChanged.Quality`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs` — forward `msg.Quality`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs` — push real quality;
+  `ToSnapshot` maps `WorstInputStatusCode`; `OnEngineEmission` routes `QualityChanged → AlarmQualityUpdate`.
+
+**Tests (RED-first):** engine — transition carries `Uncertain` worst; `Bad` input with no transition emits
+`QualityChanged(Bad)`; restore emits `QualityChanged(Good)`; no spurious emit when the bucket is unchanged.
+`ScriptedAlarmSource` — `QualityChanged` raises no `OnAlarmEvent`. Mux — `DependencyValueChanged` carries the
+published quality. Host — `Bad` dependency → `AlarmQualityUpdate(Bad)`, no `/alerts` publish; `ToSnapshot`
+maps the event's worst quality.
+
+**Coverage boundary → Layer 4 (#481).** L3 covers inputs whose driver **publishes a Bad/Uncertain-status
+data change** (the mux quality path). It does **not** cover a driver **comms loss**: a poll driver
+(Modbus/S7) whose device goes unreachable emits only `ConnectivityChanged` and goes silent on the value feed
+(`DriverInstanceActor.Reconnecting`), so the scripted engine keeps the last-known Good value and the
+condition stays Good — the same silent-feed problem native solved in L2, but native's `OnDriverConnectivityChanged`
+bridge fans only to **native** condition nodes (`_alarmNodeIdByDriverRef`), not into the mux the scripted
+engine reads. Bridging connectivity into scripted inputs — plus resolving the null-value/cold-start
+asymmetry (a runtime `Bad` with a null value is currently indistinguishable from cold start and contributes
+`Good`) and its ripple into virtual-tag quality — is **Gitea #481 (Layer 4)**. Found by the post-implementation
+code review; the code as shipped faithfully implements #478's written scope (mux-delivered input quality).

docs/plans/2026-07-20-localdb-adoption-phase1.md

@@ -0,0 +1,706 @@
+# OtOpcUa LocalDb Adoption — Phase 1 Implementation Plan
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+>
+> **Execution model:** This plan is optimized for **Claude Opus** agents (`claude --model opus`).
+> Dispatch implementer subagents on Opus for every task classified `high-risk`; `small`/`trivial`
+> tasks may use Sonnet. Work on branch **`feat/localdb-phase1`** in this repo (`~/Desktop/OtOpcUa`,
+> remote `lmxopcua`). Do NOT merge to `master` as part of this plan — stop at the DoD task and
+> report.
+>
+> **Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md`.
+> Read it before Task 0. The reference adoption is ScadaBridge (merged PR #23) — when in doubt,
+> mirror `~/Desktop/ScadaBridge` (files named per task below).
+
+**Goal:** Give every driver-role OtOpcUa node a consolidated `ZB.MOM.WW.LocalDb` database that caches
+the deployed-configuration artifact (chunked), boots from that cache when central SQL Server is
+unreachable, and optionally replicates it to the node's redundant pair peer over the library's gRPC
+sync (default-OFF, fail-closed bearer auth).
+
+**Architecture:** `AddZbLocalDb`/`AddZbLocalDbReplication` wired in a new `LocalDbRegistration`
+(mirroring `SecretsRegistration`), DDL + `RegisterReplicated` in `LocalDbSetup.OnReady`, a dedicated
+Kestrel h2c listener for `MapZbLocalDbSync` gated on `LocalDb:SyncListenPort`, a consumer-supplied
+fail-closed bearer interceptor, and an `IDeploymentArtifactCache` seam written by `DriverHostActor`
+after each successful apply and read as a boot fallback. The dormant LiteDB LocalCache subsystem is
+deleted as superseded.
+
+**Tech stack:** .NET 10, `ZB.MOM.WW.LocalDb`/`.Replication`/`.Contracts` **0.1.1** (Gitea feed),
+`Grpc.AspNetCore` 2.76.0, Akka.NET (existing), xunit.v3 (Host.IntegrationTests) + xunit2 TestKit
+(actor tests).
+
+**Hard rules (from the ScadaBridge adoption — violating any of these is a defect):**
+1. Order inside `OnReady` is load-bearing: **DDL → `RegisterReplicated` → writes**. Rows written
+   before registration are never captured or snapshotted — silently, forever.
+2. No autoincrement PKs, no BLOB columns in replicated tables.
+3. The library's passive sync endpoint verifies **no** ApiKey — the host interceptor is the only
+   auth, and it must be fail-closed (no key configured ⇒ deny everything).
+4. `ILocalDb.CreateConnection()` returns an **already-open**, pragma-configured connection with the
+   `zb_hlc_next()` UDF. Never call `Open()` on it. There is **no in-memory mode** — tests use temp
+   files + `SqliteConnection.ClearAllPools()` before delete.
+5. `Grpc.Core.Testing` does not exist on grpc-dotnet — hand-roll fake `ServerCallContext`s.
+6. Explicit Kestrel `Listen*` calls make Kestrel **ignore `ASPNETCORE_URLS`** — when adding the sync
+   listener you must re-bind the existing HTTP port in the same block, and verify it on the rig.
+7. Never run host `sqlite3` against a live bind-mounted WAL DB (copy the `db`/`-wal`/`-shm` triplet
+   with `cp` instead). `aspnet:10.0` containers have no `curl` — use a
+   `curlimages/curl` sidecar with `--network container:<name>`.
+
+---
+
+### Task 0: Preflight + recon (produces `docs/plans/2026-07-20-localdb-phase1-recon.md`)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (everything downstream consumes its findings)
+
+**Files:**
+- Create: `docs/plans/2026-07-20-localdb-phase1-recon.md`
+- Read-only: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs`,
+  `Directory.Packages.props`, `docker-dev/docker-compose.yml`,
+  the observability registration (`AddOtOpcUaObservability` implementation),
+  `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/*`
+
+**Step 1: Verify the feed packages restore.**
+
+Run:
+```bash
+cd ~/Desktop/OtOpcUa && dotnet package search ZB.MOM.WW.LocalDb --source dohertj2-gitea --format json | head -50
+```
+Expected: `ZB.MOM.WW.LocalDb`, `.Replication`, `.Contracts` at `0.1.1`. If absent → STOP, report.
+
+**Step 2: Map and record, with `file:line` citations, each of:**
+1. **Deploy fetch path:** where `DriverHostActor` loads `Deployment.ArtifactBlob` (the
+   `_dbFactory.CreateDbContext()` sites, ~L1472/1543), the artifact's runtime type
+   (`DeploymentArtifact` — how it's deserialized from the blob), the type/format of
+   `DeploymentId` and `RevisionHash`, and where a successful apply completes (the point after
+   which the cache write belongs).
+2. **Cold-boot path:** what the actor does at startup before any `DispatchDeployment` arrives —
+   does it query central SQL for the current deployment? Where does the failure path land
+   (the `Stale` state, `DbHealthProbeActor` interaction, L1345 dispatch-ignore)? Identify the
+   exact seam where "central fetch failed at boot" is known — that is where the cache fallback goes.
+3. **Cluster identity:** how a driver node knows its `ClusterId` (config key / `ClusterNode` row /
+   `IClusterRoleInfo`) — the cache is keyed by it.
+4. **DriverHostActor construction:** how its dependencies are injected
+   (`WithOtOpcUaRuntimeActors`, `Props` wiring) so `IDeploymentArtifactCache` can be threaded in.
+   ⚠ `Props.Create` builds an expression tree — adding a parameter silently rebinds
+   out-of-position named args at call sites; list every construction site.
+5. **Telemetry allowlist:** does `AddOtOpcUaObservability` restrict meters
+   (`ZbTelemetryOptions.Meters` or equivalent)? If yes, record where the list lives.
+6. **Kestrel/URLs:** confirm the Host binds via `ASPNETCORE_URLS=http://+:9000` with no
+   `ConfigureKestrel` calls; record any existing `builder.WebHost` usage.
+7. **LiteDB LocalCache:** confirm (grep) that nothing outside
+   `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/` and its tests references
+   `ILocalConfigCache`/`GenerationSealedCache`/`ResilientConfigReader`/`LiteDB`.
+8. **docker-dev volumes:** whether nodes have a writable data volume; record what `LocalDb:Path`
+   should be per node and how env vars are passed (`Cluster__*` style double-underscore).
+
+**Step 3: STOP conditions — halt the plan and report instead of improvising if:**
+- the artifact apply path cannot be given a post-apply hook without restructuring the actor;
+- `DeploymentId`/cluster identity are not stable strings/GUIDs;
+- LiteDB LocalCache turns out to be referenced by live code.
+
+**Step 4: Commit.**
+```bash
+git add docs/plans/2026-07-20-localdb-phase1-recon.md
+git commit -m "docs(localdb): phase-1 recon findings"
+```
+
+---
+
+### Task 1: Package references and pins
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+**Parallelizable with:** none (all code tasks build on it)
+
+**Files:**
+- Modify: `Directory.Packages.props`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ZB.MOM.WW.OtOpcUa.Runtime.csproj`
+
+**Step 1:** Add to `Directory.Packages.props` (versions exact):
+```xml
+<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.1" />
+<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.1" />
+<PackageVersion Include="Grpc.AspNetCore" Version="2.76.0" />
+```
+Check the existing `Grpc.Net.Client`/`Grpc.Core.Api` are already ≥ 2.76.0 and
+`Google.Protobuf` ≥ 3.34.1 (the LocalDb.Replication nuspec floors); raise if not. Confirm a
+`SQLitePCLRaw` pin ≥ 2.1.12 exists for the transitive `Microsoft.Data.Sqlite` chain (the family
+advisory GHSA-2m69-gcr7-jv3q); Core.AlarmHistorian already pins `bundle_e_sqlite3` 2.1.12 — add
+`SQLitePCLRaw.lib.e_sqlite3` 2.1.12 as an explicit reference in the Host if the audit flags it.
+
+**Step 2:** Host csproj: `ZB.MOM.WW.LocalDb`, `ZB.MOM.WW.LocalDb.Replication`, `Grpc.AspNetCore`.
+Runtime csproj: `ZB.MOM.WW.LocalDb` only (it needs `ILocalDb`, not the sync engine).
+
+**Step 3:** `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → 0 warnings/errors (repo builds warnings-as-errors).
+
+**Step 4: Commit.** `git commit -m "build(localdb): reference ZB.MOM.WW.LocalDb 0.1.1 + Grpc.AspNetCore"`
+
+---
+
+### Task 2: Schema + `LocalDbSetup.OnReady` + registration tests
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 3
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/DeploymentCacheSchema.cs`
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/LocalDbSetupTests.cs` (create the test project
+  reference layout to match neighboring Host unit tests; if Host has no unit-test project, place in
+  the closest existing one and note the deviation)
+
+**Step 1: Write the failing tests first** — using a real temp-file `ILocalDb` (build it via
+`new ServiceCollection().AddZbLocalDb(config, LocalDbSetup.OnReady)` with `LocalDb:Path` pointed at
+a temp file; dispose + `SqliteConnection.ClearAllPools()` in cleanup):
+- `OnReady_RegistersExactlyTheTwoDeploymentTables` — `db.ReplicatedTables.Keys` ordinal-sorted
+  equals `["deployment_artifacts", "deployment_pointer"]` (assert BOTH directions: no fewer, no more).
+- `DeploymentArtifacts_PkIsDeploymentIdPlusChunkIndex` and `DeploymentPointer_PkIsClusterId`
+  (pin `ReplicatedTable.PkColumns`).
+- `RowsWrittenAfterOnReady_EnterTheOplog` — insert a row, assert `SELECT COUNT(*) FROM __localdb_oplog` ≥ 1
+  (pins the DDL→register ordering; this is the assertion that catches a silently-broken CDC).
+
+**Step 2: Run tests, watch them fail** (types don't exist yet).
+
+**Step 3: Implement.** `DeploymentCacheSchema` depends only on `Microsoft.Data.Sqlite` (the
+ScadaBridge `*Schema.Apply` pattern — self-sufficient for direct construction in tests):
+
+```csharp
+namespace ZB.MOM.WW.OtOpcUa.Runtime.Deployment;
+
+public static class DeploymentCacheSchema
+{
+    public static void Apply(SqliteConnection connection)
+    {
+        using var cmd = connection.CreateCommand();
+        cmd.CommandText = """
+            CREATE TABLE IF NOT EXISTS deployment_artifacts (
+                deployment_id   TEXT    NOT NULL,
+                chunk_index     INTEGER NOT NULL,
+                cluster_id      TEXT    NOT NULL,
+                revision_hash   TEXT    NOT NULL,
+                chunk_count     INTEGER NOT NULL,
+                chunk_base64    TEXT    NOT NULL,
+                cached_at_utc   TEXT    NOT NULL,
+                PRIMARY KEY (deployment_id, chunk_index)
+            );
+            CREATE TABLE IF NOT EXISTS deployment_pointer (
+                cluster_id        TEXT NOT NULL PRIMARY KEY,
+                deployment_id     TEXT NOT NULL,
+                revision_hash     TEXT NOT NULL,
+                artifact_sha256   TEXT NOT NULL,
+                applied_at_utc    TEXT NOT NULL
+            );
+            """;
+        cmd.ExecuteNonQuery();
+    }
+}
+```
+
+`LocalDbSetup` (Host):
+```csharp
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+internal static class LocalDbSetup
+{
+    // ORDER IS LOAD-BEARING: DDL, then RegisterReplicated, then (nothing else in Phase 1).
+    // Rows written before RegisterReplicated are invisible to the peer forever, silently.
+    public static void OnReady(ILocalDb db)
+    {
+        using var connection = db.CreateConnection(); // already open — do NOT call Open()
+        DeploymentCacheSchema.Apply(connection);
+        db.RegisterReplicated("deployment_artifacts");
+        db.RegisterReplicated("deployment_pointer");
+    }
+}
+```
+
+**Step 4:** `dotnet test --filter "FullyQualifiedName~LocalDbSetupTests"` → PASS.
+
+**Step 5: Commit.** `git commit -m "feat(localdb): deployment-cache schema + OnReady registration"`
+
+---
+
+### Task 3: Fail-closed sync auth interceptor + tests
+
+**Classification:** standard
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 2
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/LocalDbSyncAuthInterceptorTests.cs`
+
+Port ScadaBridge's `src/ZB.MOM.WW.ScadaBridge.Host/LocalDbSyncAuthInterceptor.cs` (read it first;
+keep behavior identical, adjust namespace only). Semantics to preserve exactly:
+- `ServicePrefix = "/localdb_sync.v1.LocalDbSync/"`; any other method path passes through untouched.
+- Expected token from `IOptions<ReplicationOptions>.Value.ApiKey`.
+- **Fail-closed:** null/empty configured key ⇒ `RpcException(new Status(StatusCode.PermissionDenied, ...))`
+  for every sync call. Missing/mismatched bearer ⇒ same.
+- `CryptographicOperations.FixedTimeEquals` over UTF-8 bytes; header `authorization`,
+  case-insensitive, `"Bearer "` prefix. Override all four server handler kinds.
+
+**Tests** (hand-rolled fake `ServerCallContext` — `Grpc.Core.Testing` does not exist on grpc-dotnet;
+copy ScadaBridge's `LocalDbSyncAuthInterceptorTests.cs` fake): non-sync method passes with no key;
+sync + no configured key → PermissionDenied; wrong bearer → PermissionDenied; correct bearer → passes.
+
+Write tests first (fail), implement, `dotnet test --filter "FullyQualifiedName~LocalDbSyncAuthInterceptor"`,
+commit `feat(localdb): fail-closed bearer interceptor for the sync endpoint`.
+
+---
+
+### Task 4: `LocalDbRegistration` + Program.cs DI wiring + config defaults
+
+**Classification:** high-risk (touches Program.cs / role gating)
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (Task 5 edits the same file)
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (inside the `hasDriver` branch)
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json`
+
+**Step 1:** `LocalDbRegistration`, mirroring `SecretsRegistration`'s shape:
+```csharp
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+internal static class LocalDbRegistration
+{
+    /// Driver-role nodes only. Storage is unconditional; replication stays inert until
+    /// LocalDb:Replication:PeerAddress (initiator) / LocalDb:SyncListenPort (listener) are set.
+    public static IServiceCollection AddOtOpcUaLocalDb(
+        this IServiceCollection services, IConfiguration configuration)
+    {
+        services.AddZbLocalDb(configuration, LocalDbSetup.OnReady);
+        services.AddZbLocalDbReplication(configuration);
+        return services;
+    }
+
+    public static int SyncListenPort(IConfiguration configuration) =>
+        configuration.GetValue<int>("LocalDb:SyncListenPort");
+}
+```
+
+**Step 2:** Program.cs, in the `hasDriver` block (near `AddAlarmHistorian`):
+`builder.Services.AddOtOpcUaLocalDb(builder.Configuration);` and
+`builder.Services.AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>());`
+(AddGrpc only under `hasDriver` — admin-only nodes expose no sync surface).
+
+**Step 3:** appsettings.json:
+```json
+"LocalDb": {
+  "Path": "./data/otopcua-localdb.db",
+  "SyncListenPort": 0,
+  "Replication": {}
+}
+```
+`LocalDb:Path` is `ValidateOnStart`-required once `AddZbLocalDb` runs — since registration is
+driver-gated, admin-only configs need nothing. Check the role-overlay files
+(`appsettings.driver.json` / `appsettings.admin-driver.json`) and any deploy templates for
+conflicting `LocalDb` keys.
+
+**Step 4:** Build + full Host unit tests. Verify an admin-only graph doesn't require the key: this
+is pinned properly in Task 10's integration tests, but do a quick
+`OTOPCUA_ROLES=admin dotnet run` smoke only if cheap; otherwise rely on Task 10.
+
+**Step 5: Commit.** `feat(localdb): wire AddOtOpcUaLocalDb into the driver role`
+
+---
+
+### Task 5: Dedicated h2c sync listener + endpoint mapping (THE Kestrel task)
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (Program.cs)
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`
+
+**Why high-risk:** the Host today binds exclusively via `ASPNETCORE_URLS=http://+:9000`. Any
+explicit `ConfigureKestrel(... Listen*)` makes Kestrel **ignore URLs entirely** (it logs
+"Overriding address(es)"), which would silently kill the AdminUI/deploy API behind Traefik.
+
+**Step 1:** Add, before `builder.Build()`:
+```csharp
+var syncPort = LocalDbRegistration.SyncListenPort(builder.Configuration);
+if (hasDriver && syncPort > 0)
+{
+    // Explicit Listen* replaces ASPNETCORE_URLS wholesale, so re-bind the primary HTTP
+    // endpoint here too. Parse it from the configured URLs rather than hard-coding 9000.
+    var urls = builder.Configuration["ASPNETCORE_URLS"] ?? builder.Configuration["urls"] ?? "http://+:9000";
+    var httpPort = new Uri(urls.Split(';')[0].Replace("+", "localhost").Replace("*", "localhost")).Port;
+    builder.WebHost.ConfigureKestrel(k =>
+    {
+        k.ListenAnyIP(httpPort);                                            // HTTP/1.1 (existing surface)
+        k.ListenAnyIP(syncPort, o => o.Protocols = HttpProtocols.Http2);    // h2c, prior-knowledge gRPC
+    });
+}
+```
+When `syncPort == 0` (the default) nothing changes — URLs binding stays untouched. Cleartext
+`Http1AndHttp2` cannot serve prior-knowledge h2c, hence the dedicated `Http2`-only listener.
+
+**Step 2:** After the app pipeline's other `Map*` calls, driver-gated:
+```csharp
+if (hasDriver && syncPort > 0)
+{
+    app.MapZbLocalDbSync();
+}
+```
+(Mapping is harmless when unauthenticated — the interceptor fail-closes — but gating on the port
+keeps admin-only and default-OFF graphs entirely free of the endpoint.)
+
+**Step 3: Verify both surfaces.**
+```bash
+dotnet build ZB.MOM.WW.OtOpcUa.slnx
+```
+Then a local smoke with the port on:
+`ASPNETCORE_URLS=http://+:9000 OTOPCUA_ROLES=driver LocalDb__SyncListenPort=9001 dotnet run --project src/Server/ZB.MOM.WW.OtOpcUa.Host ...`
+— expect startup logs to show BOTH `:9000` and `:9001` bound, and `curl -s localhost:9000/healthz`
+(or the mapped health route) still answering. If the app needs SQL to boot, defer the smoke to the
+Task 12 rig check but say so explicitly in the task notes.
+
+**Step 4: Commit.** `feat(localdb): dedicated h2c listener + MapZbLocalDbSync gated on LocalDb:SyncListenPort`
+
+---
+
+### Task 6: `IDeploymentArtifactCache` + chunked LocalDb implementation + tests
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 3, Task 5
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactCache.cs`
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/LocalDbDeploymentArtifactCache.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Deployment/LocalDbDeploymentArtifactCacheTests.cs`
+
+**Step 1: Failing tests** (real temp-file `ILocalDb` with `LocalDbSetup.OnReady` applied):
+- round-trip: store a 300 KiB random artifact → `GetCurrentAsync` returns byte-identical payload
+  (forces ≥ 3 chunks at 128 KiB);
+- retention: store 3 deployments for one cluster → only the newest 2 remain in
+  `deployment_artifacts`; pointer names the newest;
+- integrity: corrupt one chunk row via SQL → `GetCurrentAsync` returns null (miss), never a
+  truncated artifact;
+- missing pointer → null.
+
+**Step 2: Implement:**
+```csharp
+public interface IDeploymentArtifactCache
+{
+    Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
+                    byte[] artifact, CancellationToken ct = default);
+    Task<CachedDeploymentArtifact?> GetCurrentAsync(string clusterId, CancellationToken ct = default);
+}
+
+public sealed record CachedDeploymentArtifact(
+    string DeploymentId, string RevisionHash, byte[] Artifact, DateTimeOffset AppliedAtUtc);
+```
+`LocalDbDeploymentArtifactCache(ILocalDb db)`:
+- `StoreAsync`: one `ILocalDbTransaction` — delete any existing chunks for this `deployment_id`
+  (idempotent re-store), insert chunks (raw 128 * 1024 bytes per chunk, `Convert.ToBase64String`),
+  upsert the pointer (`INSERT ... ON CONFLICT(cluster_id) DO UPDATE`), then prune: delete
+  `deployment_artifacts` rows whose `deployment_id` is not among the newest 2 `cached_at_utc` for
+  this `cluster_id`. SHA-256 over the raw artifact into the pointer. ISO-8601 UTC timestamps.
+- `GetCurrentAsync`: read pointer; read chunks `ORDER BY chunk_index`; verify count ==
+  `chunk_count` and SHA-256 matches; return null on any mismatch (log a warning naming which check
+  failed).
+- Use `db.ExecuteAsync`/`QueryAsync` with anonymous-object parameters (`@Name` markers). Remember
+  a `Dictionary<string,string>` parameter **throws** — use anonymous objects.
+
+**Step 3:** run tests → PASS. **Step 4: Commit.**
+`feat(localdb): chunked deployment-artifact cache over ILocalDb`
+
+---
+
+### Task 7: Cache write path in `DriverHostActor`
+
+**Classification:** high-risk (actor model, `Props` expression-tree trap)
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:** (exact edit sites come from the Task 0 recon doc — cite it in the commit)
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
+- Modify: the actor-registration site (`WithOtOpcUaRuntimeActors` / DI extension) to provide
+  `IDeploymentArtifactCache`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` or `LocalDbRegistration` — register
+  `IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache` singleton (driver branch)
+- Test: the **xunit2** TestKit project used for existing DriverHostActor tests (recon names it)
+
+**Steps:**
+1. **Failing test:** after a successful artifact apply, the cache received `StoreAsync` with the
+   applied deployment's id/hash/bytes (inject a recording fake `IDeploymentArtifactCache`).
+   Also: a cache that throws on `StoreAsync` does NOT fail the apply (apply result unchanged, error
+   logged).
+2. Thread the dependency through the actor's constructor. ⚠ `Props.Create` is an expression tree:
+   after adding the parameter, re-check **every** construction site for positional/named-arg
+   rebinding (the recon lists them) — do not rely on the compiler.
+3. Invoke `StoreAsync` fire-and-forget (`PipeTo`-style or a guarded `Task.Run` per the actor's
+   existing async conventions — match whatever pattern the actor already uses for side-effect IO)
+   at the post-apply point identified in recon. Cluster id from the recon-identified source.
+4. Run the actor test suite for this project. Commit:
+   `feat(localdb): DriverHostActor stores applied artifacts in the pair-local cache`
+
+---
+
+### Task 8: Boot-from-cache read path + running-from-cache signal
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
+- Test: same xunit2 TestKit project as Task 7
+
+**Steps:**
+1. **Failing tests:**
+   - central fetch fails at startup AND cache has an artifact → the actor applies the cached
+     artifact (assert via whatever observable the apply already exposes) and logs/flags
+     running-from-cache;
+   - central fetch fails AND cache empty → today's behavior exactly (Stale, no apply);
+   - central fetch **succeeds** → cache is NOT consulted (fresh config wins; assert the fake
+     cache's `GetCurrentAsync` was never called on the happy path).
+2. Implement at the recon-identified boot-failure seam. The signal: a log warning at minimum plus,
+   if the actor already publishes health/status (recon says how), a `RunningFromCache` marker on it.
+   Do NOT touch `DispatchDeployment` handling — a new deployment still requires central.
+3. Run tests; commit `feat(localdb): boot from the pair-local artifact cache when central SQL is unreachable`.
+
+---
+
+### Task 9: Delete the dormant LiteDB LocalCache subsystem
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+**Parallelizable with:** Task 6, Task 10, Task 11
+
+**Files:**
+- Delete: `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/` (entire directory:
+  `ILocalConfigCache`, `LiteDbConfigCache`, `GenerationSealedCache`, `ResilientConfigReader`,
+  `GenerationSnapshot`, `StaleConfigFlag`, exceptions)
+- Delete: `tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/GenerationSealedCacheTests.cs`,
+  `ResilientConfigReaderTests.cs`
+- Modify: `Directory.Packages.props` + the Configuration csproj — remove the `LiteDB` package if
+  nothing else references it (grep first)
+
+**Steps:** Confirm the Task 0 recon's "nothing references it" finding still holds
+(`grep -rn "ILocalConfigCache\|GenerationSealedCache\|ResilientConfigReader\|LiteDB" src/ tests/`),
+delete, build the full solution, run the Configuration test project. DoD phrasing: **no references
+from code** — leave any explanatory prose/comments that point readers at LocalDb instead. Add one
+line to the recon doc noting LiteDB's removal. Commit:
+`refactor(localdb): delete the dormant LiteDB LocalCache (superseded by ZB.MOM.WW.LocalDb)`
+
+---
+
+### Task 10: Health check + telemetry meter allowlist
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 9, Task 11
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/LocalDbReplicationHealthCheck.cs` (or the
+  directory `AddOtOpcUaHealth` uses — recon names it)
+- Modify: the `AddOtOpcUaHealth` registration (driver branch)
+- Modify: the observability meter list IF Task 0 found an allowlist
+
+**Steps:**
+1. Failing unit tests: replication unconfigured (no peer, no listener) → `Healthy` (default-OFF
+   must not degrade a plain node); peer configured + `ISyncStatus.Connected == false` → `Degraded`;
+   connected + backlog `null` → `Degraded` (unknown backlog is not healthy); connected + backlog
+   small → `Healthy`.
+2. Implement against `ISyncStatus` + `IOptions<ReplicationOptions>` (peer-configured = non-empty
+   `PeerAddress` OR `SyncListenPort > 0` — pass the latter in via options/config).
+3. **Meter allowlist:** if `AddOtOpcUaObservability` restricts meters, add
+   `"ZB.MOM.WW.LocalDb.Replication"` (use `LocalDbMetrics.MeterName`) — this is a silent allowlist;
+   the ScadaBridge live gate is the proof it bites. If there is no allowlist, record that in the
+   recon doc and skip.
+4. Commit `feat(localdb): replication health check + meter export`.
+
+---
+
+### Task 11: DI-pin integration tests over the real Program.cs
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 9, Task 10
+
+**Files:**
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbWiringTests.cs` (xunit.v3,
+  `WebApplicationFactory<Program>` — the project already builds the real Program.cs; set
+  `OTOPCUA_ROLES` per test case the way neighboring tests do)
+
+**Pins (each its own test):**
+1. Driver graph: `ILocalDb` resolves, is a singleton, `ReplicatedTables` ordinal-sorted ==
+   `["deployment_artifacts", "deployment_pointer"]` — exact set, both directions load-bearing.
+2. Driver graph: `IDeploymentArtifactCache` resolves to `LocalDbDeploymentArtifactCache`.
+3. Default-OFF pin: `ISyncStatus` resolves with `Connected == false`, `PeerNodeId == null`.
+4. Admin-only graph: `ILocalDb` is NOT registered (`GetService<ILocalDb>()` is null) and boot does
+   not demand `LocalDb:Path`.
+5. Health: the LocalDb health check is registered in the driver graph.
+
+Point `LocalDb:Path` at a per-test temp file via the factory's config overrides;
+`ClearAllPools` + delete in cleanup. These tests exist because DI extensions without a
+container-built test have shipped inert three times in this family (Secrets 0.2.0/0.2.2,
+ScadaBridge#22). Commit `test(localdb): DI pins over the real host graph`.
+
+---
+
+### Task 12: Two-node convergence harness + tests
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min (harness) + ~4 min (scenarios) — split the commit if needed
+**Parallelizable with:** none (depends on Tasks 2, 3, 5, 6)
+
+**Files:**
+- Create: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairHarness.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairConvergenceTests.cs`
+
+**Harness:** copy the pattern from
+`~/Desktop/ScadaBridge/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs` and
+the library's `ConvergenceFixture` (`~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/tests/.../Convergence/ConvergenceFixture.cs`):
+- Two full stacks over a real loopback Kestrel h2c socket (port 0), **through the real
+  `LocalDbSyncAuthInterceptor`** with a shared test key; node A initiator, node B passive.
+- Both initialized via the **production `LocalDbSetup.OnReady`** — a hand-written schema proves
+  only that the test agrees with itself.
+- DBs owned by the harness, registered as pre-constructed singletons (so MS.DI won't dispose them
+  on host teardown); `KillTransportAsync`/`RestartTransportAsync`; tight `FlushInterval` (50 ms),
+  bounded backoff (2 s); temp files, `ClearAllPools` cleanup; a collection definition to serialize
+  these tests.
+
+**Scenarios:**
+1. `ArtifactStoredOnA_ConvergesToB_ByteIdentical` — store a multi-chunk artifact via
+   `LocalDbDeploymentArtifactCache` on A; B's cache returns byte-identical bytes; both nodes'
+   `__localdb_row_version` rows for the pointer carry the **same HLC and origin node id** (B holds
+   A's row, not a re-derived one).
+2. `RetentionPruneOnA_TombstonesReachB` — third deployment on A prunes the first; B's chunk rows
+   for the pruned deployment disappear and `__localdb_row_version WHERE is_tombstone=1` rows exist
+   on B.
+3. `WritesWhileTransportDown_SurviveRejoin` — kill transport, store on A, restart, converge.
+4. `WrongApiKey_NeverConverges` — harness with mismatched keys: assert **no** convergence after a
+   bounded wait AND (positive control) that the same scenario with matching keys converges — an
+   absence assertion without a positive control passed vacuously in ScadaBridge.
+
+**DoD within this task:** temporarily comment out the two `RegisterReplicated` calls and confirm
+scenarios 1–3 go red (run locally, do not commit the red state); restore. Record the red/green
+evidence in the task notes. Commit `test(localdb): 2-node convergence harness + scenarios`.
+
+---
+
+### Task 13: docker-dev rig configuration
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 14
+
+**Files:**
+- Modify: `docker-dev/docker-compose.yml`
+
+**Steps** (adjust to the recon's findings on volumes/env style):
+1. Every driver node: a named volume (or existing data mount) backing `/app/data`, and
+   `LocalDb__Path=/app/data/otopcua-localdb.db`.
+2. **site-a pair only** (mirror the ScadaBridge default-OFF posture — site-b stays unreplicated as
+   the pin):
+   - site-a-1: `LocalDb__SyncListenPort=9001`, `LocalDb__Replication__PeerAddress=http://site-a-2:9001`,
+     `LocalDb__Replication__ApiKey=dev-site-a-localdb-sync-key`, `LocalDb__Replication__MaxBatchSize=16`
+   - site-a-2: `LocalDb__SyncListenPort=9001`, same `ApiKey`, same `MaxBatchSize`, **no PeerAddress**
+     (passive; the stream is bidirectional).
+   The key must be byte-identical on both — the interceptor fail-closes on any mismatch and the
+   pair silently stops converging.
+3. `MaxBatchSize=16` because batching is row-count-only against gRPC's 4 MB cap and artifact chunk
+   rows are ≈ 171 KB (16 × 171 KB ≈ 2.7 MB worst case).
+4. `docker compose config` to validate; do NOT bring the rig up in this task (the live gate task
+   owns rig runs).
+5. Commit `chore(localdb): rig config — consolidated path everywhere, replication on the site-a pair`.
+
+---
+
+### Task 14: Documentation
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 13
+
+**Files:**
+- Create: `docs/operations/2026-07-20-localdb-pair-replication.md` (runbook: enable/disable,
+  ApiKey handling via `${secret:}`, stop/start-together rule, tombstone-retention resurrection
+  window, MaxBatchSize row-count-vs-4MB, never host-`sqlite3` a live WAL DB / cp-triplet recipe)
+- Modify: `docs/Redundancy.md` (a short "pair-local config cache" section: what replicates, what
+  boot-from-cache does and does not cover)
+- Modify: `CLAUDE.md` (this repo): LocalDb adoption row/paragraph — state Phase 1 scope, default-OFF,
+  rig-only enablement
+- Modify: `docs/Configuration.md` if it documents config keys (add the `LocalDb` section)
+
+Commit `docs(localdb): phase-1 runbook + redundancy/config docs`.
+
+---
+
+### Task 15: DoD sweep (offline)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (final offline task)
+
+**Steps:**
+1. `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → **0 warnings** (warnings-as-errors).
+2. `dotnet test ZB.MOM.WW.OtOpcUa.slnx` → full suite green; record counts. Baseline any
+   pre-existing failures BEFORE this branch (`git stash` → run → unstash) rather than assuming;
+   report deltas only.
+3. Greps (phrase as "no references from code"; explanatory comments may remain):
+   `grep -rn "LiteDB\|ILocalConfigCache\|GenerationSealedCache" src/ tests/` → no code references.
+4. Confirm the positive-control evidence from Task 12 is recorded.
+5. Update `docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json` statuses.
+6. Commit `chore(localdb): phase-1 DoD sweep` and STOP. Report: branch name, commit list, test
+   counts, and that the live gate (Task 16) needs the rig + operator go-ahead.
+
+---
+
+### Task 16: Live gate on the docker-dev rig (run only with explicit user go-ahead)
+
+**Classification:** high-risk (rig)
+**Estimated implement time:** ~30 min wall-clock (mostly waiting)
+**Parallelizable with:** none
+
+**Preamble:** All DB inspection via `cp` of the `db`/`-wal`/`-shm` triplet out of the container
+(`docker cp`) and querying the copy — never host `sqlite3` on the live file (it poisons the WAL
+across virtiofs; root cause of the 2026-07-20 ScadaBridge incident). Metrics via
+`docker run --rm --network container:<node> curlimages/curl:latest -s localhost:<port>/metrics`.
+If an anomaly appears within seconds of one of your own measurements, suspect the measurement
+first — restart both nodes and re-run untouched before blaming the library.
+
+**Checks (all must PASS; record evidence in `docs/plans/2026-07-20-localdb-phase1-live-gate.md`):**
+1. Rig up, site-a pair healthy, `/metrics` on both shows `localdb_` series (proves the meter
+   allowlist work).
+2. Deploy a config through the normal deploy API → both site-a nodes' `deployment_pointer` +
+   `deployment_artifacts` rows are byte-identical with **identical HLC + origin node id** (one
+   node's row replicated, not two independent derivations — both nodes also locally store after
+   apply, so expect LWW-converged identical content either way; assert convergence, and note which
+   origin won).
+3. **Boot-from-cache:** stop the SQL container; restart site-a-2; it comes up serving the cached
+   config with the running-from-cache signal in its log; start SQL again; node recovers fresh
+   behavior.
+4. **Replicated-cache payoff:** wipe site-a-2's local DB file (container stopped), restart it with
+   SQL **up**, let replication repopulate the cache from site-a-1, then repeat check 3's SQL-down
+   restart — it boots from a cache it never wrote itself.
+5. Transport kill (pause site-a-2 container): store/deploy on a-1, oplog depth rises; unpause;
+   drains to 0; converged.
+6. Retention: deploy a 3rd config; pruned deployment's chunks gone on BOTH nodes with tombstone
+   rows present on both.
+7. Both-nodes-together restart: clean rejoin, identical counts, zero
+   `disk I/O error`/`SQLITE_IOERR`/`corrupt` in either log.
+8. site-b pair (default-OFF pin): LocalDb file exists and works locally, `ISyncStatus` health
+   Healthy, no sync connections, no listener on 9001.
+
+Record PASS/FAIL per check with the actual evidence (row counts, md5s, log lines). Commit the gate
+doc. Do not merge — report.
+
+---
+
+## Task persistence
+
+Tasks file: `docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json` (same directory).
+Update statuses as tasks complete; any deviation from this plan gets a `deviation` note on the task
+(the ScadaBridge tasks.json convention — the record of *why* is what makes the plan auditable).

docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json

@@ -0,0 +1,153 @@
+{
+  "planPath": "docs/plans/2026-07-20-localdb-adoption-phase1.md",
+  "tasks": [
+    {
+      "id": 0,
+      "subject": "Task 0: Preflight + recon (findings doc, STOP conditions)",
+      "status": "completed",
+      "notes": "Feed verified: LocalDb/.Contracts/.Replication all 0.1.1. All 3 STOP conditions clear. Findings: docs/plans/2026-07-20-localdb-phase1-recon.md. 8 deviations recorded (D-1..D-8); D-1/D-3/D-5/D-6 change downstream work materially.",
+      "deviation": "D-1 cache read cannot be keyed by ClusterId at the boot seam (unkeyed newest-pointer read instead); D-3 a third LiteDB test file must be deleted; D-5 WebApplicationFactory<Program> is deliberately unused in this repo (use TwoNodeClusterHarness); D-6 driver-only nodes have no ASPNETCORE_URLS so the Kestrel re-bind fallback is wrong. See recon doc \u00a79."
+    },
+    {
+      "id": 1,
+      "subject": "Task 1: Package references and pins (LocalDb 0.1.1, Grpc.AspNetCore, SQLitePCLRaw)",
+      "status": "completed",
+      "blockedBy": [
+        0
+      ]
+    },
+    {
+      "id": 2,
+      "subject": "Task 2: Schema + LocalDbSetup.OnReady + registration tests",
+      "status": "completed",
+      "blockedBy": [
+        1
+      ]
+    },
+    {
+      "id": 3,
+      "subject": "Task 3: Fail-closed sync auth interceptor + tests",
+      "status": "completed",
+      "blockedBy": [
+        1
+      ]
+    },
+    {
+      "id": 4,
+      "subject": "Task 4: LocalDbRegistration + Program.cs DI wiring + config defaults",
+      "status": "completed",
+      "blockedBy": [
+        2,
+        3
+      ]
+    },
+    {
+      "id": 5,
+      "subject": "Task 5: Dedicated h2c sync listener + MapZbLocalDbSync (Kestrel URLs-override risk)",
+      "status": "completed",
+      "blockedBy": [
+        4
+      ]
+    },
+    {
+      "id": 6,
+      "subject": "Task 6: IDeploymentArtifactCache + chunked LocalDb implementation + tests",
+      "status": "completed",
+      "blockedBy": [
+        2
+      ]
+    },
+    {
+      "id": 7,
+      "subject": "Task 7: Cache write path in DriverHostActor (Props expression-tree trap)",
+      "status": "completed",
+      "blockedBy": [
+        0,
+        6
+      ]
+    },
+    {
+      "id": 8,
+      "subject": "Task 8: Boot-from-cache read path + running-from-cache signal",
+      "status": "completed",
+      "blockedBy": [
+        7
+      ]
+    },
+    {
+      "id": 9,
+      "subject": "Task 9: Delete the dormant LiteDB LocalCache subsystem",
+      "status": "completed",
+      "blockedBy": [
+        0
+      ]
+    },
+    {
+      "id": 10,
+      "subject": "Task 10: Health check + telemetry meter allowlist",
+      "status": "completed",
+      "blockedBy": [
+        4
+      ]
+    },
+    {
+      "id": 11,
+      "subject": "Task 11: DI-pin integration tests over the real Program.cs",
+      "status": "completed",
+      "blockedBy": [
+        5,
+        10
+      ]
+    },
+    {
+      "id": 12,
+      "subject": "Task 12: Two-node convergence harness + scenarios (+ positive control)",
+      "status": "completed",
+      "blockedBy": [
+        5,
+        6
+      ]
+    },
+    {
+      "id": 13,
+      "subject": "Task 13: docker-dev rig configuration (site-a pair on, site-b pin off)",
+      "status": "completed",
+      "blockedBy": [
+        5
+      ]
+    },
+    {
+      "id": 14,
+      "subject": "Task 14: Documentation (runbook, Redundancy.md, CLAUDE.md)",
+      "status": "completed",
+      "blockedBy": [
+        8
+      ]
+    },
+    {
+      "id": 15,
+      "subject": "Task 15: DoD sweep (offline) \u2014 STOP and report after this",
+      "status": "completed",
+      "blockedBy": [
+        7,
+        8,
+        9,
+        11,
+        12,
+        13,
+        14
+      ],
+      "notes": "Offline DoD sweep: full-solution build 0 errors (824 pre-existing OTOPCUA0001 analyzer warnings in driver *test* projects; none reference any LocalDb/DeploymentCache file). Grep: no LiteDB/ILocalConfigCache/GenerationSealedCache code refs (2 explanatory doc-comment lines in ILdapGroupRoleMappingService remain, allowed). Runtime.Tests 407/0/31 twice (the previously-flagged intermittent did NOT reproduce). Host.IntegrationTests LocalDb subset 40/0/0. Full-solution `dotnet test` with stash-baseline NOT run: many suites are infra-gated (driver fixtures, full Akka mesh, LDAP real-bind, shared-SQL DB tests) and cannot run offline on macOS \u2014 deferred to Task 16 / CI. Task 12 positive-control evidence recorded in commit afa5be71. Did NOT merge to master \u2014 stopped here per plan."
+    },
+    {
+      "id": 16,
+      "subject": "Task 16: Live gate on the docker-dev rig (needs explicit user go-ahead)",
+      "status": "completed",
+      "blockedBy": [
+        15
+      ],
+      "notes": "Live gate RAN on the docker-dev rig with explicit user go-ahead. 8/8 checks pass (check 4 with a documented limitation). Caught + fixed FOUR real defects offline tests missed (4b2f0e6e NU1101 packageSourceMapping, ce9fa07f ASPNETCORE_HTTP_PORTS re-bind, 9137cb41 empty-address-space-on-cache-boot, c6a9f93a cache-not-repopulated-on-RestoreApplied), each with a regression test. Two documented limitations (follow-ups): no replication back-fill of a fully-wiped node; oplog growth on default-OFF nodes. Post-fix: full build 0 errors, Runtime.Tests 409/0, all LocalDb Host.IntegrationTests green; only consistent failure is the infra-gated AbCip_Green_AgainstSim (sim not up). Evidence: 2026-07-20-localdb-phase1-live-gate.md. NOT merged."
+    }
+  ],
+  "lastUpdated": "2026-07-20T00:00:00Z"
+}

docs/plans/2026-07-20-localdb-adoption-phase2.md

@@ -0,0 +1,293 @@
+# OtOpcUa LocalDb Adoption — Phase 2 Implementation Plan (alarm-historian store-and-forward)
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+>
+> **Execution model:** Optimized for **Claude Opus** agents (`claude --model opus`); dispatch
+> `high-risk` tasks on Opus. Branch **`feat/localdb-phase2`** in `~/Desktop/OtOpcUa` (remote
+> `lmxopcua`). **Prerequisite: Phase 1 (`docs/plans/2026-07-20-localdb-adoption-phase1.md`) is
+> merged (or this branch is stacked on it) and its live gate passed.** Do not merge as part of this
+> plan; stop at the DoD task.
+>
+> **Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md` §3.4,
+> D8, D9. Reference implementation for "replace a bespoke store" phasing: ScadaBridge Phase 2
+> (`~/Desktop/ScadaBridge/docs/plans/2026-07-19-localdb-adoption-phase2.md` + its live gate doc).
+
+**Goal:** Move the alarm-historian store-and-forward buffer (today the standalone
+`alarm-historian.db` owned by `SqliteStoreAndForwardSink`) into the consolidated LocalDb file as a
+replicated table with a primary-gated drain, so a redundant pair no longer loses buffered alarm
+history when a node dies — and delete the sink's bespoke file/connection management outright.
+
+**Architecture:** `alarm_sf_events` (TEXT GUID PK) registered in `LocalDbSetup.OnReady`; the sink
+rewired onto `ILocalDb` behind its unchanged public seam; drain gated on the delivered-snapshot
+Primary role via `PrimaryGatePolicy` (at-least-once across failover, accepted and documented); a
+one-time idempotent migrator from the legacy file, running **after** registration.
+
+**Risk framing (from ScadaBridge):** Phase 2 replaces a *working* mechanism — a harder risk class
+than Phase 1's "add where none existed." Cutover (delete + rewire) lands in **one commit**; there is
+no dual-mechanism period, and the cutover is the test.
+
+**Hard rules:** identical to Phase 1's list (OnReady ordering, no autoincrement/BLOB, fail-closed
+auth already in place, no in-memory SQLite in tests, cp-triplet-only rig inspection), plus:
+- **Legacy-copy column lists must INTERSECT** with what the legacy file actually has
+  (`pragma_table_info` probe) — a missing column throws and readers silently discard every row.
+- Absence assertions need a positive control.
+- DoD greps phrased as "no references from code" (explanatory comments may survive).
+
+---
+
+### Task 0: Recon (produces `docs/plans/2026-07-20-localdb-phase2-recon.md`)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Create: `docs/plans/2026-07-20-localdb-phase2-recon.md`
+- Read-only: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs` (and the
+  whole `Core.AlarmHistorian` project), `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/AlarmHistorianOptions.cs`,
+  the drain worker (whatever forwards batches to `GatewayHistorian`/`SendEvent`),
+  `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/PrimaryGatePolicy.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` (how `_localRole` +
+  driver-member-count reach the gate today)
+
+**Record, with `file:line` citations:**
+1. The sink's exact table schema: name, columns, PK type (**autoincrement?** — decides whether the
+   migrator needs deterministic `mig-{node}-{legacyId}` ids), indices, and any BLOB columns
+   (**STOP condition:** a BLOB payload column cannot be registered — it must become base64 TEXT in
+   the new schema, and the recon must size the largest realistic payload against the 171 KB-ish
+   chunk guidance; alarm events are small JSON, so expect this to be fine, but verify).
+2. The public seam: the interface the drain worker and producers use (e.g. `IAlarmHistorianSink` /
+   enqueue+dequeue+markDelivered+deadLetter methods), so the rewire can keep it byte-compatible.
+3. Semantics to preserve: `Capacity` (1,000,000) enforcement, `MaxAttempts` (10), dead-letter
+   retention (30 d), `BatchSize` (100), `DrainIntervalSeconds` (5) — where each lives.
+4. The drain worker's lifecycle: hosted service or actor? Where a Primary-role check can be
+   injected, and how the delivered-snapshot role (`RedundancyStateChanged` cache) is accessible
+   from it (via `DriverHostActor`, a shared status service, or a message). If the role is only
+   available inside `DriverHostActor`, note the cleanest bridge (e.g. an `IRedundancyRoleView`
+   singleton the actor updates) — that becomes Task 4's shape.
+5. Whether the sink is constructed per-node config path (`AlarmHistorian:DatabasePath`) anywhere
+   else (tests, tooling).
+6. How `AlarmHistorian:Enabled=false` short-circuits (NullAlarmHistorianSink) — the rewire must
+   keep the disabled path allocating no LocalDb tables? No: tables are created unconditionally in
+   `OnReady` (cheap, empty); only the sink/drain stay Null. Note this in the doc.
+
+Commit: `docs(localdb): phase-2 recon findings`.
+
+---
+
+### Task 1: `alarm_sf_events` schema + registration (+ tests)
+
+**Classification:** standard
+**Estimated implement time:** ~4 min
+**Parallelizable with:** none (Task 2 depends on it)
+
+**Files:**
+- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmSfSchema.cs` (depends only on
+  `Microsoft.Data.Sqlite`)
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs`
+- Test: extend `LocalDbSetupTests`
+
+Schema shape (adjust column names to the recon's findings — preserve today's semantics):
+```sql
+CREATE TABLE IF NOT EXISTS alarm_sf_events (
+    id               TEXT    NOT NULL PRIMARY KEY,   -- app-minted GUID (never autoincrement)
+    payload_json     TEXT    NOT NULL,
+    enqueued_at_utc  TEXT    NOT NULL,
+    attempts         INTEGER NOT NULL DEFAULT 0,
+    status           TEXT    NOT NULL DEFAULT 'pending',  -- pending | delivered | dead
+    last_attempt_utc TEXT    NULL,
+    dead_at_utc      TEXT    NULL
+);
+CREATE INDEX IF NOT EXISTS ix_alarm_sf_events_status ON alarm_sf_events(status, enqueued_at_utc);
+```
+`OnReady` order becomes: Phase-1 DDL → `AlarmSfSchema.Apply` → the two Phase-1
+`RegisterReplicated` calls → `RegisterReplicated("alarm_sf_events")` → **migrator (Task 5) last**.
+(All DDL may run before all registrations; the invariant is registration-before-writes.)
+
+TDD: failing test first — exact replicated set becomes
+`["alarm_sf_events", "deployment_artifacts", "deployment_pointer"]` (ordinal-sorted; update the
+Phase-1 exact-set pins in the same commit — they are *supposed* to go red here, that's them
+working). Oplog-capture test for an `alarm_sf_events` insert. Commit
+`feat(localdb): alarm_sf_events replicated table`.
+
+---
+
+### Task 2: Rewire the sink onto `ILocalDb` + delete bespoke file management (the cutover commit, part 1 of 2 — see Task 3)
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs` (or replace
+  with `LocalDbStoreAndForwardSink.cs` — keep the public seam identical either way)
+- Modify: its registration (`AddAlarmHistorian`) to inject `ILocalDb`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/AlarmHistorianOptions.cs` — remove
+  `DatabasePath` (a breaking config key removal: note it in the runbook/CHANGELOG task)
+- Test: the sink's existing unit tests, rewired to a temp-file `ILocalDb` via `TestLocalDb`-style
+  helper (create `tests/.../TestSupport` helper if none exists — real DB, never a stub: a stubbed
+  bare `SqliteConnection` lacks `zb_hlc_next()` and fails closed on registered tables)
+
+**Steps:**
+1. Write/port failing tests for the seam's semantics: enqueue, drain batch of `BatchSize`,
+   `MaxAttempts` → dead-letter, capacity enforcement, dead-letter retention purge.
+2. Implement over `ILocalDb.ExecuteAsync/QueryAsync` (anonymous-object params). Delete the private
+   connection/pragma/file-open code and any `PRAGMA journal_mode` calls (LocalDb owns pragmas).
+   GUIDs minted at enqueue (`Guid.NewGuid().ToString("N")`).
+3. `Capacity` enforcement: count-based insert guard (preserve today's overflow behavior per recon).
+4. Core.AlarmHistorian gains a package ref on core `ZB.MOM.WW.LocalDb` (interface only).
+5. Build + project tests green. **Commit together with Task 3** if the drain gate can't compile
+   separately (the ScadaBridge tasks-14/15/16 circular-dependency landmine — check before assuming
+   they're independent commits).
+
+---
+
+### Task 3: Primary-gated drain
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:** (exact shape from recon item 4)
+- Modify: the drain worker
+- Create (if recon says so): `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/IRedundancyRoleView.cs`
+  — a singleton snapshot (`RedundancyRole? LocalRole`, `int DriverMemberCount`) updated by
+  `DriverHostActor` where it already caches `_localRole`
+- Test: drain-worker tests + an actor test pinning that `DriverHostActor` publishes role changes to
+  the view
+
+**Semantics:**
+- Drain runs only when `PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount)` is
+  true — same policy, same boot-window posture (unknown role drains only when the node is alone).
+- Delivered/dead-letter marks are row UPDATEs → they replicate, so the standby's copy tracks drain
+  progress and does not re-deliver already-marked rows after failover.
+- **At-least-once across failover is accepted:** rows delivered on the old primary whose
+  `delivered` mark hadn't replicated yet will be re-sent by the new primary. Document in the
+  runbook (Task 7); do NOT build dedup.
+- When replication is OFF (default), the gate still applies but `driverMemberCount` for a solo
+  node keeps today's behavior — verify with a test: single-node, role unknown → drains (no
+  regression for unpaired deployments).
+
+TDD: failing tests — secondary role does not drain; primary drains; unknown+alone drains;
+unknown+paired does not. Commit (with Task 2 if coupled):
+`feat(localdb): alarm S&F on LocalDb with primary-gated drain (cutover)`.
+
+---
+
+### Task 4: One-time legacy migrator (`alarm-historian.db` → consolidated)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 6
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/AlarmSfLegacyMigrator.cs`
+- Modify: `LocalDbSetup.OnReady` — call it **last**, after all `RegisterReplicated` calls
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/AlarmSfLegacyMigratorTests.cs`
+
+Mirror `~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs`:
+- Source path from the pre-removal `AlarmHistorian:DatabasePath` default (`alarm-historian.db`) —
+  read the raw config key even though the option property is gone.
+- `ShouldMigrate`: skip if file missing or `<file>.migrated` exists. `:memory:`/`file:` sources → no-op.
+- **If the legacy PK is autoincrement (recon):** deterministic ids `mig-{NodeName}-{legacyId}`
+  (node name from the recon-identified config key) — rerunnable without duplicates under
+  `INSERT OR IGNORE`, and no cross-node collision. If already GUIDs, copy as-is.
+- One transaction for the whole copy; `File.Move(path, path + ".migrated")` only after commit;
+  failure throws out of `OnReady` → boot fails, legacy untouched.
+- **Column intersection** via `pragma_table_info` on the legacy table; required-PK guard.
+- Both pair nodes migrate independently; their rows have distinct ids (node-prefixed), so the
+  merged buffer is the union — expected; note that the new primary will drain the standby's
+  migrated rows too.
+
+TDD: failing tests — happy path row counts; idempotent re-run; failure leaves legacy file
+untouched; migrated rows **enter the oplog** (assert `__localdb_oplog` — the registration-order
+pin); older legacy file missing a column still migrates (intersection). Commit
+`feat(localdb): one-time alarm-historian.db migrator`.
+
+---
+
+### Task 5: Convergence + failover scenarios in the pair harness
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 4
+
+**Files:**
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/AlarmSfConvergenceTests.cs`
+  (reuses the Phase-1 `LocalDbPairHarness`)
+
+**Scenarios:**
+1. `AlarmBurstOnA_ConvergesToB_AndOplogDrains` — enqueue N events on A; identical rowset on B;
+   oplog → 0.
+2. `DeliveredMarksReplicate` — mark rows delivered on A; B's copies show delivered (the
+   no-redeliver-after-failover property, asserted at the data layer).
+3. `WritesWhileTransportDown_SurviveRejoin`.
+4. Positive control: with `RegisterReplicated("alarm_sf_events")` commented out, scenarios 1–3 go
+   red (run locally, record, restore — do not commit red).
+
+Commit `test(localdb): alarm S&F convergence scenarios`.
+
+---
+
+### Task 6: Rig config + docs
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 4
+
+**Files:**
+- Modify: `docker-dev/docker-compose.yml` — enable `AlarmHistorian__Enabled=true` on the site-a
+  pair if not already (the gate needs a live buffer); remove any `AlarmHistorian__DatabasePath`
+  env vars (key deleted).
+- Modify: `docs/operations/2026-07-20-localdb-pair-replication.md` — add: alarm S&F replication
+  semantics, at-least-once-across-failover statement, migrator behavior (`.migrated` sidecar),
+  `DatabasePath` key removal.
+- Modify: `docs/AlarmHistorian.md` + `CLAUDE.md` — sink now lives in the consolidated LocalDb;
+  drain is primary-gated.
+
+Commit `docs+chore(localdb): phase-2 rig config + docs`.
+
+---
+
+### Task 7: DoD sweep (offline)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+1. Full solution build → 0 warnings; full test suite green (deltas vs pre-branch baseline only).
+2. Greps, phrased as "no references from **code**": the old bespoke connection management
+   (`AlarmHistorian:DatabasePath`, direct `new SqliteConnection` inside Core.AlarmHistorian except
+   via schema helpers/tests) — explanatory comments may remain.
+3. Positive-control evidence from Task 5 recorded.
+4. Exact-set replicated-tables pin = 3 tables, both directions.
+5. Update `…phase2.md.tasks.json`; commit `chore(localdb): phase-2 DoD sweep`; STOP and report.
+
+---
+
+### Task 8: Live gate on the docker-dev rig (run only with explicit user go-ahead)
+
+**Classification:** high-risk (rig)
+**Estimated implement time:** ~30 min wall-clock
+**Parallelizable with:** none
+
+Same inspection rules as Phase 1's gate (cp-triplet, curl sidecar, observer-suspicion rule).
+Record in `docs/plans/2026-07-20-localdb-phase2-live-gate.md`:
+1. Migration ran: `.migrated` sidecar present on both site-a nodes; row counts match legacy.
+2. Alarm burst (drive a real driver alarm or the historian-gateway-unreachable path) converges:
+   identical rowsets, oplog drains to 0, dead letters 0.
+3. Only the primary drains: stop the historian gateway egress, buffer builds on BOTH nodes'
+   tables (replicated), but only the primary's drain worker logs attempts.
+4. Failover: stop the primary; the standby (new primary) resumes draining the shared buffer;
+   count of double-delivered events observed and recorded (at-least-once evidence, not a failure).
+5. Both-nodes-together restart clean; zero `disk I/O error`/`SQLITE_IOERR` in logs.
+6. site-b (default-OFF pin): sink works locally, no sync traffic.
+
+Commit the gate doc; report; do not merge.
+
+---
+
+## Task persistence
+
+Tasks file: `docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json`. Record deviations per
+task — especially if Tasks 2/3 had to land as one commit (the expected outcome).

docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json

@@ -0,0 +1,15 @@
+{
+  "planPath": "docs/plans/2026-07-20-localdb-adoption-phase2.md",
+  "tasks": [
+    { "id": 0, "subject": "Task 0: Recon — sink schema, seam, drain lifecycle, role-view bridge (STOP conditions)", "status": "pending" },
+    { "id": 1, "subject": "Task 1: alarm_sf_events schema + registration (+ exact-set pin update)", "status": "pending", "blockedBy": [0] },
+    { "id": 2, "subject": "Task 2: Rewire sink onto ILocalDb + delete bespoke file management (cutover 1/2)", "status": "pending", "blockedBy": [1] },
+    { "id": 3, "subject": "Task 3: Primary-gated drain via PrimaryGatePolicy (cutover 2/2 — may co-commit with Task 2)", "status": "pending", "blockedBy": [2] },
+    { "id": 4, "subject": "Task 4: One-time alarm-historian.db legacy migrator", "status": "pending", "blockedBy": [3] },
+    { "id": 5, "subject": "Task 5: Convergence + failover scenarios in the pair harness (+ positive control)", "status": "pending", "blockedBy": [3] },
+    { "id": 6, "subject": "Task 6: Rig config + docs", "status": "pending", "blockedBy": [3] },
+    { "id": 7, "subject": "Task 7: DoD sweep (offline) — STOP and report after this", "status": "pending", "blockedBy": [4, 5, 6] },
+    { "id": 8, "subject": "Task 8: Live gate on the docker-dev rig (needs explicit user go-ahead)", "status": "pending", "blockedBy": [7] }
+  ],
+  "lastUpdated": "2026-07-20T00:00:00Z"
+}

docs/plans/2026-07-20-localdb-phase1-live-gate.md

@@ -0,0 +1,93 @@
+# LocalDb Phase 1 — live gate (docker-dev rig)
+
+> Task 16 of `2026-07-20-localdb-adoption-phase1.md`. Evidence log for the 8 live-rig checks.
+> Rig: local `docker-dev/docker-compose.yml` (6 host nodes, one shared SQL). site-a pair replicates;
+> central + site-b are default-OFF.
+>
+> **Safety rules followed:** all DB inspection via `docker cp` of the `db`/`-wal`/`-shm` triplet out
+> of the container, querying the copy — never host `sqlite3` on the live WAL file. Metrics via a
+> `curlimages/curl` sidecar with `--network container:<node>` (`aspnet:10.0` has no curl).
+
+## Rig build/up
+
+Two real defects surfaced by the rebuild (both fixed on-branch, re-verified):
+
+1. **`NU1101` — packageSourceMapping gap** (`4b2f0e6e`). `ZB.MOM.WW.LocalDb*` was not mapped to the
+   `dohertj2-gitea` feed, so any *clean* restore (Docker image build, CI, fresh clone) failed. Local
+   dev builds masked it via the warm global NuGet cache. Fixed by adding the two patterns; verified
+   with a temp-packages-dir restore.
+2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`). The Kestrel re-bind read only
+   `urls`/`ASPNETCORE_URLS`; the aspnet:8.0+ base image sets `ASPNETCORE_HTTP_PORTS=8080` (all
+   interfaces) as the container default instead. A driver node that never sets `URLS` fell through to
+   `localhost:5000`, silently moving its health/metrics surface to loopback. Central nodes were fine
+   (they set `ASPNETCORE_URLS=9000` explicitly). Fixed to fall through URLS → HTTP_PORTS/HTTPS_PORTS →
+   5000; 4 new unit tests. After the fix, driver nodes bind `[::]:8080` + `[::]:9001` (site-a) /
+   `[::]:8080` only (site-b).
+
+Rig: 6 nodes rebuilt from branch `feat/localdb-phase1` @ `ce9fa07f`, all up.
+
+## Checks
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 1 | Rig up, site-a pair healthy, `/metrics` shows `localdb_` series | ✅ PASS | Both site-a nodes `/healthz` 200 Healthy; `/metrics` shows `localdb_oplog_depth` + `localdb_sync_reconnects_total` on meter `ZB.MOM.WW.LocalDb.Replication` (allowlist works); site-a-2 served `/localdb_sync.v1.LocalDbSync/Sync` with routing `match_status=success` (interceptor passed, session established); both oplogs depth 0 |
+| 2 | Deploy → both site-a nodes' pointer + artifacts byte-identical, same HLC + origin | ✅ PASS | Deploy `6e687451…` (rev `efb04c79…`) via `POST :9200/api/deployments` → both site-a-1 and site-a-2 `deployment_pointer` = `SITE-A / 6e687451… / efb04c79… / sha EFB04C79…` (identical); artifact 1 chunk each; **pointer `__localdb_row_version` identical on both: hlc `116955620225646592`, origin node_id `a0576df6-…`, tombstone 0** — one row replicated, not two derivations; both oplogs depth 0 |
+| 3 | Boot-from-cache: SQL down, restart site-a-2, serves cached config w/ signal | ✅ PASS (found + fixed a defect) | With SQL stopped, site-a-2 restart logged `RUNNING FROM CACHE — … booted deployment 6e687451…`; **address space materialised from the cached blob**: `AddressSpaceApplier: applied plan (added=18)` / `OpcUaPublish: applied rebuild (added=18)`. Browse confirms **16 nodes on site-a-2 = 16 on site-a-1**. **Defect the gate caught (`9137cb41`):** pre-fix the rebuild re-read the artifact from the down ConfigDb and no-op'd, so the cache-booted node served **1** node vs the healthy peer's 16 — it logged success but browsed empty. Fixed by passing the in-hand cached blob to `RebuildAddressSpace`; regression test added |
+| 4 | Replicated-cache payoff: wipe a-2 DB, repopulate, then SQL-down boot | ⚠️ PARTIAL — 1 defect fixed; the limitation is since **CLOSED** (LocalDb 0.1.2 — see Summary) | Wiped a-2's LocalDb volume, restarted with **SQL down**: sync session re-established (a-2 served a `Sync` call) but a-2's cache stayed **0/0/0** and a-1's oplog was **0** — the deploy's rows had been acked by the old a-2 and pruned, so there was no delta to send and **no snapshot-resync fired**. ⇒ **Limitation: pure replication does not back-fill a fully-wiped, already-converged node** (library delta-replication; snapshot-resync is gated on the oplog cap, which the running engine never hits). Separately found a **defect (`c6a9f93a`)**: recovery via `RestoreApplied` never wrote the cache (only fresh `ApplyAndAck` did), so a wiped node stayed cache-less. Fixed: `RestoreApplied` now re-caches. Verified — after the fix, wiped a-2 restarted with SQL up: `restored served state … on bootstrap` → cache repopulated `SITE-A / 6e687451`, chunks=1. **Net: a wiped node self-heals its cache from central on next boot (regaining boot-from-cache for future outages); it is not back-filled by its peer during the same outage.** |
+| 5 | Transport kill: pause a-2, oplog rises; unpause, drains to 0 | ✅ PASS | `docker pause` on a-2 severed the sync transport; during the window a-2 accumulated **oplog=2** unacked local writes (its re-cache) while a-1 held 0. `docker unpause` → a-2's oplog **drained to 0 within 5s**, both converged on the same pointer. Note: the rig's redeploys carry an identical revision, so a fresh cache write on a-1 short-circuits (no-op) — the store-during-partition variant is covered by the automated harness scenario `WritesWhileTransportDown_SurviveRejoin` (proven red without `RegisterReplicated`) |
+| 6 | Retention: 3rd deploy prunes oldest; chunks gone + tombstones on BOTH nodes | ✅ PASS | Cached 3 distinct deployment-ids (via deploy + restart re-cache). Both nodes retain exactly the newest 2 (`475df87a`, `962962dc`); the oldest `6e687451`'s chunks are **0 on both** (pruned); **1 artifact tombstone on both** — the prune replicated as a tombstone, not left as a live chunk |
+| 7 | Both-nodes-together restart: clean rejoin, identical counts, no I/O errors | ✅ PASS | Restarted site-a-1 + site-a-2 together: **0** `disk I/O error`/`SQLITE_IOERR`/`corrupt` lines in either log; both re-cached and converged to **identical** counts (ptr `962962dc…`, chunks=2, rowver=3). Byte-identical convergence preserved across a simultaneous restart |
+| 8 | site-b default-OFF pin: cache works locally, Healthy, no sync listener on 9001 | ✅ PASS (+1 finding, since **CLOSED** by the skip-if-unchanged cache write — see Summary) | site-b-1 cache works locally (`SITE-B / 6e687451`, chunks=1, from its own apply); listeners **`[::]:8080` only — no 9001**; port 9001 **not bound** (`/proc/net/tcp`); `/healthz` **200**; no sync/reconnect metric series. **Finding (follow-up, not a blocker):** `localdb_oplog_depth=5` — because `OnReady` registers replication unconditionally, a default-OFF node captures cache writes into its oplog but has no peer to ack/prune them, so the oplog grows slowly (≈2 rows/deploy, amplified by the re-cache-on-restore fix). Bounded by the library `MaxOplogRows` cap (1M); worth a periodic pruner or skip-if-unchanged in a follow-up |
+
+## Summary
+
+**8/8 checks pass** (check 4 with a documented limitation). The gate exercised the real image built
+from the branch, the real interceptor over a real loopback+cross-container h2c transport, and real
+`docker cp`-triplet DB inspection — and caught **four real defects that every offline test had
+missed**, each now fixed on-branch with a regression test:
+
+1. **`NU1101` packageSourceMapping gap** (`4b2f0e6e`) — `ZB.MOM.WW.LocalDb*` unmapped to the gitea
+   feed; every clean restore (Docker, CI, fresh clone) failed. Masked locally by the warm NuGet cache.
+2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`) — the sync-listener re-bind ignored the
+   aspnet:8.0+ container-default port var, silently moving driver health/metrics to loopback:5000.
+3. **Empty address space on boot-from-cache** (`9137cb41`) — the rebuild re-read the artifact from the
+   down ConfigDb and no-op'd, so a cache-booted node logged success but served clients **1 node vs the
+   healthy peer's 16**. The single most important find: boot-from-cache was half-working.
+4. **Cache not repopulated on `RestoreApplied`** (`c6a9f93a`) — a wiped/fresh-volume node recovered its
+   served state from central but stayed cache-less, unable to boot-from-cache on the next outage.
+
+**Two documented limitations (follow-ups, not blockers) — both since CLOSED:**
+
+- **No replication back-fill of a fully-wiped node** (check 4): a peer's already-acked cache rows are
+  pruned from its oplog and snapshot-resync is gated on the (never-hit) oplog cap, so a wiped node is
+  not healed by its peer — it self-heals from central on next boot instead.
+  → **FIXED in `ZB.MOM.WW.LocalDb` 0.1.2 + 0.1.3** (scadaproj `cad3bcb` + `3b7489a`), live-gated in
+  `2026-07-21-localdb-followups-live-gate.md`. The diagnosis in this doc was
+  slightly off: the cap was not the gate. `ComputeSnapshotRequiredAsync` measured the peer's gap
+  against the *oldest surviving oplog row* and read an empty oplog as "no gap possible" — and an
+  empty oplog is the **steady state of a converged pair**, since ack-pruning deletes everything the
+  peer has confirmed. So the healthy state was the one state that could not heal a wiped peer. With
+  an empty oplog the gap is now measured against `last_acked_seq`. Covered at both levels:
+  `SnapshotResyncTests.WipedPeer_AfterEverythingWasAckedAndPruned_StillGetsSnapshot` in the library,
+  and `LocalDbPairConvergenceTests.WipedNode_IsBackFilledByItsPeer_WithoutAnyNewDeploy` here —
+  the latter verified RED against the pinned 0.1.1 and green on 0.1.2. **0.1.3 closes a second,
+  deeper half that only became reachable once back-fill worked:** the rebuilt node's OWN writes were
+  silently dropped, because `last_applied_remote_seq` is a watermark in the *peer's* seq space and a
+  rebuilt peer numbers from 1 again. The follow-up live gate caught that one — offline tests could
+  not have, since the scenario had no test while it was still a documented limitation.
+- **Oplog growth on default-OFF nodes** (check 8): `OnReady` registers replication unconditionally, so
+  a node with no peer accumulates unacked/unpruned oplog rows (~2/deploy). Slow, bounded by the 1M cap;
+  a periodic pruner or a skip-if-unchanged cache write would close it.
+  → **FIXED at the source**: `LocalDbDeploymentArtifactCache.StoreAsync` now **skips entirely** when
+  the pointer already names this deployment/revision, the SHA matches the bytes, and the expected
+  chunk count is present — so re-caching an artifact the node already holds (every restart's
+  boot-from-cache, every `RestoreApplied`) writes nothing and mints no oplog rows. Two guard tests
+  pin what the skip must NOT do (stale bytes under an unchanged revision; a matching pointer over
+  missing chunks), both verified RED against a naive pointer-only skip.
+  Also correcting this row's bound: the growth was never headed for the 1M cap. `AddZbLocalDbReplication`
+  is registered unconditionally, so `MaintenanceBackgroundService` runs on default-OFF nodes too and
+  the **7-day `MaxOplogAge`** cap prunes — the true bound was ~7 days of writes, not 1M rows.
+
+Merged to `master` as `c957db52` (2026-07-21) and pushed. Both follow-ups closed on
+`fix/localdb-phase1-followups` (2026-07-21) and live-gated — see
+`2026-07-21-localdb-followups-live-gate.md`, which found a third defect in the process.

docs/plans/2026-07-20-localdb-phase1-progress.md

@@ -0,0 +1,72 @@
+# LocalDb Phase 1 — Execution Progress (resume state)
+
+**Branch:** `feat/localdb-phase1` (off `master`). Working tree CLEAN, all work committed, nothing pushed.
+**Plan:** `docs/plans/2026-07-20-localdb-adoption-phase1.md`
+**Recon:** `docs/plans/2026-07-20-localdb-phase1-recon.md` (READ THIS — it has the 8 deviations D-1..D-8 that override the plan text).
+**Skill in use:** `superpowers-extended-cc:executing-plans` — batch, verify, commit per task, report between batches.
+**Do NOT merge to master** — plan stops at Task 15 (DoD) and reports; Task 16 is a rig live-gate needing explicit user go-ahead.
+
+## Status: Tasks 0–9 DONE (10 commits). Tasks 10–16 REMAIN.
+
+| # | Task | State | Commit |
+|---|---|---|---|
+| 0 | Preflight + recon | ✅ | `42f85507` |
+| 1 | Package refs (LocalDb 0.1.1, Grpc.AspNetCore 2.76.0) | ✅ | `44255fcb` |
+| 2 | Schema + LocalDbSetup.OnReady + tests | ✅ | `3b65c24b` |
+| 3 | Fail-closed sync auth interceptor + tests | ✅ | `3d29be83` |
+| 4 | LocalDbRegistration + Program.cs DI + appsettings | ✅ | `6aff9a83` |
+| 5 | h2c sync listener + MapZbLocalDbSync (Kestrel) | ✅ | `b9ddf20e` |
+| (fix) | rename Runtime.Deployment → Runtime.DeploymentCache | ✅ | `e771a11a` |
+| 6 | IDeploymentArtifactCache + chunked impl + tests | ✅ | `a38a52b8` |
+| 9 | Delete LiteDB LocalCache | ✅ | `2bae1b4c` |
+| 7 | Cache write path in DriverHostActor | ✅ | `1becf591` |
+| 8 | Boot-from-cache + RunningFromCache signal | ✅ | `a27eff32` |
+| 10 | Health check + telemetry meter allowlist | ⬜ TODO | |
+| 11 | DI-pin integration tests over real host graph | ⬜ TODO | |
+| 12 | 2-node convergence harness + scenarios | ⬜ TODO | |
+| 13 | docker-dev rig config | ⬜ TODO | |
+| 14 | Documentation | ⬜ TODO | |
+| 15 | DoD sweep (offline) — STOP + report | ⬜ TODO | |
+| 16 | Live gate on docker-dev rig — NEEDS USER GO-AHEAD | ⬜ TODO | |
+
+Native task list (TaskCreate ids 1–17 map to plan tasks 0–16) tracks the same; tasks.json in plan dir mirrors it.
+
+## What exists now (key files)
+
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/DeploymentCacheSchema.cs` — DDL, namespace `Runtime.DeploymentCache`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactCache.cs` — `StoreAsync` / `GetCurrentAsync` / **`GetCurrentUnkeyedAsync`** (D-1 addition) + `CachedDeploymentArtifact` record.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/LocalDbDeploymentArtifactCache.cs` — chunked impl. NOTE the prune guard `AND deployment_id <> @DeploymentId` (bug I found: pointer-target could be pruned under same-tick GUIDs → permanent silent miss).
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs` — `OnReady` (DDL→RegisterReplicated). PUBLIC (Host has no InternalsVisibleTo).
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs` — `AddOtOpcUaLocalDb`, `SyncListenPort(config)`. Registers `IDeploymentArtifactCache → LocalDbDeploymentArtifactCache` singleton.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs` — fail-closed bearer.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/KestrelHttpBinding.cs` — URL parse/re-bind helper (D-6).
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` — LocalDb wired in `hasDriver` branch (~L176 after AddAlarmHistorian: `AddOtOpcUaLocalDb` + `AddGrpc(interceptor)`); Kestrel re-bind block before `builder.Build()`; `MapZbLocalDbSync()` gated `hasDriver && syncListenPort>0` before `MapOtOpcUaHealth()`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json` — `LocalDb` section (Path, SyncListenPort:0, Replication{}), between AlarmHistorian and Deployment.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` — cache dep threaded LAST in Props/ctor/forwarding; `CacheAppliedArtifact` (write, after PushDesiredSubscriptions, empty-blob skip, own try/catch); `TryBootFromCache` + `ApplyCachedArtifact` (in Bootstrap catch); `PushDesiredSubscriptionsFromArtifact` split out; `_isRunningFromCache` field; `SingleClusterCacheKey="__single"`; `ReconcileDrivers` now returns `byte[]?`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs` — resolves `IDeploymentArtifactCache` (optional, nullable) and passes into DriverHostActor.Props.
+- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Interfaces/NodeDiagnosticsSnapshot.cs` — added `bool RunningFromCache = false`.
+- Tests (all green): `Host.IntegrationTests/LocalDbSetupTests.cs`, `LocalDbSyncAuthInterceptorTests.cs`, `LocalDbSyncListenerTests.cs`; `Runtime.Tests/Deployment/LocalDbDeploymentArtifactCacheTests.cs`, `Runtime.Tests/Drivers/DriverHostActorArtifactCacheTests.cs`, `DriverHostActorBootFromCacheTests.cs`.
+
+## Load-bearing facts for the remaining tasks
+
+- **Namespace trap (recurs!):** any `Deployment` child namespace under `Runtime` OR `Runtime.Tests` shadows the `Deployment` EF entity → CS0118 across the assembly. Use `...DeploymentCache`. Only a FULL-SOLUTION build catches it (per-project builds pass).
+- **xUnit1051 is an error** in these test projects: every `ILocalDb`/DB call in a test must pass `TestContext.Current.CancellationToken` (xunit.v3 Host.IntegrationTests) — but `Runtime.Tests` is **xunit v2** (no `TestContext.Current`).
+- **Test project reality (D-5):** NO `Host.Tests` project; NO `WebApplicationFactory<Program>` (deliberately — `TwoNodeClusterHarness.cs:48` explains why). Host unit tests live in `Host.IntegrationTests` (xunit.v3). Actor tests in `Runtime.Tests` (xunit v2 + Akka.TestKit.Xunit2).
+- **Task 10 (D-7):** `AddOtOpcUaHealth()` takes NO args, called unconditionally at `Program.cs:365`. DON'T change its signature — register the LocalDb check unconditionally and return `Healthy` when LocalDb absent (ActiveNodeHealthCheck precedent). Health check classes: there are ZERO in-repo today; `Host/Health/` has only `HealthEndpoints.cs`. Meter allowlist is REAL and REQUIRED: `Host/Observability/ObservabilityExtensions.cs:30` `o.Meters = [OtOpcUaTelemetry.MeterName]` — add `LocalDbMetrics.MeterName` (from `ZB.MOM.WW.LocalDb.Replication`). Health check reads `ISyncStatus` + `IOptions<ReplicationOptions>`.
+- **Task 11:** copy the `TwoNodeClusterHarness` direct-host-build pattern, NOT WebApplicationFactory. Pins: `ILocalDb` singleton + `ReplicatedTables` == `["deployment_artifacts","deployment_pointer"]`; `IDeploymentArtifactCache`→concrete; `ISyncStatus` default-OFF (`Connected==false`); admin-only graph has NO `ILocalDb` + doesn't demand `LocalDb:Path`; health check registered. Set `OTOPCUA_ROLES` per case; `LocalDb:Path`→temp file, `ClearAllPools`+delete triplet in cleanup.
+- **Task 12:** harness copied from `~/Desktop/ScadaBridge/tests/.../LocalDbSitePairHarness.cs` + `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/tests/.../Convergence/ConvergenceFixture.cs`. Both nodes init via PRODUCTION `LocalDbSetup.OnReady`. Real loopback Kestrel h2c through the REAL interceptor. DBs as pre-constructed singletons. Positive control: comment out both `RegisterReplicated`, confirm scenarios 1–3 red, restore (don't commit red).
+- **Task 13 (D-8):** NO host node has a writable volume today — must ADD named volumes. `docker-dev/docker-compose.yml` services: `central-1/2` (admin,driver, MAIN), `site-a-1/2` (driver, SITE-A), `site-b-1/2` (driver, SITE-B). Env style `Section__Key`. `9001` free container-internal. Enable replication on **site-a pair only** (a-1 initiator w/ PeerAddress, a-2 passive, byte-identical ApiKey, MaxBatchSize=16). site-b stays OFF = default-OFF pin. Validate `docker compose config`; DON'T bring rig up.
+- **Task 15:** baseline pre-existing test failures FIRST (`git stash` → run full `dotnet test` → unstash), report deltas only. **KNOWN intermittent:** one `Runtime.Tests` failure seen ~3× across runs (mine + subagent's), never reproduced under trx logger, NOT one of the new tests — investigate/baseline here.
+- **Warnings-as-errors** across src + most test projects. Pre-existing CS0618/xUnit1051 warnings exist ONLY in `Client.Shared*` (which don't set TreatWarningsAsErrors) — not mine, ignore.
+- **SQLitePCLRaw:** no pin needed — LocalDb 0.1.1 nuspec floors `lib.e_sqlite3` at 2.1.12 (CVE-fixed). Confirmed.
+
+## Follow-ups logged (NOT in scope, don't action without asking)
+
+- `Polly.Core` in `Configuration.csproj` now orphaned (ResilientConfigReader was its only real consumer). Left in place — recorded in recon §7.3b.
+- Pre-existing: `TryRecoverFromStale` marks Applied w/o reconciling drivers (recon §3.5); `Install-Services.ps1` gives driver-only nodes no `ASPNETCORE_URLS` (recon §6.4, worked around by D-6).
+
+## Verify-anything-quick
+
+Full build: `dotnet build ZB.MOM.WW.OtOpcUa.slnx` (expect 0 errors). LocalDb tests:
+`dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LocalDb"` and
+`dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~ArtifactCache|FullyQualifiedName~BootFromCache"`.

docs/plans/2026-07-20-localdb-phase1-recon.md

@@ -0,0 +1,617 @@
+# LocalDb Phase 1 — Task 0 Recon Findings
+
+**Date:** 2026-07-20
+**Branch:** `feat/localdb-phase1` (based on `master`)
+**Plan:** `docs/plans/2026-07-20-localdb-adoption-phase1.md`
+**Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md`
+
+All `file:line` citations are against the branch base commit.
+
+---
+
+## Step 1 — Feed verification: PASS
+
+```
+$ dotnet package search ZB.MOM.WW.LocalDb --source dohertj2-gitea --format json
+ZB.MOM.WW.LocalDb              0.1.1
+ZB.MOM.WW.LocalDb.Contracts    0.1.1
+ZB.MOM.WW.LocalDb.Replication  0.1.1
+```
+
+All three present at **0.1.1**. No STOP. (Note: the local NuGet cache holds both `0.1.0` and `0.1.1`
+— pin exactly `0.1.1` so a stale cache cannot win.)
+
+---
+
+## Step 3 — STOP conditions: NONE TRIGGERED
+
+| STOP condition | Verdict |
+|---|---|
+| Artifact apply path cannot take a post-apply hook without restructuring the actor | **Clear** — `ApplyAndAck` has a clean insertion point (§2.2) |
+| `DeploymentId` / cluster identity are not stable strings/GUIDs | **Clear** — both are stable strings, but cluster identity is **not resolvable at boot**; see D-1 |
+| LiteDB LocalCache is referenced by live code | **Clear** — zero live-code references (§7) |
+
+Three STOP conditions clear. **Eight deviations** from the plan's stated assumptions are recorded in
+§9 — four of them (D-1, D-3, D-5, D-6) change the work materially.
+
+---
+
+## 1. Deploy fetch path
+
+### 1.1 Blob-load sites — there are **five** `CreateDbContext()` calls, not two
+
+| Line | Method | Purpose |
+|---|---|---|
+| `DriverHostActor.cs:513` | `Bootstrap()` | `NodeDeploymentStates` + `Deployments` at PreStart |
+| `DriverHostActor.cs:1472` | `ReconcileDrivers(DeploymentId)` | **loads `ArtifactBlob`** |
+| `DriverHostActor.cs:1543` | `PushDesiredSubscriptions(DeploymentId)` | **loads `ArtifactBlob` again** |
+| `DriverHostActor.cs:2000` | `TryRecoverFromStale()` | latest `Sealed` deployment (id + hash only, no blob) |
+| `DriverHostActor.cs:2029` | `UpsertNodeDeploymentState(...)` | writes `NodeDeploymentState` |
+
+`DriverHostActor.cs:1472-1476`:
+```csharp
+using var db = _dbFactory.CreateDbContext();
+blob = db.Deployments.AsNoTracking()
+    .Where(d => d.DeploymentId == deploymentId.Value)
+    .Select(d => d.ArtifactBlob)
+    .FirstOrDefault() ?? Array.Empty<byte>();
+```
+`:1543-1547` is byte-identical apart from the local. **The same blob is read twice per apply.**
+
+### 1.2 Artifact runtime type — there is no instance type
+
+`DeploymentArtifact` (`DeploymentArtifact.cs:48`) is a **static parser class** over
+`ReadOnlySpan<byte>`. The blob is never materialised into an object graph.
+
+- Storage: `byte[]` — `Deployment.cs:29` `public byte[] ArtifactBlob { get; init; } = Array.Empty<byte>();`
+- Format: **UTF-8 JSON**, `JsonDocument.Parse` (`DeploymentArtifact.cs:66`, `:490`). Not MessagePack.
+- Leniency contract: empty/malformed blob → empty result, **never throws** (`:62`, `:537-541`).
+
+**Consequence for the cache:** we cache raw `byte[]`. No serializer coupling. Good.
+
+### 1.3 Identity types
+
+| Type | Definition | Shape |
+|---|---|---|
+| `DeploymentId` | `Commons/Types/DeploymentId.cs:3` — `readonly record struct DeploymentId(Guid Value)` | `ToString()` → `Value.ToString("N")`, 32 hex chars, **no hyphens** (`:10`) |
+| `RevisionHash` | `Commons/Types/RevisionHash.cs:8` — `readonly record struct RevisionHash(string Value)` | SHA-256 hex, **lowercase 64 chars**, no `0x`; empty invalid (`:3-6`) |
+
+Both are stable, filesystem/SQL-safe strings. Use `DeploymentId.ToString()` (the "N" form) as the
+`deployment_id` TEXT column value.
+
+---
+
+## 2. Post-apply point
+
+### 2.1 `ApplyAndAck` control flow — `DriverHostActor.cs:1413-1458`
+
+```
+1415  _applyingDeploymentId = deploymentId;
+1416  Become(Applying);
+1425  UpsertNodeDeploymentState(..., Applying, null)
+1427  try {
+1429      ReconcileDrivers(deploymentId);            // blob fetch #1 — SWALLOWS DB errors
+1430      _currentRevision = revision;
+1431      UpsertNodeDeploymentState(..., Applied, null);
+1432      SendAck(..., ApplyAckOutcome.Applied, null, correlation);
+1436      _opcUaPublishActor?.Tell(RebuildAddressSpace(...));
+1439      PushDesiredSubscriptions(deploymentId);    // blob fetch #2 — SWALLOWS DB errors
+1440      OtOpcUaTelemetry.DeploymentApplied.Add(1, "outcome"="ack");
+1441      _log.Info("applied deployment ...");
+1443  }
+1444  catch (Exception ex) { ...Failed ACK... }
+1452  finally { _applyingDeploymentId = null; Become(Steady); }
+```
+
+### 2.2 Where the cache write belongs
+
+**Immediately after `DriverHostActor.cs:1439`**, before `:1440` — but **wrapped in its own
+try/catch**. Reason: `:1432` already sent an `Applied` ACK to the coordinator. An unguarded cache
+write that throws would fall into the `catch` at `:1444` and send a *second*, contradictory
+`Failed` ACK for a deployment the fleet already believes is applied.
+
+### 2.3 Paths that must NOT write to the cache
+
+1. **`catch` at `:1444`** — any apply failure.
+2. **Idempotent short-circuit at `:1402-1409`** — `HandleDispatchFromSteady` returns early with an
+   `Applied` ACK when `_currentRevision == msg.RevisionHash`. `ApplyAndAck` is never entered and no
+   blob is read.
+3. **⚠ THE SILENT-DEGRADATION TRAP.** `ReconcileDrivers:1478-1483` and
+   `PushDesiredSubscriptions:1549-1553` catch DB failures, log a **warning**, and `return` **without
+   rethrowing**. `ApplyAndAck` therefore reaches `:1440` and declares success having applied **zero
+   drivers** when the blob load failed. Caching "whatever we read" would **persist an empty artifact
+   as a successful apply** — poisoning the boot-from-cache path with a config that starts no drivers.
+
+   **Mitigation (load-bearing):** the cache write must be gated on the blob having actually loaded
+   non-empty. Have `ReconcileDrivers` surface the loaded blob upward (a field or an out-param)
+   rather than re-reading it a third time, and skip the cache write when it is
+   `Array.Empty<byte>()`. This also removes the duplicate read noted in §1.1.
+
+4. **`RestoreApplied` (`:1514-1528`)** — the bootstrap replay path. Restoring from central; caching
+   here is harmless and arguably desirable, but it is a distinct seam. **Phase 1 scope: skip it.**
+5. **`TryRecoverFromStale` (`:1996-2022`)** — never reads the artifact at all (see §3.4).
+
+---
+
+## 3. Cold-boot path
+
+### 3.1 PreStart — `DriverHostActor.cs:415-436`
+
+Ctor already did `Become(Steady)` at `:411`. PreStart subscribes to topics, spawns the virtual-tag
+and scripted-alarm hosts, then calls `Bootstrap()` at `:435`.
+
+### 3.2 `Bootstrap()` — `DriverHostActor.cs:506-565` — **yes, it queries central SQL at boot**
+
+Two queries inside one `try`: latest `NodeDeploymentState` for self (`:514-518`), then the matching
+`Deployment` row (`:527-532`). Four outcomes: no prior deployments → Steady; `Applied` →
+`RestoreApplied` (`:544`); `Applying` orphan → `ApplyAndAck` replay (`:549`); `Failed` → Steady.
+
+### 3.3 **THE SEAM** — `DriverHostActor.cs:560-564`
+
+```csharp
+catch (Exception ex)
+{
+    _log.Warning(ex, "DriverHost {Node}: ConfigDb unreachable on bootstrap; entering Stale", _localNode);
+    Become(Stale);
+}
+```
+
+**This is the exact point where "central fetch failed at boot" is known.** The cache fallback goes
+between `:562` and `:563`: attempt a local-cache load, and only `Become(Stale)` if the cache is also
+a miss.
+
+⚠ The catch also fires if the *second* query throws after the first succeeded, so `latest` is not in
+scope-valid state from the catch. **The cache must be self-describing** — it must carry its own
+`DeploymentId` + `RevisionHash`, because at this seam the actor has no idea what it should be
+restoring. The planned `deployment_pointer` shape already satisfies this.
+
+### 3.4 The `Stale` state — `DriverHostActor.cs:1343-1379`
+
+`ReconnectInterval = 30s` (`:55`), retry timer started at `:1378`.
+
+The dispatch-ignore at **`:1345-1348`** discards `DispatchDeployment` entirely — no ACK, no
+buffering (contrast `Applying:599` which does `Self.Forward(msg)`). A Stale node looks like a
+timeout to the coordinator. Phase 1 does **not** change this (per design §3.3: a *new* deployment
+still requires central).
+
+`RouteNodeWrite` fast-fails at `:1359-1360` with `"driver host stale (config DB unreachable)"`.
+
+### 3.5 Pre-existing gap worth recording (not Phase 1 scope)
+
+`TryRecoverFromStale` (`:1996-2022`) sets `_currentRevision` and marks `Applied` **without calling
+`ReconcileDrivers` or `PushDesiredSubscriptions`** — documented at `:1692-1695` and `:1706`. A
+Stale-recovered node reports Applied while running **zero drivers**, and
+`HandleDispatchFromSteady:1402` then short-circuits on revision match. Boot-from-cache largely
+neutralises this in practice, but the bookkeeping remains optimistic. **Logged as a follow-up, not
+fixed here.**
+
+### 3.6 `DbHealthProbeActor`
+
+**No interaction with `DriverHostActor` whatsoever.** Spawned as a sibling
+(`ServiceCollectionExtensions.cs:243-246`), consumed only by `OpcUaPublishActor`
+(`OpcUaPublishActor.cs:105`, `:256`, `:528`, `:543-544`). `DriverHostActor` runs its own independent
+30s `retry-db` timer. If running-from-cache should later influence OPC UA ServiceLevel,
+`OpcUaPublishActor`'s DB-health seam is the precedent to mirror.
+
+---
+
+## 4. Cluster identity — **the one real design problem (see D-1)**
+
+**There is no `ClusterId` field on `DriverHostActor` and no accessor for it.** The actor knows only
+`_localNode` (`DriverHostActor.cs:61`, type `Commons.Types.NodeId`, aliased at `:28`).
+
+`ClusterId` is resolved **per-artifact**, by `DeploymentArtifact.ResolveClusterScope(blob, nodeId)`
+(`DeploymentArtifact.cs:485-516`) — it scans the artifact's `Nodes[]` array for a matching `NodeId`
+and reads that element's `ClusterId`. Types: `ClusterScope` (`:46`), `ClusterFilterMode` (`:28`).
+
+`IClusterRoleInfo` (`Commons/Interfaces/IClusterRoleInfo.cs:12`) exposes `LocalNode`, `LocalRoles`,
+`HasRole`, `MembersWithRole`, `RoleLeader`, `RoleLeaderChanged` — **no `ClusterId`**.
+`docker-dev/docker-compose.yml` has `Cluster__Hostname`/`Port`/`Roles`/`SeedNodes` — **no
+`Cluster__ClusterId`**. The `ClusterNode` entity
+(`Configuration/Entities/ClusterNode.cs:7,10`) carries both, but lives in central SQL — exactly
+what is unreachable at the seam that needs it.
+
+> **The bind:** the design keys the cache by `ClusterId` so a replicated pair shares one cache
+> entry. But `ClusterId` is only discoverable *from a blob you already have* or *from central SQL*.
+> At the §3.3 boot-failure seam we have neither.
+
+**Resolution adopted (D-1):** keep `cluster_id` as the `deployment_pointer` PK — pair sharing is the
+whole point of replication and must not be given up. Resolve it at each end as follows:
+
+- **Write path** (`ApplyAndAck`, §2.2): call `DeploymentArtifact.ResolveClusterScope(blob, nodeId)`
+  on the blob just applied. `ClusterFilterMode.ScopeTo` → use its `ClusterId`. `None` (single-cluster
+  artifact) or `Suppress` → fall back to the literal `"__single"` sentinel, matching the actor's own
+  existing degenerate-case convention at `:1827-1830`.
+- **Read path** (boot seam, §3.3): we cannot compute the key, so **do not key the read**. Select the
+  single `deployment_pointer` row, ordered by `applied_at_utc DESC`, `LIMIT 1`. A node belongs to
+  exactly one cluster and its pair peer replicates the same cluster's row, so the table holds one
+  row in every real topology. If more than one row is present (a node was re-homed between
+  clusters), take the newest and **log a warning naming both cluster ids** — an operator-visible
+  signal rather than a silent wrong-config boot.
+- Optional escape hatch: honour a `Cluster:ClusterId` config key when present. **Deferred** — not
+  needed for the rig, and adding a required key contradicts the design's "storage ships
+  unconditionally" posture.
+
+This preserves the replication payoff (design §3.3) and the schema exactly as specified, and
+confines the change to the read query's `WHERE` clause.
+
+---
+
+## 5. `DriverHostActor` construction — the `Props` expression tree
+
+### 5.1 The factory — `DriverHostActor.cs:326-346`
+
+`static Props(...)` takes **16 parameters**, 14 optional, and forwards them **fully positionally**
+into `Akka.Actor.Props.Create(() => new DriverHostActor(...))` at **`:343-346`**. Ctor at
+`:375-412`, assignments `:393-408`.
+
+> ⚠ Six `IActorRef?` parameters and three interface-typed parameters mean many mis-bindings are
+> **type-compatible and therefore compile clean**. Callers use *named* arguments (safe); the
+> internal forwarding at `:343-346` is *positional* (unsafe).
+>
+> **Rule for Task 7: append the new parameter LAST in the `Props` signature, LAST in the ctor
+> signature, and LAST in the positional forwarding list at `:346`. Change nothing else.**
+
+### 5.2 Construction sites — **27 total** (1 production + 26 test)
+
+**Production (1):** `Runtime/ServiceCollectionExtensions.cs:332`, inside `WithOtOpcUaRuntimeActors`
+(`:193`); registered as `DriverHostActorKey` (`:344`). All args named except the first three. Does
+not pass `virtualTagHostOverride`, `scriptedAlarmHostOverride`, or `driverMemberCountProvider`.
+
+`WithOtOpcUaRuntimeActors` callers: `Host/Program.cs:331`,
+`Host.IntegrationTests/TwoNodeClusterHarness.cs:398`, `ServiceCollectionExtensionsTests.cs:42,144`.
+
+**Tests (26)**, all in `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/`:
+
+| File | Lines |
+|---|---|
+| `Drivers/DriverHostActorTests.cs` | 28, 60, 81, 118, 143, 189 |
+| `Drivers/DriverHostActorReconcileTests.cs` | 37, 59, 90, 119, 156, 193 |
+| `Drivers/DriverHostActorDiscoveryTests.cs` | 385, 671, 748, 1023 |
+| `Drivers/DriverHostActorWriteRoutingTests.cs` | 148, 208 |
+| `Drivers/DriverHostActorPrimaryGateTests.cs` | 228, 247 |
+| `Drivers/DriverHostActorVirtualTagTests.cs` | 44, 76 |
+| `Drivers/DriverHostActorNativeAlarmTests.cs` | 350 |
+| `Drivers/DriverHostActorNativeAlarmAckRoutingTests.cs` | 126 |
+| `Drivers/DriverHostActorProbeResultDropTests.cs` | 66 |
+| `Drivers/DriverHostActorHistoryWriterTests.cs` | 59 |
+| `Drivers/DiscoveryInjectionEndToEndTests.cs` | 219 |
+| `VirtualTags/DependencyMuxActorTests.cs` | 139 |
+
+No site calls `Props.Create<DriverHostActor>()` or `new DriverHostActor(...)` directly — the only
+`new DriverHostActor` in the tree is inside the factory at `:343`. Because the new parameter is
+optional and appended last, **no test site should need editing.**
+
+### 5.3 Status object for the running-from-cache signal
+
+`NodeDiagnosticsSnapshot` (`Commons/Interfaces/NodeDiagnosticsSnapshot.cs:17-22`) — built in
+`HandleGetDiagnostics` (`DriverHostActor.cs:1381-1398`), a positional record of
+`(NodeId, RevisionHash? CurrentRevision, IReadOnlyList<DriverInstanceDiagnostics> Drivers, DateTime AsOfUtc)`.
+`GetDiagnostics` is handled in **all three states** (`Steady:570`, `Applying:601`, `Stale:1349`), so
+a flag here is observable regardless of state. **This is the natural home for `RunningFromCache`**
+(append last to the record). Consumer: `IFleetDiagnosticsClient` → AdminUI.
+
+`IDriverHealthPublisher` (`:73`, defaulted `:402`) is **not** a fit — the host never calls it; it
+only passes it down to `DriverInstanceActor` children (`:1845`, `:1859`). Per-driver, not per-node.
+
+### 5.4 Async / side-effect IO convention
+
+The actor is **overwhelmingly synchronous-blocking on DB IO** — all five `CreateDbContext()` sites
+use `using var db = ...` with synchronous LINQ on the actor thread. No `async`/`await` in the file,
+no `Task.Run`.
+
+The single async pattern is `ContinueWith(..., TaskScheduler.Default).PipeTo(replyTo)` in
+`HandleRouteNodeWrite` (`:1188-1200`), with an explicit `Sender` capture before the continuation.
+
+**Recommendation for Task 7:** a **synchronous** cache write at the §2.2 point is *consistent with
+the file's existing style* and is correct there — the actor is mid-apply, holds `Applying`, and
+every other DB call on that path already blocks. A fire-and-forget `PipeTo(Self)` would need a
+handler registered in `Steady`, `Applying` **and** `Stale`, or it dead-letters after the
+`Become(Steady)` at `:1456`. **Prefer synchronous + try/catch.** (This deviates from the plan's
+"fire-and-forget"; see D-4.)
+
+`IWithTimers` is already implemented (`:50`, `:282`); `"retry-db"` key in use at `:1378`/`:2009`.
+
+---
+
+## 6. Host composition root
+
+### 6.1 Role flag — `Program.cs:47-50`
+
+```csharp
+var roles = RoleParser.Parse(Environment.GetEnvironmentVariable("OTOPCUA_ROLES"));
+var hasAdmin = roles.Contains("admin");
+var hasDriver = roles.Contains("driver");
+```
+Allowed roles: `admin`, `driver`, `dev` (`Cluster/RoleParser.cs:5-8`); unknown throws (`:25-27`).
+
+### 6.2 `hasDriver` branch — `Program.cs:131-310`
+
+Anchor for the new call: **`Program.cs:173-175`** `builder.Services.AddAlarmHistorian(...)`, inside
+the config-gated cluster at `:154-195`. Must land **before** `AddAkka` at `:314`.
+
+### 6.3 Pipeline — `Program.cs:368-404`
+
+`:383` `MapStaticAssets` · `:385-396` `if (hasAdmin) { ... }` · **`:398` `MapOtOpcUaHealth()`** ·
+`:399` `MapOtOpcUaMetrics()` · `:404` `RunAsync()`.
+
+⚠ **There is no `hasDriver` block in the pipeline half** — only `hasAdmin` is branched on. A
+driver-gated `app.Map*` is a new construct; it goes between `:396` and `:398`.
+
+`Program.cs:406-410` re-exports `public partial class Program`.
+
+### 6.4 Kestrel / URLs — **CONFIRMED: no Kestrel configuration exists**
+
+- `Program.cs:57` `builder.WebHost.UseStaticWebAssets();` — the **only** `builder.WebHost` call.
+- **Zero** `ConfigureKestrel` / `UseUrls` / `ListenAnyIP` / `UseKestrel` anywhere in `src/`.
+- Binding is exclusively via `ASPNETCORE_URLS`: `docker-compose.yml:173` (central-1) and `:240`
+  (central-2) set `http://+:9000`; `launchSettings.json:7` sets `http://localhost:9000`.
+
+Two traps for Task 5:
+
+1. ⚠ **`scripts/install/Install-Services.ps1:186`** (`$hostEnv += "ASPNETCORE_URLS=..."`) is inside
+   `if ($hasAdmin)` at `:185`. **A driver-only Windows-service node gets no `ASPNETCORE_URLS` at
+   all** and binds the ASP.NET default. The plan's parse-URLs-then-rebind snippet would re-bind
+   `http://+:9000` there — a port that node never listened on. The `?? "http://+:9000"` fallback in
+   the plan's snippet is therefore **wrong for driver-only nodes**; see D-6.
+2. ⚠ **`Host.IntegrationTests/TwoNodeClusterHarness.cs:331`** already calls
+   `builder.WebHost.UseKestrel(o => o.Listen(IPAddress.Parse(LoopbackHost), 0));`. Adding a
+   production `ConfigureKestrel` gives two competing explicit configurations in integration tests —
+   last-one-wins. The `syncPort > 0` gate keeps this dormant by default (harness sets no sync port),
+   but any harness that *does* set one will fight the port-0 binding.
+
+**Health route for smoke tests:** `/healthz` — liveness, runs no checks, always 200 while the
+process is up, `AllowAnonymous`. (`Health/HealthEndpoints.cs:48` → `MapZbHealth()`; routes from the
+`ZB.MOM.WW.Health` package.) Also `/health/ready`, `/health/active`, and `/metrics` (`Program.cs:399`).
+
+### 6.5 `SecretsRegistration` template — `Host/Configuration/SecretsRegistration.cs`
+
+Shape for `LocalDbRegistration` to mirror: `public static class` in
+`ZB.MOM.WW.OtOpcUa.Host.Configuration`; `public const string` section/key path constants; a
+`public static bool IsXEnabled(IConfiguration)` predicate using
+`GetValue(key, defaultValue: false)` (**default-deny**) that `Program.cs` can also call;
+`AddOtOpcUaX(this IServiceCollection, IConfiguration)` returning `IServiceCollection`, both args
+`ArgumentNullException.ThrowIfNull`-guarded; XML docs with `<remarks><para>` explaining *why* the
+gate is default-deny.
+
+Call-site precedent: `Program.cs:363` `AddOtOpcUaSecrets(...)`; predicate reused inside the `AddAkka`
+lambda at `Program.cs:323`.
+
+### 6.6 Telemetry meter allowlist — **EXISTS. Task 10 step 3 is REQUIRED, not conditional.**
+
+`Host/Observability/ObservabilityExtensions.cs:25-38`:
+```csharp
+return services.AddZbTelemetry(o =>
+{
+    o.ServiceName = "otopcua";
+    o.Meters = [OtOpcUaTelemetry.MeterName];          // <-- ObservabilityExtensions.cs:30
+    o.ActivitySources = [OtOpcUaTelemetry.ActivitySourceName];
+```
+
+`o.Meters` is a **strict allowlist** — `ZbTelemetryExtensions.cs:51-54` iterates it calling
+`metrics.AddMeter(name)`; **no wildcard support**. Anything unlisted is silently not exported.
+Single element today: `OtOpcUaTelemetry.MeterName` = `"ZB.MOM.WW.OtOpcUa"`
+(`Commons/Observability/OtOpcUaTelemetry.cs:19`).
+
+**Action:** add `LocalDbMetrics.MeterName` at `ObservabilityExtensions.cs:30`. This is exactly the
+omission the ScadaBridge live gate caught.
+
+### 6.7 Health registration — `Host/Health/HealthEndpoints.cs:19-41`
+
+`AddOtOpcUaHealth()` **takes no parameters** and is called **unconditionally** at `Program.cs:365`,
+outside both role blocks. Three `.AddTypeActivatedCheck<>` calls (`configdb`, `akka`,
+`admin-leader`); **no registration-time role gating exists** — `ActiveNodeHealthCheck` does runtime
+role scoping instead (returns `Healthy` when the node lacks the role).
+
+⚠ **There are zero `IHealthCheck` implementations in this repo** — all three come from the
+`ZB.MOM.WW.Health.*` packages (`Directory.Packages.props:124-126`, v0.1.0). A LocalDb check would be
+the **first health-check class in the tree**; `Host/Health/` holds only `HealthEndpoints.cs` today.
+
+**Consequence for Task 10:** driver-gating the new check requires changing
+`AddOtOpcUaHealth()`'s signature (add `bool hasDriver` or `IConfiguration`) and its `Program.cs:365`
+call site. **Preferred alternative:** follow the `ActiveNodeHealthCheck` precedent — register
+unconditionally and have the check return `Healthy` when LocalDb is not registered. That keeps the
+signature and matches the "default-OFF must not degrade a plain node" requirement for free. See D-7.
+
+### 6.8 Config files
+
+| File | Top-level sections |
+|---|---|
+| `appsettings.json` | `Serilog`, `Security`, `Secrets`, `ServerHistorian`, `ContinuousHistorization`, `AlarmHistorian`, `Deployment` |
+| `appsettings.driver.json` / `.admin.json` / `.admin-driver.json` / `.Development.json` | `Serilog`, `Security` only |
+
+Overlay selection: `Program.cs:67` sorts roles ordinally and joins with `-` → `admin,driver`
+becomes `appsettings.admin-driver.json`, added `optional: true` at `:70`; env vars (`:77`) and CLI
+args (`:78`) are re-appended after, so deployment overrides outrank the overlay.
+
+**`LocalDb` key search: CONFIRMED ABSENT.** Case-insensitive `localdb` across all `*.json`,
+`*.yml`, `*.ps1`, `*.csproj`, `*.props` matches only the two Phase-1/Phase-2 planning artifacts. No
+conflicting keys in any overlay or in `Install-Services.ps1`.
+
+**Placement convention:** each feature section opens with a `"_comment"` string then `"Enabled":
+false`; per-field caveats use sibling `"_<Field>Comment"` keys (`appsettings.json:34`, `:45`). Place
+`LocalDb` between `AlarmHistorian` (ends `:61`) and `Deployment` (`:62`).
+
+---
+
+## 7. LiteDB LocalCache — dormant, safe to delete
+
+### 7.1 Files (6) under `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/`
+
+`ILocalConfigCache.cs`, `GenerationSnapshot.cs`, `StaleConfigFlag.cs`, `LiteDbConfigCache.cs`
+(also declares `LocalConfigCacheCorruptException` at `:128`), `GenerationSealedCache.cs`,
+`ResilientConfigReader.cs`. Single namespace `...Configuration.LocalCache`. The only two
+`using LiteDB;` statements in `src/` are `LiteDbConfigCache.cs:1` and `GenerationSealedCache.cs:1`.
+
+### 7.2 Reference classification — **no live-code references. NO STOP.**
+
+- **(c) live code elsewhere: ZERO.** Also confirms full dormancy: `SealedBootstrap` (referenced by
+  `docs/v2/v2-release-readiness.md:46`) has **zero hits** in `src/` and `tests/` — it no longer exists.
+- Two benign near-misses:
+  - `Configuration/Services/ILdapGroupRoleMappingService.cs:14` — the string `ResilientConfigReader`
+    inside an **XML doc comment**. Prose only. Needs a comment edit or the doc goes stale.
+  - `Configuration/ZB.MOM.WW.OtOpcUa.Configuration.csproj:29` — `<PackageReference Include="LiteDB"/>`,
+    the only one in the repo.
+- Dozens of `LocalConfigCacheCorruptException` hits under `src/**/bin/**/*.xml` are **generated doc
+  build artifacts** (`GenerateDocumentationFile=true`), not source.
+
+### 7.3 ⚠ Test-file drift — the plan's delete list is incomplete
+
+Plan `…phase1.md:484-485` names only `GenerationSealedCacheTests.cs` and
+`ResilientConfigReaderTests.cs`. Actual test files in
+`tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/`:
+
+| File | Status |
+|---|---|
+| `GenerationSealedCacheTests.cs` | in plan ✅ |
+| `ResilientConfigReaderTests.cs` | in plan ✅ — note `StaleConfigFlagTests` is co-located here at `:323` |
+| **`LiteDbConfigCacheTests.cs`** | **MISSING from the plan** — will fail to compile if the source is deleted (`:125` calls `new LiteDB.LiteDatabase(...)` directly) |
+
+See D-3.
+
+### 7.3b Outcome (Task 9, executed 2026-07-20)
+
+Deleted: the 6 `LocalCache/` sources and **all three** test files (including
+`LiteDbConfigCacheTests.cs`, which the plan omitted — D-3). `LiteDB` removed from both
+`Configuration.csproj` and `Directory.Packages.props`. The stale XML-doc reference at
+`ILdapGroupRoleMappingService.cs:14` was rewritten to state the present truth (no fallback exists;
+reviving it means an admin-side cache on LocalDb, not restoring the old pipeline) rather than
+deleted, per the DoD's "explanatory prose may remain". Configuration tests: 92/92 green.
+
+**Follow-up found while executing:** `Polly.Core` in `Configuration.csproj` is now orphaned too —
+`ResilientConfigReader` was its only real consumer in that project (remaining `Polly` mentions are
+comments). Left in place deliberately: removing it is outside Task 9's scope and could break a
+consumer relying on the transitive flow. Worth a separate cleanup.
+
+### 7.4 Package removal
+
+- `PackageReference`: `Configuration.csproj:29` — only consumer.
+- `PackageVersion`: `Directory.Packages.props:35` — `LiteDB` `5.0.21`.
+
+Both safe to delete; no transitive consumer.
+
+---
+
+## 8. Test projects, rig, packages
+
+### 8.1 Test project layout
+
+| Question | Answer |
+|---|---|
+| `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/` | **DOES NOT EXIST** — there is no Host *unit*-test project at all |
+| Closest Host project | `Host.IntegrationTests` — despite the name it carries plenty of pure unit tests (`LdapOptionsValidatorTests`, `ServerHistorianOptionsValidatorTests`, `ResilienceInvokerFactoryRegistrationTests`, …) |
+| `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/` | **EXISTS** |
+| DriverHostActor tests | `Runtime.Tests/Drivers/` — 11 `DriverHostActor*Tests.cs` files. **xunit v2** (`xunit` 2.9.3), references **`Akka.TestKit.Xunit2`** ✅ |
+| `Host.IntegrationTests` xunit version | **xunit.v3** 1.1.0; SDK `Microsoft.NET.Sdk.Web` |
+
+`Directory.Packages.props` documents that AdminUI.Tests / ControlPlane.Tests / **Runtime.Tests** are
+the three xunit-v2 holdouts, held there by `Akka.TestKit.Xunit2` (verified 2026-07-13: no xunit.v3
+Akka TestKit exists). This matches the design's "actor tests live in an xunit2 TestKit project".
+
+### 8.2 ⚠ `WebApplicationFactory<Program>` is **deliberately not used** — Task 11 must be rewritten
+
+The only mention of the type in the entire `tests/` tree is a comment explaining why it is avoided:
+
+`Host.IntegrationTests/TwoNodeClusterHarness.cs:48`
+```
+/// <para>Why not <c>WebApplicationFactory&lt;Program&gt;</c>? Program.cs reads <c>OTOPCUA_ROLES</c> ...
+```
+
+The actual pattern builds the host directly — `TwoNodeClusterHarness.cs:329`:
+```csharp
+var builder = WebApplication.CreateBuilder(new WebApplicationOptions { Args = [] });
+```
+`Microsoft.AspNetCore.Mvc.Testing` is still referenced (csproj `:15`) but only for transitive
+test-host plumbing. **Copy `TwoNodeClusterHarness`, not `WebApplicationFactory`.** See D-5.
+
+`Host.IntegrationTests` also has its own `docker-compose.yml` (sql `14331`, ldap `3894`) gated by
+`DockerFixtureAvailability.cs`.
+
+### 8.3 docker-dev rig topology — plan's site-a/site-b assumption is **correct**
+
+Single Akka mesh (`otopcua`), seeded by `central-1`. Tenant separation is by `ServerCluster.ClusterId`
+rows (MAIN / SITE-A / SITE-B) inside **one shared ConfigDb** — not separate meshes.
+
+| Service | Roles | Cluster |
+|---|---|---|
+| `central-1`, `central-2` | `admin,driver` | MAIN |
+| `site-a-1`, `site-a-2` | `driver` | SITE-A |
+| `site-b-1`, `site-b-2` | `driver` | SITE-B |
+| `sql`, `migrator`, `cluster-seed`, `traefik` | — | infra |
+
+**There is no admin-only service** — the "admin" nodes are fused `admin,driver`. All six host nodes
+share the `&otopcua-host` YAML anchor.
+
+⚠ **NO HOST NODE HAS A WRITABLE VOLUME.** The only `volumes:` in the file are `sql` →
+`otopcua-mssql-data`, plus two read-only binds (`./seed:ro`, `./traefik-dynamic.yml:ro`). All six
+nodes write to the container's ephemeral layer, destroyed on recreate. This is already an accepted
+pattern (per-container `Secrets:SqlitePath=otopcua-secrets.db`, replicated over Akka rather than
+persisted). **Task 13 must add six named volumes** — there is nothing to piggyback on. See D-8.
+
+**Env style: confirmed double-underscore `Section__Key`**, with array indices and dictionary keys as
+a third segment (`Cluster__SeedNodes__0`, `Security__Ldap__GroupToRole__ReadOnly`). Exceptions that
+are plain SCREAMING_SNAKE (read directly, not config-bound): `OTOPCUA_ROLES`,
+`ZB_SECRETS_MASTER_KEY`, `GALAXY_MXGW_API_KEY`, `OTOPCUA_CONFIG_CONNECTION`, `ASPNETCORE_URLS`.
+
+**Ports.** Host-published: `14330`→sql, `4840`-`4845`→the six nodes' OPC UA, `9200`→traefik web,
+`8089`→traefik dashboard. Container-internal on every node: `4053` (Akka remoting), `9000`
+(Kestrel), `4840` (OPC UA). Also reserved on this machine: `14331`/`3894` (Host.IntegrationTests
+compose), `80`/`8080` (sibling scadabridge/scadalink stack), `10.100.0.35:3893` (shared GLAuth).
+
+**`9001` (the plan's choice) is free** container-internal and need not be published. ✅
+
+### 8.4 Package versions — `Directory.Packages.props`
+
+| Package | Version | Line |
+|---|---|---|
+| `Grpc.Core.Api` | 2.76.0 | 32 |
+| `Grpc.Net.Client` | 2.76.0 | 33 |
+| **`Grpc.AspNetCore`** | **NOT PRESENT** | — |
+| `Google.Protobuf` | 3.34.1 | 31 |
+| `Microsoft.Data.Sqlite` | 10.0.7 | ~55 |
+| `SQLitePCLRaw.bundle_e_sqlite3` | 2.1.12 | ~108 |
+| `LiteDB` | 5.0.21 | 35 |
+
+Both Grpc floors and the Protobuf floor are already satisfied — **no bumps needed**, only the new
+`Grpc.AspNetCore` entry (use 2.76.0 to match the train; watch for NU1605/NU1608 if it wants a
+Protobuf newer than the pinned 3.34.1).
+
+⚠ **`SQLitePCLRaw` is a surgical direct pin, not a transitive one.**
+`CentralPackageTransitivePinningEnabled` is deliberately **off** (it breaks the Roslyn version
+split). The 2.1.12 pin is promoted by a **direct `PackageReference` in `Core.AlarmHistorian`**
+(`Core.AlarmHistorian.csproj:15` + `:21`) — today the only Sqlite consumer in `src/`.
+
+> **Any new project that pulls `Microsoft.Data.Sqlite` MUST also take a direct
+> `<PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3"/>`,** or it silently resolves the
+> vulnerable 2.1.11 native bundle. This applies to **Host** and **Runtime** (both gain `ZB.MOM.WW.LocalDb`,
+> which depends on `Microsoft.Data.Sqlite`) and to every test project that builds a real `ILocalDb`.
+> Copy the `Core.AlarmHistorian.csproj:15,21` pair exactly. The plan's Task 1 mentions
+> `SQLitePCLRaw.lib.e_sqlite3` "if the audit flags it" — the correct package is
+> **`bundle_e_sqlite3`**, and it is **required, not conditional**.
+
+---
+
+## 9. Deviations from the plan
+
+Recorded per the tasks.json `deviation` convention. **D-1, D-3, D-5, D-6 change the work
+materially.**
+
+| # | Plan says | Reality | Resolution |
+|---|---|---|---|
+| **D-1** | Cache keyed by `ClusterId`; "cluster identity" is a recon item with a STOP condition | `ClusterId` is stable, but **not resolvable at the boot seam** — it lives inside the artifact or in the unreachable central DB (§4) | Keep `cluster_id` as PK (pair sharing is the point). **Write:** resolve via `DeploymentArtifact.ResolveClusterScope`, `"__single"` sentinel for the degenerate case. **Read:** unkeyed `ORDER BY applied_at_utc DESC LIMIT 1`, warn if >1 row. No STOP. |
+| **D-2** | Cache write "fire-and-forget (`PipeTo`-style or guarded `Task.Run`)" | The actor has **no async DB IO at all**; a `PipeTo(Self)` needs handlers in all three states or it dead-letters after `Become(Steady)` (§5.4) | Synchronous write in its own try/catch at `:1439`. Matches the file's style; failure still cannot fail the apply. |
+| **D-3** | Task 9 deletes 2 test files | **3** test files reference the subsystem — `LiteDbConfigCacheTests.cs` is missing from the list (§7.3) | Delete all three. Also fix the stale XML-doc mention at `ILdapGroupRoleMappingService.cs:14`. |
+| **D-4** | Cache the artifact after a successful apply | `ReconcileDrivers`/`PushDesiredSubscriptions` **swallow** DB errors, so a "successful" apply can have loaded an **empty blob** and started zero drivers (§2.3) | Gate the write on a non-empty blob; surface the blob from `ReconcileDrivers` instead of a third read. **Load-bearing — without it the cache can be poisoned with an empty config.** |
+| **D-5** | Task 11 uses `WebApplicationFactory<Program>` | Deliberately **not used** in this repo; `TwoNodeClusterHarness.cs:48` documents why (Program.cs reads `OTOPCUA_ROLES` from the environment) (§8.2) | Rewrite Task 11 against the `TwoNodeClusterHarness` direct-host-build pattern. |
+| **D-6** | Task 5 re-binds `ASPNETCORE_URLS ?? "http://+:9000"` | `Install-Services.ps1:185-186` sets `ASPNETCORE_URLS` **only when `hasAdmin`** — the fallback would bind a driver-only node to a port it never served (§6.4) | Re-bind only when a URL is actually configured; when absent, add **only** the sync listener. Also note `TwoNodeClusterHarness.cs:331` already calls `UseKestrel(Listen(…, 0))`. |
+| **D-7** | Task 10 driver-gates the health check | `AddOtOpcUaHealth()` takes no args and is called unconditionally at `Program.cs:365`; no registration-time gating exists anywhere (§6.7) | Register unconditionally; return `Healthy` when LocalDb is absent — the `ActiveNodeHealthCheck` precedent. Avoids a signature change and satisfies "default-OFF must not degrade a plain node". |
+| **D-8** | Task 13 "existing data mount" | **No host node has any writable volume** (§8.3) | Add six named volumes. Note: `9001` is free container-internal. Meter allowlist (§6.6) is **required**, not conditional. |
+
+### Follow-ups logged, not fixed here
+
+1. **`TryRecoverFromStale` marks `Applied` without reconciling drivers** (§3.5) — pre-existing,
+   documented in-code at `:1692-1695`. Out of Phase 1 scope.
+2. **Duplicate blob read per apply** (`:1472` and `:1543`, §1.1) — D-4's fix removes one of them as a
+   side effect.
+3. **`Install-Services.ps1` gives driver-only nodes no `ASPNETCORE_URLS`** (§6.4) — pre-existing;
+   D-6 works around it rather than fixing it.

docs/plans/2026-07-21-issue-485-live-gate.md

@@ -0,0 +1,76 @@
+# Issue #485 — live gate (docker-dev rig)
+
+> Verifies the two fixes for #485 — `OpcUaPublishActor` (master `c7f5b9cf`) and `DriverHostActor`
+> (master `5aad1e33`) — on the local `docker-dev` rig: 6 nodes, shared SQL, real Modbus fixture on
+> `10.100.0.35:5020`.
+>
+> **Safety rules followed**: all DB inspection + mutation via `sqlcmd` inside the SQL container; the
+> rig's Modbus fixture is read-only for this gate (no writes to the PLC sim).
+
+## Result
+
+**Both fixes verified, all three guards observed firing in production, with a live RED control that
+was not staged.** The gate also confirmed a **pre-existing** defect the fixes do not address — the node
+records `Applied` for a deployment it skipped (see "Finding" below).
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 1 | RED control — the bug in its natural habitat, unforced | ✅ | site-a-1 hit #485 during the *previous* gate's SQL flapping and had been serving an **empty address space for 34 minutes**: `failed to load artifact … rebuild becomes no-op` followed immediately by `applied plan (kind=PureRemove, added=0, removed=17, rebuild=True)`. Its 17 `ns=2` nodes were gone; a browse returned only the root `OtOpcUa` folder. Nothing was staged to produce this — it was already true on the pre-fix image. |
+| 2 | Rebuilt on the fixed image, the node recovers | ✅ | After `docker compose build` + `--force-recreate`, site-a-1 restored to `raw subtree materialised (containers=16, tags=1)` — 18 `ns=2` nodes. |
+| 3 | Address-space guard fires; nodes survive | ✅ | site-a-1, empty artifact: `OpcUaPublish: no usable artifact for deployment f4b9dd63… — KEEPING the currently served address space rather than tearing it down`. **No `PureRemove`.** Browse before vs after: 18 vs 18, `diff` **identical**. |
+| 4 | Reconcile guard fires; drivers survive | ✅ | `artifact for f4b9dd63… read back empty; skipping reconcile and keeping the 1 running driver(s)`, and the apply still reported `children=1`. Repeated on central-2 with a bigger blast radius: `keeping the 5 running driver(s)`, `children=5`. |
+| 5 | SubscribeBulk guard fires; **live values keep flowing** | ✅ | central-2, `artifact for 8c36a673… read back empty; skipping SubscribeBulk and keeping the live subscriptions`. The decisive proof is the value, not the log: `ns=2;s=pymodbus/plc/HR200X` read **56278 @ 06:59:51** before the induction, then **56336 @ 07:00:21** and **56354 @ 07:00:30** after it (induction at 07:00:07). The subscription was still delivering fresh Modbus data across the event. |
+| 6 | Positive control — a readable artifact still applies | ✅ | Immediately after the guards fired, a genuine deploy applied normally on both nodes: site-a-1 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 1 references`; central-2 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 5 references across 4 driver(s)`. The guards suppress the teardown, **not** deployment. |
+| 7 | Control — only the node that saw empty bytes took the guard path | ✅ | Of the six nodes, exactly **one** logged `read back empty` (the paused one); the other five read the real artifact before it was blanked and applied it normally. The induction window is what produced the condition, not a global change. |
+
+## How the failure was induced deterministically
+
+The original incident was a race (a transient ConfigDb blip landing between the apply and the artifact
+read). Racing it again would be unreliable, so the gate drives the **empty-bytes** branch instead — which
+is byte-for-byte what the code sees when the row cannot be read, and is the branch the driver-side guards
+key on (their *throw* paths already returned early before this work):
+
+1. `docker pause` the target node, freezing it mid-cluster.
+2. `POST /api/deployments` — the deployment seals and dispatches while the node cannot process it.
+3. `UPDATE Deployment SET ArtifactBlob = 0x WHERE DeploymentId = <new>`.
+4. `docker unpause` — the node now processes the dispatch and reads an empty artifact.
+
+The pause measured **<1s** on both runs, well inside `acceptable-heartbeat-pause = 10s` and SBR's
+`stable-after = 15s`, so the node was never flagged unreachable and no failover was triggered.
+
+## Finding — the node records `Applied` for a deployment it skipped (pre-existing, NOT fixed here)
+
+Both guards leave the node running its previous configuration, which is the intent. But `ApplyAndAck`
+advances `_currentRevision = revision` and writes `NodeDeploymentState = Applied` **before** it knows
+whether anything was applied — that assignment sits immediately after `ReconcileDrivers`, which returns
+`null` on both the (pre-existing) throw path and the new empty path:
+
+```
+[07:00:07 WRN] DriverHost central-2:4053: artifact … read back empty; skipping reconcile …
+[07:00:07 INF] DriverHost central-2:4053: applied deployment 8c36a673… (rev 03b33904…, children=5)
+```
+
+`NodeDeploymentState` for that deployment reads `Status = Applied` for central-2. The node now claims a
+revision whose configuration it never applied — and because `HandleDispatchFromSteady` short-circuits on
+a revision match (`dispatch matches current rev; immediate ACK`), **re-dispatching that same revision is
+a no-op**, so the node cannot self-heal. Only a later deployment with a *different* revision recovers it
+(check 6 confirms that path works).
+
+This is pre-existing — the throw path has always done it — and is strictly a reporting/convergence bug,
+not a data-plane one: the node keeps serving its last good configuration throughout. The honest fix is to
+ACK `Failed` and leave `_currentRevision` alone when the artifact could not be read, so the normal deploy
+retry heals it. Tracked separately rather than folded in here, because it changes deploy ACK semantics
+fleet-wide.
+
+## Addendum — issue #486 fix, live-gated (2026-07-21)
+
+The finding above is now fixed (`DriverHostActor.ApplyAndAck` fails the apply when `ReconcileDrivers`
+returns null) and gated on the same rig with the same induction.
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 8 | Regression — a normal deploy still ACKs Applied fleet-wide | ✅ | The change alters ACK semantics, so this is the risk that matters. A genuine deploy recorded `Status = Applied` on **all six** nodes. |
+| 9 | An unreadable artifact now ACKs Failed, with a reason | ✅ | central-2 recorded `Status = Failed` + `the deployment artifact could not be read (ConfigDb unreachable, or the row is missing/empty); nothing was applied`. The five nodes that read the real artifact recorded Applied — so the fleet view now distinguishes them. Pre-fix, all six claimed Applied. |
+| 10 | The revision is NOT advanced | ✅ | `apply of a855dd9e… FAILED — … Staying on revision 5bc0d9f3…` — the **previous** revision, not the dispatched `583122b7…`. This is what stops `HandleDispatchFromSteady` from short-circuiting a retry of that revision. |
+| 11 | The node keeps serving throughout (#485 still holds) | ✅ | The #485 guard fired first (`keeping the 5 running driver(s)`), and `ns=2;s=pymodbus/plc/HR200X` read **59603 @ 07:27:28**, live and current, after the Failed apply. Failing the apply reports the truth; it does not tear anything down. |
+| 12 | The node recovers on the next real deploy | ✅ | The following deployment applied on central-2 (`children=5`) and recorded Applied on all six nodes. |

docs/plans/2026-07-21-localdb-followups-live-gate.md

@@ -0,0 +1,86 @@
+# LocalDb Phase 1 follow-ups — live gate (docker-dev rig)
+
+> Closes the two limitations left open by `2026-07-20-localdb-phase1-live-gate.md` (checks 4 and 8).
+> Rig: local `docker-dev/docker-compose.yml`, 6 nodes rebuilt from `fix/localdb-phase1-followups`.
+> site-a pair replicates; central + site-b stay default-OFF.
+>
+> **Safety rules followed** (same as the Phase 1 gate): all DB inspection via `docker cp` of the
+> `db`/`-wal`/`-shm` triplet, querying the copy — never host `sqlite3` on the live WAL file.
+
+## Result
+
+**Both follow-ups closed, and the gate found a third defect** — one that was *unreachable* until the
+first fix landed. Library `ZB.MOM.WW.LocalDb` went `0.1.1` → `0.1.2` → `0.1.3` in the course of this
+gate; both consumers (OtOpcUa, ScadaBridge) are pinned to `0.1.3`.
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 8 | Oplog stops growing on a default-OFF node | ✅ PASS | site-b-1 restarted twice with an unchanged cached artifact: `oplog_depth` **7 → 7 → 7**, `deployment_pointer.applied_at_utc` frozen at `05:34:58` across both — the re-cache was skipped outright. Repeated on the `0.1.3` build after a real deploy: **13 → 13**. Pre-fix this grew ~2 rows per restart forever, because a default-OFF node has no peer to ack and prune them. |
+| 8b | …but a genuinely new deployment still writes (positive control) | ✅ PASS | A real config change (renamed tag `SITEA-tag`) produced rev `822bdf34`, then `002c16b3`. site-b-1's oplog **7 → 10 → 13** and its pointer advanced each time. The skip suppresses *identical* re-writes only; it is not a dead cache-write path. |
+| 4 | A wiped node is back-filled by its peer, no new deploy, central down | ✅ PASS (was ⚠️ PARTIAL) | Precondition asserted first — site-a-1's oplog **0**, `needs_snapshot 0`: the exact state that used to make back-fill impossible. Then SQL stopped, site-a-2's LocalDb **volume destroyed**, container recreated. a-1 logged `Snapshot sent (as_of_seq 0, 4 rows)` — a line that could not appear on `0.1.1`. a-2 came back with pointer `SITE-A / 475df87a @ 03:09:12` (a-1's **original** timestamp, not a re-derived one), chunks 2, and a `__localdb_row_version` dump **byte-identical to a-1's**, carrying origin node ids `6812ca9c` and `935659a5` — the rebuilt node's own id appears nowhere, so the rows were replicated, not recreated. |
+| 4b | …and the back-filled node then boots from cache | ✅ PASS | With SQL still down, site-a-2 logged `RUNNING FROM CACHE — … booted deployment 475df87a… cached at 2026-07-21T03:09:12`, `raw subtree materialised (containers=16, tags=1)`. OPC UA browse of both nodes: **17 ns=2 nodes each, diff-identical** — the rebuilt node serves exactly what its healthy peer does, from a configuration it never applied itself and could not have fetched. |
+| 4c | The rebuilt node's OWN writes reach its peer | ✅ PASS **after a third fix** (see below) | On `0.1.2` this failed: a-1 held `last_applied_remote = 10` while the rebuilt a-2's oplog was seqs **1,2,3** and its `last_acked` stayed **0** — never accepted. On `0.1.3`, a-1 logs `Replication peer identity changed (ebb10854 -> c85d8b1d): the peer was rebuilt … Resetting the inbound watermark`, and after the next deploy a-1 shows `last_applied_remote = 3` / a-2 shows `last_acked = 3` with a drained oplog and identical row_versions. |
+
+## The third defect (the reason this gate was worth running)
+
+`0.1.2` made back-fill work, which for the first time let a rebuilt node be observed *writing*. It
+turned out its writes went nowhere.
+
+`last_applied_remote_seq` is a watermark in the **peer's** seq space, and a rebuilt peer is a new
+database whose oplog numbers from 1. Two independent bugs stacked, and fixing either alone leaves the
+data stuck:
+
+1. The healthy node kept advertising the **old** peer's watermark. The *sending* side seeds its pump
+   directly from that number (`_sentThruSeq`), so the rebuilt node skipped its entire oplog.
+   → the watermark (and the observed peer clock) is now reset when the peer's node id changes.
+   `last_acked_seq` is deliberately **not** reset: it describes our own oplog's pruned horizon and is
+   exactly what `0.1.2` keys the snapshot decision off — clearing it would switch back-fill back off.
+2. A peer claiming to have applied more of our stream than we have ever produced can only be
+   remembering a previous incarnation of *us*. → that claim is now rejected in favour of pumping from
+   the start; LWW makes the re-send harmless. This mirrors the clamp `RecordPeerAckAsync` already
+   applies to acks.
+
+Both were reproduced offline as a RED test before fixing
+(`SnapshotResyncTests.RebuiltPeer_OwnWritesReachTheHealthyNode_DespiteTheOldPeersWatermark`).
+
+**Why offline tests could not have found it first:** the wiped-peer scenario had no test because the
+scenario was believed impossible to recover from — it was a *documented limitation*, not a bug. The
+gate is what turned the limitation into a reproducible case, and the second defect only existed
+behind the first.
+
+## Unrelated finding (NOT a follow-up of this work — pre-existing; `lmxopcua` issue #485, since FIXED)
+
+While SQL was being stopped and started to drive check 4, site-b-1 took a deploy whose artifact load
+hit a transient ConfigDb error, and **emptied its served address space**:
+
+```
+[06:11:20 ERR] An error occurred using the connection to database 'OtOpcUa' on server 'sql,1433'.
+[06:11:20 WRN] OpcUaPublish: failed to load artifact for deployment 43d54397…; rebuild becomes no-op
+[06:11:20 INF] AddressSpaceApplier: applied plan (kind=PureRemove, added=0, removed=16, rebuild=True)
+```
+
+The log says "rebuild becomes no-op" and the applier then removed all 16 nodes — those two disagree.
+The cache layer behaved correctly and refused to store the bad artifact ("not caching deployment … its
+artifact was empty or could not be loaded, so it is not a usable last-known-good configuration"), and
+the node recovered fully on restart (`containers=16`). But a transient ConfigDb blip emptying the
+served address space instead of holding last-known-good is the same failure *class* as the Phase 1
+gate's check-3 defect. Untouched by this work — site-b-1 has replication off entirely — and induced by
+this gate's own SQL flapping.
+
+**Fixed** (issue #485): `OpcUaPublishActor.HandleRebuild` now treats an artifact it could not obtain as
+*no answer* rather than *an empty configuration*, and abandons the rebuild with the served address space
+(and `_lastApplied`) intact. Nothing legitimate produces a zero-length blob — an operator who really
+deletes everything still deploys a JSON document with empty arrays — so "no bytes" can only mean the read
+failed. The log excerpt above is now impossible: the `PureRemove` never runs, and the loader's own
+"the rebuild is abandoned" line is finally true. Covered by
+`OpcUaPublishActorRebuildTests.Rebuild_keeps_the_last_known_good_address_space_when_the_artifact_load_fails`,
+with a positive control proving a *readable* artifact that genuinely drops an equipment still tears its
+subtree down.
+
+The **driver-side half** of the same mistake was fixed alongside it: `DriverHostActor.ReconcileDrivers`
+and `PushDesiredSubscriptionsFromArtifact` read the artifact the same way, so empty bytes meant zero
+driver specs — `DriverSpawnPlanner` planned every running child for `StopChild` — and then an empty
+desired set dropped each surviving driver's live subscription handle. A node lost its entire field I/O
+and still ACKed Applied. Both now skip on empty. Covered by `DriverHostActorUnreadableArtifactTests`,
+whose absence assertion is calibrated by a control that observes the very same unsubscribe *arriving*
+inside the settle window (without it the assertion passed on a race).

docs/security.md

@@ -380,6 +380,55 @@ polling the node.
 
 ---
 
+## Config secrets (`${secret:}` delivery)
+
+OtOpcUa never commits secret values to `appsettings*.json`. Every real secret is
+supplied at deploy time — either as a plain environment variable, or as a
+`${secret:<name>}` token backed by the shared **`ZB.MOM.WW.Secrets`** encrypted store.
+
+A pre-host expander (`SecretReferenceExpander.ExpandConfigurationAsync`, wired in
+`OtOpcUa.Host/Program.cs`) walks the assembled configuration and rewrites every
+`${secret:<name>}` token into its resolved plaintext **before** the host is built —
+so every downstream binder/validator (`AddZbSerilog`, `AddOtOpcUaConfigDb`, the first
+`ValidateOnStart`) sees resolved values. The store's key-encryption key (KEK) comes from
+the `ZB_SECRETS_MASTER_KEY` environment variable (`Secrets:MasterKey:Source=Environment`);
+the encrypted SQLite store lives at `Secrets:SqlitePath`.
+
+The expander is **fail-closed and section-agnostic**: a `${secret:<name>}` token whose
+secret is absent throws `SecretNotFoundException` at startup, regardless of which feature
+owns the key or whether that feature is enabled. Keys prefixed with `_` (comment keys) are
+skipped, so a `${secret:...}` example inside a `_…Comment` value is never resolved.
+
+The five config secrets and their canonical secret names:
+
+| Config key | Owner | Secret name |
+|---|---|---|
+| `Security:Jwt:SigningKey` | `JwtOptions` | `otopcua/jwt/signing-key` |
+| `Security:Ldap:ServiceAccountPassword` | `LdapOptions` | `otopcua/ldap/service-account-password` |
+| `Security:DeployApiKey` | `DeployApiEndpoints` | `otopcua/deploy/api-key` |
+| `ConnectionStrings:ConfigDb` | `AddOtOpcUaConfigDb` | `otopcua/sql/configdb-connstr` |
+| `ServerHistorian:ApiKey` | `ServerHistorianOptions` | `otopcua/historian/api-key` |
+
+**Delivery.** By default these are delivered as plain environment variables (e.g.
+`ServerHistorian__ApiKey=histgw_…`), never committed. To deliver one from the encrypted
+store instead, seed it once with the `secret` CLI (from `ZB.MOM.WW.Secrets`), then supply
+the token in place of the literal — via env var or a deployment appsettings overlay:
+
+```
+secret set otopcua/historian/api-key <value>       # seed the encrypted store first
+ServerHistorian__ApiKey='${secret:otopcua/historian/api-key}'
+```
+
+Only switch a value to a `${secret:}` token **after** the secret is seeded — an unseeded
+token fails the boot fail-closed. Do not commit the KEK (`ZB_SECRETS_MASTER_KEY`) or any
+secret value.
+
+For the production posture needed so every clustered node (admin, driver, and any
+fused role) resolves the same secrets from the same store, see
+[`docs/operations/2026-07-16-secrets-clustered-master-key.md`](operations/2026-07-16-secrets-clustered-master-key.md).
+
+---
+
 ## Troubleshooting
 
 ### Certificate trust failure

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Interfaces/NodeDiagnosticsSnapshot.cs

@@ -14,8 +14,16 @@ public sealed record DriverInstanceDiagnostics(
 /// Per-node diagnostics returned by <c>IFleetDiagnosticsClient</c>. Populated by the node's
 /// local <c>DriverHostActor</c> via a request/response over Akka.
 /// </summary>
+/// <param name="RunningFromCache">
+///     True when the node booted its configuration from the node-local artifact cache because the
+///     central ConfigDb was unreachable. Such a node looks entirely healthy — full address space,
+///     live values — but its configuration is frozen and no deployment can reach it, so the state
+///     needs to be explicitly visible rather than inferred. Defaults to false, which is also what
+///     every node that booted normally reports.
+/// </param>
 public sealed record NodeDiagnosticsSnapshot(
     NodeId NodeId,
     RevisionHash? CurrentRevision,
     IReadOnlyList<DriverInstanceDiagnostics> Drivers,
-    DateTime AsOfUtc);
+    DateTime AsOfUtc,
+    bool RunningFromCache = false);

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/AlarmConditionSnapshot.cs

@@ -16,6 +16,12 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
 /// <param name="Shelving">The shelving mode (ShelvingState): unshelved, one-shot, or timed.</param>
 /// <param name="Severity">OPC UA severity on the 1..1000 scale (the SDK <c>SetSeverity</c> input).</param>
 /// <param name="Message">The human-readable condition message (LocalizedText payload).</param>
+/// <param name="Quality">
+///     Quality of the condition's source data (Part 9 <c>ConditionType.Quality</c>). Carried so a
+///     comms-lost native source can report a non-<c>Good</c> condition instead of the accidentally-Good
+///     default (issue #477). It is a pure annotation — it never alters Active/Acked/Retain. Defaults to
+///     <see cref="OpcUaQuality.Good"/> so scripted-alarm callers (which stay Good in v1) need not supply it.
+/// </param>
 public sealed record AlarmConditionSnapshot(
     bool Active,
     bool Acknowledged,
@@ -23,7 +29,8 @@ public sealed record AlarmConditionSnapshot(
     bool Enabled,
     AlarmShelvingKind Shelving,
     ushort Severity,
-    string Message);
+    string Message,
+    OpcUaQuality Quality = OpcUaQuality.Good);
 
 /// <summary>
 ///     Commons-local mirror of the Core <c>ShelvingKind</c> enum so this assembly carries no

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/DeferredAddressSpaceSink.cs

@@ -30,6 +30,9 @@ public sealed class DeferredAddressSpaceSink : IOpcUaAddressSpaceSink, ISurgical
     public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
         => _inner.WriteAlarmCondition(alarmNodeId, state, sourceTimestampUtc, realm);
 
+    public void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
+        => _inner.WriteAlarmQuality(alarmNodeId, quality, sourceTimestampUtc, realm);
+
     /// <inheritdoc />
     public void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, AddressSpaceRealm realm, bool isNative = false)
         => _inner.MaterialiseAlarmCondition(alarmNodeId, equipmentNodeId, displayName, alarmType, severity, realm, isNative);

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IOpcUaAddressSpaceSink.cs

@@ -33,6 +33,17 @@ public interface IOpcUaAddressSpaceSink
     /// the condition was materialised under).</param>
     void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm);
 
+    /// <summary>#477 — annotate a materialised condition's source-data quality OUT OF BAND from any alarm
+    /// transition (used by the driver-connectivity path: comms lost → <see cref="OpcUaQuality.Bad"/>,
+    /// restored → <see cref="OpcUaQuality.Good"/>). Sets ONLY the condition's Quality — never
+    /// Active/Acked/Severity/Retain (a comms-lost active alarm stays active) — and fires one Part 9 event
+    /// only on a quality-bucket change. A no-op for an unmaterialised / non-condition node.</summary>
+    /// <param name="alarmNodeId">The condition node id (RawPath for a native alarm).</param>
+    /// <param name="quality">The source-data quality to annotate.</param>
+    /// <param name="sourceTimestampUtc">The connectivity transition timestamp in UTC.</param>
+    /// <param name="realm">The namespace realm the condition was materialised under.</param>
+    void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm);
+
     /// <summary>
     ///     Materialise a real OPC UA Part 9 alarm-condition node under its equipment folder so clients
     ///     can browse it as a proper condition (with basic Active/Ack state). The node id equals the
@@ -165,6 +176,7 @@ public sealed class NullOpcUaAddressSpaceSink : IOpcUaAddressSpaceSink
 
     /// <inheritdoc />
     public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
+    public void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
 
     /// <inheritdoc />
     public void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, AddressSpaceRealm realm, bool isNative = false) { }

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/GenerationSealedCache.cs

@@ -1,197 +0,0 @@
-using LiteDB;
-
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Generation-sealed LiteDB cache per <c>docs/v2/plan.md</c> and Phase 6.1
-///     Stream D.1. Each published generation writes one <b>read-only</b> LiteDB file under
-///     <c>&lt;cache-root&gt;/&lt;clusterId&gt;/&lt;generationId&gt;.db</c>. A per-cluster
-///     <c>CURRENT</c> text file holds the currently-active generation id; it is updated
-///     atomically (temp file + <see cref="File.Replace(string, string, string?)"/>) only after
-///     the sealed file is fully written.
-/// </summary>
-/// <remarks>
-///     <para>Mixed-generation reads are impossible: any read opens the single file pointed to
-///     by <c>CURRENT</c>, which is a coherent snapshot. Corruption of the CURRENT file or the
-///     sealed file surfaces as <see cref="GenerationCacheUnavailableException"/> — the reader
-///     fails closed rather than silently falling back to an older generation. Recovery path
-///     is to re-fetch from the central DB (and the Phase 6.1 Stream C <c>UsingStaleConfig</c>
-///     flag goes true until that succeeds).</para>
-///
-///     <para>This cache is the read-path fallback when the central DB is unreachable. The
-///     write path (draft edits, publish) bypasses the cache and fails hard on DB outage per
-///     Stream D.2 — inconsistent writes are worse than a temporary inability to edit.</para>
-/// </remarks>
-public sealed class GenerationSealedCache
-{
-    private const string CollectionName = "generation";
-    private const string CurrentPointerFileName = "CURRENT";
-    private readonly string _cacheRoot;
-
-    // Private per-database BsonMapper with the entity pre-registered. LiteDB's default
-    // BsonMapper.Global is a process-wide singleton whose lazy per-type member registration is
-    // not thread-safe across concurrently-constructed LiteDatabase instances; a seal racing a
-    // read (or this cache racing LiteDbConfigCache) corrupts the global mapper, surfacing as
-    // "Member … not found on BsonMapper" or a bogus duplicate-_id insert.
-    private static BsonMapper BuildMapper()
-    {
-        var mapper = new BsonMapper();
-        mapper.Entity<GenerationSnapshot>();
-        return mapper;
-    }
-
-    /// <summary>Root directory for all clusters' sealed caches.</summary>
-    public string CacheRoot => _cacheRoot;
-
-    /// <summary>Initializes a new instance of the GenerationSealedCache class.</summary>
-    /// <param name="cacheRoot">The root directory for the cache.</param>
-    public GenerationSealedCache(string cacheRoot)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(cacheRoot);
-        _cacheRoot = cacheRoot;
-        Directory.CreateDirectory(_cacheRoot);
-    }
-
-    /// <summary>
-    ///     Seal a generation: write the snapshot to <c>&lt;cluster&gt;/&lt;generationId&gt;.db</c>,
-    ///     mark the file read-only, then atomically publish the <c>CURRENT</c> pointer. Existing
-    ///     sealed files for prior generations are preserved (prune separately).
-    /// </summary>
-    /// <param name="snapshot">The generation snapshot to seal.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task representing the asynchronous operation.</returns>
-    public async Task SealAsync(GenerationSnapshot snapshot, CancellationToken ct = default)
-    {
-        ArgumentNullException.ThrowIfNull(snapshot);
-        ct.ThrowIfCancellationRequested();
-
-        var clusterDir = Path.Combine(_cacheRoot, snapshot.ClusterId);
-        Directory.CreateDirectory(clusterDir);
-        var sealedPath = Path.Combine(clusterDir, $"{snapshot.GenerationId}.db");
-
-        if (File.Exists(sealedPath))
-        {
-            // Already sealed — idempotent. Treat as no-op + update pointer in case an earlier
-            // seal succeeded but the pointer update failed (crash recovery).
-            WritePointerAtomically(clusterDir, snapshot.GenerationId);
-            return;
-        }
-
-        var tmpPath = sealedPath + ".tmp";
-        try
-        {
-            using (var db = new LiteDatabase(new ConnectionString { Filename = tmpPath, Upgrade = false }, BuildMapper()))
-            {
-                var col = db.GetCollection<GenerationSnapshot>(CollectionName);
-                col.Insert(snapshot);
-            }
-
-            File.Move(tmpPath, sealedPath);
-            File.SetAttributes(sealedPath, File.GetAttributes(sealedPath) | FileAttributes.ReadOnly);
-            WritePointerAtomically(clusterDir, snapshot.GenerationId);
-        }
-        catch
-        {
-            try { if (File.Exists(tmpPath)) File.Delete(tmpPath); } catch { /* best-effort */ }
-            throw;
-        }
-
-        await Task.CompletedTask;
-    }
-
-    /// <summary>
-    ///     Read the current sealed snapshot for <paramref name="clusterId"/>. Throws
-    ///     <see cref="GenerationCacheUnavailableException"/> when the pointer is missing
-    ///     (first-boot-no-snapshot case) or when the sealed file is corrupt. Never silently
-    ///     falls back to a prior generation.
-    /// </summary>
-    /// <param name="clusterId">The cluster ID to read the snapshot for.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task representing the asynchronous operation containing the generation snapshot.</returns>
-    public Task<GenerationSnapshot> ReadCurrentAsync(string clusterId, CancellationToken ct = default)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
-        ct.ThrowIfCancellationRequested();
-
-        var clusterDir = Path.Combine(_cacheRoot, clusterId);
-        var pointerPath = Path.Combine(clusterDir, CurrentPointerFileName);
-        if (!File.Exists(pointerPath))
-            throw new GenerationCacheUnavailableException(
-                $"No sealed generation for cluster '{clusterId}' at '{clusterDir}'. First-boot case: the central DB must be reachable at least once before cache fallback is possible.");
-
-        long generationId;
-        try
-        {
-            var text = File.ReadAllText(pointerPath).Trim();
-            generationId = long.Parse(text, System.Globalization.CultureInfo.InvariantCulture);
-        }
-        catch (Exception ex)
-        {
-            throw new GenerationCacheUnavailableException(
-                $"CURRENT pointer at '{pointerPath}' is corrupt or unreadable.", ex);
-        }
-
-        var sealedPath = Path.Combine(clusterDir, $"{generationId}.db");
-        if (!File.Exists(sealedPath))
-            throw new GenerationCacheUnavailableException(
-                $"CURRENT points at generation {generationId} but '{sealedPath}' is missing — fails closed rather than serving an older generation.");
-
-        try
-        {
-            using var db = new LiteDatabase(new ConnectionString { Filename = sealedPath, ReadOnly = true }, BuildMapper());
-            var col = db.GetCollection<GenerationSnapshot>(CollectionName);
-            var snapshot = col.FindAll().FirstOrDefault()
-                ?? throw new GenerationCacheUnavailableException(
-                    $"Sealed file '{sealedPath}' contains no snapshot row — file is corrupt.");
-            return Task.FromResult(snapshot);
-        }
-        catch (GenerationCacheUnavailableException) { throw; }
-        catch (Exception ex) when (ex is LiteException or InvalidDataException or IOException
-                                      or NotSupportedException or FormatException)
-        {
-            throw new GenerationCacheUnavailableException(
-                $"Sealed file '{sealedPath}' is corrupt or unreadable — fails closed rather than falling back to an older generation.", ex);
-        }
-    }
-
-    /// <summary>Return the generation id the <c>CURRENT</c> pointer points at, or null if no pointer exists.</summary>
-    /// <param name="clusterId">The cluster ID to get the current generation ID for.</param>
-    /// <returns>The generation ID, or null if no pointer exists.</returns>
-    public long? TryGetCurrentGenerationId(string clusterId)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
-        var pointerPath = Path.Combine(_cacheRoot, clusterId, CurrentPointerFileName);
-        if (!File.Exists(pointerPath)) return null;
-        try
-        {
-            return long.Parse(File.ReadAllText(pointerPath).Trim(), System.Globalization.CultureInfo.InvariantCulture);
-        }
-        catch
-        {
-            return null;
-        }
-    }
-
-    private static void WritePointerAtomically(string clusterDir, long generationId)
-    {
-        var pointerPath = Path.Combine(clusterDir, CurrentPointerFileName);
-        var tmpPath = pointerPath + ".tmp";
-        File.WriteAllText(tmpPath, generationId.ToString(System.Globalization.CultureInfo.InvariantCulture));
-        if (File.Exists(pointerPath))
-            File.Replace(tmpPath, pointerPath, destinationBackupFileName: null);
-        else
-            File.Move(tmpPath, pointerPath);
-    }
-}
-
-/// <summary>Sealed cache is unreachable — caller must fail closed.</summary>
-public sealed class GenerationCacheUnavailableException : Exception
-{
-    /// <summary>Initializes a new instance of the GenerationCacheUnavailableException class.</summary>
-    /// <param name="message">The error message.</param>
-    public GenerationCacheUnavailableException(string message) : base(message) { }
-    /// <summary>Initializes a new instance of the GenerationCacheUnavailableException class with an inner exception.</summary>
-    /// <param name="message">The error message.</param>
-    /// <param name="inner">The inner exception.</param>
-    public GenerationCacheUnavailableException(string message, Exception inner) : base(message, inner) { }
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/GenerationSnapshot.cs

@@ -1,20 +0,0 @@
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     A self-contained snapshot of one generation — enough to rebuild the address space on a node
-///     that has lost DB connectivity. The payload is the JSON-serialized <c>sp_GetGenerationContent</c>
-///     result; the local cache doesn't inspect the shape, it just round-trips bytes.
-/// </summary>
-public sealed class GenerationSnapshot
-{
-    /// <summary>Gets or sets the auto-generated LiteDB ID.</summary>
-    public int Id { get; set; } // LiteDB auto-ID
-    /// <summary>Gets or sets the cluster identifier.</summary>
-    public required string ClusterId { get; set; }
-    /// <summary>Gets or sets the generation identifier.</summary>
-    public required long GenerationId { get; set; }
-    /// <summary>Gets or sets the time this snapshot was cached.</summary>
-    public required DateTime CachedAt { get; set; }
-    /// <summary>Gets or sets the JSON-serialized payload content.</summary>
-    public required string PayloadJson { get; set; }
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/ILocalConfigCache.cs

@@ -1,32 +0,0 @@
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Per-node local cache of the most-recently-applied generation(s). Used to bootstrap the
-///     address space when the central DB is unreachable (degraded-but-running).
-/// </summary>
-/// <remarks>
-///     <para><b>Concurrency contract:</b> implementations must serialize writes — specifically,
-///     <see cref="PutAsync"/> for the same <c>(ClusterId, GenerationId)</c> from concurrent
-///     callers must not produce duplicate rows. Reads may run concurrently with reads and writes.
-///     The <see cref="LiteDbConfigCache"/> implementation enforces this via an instance-level
-///     <see cref="SemaphoreSlim"/> around the find-then-insert/update window.</para>
-/// </remarks>
-public interface ILocalConfigCache
-{
-    /// <summary>Retrieves the most recent generation snapshot for the specified cluster.</summary>
-    /// <param name="clusterId">The cluster identifier.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>The most recent generation snapshot, or null if none exists.</returns>
-    Task<GenerationSnapshot?> GetMostRecentAsync(string clusterId, CancellationToken ct = default);
-    /// <summary>Stores a generation snapshot in the local cache.</summary>
-    /// <param name="snapshot">The generation snapshot to store.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task that represents the asynchronous operation.</returns>
-    Task PutAsync(GenerationSnapshot snapshot, CancellationToken ct = default);
-    /// <summary>Removes old generations, keeping only the most recent N.</summary>
-    /// <param name="clusterId">The cluster identifier.</param>
-    /// <param name="keepLatest">The number of latest generations to keep.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task that represents the asynchronous operation.</returns>
-    Task PruneOldGenerationsAsync(string clusterId, int keepLatest = 10, CancellationToken ct = default);
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/LiteDbConfigCache.cs

@@ -1,129 +0,0 @@
-using LiteDB;
-
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     LiteDB-backed <see cref="ILocalConfigCache"/>. One file per node (default
-///     <c>config_cache.db</c>), one collection per snapshot. Corruption surfaces as
-///     <see cref="LocalConfigCacheCorruptException"/> on construction or read — callers should
-///     delete and re-fetch from the central DB.
-/// </summary>
-public sealed class LiteDbConfigCache : ILocalConfigCache, IDisposable
-{
-    private const string CollectionName = "generations";
-
-    // LiteDB's default BsonMapper.Global is a process-wide singleton whose per-type member
-    // registration is lazy and NOT thread-safe across concurrently-constructed LiteDatabase
-    // instances. When several caches (this one + GenerationSealedCache) initialise in parallel
-    // the global mapper races, surfacing as "Member ClusterId not found on BsonMapper" or a
-    // bogus "duplicate key _id = 0" (the int auto-id mapping was lost so Insert writes a literal
-    // 0 twice). Give each database a private, pre-registered mapper so member
-    // resolution happens once, single-threaded, at construction and never touches the global.
-    private static BsonMapper BuildMapper()
-    {
-        var mapper = new BsonMapper();
-        mapper.Entity<GenerationSnapshot>();
-        return mapper;
-    }
-
-    private readonly LiteDatabase _db;
-    private readonly ILiteCollection<GenerationSnapshot> _col;
-    // PutAsync is a find-then-insert/update; without serialization, two concurrent puts for the
-    // same (ClusterId, GenerationId) can both observe `existing is null` and both Insert,
-    // producing duplicate rows. Serialize writes through this semaphore so
-    // the read-modify-write block is atomic for a given instance. LiteDB itself only locks the
-    // page-level write, not the find-then-insert window.
-    private readonly SemaphoreSlim _writeGate = new(initialCount: 1, maxCount: 1);
-
-    /// <summary>Initializes a new instance of the <see cref="LiteDbConfigCache"/> class.</summary>
-    /// <param name="dbPath">Path to the LiteDB database file.</param>
-    public LiteDbConfigCache(string dbPath)
-    {
-        // LiteDB can be tolerant of header-only corruption at construction time (it may overwrite
-        // the header and "recover"), so we force a write + read probe to fail fast on real corruption.
-        try
-        {
-            _db = new LiteDatabase(new ConnectionString { Filename = dbPath, Upgrade = true }, BuildMapper());
-            _col = _db.GetCollection<GenerationSnapshot>(CollectionName);
-            _col.EnsureIndex(s => s.ClusterId);
-            _col.EnsureIndex(s => s.GenerationId);
-            _ = _col.Count();
-        }
-        catch (Exception ex) when (ex is LiteException or InvalidDataException or IOException
-                                      or NotSupportedException or UnauthorizedAccessException
-                                      or ArgumentOutOfRangeException or FormatException)
-        {
-            _db?.Dispose();
-            throw new LocalConfigCacheCorruptException(
-                $"LiteDB cache at '{dbPath}' is corrupt or unreadable — delete the file and refetch from the central DB.",
-                ex);
-        }
-    }
-
-    /// <inheritdoc />
-    public Task<GenerationSnapshot?> GetMostRecentAsync(string clusterId, CancellationToken ct = default)
-    {
-        ct.ThrowIfCancellationRequested();
-        var snapshot = _col
-            .Find(s => s.ClusterId == clusterId)
-            .OrderByDescending(s => s.GenerationId)
-            .FirstOrDefault();
-        return Task.FromResult<GenerationSnapshot?>(snapshot);
-    }
-
-    /// <inheritdoc />
-    public async Task PutAsync(GenerationSnapshot snapshot, CancellationToken ct = default)
-    {
-        ct.ThrowIfCancellationRequested();
-        // Serialize the find-then-insert/update so concurrent callers do not observe a stale
-        // `existing is null` and both Insert. LiteDB's per-call lock is not enough — the
-        // read and the write are independent calls.
-        await _writeGate.WaitAsync(ct).ConfigureAwait(false);
-        try
-        {
-            // upsert by (ClusterId, GenerationId) — replace in place if already cached
-            var existing = _col
-                .Find(s => s.ClusterId == snapshot.ClusterId && s.GenerationId == snapshot.GenerationId)
-                .FirstOrDefault();
-
-            if (existing is null)
-                _col.Insert(snapshot);
-            else
-            {
-                snapshot.Id = existing.Id;
-                _col.Update(snapshot);
-            }
-        }
-        finally
-        {
-            _writeGate.Release();
-        }
-    }
-
-    /// <inheritdoc />
-    public Task PruneOldGenerationsAsync(string clusterId, int keepLatest = 10, CancellationToken ct = default)
-    {
-        ct.ThrowIfCancellationRequested();
-        var doomed = _col
-            .Find(s => s.ClusterId == clusterId)
-            .OrderByDescending(s => s.GenerationId)
-            .Skip(keepLatest)
-            .Select(s => s.Id)
-            .ToList();
-
-        foreach (var id in doomed)
-            _col.Delete(id);
-
-        return Task.CompletedTask;
-    }
-
-    /// <summary>Releases all resources used by the cache.</summary>
-    public void Dispose()
-    {
-        _writeGate.Dispose();
-        _db.Dispose();
-    }
-}
-
-public sealed class LocalConfigCacheCorruptException(string message, Exception inner)
-    : Exception(message, inner);

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/ResilientConfigReader.cs

@@ -1,140 +0,0 @@
-using System.Text.RegularExpressions;
-using Microsoft.Extensions.Logging;
-using Polly;
-using Polly.Retry;
-using Polly.Timeout;
-
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Wraps a central-DB fetch function with Phase 6.1 Stream D.2 resilience:
-///     <b>timeout 2 s → retry 3× jittered → fallback to sealed cache</b>. Maintains the
-///     <see cref="StaleConfigFlag"/> — fresh on central-DB success, stale on cache fallback.
-/// </summary>
-/// <remarks>
-///     <para>Read-path only per plan. The write path (draft save, publish) bypasses this
-///     wrapper entirely and fails hard on DB outage so inconsistent writes never land.</para>
-///
-///     <para>Fallback is triggered by <b>any exception</b> the fetch raises (central-DB
-///     unreachable, SqlException, timeout). If the sealed cache also fails (no pointer,
-///     corrupt file, etc.), <see cref="GenerationCacheUnavailableException"/> surfaces — caller
-///     must fail the current request (InitializeAsync for a driver, etc.).</para>
-/// </remarks>
-public sealed class ResilientConfigReader
-{
-    private readonly GenerationSealedCache _cache;
-    private readonly StaleConfigFlag _staleFlag;
-    private readonly ResiliencePipeline _pipeline;
-    private readonly ILogger<ResilientConfigReader> _logger;
-
-    /// <summary>Initializes a resilient config reader with the given cache and options.</summary>
-    /// <param name="cache">The sealed cache for fallback.</param>
-    /// <param name="staleFlag">The stale config flag to manage.</param>
-    /// <param name="logger">The logger instance.</param>
-    /// <param name="timeout">The timeout for central fetch (default 2s).</param>
-    /// <param name="retryCount">The number of retries (default 3).</param>
-    public ResilientConfigReader(
-        GenerationSealedCache cache,
-        StaleConfigFlag staleFlag,
-        ILogger<ResilientConfigReader> logger,
-        TimeSpan? timeout = null,
-        int retryCount = 3)
-    {
-        _cache = cache;
-        _staleFlag = staleFlag;
-        _logger = logger;
-        var builder = new ResiliencePipelineBuilder()
-            .AddTimeout(new TimeoutStrategyOptions { Timeout = timeout ?? TimeSpan.FromSeconds(2) });
-
-        if (retryCount > 0)
-        {
-            builder.AddRetry(new RetryStrategyOptions
-            {
-                MaxRetryAttempts = retryCount,
-                BackoffType = DelayBackoffType.Exponential,
-                UseJitter = true,
-                Delay = TimeSpan.FromMilliseconds(100),
-                MaxDelay = TimeSpan.FromSeconds(1),
-                // Handle ALL exceptions including OperationCanceledException. A SQL command-level
-                // timeout surfaces as TaskCanceledException (derives from OperationCanceledException)
-                // when the caller's token is NOT cancelled, and must be retried just like any other
-                // transient error. Polly itself checks the cancellation token between retries and
-                // stops with OperationCanceledException on genuine caller cancellation regardless of
-                // this predicate.
-                ShouldHandle = new PredicateBuilder().Handle<Exception>(),
-            });
-        }
-
-        _pipeline = builder.Build();
-    }
-
-    /// <summary>
-    ///     Redacts connection-string fragments (Password, User Id, Pwd, etc.)
-    ///     that a caller's exception message could carry. Conservative regex pass — anything
-    ///     matching <c>Key=Value</c> with a known credential key gets its value replaced.
-    /// </summary>
-    private static readonly Regex SecretsRegex = new(
-        @"(?ix)\b(Password|Pwd|User\s*Id|Uid|AccessToken|Authorization|Api[-_]?Key)\s*=\s*[^;,)\s]*",
-        RegexOptions.Compiled);
-
-    /// <summary>Redacts sensitive credential information from a message.</summary>
-    /// <param name="message">The message to scrub.</param>
-    /// <returns>The message with redacted credentials.</returns>
-    internal static string ScrubSecrets(string? message)
-    {
-        if (string.IsNullOrEmpty(message)) return message ?? string.Empty;
-        // Replace the entire matched fragment (key + value) with a redaction marker so the
-        // key name itself doesn't leak — log scrapers grep for "Password=" too.
-        return SecretsRegex.Replace(message, "[redacted credential]");
-    }
-
-    /// <summary>
-    ///     Executes a central fetch through the resilience pipeline. On full failure
-    ///     (post-retry), reads the sealed cache and extracts the requested shape.
-    /// </summary>
-    /// <typeparam name="T">The type of configuration to read.</typeparam>
-    /// <param name="clusterId">The cluster ID to fetch for.</param>
-    /// <param name="centralFetch">Function to fetch from central DB.</param>
-    /// <param name="fromSnapshot">Function to extract the config from a snapshot.</param>
-    /// <param name="cancellationToken">Cancellation token.</param>
-    /// <returns>The configuration of type T.</returns>
-    public async ValueTask<T> ReadAsync<T>(
-        string clusterId,
-        Func<CancellationToken, ValueTask<T>> centralFetch,
-        Func<GenerationSnapshot, T> fromSnapshot,
-        CancellationToken cancellationToken)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
-        ArgumentNullException.ThrowIfNull(centralFetch);
-        ArgumentNullException.ThrowIfNull(fromSnapshot);
-
-        try
-        {
-            var result = await _pipeline.ExecuteAsync(centralFetch, cancellationToken).ConfigureAwait(false);
-            _staleFlag.MarkFresh();
-            return result;
-        }
-        // Catch all exceptions that are NOT genuine caller cancellations. A SQL command-level
-        // timeout surfaces as TaskCanceledException (derives from OperationCanceledException)
-        // but the caller's token is NOT cancelled — we must fall back to the sealed cache for
-        // that case, not propagate. Only rethrow if the caller actually requested cancellation.
-        catch (Exception ex) when (ex is not OperationCanceledException || !cancellationToken.IsCancellationRequested)
-        {
-            // Do NOT pass the raw exception object — it carries the stack
-            // and inner-exception chain, and SqlException/wrapping delegates can surface
-            // connection-string fragments (Password=…, User Id=…) embedded in messages.
-            // Log only the exception type and a scrubbed message so secrets stay out of logs.
-            _logger.LogWarning(
-                "Central-DB read failed after retries ({ExceptionType}: {SanitizedMessage}); falling back to sealed cache for cluster {ClusterId}",
-                ex.GetType().Name,
-                ScrubSecrets(ex.Message),
-                clusterId);
-            // GenerationCacheUnavailableException surfaces intentionally — fails the caller's
-            // operation. StaleConfigFlag stays unchanged; the flag only flips when we actually
-            // served a cache snapshot.
-            var snapshot = await _cache.ReadCurrentAsync(clusterId, cancellationToken).ConfigureAwait(false);
-            _staleFlag.MarkStale();
-            return fromSnapshot(snapshot);
-        }
-    }
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/StaleConfigFlag.cs

@@ -1,20 +0,0 @@
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Thread-safe <c>UsingStaleConfig</c> signal per Phase 6.1 Stream D.3. Flips true whenever
-///     a read falls back to a sealed cache snapshot; flips false on the next successful central-DB
-///     round-trip. Surfaced on <c>/healthz</c> body and on the Admin <c>/hosts</c> page.
-/// </summary>
-public sealed class StaleConfigFlag
-{
-    private int _stale;
-
-    /// <summary>True when the last config read was served from the sealed cache, not the central DB.</summary>
-    public bool IsStale => Volatile.Read(ref _stale) != 0;
-
-    /// <summary>Mark the current config as stale (a read fell back to the cache).</summary>
-    public void MarkStale() => Volatile.Write(ref _stale, 1);
-
-    /// <summary>Mark the current config as fresh (a central-DB read succeeded).</summary>
-    public void MarkFresh() => Volatile.Write(ref _stale, 0);
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Services/ILdapGroupRoleMappingService.cs

@@ -10,10 +10,18 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Services;
 ///     Phase 6.2 compliance check on control/data-plane separation).
 /// </summary>
 /// <remarks>
-///     Per Phase 6.2 Stream A.2 this service is expected to run behind the Phase 6.1
-///     <c>ResilientConfigReader</c> pipeline (timeout → retry → fallback-to-cache) so a
-///     transient DB outage during sign-in falls back to the sealed snapshot rather than
-///     denying every login.
+///     <para>
+///         This service has no local-cache fallback: a DB outage during sign-in denies logins.
+///     </para>
+///     <para>
+///         The Phase 6.1 <c>ResilientConfigReader</c> pipeline (timeout → retry →
+///         fallback-to-sealed-snapshot) this was once expected to run behind was never wired to
+///         anything and was deleted along with the rest of the dormant LiteDB local cache, which
+///         <c>ZB.MOM.WW.LocalDb</c> supersedes. LocalDb caches the deployed-configuration artifact
+///         for driver-role nodes; it does not currently cover admin-plane reads like this one.
+///         Reviving the fallback means adding an admin-side cache on LocalDb, not restoring the
+///         old pipeline.
+///     </para>
 /// </remarks>
 public interface ILdapGroupRoleMappingService
 {

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/ZB.MOM.WW.OtOpcUa.Configuration.csproj

@@ -26,7 +26,6 @@
         <PackageReference Include="Microsoft.AspNetCore.DataProtection.EntityFrameworkCore"/>
         <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions"/>
         <PackageReference Include="Microsoft.Extensions.Logging.Abstractions"/>
-        <PackageReference Include="LiteDB"/>
         <PackageReference Include="Polly.Core"/>
     </ItemGroup>
 

src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/Part9StateMachine.cs

@@ -377,4 +377,9 @@ public enum EmissionKind
     Enabled,
     Disabled,
     CommentAdded,
+    /// <summary>#478 — the worst-of-input source quality changed bucket (Good/Uncertain/Bad) with no
+    /// accompanying Part 9 state transition. Delivered out of band via the dedicated
+    /// <c>WriteAlarmQuality</c> node path, never through the <c>IAlarmSource</c> fan-out (it is not a
+    /// state change and must not materialize or historize a condition).</summary>
+    QualityChanged,
 }

src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmEngine.cs

@@ -62,6 +62,17 @@ public sealed class ScriptedAlarmEngine : IDisposable
     private readonly ConcurrentDictionary<string, AlarmScratch> _scratchByAlarmId =
         new(StringComparer.Ordinal);
 
+    /// <summary>
+    ///     #478 — last emitted worst-of-input quality bucket per alarm (0 = Good, 1 = Uncertain,
+    ///     2 = Bad), computed each evaluation over the refilled read cache. A change in this bucket
+    ///     with no accompanying Part 9 state transition drives a standalone
+    ///     <see cref="EmissionKind.QualityChanged"/> emission (a Bad input freezes the condition — no
+    ///     transition — so quality can't ride one, exactly like a comms-lost native driver). Only ever
+    ///     mutated under <c>_evalGate</c>; cleared alongside <see cref="_alarms"/> on load/dispose.
+    /// </summary>
+    private readonly ConcurrentDictionary<string, int> _lastQualityBucketByAlarmId =
+        new(StringComparer.Ordinal);
+
     /// <summary>
     ///     Compile cache for every alarm predicate. Routes <see cref="LoadAsync"/>'s
     ///     <see cref="ScriptEvaluator{TContext, TResult}.Compile"/> calls through the
@@ -203,6 +214,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
             // have changed (different Inputs, different Logger), so any reuse would be
             // unsafe.
             _scratchByAlarmId.Clear();
+            _lastQualityBucketByAlarmId.Clear();
             // Dispose every compiled-predicate ALC from the prior generation BEFORE we
             // recompile this one. Skipping this is what made the earlier fix a
             // no-op in production.
@@ -412,7 +424,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
             // OnEvent dispatch until after Release() so a slow subscriber or a
             // subscriber that re-enters the engine doesn't block / deadlock.
             if (result.Emission != EmissionKind.None)
-                pending = BuildEmission(state, result.State, result.Emission);
+                pending = BuildEmission(state, result.State, result.Emission, LastWorstStatus(alarmId));
             else if (result.NoOpReason is { } reason)
             {
                 // The Part9StateMachine remarks promise a diagnostic log line for
@@ -513,13 +525,27 @@ public sealed class ScriptedAlarmEngine : IDisposable
             _ => new AlarmScratch(state.Inputs, state.Logger, _clock));
         RefillReadCache(scratch.ReadCache, state.Inputs);
 
+        // #478 — worst OPC UA quality across the alarm's inputs, computed BEFORE the readiness
+        // short-circuit so an outright-Bad input is still observed. A bucket change with no state
+        // transition is delivered out of band as a QualityChanged emission (see below).
+        var worstStatus = WorstInputStatus(scratch.ReadCache);
+        var qualityBucketChanged = TrackQualityBucket(state.Definition.AlarmId, worstStatus);
+
         // Cold-start guard — skip the predicate when any referenced upstream tag has no
         // cached value yet (the upstream subscription hasn't delivered its first push).
         // Without this, predicates that cast `(double)ctx.GetTag(path).Value` throw NRE on
         // every tick until the cache fills, spamming the log with identical stack traces.
         // Bad quality is treated the same: the input isn't available at the predicate's
         // expected type, so the only defensible move is to hold the prior condition state.
-        if (!AreInputsReady(scratch.ReadCache)) return seed;
+        if (!AreInputsReady(scratch.ReadCache))
+        {
+            // The condition is frozen (can't trust its state), but its source quality just changed
+            // bucket — annotate it out of band so a comms-lost / Bad-input scripted condition reports
+            // Bad, mirroring the native OT path.
+            if (qualityBucketChanged)
+                pendingEmissions.Add(BuildQualityEmission(state, seed, worstStatus));
+            return seed;
+        }
 
         var context = scratch.Context;
 
@@ -544,10 +570,19 @@ public sealed class ScriptedAlarmEngine : IDisposable
         }
 
         var result = Part9StateMachine.ApplyPredicate(seed, predicateTrue, nowUtc);
-        if (result.Emission != EmissionKind.None)
+        var transition = result.Emission != EmissionKind.None
+            ? BuildEmission(state, result.State, result.Emission, worstStatus)
+            : null;
+        if (transition is not null)
         {
-            var evt = BuildEmission(state, result.State, result.Emission);
-            if (evt is not null) pendingEmissions.Add(evt);
+            // A real transition carries the current worst quality so the projected full-snapshot
+            // write doesn't clobber quality back to Good (e.g. a transition while an input is Uncertain).
+            pendingEmissions.Add(transition);
+        }
+        else if (qualityBucketChanged)
+        {
+            // No transition (or a Suppressed one) but the quality bucket moved — annotate out of band.
+            pendingEmissions.Add(BuildQualityEmission(state, result.State, worstStatus));
         }
         return result.State;
     }
@@ -599,7 +634,8 @@ public sealed class ScriptedAlarmEngine : IDisposable
     ///     done by <see cref="FireEvent(ScriptedAlarmEvent)"/> AFTER the gate is
     ///     released.
     /// </summary>
-    private ScriptedAlarmEvent? BuildEmission(AlarmState state, AlarmConditionState condition, EmissionKind kind)
+    private ScriptedAlarmEvent? BuildEmission(
+        AlarmState state, AlarmConditionState condition, EmissionKind kind, uint worstInputStatus)
     {
         // Suppressed kind means shelving ate the emission — we don't fire for subscribers
         // but the state record still advanced so startup recovery reflects reality.
@@ -629,9 +665,89 @@ public sealed class ScriptedAlarmEngine : IDisposable
             // Carry the per-alarm durable-historization opt-out through to subscribers. The historian
             // adapter honors it to suppress ONLY the durable sink write; the live alerts fan-out is
             // unaffected (it is not gated on this flag).
-            HistorizeToAveva: state.Definition.HistorizeToAveva);
+            HistorizeToAveva: state.Definition.HistorizeToAveva,
+            // #478 — the worst input quality at evaluation time rides the transition so the projected
+            // full snapshot keeps quality consistent (no clobber-to-Good).
+            WorstInputStatusCode: worstInputStatus);
     }
 
+    /// <summary>
+    ///     #478 — build a standalone <see cref="EmissionKind.QualityChanged"/> event carrying the new
+    ///     worst-of-input quality. Emitted when the quality bucket moved but no Part 9 transition fired
+    ///     (a Bad input freezes the condition; a Suppressed/None transition also leaves state unchanged).
+    ///     The host routes it to the dedicated <c>WriteAlarmQuality</c> node path (annotate quality only,
+    ///     no <c>/alerts</c> row, no historian write); the <see cref="IAlarmSource"/> fan-out skips it.
+    /// </summary>
+    private ScriptedAlarmEvent BuildQualityEmission(
+        AlarmState state, AlarmConditionState condition, uint worstInputStatus)
+        => new(
+            AlarmId: state.Definition.AlarmId,
+            EquipmentPath: state.Definition.EquipmentPath,
+            AlarmName: state.Definition.AlarmName,
+            Kind: state.Definition.Kind,
+            Severity: state.Definition.Severity,
+            Message: MessageTemplate.Resolve(state.Definition.MessageTemplate, TryLookup),
+            Condition: condition,
+            Emission: EmissionKind.QualityChanged,
+            TimestampUtc: _clock(),
+            HistorizeToAveva: state.Definition.HistorizeToAveva,
+            WorstInputStatusCode: worstInputStatus);
+
+    /// <summary>Worst OPC UA StatusCode across a refilled read cache — the entry with the highest severity
+    /// bits (top 2). An input with no value yet (null snapshot/value — the cold-start placeholder, or a
+    /// not-yet-published upstream) is NOT a quality signal: it means "no data", which the
+    /// <see cref="AreInputsReady"/> guard already handles by holding the condition. Counting it as Bad here
+    /// would flash every scripted condition Bad at deploy until the first push and would flood the quality
+    /// path with load-time annotations, so unread inputs are skipped (contribute Good). Empty / all-unread
+    /// cache ⇒ Good (0).</summary>
+    private static uint WorstInputStatus(IReadOnlyDictionary<string, DataValueSnapshot> cache)
+    {
+        uint worst = 0u;
+        var worstSeverity = 0u;
+        foreach (var kv in cache)
+        {
+            if (kv.Value is null || kv.Value.Value is null) continue; // no data yet — not a quality signal
+            var status = kv.Value.StatusCode;
+            var severity = status >> 30;
+            if (severity > worstSeverity)
+            {
+                worstSeverity = severity;
+                worst = status;
+            }
+        }
+        return worst;
+    }
+
+    /// <summary>Update the tracked worst-quality bucket for an alarm; return true iff the 3-state bucket
+    /// (0 = Good, 1 = Uncertain, 2 = Bad) changed from the last observed value. Only called under
+    /// <c>_evalGate</c>.</summary>
+    private bool TrackQualityBucket(string alarmId, uint worstStatus)
+    {
+        var bucket = QualityBucket(worstStatus);
+        var prior = _lastQualityBucketByAlarmId.TryGetValue(alarmId, out var b) ? b : 0; // default Good
+        _lastQualityBucketByAlarmId[alarmId] = bucket;
+        return bucket != prior;
+    }
+
+    /// <summary>Collapse an OPC UA StatusCode's 2 severity bits (00/01/10/11) to a 3-state quality bucket
+    /// (0 = Good, 1 = Uncertain, 2 = Bad).</summary>
+    private static int QualityBucket(uint statusCode)
+    {
+        var severity = statusCode >> 30;
+        return severity >= 2 ? 2 : (int)severity;
+    }
+
+    /// <summary>#478 — a canonical worst StatusCode for an alarm's last-observed quality bucket, used by
+    /// the operator-command + shelving-timer emission paths (which don't re-read inputs) so an ack /
+    /// shelve while an input is Bad still carries Bad rather than resetting the condition to Good.</summary>
+    private uint LastWorstStatus(string alarmId)
+        => (_lastQualityBucketByAlarmId.TryGetValue(alarmId, out var bucket) ? bucket : 0) switch
+        {
+            2 => 0x80000000u, // Bad
+            1 => 0x40000000u, // Uncertain
+            _ => 0u,          // Good
+        };
+
     /// <summary>
     ///     Invoke the <see cref="OnEvent"/> handler for a built emission. Must be
     ///     called OUTSIDE <c>_evalGate</c>: a slow subscriber would otherwise
@@ -708,7 +824,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
                         _alarms[id] = state with { Condition = result.State };
                         if (result.Emission != EmissionKind.None)
                         {
-                            var evt = BuildEmission(state, result.State, result.Emission);
+                            var evt = BuildEmission(state, result.State, result.Emission, LastWorstStatus(id));
                             if (evt is not null) pending.Add(evt);
                         }
                     }
@@ -780,6 +896,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
         _alarms.Clear();
         _alarmsReferencing.Clear();
         _scratchByAlarmId.Clear();
+        _lastQualityBucketByAlarmId.Clear();
         // Dispose every compiled-predicate ALC so the engine's shutdown actually
         // releases the emitted assemblies. The drain above ensures no evaluator is
         // mid-call; CompiledScriptCache.Dispose internally guards against use-after-
@@ -851,7 +968,11 @@ public sealed record ScriptedAlarmEvent(
     EmissionKind Emission,
     DateTime TimestampUtc,
     string? Comment = null,
-    bool HistorizeToAveva = true);
+    bool HistorizeToAveva = true,
+    // #478 — the worst OPC UA StatusCode across the alarm's input tags at evaluation time. A raw uint
+    // (Core.ScriptedAlarms does not reference Commons/OpcUaQuality); the host maps it to OpcUaQuality by
+    // the top-2 severity bits. Default 0u == Good keeps every existing constructor call unchanged.
+    uint WorstInputStatusCode = 0u);
 
 /// <summary>
 ///     Upstream source abstraction — intentionally identical shape to the virtual-tag

src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmSource.cs

@@ -81,6 +81,11 @@ public sealed class ScriptedAlarmSource : IAlarmSource, IDisposable
     {
         if (_disposed) return;
 
+        // #478 — QualityChanged is a source-quality annotation, not a Part 9 state change. It is delivered
+        // out of band via the dedicated WriteAlarmQuality node path; surfacing it here would fabricate a
+        // phantom AlarmEventArgs that materializes / historizes a condition. Swallow it.
+        if (evt.Emission == EmissionKind.QualityChanged) return;
+
         foreach (var sub in _subscriptions.Values)
         {
             if (!Matches(sub, evt)) continue;

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser/GalaxyDriverBrowser.cs

@@ -5,6 +5,7 @@ using Microsoft.Extensions.Logging.Abstractions;
 using ZB.MOM.WW.MxGateway.Client;
 using ZB.MOM.WW.OtOpcUa.Commons.Browsing;
 using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
+using ZB.MOM.WW.Secrets.Abstractions;
 
 namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser;
 
@@ -34,11 +35,14 @@ public sealed class GalaxyDriverBrowser : IDriverBrowser
     };
 
     private readonly ILogger<GalaxyDriverBrowser> _logger;
+    private readonly ISecretResolver _secretResolver;
 
     /// <summary>Creates a new browser. Logger defaults to <see cref="NullLogger{T}"/>.</summary>
+    /// <param name="secretResolver">Resolves the <c>secret:</c> arm of <c>Gateway.ApiKeySecretRef</c> (DI-injected).</param>
     /// <param name="logger">Optional logger; null is allowed for unit-test construction.</param>
-    public GalaxyDriverBrowser(ILogger<GalaxyDriverBrowser>? logger = null)
+    public GalaxyDriverBrowser(ISecretResolver secretResolver, ILogger<GalaxyDriverBrowser>? logger = null)
     {
+        _secretResolver = secretResolver ?? throw new ArgumentNullException(nameof(secretResolver));
         _logger = logger ?? NullLogger<GalaxyDriverBrowser>.Instance;
     }
 
@@ -68,7 +72,7 @@ public sealed class GalaxyDriverBrowser : IDriverBrowser
         if (string.IsNullOrWhiteSpace(opts.MxAccess.ClientName))
             throw new InvalidOperationException("Galaxy browser requires MxAccess.ClientName.");
 
-        var clientOpts = BuildClientOptions(opts.Gateway);
+        var clientOpts = await BuildClientOptionsAsync(opts.Gateway, cancellationToken).ConfigureAwait(false);
 
         // 30s wall-clock budget for the connect phase, linked to the caller's token so
         // an AdminUI cancel still wins early.
@@ -116,19 +120,28 @@ public sealed class GalaxyDriverBrowser : IDriverBrowser
     ///     Build the gateway client options from the form's Gateway section. Mirrors the
     ///     runtime driver's <c>GalaxyDriver.BuildClientOptions</c> field-for-field so the
     ///     gateway sees an identical option shape. The API-key reference is resolved via
-    ///     the shared <see cref="GalaxySecretRef.ResolveApiKey"/> in Driver.Galaxy.Contracts
+    ///     the shared <see cref="GalaxySecretRef.ResolveApiKeyAsync"/> in Driver.Galaxy.Contracts
     ///     (the same resolver the runtime driver uses), so browse and runtime stay in lock-step.
+    ///     The <c>secret:</c> arm is async, so the key is resolved into a local before the
+    ///     object initializer (you can't await inside one).
     /// </summary>
-    private MxGatewayClientOptions BuildClientOptions(GalaxyGatewayOptions gw) => new()
+    private async Task<MxGatewayClientOptions> BuildClientOptionsAsync(
+        GalaxyGatewayOptions gw, CancellationToken cancellationToken)
     {
-        Endpoint = new Uri(gw.Endpoint, UriKind.Absolute),
-        ApiKey = GalaxySecretRef.ResolveApiKey(gw.ApiKeySecretRef, _logger),
-        UseTls = gw.UseTls,
-        CaCertificatePath = gw.CaCertificatePath,
-        ConnectTimeout = TimeSpan.FromSeconds(gw.ConnectTimeoutSeconds),
-        DefaultCallTimeout = TimeSpan.FromSeconds(gw.DefaultCallTimeoutSeconds),
-        StreamTimeout = gw.StreamTimeoutSeconds > 0
-            ? TimeSpan.FromSeconds(gw.StreamTimeoutSeconds)
-            : null,
-    };
+        var apiKey = await GalaxySecretRef
+            .ResolveApiKeyAsync(gw.ApiKeySecretRef, _secretResolver, _logger, cancellationToken)
+            .ConfigureAwait(false);
+        return new MxGatewayClientOptions
+        {
+            Endpoint = new Uri(gw.Endpoint, UriKind.Absolute),
+            ApiKey = apiKey,
+            UseTls = gw.UseTls,
+            CaCertificatePath = gw.CaCertificatePath,
+            ConnectTimeout = TimeSpan.FromSeconds(gw.ConnectTimeoutSeconds),
+            DefaultCallTimeout = TimeSpan.FromSeconds(gw.DefaultCallTimeoutSeconds),
+            StreamTimeout = gw.StreamTimeoutSeconds > 0
+                ? TimeSpan.FromSeconds(gw.StreamTimeoutSeconds)
+                : null,
+        };
+    }
 }

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/GalaxyDriverOptions.cs

@@ -43,12 +43,14 @@ public sealed record GalaxyDriverOptions(
 
 /// <summary>
 ///     Connection details for the MxAccess gateway. <see cref="ApiKeySecretRef"/> is
-///     resolved by <see cref="GalaxySecretRef.ResolveApiKey"/> at InitializeAsync time. Four forms
+///     resolved by <see cref="GalaxySecretRef.ResolveApiKeyAsync"/> at InitializeAsync time. Five forms
 ///     supported, in priority order:
 ///     <list type="bullet">
 ///       <item><c>env:NAME</c> — read from an environment variable (recommended for
 ///         production; the central config DB holds only the indirection, not the key).</item>
 ///       <item><c>file:PATH</c> — read from an ACL'd file outside the repo.</item>
+///       <item><c>secret:NAME</c> — resolved through the shared <c>ZB.MOM.WW.Secrets</c>
+///         encrypted store; fail-closed if absent (the production path).</item>
 ///       <item><c>dev:KEY</c> — explicit cleartext opt-in for dev rigs / parity tests;
 ///         no startup warning.</item>
 ///       <item>Anything else — treated as a literal cleartext API key for back-compat.

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/GalaxySecretRef.cs

@@ -1,9 +1,10 @@
 using Microsoft.Extensions.Logging;
+using ZB.MOM.WW.Secrets.Abstractions;
 
 namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
 
 /// <summary>
-///     Resolves <c>Gateway.ApiKeySecretRef</c> to the actual API-key string. Four
+///     Resolves <c>Gateway.ApiKeySecretRef</c> to the actual API-key string. Five
 ///     forms supported, evaluated in order:
 ///     <list type="number">
 ///       <item><c>env:NAME</c> — reads <c>Environment.GetEnvironmentVariable(NAME)</c>.
@@ -15,12 +16,17 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
 ///       <item><c>dev:KEY</c> — explicit cleartext literal. The <c>dev:</c> prefix
 ///         is a deliberate opt-in signal (dev box, parity rig) so the resolver
 ///         doesn't emit a warning; production should never use this arm.</item>
+///       <item><c>secret:NAME</c> — resolves NAME through the shared
+///         <c>ZB.MOM.WW.Secrets</c> <see cref="ISecretResolver"/> (the encrypted-at-rest
+///         store). Fail-closed: a <c>secret:</c> ref whose secret is absent/tombstoned
+///         throws rather than falling through to the literal arm — the production path
+///         that retires the cleartext <c>dev:</c>/literal-in-DB model.</item>
 ///       <item>Anything else — used as the literal API key for back-compat with
 ///         configs that pre-date this resolver. When a logger is supplied the
 ///         resolver emits a startup warning so an operator who accidentally
 ///         committed a cleartext key sees it.</item>
 ///     </list>
-///     A future PR can swap any of these arms for a DPAPI-backed lookup without
+///     A future PR can swap any of these arms for a different backing store without
 ///     changing the call site.
 /// </summary>
 /// <remarks>
@@ -31,18 +37,26 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
 public static class GalaxySecretRef
 {
     /// <summary>
-    ///     Resolves the supplied secret reference. When the ref falls through to the
-    ///     back-compat literal arm (an unprefixed cleartext API key in
-    ///     <c>DriverConfig</c> JSON) and a <paramref name="logger"/> is supplied, emits
-    ///     a <see cref="LogLevel.Warning"/>. The <c>dev:</c> prefix is the explicit
-    ///     opt-in path that doesn't warn.
+    ///     Resolves the supplied secret reference. The <c>secret:NAME</c> arm resolves
+    ///     through <paramref name="resolver"/> and is fail-closed (throws when the secret
+    ///     is absent). When the ref falls through to the back-compat literal arm (an
+    ///     unprefixed cleartext API key in <c>DriverConfig</c> JSON) and a
+    ///     <paramref name="logger"/> is supplied, emits a <see cref="LogLevel.Warning"/>.
+    ///     The <c>dev:</c> prefix is the explicit opt-in path that doesn't warn.
     /// </summary>
     /// <param name="secretRef">The secret reference string to resolve.</param>
+    /// <param name="resolver">The shared secret resolver used by the <c>secret:</c> arm.</param>
     /// <param name="logger">Optional logger for warning on cleartext keys.</param>
+    /// <param name="ct">Cancellation token for the async <c>secret:</c> resolution.</param>
     /// <returns>The resolved API-key string.</returns>
-    public static string ResolveApiKey(string secretRef, ILogger? logger = null)
+    public static async Task<string> ResolveApiKeyAsync(
+        string secretRef,
+        ISecretResolver resolver,
+        ILogger? logger = null,
+        CancellationToken ct = default)
     {
         ArgumentException.ThrowIfNullOrEmpty(secretRef);
+        ArgumentNullException.ThrowIfNull(resolver);
 
         if (secretRef.StartsWith("env:", StringComparison.OrdinalIgnoreCase))
         {
@@ -76,6 +90,19 @@ public static class GalaxySecretRef
             return secretRef[4..];
         }
 
+        if (secretRef.StartsWith("secret:", StringComparison.OrdinalIgnoreCase))
+        {
+            // Production path: resolve the name through the shared encrypted secret store.
+            // Fail-closed — an absent/tombstoned secret throws rather than falling through
+            // to the literal arm (which would silently treat the ref string as the key).
+            var name = secretRef["secret:".Length..];
+            var value = await resolver.GetAsync(new SecretName(name), ct).ConfigureAwait(false);
+            return !string.IsNullOrEmpty(value)
+                ? value
+                : throw new InvalidOperationException(
+                    $"Galaxy.Gateway.ApiKeySecretRef='{secretRef}' resolves secret '{name}', but it is absent from the store (fail-closed).");
+        }
+
         // Back-compat literal arm. An unprefixed string is treated as the literal
         // API key — but emit a warning so an operator who accidentally committed a
         // cleartext key into DriverConfig sees it. Use the dev: prefix to suppress

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts.csproj

@@ -13,5 +13,6 @@
         <ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Abstractions\ZB.MOM.WW.OtOpcUa.Core.Abstractions.csproj" />
         <!-- Logging abstraction needed by GalaxySecretRef.ResolveApiKey's optional warning logger. -->
         <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
+        <PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" />
     </ItemGroup>
 </Project>

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/GalaxyDriver.cs

@@ -8,6 +8,7 @@ using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browse;
 using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
 using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Health;
 using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
+using ZB.MOM.WW.Secrets.Abstractions;
 
 namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy;
 
@@ -43,6 +44,12 @@ public sealed class GalaxyDriver
     private readonly GalaxyDriverOptions _options;
     private readonly ILogger<GalaxyDriver> _logger;
 
+    // Resolves the Gateway.ApiKeySecretRef secret: arm through the shared encrypted store.
+    // Injected via ctor (the production factory pulls it from DI). The internal test ctor
+    // defaults it to a null-object resolver so the 45 seam-injecting test call sites keep
+    // compiling; those tests never use a secret: ref (they inject seams or use env/file/literal).
+    private readonly ISecretResolver _secretResolver;
+
     // PR 4.1 — IGalaxyHierarchySource is the test seam for browse. When null, the driver
     // lazily builds a GatewayGalaxyHierarchySource around a GalaxyRepositoryClient on
     // first DiscoverAsync. Tests inject a fake source via the internal ctor to exercise
@@ -147,14 +154,17 @@ public sealed class GalaxyDriver
     /// <summary>Initializes a new instance of the <see cref="GalaxyDriver"/> class.</summary>
     /// <param name="driverInstanceId">The unique identifier for this driver instance.</param>
     /// <param name="options">The Galaxy driver configuration options.</param>
+    /// <param name="secretResolver">Resolves the <c>secret:</c> arm of <c>Gateway.ApiKeySecretRef</c>.</param>
     /// <param name="logger">Optional logger instance for diagnostics.</param>
     public GalaxyDriver(
         string driverInstanceId,
         GalaxyDriverOptions options,
+        ISecretResolver secretResolver,
         ILogger<GalaxyDriver>? logger = null)
         : this(driverInstanceId, options,
               hierarchySource: null, dataReader: null, dataWriter: null, subscriber: null,
-              alarmAcknowledger: null, alarmFeed: null, logger)
+              alarmAcknowledger: null, alarmFeed: null, logger,
+              secretResolver: secretResolver ?? throw new ArgumentNullException(nameof(secretResolver)))
     {
     }
 
@@ -173,6 +183,11 @@ public sealed class GalaxyDriver
     /// <param name="alarmAcknowledger">Optional custom alarm acknowledger for testing.</param>
     /// <param name="alarmFeed">Optional custom alarm feed for testing.</param>
     /// <param name="logger">Optional logger instance for diagnostics.</param>
+    /// <param name="secretResolver">
+    ///     Optional secret resolver for the <c>secret:</c> API-key arm. Defaults to a
+    ///     null-object resolver (returns null for every name) so seam-injecting tests that
+    ///     don't exercise a <c>secret:</c> ref need not supply one.
+    /// </param>
     internal GalaxyDriver(
         string driverInstanceId,
         GalaxyDriverOptions options,
@@ -182,13 +197,15 @@ public sealed class GalaxyDriver
         IGalaxySubscriber? subscriber = null,
         IGalaxyAlarmAcknowledger? alarmAcknowledger = null,
         IGalaxyAlarmFeed? alarmFeed = null,
-        ILogger<GalaxyDriver>? logger = null)
+        ILogger<GalaxyDriver>? logger = null,
+        ISecretResolver? secretResolver = null)
     {
         _driverInstanceId = !string.IsNullOrWhiteSpace(driverInstanceId)
             ? driverInstanceId
             : throw new ArgumentException("Driver instance id required.", nameof(driverInstanceId));
         _options = options ?? throw new ArgumentNullException(nameof(options));
         _logger = logger ?? NullLogger<GalaxyDriver>.Instance;
+        _secretResolver = secretResolver ?? NullSecretResolver.Instance;
         _hierarchySource = hierarchySource;
         _dataReader = dataReader;
         _dataWriter = dataWriter;
@@ -316,7 +333,7 @@ public sealed class GalaxyDriver
                 _driverInstanceId);
         }
 
-        StartDeployWatcher();
+        await StartDeployWatcherAsync(cancellationToken).ConfigureAwait(false);
         _logger.LogInformation(
             "GalaxyDriver {InstanceId} initialized — endpoint={Endpoint} clientName={ClientName}",
             _driverInstanceId, _options.Gateway.Endpoint, _options.MxAccess.ClientName);
@@ -331,7 +348,7 @@ public sealed class GalaxyDriver
     /// </summary>
     private async Task BuildProductionRuntimeAsync(CancellationToken cancellationToken)
     {
-        var clientOptions = BuildClientOptions(_options.Gateway);
+        var clientOptions = await BuildClientOptionsAsync(_options.Gateway, cancellationToken).ConfigureAwait(false);
         _ownedMxClient = MxGatewayClient.Create(clientOptions);
         _ownedMxSession = new GalaxyMxSession(_options.MxAccess, _logger);
         await _ownedMxSession.ConnectAsync(clientOptions, cancellationToken).ConfigureAwait(false);
@@ -386,7 +403,7 @@ public sealed class GalaxyDriver
     private async Task ReopenAsync(CancellationToken cancellationToken)
     {
         if (_ownedMxSession is null) return;
-        var clientOptions = BuildClientOptions(_options.Gateway);
+        var clientOptions = await BuildClientOptionsAsync(_options.Gateway, cancellationToken).ConfigureAwait(false);
         await _ownedMxSession.RecreateAsync(clientOptions, cancellationToken).ConfigureAwait(false);
         // The recreated session invalidates every prior gw item handle; drop the writer's handle/advise
         // caches so the next write re-AddItems + re-AdviseSupervisory against the fresh session.
@@ -527,32 +544,42 @@ public sealed class GalaxyDriver
         }
     }
 
-    private MxGatewayClientOptions BuildClientOptions(GalaxyGatewayOptions gw) => new()
+    private async Task<MxGatewayClientOptions> BuildClientOptionsAsync(
+        GalaxyGatewayOptions gw, CancellationToken cancellationToken)
     {
-        Endpoint = new Uri(gw.Endpoint, UriKind.Absolute),
-        // Pass the logger so the literal-arm cleartext fallback surfaces a startup
-        // warning rather than silently shipping the key. The resolver lives in
-        // Driver.Galaxy.Contracts (GalaxySecretRef) so the runtime driver and the
-        // AdminUI browser share one implementation.
-        ApiKey = GalaxySecretRef.ResolveApiKey(gw.ApiKeySecretRef, _logger),
-        UseTls = gw.UseTls,
-        CaCertificatePath = gw.CaCertificatePath,
-        ConnectTimeout = TimeSpan.FromSeconds(gw.ConnectTimeoutSeconds),
-        DefaultCallTimeout = TimeSpan.FromSeconds(gw.DefaultCallTimeoutSeconds),
-        StreamTimeout = gw.StreamTimeoutSeconds > 0 ? TimeSpan.FromSeconds(gw.StreamTimeoutSeconds) : null,
-    };
+        // Resolve the API-key ref BEFORE the object initializer — the secret: arm is
+        // async and you can't await inside an initializer. Pass the logger so the
+        // literal-arm cleartext fallback surfaces a startup warning rather than
+        // silently shipping the key. The resolver lives in Driver.Galaxy.Contracts
+        // (GalaxySecretRef) so the runtime driver and the AdminUI browser share one
+        // implementation; the secret: arm resolves through the shared ISecretResolver.
+        var apiKey = await GalaxySecretRef
+            .ResolveApiKeyAsync(gw.ApiKeySecretRef, _secretResolver, _logger, cancellationToken)
+            .ConfigureAwait(false);
+        return new MxGatewayClientOptions
+        {
+            Endpoint = new Uri(gw.Endpoint, UriKind.Absolute),
+            ApiKey = apiKey,
+            UseTls = gw.UseTls,
+            CaCertificatePath = gw.CaCertificatePath,
+            ConnectTimeout = TimeSpan.FromSeconds(gw.ConnectTimeoutSeconds),
+            DefaultCallTimeout = TimeSpan.FromSeconds(gw.DefaultCallTimeoutSeconds),
+            StreamTimeout = gw.StreamTimeoutSeconds > 0 ? TimeSpan.FromSeconds(gw.StreamTimeoutSeconds) : null,
+        };
+    }
 
-    private void StartDeployWatcher()
+    private async Task StartDeployWatcherAsync(CancellationToken cancellationToken)
     {
         if (!_options.Repository.WatchDeployEvents) return;
         if (_ownedRepositoryClient is null && _hierarchySource is null) return;
 
         // Reuse the lazily-built repository client (DiscoverAsync constructs it on demand).
         // If discovery hasn't run yet, build the client here so the watcher has a target.
-        // Guard with ??= so if BuildDefaultHierarchySource later runs it reuses this client
-        // rather than overwriting the field and leaking the first instance.
+        // Build under a null-check (not ??=) so if BuildDefaultHierarchySourceAsync later
+        // runs it reuses this client rather than overwriting the field and leaking the
+        // first instance — the client-options build is now async (secret: arm).
         _ownedRepositoryClient ??= ZB.MOM.WW.MxGateway.Client.GalaxyRepositoryClient.Create(
-            BuildClientOptions(_options.Gateway));
+            await BuildClientOptionsAsync(_options.Gateway, cancellationToken).ConfigureAwait(false));
 
         var source = new GatewayGalaxyDeployWatchSource(_ownedRepositoryClient);
         _deployWatcher = new DeployWatcher(source, _logger);
@@ -691,7 +718,8 @@ public sealed class GalaxyDriver
         // After discovery, SyncPlatformsAsync refreshes the probe watcher's membership so
         // newly added $WinPlatform / $AppEngine objects start advising their ScanState attribute.
         var capturingBuilder = new SecurityCapturingBuilder(builder, _securityByFullRef);
-        var source = _hierarchySource ??= BuildDefaultHierarchySource();
+        var source = _hierarchySource ??=
+            await BuildDefaultHierarchySourceAsync(cancellationToken).ConfigureAwait(false);
         // Thread the v3 reverse resolver so the discoverer emits each authored attribute's RawPath as
         // its node reference (FullName) — the security map SecurityCapturingBuilder captures is then
         // keyed by RawPath, matching what WriteAsync resolves against.
@@ -1292,12 +1320,14 @@ public sealed class GalaxyDriver
     ///     <see cref="Dispose"/>. Tests bypass this by injecting their own source via the
     ///     internal ctor.
     /// </summary>
-    private IGalaxyHierarchySource BuildDefaultHierarchySource()
+    private async Task<IGalaxyHierarchySource> BuildDefaultHierarchySourceAsync(CancellationToken cancellationToken)
     {
-        // Reuse a client that StartDeployWatcher may have already created (??=) rather
+        // Reuse a client that StartDeployWatcherAsync may have already created (??=) rather
         // than always overwriting the field and leaking the first instance. Both paths
-        // produce equivalent clients from the same options.
-        _ownedRepositoryClient ??= GalaxyRepositoryClient.Create(BuildClientOptions(_options.Gateway));
+        // produce equivalent clients from the same options. The client-options build is
+        // async now (secret: arm resolves through the shared ISecretResolver).
+        _ownedRepositoryClient ??= GalaxyRepositoryClient.Create(
+            await BuildClientOptionsAsync(_options.Gateway, cancellationToken).ConfigureAwait(false));
         return new TracedGalaxyHierarchySource(
             new GatewayGalaxyHierarchySource(_ownedRepositoryClient), _options.MxAccess.ClientName);
     }

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/GalaxyDriverFactoryExtensions.cs

@@ -4,6 +4,7 @@ using Microsoft.Extensions.Logging;
 using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
 using ZB.MOM.WW.OtOpcUa.Core.Hosting;
 using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
+using ZB.MOM.WW.Secrets.Abstractions;
 
 namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy;
 
@@ -23,30 +24,45 @@ public static class GalaxyDriverFactoryExtensions
 {
     public const string DriverTypeName = "GalaxyMxGateway";
 
-    /// <summary>Registers the Galaxy driver factory with the given registry and optional logger factory.</summary>
+    /// <summary>Registers the Galaxy driver factory with the given registry, secret resolver, and optional logger factory.</summary>
     /// <param name="registry">The driver factory registry to register with.</param>
+    /// <param name="secretResolver">
+    ///     The shared secret resolver (from DI) used to resolve the <c>secret:</c> arm of
+    ///     each instance's <c>Gateway.ApiKeySecretRef</c>.
+    /// </param>
     /// <param name="loggerFactory">The optional logger factory for creating drivers.</param>
-    public static void Register(DriverFactoryRegistry registry, ILoggerFactory? loggerFactory = null)
+    public static void Register(
+        DriverFactoryRegistry registry,
+        ISecretResolver secretResolver,
+        ILoggerFactory? loggerFactory = null)
     {
         ArgumentNullException.ThrowIfNull(registry);
-        registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory));
+        ArgumentNullException.ThrowIfNull(secretResolver);
+        registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory, secretResolver));
     }
 
-    /// <summary>Convenience for tests + standalone callers.</summary>
+    /// <summary>
+    ///     Convenience for tests + standalone callers. Uses the <see cref="NullSecretResolver"/>
+    ///     null-object, so the <c>secret:</c> API-key arm resolves fail-closed (absent) — callers
+    ///     that need a working <c>secret:</c> ref must use the <see cref="Register"/> path that
+    ///     threads the DI resolver.
+    /// </summary>
     /// <param name="driverInstanceId">The unique identifier for the driver instance.</param>
     /// <param name="driverConfigJson">The driver configuration in JSON format.</param>
     /// <returns>The constructed Galaxy driver instance.</returns>
     public static GalaxyDriver CreateInstance(string driverInstanceId, string driverConfigJson)
-        => CreateInstance(driverInstanceId, driverConfigJson, loggerFactory: null);
+        => CreateInstance(driverInstanceId, driverConfigJson, loggerFactory: null, NullSecretResolver.Instance);
 
-    /// <summary>Creates a Galaxy driver instance from configuration JSON with optional logger factory.</summary>
+    /// <summary>Creates a Galaxy driver instance from configuration JSON with optional logger factory and a secret resolver.</summary>
     /// <param name="driverInstanceId">The unique identifier for the driver instance.</param>
     /// <param name="driverConfigJson">The driver configuration in JSON format.</param>
     /// <param name="loggerFactory">The optional logger factory for creating drivers.</param>
+    /// <param name="secretResolver">The secret resolver for the driver's <c>secret:</c> API-key arm.</param>
     /// <returns>The constructed Galaxy driver instance.</returns>
     public static GalaxyDriver CreateInstance(
-        string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory)
+        string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory, ISecretResolver secretResolver)
     {
+        ArgumentNullException.ThrowIfNull(secretResolver);
         ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
         ArgumentException.ThrowIfNullOrWhiteSpace(driverConfigJson);
 
@@ -88,7 +104,7 @@ public static class GalaxyDriverFactoryExtensions
             RawTags = dto.RawTags ?? [],
         };
 
-        return new GalaxyDriver(driverInstanceId, options, loggerFactory?.CreateLogger<GalaxyDriver>());
+        return new GalaxyDriver(driverInstanceId, options, secretResolver, loggerFactory?.CreateLogger<GalaxyDriver>());
     }
 
     private static readonly JsonSerializerOptions JsonOptions = new()

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/NullSecretResolver.cs

@@ -0,0 +1,26 @@
+using ZB.MOM.WW.Secrets.Abstractions;
+
+namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy;
+
+/// <summary>
+///     A null-object <see cref="ISecretResolver"/> that resolves every name to <c>null</c>
+///     (absent). Used as the default for the internal test ctor and the parse-only
+///     <c>CreateInstance</c> path so callers that never exercise a <c>secret:</c> API-key
+///     ref need not thread a real resolver. A production <c>GalaxyDriver</c> always receives
+///     the DI-registered resolver via the factory; a <c>secret:</c> ref resolved against this
+///     null object throws fail-closed (the secret is reported absent), which is the correct
+///     behaviour for a mis-wired deployment.
+/// </summary>
+internal sealed class NullSecretResolver : ISecretResolver
+{
+    /// <summary>The shared singleton instance.</summary>
+    public static readonly NullSecretResolver Instance = new();
+
+    private NullSecretResolver()
+    {
+    }
+
+    /// <inheritdoc />
+    public Task<string?> GetAsync(SecretName name, CancellationToken ct) =>
+        Task.FromResult<string?>(null);
+}

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Contracts/OpcUaClientDriverOptions.cs

@@ -14,7 +14,9 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
 ///     exposes PLC data" flow. Tier A (pure managed, OPC Foundation reference SDK); universal
 ///     protections cover it.
 /// </remarks>
-public sealed class OpcUaClientDriverOptions
+// A record (not a plain class) so G-2b's OpcUaClientSecretResolution can produce a
+// credential-resolved copy with a `with` expression — every property keeps its init setter.
+public sealed record OpcUaClientDriverOptions
 {
     /// <summary>
     ///     Remote OPC UA endpoint URL, e.g. <c>opc.tcp://plc.internal:4840</c>. Convenience

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/NullSecretResolver.cs

@@ -0,0 +1,26 @@
+using ZB.MOM.WW.Secrets.Abstractions;
+
+namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
+
+/// <summary>
+///     A null-object <see cref="ISecretResolver"/> that resolves every name to <c>null</c>
+///     (absent). Backs the driver's default (test/parse-only) construction path so callers that
+///     never author a <c>secret:</c> credential ref need not thread a real resolver. A production
+///     <c>OpcUaClientDriver</c> always receives the DI-registered resolver via the factory; a
+///     <c>secret:</c> ref resolved against this null object throws fail-closed (the secret is
+///     reported absent), which is the correct behaviour for a mis-wired deployment — the
+///     <c>secret:</c> literal is never sent verbatim as a password.
+/// </summary>
+internal sealed class NullSecretResolver : ISecretResolver
+{
+    /// <summary>The shared singleton instance.</summary>
+    public static readonly NullSecretResolver Instance = new();
+
+    private NullSecretResolver()
+    {
+    }
+
+    /// <inheritdoc />
+    public Task<string?> GetAsync(SecretName name, CancellationToken ct) =>
+        Task.FromResult<string?>(null);
+}

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriver.cs

@@ -4,6 +4,7 @@ using Opc.Ua;
 using Opc.Ua.Client;
 using Opc.Ua.Configuration;
 using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
+using ZB.MOM.WW.Secrets.Abstractions;
 
 namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
 
@@ -36,15 +37,25 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
     /// <param name="options">Driver configuration.</param>
     /// <param name="driverInstanceId">Stable logical ID from the config DB.</param>
     /// <param name="logger">Optional logger; defaults to NullLogger when not supplied.</param>
+    /// <param name="secretResolver">
+    ///     Optional shared secret resolver used to resolve <c>secret:</c>-prefixed
+    ///     <see cref="OpcUaClientDriverOptions.Password"/> /
+    ///     <see cref="OpcUaClientDriverOptions.UserCertificatePassword"/> at session-open.
+    ///     Defaults to the <see cref="NullSecretResolver"/> null-object (which reports every
+    ///     secret absent) for the test/parse-only path; the production factory threads the
+    ///     DI-registered resolver. A <c>secret:</c> ref against the null-object fails closed.
+    /// </param>
     public OpcUaClientDriver(OpcUaClientDriverOptions options, string driverInstanceId,
-        ILogger<OpcUaClientDriver>? logger = null)
+        ILogger<OpcUaClientDriver>? logger = null, ISecretResolver? secretResolver = null)
     {
         _options = options;
         _driverInstanceId = driverInstanceId;
         _logger = logger ?? NullLogger<OpcUaClientDriver>.Instance;
+        _secretResolver = secretResolver ?? NullSecretResolver.Instance;
     }
 
     private readonly OpcUaClientDriverOptions _options;
+    private readonly ISecretResolver _secretResolver;
     private readonly string _driverInstanceId;
     // ---- IAlarmSource state ----
 
@@ -159,7 +170,18 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
             var appConfig = await BuildApplicationConfigurationAsync(cancellationToken).ConfigureAwait(false);
             var candidates = ResolveEndpointCandidates(_options);
 
-            var identity = BuildUserIdentity(_options);
+            // G-2b: resolve any secret:-prefixed Password / UserCertificatePassword through the
+            // shared secret store ONCE, here in the async connect flow, just before the credentials
+            // are consumed. _options stays the raw config (with secret: refs) — only this
+            // connect-scoped local carries plaintext, and only for the duration of the connect.
+            // Resolving on every InitializeAsync (a full reconnect calls it) picks up rotations.
+            // Both credential consumers — the Username password below and the Certificate password
+            // in BuildCertificateIdentity — flow through BuildUserIdentity(resolvedOptions), so a
+            // single resolve covers both paths.
+            var resolvedOptions = await OpcUaClientSecretResolution
+                .ResolveSecretRefsAsync(_options, _secretResolver, cancellationToken).ConfigureAwait(false);
+
+            var identity = BuildUserIdentity(resolvedOptions);
 
             // Failover sweep: try each endpoint in order, return the session from the first
             // one that successfully connects. Per-endpoint failures are captured so the final

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriverFactoryExtensions.cs

@@ -2,6 +2,7 @@ using System.Text.Json;
 using System.Text.Json.Serialization;
 using Microsoft.Extensions.Logging;
 using ZB.MOM.WW.OtOpcUa.Core.Hosting;
+using ZB.MOM.WW.Secrets.Abstractions;
 
 namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
 
@@ -31,22 +32,37 @@ public static class OpcUaClientDriverFactoryExtensions
         Converters = { new JsonStringEnumConverter() },
     };
 
-    /// <summary>Register the OpcUaClient factory with the driver registry.</summary>
+    /// <summary>Register the OpcUaClient factory with the driver registry, threading the shared secret resolver.</summary>
     /// <param name="registry">The driver factory registry to register with.</param>
     /// <param name="loggerFactory">Optional logger factory used to create per-instance loggers.</param>
-    public static void Register(DriverFactoryRegistry registry, ILoggerFactory? loggerFactory = null)
+    /// <param name="secretResolver">
+    ///     The shared secret resolver (from DI) used to resolve the <c>secret:</c> arm of each
+    ///     instance's <c>Password</c> / <c>UserCertificatePassword</c> at session-open. Defaults to
+    ///     the <see cref="NullSecretResolver"/> null-object for the test/standalone path — callers
+    ///     that need a working <c>secret:</c> credential ref must thread the DI resolver.
+    /// </param>
+    public static void Register(
+        DriverFactoryRegistry registry,
+        ILoggerFactory? loggerFactory = null,
+        ISecretResolver? secretResolver = null)
     {
         ArgumentNullException.ThrowIfNull(registry);
-        registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory));
+        var resolver = secretResolver ?? NullSecretResolver.Instance;
+        registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory, resolver));
     }
 
     /// <summary>Public for the Server-side bootstrapper + test consumers.</summary>
     /// <param name="driverInstanceId">The unique identifier for the driver instance.</param>
     /// <param name="driverConfigJson">The JSON configuration string for the driver.</param>
     /// <param name="loggerFactory">Optional logger factory for the per-instance logger.</param>
+    /// <param name="secretResolver">
+    ///     Optional shared secret resolver for the driver's <c>secret:</c> credential arm; defaults
+    ///     to the <see cref="NullSecretResolver"/> null-object (fail-closed on a <c>secret:</c> ref).
+    /// </param>
     /// <returns>A configured <see cref="OpcUaClientDriver"/>.</returns>
     public static OpcUaClientDriver CreateInstance(
-        string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory = null)
+        string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory = null,
+        ISecretResolver? secretResolver = null)
     {
         ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
         ArgumentException.ThrowIfNullOrWhiteSpace(driverConfigJson);
@@ -55,6 +71,8 @@ public static class OpcUaClientDriverFactoryExtensions
             ?? throw new InvalidOperationException(
                 $"OpcUaClient driver config for '{driverInstanceId}' deserialised to null");
 
-        return new OpcUaClientDriver(options, driverInstanceId, loggerFactory?.CreateLogger<OpcUaClientDriver>());
+        return new OpcUaClientDriver(
+            options, driverInstanceId, loggerFactory?.CreateLogger<OpcUaClientDriver>(),
+            secretResolver ?? NullSecretResolver.Instance);
     }
 }

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriverProbe.cs

@@ -34,6 +34,12 @@ public sealed class OpcUaClientDriverProbe : IDriverProbe
     /// <inheritdoc />
     public async Task<DriverProbeResult> ProbeAsync(string configJson, TimeSpan timeout, CancellationToken ct)
     {
+        // G-2b note: the probe does an UNAUTHENTICATED GetEndpoints discovery preflight
+        // (BuildMinimalAppConfig sets no user identity) and never reads opts.Password /
+        // opts.UserCertificatePassword. Those credentials — including any secret: refs — are
+        // therefore intentionally NOT resolved here; resolving them would be dead code. Secret
+        // resolution happens lazily in OpcUaClientDriver's async session-open path, where the
+        // credentials are actually consumed.
         OpcUaClientDriverOptions? opts;
         try { opts = JsonSerializer.Deserialize<OpcUaClientDriverOptions>(configJson, _opts); }
         catch (Exception ex) { return new(false, $"Config JSON is invalid: {ex.Message}", null); }

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientSecretResolution.cs

@@ -0,0 +1,90 @@
+using ZB.MOM.WW.Secrets.Abstractions;
+
+namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
+
+/// <summary>
+///     Layer-B (G-2b) secret resolution for the OPC UA Client driver's credential fields. Resolves
+///     <c>secret:</c>-prefixed <see cref="OpcUaClientDriverOptions.Password"/> and
+///     <see cref="OpcUaClientDriverOptions.UserCertificatePassword"/> through the shared
+///     <see cref="ISecretResolver"/> (the encrypted-at-rest store) just before the async
+///     session-open path consumes them — retiring the cleartext password-in-DB model for
+///     production.
+/// </summary>
+/// <remarks>
+///     Resolution is lazy (called from the connect flow, not at config deserialization) so a
+///     full reconnect via <c>InitializeAsync</c> re-resolves and picks up secret rotations,
+///     mirroring the Galaxy driver's re-resolve-on-reconnect behaviour. Only <c>secret:</c>-prefixed
+///     values are resolved; a null/empty or non-<c>secret:</c> value passes through verbatim so the
+///     literal-password back-compat path is preserved. The <c>secret:</c> arm is <b>fail-closed</b>:
+///     an absent/tombstoned secret throws rather than leaving the <c>secret:</c> literal in place,
+///     which would otherwise be sent verbatim to the remote server as the password.
+/// </remarks>
+internal static class OpcUaClientSecretResolution
+{
+    private const string SecretPrefix = "secret:";
+
+    /// <summary>
+    ///     Return a copy of <paramref name="options"/> with any <c>secret:</c>-prefixed
+    ///     <see cref="OpcUaClientDriverOptions.Password"/> /
+    ///     <see cref="OpcUaClientDriverOptions.UserCertificatePassword"/> resolved to their
+    ///     plaintext via <paramref name="resolver"/>. Non-secret / null / empty fields are
+    ///     returned unchanged.
+    /// </summary>
+    /// <param name="options">The raw driver options (credential fields may carry <c>secret:</c> refs).</param>
+    /// <param name="resolver">The shared secret resolver used by the <c>secret:</c> arm.</param>
+    /// <param name="ct">Cancellation token for the async secret resolution.</param>
+    /// <returns>An options copy whose credential fields carry resolved plaintext.</returns>
+    /// <exception cref="InvalidOperationException">
+    ///     A <c>secret:</c> ref names a secret that is absent/tombstoned in the store (fail-closed).
+    /// </exception>
+    internal static async Task<OpcUaClientDriverOptions> ResolveSecretRefsAsync(
+        OpcUaClientDriverOptions options, ISecretResolver resolver, CancellationToken ct)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+        ArgumentNullException.ThrowIfNull(resolver);
+
+        var password = await ResolveFieldAsync(options.Password, nameof(options.Password), resolver, ct)
+            .ConfigureAwait(false);
+        var certPassword = await ResolveFieldAsync(
+            options.UserCertificatePassword, nameof(options.UserCertificatePassword), resolver, ct)
+            .ConfigureAwait(false);
+
+        // Only re-materialize when something actually changed — a `with` on the reference-equal
+        // strings is harmless, but skipping it keeps the common (no-secret) case allocation-free.
+        if (ReferenceEquals(password, options.Password)
+            && ReferenceEquals(certPassword, options.UserCertificatePassword))
+        {
+            return options;
+        }
+
+        return options with { Password = password, UserCertificatePassword = certPassword };
+    }
+
+    /// <summary>
+    ///     Resolve a single credential field. Null/empty or non-<c>secret:</c> values pass through
+    ///     unchanged (the reference-equal original is returned). A <c>secret:NAME</c> value is
+    ///     resolved through <paramref name="resolver"/> and is fail-closed when the secret is absent.
+    /// </summary>
+    /// <param name="value">The raw field value (may be a <c>secret:</c> ref).</param>
+    /// <param name="fieldName">The field name, used in the fail-closed exception message.</param>
+    /// <param name="resolver">The shared secret resolver.</param>
+    /// <param name="ct">Cancellation token for the async resolution.</param>
+    /// <returns>The resolved plaintext, or the original value when it is not a <c>secret:</c> ref.</returns>
+    private static async Task<string?> ResolveFieldAsync(
+        string? value, string fieldName, ISecretResolver resolver, CancellationToken ct)
+    {
+        if (string.IsNullOrEmpty(value)
+            || !value.StartsWith(SecretPrefix, StringComparison.OrdinalIgnoreCase))
+        {
+            return value;
+        }
+
+        var name = value[SecretPrefix.Length..];
+        var resolved = await resolver.GetAsync(new SecretName(name), ct).ConfigureAwait(false);
+        return !string.IsNullOrEmpty(resolved)
+            ? resolved
+            : throw new InvalidOperationException(
+                $"OpcUaClientDriverOptions.{fieldName}='{value}' resolves secret '{name}', but it is " +
+                "absent from the store (fail-closed).");
+    }
+}

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.csproj

@@ -21,6 +21,7 @@
     <ItemGroup>
         <PackageReference Include="OPCFoundation.NetStandard.Opc.Ua.Client"/>
         <PackageReference Include="OPCFoundation.NetStandard.Opc.Ua.Configuration"/>
+        <PackageReference Include="ZB.MOM.WW.Secrets.Abstractions"/>
     </ItemGroup>
 
     <ItemGroup>

src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/App.razor

@@ -19,7 +19,11 @@
     <HeadOutlet/>
 </head>
 <body>
-    <Routes/>
+    @* No AdditionalAssemblies: the Secrets.Ui RCL's routable page is NOT routed directly in this
+       host — Pages/Secrets.razor wraps it so the route can carry this host's per-page
+       @rendermode (lmxopcua#483); registering the RCL routes too would make /admin/secrets
+       ambiguous. *@
+    <Routes />
     <script src="_content/ZB.MOM.WW.OtOpcUa.AdminUI/lib/bootstrap/js/bootstrap.bundle.min.js"></script>
     <ThemeScripts />
     <script src="_content/ZB.MOM.WW.OtOpcUa.AdminUI/js/monaco-init.js"></script>

src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Layout/MainLayout.razor

@@ -20,6 +20,7 @@
             <NavRailItem Href="/reservations" Text="Reservations" />
             <NavRailItem Href="/certificates" Text="Certificates" />
             <NavRailItem Href="/role-grants" Text="Role grants" />
+            <NavRailItem Href="/admin/secrets" Text="Secrets" />
         </NavRailSection>
         <NavRailSection Title="Scripting" Key="scripting">
             <NavRailItem Href="/scripts" Text="Scripts" />

src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/SecretsAdmin.razor

@@ -0,0 +1,20 @@
+@page "/admin/secrets"
+@attribute [Authorize(Policy = ZB.MOM.WW.Secrets.Ui.SecretsAuthorization.ManagePolicy)]
+@rendermode RenderMode.InteractiveServer
+@using ZB.MOM.WW.Secrets.Ui
+
+@* Host-side wrapper for the ZB.MOM.WW.Secrets.Ui secrets page (lmxopcua#483).
+
+   This host's <Routes/> is deliberately static (cookie SignInAsync needs SSR), so every
+   interactive page opts in per-page with @rendermode — a directive the RCL's own routable
+   SecretsPage cannot carry, because the other three family hosts render it under a globally
+   interactive router where a nested render mode throws. Routing straight to the RCL page
+   therefore produced a statically-rendered, dead page here: no circuit, @onclick inert.
+
+   So this wrapper owns the /admin/secrets route in THIS host (the RCL assembly is no longer
+   passed to either AdditionalAssemblies registration — doing both would make the route
+   ambiguous), carries the standard per-page render mode, re-states the RCL page's own
+   authorization policy (the router only enforces [Authorize] on the routed page itself,
+   which is now this one), and renders the RCL page as an ordinary child component. *@
+
+<SecretsPage />

src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/EndpointRouteBuilderExtensions.cs

@@ -1,3 +1,4 @@
+using Microsoft.AspNetCore.Authorization;
 using Microsoft.AspNetCore.Builder;
 using Microsoft.AspNetCore.Components;
 using Microsoft.AspNetCore.Components.Web;
@@ -10,6 +11,7 @@ using ZB.MOM.WW.OtOpcUa.Commons.Browsing;
 using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
 using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser;
 using ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser;
+using ZB.MOM.WW.Secrets.Ui;
 
 namespace ZB.MOM.WW.OtOpcUa.AdminUI;
 
@@ -28,6 +30,12 @@ public static class EndpointRouteBuilderExtensions
         // Razor class library static assets (_content/ZB.MOM.WW.OtOpcUa.AdminUI/**) are
         // served via the Host's app.UseStaticFiles() middleware which must run BEFORE
         // UseAuthentication() — see Program.cs.
+        // The Secrets.Ui RCL's routable page is deliberately NOT registered here (lmxopcua#483):
+        // this host's router is static with per-page @rendermode opt-in, so routing straight to
+        // the RCL page rendered it without a circuit — it displayed but every @onclick was dead.
+        // Pages/Secrets.razor in THIS assembly owns /admin/secrets instead (wrapping the RCL
+        // page with the render mode); registering the RCL routes too would make the route
+        // ambiguous.
         app.MapRazorComponents<TApp>()
             .AddInteractiveServerRenderMode();
         return app;
@@ -43,6 +51,15 @@ public static class EndpointRouteBuilderExtensions
         services.AddRazorComponents().AddInteractiveServerComponents();
         services.AddOtOpcUaDriverStatusServices();
 
+        // Secrets-management UI (ZB.MOM.WW.Secrets.Ui RCL, mounted at /admin/secrets). Register its
+        // "secrets:manage"/"secrets:reveal" authorization policies additively onto the AuthorizationOptions
+        // that AddOtOpcUaAuth (called just before AddAdminUI on admin nodes) sets up — Configure<T> stacks,
+        // so this ADDS the two policies without disturbing FleetAdmin/DriverOperator/ConfigEditor. The
+        // policies' AdminRole = "Administrator" reads the same ClaimTypes.Role claim as FleetAdmin, so an
+        // existing Administrator satisfies them with no new group→role mapping. Registered here (the AdminUI
+        // composition layer that already references the RCL) rather than in the core Security lib.
+        services.Configure<AuthorizationOptions>(o => o.AddSecretsAuthorization());
+
         // Browse pipeline — see docs/plans/2026-05-28-driver-browsers-design.md
         services.AddSingleton<Browsing.BrowseSessionRegistry>();
         services.AddHostedService<Browsing.BrowseSessionReaper>();

src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ZB.MOM.WW.OtOpcUa.AdminUI.csproj

@@ -10,6 +10,7 @@
         <FrameworkReference Include="Microsoft.AspNetCore.App"/>
         <PackageReference Include="Microsoft.AspNetCore.SignalR.Client"/>
         <PackageReference Include="ZB.MOM.WW.Theme"/>
+        <PackageReference Include="ZB.MOM.WW.Secrets.Ui"/>
     </ItemGroup>
 
     <ItemGroup>

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/KestrelHttpBinding.cs

@@ -0,0 +1,102 @@
+using System.Net;
+using Microsoft.AspNetCore.Server.Kestrel.Core;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     One HTTP endpoint the host was asked to serve, parsed out of the configured URL list so it
+///     can be re-bound explicitly.
+/// </summary>
+/// <param name="Host">The host component as written — <c>+</c>, <c>*</c>, a hostname, or an IP.</param>
+/// <param name="Port">The port.</param>
+/// <param name="IsSecure">True when the URL used the <c>https</c> scheme.</param>
+public sealed record KestrelHttpBinding(string Host, int Port, bool IsSecure)
+{
+    /// <summary>
+    ///     Parses a semicolon-separated URL list (the <c>ASPNETCORE_URLS</c> / <c>urls</c> format)
+    ///     into bindings. Unparseable entries are skipped rather than throwing — a malformed URL
+    ///     must not take the process down at startup.
+    /// </summary>
+    /// <param name="urls">The configured URL list, or null/empty when none is configured.</param>
+    /// <returns>The parsed bindings, in the order given; empty when nothing is configured.</returns>
+    public static IReadOnlyList<KestrelHttpBinding> Parse(string? urls)
+    {
+        if (string.IsNullOrWhiteSpace(urls))
+            return [];
+
+        var result = new List<KestrelHttpBinding>();
+        foreach (var raw in urls.Split(';', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries))
+        {
+            // Uri cannot parse the "+"/"*" wildcard hosts Kestrel accepts, so substitute a
+            // placeholder host purely to get scheme/port out, then keep the original host token.
+            var isWildcard = raw.Contains("://+", StringComparison.Ordinal)
+                          || raw.Contains("://*", StringComparison.Ordinal);
+            var probe = isWildcard
+                ? raw.Replace("://+", "://placeholder", StringComparison.Ordinal)
+                     .Replace("://*", "://placeholder", StringComparison.Ordinal)
+                : raw;
+
+            if (!Uri.TryCreate(probe, UriKind.Absolute, out var uri))
+                continue;
+
+            var host = isWildcard
+                ? raw.Contains("://+", StringComparison.Ordinal) ? "+" : "*"
+                : uri.Host;
+
+            result.Add(new KestrelHttpBinding(host, uri.Port, uri.Scheme == Uri.UriSchemeHttps));
+        }
+
+        return result;
+    }
+
+    /// <summary>
+    ///     Parses the bare-port list format of <c>ASPNETCORE_HTTP_PORTS</c> / <c>HTTP_PORTS</c> (and
+    ///     the <c>HTTPS</c> variants) into wildcard (<c>+</c>, all-interfaces) bindings — the shape
+    ///     Kestrel itself gives those variables. Modern .NET base images (aspnet:8.0+) set
+    ///     <c>ASPNETCORE_HTTP_PORTS=8080</c> as the container default <b>instead of</b>
+    ///     <c>ASPNETCORE_URLS</c>, so a node that never sets <c>URLS</c> explicitly still has a real
+    ///     configured surface here that must be re-bound, not replaced with Kestrel's localhost:5000.
+    /// </summary>
+    /// <param name="ports">A <c>;</c>- or <c>,</c>-separated list of bare ports, or null/empty.</param>
+    /// <param name="isSecure">True to mark the parsed bindings <c>https</c> (the <c>HTTPS_PORTS</c> vars).</param>
+    /// <returns>One all-interfaces binding per parseable port; empty when nothing is configured.</returns>
+    public static IReadOnlyList<KestrelHttpBinding> ParsePorts(string? ports, bool isSecure)
+    {
+        if (string.IsNullOrWhiteSpace(ports))
+            return [];
+
+        var result = new List<KestrelHttpBinding>();
+        foreach (var token in ports.Split([';', ','], StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries))
+        {
+            // Skip a malformed port rather than throwing — a bad env var must not down startup.
+            if (int.TryParse(token, out var port) && port is > 0 and <= 65535)
+                result.Add(new KestrelHttpBinding("+", port, isSecure));
+        }
+
+        return result;
+    }
+
+    /// <summary>
+    ///     Applies this binding to Kestrel, choosing the narrowest listen call the host component
+    ///     allows.
+    /// </summary>
+    /// <remarks>
+    ///     A hostname that is neither a literal IP nor <c>localhost</c> cannot be reliably mapped to
+    ///     a local interface, so it falls back to <see cref="KestrelServerOptions.ListenAnyIP"/> —
+    ///     the safe superset. Narrowing it and guessing wrong would silently stop serving.
+    /// </remarks>
+    /// <param name="options">The Kestrel options being configured.</param>
+    public void Apply(KestrelServerOptions options)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+
+        if (Host is "+" or "*")
+            options.ListenAnyIP(Port);
+        else if (IPAddress.TryParse(Host, out var address))
+            options.Listen(address, Port);
+        else if (string.Equals(Host, "localhost", StringComparison.OrdinalIgnoreCase))
+            options.ListenLocalhost(Port);
+        else
+            options.ListenAnyIP(Port);
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs

@@ -0,0 +1,101 @@
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.DependencyInjection;
+using ZB.MOM.WW.LocalDb;
+using ZB.MOM.WW.LocalDb.Replication;
+using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     Single registration point for the node-local LocalDb subsystem: the embedded SQLite store
+///     that caches the deployed-configuration artifact, plus the optional gRPC replication that
+///     mirrors it to the node's redundant pair peer.
+/// </summary>
+/// <remarks>
+///     <para>
+///         Exists as a named extension rather than inline <c>AddZbLocalDb</c> calls in
+///         <c>Program.cs</c> so the wiring is covered by a DI resolution test — <c>Program.cs</c> is
+///         top-level statements and cannot be exercised directly, which is exactly how a
+///         "registered but never resolvable" defect ships unnoticed. This family has shipped that
+///         defect three times (Secrets 0.2.0, Secrets 0.2.2, ScadaBridge #22).
+///     </para>
+///     <para>
+///         <b>Driver-role nodes only.</b> Admin-only nodes have no deployed configuration to cache,
+///         and registering here would impose the <c>LocalDb:Path</c>-required-or-no-boot constraint
+///         on them for nothing.
+///     </para>
+///     <para>
+///         <b>Storage is unconditional; replication is default-OFF.</b> A node with no
+///         <c>PeerAddress</c> and no <c>SyncListenPort</c> is simply a fast local SQLite file —
+///         <c>AddZbLocalDbReplication</c> registers the engine but it stays idle. Replication is
+///         opt-in per pair because there is no production distribution story for the sync
+///         <c>ApiKey</c> yet (the same open question the Secrets KEK has).
+///     </para>
+/// </remarks>
+public static class LocalDbRegistration
+{
+    /// <summary>Configuration section holding <c>LocalDbOptions</c>.</summary>
+    public const string LocalDbSectionPath = "LocalDb";
+
+    /// <summary>Configuration section holding <c>ReplicationOptions</c>.</summary>
+    public const string ReplicationSectionPath = "LocalDb:Replication";
+
+    /// <summary>
+    ///     Configuration key carrying the dedicated h2c sync listener's port. Zero or absent means
+    ///     no listener is bound and the passive sync endpoint is not mapped.
+    /// </summary>
+    public const string SyncListenPortKey = "LocalDb:SyncListenPort";
+
+    /// <summary>
+    ///     The port the dedicated cleartext-HTTP/2 sync listener should bind, or <c>0</c> when the
+    ///     node should not listen at all. Defaults to <c>0</c> — absent configuration must mean
+    ///     "off".
+    /// </summary>
+    /// <param name="configuration">The application configuration.</param>
+    /// <returns>The configured sync port, or <c>0</c>.</returns>
+    public static int SyncListenPort(IConfiguration configuration)
+    {
+        ArgumentNullException.ThrowIfNull(configuration);
+        return configuration.GetValue(SyncListenPortKey, defaultValue: 0);
+    }
+
+    /// <summary>
+    ///     Registers the local database (with the deployment-cache schema applied and registered for
+    ///     replication) and the replication engine.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <see cref="LocalDbSetup.OnReady"/> runs inside <c>AddZbLocalDb</c>'s singleton factory,
+    ///         before the first caller receives the <c>ILocalDb</c> — so every consumer is guaranteed
+    ///         to see the tables created and their capture triggers installed.
+    ///     </para>
+    ///     <para>
+    ///         <c>AddZbLocalDbReplication</c> is called unconditionally rather than behind an
+    ///         <c>Enabled</c> flag because the library already models "off" as "no
+    ///         <c>PeerAddress</c>": the initiator background service idles and the passive endpoint
+    ///         is only reachable if <c>Program.cs</c> maps it, which it does only when
+    ///         <see cref="SyncListenPortKey"/> is set. Adding a second gate on top would give two
+    ///         ways to express the same thing and a way for them to disagree.
+    ///     </para>
+    /// </remarks>
+    /// <param name="services">The service collection to add to.</param>
+    /// <param name="configuration">The application configuration.</param>
+    /// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
+    public static IServiceCollection AddOtOpcUaLocalDb(
+        this IServiceCollection services,
+        IConfiguration configuration)
+    {
+        ArgumentNullException.ThrowIfNull(services);
+        ArgumentNullException.ThrowIfNull(configuration);
+
+        services.AddZbLocalDb(configuration, LocalDbSetup.OnReady);
+        services.AddZbLocalDbReplication(configuration);
+
+        // The consumer-facing seam. WithOtOpcUaRuntimeActors resolves this optionally and threads it
+        // into DriverHostActor; registering the store without this leaves the cache tables present,
+        // replicating, and never written to.
+        services.AddSingleton<IDeploymentArtifactCache, LocalDbDeploymentArtifactCache>();
+
+        return services;
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs

@@ -0,0 +1,48 @@
+using ZB.MOM.WW.LocalDb;
+using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     The <c>onReady</c> callback handed to <c>AddZbLocalDb</c>: creates the deployment-cache
+///     tables and opts them into replication.
+/// </summary>
+/// <remarks>
+///     <para>
+///         Public rather than internal only so <c>LocalDbSetupTests</c> can drive the production
+///         callback directly. Initialising a test database from a hand-written copy of this schema
+///         would prove only that the test agrees with itself — the whole value of those tests is
+///         that they exercise <b>this</b> method.
+///     </para>
+/// </remarks>
+public static class LocalDbSetup
+{
+    /// <summary>
+    ///     Initialises the local database: DDL first, then replication registration.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b>THE ORDER IS LOAD-BEARING: DDL → RegisterReplicated → writes.</b>
+    ///         <c>RegisterReplicated</c> is what installs the three AFTER triggers that capture
+    ///         changes into the oplog. Any row written before that call is never captured, so it
+    ///         never reaches the peer — silently, and permanently, because nothing ever revisits
+    ///         history. Phase 1 writes nothing here; when Phase 2 adds its store-and-forward
+    ///         migrator, the migrator must run <i>after</i> both registrations for the same reason.
+    ///     </para>
+    /// </remarks>
+    /// <param name="db">The freshly constructed local database.</param>
+    public static void OnReady(ILocalDb db)
+    {
+        ArgumentNullException.ThrowIfNull(db);
+
+        // CreateConnection() hands back an already-open, pragma-configured connection carrying the
+        // zb_hlc_next() UDF the capture triggers need. Calling Open() on it would throw.
+        using (var connection = db.CreateConnection())
+        {
+            DeploymentCacheSchema.Apply(connection);
+        }
+
+        db.RegisterReplicated(DeploymentCacheSchema.ArtifactsTable);
+        db.RegisterReplicated(DeploymentCacheSchema.PointerTable);
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs

@@ -0,0 +1,162 @@
+using System.Security.Cryptography;
+using System.Text;
+using Grpc.Core;
+using Grpc.Core.Interceptors;
+using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.LocalDb.Replication;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     Gates the LocalDb passive sync endpoint. The replication library deliberately leaves inbound
+///     authentication to the host — its <c>LocalDbSyncService</c> verifies nothing — so without this
+///     interceptor anything that can reach a driver node's sync port could stream arbitrary rows
+///     into that node's deployment-artifact cache.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>Why that matters here specifically.</b> The cached artifact is what a driver node
+///         boots from when central SQL is unreachable. An attacker able to write it could choose
+///         the configuration a node comes up on during exactly the outage nobody is watching.
+///     </para>
+///     <para>
+///         <b>Scoped by method path.</b> Only calls under <c>/localdb_sync.v1.LocalDbSync/</c> are
+///         gated; every other method on the shared gRPC pipeline passes through untouched.
+///     </para>
+///     <para>
+///         <b>Fail-closed.</b> With no <c>LocalDb:Replication:ApiKey</c> configured, NO sync stream
+///         is accepted, authenticated or not. That is the deliberate choice: the alternative —
+///         treating "no key" as "no auth required" — would silently expose the endpoint on exactly
+///         the default configuration every node ships with. An operator enabling replication must
+///         set the same key on both nodes of the pair, which is already required for the initiator
+///         to dial out (<c>SyncBackgroundService</c> sends <c>Authorization: Bearer &lt;key&gt;</c>).
+///         A key typo therefore stops convergence outright rather than degrading to unauthenticated.
+///     </para>
+///     <para>
+///         Comparison is <see cref="CryptographicOperations.FixedTimeEquals"/> over UTF-8 bytes, so
+///         a wrong key cannot be recovered byte-by-byte from response timing. Length differences are
+///         unavoidably observable and are not sensitive.
+///     </para>
+/// </remarks>
+public sealed class LocalDbSyncAuthInterceptor : Interceptor
+{
+    private const string ServicePrefix = "/localdb_sync.v1.LocalDbSync/";
+    private const string AuthorizationHeader = "authorization";
+    private const string BearerPrefix = "Bearer ";
+
+    private readonly IOptions<ReplicationOptions> _options;
+    private readonly ILogger<LocalDbSyncAuthInterceptor> _logger;
+
+    /// <summary>Creates the interceptor.</summary>
+    /// <param name="options">Replication options; <c>ApiKey</c> is the expected bearer token.</param>
+    /// <param name="logger">Logger for denial diagnostics.</param>
+    public LocalDbSyncAuthInterceptor(
+        IOptions<ReplicationOptions> options,
+        ILogger<LocalDbSyncAuthInterceptor> logger)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+        ArgumentNullException.ThrowIfNull(logger);
+
+        _options = options;
+        _logger = logger;
+    }
+
+    /// <inheritdoc />
+    public override Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
+        TRequest request,
+        ServerCallContext context,
+        UnaryServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(request, context);
+    }
+
+    /// <inheritdoc />
+    public override Task DuplexStreamingServerHandler<TRequest, TResponse>(
+        IAsyncStreamReader<TRequest> requestStream,
+        IServerStreamWriter<TResponse> responseStream,
+        ServerCallContext context,
+        DuplexStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(requestStream, responseStream, context);
+    }
+
+    /// <inheritdoc />
+    public override Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
+        IAsyncStreamReader<TRequest> requestStream,
+        ServerCallContext context,
+        ClientStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(requestStream, context);
+    }
+
+    /// <inheritdoc />
+    public override Task ServerStreamingServerHandler<TRequest, TResponse>(
+        TRequest request,
+        IServerStreamWriter<TResponse> responseStream,
+        ServerCallContext context,
+        ServerStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(request, responseStream, context);
+    }
+
+    /// <summary>
+    ///     Throws <see cref="RpcException"/> with <see cref="StatusCode.PermissionDenied"/> if this
+    ///     is a sync call that does not carry the configured bearer token. Non-sync calls return
+    ///     immediately.
+    /// </summary>
+    private void Authorize(ServerCallContext context)
+    {
+        if (!context.Method.StartsWith(ServicePrefix, StringComparison.Ordinal))
+            return;
+
+        var expected = _options.Value.ApiKey;
+        if (string.IsNullOrEmpty(expected))
+        {
+            _logger.LogWarning(
+                "Rejected a LocalDb sync call to {Method}: no LocalDb:Replication:ApiKey is configured, " +
+                "so the passive sync endpoint is closed. Configure the same key on both nodes of the pair.",
+                context.Method);
+            throw new RpcException(new Status(
+                StatusCode.PermissionDenied,
+                "LocalDb sync is not accepting connections: no API key is configured on this node."));
+        }
+
+        var presented = ExtractBearerToken(context.RequestHeaders);
+        if (presented is null || !FixedTimeEquals(presented, expected))
+        {
+            _logger.LogWarning(
+                "Rejected a LocalDb sync call to {Method}: {Reason}.",
+                context.Method,
+                presented is null ? "no bearer token presented" : "bearer token did not match");
+            throw new RpcException(new Status(
+                StatusCode.PermissionDenied,
+                "LocalDb sync authentication failed."));
+        }
+    }
+
+    private static string? ExtractBearerToken(Metadata headers)
+    {
+        // gRPC lowercases header keys on the wire; compare case-insensitively anyway so a
+        // hand-built Metadata in a test behaves the same as a real request.
+        foreach (var entry in headers)
+        {
+            if (!string.Equals(entry.Key, AuthorizationHeader, StringComparison.OrdinalIgnoreCase))
+                continue;
+
+            var value = entry.Value;
+            if (value is not null && value.StartsWith(BearerPrefix, StringComparison.OrdinalIgnoreCase))
+                return value[BearerPrefix.Length..];
+        }
+
+        return null;
+    }
+
+    private static bool FixedTimeEquals(string presented, string expected)
+        => CryptographicOperations.FixedTimeEquals(
+            Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretReplicationStarter.cs

@@ -0,0 +1,55 @@
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Hosting;
+using ZB.MOM.WW.Secrets.Abstractions;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     Forces this node's secret-replication actor into existence at startup.
+/// </summary>
+/// <remarks>
+///     <para>
+///         The replication actor is created <b>lazily</b>, on the first resolution of
+///         <see cref="ISecretStore"/> — resolving the store builds <c>ReplicatingSecretStore</c>,
+///         which takes <c>ISecretReplicator</c>, whose factory touches
+///         <c>SecretReplicationActorProvider.ActorRef</c> and thereby spawns the actor.
+///     </para>
+///     <para>
+///         Laziness is correct for the library (the <c>ActorSystem</c> is usually registered after
+///         the replication call), but it makes participation in anti-entropy accidental: a node that
+///         happens never to read or write a secret — a plausible steady state for a driver-role node
+///         whose driver configs carry no <c>${secret:}</c> references — would never create the actor,
+///         never announce its manifest, and never converge. Nothing would fail; it would just
+///         silently not replicate. Resolving the store once at startup makes participation
+///         unconditional.
+///     </para>
+///     <para>
+///         <b>History — this hook has met two library defects, both fixed.</b> Against 0.2.0 it was
+///         inert: the package never bound its own <c>ISecretReplicator</c> (TryAdd registration
+///         order), so this resolve built a <c>ReplicatingSecretStore</c> around a no-op sink and
+///         spawned no actor. Against 0.2.1 it was worse: the resolve <b>deadlocked the host at
+///         startup</b> — the package's DI graph had a circular singleton dependency, invisible to
+///         the container through factory lambdas (scadaproj#1). Fixed in 0.2.2 by deferring the
+///         cycle-closing invalidator edge;
+///         <c>SecretsReplicationRegistrationTests.The_startup_hook_actually_creates_the_replication_actor</c>
+///         exercises this hook against a built provider on a real single-node cluster.
+///     </para>
+/// </remarks>
+/// <param name="services">Root provider used to resolve the (decorated) secret store exactly once.</param>
+public sealed class SecretReplicationStarter(IServiceProvider services) : IHostedService
+{
+    /// <summary>Resolves <see cref="ISecretStore"/>, which spawns the replication actor.</summary>
+    /// <param name="cancellationToken">Unused — resolution is synchronous and non-blocking.</param>
+    /// <returns>A completed task.</returns>
+    public Task StartAsync(CancellationToken cancellationToken)
+    {
+        // The resolution itself is the side effect; the instance is deliberately unused.
+        _ = services.GetRequiredService<ISecretStore>();
+        return Task.CompletedTask;
+    }
+
+    /// <summary>No-op: the actor's lifetime is the actor system's.</summary>
+    /// <param name="cancellationToken">Unused.</param>
+    /// <returns>A completed task.</returns>
+    public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs

@@ -0,0 +1,82 @@
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Hosting;
+using ZB.MOM.WW.Secrets.DependencyInjection;
+using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet.DependencyInjection;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     Single registration point for the secrets subsystem on every OtOpcUa node, admin- and
+///     driver-role alike. Exists as a named extension rather than an inline <c>AddZbSecrets</c> call
+///     in <c>Program.cs</c> so the choice of <c>ISecretStore</c> implementation is covered by a DI
+///     resolution test — <c>Program.cs</c> is top-level statements and cannot be exercised directly,
+///     which is exactly how a "registered but never resolvable" wiring defect ships unnoticed.
+/// </summary>
+public static class SecretsRegistration
+{
+    /// <summary>Configuration key gating cluster secret replication. Absent or false = off.</summary>
+    public const string ReplicationEnabledKey = "Secrets:Replication:Enabled";
+
+    /// <summary>Configuration section holding <c>AkkaSecretsReplicationOptions</c>.</summary>
+    public const string ReplicationSectionPath = "Secrets:Replication";
+
+    /// <summary>Configuration section holding the core secrets options.</summary>
+    public const string SecretsSectionPath = "Secrets";
+
+    /// <summary>
+    ///     True when cluster secret replication is switched on in configuration. Defaults to false —
+    ///     absent configuration must mean "off".
+    /// </summary>
+    /// <param name="configuration">The application configuration.</param>
+    /// <returns><c>true</c> when replication is enabled.</returns>
+    public static bool IsReplicationEnabled(IConfiguration configuration)
+    {
+        ArgumentNullException.ThrowIfNull(configuration);
+        return configuration.GetValue(ReplicationEnabledKey, defaultValue: false);
+    }
+
+    /// <summary>
+    ///     Registers the secrets subsystem. Replication is <b>opt-in</b>: unless
+    ///     <see cref="ReplicationEnabledKey"/> is true this is exactly the plain local SQLite wiring
+    ///     the host has always used.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         The gate is deliberately default-deny. This call decides which <c>ISecretStore</c> the
+    ///         container hands to <b>every</b> node, including driver-role nodes with no auth or
+    ///         AdminUI surface, where a bad store surfaces as drivers failing to open sessions rather
+    ///         than as a failing test. Enabling replication by default would change secret resolution
+    ///         across a production fleet with nobody having asked for it.
+    ///     </para>
+    ///     <para>
+    ///         <c>AddZbSecretsAkkaReplication</c> calls <c>AddZbSecrets</c> internally, so the two
+    ///         branches are alternatives, never additive — calling both would double-register.
+    ///     </para>
+    /// </remarks>
+    /// <param name="services">The service collection to add to.</param>
+    /// <param name="configuration">The application configuration.</param>
+    /// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
+    public static IServiceCollection AddOtOpcUaSecrets(
+        this IServiceCollection services,
+        IConfiguration configuration)
+    {
+        ArgumentNullException.ThrowIfNull(services);
+        ArgumentNullException.ThrowIfNull(configuration);
+
+        if (!IsReplicationEnabled(configuration))
+        {
+            services.AddZbSecrets(configuration, SecretsSectionPath);
+            return services;
+        }
+
+        services.AddZbSecretsAkkaReplication(configuration, SecretsSectionPath, ReplicationSectionPath);
+
+        // Anti-entropy participation must not hinge on this node happening to touch a secret — see
+        // SecretReplicationStarter for why the library's lazy actor creation is insufficient here.
+        services.AddSingleton<SecretReplicationStarter>();
+        services.AddHostedService(sp => sp.GetRequiredService<SecretReplicationStarter>());
+
+        return services;
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs

@@ -4,6 +4,7 @@ using Microsoft.Extensions.Logging;
 using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
 using ZB.MOM.WW.OtOpcUa.Core.Hosting;
 using ZB.MOM.WW.OtOpcUa.Core.Resilience;
+using ZB.MOM.WW.Secrets.Abstractions;
 
 namespace ZB.MOM.WW.OtOpcUa.Host.Drivers;
 
@@ -48,7 +49,10 @@ public static class DriverFactoryBootstrap
             // The calc driver needs the root script logger so a script failure fans out onto the
             // script-logs topic; resolve it here (null on nodes without the script pipeline wired).
             var scriptRoot = sp.GetService<ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptRootLogger>();
-            Register(registry, loggerFactory, scriptRoot);
+            // The Galaxy driver resolves its Gateway.ApiKeySecretRef secret: arm through the
+            // shared ISecretResolver (registered unconditionally by AddZbSecrets on every node).
+            var secretResolver = sp.GetRequiredService<ISecretResolver>();
+            Register(registry, loggerFactory, secretResolver, scriptRoot);
             return registry;
         });
         services.AddSingleton<IDriverFactory>(sp =>
@@ -131,15 +135,16 @@ public static class DriverFactoryBootstrap
     private static void Register(
         DriverFactoryRegistry registry,
         ILoggerFactory? loggerFactory,
+        ISecretResolver secretResolver,
         ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptRootLogger? scriptRoot = null)
     {
         Driver.AbCip.AbCipDriverFactoryExtensions.Register(registry);
         Driver.AbLegacy.AbLegacyDriverFactoryExtensions.Register(registry, loggerFactory);
         Driver.Calculation.CalculationDriverFactoryExtensions.Register(registry, loggerFactory, scriptRoot);
         Driver.FOCAS.FocasDriverFactoryExtensions.Register(registry);
-        Driver.Galaxy.GalaxyDriverFactoryExtensions.Register(registry, loggerFactory);
+        Driver.Galaxy.GalaxyDriverFactoryExtensions.Register(registry, secretResolver, loggerFactory);
         Driver.Modbus.ModbusDriverFactoryExtensions.Register(registry, loggerFactory);
-        Driver.OpcUaClient.OpcUaClientDriverFactoryExtensions.Register(registry, loggerFactory);
+        Driver.OpcUaClient.OpcUaClientDriverFactoryExtensions.Register(registry, loggerFactory, secretResolver);
         Driver.S7.S7DriverFactoryExtensions.Register(registry);
         Driver.TwinCAT.TwinCATDriverFactoryExtensions.Register(registry);
     }

src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs

@@ -1,10 +1,15 @@
 using Microsoft.AspNetCore.Routing;
 using Microsoft.EntityFrameworkCore;
+using Microsoft.Extensions.Configuration;
 using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Diagnostics.HealthChecks;
+using Microsoft.Extensions.Options;
 using ZB.MOM.WW.Health;
 using ZB.MOM.WW.Health.Akka;
 using ZB.MOM.WW.Health.EntityFrameworkCore;
+using ZB.MOM.WW.LocalDb.Replication;
 using ZB.MOM.WW.OtOpcUa.Configuration;
+using ZB.MOM.WW.OtOpcUa.Host.Configuration;
 
 namespace ZB.MOM.WW.OtOpcUa.Host.Health;
 
@@ -36,7 +41,20 @@ public static class HealthEndpoints
                 "admin-leader",
                 failureStatus: null,
                 tags: new[] { ZbHealthTags.Active },
-                args: "admin");
+                args: "admin")
+            // Registered unconditionally, not driver-gated. AddOtOpcUaHealth takes no role argument
+            // and runs on every node; the check itself resolves ISyncStatus optionally and reports
+            // Healthy when LocalDb is absent (admin-only graphs) or replication is default-OFF, so a
+            // plain node is never degraded by it. A factory registration keeps this no-arg signature
+            // while still reading ISyncStatus + options + the sync port from the container.
+            .Add(new HealthCheckRegistration(
+                "localdb-replication",
+                sp => new LocalDbReplicationHealthCheck(
+                    sp.GetService<ISyncStatus>(),
+                    sp.GetRequiredService<IOptions<ReplicationOptions>>(),
+                    LocalDbRegistration.SyncListenPort(sp.GetRequiredService<IConfiguration>())),
+                failureStatus: null,
+                tags: new[] { ZbHealthTags.Active }));
         return services;
     }
 

src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/LocalDbReplicationHealthCheck.cs

@@ -0,0 +1,126 @@
+using Microsoft.Extensions.Diagnostics.HealthChecks;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.LocalDb.Replication;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Health;
+
+/// <summary>
+///     Pure decision function for the LocalDb replication probe, factored out of
+///     <see cref="LocalDbReplicationHealthCheck"/> so the whole matrix is table-testable without
+///     standing up a replication engine.
+/// </summary>
+public static class LocalDbReplicationDecision
+{
+    /// <summary>
+    ///     Maps the resolved replication facts to a health status.
+    /// </summary>
+    /// <param name="peerConfigured">
+    ///     True when this node participates in replication at all — it has a peer address to dial, or
+    ///     a sync listener bound. False means default-OFF, and default-OFF must never degrade a node.
+    /// </param>
+    /// <param name="connected">Whether a sync session is currently running.</param>
+    /// <param name="oplogBacklog">
+    ///     The unacked oplog backlog, or <see langword="null"/> when it could not be read. Null is
+    ///     deliberately not treated as zero: a failed poll is "unknown", not "caught up".
+    /// </param>
+    /// <param name="degradedThreshold">
+    ///     Backlog at or above which a connected pair is judged to be falling behind.
+    /// </param>
+    /// <returns>The status plus a human-readable reason.</returns>
+    public static (HealthStatus Status, string Description) Evaluate(
+        bool peerConfigured, bool connected, long? oplogBacklog, long degradedThreshold)
+    {
+        if (!peerConfigured)
+            return (HealthStatus.Healthy, "LocalDb replication is not configured on this node (default-OFF).");
+
+        if (!connected)
+            return (HealthStatus.Degraded, "LocalDb replication is configured but no sync session is connected.");
+
+        if (oplogBacklog is null)
+            return (HealthStatus.Degraded, "LocalDb replication is connected but its oplog backlog is unknown (the poll failed).");
+
+        if (oplogBacklog.Value >= degradedThreshold)
+            return (HealthStatus.Degraded,
+                $"LocalDb replication is connected but its oplog backlog ({oplogBacklog.Value}) is at or above the degraded threshold ({degradedThreshold}).");
+
+        return (HealthStatus.Healthy, $"LocalDb replication is connected; oplog backlog {oplogBacklog.Value}.");
+    }
+}
+
+/// <summary>
+///     Reports the health of this node's LocalDb replication link. Registered unconditionally with
+///     the shared health pipeline; when LocalDb is not present at all (admin-only graphs) it reports
+///     <see cref="HealthStatus.Healthy"/> rather than degrading — matching the
+///     <c>ActiveNodeHealthCheck</c> precedent of "not applicable ⇒ Healthy".
+/// </summary>
+/// <remarks>
+///     A node running LocalDb with replication switched off (no peer, no listener) is also Healthy:
+///     that is the default posture for most of the fleet, and it must not look degraded. Only a node
+///     that is <i>meant</i> to be replicating and is not — or is connected but cannot confirm it is
+///     draining — is surfaced as Degraded. It never reports Unhealthy: a replication problem does not
+///     stop the node serving its address space, so it must not fail a readiness gate.
+/// </remarks>
+public sealed class LocalDbReplicationHealthCheck : IHealthCheck
+{
+    /// <summary>
+    ///     Backlog at or above which a connected pair is reported Degraded. Well below the library's
+    ///     <c>MaxOplogRows</c> default (1,000,000, where a snapshot resync is forced), so the probe
+    ///     flags a pair that is falling behind long before the engine itself intervenes.
+    /// </summary>
+    public const long DefaultBacklogDegradedThreshold = 100_000;
+
+    private readonly ISyncStatus? _syncStatus;
+    private readonly ReplicationOptions _options;
+    private readonly int _syncListenPort;
+    private readonly long _degradedThreshold;
+
+    /// <summary>Creates the health check.</summary>
+    /// <param name="syncStatus">
+    ///     The replication status singleton, or <see langword="null"/> when the replication engine is
+    ///     not registered (admin-only nodes) — in which case the check is a Healthy no-op.
+    /// </param>
+    /// <param name="options">The bound replication options (peer address, etc.).</param>
+    /// <param name="syncListenPort">
+    ///     The configured sync listener port; a value greater than zero counts as "replication
+    ///     configured" even when this node is the passive side with no peer address.
+    /// </param>
+    /// <param name="degradedThreshold">Backlog degraded threshold; defaults to <see cref="DefaultBacklogDegradedThreshold"/>.</param>
+    public LocalDbReplicationHealthCheck(
+        ISyncStatus? syncStatus,
+        IOptions<ReplicationOptions> options,
+        int syncListenPort,
+        long degradedThreshold = DefaultBacklogDegradedThreshold)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+
+        _syncStatus = syncStatus;
+        _options = options.Value;
+        _syncListenPort = syncListenPort;
+        _degradedThreshold = degradedThreshold;
+    }
+
+    /// <inheritdoc />
+    public Task<HealthCheckResult> CheckHealthAsync(
+        HealthCheckContext context, CancellationToken cancellationToken = default)
+    {
+        // No replication engine registered at all → not applicable → Healthy. Admin-only nodes and
+        // any graph that did not call AddOtOpcUaLocalDb land here.
+        if (_syncStatus is null)
+            return Task.FromResult(HealthCheckResult.Healthy(
+                "LocalDb replication is not present on this node."));
+
+        var peerConfigured =
+            !string.IsNullOrWhiteSpace(_options.PeerAddress) || _syncListenPort > 0;
+
+        var (status, description) = LocalDbReplicationDecision.Evaluate(
+            peerConfigured, _syncStatus.Connected, _syncStatus.OplogBacklog, _degradedThreshold);
+
+        var result = status switch
+        {
+            HealthStatus.Healthy => HealthCheckResult.Healthy(description),
+            _ => HealthCheckResult.Degraded(description),
+        };
+
+        return Task.FromResult(result);
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/ObservabilityExtensions.cs

@@ -1,4 +1,5 @@
 using Microsoft.Extensions.Configuration;
+using ZB.MOM.WW.LocalDb.Replication;
 using ZB.MOM.WW.OtOpcUa.Commons.Observability;
 using ZB.MOM.WW.Telemetry;
 
@@ -27,7 +28,11 @@ public static class ObservabilityExtensions
         return services.AddZbTelemetry(o =>
         {
             o.ServiceName = "otopcua";
-            o.Meters = [OtOpcUaTelemetry.MeterName];
+            // o.Meters is a STRICT allowlist — ZbTelemetry adds only the named meters, with no
+            // wildcard. The LocalDb replication engine publishes under its own meter name, so
+            // without this entry its localdb.sync.* / localdb.oplog.depth series are silently absent
+            // from /metrics on driver nodes (the exact omission a ScadaBridge live gate caught).
+            o.Meters = [OtOpcUaTelemetry.MeterName, LocalDbMetrics.MeterName];
             o.ActivitySources = [OtOpcUaTelemetry.ActivitySourceName];
             if (Enum.TryParse<ZbExporter>(configuration["OtOpcUa:Telemetry:Exporter"], ignoreCase: true, out var exporter))
                 o.Exporter = exporter;

src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs

@@ -1,7 +1,9 @@
 using Akka.Actor;
 using Akka.Hosting;
+using Microsoft.AspNetCore.Server.Kestrel.Core;
 using Microsoft.Extensions.DependencyInjection.Extensions;
 using Serilog;
+using ZB.MOM.WW.LocalDb.Replication;
 using Serilog.Events;
 using ZB.MOM.WW.OtOpcUa.AdminUI;
 using ZB.MOM.WW.OtOpcUa.AdminUI.Clients;
@@ -37,6 +39,12 @@ using ZB.MOM.WW.OtOpcUa.Security.Ldap;
 using ZB.MOM.WW.Configuration;
 using ZB.MOM.WW.Telemetry.Serilog;
 using ZB.MOM.WW.OtOpcUa.AdminUI.Api;
+using Microsoft.Extensions.DependencyInjection;
+using ZB.MOM.WW.Secrets.Abstractions;
+using ZB.MOM.WW.Secrets.Configuration;
+using ZB.MOM.WW.Secrets.DependencyInjection;
+using ZB.MOM.WW.Secrets.Replicator.AkkaDotNet;
+using ZB.MOM.WW.Secrets.Sqlite;
 
 // Roles drive the entire conditional wiring below — see ZB.MOM.WW.OtOpcUa.Cluster.RoleParser.
 var roles = RoleParser.Parse(Environment.GetEnvironmentVariable("OTOPCUA_ROLES"));
@@ -72,6 +80,27 @@ if (roleSuffix is not null)
     builder.Configuration.AddCommandLine(args);
 }
 
+// Pre-host ${secret:} expansion: rewrites ${secret:name} tokens in the assembled configuration
+// BEFORE the host is built, so every downstream binder/validator sees resolved plaintext.
+// Placement matters: this runs AFTER all config providers are assembled (base appsettings, the
+// role overlay, and the env-vars + command-line re-append above) but BEFORE anything READS a
+// config value (AddZbSerilog's ReadFrom.Configuration, AddOtOpcUaConfigDb, and the first
+// ValidateOnStart bind all follow). With no ${secret:} tokens present this is a no-op pass;
+// it also migrates the secrets SQLite store on startup.
+// ASP0000: DELIBERATE throwaway container — expansion must run before builder.Build(), so it
+// cannot use the app's DI. Fully disposed here; shares no singletons with the host container.
+#pragma warning disable ASP0000
+await using (var secretsProvider = new ServiceCollection()
+    .AddZbSecrets(builder.Configuration, "Secrets")
+    .BuildServiceProvider())
+#pragma warning restore ASP0000
+{
+    await secretsProvider.GetRequiredService<SqliteSecretsStoreMigrator>().MigrateAsync(default);
+    var resolver = secretsProvider.GetRequiredService<ISecretResolver>();
+    await new SecretReferenceExpander(resolver)
+        .ExpandConfigurationAsync((IConfigurationRoot)builder.Configuration, default);
+}
+
 // Anchor the process working directory to the install directory (AppContext.BaseDirectory) so every
 // relative runtime path resolves under the install dir rather than the service's startup CWD. A Windows
 // service starts with CWD=C:\Windows\System32, which otherwise scattered the Serilog rolling-file sink
@@ -147,6 +176,20 @@ if (hasDriver)
         builder.Configuration,
         (_, sp) => GatewayHistorian.CreateAlarmWriter(serverHistorianOptions, sp));
 
+    // Node-local LocalDb: caches the deployed-configuration artifact so this node can boot from
+    // its last-known-good config when central SQL is unreachable, and optionally replicates that
+    // cache to its redundant pair peer. Driver-role only — admin-only nodes have nothing to cache,
+    // and registering here would impose LocalDb:Path-required-or-no-boot on them for nothing.
+    // Storage ships unconditionally; replication stays inert until LocalDb:Replication:PeerAddress
+    // (initiator) / LocalDb:SyncListenPort (listener) are set. See LocalDbRegistration.
+    builder.Services.AddOtOpcUaLocalDb(builder.Configuration);
+
+    // gRPC server plumbing for the passive sync endpoint mapped below. Registered only under
+    // hasDriver so admin-only nodes expose no sync surface at all. The interceptor is the ONLY
+    // inbound auth on that endpoint — the replication library's LocalDbSyncService verifies
+    // nothing — and it fail-closes when no ApiKey is configured.
+    builder.Services.AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>());
+
     // Config-gated server-side HistoryRead backend. When the ServerHistorian section is enabled this
     // overrides the NullHistorianDataSource default from AddOtOpcUaRuntime (last registration wins) with
     // a read-only HistorianGateway-backed data source the node manager's HistoryRead overrides
@@ -287,6 +330,14 @@ if (hasDriver)
 builder.Services.AddAkka("otopcua", (ab, sp) =>
 {
     ab.WithOtOpcUaClusterBootstrap(sp);
+    // Secret-replication wire protocol → its own serializer, merged only when replication is on so a
+    // non-replicating node carries no serializer bindings for messages it will never see. Appended
+    // (Akka.Hosting's fallback merge — the same mode WithOtOpcUaClusterBootstrap uses for the base
+    // config) rather than a raw Config.WithFallback, which would fight the builder's own assembly.
+    // Without this the protocol DTOs would round-trip on Akka's default JSON serializer, leaving the
+    // serializer that carries secret ciphertext an inherited default rather than an explicit choice.
+    if (SecretsRegistration.IsReplicationEnabled(builder.Configuration))
+        ab.AddHocon(AkkaSecretsReplication.SerializationConfig, HoconAddMode.Append);
     if (hasAdmin)
     {
         ab.WithOtOpcUaControlPlaneSingletons();
@@ -323,9 +374,99 @@ if (hasAdmin)
     builder.Services.AddOtOpcUaAdminClients();
 }
 
+// Registered unconditionally: driver-role nodes resolve Layer-B DriverConfig secrets and have no auth/DP/AdminUI.
+// Cluster replication is opt-in behind Secrets:Replication:Enabled (default false) — see SecretsRegistration.
+builder.Services.AddOtOpcUaSecrets(builder.Configuration);
+
 builder.Services.AddOtOpcUaHealth();
 builder.Services.AddOtOpcUaObservability(builder.Configuration);
 
+// ---------------------------------------------------------------------------------------------
+// LocalDb sync listener (default-OFF).
+//
+// DANGER: any explicit Kestrel Listen* call makes Kestrel IGNORE ASPNETCORE_URLS/urls ENTIRELY
+// (it logs "Overriding address(es)"). The host has no ConfigureKestrel today and binds solely via
+// that configuration, so adding a listener naively would silently unbind the AdminUI + deploy API
+// behind Traefik. Everything the host was already asked to serve is therefore re-bound explicitly
+// in the same block.
+//
+// The listener is HTTP/2-ONLY on purpose: the sync client speaks prior-knowledge h2c, which a
+// cleartext Http1AndHttp2 endpoint cannot negotiate (there is no ALPN without TLS). Hence a
+// dedicated port rather than multiplexing onto the main one.
+//
+// When LocalDb:SyncListenPort is 0 (the default) none of this runs and URL binding is untouched.
+// ---------------------------------------------------------------------------------------------
+var syncListenPort = LocalDbRegistration.SyncListenPort(builder.Configuration);
+if (hasDriver && syncListenPort > 0)
+{
+    // Mirror Kestrel's own source precedence: URLS wins; else the HTTP_PORTS/HTTPS_PORTS bare-port
+    // vars; else Kestrel's localhost:5000 default. Missing the HTTP_PORTS leg is not academic — the
+    // aspnet:8.0+ base images set ASPNETCORE_HTTP_PORTS=8080 as the CONTAINER default in place of
+    // ASPNETCORE_URLS, so a driver node that never sets URLS is really serving :8080 on all
+    // interfaces. Falling straight through to localhost:5000 would silently move its health/metrics
+    // surface to loopback (found on the docker-dev live gate).
+    var configuredUrls = builder.Configuration["urls"]
+                      ?? builder.Configuration["ASPNETCORE_URLS"];
+    var existingBindings = KestrelHttpBinding.Parse(configuredUrls);
+
+    if (existingBindings.Count == 0)
+    {
+        var httpPorts = builder.Configuration["ASPNETCORE_HTTP_PORTS"] ?? builder.Configuration["HTTP_PORTS"];
+        var httpsPorts = builder.Configuration["ASPNETCORE_HTTPS_PORTS"] ?? builder.Configuration["HTTPS_PORTS"];
+        existingBindings =
+        [
+            .. KestrelHttpBinding.ParsePorts(httpPorts, isSecure: false),
+            .. KestrelHttpBinding.ParsePorts(httpsPorts, isSecure: true),
+        ];
+        if (existingBindings.Count > 0)
+            Log.Information(
+                "LocalDb sync listener enabled; re-binding the HTTP_PORTS surface ({HttpPorts}/{HttpsPorts}) " +
+                "alongside the sync port.", httpPorts, httpsPorts);
+    }
+
+    // A driver-only Windows-service node gets NO URLS and NO *_PORTS (Install-Services.ps1 sets URLS
+    // only under $hasAdmin), so "nothing configured" is a real, supported state — not an error.
+    // Kestrel's own default is localhost:5000; re-state it explicitly, because taking over
+    // configuration means taking over the default too. Health probes hit this port.
+    if (existingBindings.Count == 0)
+    {
+        existingBindings = [new KestrelHttpBinding("localhost", 5000, IsSecure: false)];
+        Log.Information(
+            "LocalDb sync listener enabled with no URLs configured; re-binding Kestrel's default " +
+            "http://localhost:5000 alongside the sync port.");
+    }
+
+    // Re-binding an https endpoint would need its certificate configuration replayed too, which
+    // this host does not model (TLS is terminated at Traefik). Rather than silently serve it
+    // without TLS — or drop it — refuse to take over Kestrel at all and leave the existing
+    // surface exactly as it was. The operator gets a loud, actionable error instead of an
+    // AdminUI that stopped answering.
+    if (existingBindings.Any(b => b.IsSecure))
+    {
+        Log.Error(
+            "LocalDb:SyncListenPort is set to {Port} but this host serves HTTPS endpoint(s) ({Urls}). " +
+            "Binding the sync listener requires re-binding every existing endpoint explicitly, and the " +
+            "HTTPS certificate configuration cannot be replayed safely. The sync listener is DISABLED; " +
+            "terminate TLS upstream (as the docker-dev rig does) or leave replication off on this node.",
+            syncListenPort, configuredUrls);
+        syncListenPort = 0;
+    }
+    else
+    {
+        builder.WebHost.ConfigureKestrel(kestrel =>
+        {
+            foreach (var binding in existingBindings)
+                binding.Apply(kestrel);
+
+            kestrel.ListenAnyIP(syncListenPort, o => o.Protocols = HttpProtocols.Http2);
+        });
+
+        Log.Information(
+            "LocalDb sync listener bound on :{SyncPort} (h2c); re-bound existing endpoint(s) {Urls}.",
+            syncListenPort, configuredUrls ?? "http://localhost:5000 (Kestrel default)");
+    }
+}
+
 var app = builder.Build();
 
 // AddZbSerilog registers Serilog as the MEL logging provider but does NOT assign the static
@@ -356,6 +497,15 @@ if (hasAdmin)
     app.MapOtOpcUaDeployApi(app.Configuration);
 }
 
+// Passive LocalDb sync endpoint. Gated on the same port check that bound the listener, so a
+// default-OFF or admin-only graph carries no sync surface whatsoever. Mapping it would be safe
+// even unauthenticated — LocalDbSyncAuthInterceptor fail-closes with no ApiKey configured — but
+// not mapping it at all is the stronger default.
+if (hasDriver && syncListenPort > 0)
+{
+    app.MapZbLocalDbSync();
+}
+
 app.MapOtOpcUaHealth();
 app.MapOtOpcUaMetrics();
 

src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj

@@ -33,6 +33,16 @@
         <PackageReference Include="ZB.MOM.WW.Telemetry" />
         <PackageReference Include="ZB.MOM.WW.Telemetry.Serilog" />
         <PackageReference Include="ZB.MOM.WW.Configuration" />
+        <!-- LocalDb: node-local SQLite cache (AddZbLocalDb) plus the replication sync engine and
+             its passive gRPC endpoint (AddZbLocalDbReplication / MapZbLocalDbSync). Grpc.AspNetCore
+             is required because the sync endpoint is served by this host's Kestrel, not dialled by
+             it — Grpc.Net.Client (already pinned) only covers the initiator side. -->
+        <PackageReference Include="Grpc.AspNetCore" />
+        <PackageReference Include="ZB.MOM.WW.LocalDb" />
+        <PackageReference Include="ZB.MOM.WW.LocalDb.Replication" />
+        <PackageReference Include="ZB.MOM.WW.Secrets" />
+        <PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" />
+        <PackageReference Include="ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" />
     </ItemGroup>
 
     <ItemGroup>

src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json

@@ -11,12 +11,27 @@
       "DisableLogin": false
     }
   },
+  "Secrets": {
+    "SqlitePath": "otopcua-secrets.db",
+    "MasterKey": {
+      "Source": "Environment",
+      "EnvVarName": "ZB_SECRETS_MASTER_KEY"
+    },
+    "RunMigrationsOnStartup": true,
+    "ResolveCacheTtl": "00:00:30",
+    "Replication": {
+      "_comment": "Peer-to-peer secret replication over the Akka cluster. OPT-IN: Enabled=false keeps the plain local SQLite store on every node. KNOWN NON-FUNCTIONAL against ZB.MOM.WW.Secrets.Replicator.AkkaDotNet 0.2.0 — that version's ISecretReplicator registration is shadowed by the NoOp one AddZbSecrets registers first, so enabling this decorates the store but replicates nothing and starts no actor. Do not enable until the library is fixed. All nodes must also share the same KEK.",
+      "Enabled": false,
+      "AnnounceInterval": "00:00:30",
+      "ActorName": "zb-secret-replication"
+    }
+  },
   "ServerHistorian": {
     "_comment": "Server-side HistoryRead backend (the ZB.MOM.WW.HistorianGateway gRPC client). Disabled => NullHistorianDataSource (historized nodes return GoodNoData). The gateway must run RuntimeDb:EventReadsEnabled=true for alarm-history ReadEvents, and the API key must carry historian:read + historian:write + historian:tags:write scopes.",
     "Enabled": false,
     "Endpoint": "",
     "ApiKey": "",
-    "_ApiKeyComment": "NEVER commit a real key. Supply via the environment variable ServerHistorian__ApiKey.",
+    "_ApiKeyComment": "NEVER commit a real key. Supply via env var ServerHistorian__ApiKey, either as a literal or as a ${secret:otopcua/historian/api-key} token resolved fail-closed by the pre-host secrets expander.",
     "UseTls": true,
     "AllowUntrustedServerCertificate": false,
     "CaCertificatePath": null,
@@ -44,6 +59,16 @@
     "Capacity": 1000000,
     "DeadLetterRetentionDays": 30
   },
+  "LocalDb": {
+    "_comment": "Node-local SQLite store caching the deployed-configuration artifact, so a driver node can boot from its last-known-good config when central SQL is unreachable. Registered on driver-role nodes only (see LocalDbRegistration). Storage is unconditional; replication to the redundant pair peer is default-OFF.",
+    "Path": "./data/otopcua-localdb.db",
+    "_SyncListenPortComment": "Port for the dedicated cleartext-HTTP/2 (h2c) listener serving the passive sync endpoint. 0 = no listener bound and the endpoint is not mapped. Must be a port distinct from the main HTTP listener: prior-knowledge h2c cannot share a cleartext Http1AndHttp2 port.",
+    "SyncListenPort": 0,
+    "Replication": {
+      "_comment": "Empty = passive/off. Set PeerAddress on exactly ONE node of the pair (the stream is bidirectional, so the other side still pushes its own deltas back). ApiKey must be BYTE-IDENTICAL on both nodes — the host interceptor fail-closes, so a typo silently stops convergence rather than degrading to unauthenticated. Supply it via ${secret:...} in production, never committed.",
+      "_MaxBatchSizeComment": "Row count, NOT bytes, against gRPC's 4 MB default message cap. Artifact chunk rows are ~171 KB base64, so keep this at 16 on the pair (16 x 171 KB ~= 2.7 MB worst case)."
+    }
+  },
   "Deployment": {
     "_comment": "R2-11 (05/CONV-2): deploy-gate TagConfig strictness. Warn (default) = non-blocking warnings logged + appended to the deployment result message; Error = a config with a typo'd enum or unparseable TagConfig is rejected at the draft gate. Running servers are untouched; the gate only sees re-deploys.",
     "TagConfigValidationMode": "Warn"

src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs

@@ -497,6 +497,16 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
                 condition.SetSeverity(SystemContext, MapSeverity(state.Severity));
                 condition.Message.Value = new LocalizedText(state.Message);
 
+                // #477 — project the source-data quality. A Good→Bad bucket change is a genuine condition
+                // change (it's in the delta-gate above), so it fires a Part 9 event carrying the new Quality;
+                // it never touches Active/Acked/Retain (annotation only). Quality is an optional Part 9 child —
+                // null-guard it like Confirmed/Shelving in case a leaner SDK child set omits it.
+                if (condition.Quality is not null)
+                {
+                    condition.Quality.Value = StatusFromQuality(state.Quality);
+                    if (condition.Quality.SourceTimestamp is not null) condition.Quality.SourceTimestamp.Value = sourceTimestampUtc;
+                }
+
                 // Part 9: retain the condition while it is active OR unacknowledged so a client's
                 // ConditionRefresh replays it. The event firing below also depends on this Retain being
                 // correct (a non-retained inactive+acked condition still fires its transition event, but
@@ -535,6 +545,50 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
         }
     }
 
+    /// <summary>
+    ///     #477 Layer 2 — apply a source-data quality annotation to a materialised condition, OUT OF BAND
+    ///     from any alarm transition. Used by the driver-connectivity path (comms lost → Bad, restored →
+    ///     Good) so a native condition whose device is unreachable stops reporting the accidentally-Good
+    ///     default. This sets ONLY <see cref="ConditionState.Quality"/> — it never touches
+    ///     Active / Acked / Severity / Retain (a comms-lost active alarm must stay active). A change in the
+    ///     quality bucket is a genuine Part 9 condition change, so it fires one condition event carrying the
+    ///     new Quality; an unchanged bucket suppresses (no spurious event). Unknown / unmaterialised node ⇒
+    ///     safe no-op (a mid-rebuild race must not fault a connectivity update), mirroring the other writes.
+    /// </summary>
+    /// <param name="alarmNodeId">The condition node id (RawPath for a native alarm).</param>
+    /// <param name="quality">The source-data quality to annotate.</param>
+    /// <param name="sourceTimestampUtc">Timestamp of the connectivity transition in UTC.</param>
+    /// <param name="realm">The condition's address-space realm.</param>
+    public void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
+    {
+        ArgumentException.ThrowIfNullOrEmpty(alarmNodeId);
+        EnsureAddressSpaceCreated();
+
+        var key = MapKey(realm, alarmNodeId);
+        lock (Lock)
+        {
+            // Only materialised conditions carry a Quality child; a bare value variable or a missing node is
+            // a no-op (the connectivity fan-out visits every condition the driver owns, some of which a
+            // concurrent rebuild may have just cleared).
+            if (!_alarmConditions.TryGetValue(key, out var condition) || condition.Quality is null)
+                return;
+
+            var newCode = StatusFromQuality(quality);
+            var changed = condition.Quality.Value.Code != newCode.Code;
+
+            condition.Quality.Value = newCode;
+            if (condition.Quality.SourceTimestamp is not null) condition.Quality.SourceTimestamp.Value = sourceTimestampUtc;
+
+            // Fire ONLY on a real bucket change so a steady-state connectivity re-assert doesn't spam events.
+            if (changed)
+            {
+                condition.Time.Value = sourceTimestampUtc;
+                condition.ReceiveTime.Value = sourceTimestampUtc;
+                ReportConditionEvent(condition, sourceTimestampUtc);
+            }
+        }
+    }
+
     /// <summary>
     ///     Fire a real OPC UA Part 9 condition event for one engine-driven state transition on a
     ///     materialised <see cref="AlarmConditionState"/>. The caller MUST already hold <c>Lock</c> and
@@ -650,7 +704,11 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
         bool Enabled,
         AlarmShelvingKind Shelving,
         ushort MappedSeverity,
-        string Message);
+        string Message,
+        // #477 — the source-data quality bucket. Included so a quality-only transition (e.g. a device
+        // going comms-lost: Good→Bad with no state change) is a genuine delta and fires a Part 9 event.
+        // StatusCode has value equality, so the record struct's == still holds.
+        StatusCode Quality);
 
     /// <summary>Decide whether a <see cref="WriteAlarmCondition"/> projection is a genuine state change
     /// (and so should fire a Part 9 condition event) by comparing the node's pre-projection state to the
@@ -676,7 +734,10 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
         Enabled: condition.EnabledState?.Id?.Value ?? true,
         Shelving: ReadShelvingKind(condition),
         MappedSeverity: condition.Severity?.Value ?? (ushort)0,
-        Message: condition.Message?.Value?.Text ?? string.Empty);
+        Message: condition.Message?.Value?.Text ?? string.Empty,
+        // Optional Part 9 child; a node without it reads as Good (matching the accidentally-Good default
+        // and ToConditionDelta's fold), so a snapshot Quality can't create a phantom delta against it.
+        Quality: condition.Quality?.Value ?? StatusCodes.Good);
 
     /// <summary>Build the gate-relevant slice from the incoming snapshot, normalising the two fields that
     /// the node stores in a derived form: Severity is run through <see cref="MapSeverity"/> so it matches
@@ -694,7 +755,10 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
         // node's read-back default (Unshelved).
         Shelving: condition.ShelvingState is not null ? state.Shelving : AlarmShelvingKind.Unshelved,
         MappedSeverity: (ushort)MapSeverity(state.Severity),
-        Message: state.Message ?? string.Empty);
+        Message: state.Message ?? string.Empty,
+        // If the node has no Quality child, WriteAlarmCondition's projection is a no-op there; fold to the
+        // node's read-back default (Good) so a snapshot Quality can't register a spurious delta.
+        Quality: condition.Quality is not null ? StatusFromQuality(state.Quality) : StatusCodes.Good);
 
     /// <summary>Map the live shelving state machine's CurrentState back to our 3-way
     /// <see cref="AlarmShelvingKind"/> by matching its well-known Part 9 state object id. Any node without
@@ -799,6 +863,48 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
             // (Real-server finding from the T14 integration test — not obvious from the SDK notes.)
             if (alarm.BranchId is not null) alarm.BranchId.Value = NodeId.Null;
 
+            // #473 — the mandatory BaseEventType identity fields. Create() builds these three children from
+            // the type's embedded definition but leaves them UNSET (the init string declares them mandatory
+            // with NO default), and nothing downstream fills them in: the auto-filling
+            // BaseEventState.Initialize(context, source, severity, message) overload is only used for
+            // transient events, and ReportEvent / InstanceStateSnapshot copy the children verbatim. Unset
+            // here ⇒ null on the wire on EVERY condition event, so a client cannot attribute the alarm.
+            //   SourceNode — the condition's OWN NodeId (== ConditionId): the condition IS the source. An
+            //                alarm-bearing raw tag materialises ONLY this condition, with no sibling value
+            //                variable (see AddressSpaceApplier's alarm branch), so there is no other node.
+            //   SourceName — the same identifying id string (RawPath native / ScriptedAlarmId scripted).
+            //                Deliberately the UNIQUE id, NOT the leaf: the leaf collides across devices and
+            //                is already carried by ConditionName below. See docs/AlarmTracking.md.
+            alarm.EventType.Value = alarm.TypeDefinitionId;
+            alarm.SourceNode.Value = alarm.NodeId; // Create() assigned this above; do not rebuild it
+            alarm.SourceName.Value = alarmNodeId;
+
+            // #475 — the mandatory ConditionType classification fields, unset by Create() for the same reason as
+            // the fields above (mandatory, no default, nothing downstream synthesises them) ⇒ NodeId.Null + empty
+            // text on the wire, which buckets every alarm as unclassified in a Part 9 HMI.
+            // BaseConditionClassType is Part 9's "no class modelled" value and is the honest report: we hold no
+            // classification at this seam. Deliberately NOT ProcessConditionClassType (the SDK sample's pick) — it
+            // would assert a classification we cannot back, and would be actively wrong for a Galaxy alarm whose
+            // upstream category is Safety/Diagnostics. Real per-alarm classification needs the driver's
+            // AlarmCategory, which today exists only on the runtime AlarmEventArgs transition and not on the
+            // authored composition this deploy-time seam sees — a separate feature, not a default picked here.
+            if (alarm.ConditionClassId is not null) alarm.ConditionClassId.Value = ObjectTypeIds.BaseConditionClassType;
+            if (alarm.ConditionClassName is not null) alarm.ConditionClassName.Value = new LocalizedText("BaseConditionClass");
+
+            // #477 — ConditionType.Quality (the quality of the condition's source data). Create() leaves it
+            // UNSET, and default(StatusCode) == StatusCodes.Good (0x0), so an unassigned Quality reports Good
+            // unconditionally — a wrong VALUE (not a null), which hides a comms-lost source. A NATIVE condition
+            // has no data yet at materialise (its driver hasn't confirmed connectivity), so it starts
+            // BadWaitingForInitialData — the same "no driver data yet" convention value variables use — and is
+            // driven Good by the driver-connectivity path (DriverHostActor → ProjectQuality) once Connected. A
+            // SCRIPTED condition is script-computed and always live in v1, so it starts Good. Quality is a pure
+            // annotation: it NEVER alters Active/Acked/Retain (a comms-lost active alarm must stay active).
+            if (alarm.Quality is not null)
+            {
+                alarm.Quality.Value = isNative ? StatusCodes.BadWaitingForInitialData : StatusCodes.Good;
+                if (alarm.Quality.SourceTimestamp is not null) alarm.Quality.SourceTimestamp.Value = DateTime.UtcNow;
+            }
+
             // Initial state via the SDK setters (T14: basic state only, NO event firing).
             alarm.SetEnableState(SystemContext, true);
             alarm.SetActiveState(SystemContext, false);

src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/SdkAddressSpaceSink.cs

@@ -28,6 +28,9 @@ public sealed class SdkAddressSpaceSink : IOpcUaAddressSpaceSink, ISurgicalAddre
     public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
         => _nodeManager.WriteAlarmCondition(alarmNodeId, state, sourceTimestampUtc, realm);
 
+    public void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
+        => _nodeManager.WriteAlarmQuality(alarmNodeId, quality, sourceTimestampUtc, realm);
+
     /// <inheritdoc />
     public void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, AddressSpaceRealm realm, bool isNative = false)
         => _nodeManager.MaterialiseAlarmCondition(alarmNodeId, equipmentNodeId, displayName, alarmType, severity, realm, isNative);

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/DeploymentCacheSchema.cs

@@ -0,0 +1,70 @@
+using Microsoft.Data.Sqlite;
+
+namespace ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
+
+/// <summary>
+///     DDL for the node-local deployment-artifact cache: the chunked artifact table plus the
+///     per-cluster current-deployment pointer.
+/// </summary>
+/// <remarks>
+///     <para>
+///         Deliberately depends on nothing but <see cref="SqliteConnection"/> so it can be applied
+///         to any connection — the host's <c>LocalDbSetup.OnReady</c> in production, and a bare
+///         connection in a test — without dragging the DI graph along.
+///     </para>
+///     <para>
+///         <b>Why the artifact is chunked base64 TEXT and not one BLOB row.</b> Two independent
+///         constraints force it. <c>RegisterReplicated</c> rejects BLOB columns outright; and the
+///         replication engine batches by <i>row count</i> against gRPC's 4 MB default message cap,
+///         so a single multi-megabyte artifact row would simply never be deliverable — at any
+///         batch size. A 128 KiB raw chunk is ≈ 171 KB once base64-encoded, which keeps a
+///         worst-case batch comfortably inside the cap.
+///     </para>
+///     <para>
+///         <b>Why no autoincrement PKs.</b> Convergence is last-writer-wins keyed on the primary
+///         key. Two nodes independently allocating rowid 7 for different rows would silently
+///         overwrite one another. Every key here is TEXT or a composite of caller-supplied values.
+///     </para>
+/// </remarks>
+public static class DeploymentCacheSchema
+{
+    /// <summary>Table holding the artifact bytes, split into base64 chunks.</summary>
+    public const string ArtifactsTable = "deployment_artifacts";
+
+    /// <summary>Table holding one current-deployment pointer per cluster.</summary>
+    public const string PointerTable = "deployment_pointer";
+
+    /// <summary>
+    ///     Creates both cache tables if they do not already exist. Idempotent.
+    /// </summary>
+    /// <param name="connection">
+    ///     An already-open connection. <c>ILocalDb.CreateConnection()</c> hands out open,
+    ///     pragma-configured connections — do not call <c>Open()</c> on one.
+    /// </param>
+    public static void Apply(SqliteConnection connection)
+    {
+        ArgumentNullException.ThrowIfNull(connection);
+
+        using var cmd = connection.CreateCommand();
+        cmd.CommandText = """
+            CREATE TABLE IF NOT EXISTS deployment_artifacts (
+                deployment_id   TEXT    NOT NULL,
+                chunk_index     INTEGER NOT NULL,
+                cluster_id      TEXT    NOT NULL,
+                revision_hash   TEXT    NOT NULL,
+                chunk_count     INTEGER NOT NULL,
+                chunk_base64    TEXT    NOT NULL,
+                cached_at_utc   TEXT    NOT NULL,
+                PRIMARY KEY (deployment_id, chunk_index)
+            );
+            CREATE TABLE IF NOT EXISTS deployment_pointer (
+                cluster_id        TEXT NOT NULL PRIMARY KEY,
+                deployment_id     TEXT NOT NULL,
+                revision_hash     TEXT NOT NULL,
+                artifact_sha256   TEXT NOT NULL,
+                applied_at_utc    TEXT NOT NULL
+            );
+            """;
+        cmd.ExecuteNonQuery();
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactCache.cs

@@ -0,0 +1,68 @@
+namespace ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
+
+/// <summary>
+///     Node-local durable cache of the deployment artifact this node last applied.
+/// </summary>
+/// <remarks>
+///     <para>
+///         Exists so a driver node can boot into its last-known-good address space when the central
+///         configuration database is unreachable. Without it, a control-plane outage that outlives a
+///         node restart leaves that node with no address space at all — the plant loses visibility
+///         for a reason that has nothing to do with the plant.
+///     </para>
+/// </remarks>
+public interface IDeploymentArtifactCache
+{
+    /// <summary>
+    ///     Persists <paramref name="artifact"/> as the cluster's current deployment.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         Idempotent per <paramref name="deploymentId"/>: re-storing the same deployment
+    ///         replaces its chunks rather than accumulating orphans. Retention is bounded to the two
+    ///         newest deployments per cluster, so a long-lived node cannot grow its local database
+    ///         without limit.
+    ///     </para>
+    /// </remarks>
+    Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
+                    byte[] artifact, CancellationToken ct = default);
+
+    /// <summary>
+    ///     Reads the cached artifact for <paramref name="clusterId"/>, or <see langword="null"/>
+    ///     when nothing is cached or the cached bytes fail their integrity check.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         A failed integrity check is deliberately indistinguishable from a miss. Booting a
+    ///         plant from a truncated address space is strictly worse than booting from none: the
+    ///         missing half looks like a deliberate configuration rather than an error.
+    ///     </para>
+    /// </remarks>
+    Task<CachedDeploymentArtifact?> GetCurrentAsync(string clusterId, CancellationToken ct = default);
+
+    /// <summary>
+    ///     Reads the single cached pointer without knowing the cluster id.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         The boot seam needs this. A node's cluster id is only derivable from an artifact it
+    ///         has already loaded, or from the central database — which is precisely what is
+    ///         unreachable in the scenario this cache exists to survive. So the cold path reads the
+    ///         newest pointer unkeyed.
+    ///     </para>
+    ///     <para>
+    ///         More than one pointer row means the node was re-homed between clusters. The newest
+    ///         wins, but the choice is logged as a warning naming both clusters — a node silently
+    ///         booting a neighbouring cluster's configuration is exactly the failure that must never
+    ///         be quiet.
+    ///     </para>
+    /// </remarks>
+    Task<CachedDeploymentArtifact?> GetCurrentUnkeyedAsync(CancellationToken ct = default);
+}
+
+/// <summary>
+///     A deployment artifact recovered from the node-local cache, with the metadata needed to decide
+///     whether it is still current once the control plane is reachable again.
+/// </summary>
+public sealed record CachedDeploymentArtifact(
+    string DeploymentId, string RevisionHash, byte[] Artifact, DateTimeOffset AppliedAtUtc);

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/LocalDbDeploymentArtifactCache.cs

@@ -0,0 +1,400 @@
+using System.Globalization;
+using System.Security.Cryptography;
+using Microsoft.Extensions.Logging;
+using ZB.MOM.WW.LocalDb;
+
+namespace ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
+
+/// <summary>
+///     <see cref="IDeploymentArtifactCache"/> backed by the replicated LocalDb deployment-cache
+///     tables.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>Why the artifact is stored as base64 chunks.</b> See
+///         <see cref="DeploymentCacheSchema"/> — replication rejects BLOB columns and batches by row
+///         count against gRPC's message cap, so a whole-artifact row could never be delivered.
+///         Everything in this class that looks like ceremony (chunk indices, a chunk count, a
+///         separate SHA over the raw bytes) exists to make that split safely reversible.
+///     </para>
+/// </remarks>
+public sealed class LocalDbDeploymentArtifactCache : IDeploymentArtifactCache
+{
+    /// <summary>
+    ///     Raw bytes per chunk, before base64 expansion.
+    /// </summary>
+    /// <remarks>
+    ///     128 KiB raw is ≈ 171 KB encoded, which keeps a worst-case replication batch inside gRPC's
+    ///     4 MB default cap with room to spare.
+    /// </remarks>
+    private const int ChunkSize = 128 * 1024;
+
+    /// <summary>
+    ///     How many deployments per cluster survive a store.
+    /// </summary>
+    /// <remarks>
+    ///     Two, not one: the previous artifact is what a rollback would need, and keeping it costs
+    ///     one artifact's worth of disk. Three would only add a generation nobody rolls back to.
+    /// </remarks>
+    private const int RetainedDeployments = 2;
+
+    private readonly ILocalDb _db;
+    private readonly ILogger<LocalDbDeploymentArtifactCache> _logger;
+
+    public LocalDbDeploymentArtifactCache(ILocalDb db, ILogger<LocalDbDeploymentArtifactCache> logger)
+    {
+        ArgumentNullException.ThrowIfNull(db);
+        ArgumentNullException.ThrowIfNull(logger);
+
+        _db = db;
+        _logger = logger;
+    }
+
+    /// <inheritdoc/>
+    public async Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
+                                 byte[] artifact, CancellationToken ct = default)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
+        ArgumentException.ThrowIfNullOrWhiteSpace(deploymentId);
+        ArgumentException.ThrowIfNullOrWhiteSpace(revisionHash);
+        ArgumentNullException.ThrowIfNull(artifact);
+
+        var sha = Convert.ToHexString(SHA256.HashData(artifact));
+        var cachedAtUtc = FormatTimestamp(DateTimeOffset.UtcNow);
+        var chunkCount = (artifact.Length + ChunkSize - 1) / ChunkSize;
+
+        if (await IsAlreadyCachedAsync(clusterId, deploymentId, revisionHash, sha, chunkCount, ct))
+        {
+            _logger.LogDebug(
+                "Deployment {DeploymentId} (revision {RevisionHash}) is already cached intact for cluster {ClusterId}; skipping the re-store.",
+                deploymentId, revisionHash, clusterId);
+            return;
+        }
+
+        // One transaction for the whole store. A pointer that commits without its chunks — or
+        // chunks that commit without the pointer — is a cache that reads back as a corrupt hit
+        // rather than a clean miss, which is the one outcome this type exists to prevent.
+        await using var tx = await _db.BeginTransactionAsync(ct);
+
+        // Delete-then-insert rather than upsert-per-chunk: a re-store with FEWER chunks than last
+        // time would otherwise leave the old tail behind, and those orphans read back as a
+        // chunk-count mismatch forever.
+        await tx.ExecuteAsync(
+            "DELETE FROM deployment_artifacts WHERE deployment_id = @DeploymentId",
+            new { DeploymentId = deploymentId },
+            ct);
+
+        for (var index = 0; index < chunkCount; index++)
+        {
+            var offset = index * ChunkSize;
+            var length = Math.Min(ChunkSize, artifact.Length - offset);
+
+            await tx.ExecuteAsync(
+                """
+                INSERT INTO deployment_artifacts
+                    (deployment_id, chunk_index, cluster_id, revision_hash, chunk_count,
+                     chunk_base64, cached_at_utc)
+                VALUES
+                    (@DeploymentId, @ChunkIndex, @ClusterId, @RevisionHash, @ChunkCount,
+                     @ChunkBase64, @CachedAtUtc)
+                """,
+                new
+                {
+                    DeploymentId = deploymentId,
+                    ChunkIndex = index,
+                    ClusterId = clusterId,
+                    RevisionHash = revisionHash,
+                    ChunkCount = chunkCount,
+                    ChunkBase64 = Convert.ToBase64String(artifact, offset, length),
+                    CachedAtUtc = cachedAtUtc,
+                },
+                ct);
+        }
+
+        await tx.ExecuteAsync(
+            """
+            INSERT INTO deployment_pointer
+                (cluster_id, deployment_id, revision_hash, artifact_sha256, applied_at_utc)
+            VALUES
+                (@ClusterId, @DeploymentId, @RevisionHash, @Sha, @AppliedAtUtc)
+            ON CONFLICT(cluster_id) DO UPDATE SET
+                deployment_id   = excluded.deployment_id,
+                revision_hash   = excluded.revision_hash,
+                artifact_sha256 = excluded.artifact_sha256,
+                applied_at_utc  = excluded.applied_at_utc
+            """,
+            new
+            {
+                ClusterId = clusterId,
+                DeploymentId = deploymentId,
+                RevisionHash = revisionHash,
+                Sha = sha,
+                AppliedAtUtc = cachedAtUtc,
+            },
+            ct);
+
+        // Prune inside the same transaction so the cache is never briefly unbounded.
+        //
+        // The `deployment_id <> @DeploymentId` clause is load-bearing, not belt-and-braces: it makes
+        // "the deployment the pointer names is always present" a structural invariant instead of a
+        // consequence of clock resolution. Without it, three stores landing inside one timestamp
+        // tick fall through to the `deployment_id DESC` tiebreak — and since real deployment ids are
+        // GUIDs, that ordering is effectively random, so the row just written can lose. The pointer
+        // would then name chunks that no longer exist, which reads back as a cache miss on every
+        // subsequent boot until the next deploy overwrites it. Silent and permanent: exactly the
+        // failure this cache exists to prevent.
+        await tx.ExecuteAsync(
+            $"""
+             DELETE FROM deployment_artifacts
+             WHERE cluster_id = @ClusterId
+               AND deployment_id <> @DeploymentId
+               AND deployment_id NOT IN (
+                   SELECT deployment_id
+                   FROM deployment_artifacts
+                   WHERE cluster_id = @ClusterId
+                   GROUP BY deployment_id
+                   ORDER BY MAX(cached_at_utc) DESC, deployment_id DESC
+                   LIMIT {RetainedDeployments}
+               )
+             """,
+            new { ClusterId = clusterId, DeploymentId = deploymentId },
+            ct);
+
+        await tx.CommitAsync(ct);
+    }
+
+    /// <summary>
+    ///     Whether this exact artifact is already cached <b>and readable</b> for the cluster, so
+    ///     storing it again would rewrite every row to its current value.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b>Why this exists.</b> Every write to a replicated table fires a capture trigger and
+    ///         mints an oplog row. Re-caching an artifact the node already holds — which is what a
+    ///         restart's boot-from-cache and every <c>RestoreApplied</c> recovery does — would
+    ///         therefore cost a delete plus an insert per chunk plus a pointer update, all carrying
+    ///         values identical to what is already there. On a replicating pair that is pure churn;
+    ///         on an unpaired (default-OFF) node there is no peer to ack those rows, so they
+    ///         accumulate in the oplog until the library's age cap prunes them. The Phase 1 live gate
+    ///         saw exactly that (check 8).
+    ///     </para>
+    ///     <para>
+    ///         <b>Identity is over the bytes, not the ids.</b> The pointer must match on
+    ///         <c>artifact_sha256</c> as well as the deployment id and revision hash: a re-composed
+    ///         artifact can legitimately carry the same ids with different bytes, and keeping the
+    ///         stale copy would serve a boot-from-cache node the wrong address space while every
+    ///         id-shaped signal claimed it was right.
+    ///     </para>
+    ///     <para>
+    ///         <b>The chunk count is load-bearing.</b> A pointer can name a deployment whose chunks
+    ///         are missing or partial — a half-replicated artifact, or a prune that raced the
+    ///         pointer. Skipping on the pointer alone would make that state permanent, because the
+    ///         re-store that repairs it is the call being skipped.
+    ///     </para>
+    ///     <para>
+    ///         <b>Accepted consequence:</b> a skipped store leaves <c>applied_at_utc</c> at the
+    ///         moment the artifact was <i>first</i> cached rather than last confirmed. Refreshing it
+    ///         would mint the very oplog row this check exists to avoid. Nothing reads it as a
+    ///         liveness signal; retention ranks distinct deployments, which are unaffected.
+    ///     </para>
+    /// </remarks>
+    private async Task<bool> IsAlreadyCachedAsync(
+        string clusterId, string deploymentId, string revisionHash, string sha, int chunkCount,
+        CancellationToken ct)
+    {
+        var matches = await _db.QueryAsync(
+            """
+            SELECT (
+                SELECT COUNT(*)
+                FROM deployment_artifacts
+                WHERE cluster_id = @ClusterId
+                  AND deployment_id = @DeploymentId
+                  AND chunk_count = @ChunkCount
+            )
+            FROM deployment_pointer
+            WHERE cluster_id = @ClusterId
+              AND deployment_id = @DeploymentId
+              AND revision_hash = @RevisionHash
+              AND artifact_sha256 = @Sha
+            """,
+            r => r.GetInt32(0),
+            new
+            {
+                ClusterId = clusterId,
+                DeploymentId = deploymentId,
+                RevisionHash = revisionHash,
+                Sha = sha,
+                ChunkCount = chunkCount,
+            },
+            ct);
+
+        return matches.Count == 1 && matches[0] == chunkCount;
+    }
+
+    /// <inheritdoc/>
+    public async Task<CachedDeploymentArtifact?> GetCurrentAsync(
+        string clusterId, CancellationToken ct = default)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
+
+        var pointers = await _db.QueryAsync(
+            """
+            SELECT deployment_id, revision_hash, artifact_sha256, applied_at_utc
+            FROM deployment_pointer
+            WHERE cluster_id = @ClusterId
+            """,
+            ReadPointer,
+            new { ClusterId = clusterId },
+            ct);
+
+        if (pointers.Count == 0)
+        {
+            _logger.LogDebug("No cached deployment pointer for cluster {ClusterId}.", clusterId);
+            return null;
+        }
+
+        return await ReassembleAsync(clusterId, pointers[0], ct);
+    }
+
+    /// <inheritdoc/>
+    public async Task<CachedDeploymentArtifact?> GetCurrentUnkeyedAsync(CancellationToken ct = default)
+    {
+        // applied_at_utc is round-trip ISO-8601 UTC, so ordering it as TEXT is chronological —
+        // that is the reason the format is pinned rather than left to the current culture.
+        var pointers = await _db.QueryAsync(
+            """
+            SELECT cluster_id, deployment_id, revision_hash, artifact_sha256, applied_at_utc
+            FROM deployment_pointer
+            ORDER BY applied_at_utc DESC
+            """,
+            r => (ClusterId: r.GetString(0), Pointer: new PointerRow(
+                DeploymentId: r.GetString(1),
+                RevisionHash: r.GetString(2),
+                ArtifactSha256: r.GetString(3),
+                AppliedAtUtc: r.GetString(4))),
+            parameters: null,
+            ct);
+
+        if (pointers.Count == 0)
+        {
+            _logger.LogDebug("No cached deployment pointer of any cluster.");
+            return null;
+        }
+
+        if (pointers.Count > 1)
+        {
+            // The node was re-homed between clusters. Booting the newest is the only defensible
+            // choice, but it must never be a silent one — a wrong-cluster address space presents as
+            // a plausible configuration, so the operator needs the cluster names to spot it.
+            _logger.LogWarning(
+                "Local deployment cache holds pointers for {PointerCount} clusters ({ClusterIds}); " +
+                "booting the newest ({ChosenClusterId}). This node appears to have been re-homed — " +
+                "clear the stale cache if that is unexpected.",
+                pointers.Count,
+                string.Join(", ", pointers.Select(p => p.ClusterId)),
+                pointers[0].ClusterId);
+        }
+
+        return await ReassembleAsync(pointers[0].ClusterId, pointers[0].Pointer, ct);
+    }
+
+    /// <summary>
+    ///     Rebuilds the artifact the pointer names, returning <see langword="null"/> unless every
+    ///     integrity check passes.
+    /// </summary>
+    private async Task<CachedDeploymentArtifact?> ReassembleAsync(
+        string clusterId, PointerRow pointer, CancellationToken ct)
+    {
+        var chunks = await _db.QueryAsync(
+            """
+            SELECT chunk_count, chunk_base64
+            FROM deployment_artifacts
+            WHERE deployment_id = @DeploymentId
+            ORDER BY chunk_index
+            """,
+            r => (ChunkCount: r.GetInt32(0), Base64: r.GetString(1)),
+            new { pointer.DeploymentId },
+            ct);
+
+        // The chunk_count carried on every row is what makes a partial replica detectable. Without
+        // it a missing tail chunk is indistinguishable from a shorter artifact.
+        var expectedChunkCount = chunks.Count > 0 ? chunks[0].ChunkCount : 0;
+
+        if (chunks.Count != expectedChunkCount)
+        {
+            _logger.LogWarning(
+                "Cached deployment {DeploymentId} for cluster {ClusterId} is incomplete: found " +
+                "{FoundChunks} of {ExpectedChunks} chunks. Treating as a cache miss.",
+                pointer.DeploymentId, clusterId, chunks.Count, expectedChunkCount);
+            return null;
+        }
+
+        byte[] artifact;
+        try
+        {
+            artifact = Decode(chunks.Select(c => c.Base64));
+        }
+        catch (FormatException ex)
+        {
+            _logger.LogWarning(
+                ex,
+                "Cached deployment {DeploymentId} for cluster {ClusterId} has a chunk that is not " +
+                "valid base64. Treating as a cache miss.",
+                pointer.DeploymentId, clusterId);
+            return null;
+        }
+
+        var actualSha = Convert.ToHexString(SHA256.HashData(artifact));
+        if (!string.Equals(actualSha, pointer.ArtifactSha256, StringComparison.OrdinalIgnoreCase))
+        {
+            _logger.LogWarning(
+                "Cached deployment {DeploymentId} for cluster {ClusterId} failed its SHA-256 check " +
+                "(expected {ExpectedSha}, computed {ActualSha}). Treating as a cache miss.",
+                pointer.DeploymentId, clusterId, pointer.ArtifactSha256, actualSha);
+            return null;
+        }
+
+        if (!DateTimeOffset.TryParse(pointer.AppliedAtUtc, CultureInfo.InvariantCulture,
+                DateTimeStyles.RoundtripKind, out var appliedAtUtc))
+        {
+            _logger.LogWarning(
+                "Cached deployment {DeploymentId} for cluster {ClusterId} has an unparseable " +
+                "applied_at_utc ({AppliedAtUtc}). Treating as a cache miss.",
+                pointer.DeploymentId, clusterId, pointer.AppliedAtUtc);
+            return null;
+        }
+
+        return new CachedDeploymentArtifact(
+            pointer.DeploymentId, pointer.RevisionHash, artifact, appliedAtUtc);
+    }
+
+    private static byte[] Decode(IEnumerable<string> chunksInOrder)
+    {
+        using var buffer = new MemoryStream();
+
+        foreach (var chunk in chunksInOrder)
+        {
+            var bytes = Convert.FromBase64String(chunk);
+            buffer.Write(bytes, 0, bytes.Length);
+        }
+
+        return buffer.ToArray();
+    }
+
+    private static PointerRow ReadPointer(Microsoft.Data.Sqlite.SqliteDataReader reader)
+        => new(
+            DeploymentId: reader.GetString(0),
+            RevisionHash: reader.GetString(1),
+            ArtifactSha256: reader.GetString(2),
+            AppliedAtUtc: reader.GetString(3));
+
+    /// <summary>
+    ///     Round-trip ("O") UTC, so lexicographic ordering of the stored TEXT is chronological
+    ///     ordering — the retention query sorts on it directly.
+    /// </summary>
+    private static string FormatTimestamp(DateTimeOffset value)
+        => value.ToUniversalTime().ToString("O", CultureInfo.InvariantCulture);
+
+    private sealed record PointerRow(
+        string DeploymentId, string RevisionHash, string ArtifactSha256, string AppliedAtUtc);
+}

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs

@@ -23,6 +23,7 @@ using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
 using ZB.MOM.WW.OtOpcUa.Core.Scripting;
 using ZB.MOM.WW.OtOpcUa.Core.VirtualTags;
 using ZB.MOM.WW.OtOpcUa.OpcUaServer;
+using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
 using ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms;
 using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
 using CommonsNodeId = ZB.MOM.WW.OtOpcUa.Commons.Types.NodeId;
@@ -57,6 +58,31 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
     /// <summary>Publishing interval handed to each driver's SubscribeBulk pass after an apply.</summary>
     private static readonly TimeSpan SubscriptionPublishingInterval = TimeSpan.FromSeconds(1);
 
+    /// <summary>
+    ///     Cache key used when an artifact carries no cluster scoping (single-cluster or unscoped
+    ///     deployments). A literal rather than an empty string so the row is visibly deliberate when
+    ///     someone reads the table during an incident.
+    /// </summary>
+    private const string SingleClusterCacheKey = "__single";
+
+    /// <summary>
+    ///     Node-local cache of applied deployment artifacts, or null when this node has none
+    ///     (admin-only graphs, and tests that do not exercise it).
+    /// </summary>
+    private readonly IDeploymentArtifactCache? _deploymentArtifactCache;
+
+    /// <summary>
+    ///     True once this node has booted a cached artifact because central SQL was unreachable.
+    /// </summary>
+    /// <remarks>
+    ///     Surfaced on <see cref="NodeDiagnosticsSnapshot"/> because a node running from cache looks
+    ///     completely healthy from the outside — it serves a full address space with live values.
+    ///     The difference is that its configuration is frozen at whatever the cache held, and no
+    ///     deployment can reach it. Without an explicit signal that is invisible until someone
+    ///     wonders why a deploy "succeeded" everywhere but did not take effect here.
+    /// </remarks>
+    private bool _isRunningFromCache;
+
     private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
     private readonly CommonsNodeId _localNode;
     private readonly IActorRef? _coordinatorOverride;
@@ -339,11 +365,18 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         ScriptRootLogger? scriptRootLogger = null,
         IActorRef? scriptedAlarmHostOverride = null,
         IDriverCapabilityInvokerFactory? invokerFactory = null,
-        Func<int>? driverMemberCountProvider = null) =>
+        Func<int>? driverMemberCountProvider = null,
+        IDeploymentArtifactCache? deploymentArtifactCache = null) =>
+        // WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
+        // expression tree. Six IActorRef? parameters and several interface-typed ones mean a
+        // mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
+        // the wrong dependency at runtime. New parameters go LAST in all three places — this
+        // signature, the constructor's, and this call — and nothing else moves.
         Akka.Actor.Props.Create(() => new DriverHostActor(
             dbFactory, localNode, coordinator, driverFactory, localRoles, dependencyMux, opcUaPublishActor,
             healthPublisher, virtualTagEvaluator, historyWriter, virtualTagHostOverride,
-            loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider));
+            loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider,
+            deploymentArtifactCache));
 
     /// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
     /// <param name="dbFactory">Database context factory for configuration database access.</param>
@@ -372,6 +405,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
     /// <param name="driverMemberCountProvider">Test seam (archreview 03/S4): overrides the count of Up
     /// <c>driver</c>-role cluster members the Primary gate reads while the role is unknown. When null the
     /// default reads <c>Cluster.Get(Context.System).State.Members</c> (0 on a non-cluster ActorRefProvider).</param>
+    /// <param name="deploymentArtifactCache">Optional node-local cache of applied deployment artifacts.
+    /// When supplied, each successful apply stores its artifact so the node can boot from its
+    /// last-known-good configuration while central SQL is unreachable. Null on admin-only nodes and in
+    /// tests that do not exercise the cache — caching is then simply skipped.</param>
     public DriverHostActor(
         IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
         CommonsNodeId localNode,
@@ -388,8 +425,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         ScriptRootLogger? scriptRootLogger = null,
         IActorRef? scriptedAlarmHostOverride = null,
         IDriverCapabilityInvokerFactory? invokerFactory = null,
-        Func<int>? driverMemberCountProvider = null)
+        Func<int>? driverMemberCountProvider = null,
+        IDeploymentArtifactCache? deploymentArtifactCache = null)
     {
+        _deploymentArtifactCache = deploymentArtifactCache;
         _dbFactory = dbFactory;
         _localNode = localNode;
         _coordinatorOverride = coordinator;
@@ -541,7 +580,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
                     // space were lost on restart. Re-spawn + re-materialise + re-subscribe from the
                     // applied deployment so a restarted/rebuilt node restores its served state instead
                     // of waiting for a config change (whose identical-config revision would no-op).
-                    RestoreApplied(new DeploymentId(latest.DeploymentId));
+                    RestoreApplied(new DeploymentId(latest.DeploymentId), revision);
                     break;
                 case NodeDeploymentStatus.Applying:
                     _log.Warning("DriverHost {Node}: found orphan Applying row for deployment {Id}; replaying",
@@ -560,16 +599,129 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         catch (Exception ex)
         {
             _log.Warning(ex, "DriverHost {Node}: ConfigDb unreachable on bootstrap; entering Stale", _localNode);
-            Become(Stale);
+
+            // Central is unreachable, so try the node-local cache before giving up. On a hit this
+            // node serves its last-known-good configuration through the outage instead of coming up
+            // with an empty address space. On a miss, behaviour is exactly what it always was.
+            if (!TryBootFromCache())
+                Become(Stale);
         }
     }
 
+    /// <summary>
+    ///     Last-resort boot path: apply the artifact cached by a previous successful deploy (or
+    ///     replicated from this node's pair peer) when central SQL cannot be reached.
+    /// </summary>
+    /// <returns>
+    ///     <see langword="true"/> when a cached artifact was applied and the actor is now Steady;
+    ///     <see langword="false"/> when the caller should fall through to <c>Stale</c>.
+    /// </returns>
+    /// <remarks>
+    ///     <para>
+    ///         <b>The read is unkeyed.</b> The cache is keyed by ClusterId so a pair shares one
+    ///         entry, but ClusterId is only derivable from an artifact you already hold or from the
+    ///         central DB — and at this seam we have neither. So the newest pointer row wins, which
+    ///         is correct in every real topology because a node belongs to one cluster and its peer
+    ///         replicates that same cluster's row. A node re-homed between clusters is the only
+    ///         ambiguous case, and the cache logs a warning naming both.
+    ///     </para>
+    ///     <para>
+    ///         <b>This does not make the node current.</b> <c>_currentRevision</c> is set from the
+    ///         cached artifact, so a subsequent dispatch of that same revision correctly no-ops,
+    ///         while any NEW revision still requires central — the cache holds the past, not the
+    ///         future. Dispatch handling is unchanged.
+    ///     </para>
+    ///     <para>
+    ///         Never throws: a fault here must degrade to Stale, which is exactly where the node
+    ///         would have been without a cache at all.
+    ///     </para>
+    /// </remarks>
+    private bool TryBootFromCache()
+    {
+        if (_deploymentArtifactCache is null)
+            return false;
+
+        try
+        {
+            var cached = _deploymentArtifactCache.GetCurrentUnkeyedAsync().GetAwaiter().GetResult();
+            if (cached is null)
+            {
+                _log.Info(
+                    "DriverHost {Node}: no cached deployment available; entering Stale with no configuration.",
+                    _localNode);
+                return false;
+            }
+
+            var deploymentId = DeploymentId.Parse(cached.DeploymentId);
+            var revision = RevisionHash.Parse(cached.RevisionHash);
+
+            // Steady, not Stale: this node is serving a real configuration. The retry-db timer that
+            // Stale would have started does not run here, so recovery rides on the next dispatch —
+            // matching how a normally-booted node behaves.
+            _currentRevision = revision;
+            Become(Steady);
+
+            ApplyCachedArtifact(deploymentId, cached.Artifact);
+
+            _isRunningFromCache = true;
+            _log.Warning(
+                "DriverHost {Node}: RUNNING FROM CACHE — central ConfigDb is unreachable, so this node " +
+                "booted deployment {Id} (rev {Rev}) cached at {CachedAtUtc:o} from its local database. " +
+                "Configuration changes cannot be applied until the ConfigDb is reachable again.",
+                _localNode, deploymentId, revision, cached.AppliedAtUtc);
+
+            return true;
+        }
+        catch (Exception ex)
+        {
+            _log.Error(ex,
+                "DriverHost {Node}: failed to boot from the local deployment cache; falling back to Stale.",
+                _localNode);
+            return false;
+        }
+    }
+
+    /// <summary>
+    ///     Applies an artifact already in hand — no ConfigDb read, no ACK.
+    /// </summary>
+    /// <remarks>
+    ///     Mirrors <see cref="RestoreApplied"/>, but sources the artifact from the cache rather than
+    ///     re-reading it from a database that is by definition unreachable here. It also skips
+    ///     <c>UpsertNodeDeploymentState</c> and <c>SendAck</c> for the same reason: both write to or
+    ///     depend on central.
+    /// </remarks>
+    private void ApplyCachedArtifact(DeploymentId deploymentId, byte[] blob)
+    {
+        var correlation = CorrelationId.NewId();
+
+        var specs = DeploymentArtifact.ParseDriverInstances(blob, _localNode.Value);
+        var snapshots = _children.ToDictionary(
+            kv => kv.Key,
+            kv => new DriverChildSnapshot(kv.Value.DriverType, kv.Value.LastConfigJson, kv.Value.ResilienceConfig),
+            StringComparer.Ordinal);
+        var plan = DriverSpawnPlanner.Compute(snapshots, specs);
+
+        foreach (var id in plan.ToStop) StopChild(id);
+        foreach (var spec in plan.ToApplyDelta) ApplyChildDelta(spec);
+        foreach (var spec in plan.ToSpawn) SpawnChild(spec);
+
+        // Hand the cached blob to the rebuild so it materialises the client-facing address space from
+        // it directly. Passing only the deploymentId would drive a ConfigDb read — unreachable here by
+        // definition — and the rebuild would no-op, leaving OPC UA clients browsing an empty server
+        // while the drivers below poll happily. (Found on the docker-dev live gate.)
+        _opcUaPublishActor?.Tell(
+            new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RebuildAddressSpace(correlation, deploymentId, blob));
+
+        PushDesiredSubscriptionsFromArtifact(deploymentId, blob);
+    }
+
     private void Steady()
     {
         Receive<DispatchDeployment>(HandleDispatchFromSteady);
         Receive<GetDiagnostics>(HandleGetDiagnostics);
         Receive<DriverInstanceActor.AttributeValuePublished>(ForwardToMux);
         Receive<DriverInstanceActor.AttributeAlarmPublished>(ForwardNativeAlarm);
+        Receive<DriverInstanceActor.ConnectivityChanged>(OnDriverConnectivityChanged);
         Receive<DriverInstanceActor.DiscoveredNodesReady>(HandleDiscoveredNodes);
         Receive<DriverInstanceActor.DeltaApplied>(HandleDeltaApplied);
         Receive<RestartDriver>(HandleRestartDriver);
@@ -600,6 +752,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         Receive<GetDiagnostics>(HandleGetDiagnostics);
         Receive<DriverInstanceActor.AttributeValuePublished>(ForwardToMux);
         Receive<DriverInstanceActor.AttributeAlarmPublished>(ForwardNativeAlarm);
+        Receive<DriverInstanceActor.ConnectivityChanged>(OnDriverConnectivityChanged);
         Receive<DriverInstanceActor.DiscoveredNodesReady>(HandleDiscoveredNodes);
         Receive<DriverInstanceActor.DeltaApplied>(HandleDeltaApplied);
         Receive<RestartDriver>(HandleRestartDriver);
@@ -1010,6 +1163,46 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
     ///         signal the inbound-write gate uses — only the Primary publishes the single fleet-wide copy.
     ///     </para>
     /// </summary>
+    /// <summary>
+    ///     #477 — a child driver's connectivity transition. Annotates the source-data Quality of EVERY native
+    ///     alarm condition the driver owns: comms lost → <see cref="OpcUaQuality.Bad"/>, restored →
+    ///     <see cref="OpcUaQuality.Good"/>. This is the ONLY signal for a comms-lost native source, because a
+    ///     disconnected driver emits no alarm transitions — the alarm feed goes silent, so without this a
+    ///     comms-lost condition would keep reporting the accidentally-Good default forever.
+    ///     <para>
+    ///         UNGATED by redundancy role (like the condition write in <see cref="ForwardNativeAlarm"/>): a
+    ///         Secondary keeps its address space — including condition quality — warm for failover. Quality is
+    ///         a pure annotation: <c>WriteAlarmQuality</c> touches ONLY the condition's Quality, never its
+    ///         Active/Acked/Retain, and fires a Part 9 event only on a real quality-bucket change. No cluster
+    ///         <c>alerts</c> row is published here — driver comms health has its own status surface
+    ///         (<see cref="IDriverHealthPublisher"/>); a row per condition would be alarm-fatigue.
+    ///     </para>
+    /// </summary>
+    private void OnDriverConnectivityChanged(DriverInstanceActor.ConnectivityChanged msg)
+    {
+        if (_opcUaPublishActor is null) return;
+
+        var quality = msg.Connected ? OpcUaQuality.Good : OpcUaQuality.Bad;
+        var ts = DateTime.UtcNow;
+        var annotated = 0;
+        // Fan out to every condition this driver owns. _alarmNodeIdByDriverRef is keyed by
+        // (DriverInstanceId, RawPath); one driver ref can back several condition NodeIds (identical machines).
+        foreach (var ((driverId, _), nodeIds) in _alarmNodeIdByDriverRef)
+        {
+            if (driverId != msg.DriverInstanceId) continue;
+            foreach (var n in nodeIds)
+            {
+                _opcUaPublishActor.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.AlarmQualityUpdate(
+                    n.NodeId, quality, ts, n.Realm));
+                annotated++;
+            }
+        }
+
+        if (annotated > 0)
+            _log.Debug("DriverHost {Node}: driver {Driver} {State} — annotated {Count} native condition(s) {Quality}",
+                _localNode, msg.DriverInstanceId, msg.Connected ? "connected" : "disconnected", annotated, quality);
+    }
+
     private void ForwardNativeAlarm(DriverInstanceActor.AttributeAlarmPublished msg)
     {
         if (_opcUaPublishActor is null) return;
@@ -1326,6 +1519,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         // applied yet, so the equipment can't be resolved). Drop it — the re-discovery loop re-sends it
         // and the post-recovery re-apply self-heals it once an apply runs (matches the no-op drops above).
         Receive<DriverInstanceActor.DiscoveredNodesReady>(_ => { });
+        // A child connectivity transition while the host is Stale has no live address space to annotate — drop
+        // it (the post-recovery rebuild re-materialises conditions, and the child re-announces on its next entry).
+        Receive<DriverInstanceActor.ConnectivityChanged>(_ => { });
         // A late DeltaApplied (an apply completed just before the DB went Stale) — re-register the driver's
         // mux adapter anyway; it simply re-reads the driver's current refs (harmless, no DB access).
         Receive<DriverInstanceActor.DeltaApplied>(HandleDeltaApplied);
@@ -1348,7 +1544,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
             NodeId: _localNode,
             CurrentRevision: _currentRevision,
             Drivers: drivers,
-            AsOfUtc: DateTime.UtcNow);
+            AsOfUtc: DateTime.UtcNow,
+            RunningFromCache: _isRunningFromCache);
         Sender.Tell(snapshot);
     }
 
@@ -1381,7 +1578,30 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
 
         try
         {
-            ReconcileDrivers(deploymentId);
+            var appliedBlob = ReconcileDrivers(deploymentId);
+
+            // Issue #486 — a null blob means the artifact could not be read (unreachable ConfigDb, or the
+            // empty bytes a missing row yields), so NOTHING was applied. Reporting Applied here used to
+            // advance _currentRevision too, and HandleDispatchFromSteady short-circuits on a revision
+            // match — so the node claimed a configuration it never applied AND could never be healed by
+            // re-dispatching that revision. Failing the apply keeps the revision where it was, which is
+            // what makes the retry land instead of being waved through. The node keeps serving its
+            // last-known-good address space, drivers and subscriptions throughout (issue #485) — this is
+            // about telling the truth, not about tearing anything down.
+            if (appliedBlob is null)
+            {
+                const string reason =
+                    "the deployment artifact could not be read (ConfigDb unreachable, or the row is missing/empty); nothing was applied";
+                UpsertNodeDeploymentState(deploymentId, NodeDeploymentStatus.Failed, reason);
+                SendAck(deploymentId, ApplyAckOutcome.Failed, reason, correlation);
+                OtOpcUaTelemetry.DeploymentApplied.Add(1, new KeyValuePair<string, object?>("outcome", "reject"));
+                span?.SetStatus(ActivityStatusCode.Error, reason);
+                _log.Error(
+                    "DriverHost {Node}: apply of {Id} FAILED — {Reason}. Staying on revision {Rev} and continuing to serve it; re-dispatch once the ConfigDb is readable",
+                    _localNode, deploymentId, reason, _currentRevision);
+                return;
+            }
+
             _currentRevision = revision;
             UpsertNodeDeploymentState(deploymentId, NodeDeploymentStatus.Applied, failureReason: null);
             SendAck(deploymentId, ApplyAckOutcome.Applied, failureReason: null, correlation);
@@ -1392,6 +1612,12 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
             // SubscribeBulk pass: hand each driver its desired tag references so live values flow into
             // the just-rebuilt address space instead of staying BadWaitingForInitialData.
             PushDesiredSubscriptions(deploymentId);
+            CacheAppliedArtifact(deploymentId, revision, appliedBlob);
+            // Reaching here means the artifact came from central, so the node is no longer serving a
+            // cache-sourced configuration. Note this only clears on a real apply: a dispatch of the
+            // revision already booted from cache short-circuits in HandleDispatchFromSteady without
+            // touching the ConfigDb, and the flag correctly stays set.
+            _isRunningFromCache = false;
             OtOpcUaTelemetry.DeploymentApplied.Add(1, new KeyValuePair<string, object?>("outcome", "ack"));
             _log.Info("DriverHost {Node}: applied deployment {Id} (rev {Rev}, children={Count})",
                 _localNode, deploymentId, revision, _children.Count);
@@ -1415,11 +1641,24 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
     /// <summary>
     ///     Read the deployment artifact + reconcile the set of running <see cref="DriverInstanceActor"/>
     ///     children. Spawn missing, ApplyDelta on config change, stop removed/disabled drivers.
-    ///     When the artifact blob is empty (legacy ControlPlane tests, smoke fixtures) or the
-    ///     configured <see cref="IDriverFactory"/> can't materialise any of the requested
-    ///     types, this is effectively a no-op.
+    ///     When the artifact blob is empty (legacy ControlPlane tests, smoke fixtures) the reconcile is
+    ///     SKIPPED outright — see the issue #485 guard below — and when the configured
+    ///     <see cref="IDriverFactory"/> can't materialise any of the requested types this is a no-op.
     /// </summary>
-    private void ReconcileDrivers(DeploymentId deploymentId)
+    /// <returns>
+    ///     The artifact blob that was reconciled, or <see langword="null"/> when it could not be
+    ///     loaded at all.
+    /// </returns>
+    /// <remarks>
+    ///     Returning the blob rather than swallowing it is load-bearing for the artifact cache. This
+    ///     method catches its own DB failures, logs a warning and returns WITHOUT rethrowing — so
+    ///     <see cref="ApplyAndAck"/> proceeds to its success path, ACKs Applied and logs success
+    ///     having spawned zero drivers. A cache write that trusted "the apply succeeded" would then
+    ///     persist an empty artifact as this node's last-known-good configuration, and the node would
+    ///     later boot from it into an empty address space. The caller distinguishes the two cases by
+    ///     this return value.
+    /// </remarks>
+    private byte[]? ReconcileDrivers(DeploymentId deploymentId)
     {
         byte[] blob;
         try
@@ -1434,7 +1673,21 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         {
             _log.Warning(ex, "DriverHost {Node}: failed to load artifact for {Id}; skipping reconcile",
                 _localNode, deploymentId);
-            return;
+            return null;
+        }
+
+        // Issue #485 — no bytes is no answer, not "a configuration with no drivers". Parsing zero bytes
+        // yields zero specs, and DriverSpawnPlanner then plans EVERY running child for StopChild: a missing
+        // deployment row (or a half-written one) silently takes this node's entire field I/O down while the
+        // apply still ACKs Applied. Nothing legitimate produces a zero-length blob — deleting the last
+        // driver still deploys a JSON document with empty arrays — so treat it exactly like the load
+        // failure above and leave the running children alone.
+        if (blob.Length == 0)
+        {
+            _log.Warning(
+                "DriverHost {Node}: artifact for {Id} read back empty; skipping reconcile and keeping the {Count} running driver(s)",
+                _localNode, deploymentId, _children.Count);
+            return null;
         }
 
         var specs = DeploymentArtifact.ParseDriverInstances(blob, _localNode.Value);
@@ -1455,6 +1708,74 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         // sequenced AFTER ReinitializeAsync rebuilds DependencyRefs — a synchronous re-register HERE would
         // re-read the STALE ref set (ApplyDelta runs asynchronously on the child), so it is deliberately NOT
         // done inline.
+
+        return blob;
+    }
+
+    /// <summary>
+    ///     Store a successfully applied artifact in the node-local cache, so this node can boot from
+    ///     it while central SQL is unreachable.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b>Never throws.</b> By the time this runs the deployment is already recorded Applied
+    ///         in central SQL and an Applied ACK has been sent to the coordinator. An exception
+    ///         escaping here would unwind into <see cref="ApplyAndAck"/>'s catch and send a second,
+    ///         contradictory Failed ACK for a deployment the fleet already believes is live.
+    ///     </para>
+    ///     <para>
+    ///         <b>An empty or unloadable blob is skipped, not cached.</b> See
+    ///         <see cref="ReconcileDrivers"/>: a DB failure there degrades silently, so "the apply
+    ///         succeeded" does not imply "a real configuration was applied". Caching an empty
+    ///         artifact would make it this node's last-known-good and boot it into an empty address
+    ///         space during the next outage — strictly worse than having no cache at all.
+    ///     </para>
+    ///     <para>
+    ///         Runs synchronously on the actor thread. That matches every other DB call on this path
+    ///         (all of which already block), and the alternative — piping the result back as a
+    ///         self-message — would need a handler registered in Steady, Applying AND Stale or it
+    ///         dead-letters after the <c>Become(Steady)</c> in the enclosing finally.
+    ///     </para>
+    /// </remarks>
+    private void CacheAppliedArtifact(DeploymentId deploymentId, RevisionHash revision, byte[]? blob)
+    {
+        if (_deploymentArtifactCache is null)
+            return;
+
+        if (blob is null || blob.Length == 0)
+        {
+            _log.Warning(
+                "DriverHost {Node}: not caching deployment {Id} — its artifact was empty or could not " +
+                "be loaded, so it is not a usable last-known-good configuration.",
+                _localNode, deploymentId);
+            return;
+        }
+
+        try
+        {
+            // The node's ClusterId is carried inside the artifact (ClusterNode rows), not in config
+            // or on IClusterRoleInfo. A single-cluster or unscoped artifact has none, so it shares
+            // one sentinel key — the pair still converges on a single pointer row either way.
+            var scope = DeploymentArtifact.ResolveClusterScope(blob, _localNode.Value);
+            var clusterId = scope.Mode == ClusterFilterMode.ScopeTo && !string.IsNullOrWhiteSpace(scope.ClusterId)
+                ? scope.ClusterId
+                : SingleClusterCacheKey;
+
+            _deploymentArtifactCache
+                .StoreAsync(clusterId, deploymentId.ToString(), revision.Value, blob)
+                .GetAwaiter()
+                .GetResult();
+
+            _log.Debug("DriverHost {Node}: cached deployment {Id} (rev {Rev}, cluster {ClusterId}, {Bytes} bytes)",
+                _localNode, deploymentId, revision, clusterId, blob.Length);
+        }
+        catch (Exception ex)
+        {
+            _log.Error(ex,
+                "DriverHost {Node}: failed to cache applied deployment {Id}. The apply itself succeeded " +
+                "and is unaffected; this node simply has no local fallback for that revision.",
+                _localNode, deploymentId);
+        }
     }
 
     /// <summary>
@@ -1466,14 +1787,24 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
     ///     drivers, rebuilds the address space from the applied artifact, and re-pushes SubscribeBulk.
     ///     No re-ack: the deployment is already Applied.
     /// </summary>
-    private void RestoreApplied(DeploymentId deploymentId)
+    private void RestoreApplied(DeploymentId deploymentId, RevisionHash? revision)
     {
         var correlation = CorrelationId.NewId();
         try
         {
-            ReconcileDrivers(deploymentId);
-            _opcUaPublishActor?.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RebuildAddressSpace(correlation, deploymentId));
+            var appliedBlob = ReconcileDrivers(deploymentId);
+            _opcUaPublishActor?.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RebuildAddressSpace(correlation, deploymentId, appliedBlob));
             PushDesiredSubscriptions(deploymentId);
+            // Populate the node-local cache from the artifact this restored node is now serving. The
+            // cache invariant is "holds what the node currently serves"; without this only a FRESH
+            // apply (ApplyAndAck) ever wrote it, so a node whose cache was lost — a wiped/fresh volume,
+            // a disk failure — would recover its served state here yet stay cache-less, unable to
+            // boot-from-cache on the NEXT central outage until some future new deploy happened to land.
+            // Replication does not heal that gap either: a peer's already-acked rows are pruned from its
+            // oplog, so a fully-wiped node is never back-filled. Re-caching on restore is what closes
+            // it. (Found on the docker-dev live gate.)
+            if (revision is { } rev)
+                CacheAppliedArtifact(deploymentId, rev, appliedBlob);
             _log.Info("DriverHost {Node}: restored served state for applied deployment {Id} on bootstrap", _localNode, deploymentId);
         }
         catch (Exception ex)
@@ -1507,6 +1838,31 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
             return;
         }
 
+        PushDesiredSubscriptionsFromArtifact(deploymentId, blob);
+    }
+
+    /// <summary>
+    ///     The artifact-processing half of <see cref="PushDesiredSubscriptions"/>, split out so the
+    ///     boot-from-cache path can drive it with bytes already in hand.
+    /// </summary>
+    /// <remarks>
+    ///     Separated because the cache path runs precisely when the ConfigDb read above cannot
+    ///     succeed — re-reading the artifact there would fail by definition.
+    /// </remarks>
+    private void PushDesiredSubscriptionsFromArtifact(DeploymentId deploymentId, byte[] blob)
+    {
+        // Issue #485 — the same "no bytes is no answer" rule as ReconcileDrivers. An empty artifact parses
+        // to a composition with no raw tags, which hands every child an EMPTY desired set — and an empty set
+        // drops the live subscription handle rather than re-subscribing. Reachable independently of the
+        // reconcile guard: this method does its OWN ConfigDb read, so the row can go missing between the two.
+        if (blob.Length == 0)
+        {
+            _log.Warning(
+                "DriverHost {Node}: artifact for {Id} read back empty; skipping SubscribeBulk and keeping the live subscriptions",
+                _localNode, deploymentId);
+            return;
+        }
+
         AddressSpaceComposition composition;
         try
         {

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs

@@ -46,6 +46,12 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
     public sealed record InitializeSucceeded(int Generation);
     public sealed record InitializeFailed(string Reason, int Generation);
     public sealed record DisconnectObserved(string Reason);
+    /// <summary>#477 — sent to the parent (<see cref="DriverHostActor"/>) on every connectivity transition:
+    /// <c>Connected=true</c> on entering the Connected state, <c>false</c> on entering Reconnecting. The host
+    /// annotates this driver's native alarm conditions' source-data Quality from it (comms lost → Bad,
+    /// restored → Good) — independently of alarm transitions, since a comms-lost driver emits no alarm
+    /// events. Fire-and-forget, mirroring <see cref="DeltaApplied"/>.</summary>
+    public sealed record ConnectivityChanged(string DriverInstanceId, bool Connected);
     public sealed record ApplyDelta(string DriverConfigJson, CorrelationId Correlation);
     public sealed record ApplyResult(bool Success, string? Reason, CorrelationId Correlation);
     /// <summary>
@@ -413,6 +419,11 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
 
     private void Connected()
     {
+        // #477 — announce connectivity to the host so it can clear any comms-lost Quality annotation on this
+        // driver's native alarm conditions (Bad → Good). Fire-and-forget; the host defaults conditions to a
+        // non-Good "waiting for initial data" quality at materialise, and this is what confirms them Good.
+        Context.Parent.Tell(new ConnectivityChanged(_driverInstanceId, Connected: true));
+
         ReceiveAsync<ApplyDelta>(HandleApplyDeltaAsync);
         Receive<DisconnectObserved>(msg =>
         {
@@ -483,6 +494,10 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
 
     private void Reconnecting()
     {
+        // #477 — announce comms loss to the host so it annotates this driver's native alarm conditions Bad
+        // (a comms-lost driver emits no alarm events, so this is the ONLY signal that the source is unreachable).
+        Context.Parent.Tell(new ConnectivityChanged(_driverInstanceId, Connected: false));
+
         Receive<RetryConnect>(_ => InitializeAsync(_currentConfigJson ?? "{}"));
         // Fast-fail writes while reconnecting (same reason as Connecting — avoids the 8s host Ask
         // timeout on an inbound write to a transiently-down driver). Synchronous Receive.

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ServerHistorianOptions.cs

@@ -40,8 +40,10 @@ public sealed class ServerHistorianOptions
     /// <summary>
     ///     The peppered-HMAC API key (<c>histgw_&lt;id&gt;_&lt;secret&gt;</c>) the gateway validates
     ///     in the <c>Authorization: Bearer</c> header. Supply via the environment variable
-    ///     <c>ServerHistorian__ApiKey</c> — never commit it to config. Required when
-    ///     <see cref="Enabled"/> is <c>true</c>.
+    ///     <c>ServerHistorian__ApiKey</c> — never commit it to config. The value may be a literal
+    ///     key or a <c>${secret:otopcua/historian/api-key}</c> token, which the pre-host secrets
+    ///     expander resolves fail-closed (throwing if the secret is absent) before this options
+    ///     class binds. Required when <see cref="Enabled"/> is <c>true</c>.
     /// </summary>
     public string ApiKey { get; init; } = "";
 

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs

@@ -58,13 +58,32 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
     /// <param name="Realm">The namespace realm the condition lives in — <see cref="AddressSpaceRealm.Uns"/> for
     /// scripted alarms (default), <see cref="AddressSpaceRealm.Raw"/> for v3 native raw conditions.</param>
     public sealed record AlarmStateUpdate(string AlarmNodeId, AlarmConditionSnapshot State, DateTime TimestampUtc, AddressSpaceRealm Realm);
+    /// <summary>#477 — annotate a materialised condition's source-data Quality out of band from any alarm
+    /// transition (the driver-connectivity path: comms lost → <see cref="OpcUaQuality.Bad"/>, restored →
+    /// <see cref="OpcUaQuality.Good"/>). Routed to <see cref="IOpcUaAddressSpaceSink.WriteAlarmQuality"/>,
+    /// which sets ONLY Quality and fires one Part 9 event on a quality-bucket change.</summary>
+    /// <param name="AlarmNodeId">The condition node id (RawPath for a native alarm).</param>
+    /// <param name="Quality">The source-data quality to annotate.</param>
+    /// <param name="TimestampUtc">The connectivity transition timestamp in UTC.</param>
+    /// <param name="Realm">The namespace realm the condition lives in (<see cref="AddressSpaceRealm.Raw"/> for native).</param>
+    public sealed record AlarmQualityUpdate(string AlarmNodeId, OpcUaQuality Quality, DateTime TimestampUtc, AddressSpaceRealm Realm);
     /// <summary>
     ///     Triggers an address-space rebuild. <paramref name="DeploymentId"/> is the deployment
     ///     just applied by the host; the rebuild loads THAT artifact so materialisation matches the
     ///     applied config + the SubscribeBulk pass. It is null only for legacy/dev callers, which
     ///     fall back to the latest sealed deployment (lags a not-yet-sealed apply by one revision).
     /// </summary>
-    public sealed record RebuildAddressSpace(CorrelationId Correlation, DeploymentId? DeploymentId = null);
+    /// <param name="Correlation">Correlation id for tracing this rebuild.</param>
+    /// <param name="DeploymentId">The applied deployment whose artifact to materialise, or null for the latest-sealed fallback.</param>
+    /// <param name="Artifact">
+    ///     The artifact bytes already in hand, used INSTEAD of loading them from the ConfigDb. This is
+    ///     what makes boot-from-cache actually serve its address space: on a central-SQL outage the
+    ///     host has the cached blob but <see cref="DeploymentId"/> alone would drive a ConfigDb read
+    ///     that cannot succeed, leaving the rebuild a no-op and clients browsing an empty server. Null
+    ///     on the normal path, where loading from the ConfigDb by id is correct.
+    /// </param>
+    public sealed record RebuildAddressSpace(
+        CorrelationId Correlation, DeploymentId? DeploymentId = null, byte[]? Artifact = null);
 
     /// <summary>Inject driver-discovered nodes (FixedTree) under an equipment at runtime (post-connect).</summary>
     /// <param name="EquipmentRootNodeId">The OPC UA NodeId of the equipment root folder to inject the
@@ -93,6 +112,10 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
     private int _writes;
     private byte _lastServiceLevel;
     private bool _publishedAtLeastOnce;
+    /// <summary>True once a real composition has been applied, i.e. there is a served address space worth
+    /// protecting. Distinguishes "the artifact went missing under a running server" (issue #485 — warn, and
+    /// hold what is materialised) from "nothing has been deployed here yet" (a quiet no-op at boot).</summary>
+    private bool _hasAppliedComposition;
     private DbHealthProbeActor.DbHealthStatus? _lastDbHealth;
     private RedundancyStateChanged? _lastSnapshot;
     private (bool Ok, DateTime At)? _probeAboutMe;
@@ -239,6 +262,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
 
         Receive<AttributeValueUpdate>(HandleAttributeUpdate);
         Receive<AlarmStateUpdate>(HandleAlarmUpdate);
+        Receive<AlarmQualityUpdate>(HandleAlarmQualityUpdate);
         Receive<RebuildAddressSpace>(HandleRebuild);
         Receive<MaterialiseDiscoveredNodes>(HandleMaterialiseDiscovered);
         Receive<ServiceLevelChanged>(HandleServiceLevelChanged);
@@ -296,6 +320,20 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
         }
     }
 
+    private void HandleAlarmQualityUpdate(AlarmQualityUpdate msg)
+    {
+        try
+        {
+            _sink.WriteAlarmQuality(msg.AlarmNodeId, msg.Quality, msg.TimestampUtc, msg.Realm);
+            Interlocked.Increment(ref _writes);
+            OtOpcUaTelemetry.OpcUaSinkWrite.Add(1, new KeyValuePair<string, object?>("kind", "alarm-quality"));
+        }
+        catch (Exception ex)
+        {
+            _log.Warning(ex, "OpcUaPublish: sink.WriteAlarmQuality threw for {Node}", msg.AlarmNodeId);
+        }
+    }
+
     private void HandleRebuild(RebuildAddressSpace msg)
     {
         using var span = OtOpcUaTelemetry.StartAddressSpaceRebuildSpan();
@@ -321,13 +359,37 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
 
         try
         {
-            // Prefer the artifact of the deployment the host just applied — at apply time it is not
-            // yet Sealed, so LoadLatestArtifact would return the PREVIOUS revision and materialise a
-            // stale composition (variables that don't match the SubscribeBulk refs). Fall back to
-            // latest-sealed only for legacy callers that don't carry a DeploymentId.
-            var artifact = msg.DeploymentId is { } depId
-                ? LoadArtifact(depId)
-                : LoadLatestArtifact();
+            // An in-hand artifact (boot-from-cache) wins: the host already has the cached bytes, and
+            // the ConfigDb read the DeploymentId path would do is exactly what is unreachable during
+            // the outage this exists to survive. Otherwise prefer the artifact of the deployment the
+            // host just applied — at apply time it is not yet Sealed, so LoadLatestArtifact would
+            // return the PREVIOUS revision and materialise a stale composition (variables that don't
+            // match the SubscribeBulk refs). Fall back to latest-sealed only for legacy callers that
+            // carry neither.
+            var artifact = msg.Artifact
+                ?? (msg.DeploymentId is { } depId ? LoadArtifact(depId) : LoadLatestArtifact());
+
+            // Issue #485 — an artifact we could not obtain is NOT an empty configuration. Parsing zero
+            // bytes yields an empty composition, which the planner diffs against the live one as a
+            // PureRemove and the applier then faithfully tears down: a transient ConfigDb blip empties the
+            // served address space. Nothing legitimate produces a zero-length blob (an operator who really
+            // deletes everything still deploys a JSON document with empty arrays), so the only honest
+            // reading of "no bytes" is "no answer" — hold the last-known-good address space and let the
+            // next deploy (or the next boot) supply a real one. Returning here also leaves _lastApplied
+            // intact, so the retry diffs against what is actually materialised.
+            if (artifact is null or { Length: 0 })
+            {
+                if (_hasAppliedComposition)
+                    _log.Warning(
+                        "OpcUaPublish: no usable artifact for deployment {Id} (correlation={Correlation}) — KEEPING the currently served address space rather than tearing it down",
+                        msg.DeploymentId, msg.Correlation);
+                else
+                    _log.Debug(
+                        "OpcUaPublish: no artifact to materialise yet (deployment={Id}, correlation={Correlation})",
+                        msg.DeploymentId, msg.Correlation);
+                return;
+            }
+
             var composition = _localNode is { } ln
                 ? DeploymentArtifact.ParseComposition(artifact, ln.Value,
                     inconsistency => _log.Warning("OpcUaPublish {Node}: cross-cluster binding — {Message}", ln, inconsistency))
@@ -343,6 +405,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
 
             var outcome = _applier.Apply(plan);
             _lastApplied = composition;
+            _hasAppliedComposition = true;
 
             // Sum swallowed per-node materialise failures across every pass together with Apply's own
             // removal-pass tally (archreview 01/S-1). A degraded apply used to vanish into per-node
@@ -407,7 +470,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
     }
 
     /// <summary>Read a specific deployment's artifact blob from ConfigDb (the one just applied,
-    /// which may not be Sealed yet). Empty array on any failure — parser treats it as "no composition".</summary>
+    /// which may not be Sealed yet). Empty array on any failure — <see cref="HandleRebuild"/> treats that as
+    /// "no answer" and abandons the rebuild, leaving the served address space untouched (issue #485).</summary>
     private byte[] LoadArtifact(DeploymentId deploymentId)
     {
         try
@@ -420,13 +484,13 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
         }
         catch (Exception ex)
         {
-            _log.Warning(ex, "OpcUaPublish: failed to load artifact for deployment {Id}; rebuild becomes no-op", deploymentId);
+            _log.Warning(ex, "OpcUaPublish: failed to load artifact for deployment {Id}; the rebuild is abandoned", deploymentId);
             return Array.Empty<byte>();
         }
     }
 
     /// <summary>Read the most recent <c>Sealed</c> deployment's artifact blob from ConfigDb.
-    /// Empty array on any failure — the parser treats empty blob as "no composition".</summary>
+    /// Empty array on any failure — see <see cref="LoadArtifact"/> for how that is handled.</summary>
     private byte[] LoadLatestArtifact()
     {
         try
@@ -440,7 +504,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
         }
         catch (Exception ex)
         {
-            _log.Warning(ex, "OpcUaPublish: failed to load latest deployment artifact; rebuild becomes no-op");
+            _log.Warning(ex, "OpcUaPublish: failed to load latest deployment artifact; the rebuild is abandoned");
             return Array.Empty<byte>();
         }
     }

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs

@@ -278,11 +278,31 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
 
     private void OnDependencyChanged(VirtualTagActor.DependencyValueChanged msg)
     {
-        // Feed the live value into the upstream the engine subscribes from. StatusCode 0 = Good; the
-        // mux only forwards values it received from a driver publish, so we treat them as Good-quality.
-        _upstream.Push(msg.TagId, new DataValueSnapshot(msg.Value, 0u, msg.TimestampUtc, msg.TimestampUtc));
+        // Feed the live value into the upstream the engine subscribes from. #478 — carry the source
+        // driver's quality (mapped to an OPC UA StatusCode) so the engine can derive a scripted
+        // condition's worst-of-input quality; a Bad/Uncertain input is no longer silently treated as Good.
+        _upstream.Push(msg.TagId,
+            new DataValueSnapshot(msg.Value, StatusFromQuality(msg.Quality), msg.TimestampUtc, msg.TimestampUtc));
     }
 
+    /// <summary>#478 — map the 3-state <see cref="OpcUaQuality"/> to an OPC UA StatusCode (severity bits)
+    /// for the engine's read cache. The inverse of <see cref="QualityFromStatus"/>.</summary>
+    private static uint StatusFromQuality(OpcUaQuality quality) => quality switch
+    {
+        OpcUaQuality.Bad => 0x80000000u,
+        OpcUaQuality.Uncertain => 0x40000000u,
+        _ => 0u, // Good
+    };
+
+    /// <summary>#478 — collapse an engine <see cref="ScriptedAlarmEvent.WorstInputStatusCode"/> (top-2
+    /// severity bits) back to the 3-state <see cref="OpcUaQuality"/> the Commons snapshot / node path use.</summary>
+    private static OpcUaQuality QualityFromStatus(uint statusCode) => (statusCode >> 30) switch
+    {
+        0 => OpcUaQuality.Good,
+        1 => OpcUaQuality.Uncertain,
+        _ => OpcUaQuality.Bad,
+    };
+
     private void OnEngineEmission(EngineEmission msg)
     {
         var e = msg.Event;
@@ -294,6 +314,21 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
             return;
         }
 
+        // #478 — QualityChanged is a source-quality annotation (worst-of-input bucket moved with no Part 9
+        // state transition — e.g. an input went Bad, freezing the condition). Route it to the dedicated
+        // WriteAlarmQuality node path (sets ONLY Quality, one Part 9 event on a bucket change): NO full-state
+        // projection (would clobber severity/message) and NO /alerts row (it is not a state transition, so it
+        // must not historize). Same out-of-band quality path native comms-loss uses (#477 Layer 2).
+        if (e.Emission == EmissionKind.QualityChanged)
+        {
+            _publishActor.Tell(new OpcUaPublishActor.AlarmQualityUpdate(
+                AlarmNodeId: e.AlarmId,
+                Quality: QualityFromStatus(e.WorstInputStatusCode),
+                TimestampUtc: e.TimestampUtc,
+                Realm: AddressSpaceRealm.Uns));
+            return;
+        }
+
         // Bridge to OPC UA: project the FULL Part 9 condition state (enabled/active/acked/confirmed/
         // shelving/severity/message) onto the materialised condition node via the Commons snapshot.
         // e.AlarmId is the materialised condition's NodeId (T14 aligned it to the ScriptedAlarmId).
@@ -539,7 +574,11 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
         Enabled: e.Condition.Enabled == AlarmEnabledState.Enabled,
         Shelving: MapShelving(e.Condition.Shelving.Kind),
         Severity: (ushort)SeverityToInt(e.Severity),
-        Message: e.Message);
+        Message: e.Message,
+        // #478 — the condition's Quality is the worst quality across the script's input tags at evaluation
+        // time (carried on the event by the engine). A transition fired while an input is Uncertain projects
+        // Uncertain here so the full-snapshot write doesn't clobber quality back to Good.
+        Quality: QualityFromStatus(e.WorstInputStatusCode));
 
     /// <summary>Maps the Core <see cref="ShelvingKind"/> onto the Commons <see cref="AlarmShelvingKind"/>
     /// mirror (the Commons assembly can't see the Core enum).</summary>

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs

@@ -15,6 +15,7 @@ using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
 using ZB.MOM.WW.OtOpcUa.Core.Scripting;
 using ZB.MOM.WW.OtOpcUa.Core.VirtualTags;
 using ZB.MOM.WW.OtOpcUa.OpcUaServer;
+using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
 using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
 using ZB.MOM.WW.OtOpcUa.Runtime.Health;
 using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
@@ -221,6 +222,11 @@ public static class ServiceCollectionExtensions
             var serviceLevel = resolver.GetService<IServiceLevelPublisher>() ?? NullServiceLevelPublisher.Instance;
             var loggerFactory = resolver.GetService<ILoggerFactory>() ?? NullLoggerFactory.Instance;
             var healthPublisher = resolver.GetService<IDriverHealthPublisher>() ?? NullDriverHealthPublisher.Instance;
+            // Node-local deployment-artifact cache. Registered by the Host's AddOtOpcUaLocalDb on
+            // driver-role nodes only; deliberately left null elsewhere (admin-only graphs, test
+            // harnesses) rather than given a null-object, so DriverHostActor skips caching outright
+            // instead of pretending to cache into a sink that drops everything.
+            var deploymentArtifactCache = resolver.GetService<IDeploymentArtifactCache>();
             // Root script logger backs the ScriptedAlarm host's engine + script logging. Registered in
             // Host DI inside the hasDriver block; may be absent in some role configs / test harnesses,
             // in which case the DriverHostActor gracefully skips spawning the ScriptedAlarm host.
@@ -338,7 +344,8 @@ public static class ServiceCollectionExtensions
                     historyWriter: historyWriter,
                     loggerFactory: loggerFactory,
                     scriptRootLogger: scriptRootLogger,
-                    invokerFactory: invokerFactory),
+                    invokerFactory: invokerFactory,
+                    deploymentArtifactCache: deploymentArtifactCache),
                 DriverHostActorName);
             registry.Register<DriverHostActorKey>(driverHost);
 

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs

@@ -100,7 +100,7 @@ public sealed class DependencyMuxActor : ReceiveActor
             // space carries thousands of tags and only a fraction feed virtual-tag expressions.
             return;
         }
-        var dep = new VirtualTagActor.DependencyValueChanged(msg.FullReference, msg.Value, msg.TimestampUtc);
+        var dep = new VirtualTagActor.DependencyValueChanged(msg.FullReference, msg.Value, msg.TimestampUtc, msg.Quality);
         foreach (var sub in subscribers)
         {
             sub.Tell(dep);

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs

@@ -29,7 +29,11 @@ public sealed class VirtualTagActor : ReceiveActor
 {
     public const string ScriptLogsTopic = "script-logs";
 
-    public sealed record DependencyValueChanged(string TagId, object? Value, DateTime TimestampUtc);
+    /// <summary>A dependency's value changed. <paramref name="Quality"/> (#478) carries the source
+    /// driver's OPC UA quality so the scripted-alarm host can derive a condition's worst-of-input quality;
+    /// it defaults to <see cref="OpcUaQuality.Good"/>, and the virtual-tag engine ignores it.</summary>
+    public sealed record DependencyValueChanged(
+        string TagId, object? Value, DateTime TimestampUtc, OpcUaQuality Quality = OpcUaQuality.Good);
 
     /// <summary>Re-assert the last emitted value/quality to the parent bridge, bypassing the value dedup.
     /// Sent by <see cref="VirtualTagHostActor"/> on every apply (deploy) to a surviving child: a deploy

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ZB.MOM.WW.OtOpcUa.Runtime.csproj

@@ -8,6 +8,9 @@
     <ItemGroup>
         <PackageReference Include="Akka.Hosting"/>
         <PackageReference Include="Akka.Cluster.Tools"/>
+        <!-- Core LocalDb only (ILocalDb + the connection/transaction seam) — the deployment-artifact
+             cache lives here, but the sync engine and its gRPC endpoint are the Host's concern. -->
+        <PackageReference Include="ZB.MOM.WW.LocalDb" />
     </ItemGroup>
 
     <ItemGroup>

tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/OpcUa/DeferredAddressSpaceSinkTests.cs

@@ -209,6 +209,8 @@ public class DeferredAddressSpaceSinkTests
         public void WriteValue(string nodeId, object? value, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm = AddressSpaceRealm.Uns)
             => WriteValueCalled = true;
 
+        public void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
+
         public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm = AddressSpaceRealm.Uns) { }
         public void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, AddressSpaceRealm realm, bool isNative = false) { }
         public void EnsureFolder(string folderNodeId, string? parentNodeId, string displayName, AddressSpaceRealm realm = AddressSpaceRealm.Uns) { }
@@ -226,6 +228,7 @@ public class DeferredAddressSpaceSinkTests
         public bool FolderRenameCalled { get; private set; }
 
         public void WriteValue(string nodeId, object? value, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm = AddressSpaceRealm.Uns) { }
+        public void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
         public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm = AddressSpaceRealm.Uns) { }
         public void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, AddressSpaceRealm realm, bool isNative = false) { }
         public void EnsureFolder(string folderNodeId, string? parentNodeId, string displayName, AddressSpaceRealm realm = AddressSpaceRealm.Uns) { }

tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/GenerationSealedCacheTests.cs

@@ -1,168 +0,0 @@
-using Shouldly;
-using Xunit;
-using ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-namespace ZB.MOM.WW.OtOpcUa.Configuration.Tests;
-
-[Trait("Category", "Unit")]
-public sealed class GenerationSealedCacheTests : IDisposable
-{
-    private readonly string _root = Path.Combine(Path.GetTempPath(), $"otopcua-sealed-{Guid.NewGuid():N}");
-
-    /// <summary>Cleans up temporary directory after test execution.</summary>
-    public void Dispose()
-    {
-        try
-        {
-            if (!Directory.Exists(_root)) return;
-            // Remove ReadOnly attribute first so Directory.Delete can clean sealed files.
-            foreach (var f in Directory.EnumerateFiles(_root, "*", SearchOption.AllDirectories))
-                File.SetAttributes(f, FileAttributes.Normal);
-            Directory.Delete(_root, recursive: true);
-        }
-        catch { /* best-effort cleanup */ }
-    }
-
-    private GenerationSnapshot MakeSnapshot(string clusterId, long generationId, string payload = "{\"sample\":true}") =>
-        new()
-        {
-            ClusterId = clusterId,
-            GenerationId = generationId,
-            CachedAt = DateTime.UtcNow,
-            PayloadJson = payload,
-        };
-
-    /// <summary>Verifies that reading a snapshot on first boot with no existing snapshot throws.</summary>
-    [Fact]
-    public async Task FirstBoot_NoSnapshot_ReadThrows()
-    {
-        var cache = new GenerationSealedCache(_root);
-
-        await Should.ThrowAsync<GenerationCacheUnavailableException>(
-            () => cache.ReadCurrentAsync("cluster-a"));
-    }
-
-    /// <summary>Verifies that sealed snapshots can be read back correctly.</summary>
-    [Fact]
-    public async Task SealThenRead_RoundTrips()
-    {
-        var cache = new GenerationSealedCache(_root);
-        var snapshot = MakeSnapshot("cluster-a", 42, "{\"hello\":\"world\"}");
-
-        await cache.SealAsync(snapshot);
-
-        var read = await cache.ReadCurrentAsync("cluster-a");
-        read.GenerationId.ShouldBe(42);
-        read.ClusterId.ShouldBe("cluster-a");
-        read.PayloadJson.ShouldBe("{\"hello\":\"world\"}");
-    }
-
-    /// <summary>Verifies that sealed files are marked read-only on disk.</summary>
-    [Fact]
-    public async Task SealedFile_IsReadOnly_OnDisk()
-    {
-        var cache = new GenerationSealedCache(_root);
-        await cache.SealAsync(MakeSnapshot("cluster-a", 5));
-
-        var sealedPath = Path.Combine(_root, "cluster-a", "5.db");
-        File.Exists(sealedPath).ShouldBeTrue();
-        var attrs = File.GetAttributes(sealedPath);
-        attrs.HasFlag(FileAttributes.ReadOnly).ShouldBeTrue("sealed file must be read-only");
-    }
-
-    /// <summary>Verifies that the current generation pointer advances when a new generation is sealed.</summary>
-    [Fact]
-    public async Task SealingTwoGenerations_PointerAdvances_ToLatest()
-    {
-        var cache = new GenerationSealedCache(_root);
-        await cache.SealAsync(MakeSnapshot("cluster-a", 1));
-        await cache.SealAsync(MakeSnapshot("cluster-a", 2));
-
-        cache.TryGetCurrentGenerationId("cluster-a").ShouldBe(2);
-        var read = await cache.ReadCurrentAsync("cluster-a");
-        read.GenerationId.ShouldBe(2);
-    }
-
-    /// <summary>Verifies that prior generation files are preserved after a new seal.</summary>
-    [Fact]
-    public async Task PriorGenerationFile_Survives_AfterNewSeal()
-    {
-        var cache = new GenerationSealedCache(_root);
-        await cache.SealAsync(MakeSnapshot("cluster-a", 1));
-        await cache.SealAsync(MakeSnapshot("cluster-a", 2));
-
-        File.Exists(Path.Combine(_root, "cluster-a", "1.db")).ShouldBeTrue(
-            "prior generations preserved for audit; pruning is separate");
-        File.Exists(Path.Combine(_root, "cluster-a", "2.db")).ShouldBeTrue();
-    }
-
-    /// <summary>Verifies that reading a corrupt sealed file fails safely.</summary>
-    [Fact]
-    public async Task CorruptSealedFile_ReadFailsClosed()
-    {
-        var cache = new GenerationSealedCache(_root);
-        await cache.SealAsync(MakeSnapshot("cluster-a", 7));
-
-        // Corrupt the sealed file: clear read-only, truncate, leave pointer intact.
-        var sealedPath = Path.Combine(_root, "cluster-a", "7.db");
-        File.SetAttributes(sealedPath, FileAttributes.Normal);
-        File.WriteAllBytes(sealedPath, [0x00, 0x01, 0x02]);
-
-        await Should.ThrowAsync<GenerationCacheUnavailableException>(
-            () => cache.ReadCurrentAsync("cluster-a"));
-    }
-
-    /// <summary>Verifies that reading with a missing sealed file fails safely.</summary>
-    [Fact]
-    public async Task MissingSealedFile_ReadFailsClosed()
-    {
-        var cache = new GenerationSealedCache(_root);
-        await cache.SealAsync(MakeSnapshot("cluster-a", 3));
-
-        // Delete the sealed file but leave the pointer — corruption scenario.
-        var sealedPath = Path.Combine(_root, "cluster-a", "3.db");
-        File.SetAttributes(sealedPath, FileAttributes.Normal);
-        File.Delete(sealedPath);
-
-        await Should.ThrowAsync<GenerationCacheUnavailableException>(
-            () => cache.ReadCurrentAsync("cluster-a"));
-    }
-
-    /// <summary>Verifies that reading with a corrupt pointer file fails safely.</summary>
-    [Fact]
-    public async Task CorruptPointerFile_ReadFailsClosed()
-    {
-        var cache = new GenerationSealedCache(_root);
-        await cache.SealAsync(MakeSnapshot("cluster-a", 9));
-
-        var pointerPath = Path.Combine(_root, "cluster-a", "CURRENT");
-        File.WriteAllText(pointerPath, "not-a-number");
-
-        await Should.ThrowAsync<GenerationCacheUnavailableException>(
-            () => cache.ReadCurrentAsync("cluster-a"));
-    }
-
-    /// <summary>Verifies that sealing the same generation twice is idempotent.</summary>
-    [Fact]
-    public async Task SealSameGenerationTwice_IsIdempotent()
-    {
-        var cache = new GenerationSealedCache(_root);
-        await cache.SealAsync(MakeSnapshot("cluster-a", 11));
-        await cache.SealAsync(MakeSnapshot("cluster-a", 11, "{\"v\":2}"));
-
-        var read = await cache.ReadCurrentAsync("cluster-a");
-        read.PayloadJson.ShouldBe("{\"sample\":true}", "sealed file is immutable; second seal no-ops");
-    }
-
-    /// <summary>Verifies that independent clusters do not interfere with each other.</summary>
-    [Fact]
-    public async Task IndependentClusters_DoNotInterfere()
-    {
-        var cache = new GenerationSealedCache(_root);
-        await cache.SealAsync(MakeSnapshot("cluster-a", 1));
-        await cache.SealAsync(MakeSnapshot("cluster-b", 10));
-
-        (await cache.ReadCurrentAsync("cluster-a")).GenerationId.ShouldBe(1);
-        (await cache.ReadCurrentAsync("cluster-b")).GenerationId.ShouldBe(10);
-    }
-}

tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/LiteDbConfigCacheTests.cs

@@ -1,192 +0,0 @@
-using Shouldly;
-using Xunit;
-using ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-namespace ZB.MOM.WW.OtOpcUa.Configuration.Tests;
-
-[Trait("Category", "Unit")]
-public sealed class LiteDbConfigCacheTests : IDisposable
-{
-    private readonly string _dbPath = Path.Combine(Path.GetTempPath(), $"otopcua-cache-test-{Guid.NewGuid():N}.db");
-
-    /// <summary>Cleans up the temporary database file.</summary>
-    public void Dispose()
-    {
-        if (File.Exists(_dbPath)) File.Delete(_dbPath);
-    }
-
-    private GenerationSnapshot Snapshot(string cluster, long gen) => new()
-    {
-        ClusterId = cluster,
-        GenerationId = gen,
-        CachedAt = DateTime.UtcNow,
-        PayloadJson = $"{{\"g\":{gen}}}",
-    };
-
-    /// <summary>Verifies that payload is preserved through a write-then-read cycle.</summary>
-    [Fact]
-    public async Task Roundtrip_preserves_payload()
-    {
-        using var cache = new LiteDbConfigCache(_dbPath);
-        var put = Snapshot("c-1", 42);
-        await cache.PutAsync(put);
-
-        var got = await cache.GetMostRecentAsync("c-1");
-        got.ShouldNotBeNull();
-        got!.GenerationId.ShouldBe(42);
-        got.PayloadJson.ShouldBe(put.PayloadJson);
-    }
-
-    /// <summary>Verifies that GetMostRecentAsync returns the latest generation when multiple exist.</summary>
-    [Fact]
-    public async Task GetMostRecent_returns_latest_when_multiple_generations_present()
-    {
-        using var cache = new LiteDbConfigCache(_dbPath);
-        foreach (var g in new long[] { 10, 20, 15 })
-            await cache.PutAsync(Snapshot("c-1", g));
-
-        var got = await cache.GetMostRecentAsync("c-1");
-        got!.GenerationId.ShouldBe(20);
-    }
-
-    /// <summary>Verifies that GetMostRecentAsync returns null for an unknown cluster.</summary>
-    [Fact]
-    public async Task GetMostRecent_returns_null_for_unknown_cluster()
-    {
-        using var cache = new LiteDbConfigCache(_dbPath);
-        (await cache.GetMostRecentAsync("ghost")).ShouldBeNull();
-    }
-
-    /// <summary>Verifies that Prune keeps the latest N generations and drops older ones.</summary>
-    [Fact]
-    public async Task Prune_keeps_latest_N_and_drops_older()
-    {
-        using var cache = new LiteDbConfigCache(_dbPath);
-        for (long g = 1; g <= 15; g++)
-            await cache.PutAsync(Snapshot("c-1", g));
-
-        await cache.PruneOldGenerationsAsync("c-1", keepLatest: 10);
-
-        (await cache.GetMostRecentAsync("c-1"))!.GenerationId.ShouldBe(15);
-
-        // Drop them one by one and count — should be exactly 10 remaining
-        var count = 0;
-        while (await cache.GetMostRecentAsync("c-1") is not null)
-        {
-            count++;
-            await cache.PruneOldGenerationsAsync("c-1", keepLatest: Math.Max(0, 10 - count));
-            if (count > 20) break; // safety
-        }
-        count.ShouldBe(10);
-    }
-
-    /// <summary>Verifies that writing the same cluster/generation twice replaces rather than duplicates.</summary>
-    [Fact]
-    public async Task Put_same_cluster_generation_twice_replaces_not_duplicates()
-    {
-        using var cache = new LiteDbConfigCache(_dbPath);
-        var first = Snapshot("c-1", 1);
-        first.PayloadJson = "{\"v\":1}";
-        await cache.PutAsync(first);
-
-        var second = Snapshot("c-1", 1);
-        second.PayloadJson = "{\"v\":2}";
-        await cache.PutAsync(second);
-
-        (await cache.GetMostRecentAsync("c-1"))!.PayloadJson.ShouldBe("{\"v\":2}");
-    }
-
-    // ------------------------------------------------------------------------------------
-    // Configuration-005 — concurrent PutAsync for the same (ClusterId, GenerationId) must
-    // not produce duplicate rows. The original find-then-insert was non-atomic so two racing
-    // callers could both observe `existing is null` and both Insert.
-    // ------------------------------------------------------------------------------------
-    /// <summary>Verifies that concurrent PutAsync calls for the same cluster and generation do not create duplicates.</summary>
-    [Fact]
-    public async Task PutAsync_concurrent_for_same_cluster_and_generation_does_not_duplicate()
-    {
-        using var cache = new LiteDbConfigCache(_dbPath);
-        // Pre-seed gen=99 so prune keepLatest:1 has a sentinel that survives independent of
-        // any potential duplicate (gen=42) row count.
-        await cache.PutAsync(Snapshot("c-1", 99));
-
-        // Many parallel writes for the same key. Without serialization, racing find-then-insert
-        // would Insert multiple rows for the same (ClusterId, GenerationId=42).
-        var tasks = Enumerable.Range(0, 64).Select(_ => Task.Run(async () =>
-        {
-            var s = Snapshot("c-1", 42);
-            await cache.PutAsync(s);
-        })).ToArray();
-
-        await Task.WhenAll(tasks);
-
-        // Count rows for gen=42 directly by inspecting the LiteDB file via a fresh handle.
-        cache.Dispose();
-        using var verify = new LiteDB.LiteDatabase(_dbPath);
-        var col = verify.GetCollection<GenerationSnapshot>("generations");
-        var gen42Count = col.Find(s => s.ClusterId == "c-1" && s.GenerationId == 42).Count();
-        gen42Count.ShouldBe(1,
-            $"PutAsync must upsert atomically — found {gen42Count} rows for (c-1, gen=42) after 64 concurrent puts");
-    }
-
-    // ------------------------------------------------------------------------------------
-    // Configuration-012 — the per-instance _writeGate (Configuration-005) does not protect
-    // against LiteDB's process-wide BsonMapper.Global lazy-init race. Many cache INSTANCES
-    // constructed + driven concurrently corrupt the shared global mapper, surfacing as
-    // "Member ClusterId not found on BsonMapper" or a bogus "duplicate key _id = 0". A private
-    // per-database mapper with the entity pre-registered fixes it.
-    // ------------------------------------------------------------------------------------
-    /// <summary>Verifies that many cache instances constructed and driven concurrently do not
-    /// corrupt LiteDB's shared global BsonMapper — each Put/Get round-trips its own payload and
-    /// no insert throws a member-not-found or duplicate-_id exception.</summary>
-    [Fact]
-    public async Task Concurrent_cache_instances_do_not_race_the_shared_bson_mapper()
-    {
-        var paths = new List<string>();
-        try
-        {
-            var outer = Enumerable.Range(0, 24).Select(i => Task.Run(async () =>
-            {
-                var path = Path.Combine(Path.GetTempPath(), $"otopcua-cache-mapperrace-{Guid.NewGuid():N}.db");
-                lock (paths) paths.Add(path);
-
-                using var cache = new LiteDbConfigCache(path);
-                // Pre-seed a sentinel, then hammer one (cluster, gen) from many threads.
-                await cache.PutAsync(Snapshot($"c-{i}", 99));
-                var inner = Enumerable.Range(0, 16)
-                    .Select(_ => Task.Run(() => cache.PutAsync(Snapshot($"c-{i}", 42))))
-                    .ToArray();
-                await Task.WhenAll(inner);
-
-                var got = await cache.GetMostRecentAsync($"c-{i}");
-                got.ShouldNotBeNull();
-                got!.GenerationId.ShouldBe(99); // 99 > 42, latest by GenerationId
-            })).ToArray();
-
-            // The unfixed code throws LiteException / NotSupportedException out of these tasks under
-            // the global-mapper race; the fixed code completes cleanly.
-            await Task.WhenAll(outer);
-        }
-        finally
-        {
-            foreach (var p in paths)
-                if (File.Exists(p)) File.Delete(p);
-        }
-    }
-
-    /// <summary>Verifies that a corrupted cache file surfaces as LocalConfigCacheCorruptException.</summary>
-    [Fact]
-    public void Corrupt_file_surfaces_as_LocalConfigCacheCorruptException()
-    {
-        // Write a file large enough to look like a LiteDB page but with garbage contents so page
-        // deserialization fails on the first read probe.
-        File.WriteAllBytes(_dbPath, new byte[8192]);
-        Array.Fill<byte>(File.ReadAllBytes(_dbPath), 0xAB);
-        using (var fs = File.OpenWrite(_dbPath))
-        {
-            fs.Write(new byte[8192].Select(_ => (byte)0xAB).ToArray());
-        }
-
-        Should.Throw<LocalConfigCacheCorruptException>(() => new LiteDbConfigCache(_dbPath));
-    }
-}

tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/ResilientConfigReaderTests.cs

@@ -1,355 +0,0 @@
-using Microsoft.Extensions.Logging;
-using Microsoft.Extensions.Logging.Abstractions;
-using Polly.Timeout;
-using Shouldly;
-using Xunit;
-using ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-namespace ZB.MOM.WW.OtOpcUa.Configuration.Tests;
-
-[Trait("Category", "Unit")]
-public sealed class ResilientConfigReaderTests : IDisposable
-{
-    private readonly string _root = Path.Combine(Path.GetTempPath(), $"otopcua-reader-{Guid.NewGuid():N}");
-
-    /// <summary>Disposes temporary test files.</summary>
-    public void Dispose()
-    {
-        try
-        {
-            if (!Directory.Exists(_root)) return;
-            foreach (var f in Directory.EnumerateFiles(_root, "*", SearchOption.AllDirectories))
-                File.SetAttributes(f, FileAttributes.Normal);
-            Directory.Delete(_root, recursive: true);
-        }
-        catch { /* best-effort */ }
-    }
-
-    /// <summary>Verifies that successful central DB reads return value and mark fresh.</summary>
-    [Fact]
-    public async Task CentralDbSucceeds_ReturnsValue_MarksFresh()
-    {
-        var cache = new GenerationSealedCache(_root);
-        var flag = new StaleConfigFlag { };
-        flag.MarkStale(); // pre-existing stale state
-        var reader = new ResilientConfigReader(cache, flag, NullLogger<ResilientConfigReader>.Instance);
-
-        var result = await reader.ReadAsync(
-            "cluster-a",
-            _ => ValueTask.FromResult("fresh-from-db"),
-            _ => "from-cache",
-            CancellationToken.None);
-
-        result.ShouldBe("fresh-from-db");
-        flag.IsStale.ShouldBeFalse("successful central-DB read clears stale flag");
-    }
-
-    /// <summary>Verifies that exhausted retries fall back to cache and mark stale.</summary>
-    [Fact]
-    public async Task CentralDbFails_ExhaustsRetries_FallsBackToCache_MarksStale()
-    {
-        var cache = new GenerationSealedCache(_root);
-        await cache.SealAsync(new GenerationSnapshot
-        {
-            ClusterId = "cluster-a", GenerationId = 99, CachedAt = DateTime.UtcNow,
-            PayloadJson = "{\"cached\":true}",
-        });
-        var flag = new StaleConfigFlag();
-        var reader = new ResilientConfigReader(cache, flag, NullLogger<ResilientConfigReader>.Instance,
-            timeout: TimeSpan.FromSeconds(10), retryCount: 2);
-        var attempts = 0;
-
-        var result = await reader.ReadAsync(
-            "cluster-a",
-            _ =>
-            {
-                attempts++;
-                throw new InvalidOperationException("SQL dead");
-#pragma warning disable CS0162
-                return ValueTask.FromResult("never");
-#pragma warning restore CS0162
-            },
-            snap => snap.PayloadJson,
-            CancellationToken.None);
-
-        attempts.ShouldBe(3, "1 initial + 2 retries = 3 attempts");
-        result.ShouldBe("{\"cached\":true}");
-        flag.IsStale.ShouldBeTrue("cache fallback flips stale flag true");
-    }
-
-    /// <summary>Verifies that DB failure with unavailable cache throws.</summary>
-    [Fact]
-    public async Task CentralDbFails_AndCacheAlsoUnavailable_Throws()
-    {
-        var cache = new GenerationSealedCache(_root);
-        var flag = new StaleConfigFlag();
-        var reader = new ResilientConfigReader(cache, flag, NullLogger<ResilientConfigReader>.Instance,
-            timeout: TimeSpan.FromSeconds(10), retryCount: 0);
-
-        await Should.ThrowAsync<GenerationCacheUnavailableException>(async () =>
-        {
-            await reader.ReadAsync<string>(
-                "cluster-a",
-                _ => throw new InvalidOperationException("SQL dead"),
-                _ => "never",
-                CancellationToken.None);
-        });
-
-        flag.IsStale.ShouldBeFalse("no snapshot ever served, so flag stays whatever it was");
-    }
-
-    /// <summary>Verifies that cancellation is not retried.</summary>
-    [Fact]
-    public async Task Cancellation_NotRetried()
-    {
-        var cache = new GenerationSealedCache(_root);
-        var flag = new StaleConfigFlag();
-        var reader = new ResilientConfigReader(cache, flag, NullLogger<ResilientConfigReader>.Instance,
-            timeout: TimeSpan.FromSeconds(10), retryCount: 5);
-        using var cts = new CancellationTokenSource();
-        cts.Cancel();
-        var attempts = 0;
-
-        await Should.ThrowAsync<OperationCanceledException>(async () =>
-        {
-            await reader.ReadAsync<string>(
-                "cluster-a",
-                ct =>
-                {
-                    attempts++;
-                    ct.ThrowIfCancellationRequested();
-                    return ValueTask.FromResult("ok");
-                },
-                _ => "cache",
-                cts.Token);
-        });
-
-        attempts.ShouldBeLessThanOrEqualTo(1);
-    }
-
-    // ------------------------------------------------------------------------------------
-    // Configuration-006 — command-timeout TaskCanceledException and TimeoutRejectedException
-    // must fall back to the sealed cache, not propagate as caller cancellation.
-    // ------------------------------------------------------------------------------------
-
-    /// <summary>Verifies that command timeout TaskCanceledException falls back to cache.</summary>
-    [Fact]
-    public async Task CommandTimeout_TaskCanceledException_FallsBackToCache()
-    {
-        // A SQL command-level timeout surfaces as a TaskCanceledException thrown by the
-        // delegate itself (not triggered by the caller's CancellationToken). It must be
-        // treated as a transient failure and trigger the cache fallback, not be mistaken
-        // for genuine caller cancellation and propagated.
-        var cache = new GenerationSealedCache(_root);
-        await cache.SealAsync(new GenerationSnapshot
-        {
-            ClusterId = "cluster-b", GenerationId = 7, CachedAt = DateTime.UtcNow,
-            PayloadJson = "{\"from\":\"cache\"}",
-        });
-        var flag = new StaleConfigFlag();
-        var reader = new ResilientConfigReader(cache, flag, NullLogger<ResilientConfigReader>.Instance,
-            timeout: TimeSpan.FromSeconds(10), retryCount: 0);
-
-        // Simulate a command-level timeout: TaskCanceledException with no linked token.
-        var result = await reader.ReadAsync(
-            "cluster-b",
-            _ => throw new TaskCanceledException("SQL command timeout (no caller token)"),
-            snap => snap.PayloadJson,
-            CancellationToken.None); // caller token is NOT cancelled
-
-        result.ShouldBe("{\"from\":\"cache\"}",
-            "command-timeout TaskCanceledException must fall back to sealed cache");
-        flag.IsStale.ShouldBeTrue("cache fallback marks the stale flag");
-    }
-
-    /// <summary>Verifies that Polly timeout rejection falls back to cache.</summary>
-    [Fact]
-    public async Task PollyTimeout_TimeoutRejectedException_FallsBackToCache()
-    {
-        // When Polly's own timeout strategy fires it throws TimeoutRejectedException.
-        // That should trigger the cache fallback just like any other transient error.
-        var cache = new GenerationSealedCache(_root);
-        await cache.SealAsync(new GenerationSnapshot
-        {
-            ClusterId = "cluster-c", GenerationId = 8, CachedAt = DateTime.UtcNow,
-            PayloadJson = "{\"from\":\"polly-timeout-cache\"}",
-        });
-        var flag = new StaleConfigFlag();
-        // Set an extremely short Polly timeout so the async delay triggers it.
-        var reader = new ResilientConfigReader(cache, flag, NullLogger<ResilientConfigReader>.Instance,
-            timeout: TimeSpan.FromMilliseconds(10), retryCount: 0);
-
-        var result = await reader.ReadAsync(
-            "cluster-c",
-            async ct =>
-            {
-                await Task.Delay(TimeSpan.FromSeconds(5), ct); // far exceeds 10 ms timeout
-                return "never";
-            },
-            snap => snap.PayloadJson,
-            CancellationToken.None);
-
-        result.ShouldBe("{\"from\":\"polly-timeout-cache\"}",
-            "Polly TimeoutRejectedException must fall back to sealed cache");
-        flag.IsStale.ShouldBeTrue("cache fallback marks the stale flag");
-    }
-
-    // ------------------------------------------------------------------------------------
-    // Configuration-010 — fallback warning log must scrub connection-string fragments and
-    // must not include the full exception object (which carries the stack and any inner-
-    // exception chain). Project rule: no credential or connection-string fragment in logs.
-    // ------------------------------------------------------------------------------------
-
-    /// <summary>Verifies that fallback warnings do not log exceptions or password fragments.</summary>
-    [Fact]
-    public async Task FallbackWarning_does_not_log_full_exception_object_or_password_fragment()
-    {
-        var cache = new GenerationSealedCache(_root);
-        await cache.SealAsync(new GenerationSnapshot
-        {
-            ClusterId = "cluster-e", GenerationId = 1, CachedAt = DateTime.UtcNow,
-            PayloadJson = "{\"ok\":true}",
-        });
-        var flag = new StaleConfigFlag();
-        var capturing = new CapturingLogger<ResilientConfigReader>();
-        var reader = new ResilientConfigReader(cache, flag, capturing,
-            timeout: TimeSpan.FromSeconds(10), retryCount: 0);
-
-        // Simulated SqlException-style message carrying a connection-string fragment, the
-        // kind of thing a poorly-wrapped delegate could surface.
-        const string secretBearingMessage =
-            "Login failed for user 'sa'. (Server=sql.example.com,1433;User Id=sa;Password=SuperSecret123!)";
-
-        await reader.ReadAsync(
-            "cluster-e",
-            _ => throw new InvalidOperationException(secretBearingMessage),
-            snap => snap.PayloadJson,
-            CancellationToken.None);
-
-        var warning = capturing.Records.ShouldHaveSingleItem();
-        warning.LogLevel.ShouldBe(LogLevel.Warning);
-
-        // The exception object passed as the first arg to LogWarning(ex, ...) drives the
-        // formatter's stack-trace dump; capturing it lets us assert the scrubbing surface.
-        warning.Exception.ShouldBeNull(
-            "the warning must not attach the raw exception — it can carry connection-string fragments");
-
-        // The rendered message must not echo password / user-id strings even if the caller
-        // embedded them in the exception message.
-        warning.RenderedMessage.ShouldNotContain("Password=", Case.Insensitive);
-        warning.RenderedMessage.ShouldNotContain("SuperSecret123!");
-        warning.RenderedMessage.ShouldNotContain("User Id=", Case.Insensitive);
-    }
-
-    /// <summary>Verifies that caller cancellation propagates rather than falling back.</summary>
-    [Fact]
-    public async Task CallerCancellation_Propagates_NotFallback()
-    {
-        // Explicit caller cancellation must NOT fall back to the sealed cache — the
-        // caller said stop, so we must stop.
-        var cache = new GenerationSealedCache(_root);
-        await cache.SealAsync(new GenerationSnapshot
-        {
-            ClusterId = "cluster-d", GenerationId = 9, CachedAt = DateTime.UtcNow,
-            PayloadJson = "{\"should\":\"not be returned\"}",
-        });
-        var flag = new StaleConfigFlag();
-        var reader = new ResilientConfigReader(cache, flag, NullLogger<ResilientConfigReader>.Instance,
-            timeout: TimeSpan.FromSeconds(10), retryCount: 0);
-        using var cts = new CancellationTokenSource();
-        cts.Cancel();
-
-        await Should.ThrowAsync<OperationCanceledException>(async () =>
-        {
-            await reader.ReadAsync<string>(
-                "cluster-d",
-                ct =>
-                {
-                    ct.ThrowIfCancellationRequested();
-                    return ValueTask.FromResult("ok");
-                },
-                _ => "cache-should-not-be-used",
-                cts.Token);
-        });
-
-        flag.IsStale.ShouldBeFalse("no cache snapshot served on genuine cancellation");
-    }
-}
-
-/// <summary>Represents a captured log record for testing.</summary>
-internal sealed record LogRecord(LogLevel LogLevel, string RenderedMessage, Exception? Exception);
-
-/// <summary>Captures log records for assertion in tests.</summary>
-internal sealed class CapturingLogger<T> : ILogger<T>
-{
-    /// <summary>Gets the list of captured log records.</summary>
-    public List<LogRecord> Records { get; } = new();
-
-    /// <summary>Begins a scope (no-op for testing).</summary>
-    /// <typeparam name="TState">The type of the scope state.</typeparam>
-    /// <param name="state">The scope state.</param>
-    /// <returns>A disposable scope handle.</returns>
-    public IDisposable BeginScope<TState>(TState state) where TState : notnull => NullScope.Instance;
-
-    /// <summary>Returns true to enable all log levels.</summary>
-    /// <param name="logLevel">The log level to check.</param>
-    /// <returns>True to indicate the log level is enabled.</returns>
-    public bool IsEnabled(LogLevel logLevel) => true;
-
-    /// <summary>Logs a message by capturing it.</summary>
-    /// <typeparam name="TState">The type of the log state.</typeparam>
-    /// <param name="logLevel">The log level.</param>
-    /// <param name="eventId">The event identifier.</param>
-    /// <param name="state">The log state.</param>
-    /// <param name="exception">The exception, if any.</param>
-    /// <param name="formatter">Function to format the log message.</param>
-    public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func<TState, Exception?, string> formatter)
-    {
-        Records.Add(new LogRecord(logLevel, formatter(state, exception), exception));
-    }
-
-    /// <summary>No-op scope for testing.</summary>
-    private sealed class NullScope : IDisposable
-    {
-        /// <summary>Gets the singleton instance.</summary>
-        public static readonly NullScope Instance = new();
-
-        /// <summary>Disposes the scope (no-op).</summary>
-        public void Dispose() { }
-    }
-}
-
-[Trait("Category", "Unit")]
-public sealed class StaleConfigFlagTests
-{
-    /// <summary>Verifies that default state is fresh.</summary>
-    [Fact]
-    public void Default_IsFresh()
-    {
-        new StaleConfigFlag().IsStale.ShouldBeFalse();
-    }
-
-    /// <summary>Verifies that stale and fresh states toggle correctly.</summary>
-    [Fact]
-    public void MarkStale_ThenFresh_Toggles()
-    {
-        var flag = new StaleConfigFlag();
-        flag.MarkStale();
-        flag.IsStale.ShouldBeTrue();
-        flag.MarkFresh();
-        flag.IsStale.ShouldBeFalse();
-    }
-
-    /// <summary>Verifies that concurrent writes converge to the final state.</summary>
-    [Fact]
-    public void ConcurrentWrites_Converge()
-    {
-        var flag = new StaleConfigFlag();
-        Parallel.For(0, 1000, i =>
-        {
-            if (i % 2 == 0) flag.MarkStale(); else flag.MarkFresh();
-        });
-        flag.MarkFresh();
-        flag.IsStale.ShouldBeFalse();
-    }
-}

tests/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms.Tests/ScriptedAlarmEngineTests.cs

@@ -1005,6 +1005,104 @@ public sealed class ScriptedAlarmEngineTests
         // If Dispose threw or hung, the WaitAsync would surface it.
     }
 
+    // -------------------------------------------------------------------------
+    // #478 — scripted condition Quality from worst-of-input tag quality (Layer 3)
+    // -------------------------------------------------------------------------
+
+    private const uint StatusUncertain = 0x40000000u;
+    private const uint StatusBad = 0x80000000u;
+
+    /// <summary>A transition that fires while an input is Uncertain carries that worst input quality on the
+    /// emitted event (so the projected snapshot doesn't clobber quality back to Good).</summary>
+    [Fact]
+    public async Task Transition_carries_worst_input_quality()
+    {
+        var up = new FakeUpstream();
+        up.Set("Temp", 50);
+        using var eng = Build(up, out _);
+        await eng.LoadAsync([Alarm("HighTemp", """return (int)ctx.GetTag("Temp").Value > 100;""")],
+            TestContext.Current.CancellationToken);
+
+        var events = new List<ScriptedAlarmEvent>();
+        eng.OnEvent += (_, e) => events.Add(e);
+
+        // Uncertain is "ready" (only outright Bad short-circuits), so the predicate runs and fires Activated.
+        up.Push("Temp", 150, StatusUncertain);
+        await WaitForAsync(() => events.Any(e => e.Emission == EmissionKind.Activated));
+
+        var activated = events.First(e => e.Emission == EmissionKind.Activated);
+        (activated.WorstInputStatusCode >> 30).ShouldBe(1u); // Uncertain severity bucket
+    }
+
+    /// <summary>A Bad input freezes the condition (no transition) but the worst-quality bucket changed
+    /// Good→Bad, so the engine emits a standalone QualityChanged carrying Bad.</summary>
+    [Fact]
+    public async Task Bad_input_without_transition_emits_QualityChanged_Bad()
+    {
+        var up = new FakeUpstream();
+        up.Set("Temp", 50); // predicate false → inactive
+        using var eng = Build(up, out _);
+        await eng.LoadAsync([Alarm("HighTemp", """return (int)ctx.GetTag("Temp").Value > 100;""")],
+            TestContext.Current.CancellationToken);
+
+        var events = new List<ScriptedAlarmEvent>();
+        eng.OnEvent += (_, e) => events.Add(e);
+
+        up.Push("Temp", 50, StatusBad);
+        await WaitForAsync(() => events.Any(e => e.Emission == EmissionKind.QualityChanged));
+
+        var q = events.First(e => e.Emission == EmissionKind.QualityChanged);
+        (q.WorstInputStatusCode >> 30).ShouldBeGreaterThanOrEqualTo(2u); // Bad severity bucket
+        // No phantom state transition accompanied the quality change.
+        events.Any(e => e.Emission == EmissionKind.Activated || e.Emission == EmissionKind.Cleared)
+            .ShouldBeFalse();
+        eng.GetState("HighTemp")!.Active.ShouldBe(AlarmActiveState.Inactive);
+    }
+
+    /// <summary>Restoring a Good input after a Bad one flips the bucket back and emits QualityChanged(Good).</summary>
+    [Fact]
+    public async Task Restoring_good_input_emits_QualityChanged_Good()
+    {
+        var up = new FakeUpstream();
+        up.Set("Temp", 50);
+        using var eng = Build(up, out _);
+        await eng.LoadAsync([Alarm("HighTemp", """return (int)ctx.GetTag("Temp").Value > 100;""")],
+            TestContext.Current.CancellationToken);
+
+        var events = new List<ScriptedAlarmEvent>();
+        eng.OnEvent += (_, e) => events.Add(e);
+
+        up.Push("Temp", 50, StatusBad);
+        await WaitForAsync(() => events.Any(e => e.Emission == EmissionKind.QualityChanged
+            && (e.WorstInputStatusCode >> 30) >= 2u));
+        events.Clear();
+
+        up.Push("Temp", 50, 0u); // Good again, still below threshold
+        await WaitForAsync(() => events.Any(e => e.Emission == EmissionKind.QualityChanged));
+
+        var q = events.First(e => e.Emission == EmissionKind.QualityChanged);
+        (q.WorstInputStatusCode >> 30).ShouldBe(0u); // Good bucket
+    }
+
+    /// <summary>A value change that keeps the same quality bucket (Good→Good) emits no QualityChanged.</summary>
+    [Fact]
+    public async Task No_QualityChanged_when_bucket_unchanged()
+    {
+        var up = new FakeUpstream();
+        up.Set("Temp", 50);
+        using var eng = Build(up, out _);
+        await eng.LoadAsync([Alarm("HighTemp", """return (int)ctx.GetTag("Temp").Value > 100;""")],
+            TestContext.Current.CancellationToken);
+
+        var events = new List<ScriptedAlarmEvent>();
+        eng.OnEvent += (_, e) => events.Add(e);
+
+        up.Push("Temp", 60, 0u); // Good→Good, still below threshold → no transition, no quality change
+        await Task.Delay(150, TestContext.Current.CancellationToken);
+
+        events.ShouldNotContain(e => e.Emission == EmissionKind.QualityChanged);
+    }
+
     private static async Task WaitForAsync(Func<bool> cond, int timeoutMs = 2000)
     {
         var deadline = DateTime.UtcNow.AddMilliseconds(timeoutMs);

tests/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms.Tests/ScriptedAlarmSourceTests.cs

@@ -107,6 +107,28 @@ public sealed class ScriptedAlarmSourceTests
         events.Count.ShouldBe(0);
     }
 
+    /// <summary>#478 — a QualityChanged engine emission (source-quality bucket change, no state transition)
+    /// must NOT surface through the IAlarmSource fan-out: quality is delivered out of band via the
+    /// dedicated node path, never as an AlarmEventArgs (which would materialize / historize a condition).</summary>
+    [Fact]
+    public async Task QualityChanged_emission_raises_no_alarm_event()
+    {
+        var (engine, source, up) = await BuildAsync();
+        using var _e = engine;
+        using var _s = source;
+
+        var events = new List<AlarmEventArgs>();
+        source.OnAlarmEvent += (_, e) => events.Add(e);
+        await source.SubscribeAlarmsAsync([], TestContext.Current.CancellationToken);
+
+        // Drive HighTemp's input Bad: the predicate freezes (no transition), but the worst-quality bucket
+        // moves Good→Bad → the engine emits QualityChanged. The source must swallow it.
+        up.Push("Temp", 50, 0x80000000u);
+        await Task.Delay(200);
+
+        events.ShouldBeEmpty();
+    }
+
     /// <summary>Verifies that AcknowledgeAsync routes to the engine with a default user.</summary>
     [Fact]
     public async Task AcknowledgeAsync_routes_to_engine_with_default_user()

tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser.Tests/GalaxyDriverBrowserTests.cs

@@ -1,5 +1,6 @@
 using Shouldly;
 using Xunit;
+using ZB.MOM.WW.Secrets.Abstractions;
 
 namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser.Tests;
 
@@ -14,7 +15,17 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser.Tests;
 [Trait("Category", "Unit")]
 public sealed class GalaxyDriverBrowserTests
 {
-    private readonly GalaxyDriverBrowser _sut = new();
+    // These unit tests exercise only the pre-connect validation paths (empty endpoint,
+    // blank client name) that throw BEFORE the API-key secret: arm is resolved, so a
+    // null-returning stub resolver suffices — it is never consulted.
+    private readonly GalaxyDriverBrowser _sut = new(new StubSecretResolver());
+
+    /// <summary>A no-op resolver: every name resolves to null. Never consulted by these tests.</summary>
+    private sealed class StubSecretResolver : ISecretResolver
+    {
+        /// <inheritdoc />
+        public Task<string?> GetAsync(SecretName name, CancellationToken ct) => Task.FromResult<string?>(null);
+    }
 
     /// <summary>The DriverType key must match the AdminUI's persisted "GalaxyMxGateway" value
     /// so the factory wire-up picks the right browser implementation.</summary>

tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests/GalaxyDriverApiKeyResolverTests.cs

@@ -2,35 +2,48 @@ using Microsoft.Extensions.Logging;
 using Shouldly;
 using Xunit;
 using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
+using ZB.MOM.WW.Secrets.Abstractions;
 
 namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests;
 
 /// <summary>
-///     Follow-up #2 — pins the three resolution forms supported by
-///     <see cref="GalaxySecretRef.ResolveApiKey"/>: <c>env:NAME</c>, <c>file:PATH</c>,
-///     and the literal-string fallback. A future DPAPI arm slots in here without
-///     touching the call site. (The resolver was extracted from <c>GalaxyDriver</c> to
-///     the shared <c>GalaxySecretRef</c> in Driver.Galaxy.Contracts so the runtime
-///     driver and the AdminUI browser share one copy.)
+///     Follow-up #2 + G-2a — pins the five resolution forms supported by
+///     <see cref="GalaxySecretRef.ResolveApiKeyAsync"/>: <c>env:NAME</c>, <c>file:PATH</c>,
+///     <c>dev:KEY</c>, <c>secret:NAME</c> (via the shared <see cref="ISecretResolver"/>), and
+///     the literal-string fallback. The <c>secret:</c> arm is fail-closed — an absent secret
+///     throws rather than leaking the ref string as the key. (The resolver was extracted from
+///     <c>GalaxyDriver</c> to the shared <c>GalaxySecretRef</c> in Driver.Galaxy.Contracts so the
+///     runtime driver and the AdminUI browser share one copy.)
 /// </summary>
 public sealed class GalaxyDriverApiKeyResolverTests
 {
+    /// <summary>Name the fake resolver knows; matches the SecretName-normalized form.</summary>
+    private const string KnownSecretName = "galaxy/inst1/apikey";
+
+    /// <summary>The value the fake resolver returns for <see cref="KnownSecretName"/>.</summary>
+    private const string KnownSecretValue = "key-from-secret-store";
+
+    /// <summary>A fake resolver: returns a known value for one known name, null otherwise.</summary>
+    private static ISecretResolver FakeResolver() => new StubSecretResolver();
+
     /// <summary>Verifies that a literal string is returned unchanged.</summary>
     [Fact]
-    public void Literal_string_is_returned_unchanged()
+    public async Task Literal_string_is_returned_unchanged()
     {
-        GalaxySecretRef.ResolveApiKey("plain-text-key").ShouldBe("plain-text-key");
+        (await GalaxySecretRef.ResolveApiKeyAsync("plain-text-key", FakeResolver()))
+            .ShouldBe("plain-text-key");
     }
 
     /// <summary>Verifies that env: prefix resolves to an environment variable.</summary>
     [Fact]
-    public void Env_prefix_resolves_to_environment_variable()
+    public async Task Env_prefix_resolves_to_environment_variable()
     {
         const string name = "OTOPCUA_TEST_GALAXY_API_KEY";
         Environment.SetEnvironmentVariable(name, "key-from-env");
         try
         {
-            GalaxySecretRef.ResolveApiKey($"env:{name}").ShouldBe("key-from-env");
+            (await GalaxySecretRef.ResolveApiKeyAsync($"env:{name}", FakeResolver()))
+                .ShouldBe("key-from-env");
         }
         finally
         {
@@ -40,26 +53,27 @@ public sealed class GalaxyDriverApiKeyResolverTests
 
     /// <summary>Verifies that unset environment variables throw with a descriptive message.</summary>
     [Fact]
-    public void Env_prefix_unset_variable_throws_with_descriptive_message()
+    public async Task Env_prefix_unset_variable_throws_with_descriptive_message()
     {
         const string name = "OTOPCUA_TEST_GALAXY_API_KEY_UNSET";
         Environment.SetEnvironmentVariable(name, null);
 
-        var ex = Should.Throw<InvalidOperationException>(() =>
-            GalaxySecretRef.ResolveApiKey($"env:{name}"));
+        var ex = await Should.ThrowAsync<InvalidOperationException>(() =>
+            GalaxySecretRef.ResolveApiKeyAsync($"env:{name}", FakeResolver()));
         ex.Message.ShouldContain(name);
         ex.Message.ShouldContain("unset");
     }
 
     /// <summary>Verifies that file: prefix resolves to trimmed file contents.</summary>
     [Fact]
-    public void File_prefix_resolves_to_trimmed_file_contents()
+    public async Task File_prefix_resolves_to_trimmed_file_contents()
     {
         var path = Path.Combine(Path.GetTempPath(), $"galaxy-key-{Guid.NewGuid():N}.txt");
         File.WriteAllText(path, "  key-from-file  \n");
         try
         {
-            GalaxySecretRef.ResolveApiKey($"file:{path}").ShouldBe("key-from-file");
+            (await GalaxySecretRef.ResolveApiKeyAsync($"file:{path}", FakeResolver()))
+                .ShouldBe("key-from-file");
         }
         finally
         {
@@ -69,26 +83,60 @@ public sealed class GalaxyDriverApiKeyResolverTests
 
     /// <summary>Verifies that file: prefix with missing path throws.</summary>
     [Fact]
-    public void File_prefix_missing_path_throws()
+    public async Task File_prefix_missing_path_throws()
     {
         var path = Path.Combine(Path.GetTempPath(), $"does-not-exist-{Guid.NewGuid():N}.txt");
-        var ex = Should.Throw<InvalidOperationException>(() =>
-            GalaxySecretRef.ResolveApiKey($"file:{path}"));
+        var ex = await Should.ThrowAsync<InvalidOperationException>(() =>
+            GalaxySecretRef.ResolveApiKeyAsync($"file:{path}", FakeResolver()));
         ex.Message.ShouldContain(path);
         ex.Message.ShouldContain("doesn't exist");
     }
 
+    // ===== G-2a: secret: arm — resolves through the shared ISecretResolver, fail-closed =====
+
+    /// <summary>Verifies that secret: prefix resolves through the ISecretResolver.</summary>
+    [Fact]
+    public async Task Secret_prefix_resolves_through_secret_resolver()
+    {
+        (await GalaxySecretRef.ResolveApiKeyAsync($"secret:{KnownSecretName}", FakeResolver()))
+            .ShouldBe(KnownSecretValue);
+    }
+
+    /// <summary>Verifies that an absent secret throws (fail-closed) and does NOT leak the ref string.</summary>
+    [Fact]
+    public async Task Secret_prefix_absent_secret_throws_fail_closed()
+    {
+        const string missingRef = "secret:galaxy/missing/apikey";
+
+        var ex = await Should.ThrowAsync<InvalidOperationException>(() =>
+            GalaxySecretRef.ResolveApiKeyAsync(missingRef, FakeResolver()));
+
+        // Fail-closed: must throw, must reference the secret + "absent"/"fail-closed", and
+        // must NOT return the ref string as the key.
+        ex.Message.ShouldContain("galaxy/missing/apikey");
+        ex.Message.ShouldContain("absent");
+    }
+
+    /// <summary>Verifies that the secret: arm does not consult the logger's literal warning.</summary>
+    [Fact]
+    public async Task Secret_prefix_does_not_emit_literal_warning()
+    {
+        var logger = new CaptureLogger();
+        await GalaxySecretRef.ResolveApiKeyAsync($"secret:{KnownSecretName}", FakeResolver(), logger);
+        logger.Entries.ShouldNotContain(e => e.Level == LogLevel.Warning);
+    }
+
     // ===== Driver.Galaxy-010 regression: literal arm warns + dev: prefix path =====
 
     /// <summary>Verifies that literal strings emit a warning when a logger is supplied.</summary>
     [Fact]
-    public void Literal_string_emits_warning_when_logger_supplied()
+    public async Task Literal_string_emits_warning_when_logger_supplied()
     {
         // A literal API key on a production deployment means the cleartext key sits
         // in the DriverConfig JSON. The resolver must surface a warning so an
         // operator who committed one by accident sees it at startup.
         var logger = new CaptureLogger();
-        var key = GalaxySecretRef.ResolveApiKey("plain-text-key", logger);
+        var key = await GalaxySecretRef.ResolveApiKeyAsync("plain-text-key", FakeResolver(), logger);
 
         key.ShouldBe("plain-text-key");
         logger.Entries.ShouldContain(e =>
@@ -97,13 +145,13 @@ public sealed class GalaxyDriverApiKeyResolverTests
 
     /// <summary>Verifies that dev: prefix returns literal text without emitting warnings.</summary>
     [Fact]
-    public void Dev_prefix_returns_literal_without_warning()
+    public async Task Dev_prefix_returns_literal_without_warning()
     {
         // An explicit dev: prefix signals the operator knowingly opted into a literal
         // key (dev / parity rig). The resolver must accept it AND suppress the
         // warning so production logs aren't polluted on a deliberate dev choice.
         var logger = new CaptureLogger();
-        var key = GalaxySecretRef.ResolveApiKey("dev:plain-text-key", logger);
+        var key = await GalaxySecretRef.ResolveApiKeyAsync("dev:plain-text-key", FakeResolver(), logger);
 
         key.ShouldBe("plain-text-key");
         logger.Entries.ShouldNotContain(e => e.Level == LogLevel.Warning);
@@ -111,14 +159,14 @@ public sealed class GalaxyDriverApiKeyResolverTests
 
     /// <summary>Verifies that env: prefix does not emit literal string warnings.</summary>
     [Fact]
-    public void Env_prefix_does_not_emit_literal_warning()
+    public async Task Env_prefix_does_not_emit_literal_warning()
     {
         const string name = "OTOPCUA_TEST_GALAXY_API_KEY_NOWARN";
         Environment.SetEnvironmentVariable(name, "v");
         try
         {
             var logger = new CaptureLogger();
-            GalaxySecretRef.ResolveApiKey($"env:{name}", logger);
+            await GalaxySecretRef.ResolveApiKeyAsync($"env:{name}", FakeResolver(), logger);
             logger.Entries.ShouldNotContain(e => e.Level == LogLevel.Warning);
         }
         finally
@@ -127,6 +175,14 @@ public sealed class GalaxyDriverApiKeyResolverTests
         }
     }
 
+    /// <summary>Verifies that a null resolver is rejected (ArgumentNullException).</summary>
+    [Fact]
+    public async Task Null_resolver_throws()
+    {
+        await Should.ThrowAsync<ArgumentNullException>(() =>
+            GalaxySecretRef.ResolveApiKeyAsync("plain-text-key", resolver: null!));
+    }
+
     /// <summary>A test logger that captures log entries for verification.</summary>
     private sealed class CaptureLogger : ILogger
     {
@@ -144,16 +200,27 @@ public sealed class GalaxyDriverApiKeyResolverTests
             => Entries.Add((logLevel, formatter(state, exception)));
     }
 
+    /// <summary>A fake ISecretResolver returning one known value; null for every other name.</summary>
+    private sealed class StubSecretResolver : ISecretResolver
+    {
+        /// <inheritdoc />
+        public Task<string?> GetAsync(SecretName name, CancellationToken ct) =>
+            Task.FromResult<string?>(
+                string.Equals(name.Value, KnownSecretName, StringComparison.Ordinal)
+                    ? KnownSecretValue
+                    : null);
+    }
+
     /// <summary>Verifies that file: prefix with empty file throws.</summary>
     [Fact]
-    public void File_prefix_empty_file_throws()
+    public async Task File_prefix_empty_file_throws()
     {
         var path = Path.Combine(Path.GetTempPath(), $"galaxy-key-empty-{Guid.NewGuid():N}.txt");
         File.WriteAllText(path, "   \n  ");
         try
         {
-            var ex = Should.Throw<InvalidOperationException>(() =>
-                GalaxySecretRef.ResolveApiKey($"file:{path}"));
+            var ex = await Should.ThrowAsync<InvalidOperationException>(() =>
+                GalaxySecretRef.ResolveApiKeyAsync($"file:{path}", FakeResolver()));
             ex.Message.ShouldContain("empty");
         }
         finally

tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests/GalaxyDriverFactoryTests.cs

@@ -143,7 +143,7 @@ public sealed class GalaxyDriverFactoryTests
     public void Register_AddsFactoryToRegistry()
     {
         var registry = new DriverFactoryRegistry();
-        GalaxyDriverFactoryExtensions.Register(registry);
+        GalaxyDriverFactoryExtensions.Register(registry, NullSecretResolver.Instance);
 
         registry.RegisteredTypes.ShouldContain(GalaxyDriverFactoryExtensions.DriverTypeName);
 

tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests/Runtime/GalaxyDriverReadTests.cs

@@ -78,7 +78,7 @@ public sealed class GalaxyDriverReadTests
         // _dataReader and _subscriber are both null. The follow-up read path can't
         // synthesise a Read without one, so it surfaces a NotSupportedException
         // pointing at the misuse rather than NullRef'ing inside the pump path.
-        var driver = new GalaxyDriver("g", Opts());
+        var driver = new GalaxyDriver("g", Opts(), hierarchySource: null);
 
         var ex = await Should.ThrowAsync<NotSupportedException>(() =>
             driver.ReadAsync(["x"], CancellationToken.None));

tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests/Runtime/GalaxyDriverSubscribeTests.cs

@@ -242,7 +242,7 @@ public sealed class GalaxyDriverSubscribeTests
     [Fact]
     public async Task SubscribeAsync_NoSubscriber_Throws()
     {
-        using var driver = new GalaxyDriver("g", Opts());
+        using var driver = new GalaxyDriver("g", Opts(), hierarchySource: null);
         var ex = await Should.ThrowAsync<NotSupportedException>(() =>
             driver.SubscribeAsync(["x"], TimeSpan.FromSeconds(1), CancellationToken.None));
         ex.Message.ShouldContain("GalaxyDriver.SubscribeAsync requires");

tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests/Runtime/GalaxyDriverWriteTests.cs

@@ -192,7 +192,7 @@ public sealed class GalaxyDriverWriteTests
     [Fact]
     public async Task WriteAsync_NoWriter_Throws()
     {
-        var driver = new GalaxyDriver("g", Opts());
+        var driver = new GalaxyDriver("g", Opts(), hierarchySource: null);
 
         var ex = await Should.ThrowAsync<NotSupportedException>(() =>
             driver.WriteAsync([new WriteRequest("x", 1)], CancellationToken.None));

tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests/OpcUaClientSecretResolutionTests.cs

@@ -0,0 +1,95 @@
+using Shouldly;
+using Xunit;
+using ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
+using ZB.MOM.WW.Secrets.Abstractions;
+
+namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests;
+
+/// <summary>
+///     G-2b — pins <see cref="OpcUaClientSecretResolution.ResolveSecretRefsAsync"/>, the Layer-B
+///     arm that resolves <c>secret:</c>-prefixed <see cref="OpcUaClientDriverOptions.Password"/>
+///     and <see cref="OpcUaClientDriverOptions.UserCertificatePassword"/> through the shared
+///     <see cref="ISecretResolver"/> just before the async session-open consumes them. The
+///     <c>secret:</c> arm is fail-closed — an absent secret throws rather than leaving the
+///     <c>secret:</c> literal in place (which would be sent verbatim as the password).
+///     Non-secret literals and null/empty pass through unchanged.
+/// </summary>
+public sealed class OpcUaClientSecretResolutionTests
+{
+    private const string KnownPwName = "opcua/inst/password";
+    private const string KnownPwValue = "resolved-password-plaintext";
+    private const string KnownCertPwName = "opcua/inst/cert-pw";
+    private const string KnownCertPwValue = "resolved-cert-pw-plaintext";
+
+    /// <summary>A fake resolver: returns known values for two known names, null otherwise.</summary>
+    private static ISecretResolver FakeResolver() => new StubSecretResolver();
+
+    /// <summary>(a) A <c>secret:</c> Password resolves to the store's plaintext.</summary>
+    [Fact]
+    public async Task Secret_password_resolves_to_store_plaintext()
+    {
+        var options = new OpcUaClientDriverOptions { Password = $"secret:{KnownPwName}" };
+
+        var resolved = await OpcUaClientSecretResolution.ResolveSecretRefsAsync(
+            options, FakeResolver(), CancellationToken.None);
+
+        resolved.Password.ShouldBe(KnownPwValue);
+    }
+
+    /// <summary>(b) A <c>secret:</c> UserCertificatePassword resolves to the store's plaintext.</summary>
+    [Fact]
+    public async Task Secret_cert_password_resolves_to_store_plaintext()
+    {
+        var options = new OpcUaClientDriverOptions { UserCertificatePassword = $"secret:{KnownCertPwName}" };
+
+        var resolved = await OpcUaClientSecretResolution.ResolveSecretRefsAsync(
+            options, FakeResolver(), CancellationToken.None);
+
+        resolved.UserCertificatePassword.ShouldBe(KnownCertPwValue);
+    }
+
+    /// <summary>(c) A non-secret literal Password and a null field pass through unchanged.</summary>
+    [Fact]
+    public async Task Literal_and_null_pass_through_unchanged()
+    {
+        var options = new OpcUaClientDriverOptions
+        {
+            Password = "plainpw",
+            UserCertificatePassword = null,
+        };
+
+        var resolved = await OpcUaClientSecretResolution.ResolveSecretRefsAsync(
+            options, FakeResolver(), CancellationToken.None);
+
+        resolved.Password.ShouldBe("plainpw");
+        resolved.UserCertificatePassword.ShouldBeNull();
+    }
+
+    /// <summary>(d) An unknown <c>secret:</c> ref (resolver returns null) is fail-closed — it throws
+    ///     and never leaves the <c>secret:</c> literal in place.</summary>
+    [Fact]
+    public async Task Unknown_secret_ref_is_fail_closed()
+    {
+        var options = new OpcUaClientDriverOptions { Password = "secret:opcua/inst/absent" };
+
+        var ex = await Should.ThrowAsync<InvalidOperationException>(() =>
+            OpcUaClientSecretResolution.ResolveSecretRefsAsync(
+                options, FakeResolver(), CancellationToken.None));
+
+        ex.Message.ShouldContain("Password");
+        ex.Message.ShouldContain("opcua/inst/absent");
+    }
+
+    /// <summary>A fake resolver: returns known values for two known names, null otherwise.</summary>
+    private sealed class StubSecretResolver : ISecretResolver
+    {
+        /// <inheritdoc />
+        public Task<string?> GetAsync(SecretName name, CancellationToken ct) =>
+            Task.FromResult<string?>(name.Value switch
+            {
+                KnownPwName => KnownPwValue,
+                KnownCertPwName => KnownCertPwValue,
+                _ => null,
+            });
+    }
+}

tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Authorization/PageAuthorizationGuardTests.cs

@@ -47,6 +47,11 @@ public class PageAuthorizationGuardTests
         // ── FleetAdmin (1) ──
         [typeof(RoleGrants)] = AdminUiPolicies.FleetAdmin,
 
+        // ── Secrets (1): host wrapper for the ZB.MOM.WW.Secrets.Ui RCL page (lmxopcua#483) —
+        //    carries the RCL page's own policy, registered additively by AddAdminUI via
+        //    AddSecretsAuthorization(). ──
+        [typeof(SecretsAdmin)] = global::ZB.MOM.WW.Secrets.Ui.SecretsAuthorization.ManagePolicy,
+
         // ── AuthenticatedRead (16): dashboards, lists, live tails (read-only) + the dev ContextMenu demo ──
         [typeof(Home)] = AdminUiPolicies.AuthenticatedRead,
         [typeof(global::ZB.MOM.WW.OtOpcUa.AdminUI.Components.Pages.Dev.ContextMenuDemo)] = AdminUiPolicies.AuthenticatedRead,
@@ -75,6 +80,9 @@ public class PageAuthorizationGuardTests
         AdminUiPolicies.DriverOperator,
         AdminUiPolicies.ConfigEditor,
         AdminUiPolicies.AuthenticatedRead,
+        // Not an AdminUiPolicies constant: the /admin/secrets wrapper re-states the Secrets.Ui
+        // RCL page's own policy, which AddAdminUI registers via AddSecretsAuthorization().
+        global::ZB.MOM.WW.Secrets.Ui.SecretsAuthorization.ManagePolicy,
     ];
 
     private static IReadOnlyList<Type> RoutablePages() =>

tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/SecretsPageWiringTests.cs

@@ -0,0 +1,67 @@
+using System.Reflection;
+using Microsoft.AspNetCore.Authorization;
+using Microsoft.AspNetCore.Components;
+using Microsoft.AspNetCore.Components.Web;
+using Shouldly;
+using Xunit;
+using ZB.MOM.WW.OtOpcUa.AdminUI.Components.Pages;
+using ZB.MOM.WW.Secrets.Ui;
+
+namespace ZB.MOM.WW.OtOpcUa.AdminUI.Tests;
+
+/// <summary>
+///     Pins the host-side wiring that makes <c>/admin/secrets</c> interactive (lmxopcua#483).
+///     This host's router is static with per-page <c>@rendermode</c> opt-in; routing straight to
+///     the Secrets.Ui RCL's routable page rendered it with no circuit — the page displayed but
+///     every <c>@onclick</c> was dead, and nothing failed. The fix is a wrapper page in this
+///     assembly that owns the route and carries the render mode. Razor directives compile to
+///     class-level attributes, so a metadata scan is authoritative (same technique as
+///     <c>PageAuthorizationGuardTests</c>; the repo has no bUnit).
+/// </summary>
+public class SecretsPageWiringTests
+{
+    /// <summary>
+    ///     THE #483 regression pin: without a render mode this host serves the page as static SSR
+    ///     and it silently loses all interactivity.
+    /// </summary>
+    [Fact]
+    public void Wrapper_declares_the_InteractiveServer_render_mode()
+    {
+        var mode = typeof(SecretsAdmin).GetCustomAttributes<RenderModeAttribute>(inherit: false)
+            .SingleOrDefault();
+
+        mode.ShouldNotBeNull(
+            "/admin/secrets lost its @rendermode — in this host's static router that renders a " +
+            "dead page (displays, but no circuit and every @onclick inert), exactly lmxopcua#483");
+        mode.Mode.ShouldBeOfType<InteractiveServerRenderMode>();
+    }
+
+    /// <summary>
+    ///     The wrapper must shadow the RCL page's route exactly — if Secrets.Ui ever moves its
+    ///     page, the wrapper must move with it or the nav rail and bookmarks split from the RCL.
+    /// </summary>
+    [Fact]
+    public void Wrapper_owns_the_same_route_as_the_RCL_page()
+    {
+        string[] wrapper = [.. typeof(SecretsAdmin).GetCustomAttributes<RouteAttribute>(inherit: false).Select(r => r.Template)];
+        string[] rcl = [.. typeof(SecretsPage).GetCustomAttributes<RouteAttribute>(inherit: false).Select(r => r.Template)];
+
+        wrapper.ShouldBe(["/admin/secrets"]);
+        wrapper.ShouldBe(rcl, "the wrapper must track the RCL page's route exactly");
+    }
+
+    /// <summary>
+    ///     The router enforces [Authorize] only on the routed component — which is now the wrapper,
+    ///     with the RCL page rendered as a plain child whose own attribute is inert. The wrapper
+    ///     must therefore re-state the RCL page's policy verbatim, or the fix silently widens access.
+    /// </summary>
+    [Fact]
+    public void Wrapper_restates_the_RCL_pages_authorization_policy()
+    {
+        var wrapper = typeof(SecretsAdmin).GetCustomAttributes<AuthorizeAttribute>(inherit: false).Single();
+        var rcl = typeof(SecretsPage).GetCustomAttributes<AuthorizeAttribute>(inherit: false).Single();
+
+        wrapper.Policy.ShouldBe(SecretsAuthorization.ManagePolicy);
+        wrapper.Policy.ShouldBe(rcl.Policy, "the wrapper must enforce exactly what the RCL page declares");
+    }
+}

tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Uns/RawTreeServiceSecretRefRoundTripTests.cs

@@ -0,0 +1,110 @@
+using Microsoft.EntityFrameworkCore;
+using Shouldly;
+using Xunit;
+using ZB.MOM.WW.OtOpcUa.AdminUI.Uns;
+using ZB.MOM.WW.OtOpcUa.Configuration;
+
+namespace ZB.MOM.WW.OtOpcUa.AdminUI.Tests.Uns;
+
+/// <summary>
+/// Secrets-adoption (G-2c) regression guard: the AdminUI driver-config editor persists <c>DriverConfig</c>
+/// as an OPAQUE JSON string through <see cref="RawTreeService"/> (see <c>UpdateDriverAsync</c> /
+/// <c>CreateDriverAsync</c> — <c>entity.DriverConfig = driverConfigJson</c> verbatim, and
+/// <c>LoadDriverForEditAsync</c> reads it back verbatim). A <c>secret:</c>/<c>env:</c>/<c>file:</c>
+/// reference stored in a driver secret field (OpcUaClient <c>Password</c>/<c>UserCertificatePassword</c>,
+/// Galaxy <c>Gateway.ApiKeySecretRef</c>) MUST survive save→reload as the literal ref string — the save
+/// path must never resolve-then-resave cleartext (which would re-leak the secret into the config store and
+/// defeat G-2). Secret resolution belongs ONLY at driver-instantiation / Test-connect time (Tasks 7-8),
+/// never on the AdminUI persistence path — and indeed <see cref="RawTreeService"/> takes no secret resolver
+/// (its only dependency is the DbContext factory), so resolution is structurally impossible here.
+/// <para>
+/// Reuses the shared <see cref="RawTreeTestDb"/> InMemory fixture, matching the WP3 driver-config tests.
+/// </para>
+/// </summary>
+[Trait("Category", "Unit")]
+public sealed class RawTreeServiceSecretRefRoundTripTests
+{
+    private static (RawTreeService Service, string DbName) Seeded()
+    {
+        var name = RawTreeTestDb.SeedFresh();
+        return (new RawTreeService(RawTreeTestDb.Factory(name)), name);
+    }
+
+    private static byte[] RowVersionOf(string dbName, string driverId)
+    {
+        using var db = RawTreeTestDb.CreateNamed(dbName);
+        return db.DriverInstances.Single(d => d.DriverInstanceId == driverId).RowVersion;
+    }
+
+    [Fact]
+    public async Task UpdateDriver_preserves_OpcUaClient_secret_refs_verbatim_on_save_and_reload()
+    {
+        var (service, dbName) = Seeded();
+        var rv = RowVersionOf(dbName, RawTreeTestDb.FolderedDriverId);
+
+        // The editor emits the raw ref strings the user typed — they must NOT be resolved on save.
+        const string config =
+            "{\"Password\":\"secret:opcua/x/pw\",\"UserCertificatePassword\":\"secret:opcua/x/certpw\"}";
+
+        var result = await service.UpdateDriverAsync(
+            RawTreeTestDb.FolderedDriverId, "modbus-1", config, null, rv);
+        result.Ok.ShouldBeTrue();
+
+        // Reload through the same seam the modal uses.
+        var reloaded = await service.LoadDriverForEditAsync(RawTreeTestDb.FolderedDriverId);
+        reloaded.ShouldNotBeNull();
+
+        // Byte-identical: nothing on the save path rewrote the string.
+        reloaded!.DriverConfig.ShouldBe(config);
+        // The literal refs survived (not resolved to cleartext, not stripped).
+        reloaded.DriverConfig.ShouldContain("secret:opcua/x/pw");
+        reloaded.DriverConfig.ShouldContain("secret:opcua/x/certpw");
+
+        // And the raw column agrees with the projection.
+        using var db = RawTreeTestDb.CreateNamed(dbName);
+        db.DriverInstances.Single(d => d.DriverInstanceId == RawTreeTestDb.FolderedDriverId)
+            .DriverConfig.ShouldBe(config);
+    }
+
+    [Fact]
+    public async Task CreateDriver_preserves_Galaxy_ApiKeySecretRef_verbatim_on_save_and_reload()
+    {
+        var (service, dbName) = Seeded();
+
+        const string config = "{\"Gateway\":{\"ApiKeySecretRef\":\"secret:galaxy/x/apikey\"}}";
+
+        var created = await service.CreateDriverAsync(
+            RawTreeTestDb.ClusterId, RawTreeTestDb.RootFolderId, "galaxy-1", "Galaxy", config);
+        created.Ok.ShouldBeTrue();
+        created.CreatedId.ShouldNotBeNull();
+
+        var reloaded = await service.LoadDriverForEditAsync(created.CreatedId!);
+        reloaded.ShouldNotBeNull();
+
+        reloaded!.DriverConfig.ShouldBe(config);
+        reloaded.DriverConfig.ShouldContain("secret:galaxy/x/apikey");
+
+        using var db = RawTreeTestDb.CreateNamed(dbName);
+        db.DriverInstances.Single(d => d.DriverInstanceId == created.CreatedId)
+            .DriverConfig.ShouldBe(config);
+    }
+
+    [Fact]
+    public async Task UpdateDriver_preserves_env_and_file_ref_forms_verbatim()
+    {
+        var (service, dbName) = Seeded();
+        var rv = RowVersionOf(dbName, RawTreeTestDb.FolderedDriverId);
+
+        const string config =
+            "{\"Password\":\"env:OPCUA_PW\",\"UserCertificatePassword\":\"file:/run/secrets/certpw\"}";
+
+        var result = await service.UpdateDriverAsync(
+            RawTreeTestDb.FolderedDriverId, "modbus-1", config, null, rv);
+        result.Ok.ShouldBeTrue();
+
+        var reloaded = await service.LoadDriverForEditAsync(RawTreeTestDb.FolderedDriverId);
+        reloaded!.DriverConfig.ShouldBe(config);
+        reloaded.DriverConfig.ShouldContain("env:OPCUA_PW");
+        reloaded.DriverConfig.ShouldContain("file:/run/secrets/certpw");
+    }
+}

tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairConvergenceTests.cs

@@ -0,0 +1,226 @@
+using Xunit;
+using ZB.MOM.WW.LocalDb;
+using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.LocalDb;
+
+/// <summary>
+///     LocalDb Phase 1 (Task 12) — two driver nodes replicating the deployment-artifact cache over a
+///     real loopback h2c transport, through the real fail-closed interceptor.
+/// </summary>
+/// <remarks>
+///     This is the test the whole phase exists to pass: does a redundant driver pair actually share
+///     the cached configuration a node boots from when central SQL is down? Every upstream piece —
+///     the schema, the DI wiring, the interceptor, the chunked cache — can be individually green
+///     while the pair still fails to converge.
+/// </remarks>
+[Collection("LocalDbPairConvergence")]
+public sealed class LocalDbPairConvergenceTests
+{
+    private const string Cluster = "cluster-1";
+
+    /// <summary>
+    ///     A deterministic artifact large enough to force multiple 128 KiB chunks (≈ 3 here), so the
+    ///     tests exercise the chunk-split/reassemble path rather than a single-row shortcut.
+    /// </summary>
+    private static byte[] MultiChunkArtifact(int seed, int size = 300 * 1024)
+    {
+        var bytes = new byte[size];
+        new Random(seed).NextBytes(bytes);
+        return bytes;
+    }
+
+    private static async Task<byte[]?> ReadArtifactAsync(IDeploymentArtifactCache cache, string cluster)
+    {
+        var cached = await cache.GetCurrentAsync(cluster, TestContext.Current.CancellationToken);
+        return cached?.Artifact;
+    }
+
+    [Fact]
+    public async Task ArtifactStoredOnA_ConvergesToB_ByteIdentical()
+    {
+        await using var harness = new LocalDbPairHarness();
+        await harness.StartAsync();
+
+        var deploymentId = Guid.NewGuid().ToString("N");
+        var artifact = MultiChunkArtifact(seed: 1);
+        await harness.CacheA.StoreAsync(Cluster, deploymentId, "rev-1", artifact,
+            TestContext.Current.CancellationToken);
+
+        // B's cache reassembles byte-for-byte what A stored — proving the chunk rows, chunk_count,
+        // and pointer all replicated intact through the real transport + interceptor.
+        await LocalDbPairHarness.WaitUntilAsync(
+            async () =>
+            {
+                var onB = await ReadArtifactAsync(harness.CacheB, Cluster);
+                return onB is not null && onB.AsSpan().SequenceEqual(artifact);
+            },
+            "the artifact stored on node A to be byte-identical on node B");
+
+        // The pointer row on B carries A's origin HLC + node id, not a locally re-derived stamp: the
+        // row_version dumps for the pointer table are identical on both nodes. This is what
+        // distinguishes "B replicated A's row" from "both nodes happened to write equal content".
+        await LocalDbPairHarness.WaitUntilAsync(
+            async () =>
+                await LocalDbPairHarness.DumpRowVersionAsync(harness.A, DeploymentCacheSchema.PointerTable)
+                == await LocalDbPairHarness.DumpRowVersionAsync(harness.B, DeploymentCacheSchema.PointerTable),
+            "both nodes' pointer row_version (hlc + origin node id) to match");
+    }
+
+    [Fact]
+    public async Task RetentionPruneOnA_TombstonesReachB()
+    {
+        await using var harness = new LocalDbPairHarness();
+        await harness.StartAsync();
+
+        // Three deployments for one cluster. The cache retains the newest two, so the first is pruned
+        // on A — and that prune must replicate to B as tombstones, not linger as live chunks.
+        var dep1 = Guid.NewGuid().ToString("N");
+        var dep2 = Guid.NewGuid().ToString("N");
+        var dep3 = Guid.NewGuid().ToString("N");
+
+        await harness.CacheA.StoreAsync(Cluster, dep1, "rev-1", MultiChunkArtifact(1),
+            TestContext.Current.CancellationToken);
+        await harness.CacheA.StoreAsync(Cluster, dep2, "rev-2", MultiChunkArtifact(2),
+            TestContext.Current.CancellationToken);
+
+        // dep1's chunks are still present until the third store prunes them — confirm they reached B
+        // first, so the later disappearance is a real replicated prune rather than never-arrived.
+        await LocalDbPairHarness.WaitUntilAsync(
+            async () => await LocalDbPairHarness.ChunkCountAsync(harness.B, dep1) > 0,
+            "dep1's chunks to reach node B before the prune");
+
+        await harness.CacheA.StoreAsync(Cluster, dep3, "rev-3", MultiChunkArtifact(3),
+            TestContext.Current.CancellationToken);
+
+        // A pruned dep1 locally; B must follow via replicated deletes: dep1 gone, tombstones present,
+        // and the surviving newest (dep3) intact.
+        await LocalDbPairHarness.WaitUntilAsync(
+            async () =>
+                await LocalDbPairHarness.ChunkCountAsync(harness.B, dep1) == 0
+                && await LocalDbPairHarness.TombstoneCountAsync(harness.B, DeploymentCacheSchema.ArtifactsTable) > 0
+                && await LocalDbPairHarness.ChunkCountAsync(harness.B, dep3) > 0,
+            "node B to prune dep1 (with tombstones) while keeping dep3");
+    }
+
+    [Fact]
+    public async Task WipedNode_IsBackFilledByItsPeer_WithoutAnyNewDeploy()
+    {
+        // The Phase 1 live gate's check 4, as an offline scenario. A node whose LocalDb volume is
+        // lost used to come back permanently empty: the pair had already converged, so every oplog
+        // row was acked and pruned, and the healthy node had no delta left to send. Nothing short of
+        // a fresh deploy could repopulate it — which is the worst possible moment to depend on
+        // central being reachable, since the cache exists precisely for when it is not.
+        //
+        // Fixed in ZB.MOM.WW.LocalDb 0.1.2 (snapshot resync now measures the peer's gap against
+        // last_acked_seq when the oplog is empty). Pin the payoff at this level, not just the
+        // library's: what matters here is that the deployment artifact itself comes back.
+        await using var harness = new LocalDbPairHarness();
+        await harness.StartAsync();
+
+        var deploymentId = Guid.NewGuid().ToString("N");
+        var artifact = MultiChunkArtifact(seed: 20);
+        await harness.CacheA.StoreAsync(Cluster, deploymentId, "rev-1", artifact,
+            TestContext.Current.CancellationToken);
+
+        await LocalDbPairHarness.WaitUntilAsync(
+            async () => await ReadArtifactAsync(harness.CacheB, Cluster) is { } b && b.AsSpan().SequenceEqual(artifact),
+            "the artifact to converge before the wipe");
+
+        // Convergence alone is not the precondition — an emptied oplog on A is. That is the state
+        // that used to make back-fill impossible, so assert it rather than assume it.
+        await LocalDbPairHarness.WaitUntilAsync(
+            async () => await LocalDbPairHarness.OplogDepthAsync(harness.A) == 0,
+            "node A's oplog to be fully acked and pruned");
+
+        await harness.WipePassiveAsync();
+        Assert.Null(await ReadArtifactAsync(harness.CacheB, Cluster));
+
+        await harness.RestartPairAsync();
+
+        // No new deploy, no central: the rebuilt node gets its cached configuration back from its
+        // peer alone, byte-identical, and can boot from cache again.
+        await LocalDbPairHarness.WaitUntilAsync(
+            async () =>
+            {
+                var cached = await harness.CacheB.GetCurrentAsync(Cluster, TestContext.Current.CancellationToken);
+                return cached is not null
+                       && cached.DeploymentId == deploymentId
+                       && cached.Artifact.AsSpan().SequenceEqual(artifact);
+            },
+            "the wiped node to be back-filled from its peer");
+    }
+
+    [Fact]
+    public async Task WritesWhileTransportDown_SurviveRejoin()
+    {
+        await using var harness = new LocalDbPairHarness();
+        await harness.StartAsync();
+
+        var dep1 = Guid.NewGuid().ToString("N");
+        var first = MultiChunkArtifact(seed: 10);
+        await harness.CacheA.StoreAsync(Cluster, dep1, "rev-1", first,
+            TestContext.Current.CancellationToken);
+        await LocalDbPairHarness.WaitUntilAsync(
+            async () => await ReadArtifactAsync(harness.CacheB, Cluster) is { } b && b.AsSpan().SequenceEqual(first),
+            "the first artifact to converge before the outage");
+
+        // B goes offline; A keeps deploying. B's database survives the host teardown (pre-constructed
+        // instance), so the rejoin is genuine rather than a fresh node.
+        await harness.StopPassiveAsync();
+
+        var dep2 = Guid.NewGuid().ToString("N");
+        var second = MultiChunkArtifact(seed: 11);
+        await harness.CacheA.StoreAsync(Cluster, dep2, "rev-2", second,
+            TestContext.Current.CancellationToken);
+
+        await harness.RestartPairAsync();
+
+        // Everything written during the outage catches up: B now serves the newer deployment.
+        await LocalDbPairHarness.WaitUntilAsync(
+            async () =>
+            {
+                var cached = await harness.CacheB.GetCurrentAsync(Cluster, TestContext.Current.CancellationToken);
+                return cached is not null
+                       && cached.DeploymentId == dep2
+                       && cached.Artifact.AsSpan().SequenceEqual(second);
+            },
+            "node B to catch up on the deployment written while it was offline");
+    }
+
+    [Fact]
+    public async Task WrongApiKey_NeverConverges_WithMatchingKeyPositiveControl()
+    {
+        // Negative half: A and B hold different keys, so B's interceptor fail-closes on A's stream.
+        await using (var mismatched = new LocalDbPairHarness(
+            apiKeyA: LocalDbPairHarness.DefaultApiKey, apiKeyB: "a-different-key-entirely"))
+        {
+            await mismatched.StartAsync();
+
+            var deploymentId = Guid.NewGuid().ToString("N");
+            await mismatched.CacheA.StoreAsync(Cluster, deploymentId, "rev-1", MultiChunkArtifact(20),
+                TestContext.Current.CancellationToken);
+
+            var converged = await LocalDbPairHarness.HeldWithinAsync(
+                async () => await ReadArtifactAsync(mismatched.CacheB, Cluster) is not null,
+                TimeSpan.FromSeconds(5));
+
+            Assert.False(converged,
+                "a key mismatch must stop the pair converging — the interceptor is fail-closed");
+        }
+
+        // Positive control: the SAME scenario with matching keys DOES converge. Without this, the
+        // negative assertion above would pass even if convergence were broken for an unrelated reason.
+        await using var matched = new LocalDbPairHarness();
+        await matched.StartAsync();
+
+        var controlId = Guid.NewGuid().ToString("N");
+        var controlArtifact = MultiChunkArtifact(seed: 21);
+        await matched.CacheA.StoreAsync(Cluster, controlId, "rev-1", controlArtifact,
+            TestContext.Current.CancellationToken);
+
+        await LocalDbPairHarness.WaitUntilAsync(
+            async () => await ReadArtifactAsync(matched.CacheB, Cluster) is { } b && b.AsSpan().SequenceEqual(controlArtifact),
+            "the matching-key control pair to converge");
+    }
+}

tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairHarness.cs

@@ -0,0 +1,366 @@
+using System.Net;
+using Microsoft.AspNetCore.Builder;
+using Microsoft.AspNetCore.Hosting;
+using Microsoft.AspNetCore.Hosting.Server;
+using Microsoft.AspNetCore.Hosting.Server.Features;
+using Microsoft.AspNetCore.Server.Kestrel.Core;
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Logging.Abstractions;
+using Xunit;
+using ZB.MOM.WW.LocalDb;
+using ZB.MOM.WW.LocalDb.Replication;
+using ZB.MOM.WW.OtOpcUa.Host.Configuration;
+using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.LocalDb;
+
+/// <summary>
+///     Serializes the two-node convergence tests against one another: each stands up a real Kestrel
+///     h2c listener plus two SQLite files, and running them concurrently under CI contention is a
+///     flakiness risk. The rest of the assembly parallelizes normally.
+/// </summary>
+[CollectionDefinition("LocalDbPairConvergence")]
+public sealed class LocalDbPairConvergenceCollection;
+
+/// <summary>
+///     Two driver-role OtOpcUa nodes replicating the deployment-artifact cache over a REAL loopback
+///     gRPC transport (Kestrel h2c on 127.0.0.1), through the REAL fail-closed
+///     <see cref="LocalDbSyncAuthInterceptor"/>. Node A is the initiator (it dials the peer); node B
+///     is passive (it hosts <c>MapZbLocalDbSync</c>).
+/// </summary>
+/// <remarks>
+///     <para>
+///         Both databases are initialised through the production <see cref="LocalDbSetup.OnReady"/> —
+///         a hand-written schema here would prove only that the test agrees with itself. The tables,
+///         their primary keys, and the DDL→<c>RegisterReplicated</c> ordering under test are the ones
+///         the host actually runs.
+///     </para>
+///     <para>
+///         The two <see cref="ILocalDb"/> instances are owned by the harness and registered into the
+///         hosts as pre-constructed singletons. MS.DI does not dispose instances it did not create,
+///         so tearing a host down (the transport-loss scenario) leaves the databases intact and
+///         writable — which is exactly what lets node A accumulate writes while B is down.
+///     </para>
+///     <para>
+///         A real loopback socket (not an in-memory TestServer) is deliberate: killing the passive
+///         host disposes the server and closes the socket, faulting the initiator's active stream
+///         promptly. An in-memory TestServer does not model a connection drop and leaves the client
+///         hanging.
+///     </para>
+///     <para>
+///         The API keys are per-node so the wrong-key scenario can hand A and B different keys and
+///         assert non-convergence — with a matching-key positive control proving the same harness
+///         does converge when the keys agree (an absence assertion without a positive control passed
+///         vacuously in ScadaBridge).
+///     </para>
+///     <para>Offline: no docker, no external services.</para>
+/// </remarks>
+public sealed class LocalDbPairHarness : IAsyncDisposable
+{
+    /// <summary>The key both nodes share unless a test overrides one of them.</summary>
+    public const string DefaultApiKey = "otopcua-localdb-pair-convergence-key";
+
+    /// <summary>How long a scenario waits for the pair to agree (or to stay diverged) before deciding.</summary>
+    public static readonly TimeSpan ConvergeTimeout = TimeSpan.FromSeconds(30);
+
+    private readonly string _apiKeyA;
+    private readonly string _apiKeyB;
+
+    private readonly string _pathA = Path.Combine(Path.GetTempPath(), $"otopcua-pairA-{Guid.NewGuid():N}.db");
+
+    // Not readonly: WipePassiveAsync replaces node B's database wholesale, which is what a rebuilt
+    // node with a lost volume actually is — a new file, and with it a new node id.
+    private string _pathB = Path.Combine(Path.GetTempPath(), $"otopcua-pairB-{Guid.NewGuid():N}.db");
+
+    private readonly ServiceProvider _dbProviderA;
+    private ServiceProvider _dbProviderB;
+
+    private IHost? _serverHost;      // node B — passive
+    private IHost? _initiatorHost;   // node A — dials the peer
+
+    static LocalDbPairHarness() =>
+        // Grpc.Net.Client dials the loopback server over HTTP/2 cleartext (h2c).
+        AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
+
+    /// <param name="apiKeyA">Node A's replication key. Defaults to <see cref="DefaultApiKey"/>.</param>
+    /// <param name="apiKeyB">Node B's replication key. Defaults to <see cref="DefaultApiKey"/>.</param>
+    public LocalDbPairHarness(string? apiKeyA = null, string? apiKeyB = null)
+    {
+        _apiKeyA = apiKeyA ?? DefaultApiKey;
+        _apiKeyB = apiKeyB ?? DefaultApiKey;
+
+        _dbProviderA = BuildDatabaseProvider(_pathA);
+        _dbProviderB = BuildDatabaseProvider(_pathB);
+    }
+
+    /// <summary>Node A — the initiator, which dials the peer.</summary>
+    public ILocalDb A => _dbProviderA.GetRequiredService<ILocalDb>();
+
+    /// <summary>Node B — the passive node, which listens.</summary>
+    public ILocalDb B => _dbProviderB.GetRequiredService<ILocalDb>();
+
+    /// <summary>The production artifact cache over node A's database.</summary>
+    public IDeploymentArtifactCache CacheA =>
+        new LocalDbDeploymentArtifactCache(A, NullLogger<LocalDbDeploymentArtifactCache>.Instance);
+
+    /// <summary>The production artifact cache over node B's database.</summary>
+    public IDeploymentArtifactCache CacheB =>
+        new LocalDbDeploymentArtifactCache(B, NullLogger<LocalDbDeploymentArtifactCache>.Instance);
+
+    // ---- lifecycle --------------------------------------------------------------------------
+
+    /// <summary>Forces both databases to construct (running OnReady) then brings the pair online.</summary>
+    public async Task StartAsync()
+    {
+        _ = A;
+        _ = B;
+
+        await StartPassiveAsync();
+        await StartInitiatorAsync();
+    }
+
+    /// <summary>Takes node B's listener down, leaving its database intact and writable.</summary>
+    public async Task StopPassiveAsync()
+    {
+        await StopHostAsync(_serverHost);
+        _serverHost = null;
+    }
+
+    /// <summary>
+    ///     Destroys node B's database and replaces it with an empty one, leaving the pairing config
+    ///     untouched — the rebuilt-node case: a lost volume, a re-imaged host, a fresh container.
+    /// </summary>
+    /// <remarks>
+    ///     A new file, not a truncated one, because that is what the real thing is: the node id lives
+    ///     in the database, so a rebuilt node comes back with a new identity and a zero watermark. The
+    ///     caller follows with <see cref="RestartPairAsync"/> to bring the wiped node back online.
+    /// </remarks>
+    public async Task WipePassiveAsync()
+    {
+        await StopPassiveAsync();
+        await _dbProviderB.DisposeAsync();
+
+        // Pooled connections outlive the provider; without this the delete silently no-ops on
+        // Windows and the "wiped" node would still hold its rows.
+        Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
+        DeleteDatabaseFiles(_pathB);
+
+        _pathB = Path.Combine(Path.GetTempPath(), $"otopcua-pairB-{Guid.NewGuid():N}.db");
+        _dbProviderB = BuildDatabaseProvider(_pathB);
+        _ = B;
+    }
+
+    /// <summary>
+    ///     Brings node B back on a NEW loopback port and re-dials from A. The initiator re-reads the
+    ///     peer address on each reconnect, so this is a genuine rejoin over the same databases.
+    /// </summary>
+    public async Task RestartPairAsync()
+    {
+        await StartPassiveAsync();
+        await StopHostAsync(_initiatorHost);
+        _initiatorHost = null;
+        await StartInitiatorAsync();
+    }
+
+    // ---- convergence helpers ----------------------------------------------------------------
+
+    /// <summary>Polls <paramref name="condition"/> until true or the deadline passes; fails otherwise.</summary>
+    public static async Task WaitUntilAsync(Func<Task<bool>> condition, string because)
+    {
+        var deadline = DateTime.UtcNow + ConvergeTimeout;
+        while (DateTime.UtcNow < deadline)
+        {
+            if (await condition())
+                return;
+            await Task.Delay(50);
+        }
+
+        Assert.Fail($"Timed out after {ConvergeTimeout.TotalSeconds:0}s waiting for: {because}");
+    }
+
+    /// <summary>Waits out a bounded window and returns whether <paramref name="condition"/> ever held.</summary>
+    /// <remarks>
+    ///     For the negative half of the wrong-key scenario: it must NOT converge. A short, fixed
+    ///     window keeps the test quick while still giving a matching-key control ample time to
+    ///     converge (the control uses <see cref="WaitUntilAsync"/> with the full timeout).
+    /// </remarks>
+    public static async Task<bool> HeldWithinAsync(Func<Task<bool>> condition, TimeSpan window)
+    {
+        var deadline = DateTime.UtcNow + window;
+        while (DateTime.UtcNow < deadline)
+        {
+            if (await condition())
+                return true;
+            await Task.Delay(50);
+        }
+
+        return false;
+    }
+
+    /// <summary>
+    ///     Dumps the <c>__localdb_row_version</c> rows for one table, ordered by pk, as
+    ///     <c>pk_json|hlc|node_id|is_tombstone</c>. Equal dumps on both nodes prove B holds A's
+    ///     origin-stamped row (same HLC + node id), not a locally re-derived one.
+    /// </summary>
+    public static async Task<string> DumpRowVersionAsync(ILocalDb db, string table)
+    {
+        var rows = await db.QueryAsync(
+            """
+            SELECT pk_json, hlc, node_id, is_tombstone
+            FROM __localdb_row_version
+            WHERE table_name = @Table
+            ORDER BY pk_json
+            """,
+            static r => $"{r.GetString(0)}|{r.GetInt64(1)}|{r.GetString(2)}|{r.GetInt64(3)}",
+            new { Table = table });
+        return string.Join("\n", rows);
+    }
+
+    /// <summary>Counts <c>deployment_artifacts</c> chunk rows for one deployment on a node.</summary>
+    public static async Task<long> ChunkCountAsync(ILocalDb db, string deploymentId)
+    {
+        var rows = await db.QueryAsync(
+            "SELECT COUNT(*) FROM deployment_artifacts WHERE deployment_id = @DeploymentId",
+            static r => r.GetInt64(0),
+            new { DeploymentId = deploymentId });
+        return rows[0];
+    }
+
+    /// <summary>
+    ///     Counts oplog rows on a node. Zero means every local change has been acked by the peer and
+    ///     pruned — the steady state of a converged pair, and the state from which a wiped peer can
+    ///     only be healed by a snapshot.
+    /// </summary>
+    public static async Task<long> OplogDepthAsync(ILocalDb db)
+    {
+        var rows = await db.QueryAsync("SELECT COUNT(*) FROM __localdb_oplog", static r => r.GetInt64(0));
+        return rows[0];
+    }
+
+    /// <summary>Counts tombstone rows in <c>__localdb_row_version</c> for one table on a node.</summary>
+    public static async Task<long> TombstoneCountAsync(ILocalDb db, string table)
+    {
+        var rows = await db.QueryAsync(
+            """
+            SELECT COUNT(*) FROM __localdb_row_version
+            WHERE table_name = @Table AND is_tombstone = 1
+            """,
+            static r => r.GetInt64(0),
+            new { Table = table });
+        return rows[0];
+    }
+
+    // ---- fixture internals ------------------------------------------------------------------
+
+    /// <summary>
+    ///     A provider owning one deployment-cache database, initialised through the host's own
+    ///     <see cref="LocalDbSetup.OnReady"/> — same schema, same registration order.
+    /// </summary>
+    private static ServiceProvider BuildDatabaseProvider(string path)
+    {
+        var config = new ConfigurationBuilder()
+            .AddInMemoryCollection(new Dictionary<string, string?> { ["LocalDb:Path"] = path })
+            .Build();
+
+        return new ServiceCollection()
+            .AddZbLocalDb(config, LocalDbSetup.OnReady)
+            .BuildServiceProvider();
+    }
+
+    private IConfiguration ReplicationConfig(string apiKey, string? peerAddress)
+    {
+        var values = new Dictionary<string, string?>
+        {
+            // Tight flush + bounded reconnect backoff so convergence is observable well inside the
+            // poll deadline. The 60 s production default would let the doubling backoff overrun it
+            // after a peer outage.
+            ["LocalDb:Replication:FlushInterval"] = "00:00:00.050",
+            ["LocalDb:Replication:ReconnectBackoffMax"] = "00:00:02",
+            ["LocalDb:Replication:ApiKey"] = apiKey,
+        };
+
+        if (peerAddress is not null)
+            values["LocalDb:Replication:PeerAddress"] = peerAddress;
+
+        return new ConfigurationBuilder().AddInMemoryCollection(values).Build();
+    }
+
+    /// <summary>Starts node B, the passive listener, behind the real auth interceptor.</summary>
+    private async Task StartPassiveAsync()
+    {
+        var config = ReplicationConfig(_apiKeyB, peerAddress: null);
+
+        _serverHost = await new HostBuilder()
+            .ConfigureWebHost(web =>
+            {
+                web.UseKestrel(o =>
+                    o.Listen(IPAddress.Loopback, 0, listen => listen.Protocols = HttpProtocols.Http2));
+                web.ConfigureServices(services =>
+                {
+                    services.AddLogging();
+                    services.AddRouting();
+                    // The REAL interceptor, not a stand-in. If it rejected legitimate peer traffic,
+                    // every matching-key scenario would fail — which is the point.
+                    services.AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>());
+                    services.AddSingleton(B);
+                    services.AddZbLocalDbReplication(config);
+                })
+                .Configure(app =>
+                {
+                    app.UseRouting();
+                    app.UseEndpoints(e => e.MapZbLocalDbSync());
+                });
+            })
+            .StartAsync();
+    }
+
+    /// <summary>Starts node A, which dials the passive node.</summary>
+    private async Task StartInitiatorAsync()
+    {
+        var config = ReplicationConfig(_apiKeyA, PassiveAddress());
+
+        _initiatorHost = await new HostBuilder()
+            .ConfigureServices(services =>
+            {
+                services.AddLogging();
+                services.AddSingleton(A);
+                services.AddZbLocalDbReplication(config);
+            })
+            .StartAsync();
+    }
+
+    private string PassiveAddress()
+        => _serverHost!.Services.GetRequiredService<IServer>()
+            .Features.Get<IServerAddressesFeature>()!.Addresses.Single();
+
+    private static async Task StopHostAsync(IHost? host)
+    {
+        if (host is null)
+            return;
+        try { await host.StopAsync(TimeSpan.FromSeconds(5)); } catch { /* teardown */ }
+        host.Dispose();
+    }
+
+    public async ValueTask DisposeAsync()
+    {
+        await StopHostAsync(_initiatorHost);
+        await StopHostAsync(_serverHost);
+        await _dbProviderA.DisposeAsync();
+        await _dbProviderB.DisposeAsync();
+
+        Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
+        DeleteDatabaseFiles(_pathA);
+        DeleteDatabaseFiles(_pathB);
+    }
+
+    /// <summary>Deletes a SQLite database and its WAL/shm sidecars, best effort.</summary>
+    private static void DeleteDatabaseFiles(string path)
+    {
+        foreach (var suffix in new[] { "", "-wal", "-shm" })
+        {
+            try { File.Delete(path + suffix); } catch { /* best effort */ }
+        }
+    }
+}