test(harness): serialize Host.IntegrationTests; drop the reconciler's unused singleton proxy

Closes the last branch-only failures. Measured across five full-suite runs on
this branch versus two on master in a clean worktree: master failed only AbCip
in this assembly (2/2), while the branch additionally failed one or two E2E
deploy tests, rotating between DeployHappyPath, DriverReconnect and
EquipmentNamespaceMaterialization (4/4). Each passed in isolation, and the
assembly on its own was 195/201 throughout — the failures only appeared under
full-suite CPU contention.

Cause is contention, not correctness. Every test here builds a real two-node
Akka cluster — two Kestrel hosts, two ActorSystems with remoting, six admin
singletons, an EF context, a deploy pipeline — and xUnit ran them concurrently
with each other and with every other assembly. That was survivable while the
coordinator sealed instantly on an empty expected-ack set; now that it waits for
a real ack from every configured node, these tests measure an end-to-end
round-trip and starve.

Serialising this one assembly makes them deterministic: 195/201 with only
AbCip_Green_AgainstSim, and the full-solution failure set is now IDENTICAL to
master's — 7 shared, zero branch-only. Cost is wall-clock for this assembly,
40s -> 3m35s; it is a handful of heavyweight E2E tests, not a broad unit suite,
and a widened timeout alone did not fix it (tried first, at 45s).

Also drops createProxyToo for the address reconciler. Nothing resolves
ClusterNodeAddressReconcilerKey from the registry — the actor only reads cluster
state and logs — so the proxy was a second actor per node hunting for a
singleton nobody addresses. Kept because it removes dead machinery, not because
it fixed the flakiness; measured on its own, it did not.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

CLAUDE.md

@@ -219,6 +219,64 @@ The server supports configurable OPC UA transport security via the `OpcUa:Enable
 
 The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide.
 
+**Two corrections landed 2026-07-21 — both were total-outage or wrong-node defects, and both are worth knowing before touching this area.**
+
+- **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is still open** — the pathology only appears 1-vs-1 and docker-dev is a single six-node mesh, so the crash-the-oldest drill is deferred to the per-cluster mesh work.
+- **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**.
+
+**A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node’s own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering".
+
+Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
+
+**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2–7 not started).
+
+**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
+
+- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag.
+- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out.
+- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
+- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever.
+
+## LocalDb pair-local store (Phases 1 + 2)
+
+Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
+LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**:
+`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2).
+
+**Phase 1** caches the
+**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a
+driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes
+the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`,
+in `Runtime/Deployment/`) and reads it as a boot fallback, flagging running-from-cache. Wired by
+`AddOtOpcUaLocalDb` in the `hasDriver` branch only (`Host/Configuration/LocalDbRegistration.cs` +
+`LocalDbSetup.OnReady`, whose DDL→`RegisterReplicated` order is load-bearing); **admin-only nodes
+never register it.** The cache can optionally replicate to the node's redundant pair peer over the
+library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAuthInterceptor`), gated
+on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the
+docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is
+the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret
+exception).
+
+**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db`
+and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm
+history no longer takes it to the grave. `SqliteStoreAndForwardSink` became
+`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key
+`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely
+to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy
+commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the
+sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that
+`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new
+`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by
+design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the
+same event accepted on both nodes — which happens in every boot window, since
+`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the
+docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a
+deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.
+
+See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
+cache), `docs/AlarmHistorian.md`, and the runbook
+`docs/operations/2026-07-20-localdb-pair-replication.md`.
+
 ## LDAP Authentication
 
 The server uses LDAP-based user authentication via the `Security:Ldap` section in `appsettings.json`. When enabled, credentials are validated by LDAP bind against a GLAuth server, and LDAP group membership maps to OPC UA permissions: `ReadOnly` (browse/read), `WriteOperate` (write FreeAccess/Operate attributes), `WriteTune` (write Tune attributes), `WriteConfigure` (write Configure attributes), `AlarmAck` (alarm acknowledgment). `LdapOpcUaUserAuthenticator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`) implements `IOpcUaUserAuthenticator`, delegating the LDAP bind + group lookup to `OtOpcUaLdapAuthService` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs`, an `ILdapAuthService`). See `docs/security.md` for the full guide.
@@ -321,11 +379,13 @@ authorization uses the standard `AccessLevels.HistoryRead` bit set at materializ
 ### Alarm-history path (`AlarmHistorian` section)
 
 Alarm events are written through `GatewayAlarmHistorianWriter` (the gateway **`SendEvent`** path) behind
-the durable **`SqliteStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink`
-default for the SQLite store-and-forward queue, whose drain worker forwards batches to the gateway and uses
-per-event outcomes to decide retry vs. dead-letter (never throws). The `AlarmHistorian` section carries
-only the `Enabled` gate + the SQLite knobs (`DatabasePath`, `DrainIntervalSeconds`, `Capacity`,
-`DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection
+the durable **`LocalDbStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink`
+default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the
+replicated `alarm_sf_events` table (see the LocalDb section) and whose drain worker forwards batches to
+the gateway and uses per-event outcomes to decide retry vs. dead-letter (never throws). **Only the node
+holding the Primary role drains** (a gated node reports `HistorianDrainState.NotPrimary`). The
+`AlarmHistorian` section carries only the `Enabled` gate + the queue knobs (`DrainIntervalSeconds`,
+`Capacity`, `DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection
 (endpoint/key/TLS) is sourced from the `ServerHistorian` section. **Alarm-history `ReadEvents` requires the
 target gateway deployed with `RuntimeDb:EventReadsEnabled=true`** (the C2 SQL event-read workaround).
 
@@ -397,11 +457,16 @@ The `AlarmHistorian` section's old Wonderware connection keys (`Host`/`Port`/`Us
 were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from
 `ServerHistorian`. See `docs/Historian.md` for the full guide.
 
-### KNOWN LIMITATION 1 — live-validation gate (do before merging/trusting the cutover)
+### Live-validation gate — PASSED (was "KNOWN LIMITATION 1")
 
-The cutover is code-complete but **must be live-validated against a real gateway** (VPN to
-`wonder-sql-vd03`, gateway running the prerequisites above) before it is merged or trusted. Run the
-env-gated suite:
+**No longer a limitation.** The cutover was live-validated against the real gateway on
+`wonder-sql-vd03`: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a *confirmed
+protocol limitation* (the captured `CM_EVENT` wire never carries `SourceName`, so the gateway cannot
+populate `Source_Object` — not a fixable bug). Evidence:
+`docs/plans/2026-06-27-otopcua-historian-followups.md`.
+
+Kept here because it is the recipe to **re-run** the gate after any gateway or backend change (VPN to
+`wonder-sql-vd03`, gateway running the prerequisites above):
 
 ```bash
 export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222   # absolute gateway URI; absent ⇒ all live tests skip
@@ -416,7 +481,7 @@ dotnet test --filter "Category=LiveIntegration"
 The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the
 gate the operator runs on the VPN before trusting the cutover.
 
-### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending
+### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending (issue #491)
 
 The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer +
 meters). It is spawned with an **empty *initial* ref set** (`Array.Empty<string>()` in
@@ -430,4 +495,4 @@ deployed set of historized value tags from the first deploy onward (the feed is
 path is code-complete.** What remains is the **live gate**: an end-to-end verification against a real gateway
 (deploy → dependency-mux value delta → outbox append → `WriteLiveValues`) plus a restart-convergence test
 proving the recorder re-registers the deployed set after a process restart. Until that live verification runs,
-treat continuous value-capture as unproven-in-production rather than absent.
+treat continuous value-capture as unproven-in-production rather than absent. Tracked as **issue #491**.

Directory.Build.rsp

@@ -0,0 +1,12 @@
+# MSBuild auto-response file, applied to every build run from this repo root.
+#
+# Disable the MSBuild node pool. By default MSBuild leaves worker nodes resident
+# after a build so the next one starts warm. Across a session of repeated
+# solution-wide builds and test runs this repo accumulated 55 idle nodes holding
+# ~6 GB, which is not merely untidy: the integration suites assert on timeouts,
+# and memory pressure turns those into failures indistinguishable from real
+# regressions (see the Workstation-GC note in Host.IntegrationTests.csproj for
+# the same failure mode from a different cause).
+#
+# The cost is a small startup penalty per build. That is a good trade here.
+-nodeReuse:false

Directory.Packages.props

@@ -29,10 +29,10 @@
     <PackageVersion Include="coverlet.collector" Version="6.0.4" />
     <PackageVersion Include="FluentAssertions" Version="8.3.0" />
     <PackageVersion Include="Google.Protobuf" Version="3.34.1" />
+    <PackageVersion Include="Grpc.AspNetCore" Version="2.76.0" />
     <PackageVersion Include="Grpc.Core.Api" Version="2.76.0" />
     <PackageVersion Include="Grpc.Net.Client" Version="2.76.0" />
     <PackageVersion Include="libplctag" Version="1.5.2" />
-    <PackageVersion Include="LiteDB" Version="5.0.21" />
     <PackageVersion Include="MessagePack" Version="2.5.301" />
     <PackageVersion Include="Microsoft.AspNetCore.Authorization" Version="10.0.7" />
     <PackageVersion Include="Microsoft.AspNetCore.DataProtection" Version="10.0.7" />
@@ -57,6 +57,7 @@
     <PackageVersion Include="Microsoft.EntityFrameworkCore.InMemory" Version="10.0.7" />
     <PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="10.0.7" />
     <PackageVersion Include="Microsoft.FASTER.Core" Version="2.6.5" />
+    <PackageVersion Include="Microsoft.Extensions.Configuration" Version="10.0.7" />
     <PackageVersion Include="Microsoft.Extensions.Configuration.Abstractions" Version="10.0.7" />
     <PackageVersion Include="Microsoft.Extensions.Configuration.Json" Version="10.0.7" />
     <PackageVersion Include="Microsoft.Extensions.DependencyInjection" Version="10.0.7" />
@@ -103,6 +104,23 @@
       clearing the advisory without CentralPackageTransitivePinningEnabled (which breaks the Roslyn split).
     -->
     <PackageVersion Include="SQLitePCLRaw.bundle_e_sqlite3" Version="2.1.12" />
+    <!--
+      Surgical transitive pin #2. Four NU1903 high-severity advisories (GHSA-23rf-6693-g89p,
+      GHSA-8q5v-6pqq-x66h, GHSA-cvvh-rhrc-wg4q, GHSA-g8r8-53c2-pm3f) landed in the NuGet audit
+      data against System.Security.Cryptography.Xml 10.0.7, which Microsoft.AspNetCore.DataProtection
+      10.0.7 pulls in transitively for its key storage. Under TreatWarningsAsErrors every FRESH
+      restore goes red — 204 NU1903 errors across the solution — while machines with cached audit
+      data keep building, so this surfaces first on a clean clone or a docker image build, not
+      locally. Verified 2026-07-21 by restoring a detached worktree.
+
+      10.0.10 is the patched version, pinned by direct PackageReference in Core.Configuration (the
+      one project where the DataProtection chain enters). Bumping the DataProtection parent to
+      10.0.10 instead was rejected in the sister repo for a reason that applies here too: it floors
+      Microsoft.Extensions.* and, through the EFCore adapter, Microsoft.EntityFrameworkCore at
+      10.0.10, forcing a family-wide servicing bump (NU1605 downgrade cascade) that deserves its own
+      reviewed change.
+    -->
+    <PackageVersion Include="System.Security.Cryptography.Xml" Version="10.0.10" />
     <PackageVersion Include="System.CommandLine" Version="2.0.5" />
     <PackageVersion Include="System.Data.SqlClient" Version="4.9.0" />
     <PackageVersion Include="System.IdentityModel.Tokens.Jwt" Version="8.11.0" />
@@ -122,6 +140,13 @@
     <PackageVersion Include="xunit.runner.visualstudio" Version="3.0.2" />
     <PackageVersion Include="xunit.v3" Version="1.1.0" />
     <PackageVersion Include="ZB.MOM.WW.Health" Version="0.1.0" />
+    <!--
+      LocalDb: embedded SQLite node-local cache + optional 2-node gRPC replication. The core
+      package floors the native SQLitePCLRaw.lib.e_sqlite3 at 2.1.12, so consumers inherit the
+      CVE-2025-6965-fixed native binary without needing the surgical pin below.
+    -->
+    <PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.3" />
+    <PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.3" />
     <PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.1.0" />
     <PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.1.0" />
     <PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />

NuGet.config

@@ -27,6 +27,8 @@
       <package pattern="ZB.MOM.WW.HistorianGateway.Client" />
       <package pattern="ZB.MOM.WW.Secrets" />
       <package pattern="ZB.MOM.WW.Secrets.*" />
+      <package pattern="ZB.MOM.WW.LocalDb" />
+      <package pattern="ZB.MOM.WW.LocalDb.*" />
     </packageSource>
   </packageSourceMapping>
 </configuration>

docker-dev/docker-compose.yml

@@ -13,7 +13,9 @@
 #   central-1, central-2   OTOPCUA_ROLES=admin,driver — the ONLY UI + deploy
 #                          singleton, plus the MAIN cluster's OPC UA publishers.
 #                          Reachable at http://localhost:9200 (via Traefik).
-#                          central-1 is the Akka seed node; central-2 joins it.
+#                          Both are Akka seed nodes, each listing ITSELF first
+#                          (2026-07-22) so either can cold-start while the other
+#                          is down; the site nodes seed off central-1 only.
 #   site-a-1, site-a-2     OTOPCUA_ROLES=driver — driver-only members of the same
 #   site-b-1, site-b-2     mesh, scoped to SITE-A / SITE-B by ClusterId. They
 #                          serve no UI and authenticate no users; the central
@@ -136,7 +138,8 @@ services:
 
   # ── Central cluster (2-node fused admin+driver) ─────────────────────────────
   # The only UI + deploy singleton; also the MAIN cluster's OPC UA publishers.
-  # central-1 seeds the single Akka mesh that every other node joins.
+  # central-1 seeds the single Akka mesh that every other node joins; central-2 is a seed too
+  # (listing itself first), so it can form the mesh alone if central-1 is down.
 
   central-1: &otopcua-host
     build:
@@ -176,8 +179,13 @@ services:
       Cluster__Port: "4053"
       Cluster__PublicHostname: "central-1"
       # Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
-      # re-join the mesh via EITHER peer, not only central-1. With restart supervision this is
-      # the 2-node keep-oldest exit-and-rejoin recovery path.
+      # re-join the mesh via EITHER peer, not only central-1.
+      #
+      # ORDER IS LOAD-BEARING (2026-07-22): each node lists ITSELF first. Akka runs
+      # FirstSeedNodeProcess — the only path that can form a NEW cluster when no peer answers
+      # InitJoin — exclusively for seed-nodes[0]; every other node retries InitJoin forever. So a
+      # node listing its partner first cannot cold-start while that partner is down. Enforced at
+      # boot by AkkaClusterOptionsValidator; see docs/Redundancy.md.
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
       Cluster__Roles__0: "admin"
@@ -224,8 +232,15 @@ services:
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # Consolidated pair-local config cache (ZB.MOM.WW.LocalDb). Every driver-role node
+      # (central is admin,driver) gets one, on a per-node named volume so the cache survives
+      # container recreates. The central pair leaves replication OFF (default) — only the
+      # site-a pair replicates, as the enablement demo; site-b is the default-OFF pin.
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4840:4840"
+    volumes:
+      - otopcua-localdb-central-1:/app/data
 
   central-2:
     <<: *otopcua-host
@@ -241,9 +256,11 @@ services:
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "central-2"
-      # Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1.
-      Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
-      Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
+      # Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST:
+      # central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1
+      # is down (it previously listed central-1 first and simply never came Up in that case).
+      Cluster__SeedNodes__0: "akka.tcp://otopcua@central-2:4053"
+      Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "admin"
       Cluster__Roles__1: "driver"
       Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
@@ -283,8 +300,12 @@ services:
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # Pair-local config cache; central pair leaves replication OFF (see central-1).
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4841:4840"
+    volumes:
+      - otopcua-localdb-central-2:/app/data
 
   # ── Site A cluster (2-node driver-only) ─────────────────────────────────────
   # Driver-only members of the single mesh, scoped to SITE-A by ClusterId. No UI,
@@ -303,6 +324,9 @@ services:
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "site-a-1"
+      # Site nodes are deliberately NOT seeds — they join the central pair's mesh. The self-first
+      # seed rule is conditional for exactly this reason (it binds only when a node's own address
+      # is in its own seed list), so these configs are exempt rather than broken.
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
       <<: *secrets-env
@@ -313,8 +337,49 @@ services:
       # Resolved at runtime by GalaxyDriver.ResolveApiKey when a DriverInstance's
       # Gateway.ApiKeySecretRef = "env:GALAXY_MXGW_API_KEY".
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # Pair-local config cache + REPLICATION ON — the site-a pair is the enablement demo.
+      # site-a-1 is the initiator: it binds the h2c sync listener on 9001 AND dials the peer.
+      # Setting SyncListenPort makes Program.cs add a dedicated Http2-only Kestrel listener and
+      # re-bind the primary HTTP port (an explicit Listen* otherwise discards ASPNETCORE_URLS).
+      # Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
+      # buffer is a live, replicated table rather than an empty one — the live gate needs a real
+      # queue to watch converge, and needs to see that only the Primary drains it.
+      #
+      # There is no HistorianGateway on this rig, so the endpoint below is deliberately
+      # unresolvable: every drain attempt fails, which is exactly the historian-outage state the
+      # buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
+      # URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
+      # AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
+      #
+      # MaxAttempts is raised far above the production default of 10. With a permanently
+      # unreachable gateway the default would dead-letter the whole queue about five minutes into
+      # the outage, turning a buffering test into a dead-letter test.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
+      # Required whenever the section is consumed: the gateway client validates its own options at
+      # construction and throws on an empty key, so a keyless node crash-loops during Akka startup
+      # rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
+      # above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
+      # ServerHistorian__ApiKey from the environment.
+      ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
+      # Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
+      # direction, so an http endpoint with the default UseTls=true crash-loops the host.
+      ServerHistorian__UseTls: "false"
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
+      LocalDb__SyncListenPort: "9001"
+      LocalDb__Replication__PeerAddress: "http://site-a-2:9001"
+      # DEV-ONLY committed key (rig philosophy — like the SQL password + secrets KEK above). It
+      # MUST be byte-identical on both nodes: the interceptor is fail-closed, so any mismatch
+      # silently stops the pair converging. NEVER reuse this key outside this local dev rig.
+      LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
+      # Row-count batching against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so
+      # 16 × 171 KB ≈ 2.7 MB stays under the cap with headroom.
+      LocalDb__Replication__MaxBatchSize: "16"
     ports:
       - "4842:4840"
+    volumes:
+      - otopcua-localdb-site-a-1:/app/data
 
   site-a-2:
     <<: *otopcua-host
@@ -334,8 +399,43 @@ services:
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # site-a-2 is the passive half of the replicating pair: it binds the sync listener on 9001
+      # but does NOT dial (no PeerAddress). The replication stream is bidirectional, so a-1's
+      # single dial carries both directions. Same key + MaxBatchSize as a-1 (byte-identical key
+      # is mandatory — the interceptor fail-closes on a mismatch).
+      # Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
+      # buffer is a live, replicated table rather than an empty one — the live gate needs a real
+      # queue to watch converge, and needs to see that only the Primary drains it.
+      #
+      # There is no HistorianGateway on this rig, so the endpoint below is deliberately
+      # unresolvable: every drain attempt fails, which is exactly the historian-outage state the
+      # buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
+      # URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
+      # AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
+      #
+      # MaxAttempts is raised far above the production default of 10. With a permanently
+      # unreachable gateway the default would dead-letter the whole queue about five minutes into
+      # the outage, turning a buffering test into a dead-letter test.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
+      # Required whenever the section is consumed: the gateway client validates its own options at
+      # construction and throws on an empty key, so a keyless node crash-loops during Akka startup
+      # rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
+      # above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
+      # ServerHistorian__ApiKey from the environment.
+      ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
+      # Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
+      # direction, so an http endpoint with the default UseTls=true crash-loops the host.
+      ServerHistorian__UseTls: "false"
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
+      LocalDb__SyncListenPort: "9001"
+      LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
+      LocalDb__Replication__MaxBatchSize: "16"
     ports:
       - "4843:4840"
+    volumes:
+      - otopcua-localdb-site-a-2:/app/data
 
   # ── Site B cluster (2-node driver-only) ─────────────────────────────────────
 
@@ -357,8 +457,28 @@ services:
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # site-b is the default-OFF pin: it gets the pair-local cache but NO SyncListenPort and NO
+      # replication config, so no sync listener binds and ISyncStatus stays disconnected/Healthy.
+      # This is what the live gate's check 8 verifies — the cache works locally with replication off.
+      # Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
+      # must work as a plain node-local buffer here, with no sync listener and no peer traffic.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
+      # Required whenever the section is consumed: the gateway client validates its own options at
+      # construction and throws on an empty key, so a keyless node crash-loops during Akka startup
+      # rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
+      # above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
+      # ServerHistorian__ApiKey from the environment.
+      ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
+      # Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
+      # direction, so an http endpoint with the default UseTls=true crash-loops the host.
+      ServerHistorian__UseTls: "false"
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4844:4840"
+    volumes:
+      - otopcua-localdb-site-b-1:/app/data
 
   site-b-2:
     <<: *otopcua-host
@@ -378,8 +498,26 @@ services:
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # site-b default-OFF pin (see site-b-1): cache on, replication off.
+      # Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
+      # must work as a plain node-local buffer here, with no sync listener and no peer traffic.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
+      # Required whenever the section is consumed: the gateway client validates its own options at
+      # construction and throws on an empty key, so a keyless node crash-loops during Akka startup
+      # rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
+      # above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
+      # ServerHistorian__ApiKey from the environment.
+      ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
+      # Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
+      # direction, so an http endpoint with the default UseTls=true crash-loops the host.
+      ServerHistorian__UseTls: "false"
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4845:4840"
+    volumes:
+      - otopcua-localdb-site-b-2:/app/data
 
   traefik:
     image: traefik:v3.1
@@ -400,3 +538,13 @@ services:
 volumes:
   # SQL Server data dir — persists the OtOpcUa ConfigDb across container recreates.
   otopcua-mssql-data:
+  # Per-node pair-local config-cache (ZB.MOM.WW.LocalDb) files. One volume per node — the
+  # cache is node-local, not shared; it persists the cached deployment artifact (and, on the
+  # site-a pair, the replicated oplog state) across container recreates so a node can boot
+  # from cache after a restart even with central SQL down.
+  otopcua-localdb-central-1:
+  otopcua-localdb-central-2:
+  otopcua-localdb-site-a-1:
+  otopcua-localdb-site-a-2:
+  otopcua-localdb-site-b-1:
+  otopcua-localdb-site-b-2:

docker-dev/seed/seed-clusters.sql

@@ -64,13 +64,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ServerCluster WHERE ClusterId = 'SITE-B')
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-1:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('central-1:4053', 'MAIN', 'central-1', 4840, 8081, 'urn:OtOpcUa:central-1', 200, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('central-1:4053', 'MAIN', 'central-1', 4840, 8081, 4053, 'urn:OtOpcUa:central-1', 200, 1, 'docker-dev-seed');
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('central-2:4053', 'MAIN', 'central-2', 4840, 8081, 'urn:OtOpcUa:central-2', 150, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('central-2:4053', 'MAIN', 'central-2', 4840, 8081, 4053, 'urn:OtOpcUa:central-2', 150, 1, 'docker-dev-seed');
 
 ------------------------------------------------------------------------------
 -- ClusterNode — site A
@@ -78,13 +78,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053')
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-1:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('site-a-1:4053', 'SITE-A', 'site-a-1', 4840, 8081, 'urn:OtOpcUa:site-a-1', 200, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('site-a-1:4053', 'SITE-A', 'site-a-1', 4840, 8081, 4053, 'urn:OtOpcUa:site-a-1', 200, 1, 'docker-dev-seed');
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('site-a-2:4053', 'SITE-A', 'site-a-2', 4840, 8081, 'urn:OtOpcUa:site-a-2', 150, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('site-a-2:4053', 'SITE-A', 'site-a-2', 4840, 8081, 4053, 'urn:OtOpcUa:site-a-2', 150, 1, 'docker-dev-seed');
 
 ------------------------------------------------------------------------------
 -- ClusterNode — site B
@@ -92,13 +92,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053')
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-1:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('site-b-1:4053', 'SITE-B', 'site-b-1', 4840, 8081, 'urn:OtOpcUa:site-b-1', 200, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('site-b-1:4053', 'SITE-B', 'site-b-1', 4840, 8081, 4053, 'urn:OtOpcUa:site-b-1', 200, 1, 'docker-dev-seed');
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-2:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 4053, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed');
 
 ------------------------------------------------------------------------------
 -- Galaxy MxAccess gateway — INTENTIONALLY NOT SEEDED (retired 2026-06-15)

docs/AlarmHistorian.md

@@ -42,7 +42,7 @@ unless noted.
 - **`NullAlarmHistorianSink`** — the no-op default for tests and deployments
   that don't historize alarms. It is the default DI binding (registered in the
   Runtime's `AddOtOpcUaRuntime`); production overrides it with
-  `SqliteStoreAndForwardSink`.
+  `LocalDbStoreAndForwardSink`.
 - **`AlarmHistorianEvent`**
   ([`AlarmHistorianEvent.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmHistorianEvent.cs))
   — the source-agnostic event record: `AlarmId`, `EquipmentPath` (UNS path,
@@ -61,42 +61,84 @@ unless noted.
 - **`HistorianSinkStatus`** — diagnostic snapshot surfaced to the AdminUI and
   `/healthz`: `QueueDepth`, `DeadLetterDepth`, `LastDrainUtc`, `LastSuccessUtc`,
   `LastError`, `DrainState`, and `EvictedCount`.
-- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff`.
+- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff` /
+  **`NotPrimary`** (ticking, but leaving the replicated queue for the node that
+  holds the Primary role).
 
 ---
 
-## SqliteStoreAndForwardSink
+## LocalDbStoreAndForwardSink
 
-[`SqliteStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs)
-is the production `IAlarmHistorianSink`. Construction takes a SQLite database
-path, an `IAlarmHistorianWriter`, a logger, and optional `batchSize` (default
-100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30 days),
-and a test clock.
+[`LocalDbStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/LocalDbStoreAndForwardSink.cs)
+is the production `IAlarmHistorianSink`. Construction takes the node's
+`ILocalDb`, an `IAlarmHistorianWriter`, a logger, and optional `batchSize`
+(default 100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30
+days), a test clock, and a **`drainGate`**.
+
+### The drain gate
+
+`drainGate` is a `Func<bool>` consulted at the top of every tick; `false` skips
+the tick entirely, leaving the queue untouched and reporting
+`DrainState = NotPrimary`.
+
+It exists because the queue **replicates**. The Secondary holds a full copy of
+the Primary's rows, and an ungated drain there would re-deliver every event,
+continuously — not just at a failover. Runtime supplies the gate from
+`IRedundancyRoleView`, a singleton `DriverHostActor` publishes its
+`PrimaryGatePolicy` verdict to on every redundancy snapshot, so the drain agrees
+with the inbound-write and native-ack gates by construction.
+
+Two postures are deliberate:
+
+- **Unset (the default) means drain.** So does an unpublished view. A deployment
+  that runs no redundancy never publishes a role, and defaulting closed would
+  silently stop its alarm history forever.
+- **A gate that throws means "not now", never permission.** Draining on a gate
+  fault would put a pair into dual delivery exactly when its redundancy signal is
+  sick.
 
 ### Queue table
 
-The sink owns one SQLite table (created on construction, WAL journal mode):
+The sink owns one table in the node's consolidated LocalDb — created and
+registered for replication by `LocalDbSetup.OnReady`, before any consumer can
+resolve the sink:
 
 ```sql
-CREATE TABLE Queue (
-    RowId          INTEGER PRIMARY KEY AUTOINCREMENT,
-    AlarmId        TEXT    NOT NULL,
-    EnqueuedUtc    TEXT    NOT NULL,
-    PayloadJson    TEXT    NOT NULL,   -- JSON-serialized AlarmHistorianEvent
-    AttemptCount   INTEGER NOT NULL DEFAULT 0,
-    LastAttemptUtc TEXT    NULL,
-    LastError      TEXT    NULL,
-    DeadLettered   INTEGER NOT NULL DEFAULT 0
+CREATE TABLE alarm_sf_events (
+    id               TEXT    NOT NULL PRIMARY KEY,   -- SHA-256 of payload_json
+    alarm_id         TEXT    NOT NULL,
+    enqueued_at_utc  TEXT    NOT NULL,
+    payload_json     TEXT    NOT NULL,   -- JSON-serialized AlarmHistorianEvent
+    attempt_count    INTEGER NOT NULL DEFAULT 0,
+    last_attempt_utc TEXT    NULL,
+    last_error       TEXT    NULL,
+    dead_lettered    INTEGER NOT NULL DEFAULT 0
 );
-CREATE INDEX IX_Queue_Drain ON Queue (DeadLettered, RowId);
+CREATE INDEX ix_alarm_sf_events_drain
+    ON alarm_sf_events (dead_lettered, enqueued_at_utc, id);
 ```
 
-`EnqueueAsync` does a single `INSERT` on the hot path. To avoid a
-`SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory non-dead-lettered
-row counter (seeded at startup, kept current by every mutation, and re-synced
-from storage every 10,000 enqueues to defend against drift). SQLite writer
-contention is handled via `PRAGMA busy_timeout=5000` + WAL so an enqueue/drain
-collision waits out the file lock instead of failing fast.
+**The primary key is a hash of the payload, not an autoincrement rowid.** Two
+reasons. Replication converges last-writer-wins *per primary key*, so two nodes
+independently allocating rowid 7 to different alarms would silently overwrite one
+another. And an equal-payload key makes the same event arriving twice collapse
+into one row — which happens for real, because `HistorianAdapterActor`
+default-writes while its redundancy role is unknown and both nodes of a pair
+therefore accept the same fanned transition in every boot window. Two genuinely
+distinct events cannot collide: `AlarmHistorianEvent` carries a full-precision
+timestamp alongside the alarm id, kind, message and user.
+
+Ordering follows from that: the drain reads in `enqueued_at_utc` order (with `id`
+as a tiebreak so the ordering is total), because a hashed key carries no
+insertion sequence.
+
+`EnqueueAsync` does a single `INSERT … ON CONFLICT DO NOTHING` on the hot path.
+To avoid a `SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory
+non-dead-lettered row counter (seeded at startup, kept current by every mutation,
+and re-synced from storage every 10,000 enqueues to defend against drift — now
+doubly warranted, since replication applies the peer's rows straight into the
+table where no local code path can observe them). Pragmas and connection
+lifetime belong to LocalDb; the sink no longer manages either.
 
 ### Drain worker
 
@@ -104,13 +146,17 @@ collision waits out the file lock instead of failing fast.
 `System.Threading.Timer`** (not started automatically — tests drive
 `DrainOnceAsync` deterministically). Each tick:
 
+0. Consults the drain gate. A closed gate returns immediately, leaving the queue
+   untouched and `DrainState = NotPrimary`.
 1. Purges aged dead-lettered rows past the retention window.
-2. Reads up to `batchSize` non-dead-lettered rows in `RowId` order.
+2. Reads up to `batchSize` non-dead-lettered rows in `enqueued_at_utc, id` order.
 3. Rows with un-deserializable payloads are dead-lettered immediately (by their
-   own `RowId`) so they can't stall the queue head.
+   own `id`) so they can't stall the queue head.
 4. The remaining batch is handed to `IAlarmHistorianWriter.WriteBatchAsync`, and
-   each outcome is applied in one transaction: `Ack` deletes the row,
-   `PermanentFail` flips its `DeadLettered` flag, `RetryPlease` bumps its attempt
+   each outcome is applied in one transaction: `Ack` deletes the row (the delete
+   replicates as a tombstone, so the peer drops its copy and a promoted standby
+   cannot re-send it), `PermanentFail` flips its `dead_lettered` flag,
+   `RetryPlease` bumps its attempt
    count and leaves it queued. A row whose `AttemptCount` has reached the configured
    **`MaxAttempts`** cap (default 10) is dead-lettered automatically on the next drain
    tick rather than retried — this breaks infinite retry loops for poison events whose
@@ -131,7 +177,7 @@ an unobserved task exception.
 
 **The durability guarantee is bounded by `capacity` (default 1,000,000 rows).**
 When the non-dead-lettered queue reaches capacity, `EnqueueAsync` evicts the
-oldest non-dead-lettered rows (oldest `RowId` first) to make room, logs a WARN,
+oldest non-dead-lettered rows (oldest `enqueued_at_utc` first) to make room, logs a WARN,
 and increments `HistorianSinkStatus.EvictedCount`. Under a sustained historian
 outage, accepted alarm events can therefore be dropped before delivery. A
 non-zero `EvictedCount` is a data-loss signal that requires operator attention —
@@ -140,7 +186,7 @@ it surfaces silent loss without log scraping.
 ### Dead-letter + operator recovery
 
 `PermanentFail` and corrupt-payload rows are retained in-place with
-`DeadLettered = 1` for the retention window (default 30 days) so operators can
+`dead_lettered = 1` for the retention window (default 30 days) so operators can
 inspect them before the sweeper purges them. `RetryDeadLettered()` is the
 operator action (from the AdminUI) that clears the dead-letter flag and attempt
 count on every dead-lettered row, returning them to the regular queue with a
@@ -173,17 +219,28 @@ the `alerts` topic (future).
 The real sink is opt-in via the `AlarmHistorian` section of `appsettings.json`.
 When `Enabled` is `false` (the default), `AddAlarmHistorian` registers
 `NullAlarmHistorianSink` and the feature is dormant. When `Enabled` is `true`,
-`AddAlarmHistorian` constructs `SqliteStoreAndForwardSink` and registers
+`AddAlarmHistorian` constructs `LocalDbStoreAndForwardSink` and registers
 `GatewayAlarmHistorianWriter` as the `IAlarmHistorianWriter`. This section carries
-**only** the `Enabled` gate + the SQLite store-and-forward knobs — the downstream
+**only** the `Enabled` gate + the store-and-forward knobs — the downstream
 gateway connection (endpoint / key / TLS) is sourced from the `ServerHistorian`
-section (see [Historian.md](Historian.md)).
+section (see [Historian.md](Historian.md)), and the queue's storage location is the
+node's consolidated `LocalDb:Path` database (see
+[Configuration.md](Configuration.md)).
+
+> **Where the queue lives (changed).** The buffer is a table — `alarm_sf_events` — in the node's
+> consolidated LocalDb, not a standalone `alarm-historian.db`. That is what lets it **replicate to
+> the redundant pair peer**, so undelivered alarm history survives losing the node holding it.
+> Two consequences follow, both covered in the
+> [pair-replication runbook](operations/2026-07-20-localdb-pair-replication.md):
+> **only the Primary drains** (the Secondary holds a full replica and would otherwise re-send
+> everything), and **delivery is at-least-once across a failover** by design. On first boot after
+> the upgrade, `AlarmSfLegacyMigrator` copies any existing `alarm-historian.db` across and renames
+> it `.migrated`.
 
 ```json
 {
   "AlarmHistorian": {
     "Enabled": true,
-    "DatabasePath": "C:\\ProgramData\\OtOpcUa\\alarmhistorian.db",
     "BatchSize": 100,
     "DrainIntervalSeconds": 5,
     "Capacity": 1000000,
@@ -194,8 +251,7 @@ section (see [Historian.md](Historian.md)).
 
 | Key | Type | Default | Description |
 |---|---|---|---|
-| `Enabled` | bool | `false` | Enable the SQLite store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. |
-| `DatabasePath` | string | `alarm-historian.db` | Path to the SQLite queue file. Created on first use (WAL mode). Set an **absolute** path in production. |
+| `Enabled` | bool | `false` | Enable the durable store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. Requires LocalDb to be registered — an enabled sink with no `ILocalDb` fails at resolution rather than silently discarding events. |
 | `BatchSize` | int | `100` | Max rows per drain cycle handed to `IAlarmHistorianWriter.WriteBatchAsync`. |
 | `DrainIntervalSeconds` | int | `5` | Seconds between drain-worker ticks. |
 | `Capacity` | long | `1000000` | Max queued rows before the sink evicts the oldest (data-loss signal via `EvictedCount`). |
@@ -207,6 +263,15 @@ section (see [Historian.md](Historian.md)).
 > `RuntimeDb:EventReadsEnabled=true`. The old Wonderware connection keys (`SharedSecret` /
 > `AlarmHistorian:Host`/`Port`/`UseTls`/`ServerCertThumbprint`) were pruned.
 
+> **`DatabasePath` was removed.** The queue no longer owns a file. The key is still *read*, by
+> `AlarmSfLegacyMigrator`, purely to locate a pre-consolidation `alarm-historian.db` to migrate —
+> so leave it in place through the upgrade, then delete it once every node has an
+> `alarm-historian.db.migrated` sidecar. A leftover value configures nothing.
+
+> **`DrainState` gained `NotPrimary`.** A gated-off Secondary reports it instead of `Idle`, so a
+> rising queue depth there is distinguishable from a stalled drain. Both nodes reporting it means
+> nobody is draining — see the runbook.
+
 > Dev and docker-dev deployments leave `Enabled` unset (defaults to `false`) so alarm transitions historize to nowhere unless a HistorianGateway is configured.
 
 ---
@@ -221,3 +286,5 @@ section (see [Historian.md](Historian.md)).
 - [ScriptedAlarms.md](ScriptedAlarms.md) — the scripted-alarm engine that emits
   most events into this sink.
 - [ServiceHosting.md](ServiceHosting.md) — the external HistorianGateway backend.
+- [operations/2026-07-20-localdb-pair-replication.md](operations/2026-07-20-localdb-pair-replication.md)
+  — where the queue lives, the Primary-only drain, and the one-time legacy migration.

docs/AlarmTracking.md

@@ -365,12 +365,16 @@ AB CIP ALMD) route to AVEVA Historian via the HistorianGateway:
 - `IAlarmHistorianSink` is the DI-registered intake contract. The
   default binding is `NullAlarmHistorianSink` (registered in
   `ServiceCollectionExtensions.AddOtOpcUaRuntime`). Production
-  deployments override it with `SqliteStoreAndForwardSink` wrapping
+  deployments override it with `LocalDbStoreAndForwardSink` wrapping
   `GatewayAlarmHistorianWriter` (the HistorianGateway `SendEvent` path)
   — see [ServiceHosting.md](ServiceHosting.md) for the HistorianGateway setup.
-- `SqliteStoreAndForwardSink` queues each transition to a local
-  SQLite database and drains in the background via an
-  `IAlarmHistorianWriter`. **The durability guarantee is bounded**: the
+- `LocalDbStoreAndForwardSink` queues each transition into the node's
+  consolidated LocalDb — where it **replicates to the redundant pair peer**,
+  so undelivered alarm history survives losing the node holding it — and
+  drains in the background via an `IAlarmHistorianWriter`. **Only the node
+  holding the Primary role drains**, since the peer holds a full replica;
+  delivery is consequently at-least-once across a failover, by design. See
+  [AlarmHistorian.md](AlarmHistorian.md). **The durability guarantee is bounded**: the
   queue capacity defaults to 1,000,000 rows; under a sustained
   historian outage, older non-dead-lettered rows are evicted (oldest
   first) to make room for new events. The `HistorianSinkStatus.EvictedCount`
@@ -379,8 +383,9 @@ AB CIP ALMD) route to AVEVA Historian via the HistorianGateway:
   capacity, and dead-letter retention are tunable via the `AlarmHistorian`
   config section (`DrainIntervalSeconds`, `Capacity`,
   `DeadLetterRetentionDays`); `AlarmHistorianOptions.Validate()` logs a
-  startup warning for an empty `SharedSecret`, a relative `DatabasePath`,
-  or a non-positive knob.
+  startup warning for a non-positive knob. (It no longer warns about
+  `SharedSecret` or `DatabasePath` — both keys are gone: the connection
+  comes from `ServerHistorian`, and the queue lives in the LocalDb.)
 - `HistorianAdapterActor`
   (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs`)
   subscribes to the cluster `alerts` DPS topic, translates each

docs/Configuration.md

@@ -101,13 +101,15 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
 |---|---|---|---|
 | `SystemName` | string | `otopcua` | Akka actor-system name. |
 | `Hostname` | string | `0.0.0.0` | Bind hostname. |
-| `Port` | int | `4053` | Cluster transport port. |
-| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. |
-| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. |
+| `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. |
+| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. |
+| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). |
 | `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`. |
 
 > The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.
 
+> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
+
 ### `ConnectionStrings` → `ConfigDb`
 
 - **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally.
@@ -136,8 +138,9 @@ The historian backend is the external **`ZB.MOM.WW.HistorianGateway`** sidecar,
 `ZB.MOM.WW.HistorianGateway.Client` gRPC package (the retired Wonderware TCP sidecar is documented at
 [`docs/drivers/Historian.Wonderware.md`](drivers/Historian.Wonderware.md)). The OtOpcUa host reads three
 appsettings sections — `ServerHistorian` (read path + gateway connection), `ContinuousHistorization`
-(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (SQLite store-and-forward
-alarm sink draining to `SendEvent`). The gateway connection (endpoint / key / TLS) lives **only** in
+(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (the store-and-forward
+alarm sink draining to `SendEvent`; its buffer lives in the node's `LocalDb:Path` database, not a
+file of its own). The gateway connection (endpoint / key / TLS) lives **only** in
 `ServerHistorian`; the other two sections source it from there.
 
 The gateway API key is supplied via the environment variable **`ServerHistorian__ApiKey`** — never committed
@@ -146,6 +149,28 @@ key must carry the scopes `historian:read`, `historian:write`, `historian:tags:w
 [`docs/Historian.md`](Historian.md) for the full key reference, the migration note (old Wonderware keys →
 gateway keys), and the deployment prerequisites.
 
+### `LocalDb` (pair-local config cache — driver role only)
+
+Driver-role nodes keep a consolidated **`ZB.MOM.WW.LocalDb`** SQLite database that caches the deployed
+configuration artifact (chunked) so the node can **boot from cache when central SQL is unreachable**. The
+section is bound by `AddOtOpcUaLocalDb` (`Host/Configuration/LocalDbRegistration.cs`) inside the `hasDriver`
+branch — **admin-only nodes never register it, and never require `LocalDb:Path`**. Storage is unconditional;
+replication stays inert until the sync port + peer are set.
+
+| Key | Default | Meaning |
+|---|---|---|
+| `LocalDb:Path` | `./data/otopcua-localdb.db` | SQLite file path. `ValidateOnStart`-required once `AddZbLocalDb` runs (driver nodes only). In containers put it on a durable volume. |
+| `LocalDb:SyncListenPort` | `0` | `0` = replication off (no sync listener). A non-zero port binds a dedicated **h2c** sync listener; set the **same** port on both pair nodes. Setting it makes the Host add an explicit Kestrel `Listen*` and re-bind the primary HTTP port (an explicit `Listen*` otherwise discards `ASPNETCORE_URLS`). |
+| `LocalDb:Replication:PeerAddress` | `""` | The peer this node **dials** (`http://<peer>:<port>`). Set on the initiator only; leave empty on the passive node. The stream is bidirectional. |
+| `LocalDb:Replication:ApiKey` | `""` | Bearer token for the sync stream. **Fail-closed:** no key ⇒ the passive endpoint refuses everything; a mismatch ⇒ the pair silently stops converging. **Must be byte-identical on both nodes.** Supply via `${secret:...}` / env — never a cleartext literal in production. |
+| `LocalDb:Replication:MaxBatchSize` | (library default) | Rows per replication batch — **row-count-only** against gRPC's 4 MB cap. `16` for the artifact cache (chunk rows ≈171 KB, so 16 × 171 KB ≈ 2.7 MB). |
+
+Replication is **default-OFF** across the fleet; it is enabled per-pair as an opt-in. See
+[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
+for the enablement runbook (ApiKey handling, stop/start-together rule, tombstone-retention window, and the
+never-`sqlite3`-a-live-WAL-DB inspection rules) and the "pair-local config cache" section of
+[`docs/Redundancy.md`](Redundancy.md) for what boot-from-cache does and does not cover.
+
 ---
 
 ## Environment variables

docs/README.md

@@ -30,7 +30,7 @@ The project was originally called **LmxOpcUa** (a single-driver Galaxy/MXAccess
 | [Subscriptions.md](v1/Subscriptions.md) | Monitored items → `ISubscribable` + per-driver subscription refcount (v1 archive) |
 | [AlarmTracking.md](AlarmTracking.md) | `IAlarmSource` + `AlarmSurfaceInvoker` + OPC UA alarm conditions — native Galaxy alarms end-to-end (live) |
 | [AlarmTracking.md](v1/AlarmTracking.md) | Original alarm-tracking write-up (v1 archive) |
-| [AlarmHistorian.md](AlarmHistorian.md) | `Core.AlarmHistorian` store-and-forward SQLite sink — `SqliteStoreAndForwardSink`, `IAlarmHistorianWriter`, dead-letter/retry/eviction |
+| [AlarmHistorian.md](AlarmHistorian.md) | `Core.AlarmHistorian` store-and-forward sink — `LocalDbStoreAndForwardSink`, the replicated `alarm_sf_events` buffer + Primary-only drain, `IAlarmHistorianWriter`, dead-letter/retry/eviction |
 | [DataTypeMapping.md](v1/DataTypeMapping.md) | Per-driver `DriverAttributeInfo` → OPC UA variable types (v1 archive — live mapping is in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DataTypeMap.cs`) |
 | [IncrementalSync.md](IncrementalSync.md) | Address-space rebuild on redeploy + `sp_ComputeGenerationDiff` |
 | [HistoricalDataAccess.md](v1/HistoricalDataAccess.md) | `IHistoryProvider` as a per-driver optional capability (v1 archive) |

docs/Redundancy.md

@@ -6,7 +6,7 @@ OtOpcUa supports OPC UA **non-transparent** warm/hot redundancy. Two or more `Ot
 
 > **Discovery surface.** The `ServerArray` path on the `Server` object is what each node populates with self + peer `ApplicationUri`s — see `OpcUaApplicationHost.PopulateServerArray` and the per-node `PeerApplicationUris` option below. The redundancy-object-type `ServerUriArray` proper (a child of `Server.ServerRedundancy`) remains deferred pending an SDK object-type upgrade; clients should read `Server.ServerArray` for peer discovery today.
 
-> **v2 change.** v1's operator-managed `ClusterNode.RedundancyRole` column + `RedundancyCoordinator` / `ApplyLeaseRegistry` / `PeerHttpProbeLoop` are gone. Primary/secondary is now derived from **Akka cluster role-leader** for the `driver` role. The operator no longer writes a role into the DB; cluster topology (specifically the `driver` role-leader) drives ServiceLevel automatically.
+> **v2 change.** v1's operator-managed `ClusterNode.RedundancyRole` column + `RedundancyCoordinator` / `ApplyLeaseRegistry` / `PeerHttpProbeLoop` are gone. Primary/secondary is now derived from **Akka cluster membership** — the oldest Up member carrying the `driver` role. The operator no longer writes a role into the DB; cluster topology drives ServiceLevel automatically.
 
 The runtime pieces live in:
 
@@ -19,7 +19,7 @@ The runtime pieces live in:
 | `DbHealthProbeActor` | `OtOpcUa.Runtime.Health` | Per-node; runs `SELECT 1` against ConfigDb every 5s. Read by the health endpoint AND by `OpcUaPublishActor` (the `DbReachable` ServiceLevel input). |
 | `PeerProbeSupervisor` | `OtOpcUa.Runtime.Health` | Per-node; subscribes to the `redundancy-state` topic and maintains one `PeerOpcUaProbeActor` child per OTHER driver-role peer (spawn on join, stop on departure), so every node is continuously probed by its peers. |
 | `PeerOpcUaProbeActor` | `OtOpcUa.Runtime.Health` | Spawned by `PeerProbeSupervisor`; pings a peer `opc.tcp://peer:4840` with a TCP connect (2s timeout) and publishes `OpcUaProbeResult` on the `redundancy-state` topic. A full secure-channel Hello handshake is a possible future upgrade; the TCP connect is the current real probe. |
-| `ClusterRoleInfo` | `OtOpcUa.Cluster` | Live view of cluster membership + role-leader; exposes `IClusterRoleInfo` to the rest of the host. |
+| `ClusterRoleInfo` | `OtOpcUa.Cluster` | Live view of cluster membership + role-leader; exposes `IClusterRoleInfo` to the rest of the host. (Akka's role-leader is *not* what elects the redundancy Primary — see below.) |
 
 ## ServiceLevel tiers
 
@@ -38,7 +38,7 @@ The four inputs are sourced locally per driver node:
 | `DbReachable` | Local `DbHealthProbeActor` — `OpcUaPublishActor` Asks it on each `HealthTick`; an Ask timeout is treated as `Reachable=false`. |
 | `OpcUaProbeOk` | Result of a peer probing THIS node's OPC UA endpoint: `PeerProbeSupervisor` spawns one `PeerOpcUaProbeActor` per OTHER driver-role peer; each probe publishes `OpcUaProbeResult(probed-node, ok)` on the `redundancy-state` topic; the publish actor consumes only results whose target is itself. Freshness-debounced: absent or stale (>30 s) → `true` (benefit of the doubt — single-node clusters and a departed peer never demote); only an actively-observed RECENT `false` demotes. |
 | `Stale` (derived) | `!DbReachable \|\| (now − lastDbHealth.AsOfUtc) > 30 s \|\| (now − snapshotEntry.AsOfUtc) > 30 s`. |
-| `IsDriverRoleLeader` | The local node's entry in the `RedundancyStateChanged` snapshot from `RedundancyStateActor`. |
+| `IsDriverPrimary` | The local node's entry in the `RedundancyStateChanged` snapshot from `RedundancyStateActor`. |
 
 The resulting truth table (all tiers are now reachable at runtime):
 
@@ -47,8 +47,8 @@ The resulting truth table (all tiers are now reachable at runtime):
 | Down / Detached | 0 | Member status is not `Up` or `Joining` (leaving, removed, exiting), OR node has no `driver` role (Detached). Published immediately — a starting or detached node never leaves the SDK default 255. |
 | Critically degraded | 100 | ConfigDb unreachable AND data is stale. |
 | Stale | 200 | Data stale but ConfigDb reachable. |
-| Healthy follower | 240 | DB reachable + OPC UA probe ok + not stale + not role-leader. |
-| Healthy leader | 250 | Same as healthy follower + this node is the `driver` role-leader (+10 bonus). |
+| Healthy follower | 240 | DB reachable + OPC UA probe ok + not stale + not the driver Primary. |
+| Healthy primary | 250 | Same as healthy follower + this node is the driver Primary (+10 bonus). |
 
 > **Secondary 100 → 240 (behavior change).** Previously a healthy Secondary
 > published 100 (coarse role-only mapping). It now publishes **240** — both
@@ -66,9 +66,38 @@ ServiceLevel (even 0) is always published so no node lingers at the SDK default
 255.
 
 Roles come from `RedundancyStateActor.BuildSnapshot`: a node with the `driver`
-role is `Primary` when it holds the `driver` role-leader lease, otherwise
+role is `Primary` when it is the **oldest Up member carrying the `driver` role**, otherwise
 `Secondary`; a node without the `driver` role is `Detached`.
 
+**Oldest, not role leader (changed 2026-07-21).** The election previously used
+`ClusterState.RoleLeader("driver")` — the *lowest-addressed* Up member with the role. Akka offers
+both, and they name different nodes: role leader is ordered by address (host, then port), oldest by
+up-number. They agree on a freshly-formed cluster, and diverge after any restart — a restarted node
+re-joins as the *youngest* while keeping its address, so if it holds the lower address it becomes
+role leader. Meanwhile `ClusterSingletonManager` places singletons on the **oldest**, so the old
+derivation could name a Primary that was not hosting the singletons, enabling the Primary-gated data
+plane (writes, alarm acks, the alerts emit, the alarm-history drain) on the wrong node. Pinned by
+`RedundancyPrimaryElectionTests`, which forms a real two-node cluster where the oldest node holds the
+*higher* address so the two orderings genuinely disagree. `NodeRedundancyState.IsRoleLeaderForDriver`
+was renamed `IsDriverPrimary` to stop the name asserting the old derivation.
+
+> **KNOWN LIMITATION — the election is per *Akka* cluster, not per application `Cluster`.**
+> The election yields exactly **one** Primary across the whole Akka cluster. A fleet that
+> runs several application clusters (each with its own redundant pair) inside one Akka cluster —
+> which is what `docker-dev/docker-compose.yml` models, with MAIN, SITE-A and SITE-B — therefore has
+> one Primary among *all* its driver nodes, not one per pair. Every consumer of the role inherits
+> this: `ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate below.
+>
+> The unit of redundancy everywhere else in the product is the application `Cluster` (`ClusterId`) —
+> it owns drivers, devices and `ClusterNode` rows, and the LocalDb deployment-artifact cache is
+> already keyed by it *so that a pair shares one entry*. Scoping the election the same way is the
+> fix; `RedundancyStateActor` is an admin-role singleton in the same assembly as
+> `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
+>
+> Surfaced by the LocalDb Phase 2 live gate, where it stopped the alarm-history drain on every node
+> in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`. The alarm drain no longer
+> depends on this (it defers only to a node that shares its queue), but the other three gates still do.
+
 ## Data flow
 
 ```
@@ -114,13 +143,18 @@ The admin singleton is the cluster's only `RedundancyStateActor`. If the admin l
 
 Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var:
 
-```json
+```jsonc
 {
   "Cluster": {
     "Hostname": "0.0.0.0",
     "Port": 4053,
     "PublicHostname": "node-a.lan",
-    "SeedNodes": ["akka.tcp://otopcua@node-a.lan:4053"],
+    // Self FIRST, partner second — see "Bootstrap: self-first seed ordering".
+    // node-b.lan's own config lists node-b.lan first and node-a.lan second.
+    "SeedNodes": [
+      "akka.tcp://otopcua@node-a.lan:4053",
+      "akka.tcp://otopcua@node-b.lan:4053"
+    ],
     "Roles": ["admin", "driver"]
   }
 }
@@ -130,9 +164,9 @@ Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var:
 OTOPCUA_ROLES=admin,driver
 ```
 
-Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list.
+Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. List **both** peers in `SeedNodes` on **both** nodes so either can cold-start alone — **and list each node ITSELF first**: the order decides which Akka bootstrap process runs, and a node that lists its partner first can never form the cluster while that partner is down. See [Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering).
 
-There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster role-leadership). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).
+There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).
 
 > **`RedundancyStateActor` NodeId consistency (fixed).** `RedundancyStateActor` now keys each node's `NodeRedundancyState` entry by the canonical `host:port` node id (via a `ToNodeId(Address)` helper mirroring `ClusterRoleInfo.ToNodeId`). Previously it keyed by `member.Address.Host` (host-only, e.g. `central-2`); since every subscriber matches by the canonical `host:port` form, the mismatch silently meant no node ever matched its own entry — all nodes stayed at the default ServiceLevel 255 and never learned their role. This fix makes `RedundancyStateActor` consistent with the stated contract above. Additionally, `RedundancyStateActor` now **re-publishes the current snapshot on a periodic heartbeat (default 10 s)** so any node that subscribes after the last topology-change publish converges within the interval (DistributedPubSub does not replay to late subscribers).
 
@@ -152,55 +186,201 @@ Each node advertises its partner via `OpcUaApplicationHostOptions.PeerApplicatio
 
 Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpointTests` in `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/` — boots two `OpcUaApplicationHost` instances on loopback, asserts a real OPCFoundation client `Session` reading `Server.ServerArray` from Node A sees both URIs.
 
-## Split-brain
+## Split-brain / downing
 
-The split-brain resolver is **active by default**: Akka.Cluster.Hosting's `WithClustering` enables an SBR
-downing provider whenever `ClusterOptions.SplitBrainResolver` is null (it applies
-`SplitBrainResolverOption.Default`, which registers `Akka.Cluster.SBR.SplitBrainResolverProvider`), and that
-provider reads the `split-brain-resolver` HOCON block in `akka.conf`. On top of that,
-`ServiceCollectionExtensions.BuildClusterOptions` sets the typed `ClusterOptions.SplitBrainResolver`
-(`KeepOldestOption { DownIfAlone = true }`) to make the strategy **explicit in code** rather than relying on
-the framework default — it is reinforcing, not the sole activator, and yields the same effective behavior. So
-the cluster is **not** running `NoDowning`; hard-crashed nodes are downed and the cluster recovers (how it
-recovers depends on *which* node was lost — see the table below). (Only an *explicit* `NoDowning`, e.g.
-`akka.cluster.downing-provider-class = ""`, would leave both redundancy sides at ServiceLevel 240
-indefinitely.) The HOCON block carries the tuning: `active-strategy = keep-oldest`, `stable-after = 15s`,
-`keep-oldest.down-if-alone = on`, `failure-detector.threshold = 10.0` (its `active-strategy` + `down-if-alone`
-must stay consistent with the typed option; `stable-after` lives only in HOCON because the typed option can't
-express it).
+**The default downing strategy is `auto-down` (changed 2026-07-21).** It is selected by
+`Cluster:SplitBrainResolverStrategy`, and `ServiceCollectionExtensions.BuildDowningHocon` turns that
+into the `akka.cluster.downing-provider-class` actually installed:
 
-`keep-oldest` is the correct strategy for a 2-node warm-redundancy pair (`keep-majority`/`static-quorum`
-are wrong for two nodes — no majority in a 1-1 split), but its two loss cases recover **differently**, and
-this is inherent to any 2-node cluster:
-
-| Node lost | What keep-oldest does | Recovery |
+| `Cluster:SplitBrainResolverStrategy` | Provider installed | Posture |
 |---|---|---|
-| **Younger** node (crash or partition) | The oldest is on the surviving side → it stays up and downs the younger side (`DownUnreachable`). | **In-place, fast.** The oldest keeps its singletons + `driver` role-leadership; `RedundancyStateActor` re-computes from the post-loss `Cluster.State`. |
-| **Oldest** node (crash or partition) | The lone survivor is "the side **without** the oldest" → keep-oldest downs the **survivor itself** (`DownReachable "including myself"`), and `run-coordinated-shutdown-when-down = on` terminates it. `down-if-alone` does **not** rescue a lone survivor (its rescue branch needs ≥ 2 surviving members, so it is a 3+-node feature). | **Exit-and-rejoin.** The self-downed survivor **and** the crashed oldest are both restarted by the service supervisor and re-form / rejoin. There is a brief restart-window outage; there is **no** in-place takeover. |
+| `auto-down` (**default**) | `Akka.Cluster.AutoDowning` with `auto-down-unreachable-after = 15s` | **Availability.** The leader among the *reachable* members downs the unreachable peer, so a crash of **either** node — oldest included — fails over in place. |
+| `keep-oldest` | `Akka.Cluster.SBR.SplitBrainResolverProvider` reading the `split-brain-resolver` block in `akka.conf` | **Partition-safety.** A partition can never run dual-active. **Not safe for a 2-node pair** — see below. |
 
-**Recovery therefore depends on three things being in place** (the [ScadaBridge](../../ScadaBridge/CLAUDE.md)
-sister project runs the same pattern):
+Any other value **fails the host at startup** rather than falling through to a default.
+
+### Why keep-oldest is no longer the default
+
+The previous documentation in this section — and the code comments behind it — asserted that
+`keep-oldest` with `down-if-alone = on` was "the correct strategy for a 2-node warm-redundancy pair."
+**That was wrong, and the error was not conservative: it described a total-outage configuration as the
+recommended one.**
+
+In Akka.NET 1.5.62's `KeepOldest.OldestDecision`, the `down-if-alone` rescue branch requires the
+*surviving* side to hold **≥ 2 members**. In a two-node cluster a lost peer is always a 1-vs-1 split, so
+the branch never fires and the lone survivor falls through to `DownReachable` — downing **itself** — and
+`run-coordinated-shutdown-when-down = on` then terminates it. `down-if-alone` is a 3+-node feature; it
+does not do what its name suggests here. Live-proven on the sister project's rig, whose survivor logged
+`SBR took decision …DownReachable and is downing [self] including myself, [1] unreachable of [2] members`.
+See [`../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md`](../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md)
+for the upstream decision record, which OtOpcUa follows.
+
+The alternatives do not help a pair: `static-quorum` with quorum 1 hits Akka's `IsTooManyMembers` guard
+and returns `DownAll`; with quorum 2 the survivor downs itself; `keep-majority` keeps the lowest-address
+side, which just moves the fatal crash from "the oldest node" to "the other node"; `lease-majority` needs
+a shared lease store both nodes can reach.
+
+### The accepted trade
+
+Under `auto-down`, a **genuine network partition** (both nodes alive, link cut) leaves each side downing
+the other and continuing alone: **both run active**, both hold singletons, and both advertise the primary
+`ServiceLevel`. Recovery is operator-driven — after the link heals, restart **one** side; it rejoins as a
+fresh incarnation and settles as secondary. The two sides do not merge on their own, because the mutual
+downing quarantines the association.
+
+This is a deliberate choice of availability over partition-safety, matching ScadaBridge: the pairs run one
+node per VM with no external arbiter available, so a crash is a far more likely event than a partition, and
+a crash under `keep-oldest` was unrecoverable without operator action anyway.
+
+Deployments that would rather take an outage than ever run dual-active can set
+`Cluster:SplitBrainResolverStrategy: "keep-oldest"` — but should read the section above first, because for a
+two-node pair that choice means *any* crash of the oldest node is a full outage.
+
+### Recovery still depends on supervision
 
 1. **A service supervisor that restarts the process** on exit — production `Install-Services.ps1` sets
    `sc.exe failure OtOpcUaHost … actions= restart/5000/restart/30000/restart/60000`; the docker-dev rig sets
-   `restart: unless-stopped`. Without it a downed node stays down and an oldest-crash looks like a total outage.
-2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) — it
-   watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls
-   `IHostApplicationLifetime.StopApplication()` so the process exits (and the supervisor restarts it) instead of
-   idling forever with a dead actor system.
-3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join the mesh via either peer.
+   `restart: unless-stopped`.
+2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) —
+   it watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls
+   `IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead
+   of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this
+   (it is never the one downed); the **downed** node still does.
+3. **Both peers listed in `SeedNodes`** on every node, **each node listing itself first**, so a restarted
+   node can re-join via either peer and a node cold-starting while its peer is dead still forms the
+   cluster. Without the self-first ordering it waits in `InitJoin` forever (auto-down removes the crash
+   outage, not that one) — see [Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering).
 
 > **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
-> sole `driver` role-leader, and it passes — but it simulates the crash with `provider.Transport.Shutdown()`,
-> which leaves node A's `ActorSystem` **alive** (a transport partition with the oldest still running), not a
-> real process death. It is therefore **not** representative of an oldest-process crash; a real 2-container
-> `docker kill` of the oldest downs the survivor (verified 2026-07-15, see
-> `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`). Treat the recovery guarantee for
-> the oldest as "exit-and-rejoin under supervision," not "in-place failover."
+> sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with
+> `provider.Transport.Shutdown()`, which leaves node A's `ActorSystem` **alive** (a transport partition with
+> the oldest still running), not a real process death. A real 2-container `docker kill` of the oldest downed
+> the survivor (verified 2026-07-15, `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`).
+> **It is a standing example of a green test over a fatal defect**: keep the distinction between a transport
+> partition and a process death in mind when reading it.
 
-There is no operator-driven role swap during a partition. Failover / recovery is what the cluster + supervisor
-do automatically. **Instant in-place takeover on *any* single-node loss requires 3+ members** (an odd cluster
-or a lightweight witness) — a deliberate future option, not the current 2-node posture.
+`SplitBrainResolverActivationTests` pins the strategy by starting a real host through the production
+bootstrap and reading `akka.cluster.downing-provider-class` back off the running `ActorSystem` — asserting
+the typed option or the akka.conf text is not sufficient, because several HOCON fragments compete and the
+losing one still looks correct in the file.
+
+Failover during a partition is what the cluster and supervisor do automatically. For a *planned* swap —
+patching, draining a node — see [Manual failover](#manual-failover) below.
+
+## Manual failover
+
+The cluster redundancy page (`/clusters/{id}/redundancy`) carries a **Live redundancy** panel — the current
+driver Primary and every Up `driver` member, read from live cluster state — and an admin-gated **Trigger
+failover** button.
+
+The mechanism is a graceful cluster **`Leave`** of the current Primary, never a `Down`:
+`IManualFailoverService.FailOverDriverPrimaryAsync` selects the **oldest Up `driver` member** — the identical
+query to `RedundancyStateActor.SelectDriverPrimary`, pinned together by
+`ManualFailoverServiceTests.Parity_with_SelectDriverPrimary` so the node acted on is exactly the node the
+election names. The leaving node hands its singletons over through the cluster-leave phases,
+`CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it,
+and it rejoins as the **youngest** member — so the survivor is now the oldest, becomes Primary, and
+advertises `ServiceLevel` 250. OPC UA clients re-select.
+
+| Rule | Behaviour |
+|---|---|
+| Authorization | `AdminUiPolicies.FleetAdmin` (**Administrator only**) — gated in markup *and* re-checked server-side before the call. Deliberately not `ConfigEditor`, which the neighbouring cluster pages use: this restarts a production node rather than editing config, and ConfigEditor also admits Designer. |
+| Peer guard | Disabled, with the reason as its tooltip, when fewer than two Up driver members exist — with one node a "failover" is a shutdown. Re-evaluated inside the service against live state, so a stale page cannot bypass it. |
+| Confirmation | A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select). |
+| Audit | Written through the shared `ZB.MOM.WW.Audit.IAuditWriter` seam **before** the Leave — action `cluster.manual-failover`, actor, target. A *refused* failover changes nothing and writes nothing. |
+
+> **Mesh-scope caveat.** Until the per-cluster mesh work lands
+> (`docs/plans/2026-07-21-per-cluster-mesh-design.md`), the Primary is elected once per Akka mesh rather than
+> per application `Cluster` row — so on a fleet running several clusters in one mesh this button acts on the
+> whole mesh's Primary, which may belong to a different cluster than the page you are on. The panel says so.
+> Phase 6 of the mesh program removes both the caveat and this notice.
+
+> **Live gate outstanding.** The auto-down change is verified by unit tests against the effective
+> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has
+> **not** yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where
+> the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster;
+> assert the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) should run
+> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
+> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
+
+## Bootstrap: self-first seed ordering
+
+Listing both peers in `SeedNodes` does **not** mean either node can cold-start alone — and neither does any
+downing strategy: **auto-down removes the crash outage, not this one.** Akka runs a different bootstrap
+process depending on whether `seed-nodes[0]` is the node's own address:
+
+| `seed-nodes[0]` is… | Process Akka runs | Can it form a NEW cluster? |
+|---|---|---|
+| this node's own address | `FirstSeedNodeProcess` | **Yes** — it `InitJoin`s the other seeds and self-joins once `seed-node-timeout` (5 s) passes with nobody answering. |
+| some other node | `JoinSeedNodeProcess` | **No** — it retries `InitJoin` forever, however long the peer stays down. |
+
+So the ordering *is* the mechanism. **Every node that is a seed of its own mesh lists ITSELF first and its
+partner second** (decision 2026-07-22):
+
+```jsonc
+// central-1                                     // central-2
+"SeedNodes": [                                   "SeedNodes": [
+  "akka.tcp://otopcua@central-1:4053",             "akka.tcp://otopcua@central-2:4053",
+  "akka.tcp://otopcua@central-2:4053"              "akka.tcp://otopcua@central-1:4053"
+]                                                ]
+```
+
+The rule is **conditional**: it binds only when a node's own address appears in its own seed list. A
+driver-only site node is seeded solely by `central-1` (today's docker-dev topology) — it is legitimately
+not a seed of anything and is exempt. An unconditional "self must be first" rule would refuse to boot every
+site node in the fleet.
+
+`AkkaClusterOptionsValidator` enforces it at startup (`AddValidatedOptions` → `ValidateOnStart` in
+`AddOtOpcUaCluster`), because the invariant otherwise fails **silently**: the process runs, the port
+listens, the node simply never becomes a member. Identity is compared on `Cluster:PublicHostname` (falling
+back to `Cluster:Hostname` when blank) **and** `Cluster:Port` — the address Akka puts in `SelfAddress`.
+Comparing against `Cluster:Hostname` would be worse than wrong: in docker-dev it is `0.0.0.0`, which matches
+no seed URI anywhere, so the rule would silently exempt the entire rig.
+
+Sequential recovery is island-free by construction: a `FirstSeedNodeProcess` node self-joins only when **no**
+seed answered, so a peer booting after the survivor formed simply joins it as the youngest member. Both
+nodes cold-starting *simultaneously* converge on one cluster while they are mutually reachable — the
+`InitJoin` handshake resolves it before either self-join deadline expires. The residual risk is a boot-time
+**partition** (both cold-starting, mutually unreachable): both form, which is the same dual-active class the
+`auto-down` strategy already accepts, with the same recovery (restart one side).
+
+Pinned by `SelfFirstSeedBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real
+in-process clusters through the production bootstrap at production failure-detection timings: a lone
+cold-start with a dead peer forms alone; a restarted node rejoins its live peer; two nodes cold-starting
+together converge on one cluster. Its **falsifiability control** — the old peer-first ordering, same
+scenario, never comes Up — is what keeps the other three from going vacuous, and carries a positive-control
+self-join proving the node was formable all along. `AkkaClusterOptionsValidatorTests` covers the validator,
+including the site-node exemption and the `0.0.0.0` identity trap.
+
+### Why the self-form watchdog was retired (2026-07-22)
+
+The first fix for this gap was `Cluster:SelfFormAfter` + `ClusterBootstrapFallback`: wait 10 s for
+membership, then call `Cluster.Join(SelfAddress)`. **Both are deleted.** A timer sits *outside* Akka's join
+handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight" —
+and `Cluster.Join(SelfAddress)` is **not** ignored mid-handshake: it wins.
+
+> **Live-gate finding (docker-dev, 2026-07-22 — preserved because it is the evidence).** The watchdog
+> islanded a node that manual failover had bounced. `central-1` restarted, received `InitJoinAck` from
+> `central-2` at 10:49:05 — the join was in flight and healthy — but no Welcome arrived inside the window,
+> because the peer's ring still held its previous incarnation (`Exiting` → `Down` → `Removed`, retired at
+> 10:49:06). At 10:49:15 the watchdog fired and the node formed a **second cluster**, islanded until an
+> operator restarted it. Manual failover deliberately produces exactly that restart, so the two halves of
+> that plan collided. A TCP reachability guard (`ea45ace1`) patched that one shape — never self-form while
+> a seed peer accepts a connection — and the earlier live gate for the watchdog itself had genuinely
+> passed: `central-2` cold-started alone in 10 s and a non-seed `site-a-2` correctly stayed out.
+
+The reachability guard closed the observed failure, not the class: a peer can be TCP-reachable and not
+answering, or answering and slow, and the timer cannot tell. Self-first ordering has no such race because
+the decision is made *inside* the handshake, by Akka, on the actual `InitJoin` replies. It was also inert
+for every non-seed node, so after this change it could never usefully fire. `SelfFormBootstrapTests` is
+replaced by `SelfFirstSeedBootstrapTests`, whose `Restarting_node_rejoins_its_live_peer_instead_of_islanding`
+is the drilled scenario above.
+
+> **Live gate outstanding for the new mechanism.** Self-first ordering is verified here by in-process
+> clusters at production timings, and on the sister project (ScadaBridge `4a6341d8`) on real clusters *and*
+> live docker containers. It has **not** been re-drilled on the OtOpcUa docker-dev rig since the swap — the
+> `central-2` cold-start-alone drill should be repeated on the rebuilt image. Owned by
+> `docs/plans/2026-07-22-per-cluster-mesh-program.md` Phase 7 alongside the auto-down 1-vs-1 gate.
 
 ## Primary data-plane gate (writes, acks, alerts emit)
 
@@ -221,11 +401,44 @@ Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and ev
 
 - **`alerts` topic emission** — `ScriptedAlarmHostActor` and `DriverHostActor.ForwardNativeAlarm` subscribe to the `redundancy-state` DPS topic and cache the local node's `RedundancyRole`, then gate the cluster `alerts` publish through `PrimaryGatePolicy` (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain **ungated** on both nodes so the secondary is always ready to serve clients after a failover.
 - **`HistorianAdapterActor` historization** — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the `alerts` DPS topic and translates each `AlarmTransitionEvent` → `AlarmHistorianEvent` before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
+- **The alarm sink's DRAIN** — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. `LocalDbStoreAndForwardSink` takes a `drainGate` fed from `IRedundancyRoleView` — a singleton `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, because the drain runs on a timer and cannot receive `RedundancyStateChanged` itself. A gated-off node reports `HistorianDrainState.NotPrimary`. Delivery is **at-least-once across a failover** by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See [AlarmHistorian.md](AlarmHistorian.md).
 
 Net effect: each alarm transition appears **once** on `/alerts` and would historize once, not once per node.
 
 See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals.
 
+## Pair-local store (LocalDb — Phases 1 + 2)
+
+Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated
+[`ZB.MOM.WW.LocalDb`](../CLAUDE.md) SQLite database. Phase 2 added the **alarm store-and-forward
+buffer** (`alarm_sf_events`) to it alongside the config cache, so a node that dies holding
+undelivered alarm history no longer takes it to the grave — see the Primary-only drain note above
+and [AlarmHistorian.md](AlarmHistorian.md). Phase 1 caches the **deployed-configuration
+artifact** (chunked). Its job is a single failure mode redundancy does not otherwise cover: a driver
+node restarting into a **central-SQL-Server outage** would come up with no configuration at all. With
+the cache, it **boots from the last artifact it applied** instead, and logs a running-from-cache
+signal.
+
+- **What replicates.** The cache can *optionally* replicate to the node's redundant pair peer over
+  the LocalDb library's gRPC sync (HLC-stamped, last-writer-wins). Two tables replicate:
+  `deployment_artifacts` and `deployment_pointer`. Replication is **default-OFF and fail-closed** —
+  it stays inert until `LocalDb:SyncListenPort` + a peer are configured, and a missing/mismatched
+  `LocalDb:Replication:ApiKey` refuses or silently halts sync. It is enabled per-pair as an opt-in.
+- **The replicated-cache payoff.** In a pair with replication on, *either* node can be the one that
+  applied the latest deploy; the peer holds a byte-identical copy. So a node that restarts during a
+  central outage can boot the current config **even if it never applied that deploy itself** — its
+  peer did, and the row replicated.
+- **What it does NOT cover.** The cache is a **boot fallback for central-SQL outages only**, not a
+  standby data path and not a replacement for central. A **new deployment still requires central
+  SQL**; the cache is read only when the central fetch fails at startup, and the node resumes
+  fresh-config behavior once central returns. It does not replicate live tag values, alarms, or
+  historian data — only the config artifact. It is orthogonal to the ServiceLevel/primary-gate
+  logic: a node running from cache still participates in redundancy normally.
+
+Full detail: the enablement runbook at
+[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
+and the `LocalDb` section of [`docs/Configuration.md`](Configuration.md).
+
 ## Client-side failover
 
 The OtOpcUa Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` supports `-F` / `--failover-urls` for automatic client-side failover; for long-running subscriptions the CLI monitors session KeepAlive and reconnects to the next available server, recreating the subscription on the new endpoint. See [`Client.CLI.md`](Client.CLI.md).

docs/ServiceHosting.md

@@ -56,6 +56,7 @@ The host joins an Akka.NET cluster bound to the address in `appsettings.json::Cl
 }
 ```
 
+- `SeedNodes` is **ordered**: a node listed in its own seed list must be entry 0, since Akka only lets `seed-nodes[0]` form a new cluster. `AkkaClusterOptionsValidator` fails startup otherwise — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering).
 - `WithOtOpcUaClusterBootstrap` (in `OtOpcUa.Cluster`) loads the embedded HOCON (split-brain resolver, pinned dispatcher, failure detector tuning) and overlays remote endpoint + cluster options.
 - All cluster singletons + per-node actors live on this single ActorSystem — there is no second Akka instance.
 

docs/drivers/Historian.Wonderware.md

@@ -13,7 +13,7 @@ OtOpcUa now consumes the **`ZB.MOM.WW.HistorianGateway`** sidecar through the Gi
 
 - **HistoryRead** → `GatewayHistorianDataSource` over the `ServerHistorian` appsettings section.
 - **Alarm history** → `GatewayAlarmHistorianWriter` (the gateway `SendEvent` path) behind the durable
-  `SqliteStoreAndForwardSink`; alarm-history `ReadEvents` needs the gateway running
+  `LocalDbStoreAndForwardSink`; alarm-history `ReadEvents` needs the gateway running
   `RuntimeDb:EventReadsEnabled=true`.
 - **Continuous historization** → a crash-safe FasterLog outbox + `ContinuousHistorizationRecorder`
   draining to the gateway's `WriteLiveValues` (`ContinuousHistorization` section); needs the gateway

docs/operations/2026-07-20-localdb-pair-replication.md

@@ -0,0 +1,191 @@
+# LocalDb pair replication — operations runbook
+
+> **Scope.** Every driver-role OtOpcUa node keeps a consolidated
+> [`ZB.MOM.WW.LocalDb`](../../CLAUDE.md) SQLite database. It holds two things: the
+> **deployed-configuration artifact** (chunked, Phase 1) so a node can **boot from cache when
+> central SQL Server is unreachable**, and the **alarm store-and-forward buffer** (Phase 2) so a
+> node that dies holding undelivered alarm history does not take it to the grave. Both can
+> *optionally* replicate to the node's redundant pair peer over the library's gRPC sync.
+> **Replication is default-OFF and fail-closed.** This runbook covers enabling it, the operational
+> rules that keep a pair converging, and the DB-inspection safety rules.
+
+## What replicates, and what it buys you
+
+- **Replicated tables:** `deployment_artifacts` (the artifact, split into base64 chunks),
+  `deployment_pointer` (one current-deployment pointer per cluster), and `alarm_sf_events` (the
+  alarm store-and-forward buffer). Registered — in this order, after the DDL — by
+  `LocalDbSetup.OnReady`.
+- **The payoff:** in a redundant driver pair, *either* node can be the one that applied the latest
+  deploy and wrote the cache. With replication on, the peer holds a byte-identical copy. So a node
+  that restarts into a central-SQL outage can boot the current config **even if it never applied
+  that deploy itself** — its peer did, and the row replicated.
+- **Convergence model:** last-writer-wins per primary key, HLC-stamped. Both nodes also store
+  locally after every apply, so the pair converges to identical content regardless of which node
+  deployed. Retention (newest-2 deployments per cluster) prunes on one node and the deletes
+  replicate as tombstones.
+
+## Alarm store-and-forward — what replication changes
+
+Phase 2 moved the alarm buffer out of its own `alarm-historian.db` and into this database. Three
+consequences an operator needs to know:
+
+- **Only the Primary drains.** The buffer replicates, so the Secondary holds a full copy of the
+  Primary's queue. Its drain worker is gated on the same Primary decision the inbound-write and
+  native-ack gates use, and reports `DrainState = NotPrimary`. **A rising queue depth on a
+  Secondary is correct.** A rising queue depth on *both* nodes is not — see below.
+- **Delivery is at-least-once across a failover, by design.** Rows the old Primary delivered
+  moments before it died may not have had their deletes replicated yet, so the new Primary will
+  re-send them. This is accepted, not a defect: a duplicate alarm-history row is recoverable, a
+  missing one is not. There is deliberately no dedup layer.
+- **The buffer is bounded.** `AlarmHistorian:Capacity` (1,000,000 by default) still evicts the
+  oldest undelivered rows when the historian has been unreachable long enough. Watch
+  `EvictedCount` — non-zero means accepted alarm events were dropped before reaching the historian.
+
+### If BOTH nodes report `NotPrimary`
+
+Nobody is draining, and the queue is filling toward the capacity ceiling on both. The gate
+default-DENIES while the redundancy role is unknown *and* a driver peer exists, so this is what a
+redundancy snapshot that never names either node looks like — the identity-mismatch shape
+`DriverHostActor` also warns about once ("redundancy snapshot omitted this node"). Check node
+identity (`Cluster:PublicHostname` / `Cluster:Port` skew) before suspecting the sink.
+
+### One-time migration from `alarm-historian.db`
+
+On first boot after the upgrade, `AlarmSfLegacyMigrator` copies any pre-existing
+`alarm-historian.db` into `alarm_sf_events` and renames the file to `alarm-historian.db.migrated`.
+
+- **Both nodes of a pair migrate independently**, and their files overlap — row ids are derived
+  from the event payload, so the same event on both nodes converges to one row rather than
+  duplicating. The new Primary drains the union.
+- **The rename happens only after the copy commits.** A failure fails host startup with the legacy
+  file untouched; the `.migrated` sidecar is what makes a later boot a no-op.
+- **A file whose shape is unrecognised is left alone** — not renamed, not copied — so it stays
+  available for inspection.
+- The `AlarmHistorian:DatabasePath` key is **removed**. It is still *read* by the migrator, to
+  find the legacy file; it no longer configures anything. Delete it from appsettings once the
+  `.migrated` sidecar exists on every node.
+
+## Enabling replication on a pair
+
+Replication has two knobs, both under `LocalDb`:
+
+| Key | Role | Notes |
+|---|---|---|
+| `LocalDb:SyncListenPort` | binds the dedicated **h2c** sync listener | `0` (default) = no listener, replication off. Set the **same** non-zero port on **both** nodes of the pair. |
+| `LocalDb:Replication:PeerAddress` | the address this node **dials** | Set on **one** node (the initiator); leave empty on the other (passive). The stream is bidirectional — one dial carries both directions. |
+| `LocalDb:Replication:ApiKey` | bearer token for the sync stream | **Must be byte-identical on both nodes.** See fail-closed rule below. Supply via `${secret:...}` / env — never a cleartext literal in production config. |
+| `LocalDb:Replication:MaxBatchSize` | rows per replication batch | `16` for the artifact cache. Batching is **row-count-only** against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so 16 × 171 KB ≈ 2.7 MB stays under the cap. Raising it risks tripping the cap. |
+
+Example (site-a-1 initiator, site-a-2 passive):
+
+```
+site-a-1:  LocalDb__SyncListenPort=9001
+           LocalDb__Replication__PeerAddress=http://site-a-2:9001
+           LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key}
+           LocalDb__Replication__MaxBatchSize=16
+site-a-2:  LocalDb__SyncListenPort=9001
+           LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key}   # no PeerAddress
+           LocalDb__Replication__MaxBatchSize=16
+```
+
+> **Kestrel note.** Setting `SyncListenPort` makes the Host add an explicit `Listen*` for the h2c
+> sync listener. An explicit `Listen*` makes Kestrel **ignore `ASPNETCORE_URLS` entirely**, so the
+> Host also re-binds the primary HTTP port in the same block. If you change the primary port,
+> verify both `:<httpPort>` and `:<syncPort>` appear in the startup "Now listening on" lines.
+
+### The ApiKey rule (fail-closed)
+
+The replication library's passive endpoint verifies **no** authentication — the host
+`LocalDbSyncAuthInterceptor` is the only gate, and it is **fail-closed**:
+
+- **No key configured ⇒ every sync call is refused** (`PermissionDenied`). "No key" is never
+  "no auth required".
+- **A key mismatch ⇒ the pair silently stops converging.** The initiator's stream is rejected at
+  the peer; nothing errors loudly. A typo in one node's key looks exactly like "replication is
+  broken". If a pair is not converging, **check the keys match first.**
+
+## Operational rules
+
+- **Stop / start the pair together where you can.** Each node keeps working (and caching locally)
+  while its peer is down; the outage is not a data-loss event — the surviving node accumulates
+  writes and the peer catches up on rejoin. But a long-lived solo node drifts further from its
+  peer, so avoid leaving a pair split for extended periods.
+- **Tombstone-retention resurrection window.** Retention prunes to the newest 2 deployments and
+  replicates the prune as tombstones. Tombstones are themselves retained only for a bounded window.
+  If a node is offline **longer than the tombstone-retention window**, a delete that happened during
+  its outage may no longer be expressible as a tombstone on rejoin — a pruned deployment could
+  briefly reappear until the next deploy re-prunes it. Keep pair outages well inside that window.
+- **A rebuilt node is back-filled by its peer.** If a node loses its LocalDb file — a wiped volume,
+  a re-imaged host, a fresh container — it rejoins with an empty database and its peer snapshots the
+  cached configuration back to it, with no new deploy and no central SQL. Two things are worth
+  knowing: this needs **`ZB.MOM.WW.LocalDb` ≥ 0.1.3**, and it heals the *cache*, so the node regains
+  boot-from-cache for future outages. (On `0.1.1` a converged pair has pruned every oplog row on ack
+  and the wiped node was never snapshotted — it stayed empty until the next deploy. On `0.1.2` it is
+  back-filled but its OWN writes are silently dropped until its restarted seq counter climbs past the
+  peer's stale watermark, so the pair looks converged right up until the moment only the rebuilt node
+  applies a deploy.) A node
+  wiped **during** a central outage is still repopulated by its peer under 0.1.2; a node with **no**
+  peer replication configured self-heals only from central on its next successful apply.
+
+- **What boot-from-cache does NOT cover.** The cache is a *fallback for central-SQL outages at
+  boot*, not a replacement for central. A **new deployment still requires central SQL** — the cache
+  is only read when the central fetch fails at startup. When central returns, the node resumes
+  fresh-config behavior. Boot-from-cache logs a running-from-cache signal; treat a node that stays
+  on it as a central-connectivity incident, not steady state.
+
+## Inspecting the LocalDb file — safety rules
+
+The database runs in **WAL mode**. These rules exist because violating them corrupted a live DB in
+the 2026-07-20 ScadaBridge incident:
+
+- **Never run `sqlite3` on the live file** (host-side, against a bind-mounted or container path).
+  Opening a live WAL DB from a second process across virtiofs poisons the WAL. **Always copy the
+  triplet first** and query the copy:
+
+  ```bash
+  docker cp <node>:/app/data/otopcua-localdb.db      /tmp/localdb.db
+  docker cp <node>:/app/data/otopcua-localdb.db-wal  /tmp/localdb.db-wal
+  docker cp <node>:/app/data/otopcua-localdb.db-shm  /tmp/localdb.db-shm
+  sqlite3 /tmp/localdb.db 'SELECT cluster_id, deployment_id FROM deployment_pointer;'
+  ```
+
+- **Metrics** come from the container, not a host curl (`aspnet:10.0` has no `curl`):
+
+  ```bash
+  docker run --rm --network container:<node> curlimages/curl:latest -s localhost:<httpPort>/metrics | grep localdb_
+  ```
+
+- **If an anomaly appears within seconds of your own measurement, suspect the measurement.**
+  Restart both nodes and re-observe untouched before blaming the library.
+
+## Useful queries (on a copied triplet)
+
+```sql
+-- What each node thinks the current deployment is
+SELECT cluster_id, deployment_id, revision_hash, applied_at_utc FROM deployment_pointer;
+
+-- Convergence check: the pointer's origin stamp should match on both nodes
+SELECT pk_json, hlc, node_id, is_tombstone
+FROM __localdb_row_version WHERE table_name = 'deployment_pointer' ORDER BY pk_json;
+
+-- Replication backlog (should drain to 0 when a pair is caught up)
+SELECT COUNT(*) FROM __localdb_oplog;
+
+-- Undelivered alarm history, and how much of it has given up
+SELECT dead_lettered, COUNT(*) FROM alarm_sf_events GROUP BY dead_lettered;
+
+-- Why the dead-lettered rows died
+SELECT alarm_id, attempt_count, last_attempt_utc, last_error
+FROM alarm_sf_events WHERE dead_lettered = 1 ORDER BY last_attempt_utc DESC LIMIT 20;
+
+-- Convergence check for the buffer: identical dumps mean the peer holds the ORIGIN-stamped rows
+SELECT pk_json, hlc, node_id, is_tombstone
+FROM __localdb_row_version WHERE table_name = 'alarm_sf_events' ORDER BY pk_json;
+```
+
+## See also
+
+- [`docs/Redundancy.md`](../Redundancy.md) — the pair-local config cache section.
+- [`docs/Configuration.md`](../Configuration.md) — the `LocalDb` appsettings section.
+- [`docs/AlarmHistorian.md`](../AlarmHistorian.md) — the alarm sink, its knobs, and the drain.
+- The two-node convergence harness: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/`.

docs/plans/2026-07-16-v3-batch4-PR.md

@@ -94,7 +94,7 @@ dual-namespace + event-delivery), Runtime 377 (+ VT reassert regression), AdminU
 
 ## Documented follow-ups (non-blocking)
 
-- **Scripted-alarm redeploy recovery (CONFIRMED pre-existing bug, deferred — deserves its own careful fix).**
+- **Scripted-alarm redeploy recovery (CONFIRMED pre-existing bug, deferred — deserves its own careful fix). Filed as issue #487.**
   The scripted-alarm Part 9 condition node has the same redeploy-reset race the VT `ReassertValue` fix closed:
   `RebuildAddressSpace` clears `_alarmConditions` and `MaterialiseScriptedAlarms` recreates each condition
   fresh/normal, but `ScriptedAlarmEngine.LoadAsync` reloads from the persisted state and yields
@@ -106,10 +106,11 @@ dual-namespace + event-delivery), Runtime 377 (+ VT reassert regression), AdminU
   historian issue). Proposed shape: a host-driven, node-only `AlarmStateUpdate` re-assert in `OnAlarmsLoaded`
   (same safe post-materialise ordering as the VT fix), reusing `_engine.GetState(alarmId)` + `LoadedAlarmIds`,
   **never touching the `alerts` topic**.
-- Raw-tag rename hot-rebind of the renamed tag's live value without a recreate (today: deploy-THEN-recreate).
-- Absolute-path Monaco completion → RawPath (from Batch 3; the `{{equip}}/` completion works).
+- Raw-tag rename hot-rebind of the renamed tag's live value without a recreate (today: deploy-THEN-recreate). Filed as issue #489.
+- Absolute-path Monaco completion → RawPath (from Batch 3; the `{{equip}}/` completion works). Filed as issue #490.
 - Cross-repo (post-merge): ScadaBridge Data-Connection-Layer NodeId/namespace cutover (raw `s=<RawPath>` /
-  UNS `s=<Area>/<Line>/<Equipment>/<EffectiveName>`, retired `EquipmentNodeIds`) + the umbrella index
-  `../scadaproj/CLAUDE.md` OtOpcUa entry.
+  UNS `s=<Area>/<Line>/<Equipment>/<EffectiveName>`, retired `EquipmentNodeIds`) — tracked in the
+  ScadaBridge repo as its issue #14 (its #20 landed the `nsu=` binding + native-alarm routing). The umbrella
+  index `../scadaproj/CLAUDE.md` OtOpcUa entry is **DONE** (scadaproj `ede5275`).
 
 https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox

docs/plans/2026-07-20-localdb-adoption-phase1.md

@@ -0,0 +1,706 @@
+# OtOpcUa LocalDb Adoption — Phase 1 Implementation Plan
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+>
+> **Execution model:** This plan is optimized for **Claude Opus** agents (`claude --model opus`).
+> Dispatch implementer subagents on Opus for every task classified `high-risk`; `small`/`trivial`
+> tasks may use Sonnet. Work on branch **`feat/localdb-phase1`** in this repo (`~/Desktop/OtOpcUa`,
+> remote `lmxopcua`). Do NOT merge to `master` as part of this plan — stop at the DoD task and
+> report.
+>
+> **Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md`.
+> Read it before Task 0. The reference adoption is ScadaBridge (merged PR #23) — when in doubt,
+> mirror `~/Desktop/ScadaBridge` (files named per task below).
+
+**Goal:** Give every driver-role OtOpcUa node a consolidated `ZB.MOM.WW.LocalDb` database that caches
+the deployed-configuration artifact (chunked), boots from that cache when central SQL Server is
+unreachable, and optionally replicates it to the node's redundant pair peer over the library's gRPC
+sync (default-OFF, fail-closed bearer auth).
+
+**Architecture:** `AddZbLocalDb`/`AddZbLocalDbReplication` wired in a new `LocalDbRegistration`
+(mirroring `SecretsRegistration`), DDL + `RegisterReplicated` in `LocalDbSetup.OnReady`, a dedicated
+Kestrel h2c listener for `MapZbLocalDbSync` gated on `LocalDb:SyncListenPort`, a consumer-supplied
+fail-closed bearer interceptor, and an `IDeploymentArtifactCache` seam written by `DriverHostActor`
+after each successful apply and read as a boot fallback. The dormant LiteDB LocalCache subsystem is
+deleted as superseded.
+
+**Tech stack:** .NET 10, `ZB.MOM.WW.LocalDb`/`.Replication`/`.Contracts` **0.1.1** (Gitea feed),
+`Grpc.AspNetCore` 2.76.0, Akka.NET (existing), xunit.v3 (Host.IntegrationTests) + xunit2 TestKit
+(actor tests).
+
+**Hard rules (from the ScadaBridge adoption — violating any of these is a defect):**
+1. Order inside `OnReady` is load-bearing: **DDL → `RegisterReplicated` → writes**. Rows written
+   before registration are never captured or snapshotted — silently, forever.
+2. No autoincrement PKs, no BLOB columns in replicated tables.
+3. The library's passive sync endpoint verifies **no** ApiKey — the host interceptor is the only
+   auth, and it must be fail-closed (no key configured ⇒ deny everything).
+4. `ILocalDb.CreateConnection()` returns an **already-open**, pragma-configured connection with the
+   `zb_hlc_next()` UDF. Never call `Open()` on it. There is **no in-memory mode** — tests use temp
+   files + `SqliteConnection.ClearAllPools()` before delete.
+5. `Grpc.Core.Testing` does not exist on grpc-dotnet — hand-roll fake `ServerCallContext`s.
+6. Explicit Kestrel `Listen*` calls make Kestrel **ignore `ASPNETCORE_URLS`** — when adding the sync
+   listener you must re-bind the existing HTTP port in the same block, and verify it on the rig.
+7. Never run host `sqlite3` against a live bind-mounted WAL DB (copy the `db`/`-wal`/`-shm` triplet
+   with `cp` instead). `aspnet:10.0` containers have no `curl` — use a
+   `curlimages/curl` sidecar with `--network container:<name>`.
+
+---
+
+### Task 0: Preflight + recon (produces `docs/plans/2026-07-20-localdb-phase1-recon.md`)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (everything downstream consumes its findings)
+
+**Files:**
+- Create: `docs/plans/2026-07-20-localdb-phase1-recon.md`
+- Read-only: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs`,
+  `Directory.Packages.props`, `docker-dev/docker-compose.yml`,
+  the observability registration (`AddOtOpcUaObservability` implementation),
+  `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/*`
+
+**Step 1: Verify the feed packages restore.**
+
+Run:
+```bash
+cd ~/Desktop/OtOpcUa && dotnet package search ZB.MOM.WW.LocalDb --source dohertj2-gitea --format json | head -50
+```
+Expected: `ZB.MOM.WW.LocalDb`, `.Replication`, `.Contracts` at `0.1.1`. If absent → STOP, report.
+
+**Step 2: Map and record, with `file:line` citations, each of:**
+1. **Deploy fetch path:** where `DriverHostActor` loads `Deployment.ArtifactBlob` (the
+   `_dbFactory.CreateDbContext()` sites, ~L1472/1543), the artifact's runtime type
+   (`DeploymentArtifact` — how it's deserialized from the blob), the type/format of
+   `DeploymentId` and `RevisionHash`, and where a successful apply completes (the point after
+   which the cache write belongs).
+2. **Cold-boot path:** what the actor does at startup before any `DispatchDeployment` arrives —
+   does it query central SQL for the current deployment? Where does the failure path land
+   (the `Stale` state, `DbHealthProbeActor` interaction, L1345 dispatch-ignore)? Identify the
+   exact seam where "central fetch failed at boot" is known — that is where the cache fallback goes.
+3. **Cluster identity:** how a driver node knows its `ClusterId` (config key / `ClusterNode` row /
+   `IClusterRoleInfo`) — the cache is keyed by it.
+4. **DriverHostActor construction:** how its dependencies are injected
+   (`WithOtOpcUaRuntimeActors`, `Props` wiring) so `IDeploymentArtifactCache` can be threaded in.
+   ⚠ `Props.Create` builds an expression tree — adding a parameter silently rebinds
+   out-of-position named args at call sites; list every construction site.
+5. **Telemetry allowlist:** does `AddOtOpcUaObservability` restrict meters
+   (`ZbTelemetryOptions.Meters` or equivalent)? If yes, record where the list lives.
+6. **Kestrel/URLs:** confirm the Host binds via `ASPNETCORE_URLS=http://+:9000` with no
+   `ConfigureKestrel` calls; record any existing `builder.WebHost` usage.
+7. **LiteDB LocalCache:** confirm (grep) that nothing outside
+   `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/` and its tests references
+   `ILocalConfigCache`/`GenerationSealedCache`/`ResilientConfigReader`/`LiteDB`.
+8. **docker-dev volumes:** whether nodes have a writable data volume; record what `LocalDb:Path`
+   should be per node and how env vars are passed (`Cluster__*` style double-underscore).
+
+**Step 3: STOP conditions — halt the plan and report instead of improvising if:**
+- the artifact apply path cannot be given a post-apply hook without restructuring the actor;
+- `DeploymentId`/cluster identity are not stable strings/GUIDs;
+- LiteDB LocalCache turns out to be referenced by live code.
+
+**Step 4: Commit.**
+```bash
+git add docs/plans/2026-07-20-localdb-phase1-recon.md
+git commit -m "docs(localdb): phase-1 recon findings"
+```
+
+---
+
+### Task 1: Package references and pins
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+**Parallelizable with:** none (all code tasks build on it)
+
+**Files:**
+- Modify: `Directory.Packages.props`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ZB.MOM.WW.OtOpcUa.Runtime.csproj`
+
+**Step 1:** Add to `Directory.Packages.props` (versions exact):
+```xml
+<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.1" />
+<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.1" />
+<PackageVersion Include="Grpc.AspNetCore" Version="2.76.0" />
+```
+Check the existing `Grpc.Net.Client`/`Grpc.Core.Api` are already ≥ 2.76.0 and
+`Google.Protobuf` ≥ 3.34.1 (the LocalDb.Replication nuspec floors); raise if not. Confirm a
+`SQLitePCLRaw` pin ≥ 2.1.12 exists for the transitive `Microsoft.Data.Sqlite` chain (the family
+advisory GHSA-2m69-gcr7-jv3q); Core.AlarmHistorian already pins `bundle_e_sqlite3` 2.1.12 — add
+`SQLitePCLRaw.lib.e_sqlite3` 2.1.12 as an explicit reference in the Host if the audit flags it.
+
+**Step 2:** Host csproj: `ZB.MOM.WW.LocalDb`, `ZB.MOM.WW.LocalDb.Replication`, `Grpc.AspNetCore`.
+Runtime csproj: `ZB.MOM.WW.LocalDb` only (it needs `ILocalDb`, not the sync engine).
+
+**Step 3:** `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → 0 warnings/errors (repo builds warnings-as-errors).
+
+**Step 4: Commit.** `git commit -m "build(localdb): reference ZB.MOM.WW.LocalDb 0.1.1 + Grpc.AspNetCore"`
+
+---
+
+### Task 2: Schema + `LocalDbSetup.OnReady` + registration tests
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 3
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/DeploymentCacheSchema.cs`
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/LocalDbSetupTests.cs` (create the test project
+  reference layout to match neighboring Host unit tests; if Host has no unit-test project, place in
+  the closest existing one and note the deviation)
+
+**Step 1: Write the failing tests first** — using a real temp-file `ILocalDb` (build it via
+`new ServiceCollection().AddZbLocalDb(config, LocalDbSetup.OnReady)` with `LocalDb:Path` pointed at
+a temp file; dispose + `SqliteConnection.ClearAllPools()` in cleanup):
+- `OnReady_RegistersExactlyTheTwoDeploymentTables` — `db.ReplicatedTables.Keys` ordinal-sorted
+  equals `["deployment_artifacts", "deployment_pointer"]` (assert BOTH directions: no fewer, no more).
+- `DeploymentArtifacts_PkIsDeploymentIdPlusChunkIndex` and `DeploymentPointer_PkIsClusterId`
+  (pin `ReplicatedTable.PkColumns`).
+- `RowsWrittenAfterOnReady_EnterTheOplog` — insert a row, assert `SELECT COUNT(*) FROM __localdb_oplog` ≥ 1
+  (pins the DDL→register ordering; this is the assertion that catches a silently-broken CDC).
+
+**Step 2: Run tests, watch them fail** (types don't exist yet).
+
+**Step 3: Implement.** `DeploymentCacheSchema` depends only on `Microsoft.Data.Sqlite` (the
+ScadaBridge `*Schema.Apply` pattern — self-sufficient for direct construction in tests):
+
+```csharp
+namespace ZB.MOM.WW.OtOpcUa.Runtime.Deployment;
+
+public static class DeploymentCacheSchema
+{
+    public static void Apply(SqliteConnection connection)
+    {
+        using var cmd = connection.CreateCommand();
+        cmd.CommandText = """
+            CREATE TABLE IF NOT EXISTS deployment_artifacts (
+                deployment_id   TEXT    NOT NULL,
+                chunk_index     INTEGER NOT NULL,
+                cluster_id      TEXT    NOT NULL,
+                revision_hash   TEXT    NOT NULL,
+                chunk_count     INTEGER NOT NULL,
+                chunk_base64    TEXT    NOT NULL,
+                cached_at_utc   TEXT    NOT NULL,
+                PRIMARY KEY (deployment_id, chunk_index)
+            );
+            CREATE TABLE IF NOT EXISTS deployment_pointer (
+                cluster_id        TEXT NOT NULL PRIMARY KEY,
+                deployment_id     TEXT NOT NULL,
+                revision_hash     TEXT NOT NULL,
+                artifact_sha256   TEXT NOT NULL,
+                applied_at_utc    TEXT NOT NULL
+            );
+            """;
+        cmd.ExecuteNonQuery();
+    }
+}
+```
+
+`LocalDbSetup` (Host):
+```csharp
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+internal static class LocalDbSetup
+{
+    // ORDER IS LOAD-BEARING: DDL, then RegisterReplicated, then (nothing else in Phase 1).
+    // Rows written before RegisterReplicated are invisible to the peer forever, silently.
+    public static void OnReady(ILocalDb db)
+    {
+        using var connection = db.CreateConnection(); // already open — do NOT call Open()
+        DeploymentCacheSchema.Apply(connection);
+        db.RegisterReplicated("deployment_artifacts");
+        db.RegisterReplicated("deployment_pointer");
+    }
+}
+```
+
+**Step 4:** `dotnet test --filter "FullyQualifiedName~LocalDbSetupTests"` → PASS.
+
+**Step 5: Commit.** `git commit -m "feat(localdb): deployment-cache schema + OnReady registration"`
+
+---
+
+### Task 3: Fail-closed sync auth interceptor + tests
+
+**Classification:** standard
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 2
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/LocalDbSyncAuthInterceptorTests.cs`
+
+Port ScadaBridge's `src/ZB.MOM.WW.ScadaBridge.Host/LocalDbSyncAuthInterceptor.cs` (read it first;
+keep behavior identical, adjust namespace only). Semantics to preserve exactly:
+- `ServicePrefix = "/localdb_sync.v1.LocalDbSync/"`; any other method path passes through untouched.
+- Expected token from `IOptions<ReplicationOptions>.Value.ApiKey`.
+- **Fail-closed:** null/empty configured key ⇒ `RpcException(new Status(StatusCode.PermissionDenied, ...))`
+  for every sync call. Missing/mismatched bearer ⇒ same.
+- `CryptographicOperations.FixedTimeEquals` over UTF-8 bytes; header `authorization`,
+  case-insensitive, `"Bearer "` prefix. Override all four server handler kinds.
+
+**Tests** (hand-rolled fake `ServerCallContext` — `Grpc.Core.Testing` does not exist on grpc-dotnet;
+copy ScadaBridge's `LocalDbSyncAuthInterceptorTests.cs` fake): non-sync method passes with no key;
+sync + no configured key → PermissionDenied; wrong bearer → PermissionDenied; correct bearer → passes.
+
+Write tests first (fail), implement, `dotnet test --filter "FullyQualifiedName~LocalDbSyncAuthInterceptor"`,
+commit `feat(localdb): fail-closed bearer interceptor for the sync endpoint`.
+
+---
+
+### Task 4: `LocalDbRegistration` + Program.cs DI wiring + config defaults
+
+**Classification:** high-risk (touches Program.cs / role gating)
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (Task 5 edits the same file)
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (inside the `hasDriver` branch)
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json`
+
+**Step 1:** `LocalDbRegistration`, mirroring `SecretsRegistration`'s shape:
+```csharp
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+internal static class LocalDbRegistration
+{
+    /// Driver-role nodes only. Storage is unconditional; replication stays inert until
+    /// LocalDb:Replication:PeerAddress (initiator) / LocalDb:SyncListenPort (listener) are set.
+    public static IServiceCollection AddOtOpcUaLocalDb(
+        this IServiceCollection services, IConfiguration configuration)
+    {
+        services.AddZbLocalDb(configuration, LocalDbSetup.OnReady);
+        services.AddZbLocalDbReplication(configuration);
+        return services;
+    }
+
+    public static int SyncListenPort(IConfiguration configuration) =>
+        configuration.GetValue<int>("LocalDb:SyncListenPort");
+}
+```
+
+**Step 2:** Program.cs, in the `hasDriver` block (near `AddAlarmHistorian`):
+`builder.Services.AddOtOpcUaLocalDb(builder.Configuration);` and
+`builder.Services.AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>());`
+(AddGrpc only under `hasDriver` — admin-only nodes expose no sync surface).
+
+**Step 3:** appsettings.json:
+```json
+"LocalDb": {
+  "Path": "./data/otopcua-localdb.db",
+  "SyncListenPort": 0,
+  "Replication": {}
+}
+```
+`LocalDb:Path` is `ValidateOnStart`-required once `AddZbLocalDb` runs — since registration is
+driver-gated, admin-only configs need nothing. Check the role-overlay files
+(`appsettings.driver.json` / `appsettings.admin-driver.json`) and any deploy templates for
+conflicting `LocalDb` keys.
+
+**Step 4:** Build + full Host unit tests. Verify an admin-only graph doesn't require the key: this
+is pinned properly in Task 10's integration tests, but do a quick
+`OTOPCUA_ROLES=admin dotnet run` smoke only if cheap; otherwise rely on Task 10.
+
+**Step 5: Commit.** `feat(localdb): wire AddOtOpcUaLocalDb into the driver role`
+
+---
+
+### Task 5: Dedicated h2c sync listener + endpoint mapping (THE Kestrel task)
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (Program.cs)
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`
+
+**Why high-risk:** the Host today binds exclusively via `ASPNETCORE_URLS=http://+:9000`. Any
+explicit `ConfigureKestrel(... Listen*)` makes Kestrel **ignore URLs entirely** (it logs
+"Overriding address(es)"), which would silently kill the AdminUI/deploy API behind Traefik.
+
+**Step 1:** Add, before `builder.Build()`:
+```csharp
+var syncPort = LocalDbRegistration.SyncListenPort(builder.Configuration);
+if (hasDriver && syncPort > 0)
+{
+    // Explicit Listen* replaces ASPNETCORE_URLS wholesale, so re-bind the primary HTTP
+    // endpoint here too. Parse it from the configured URLs rather than hard-coding 9000.
+    var urls = builder.Configuration["ASPNETCORE_URLS"] ?? builder.Configuration["urls"] ?? "http://+:9000";
+    var httpPort = new Uri(urls.Split(';')[0].Replace("+", "localhost").Replace("*", "localhost")).Port;
+    builder.WebHost.ConfigureKestrel(k =>
+    {
+        k.ListenAnyIP(httpPort);                                            // HTTP/1.1 (existing surface)
+        k.ListenAnyIP(syncPort, o => o.Protocols = HttpProtocols.Http2);    // h2c, prior-knowledge gRPC
+    });
+}
+```
+When `syncPort == 0` (the default) nothing changes — URLs binding stays untouched. Cleartext
+`Http1AndHttp2` cannot serve prior-knowledge h2c, hence the dedicated `Http2`-only listener.
+
+**Step 2:** After the app pipeline's other `Map*` calls, driver-gated:
+```csharp
+if (hasDriver && syncPort > 0)
+{
+    app.MapZbLocalDbSync();
+}
+```
+(Mapping is harmless when unauthenticated — the interceptor fail-closes — but gating on the port
+keeps admin-only and default-OFF graphs entirely free of the endpoint.)
+
+**Step 3: Verify both surfaces.**
+```bash
+dotnet build ZB.MOM.WW.OtOpcUa.slnx
+```
+Then a local smoke with the port on:
+`ASPNETCORE_URLS=http://+:9000 OTOPCUA_ROLES=driver LocalDb__SyncListenPort=9001 dotnet run --project src/Server/ZB.MOM.WW.OtOpcUa.Host ...`
+— expect startup logs to show BOTH `:9000` and `:9001` bound, and `curl -s localhost:9000/healthz`
+(or the mapped health route) still answering. If the app needs SQL to boot, defer the smoke to the
+Task 12 rig check but say so explicitly in the task notes.
+
+**Step 4: Commit.** `feat(localdb): dedicated h2c listener + MapZbLocalDbSync gated on LocalDb:SyncListenPort`
+
+---
+
+### Task 6: `IDeploymentArtifactCache` + chunked LocalDb implementation + tests
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 3, Task 5
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactCache.cs`
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/LocalDbDeploymentArtifactCache.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Deployment/LocalDbDeploymentArtifactCacheTests.cs`
+
+**Step 1: Failing tests** (real temp-file `ILocalDb` with `LocalDbSetup.OnReady` applied):
+- round-trip: store a 300 KiB random artifact → `GetCurrentAsync` returns byte-identical payload
+  (forces ≥ 3 chunks at 128 KiB);
+- retention: store 3 deployments for one cluster → only the newest 2 remain in
+  `deployment_artifacts`; pointer names the newest;
+- integrity: corrupt one chunk row via SQL → `GetCurrentAsync` returns null (miss), never a
+  truncated artifact;
+- missing pointer → null.
+
+**Step 2: Implement:**
+```csharp
+public interface IDeploymentArtifactCache
+{
+    Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
+                    byte[] artifact, CancellationToken ct = default);
+    Task<CachedDeploymentArtifact?> GetCurrentAsync(string clusterId, CancellationToken ct = default);
+}
+
+public sealed record CachedDeploymentArtifact(
+    string DeploymentId, string RevisionHash, byte[] Artifact, DateTimeOffset AppliedAtUtc);
+```
+`LocalDbDeploymentArtifactCache(ILocalDb db)`:
+- `StoreAsync`: one `ILocalDbTransaction` — delete any existing chunks for this `deployment_id`
+  (idempotent re-store), insert chunks (raw 128 * 1024 bytes per chunk, `Convert.ToBase64String`),
+  upsert the pointer (`INSERT ... ON CONFLICT(cluster_id) DO UPDATE`), then prune: delete
+  `deployment_artifacts` rows whose `deployment_id` is not among the newest 2 `cached_at_utc` for
+  this `cluster_id`. SHA-256 over the raw artifact into the pointer. ISO-8601 UTC timestamps.
+- `GetCurrentAsync`: read pointer; read chunks `ORDER BY chunk_index`; verify count ==
+  `chunk_count` and SHA-256 matches; return null on any mismatch (log a warning naming which check
+  failed).
+- Use `db.ExecuteAsync`/`QueryAsync` with anonymous-object parameters (`@Name` markers). Remember
+  a `Dictionary<string,string>` parameter **throws** — use anonymous objects.
+
+**Step 3:** run tests → PASS. **Step 4: Commit.**
+`feat(localdb): chunked deployment-artifact cache over ILocalDb`
+
+---
+
+### Task 7: Cache write path in `DriverHostActor`
+
+**Classification:** high-risk (actor model, `Props` expression-tree trap)
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:** (exact edit sites come from the Task 0 recon doc — cite it in the commit)
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
+- Modify: the actor-registration site (`WithOtOpcUaRuntimeActors` / DI extension) to provide
+  `IDeploymentArtifactCache`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` or `LocalDbRegistration` — register
+  `IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache` singleton (driver branch)
+- Test: the **xunit2** TestKit project used for existing DriverHostActor tests (recon names it)
+
+**Steps:**
+1. **Failing test:** after a successful artifact apply, the cache received `StoreAsync` with the
+   applied deployment's id/hash/bytes (inject a recording fake `IDeploymentArtifactCache`).
+   Also: a cache that throws on `StoreAsync` does NOT fail the apply (apply result unchanged, error
+   logged).
+2. Thread the dependency through the actor's constructor. ⚠ `Props.Create` is an expression tree:
+   after adding the parameter, re-check **every** construction site for positional/named-arg
+   rebinding (the recon lists them) — do not rely on the compiler.
+3. Invoke `StoreAsync` fire-and-forget (`PipeTo`-style or a guarded `Task.Run` per the actor's
+   existing async conventions — match whatever pattern the actor already uses for side-effect IO)
+   at the post-apply point identified in recon. Cluster id from the recon-identified source.
+4. Run the actor test suite for this project. Commit:
+   `feat(localdb): DriverHostActor stores applied artifacts in the pair-local cache`
+
+---
+
+### Task 8: Boot-from-cache read path + running-from-cache signal
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
+- Test: same xunit2 TestKit project as Task 7
+
+**Steps:**
+1. **Failing tests:**
+   - central fetch fails at startup AND cache has an artifact → the actor applies the cached
+     artifact (assert via whatever observable the apply already exposes) and logs/flags
+     running-from-cache;
+   - central fetch fails AND cache empty → today's behavior exactly (Stale, no apply);
+   - central fetch **succeeds** → cache is NOT consulted (fresh config wins; assert the fake
+     cache's `GetCurrentAsync` was never called on the happy path).
+2. Implement at the recon-identified boot-failure seam. The signal: a log warning at minimum plus,
+   if the actor already publishes health/status (recon says how), a `RunningFromCache` marker on it.
+   Do NOT touch `DispatchDeployment` handling — a new deployment still requires central.
+3. Run tests; commit `feat(localdb): boot from the pair-local artifact cache when central SQL is unreachable`.
+
+---
+
+### Task 9: Delete the dormant LiteDB LocalCache subsystem
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+**Parallelizable with:** Task 6, Task 10, Task 11
+
+**Files:**
+- Delete: `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/` (entire directory:
+  `ILocalConfigCache`, `LiteDbConfigCache`, `GenerationSealedCache`, `ResilientConfigReader`,
+  `GenerationSnapshot`, `StaleConfigFlag`, exceptions)
+- Delete: `tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/GenerationSealedCacheTests.cs`,
+  `ResilientConfigReaderTests.cs`
+- Modify: `Directory.Packages.props` + the Configuration csproj — remove the `LiteDB` package if
+  nothing else references it (grep first)
+
+**Steps:** Confirm the Task 0 recon's "nothing references it" finding still holds
+(`grep -rn "ILocalConfigCache\|GenerationSealedCache\|ResilientConfigReader\|LiteDB" src/ tests/`),
+delete, build the full solution, run the Configuration test project. DoD phrasing: **no references
+from code** — leave any explanatory prose/comments that point readers at LocalDb instead. Add one
+line to the recon doc noting LiteDB's removal. Commit:
+`refactor(localdb): delete the dormant LiteDB LocalCache (superseded by ZB.MOM.WW.LocalDb)`
+
+---
+
+### Task 10: Health check + telemetry meter allowlist
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 9, Task 11
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/LocalDbReplicationHealthCheck.cs` (or the
+  directory `AddOtOpcUaHealth` uses — recon names it)
+- Modify: the `AddOtOpcUaHealth` registration (driver branch)
+- Modify: the observability meter list IF Task 0 found an allowlist
+
+**Steps:**
+1. Failing unit tests: replication unconfigured (no peer, no listener) → `Healthy` (default-OFF
+   must not degrade a plain node); peer configured + `ISyncStatus.Connected == false` → `Degraded`;
+   connected + backlog `null` → `Degraded` (unknown backlog is not healthy); connected + backlog
+   small → `Healthy`.
+2. Implement against `ISyncStatus` + `IOptions<ReplicationOptions>` (peer-configured = non-empty
+   `PeerAddress` OR `SyncListenPort > 0` — pass the latter in via options/config).
+3. **Meter allowlist:** if `AddOtOpcUaObservability` restricts meters, add
+   `"ZB.MOM.WW.LocalDb.Replication"` (use `LocalDbMetrics.MeterName`) — this is a silent allowlist;
+   the ScadaBridge live gate is the proof it bites. If there is no allowlist, record that in the
+   recon doc and skip.
+4. Commit `feat(localdb): replication health check + meter export`.
+
+---
+
+### Task 11: DI-pin integration tests over the real Program.cs
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 9, Task 10
+
+**Files:**
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbWiringTests.cs` (xunit.v3,
+  `WebApplicationFactory<Program>` — the project already builds the real Program.cs; set
+  `OTOPCUA_ROLES` per test case the way neighboring tests do)
+
+**Pins (each its own test):**
+1. Driver graph: `ILocalDb` resolves, is a singleton, `ReplicatedTables` ordinal-sorted ==
+   `["deployment_artifacts", "deployment_pointer"]` — exact set, both directions load-bearing.
+2. Driver graph: `IDeploymentArtifactCache` resolves to `LocalDbDeploymentArtifactCache`.
+3. Default-OFF pin: `ISyncStatus` resolves with `Connected == false`, `PeerNodeId == null`.
+4. Admin-only graph: `ILocalDb` is NOT registered (`GetService<ILocalDb>()` is null) and boot does
+   not demand `LocalDb:Path`.
+5. Health: the LocalDb health check is registered in the driver graph.
+
+Point `LocalDb:Path` at a per-test temp file via the factory's config overrides;
+`ClearAllPools` + delete in cleanup. These tests exist because DI extensions without a
+container-built test have shipped inert three times in this family (Secrets 0.2.0/0.2.2,
+ScadaBridge#22). Commit `test(localdb): DI pins over the real host graph`.
+
+---
+
+### Task 12: Two-node convergence harness + tests
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min (harness) + ~4 min (scenarios) — split the commit if needed
+**Parallelizable with:** none (depends on Tasks 2, 3, 5, 6)
+
+**Files:**
+- Create: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairHarness.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairConvergenceTests.cs`
+
+**Harness:** copy the pattern from
+`~/Desktop/ScadaBridge/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs` and
+the library's `ConvergenceFixture` (`~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/tests/.../Convergence/ConvergenceFixture.cs`):
+- Two full stacks over a real loopback Kestrel h2c socket (port 0), **through the real
+  `LocalDbSyncAuthInterceptor`** with a shared test key; node A initiator, node B passive.
+- Both initialized via the **production `LocalDbSetup.OnReady`** — a hand-written schema proves
+  only that the test agrees with itself.
+- DBs owned by the harness, registered as pre-constructed singletons (so MS.DI won't dispose them
+  on host teardown); `KillTransportAsync`/`RestartTransportAsync`; tight `FlushInterval` (50 ms),
+  bounded backoff (2 s); temp files, `ClearAllPools` cleanup; a collection definition to serialize
+  these tests.
+
+**Scenarios:**
+1. `ArtifactStoredOnA_ConvergesToB_ByteIdentical` — store a multi-chunk artifact via
+   `LocalDbDeploymentArtifactCache` on A; B's cache returns byte-identical bytes; both nodes'
+   `__localdb_row_version` rows for the pointer carry the **same HLC and origin node id** (B holds
+   A's row, not a re-derived one).
+2. `RetentionPruneOnA_TombstonesReachB` — third deployment on A prunes the first; B's chunk rows
+   for the pruned deployment disappear and `__localdb_row_version WHERE is_tombstone=1` rows exist
+   on B.
+3. `WritesWhileTransportDown_SurviveRejoin` — kill transport, store on A, restart, converge.
+4. `WrongApiKey_NeverConverges` — harness with mismatched keys: assert **no** convergence after a
+   bounded wait AND (positive control) that the same scenario with matching keys converges — an
+   absence assertion without a positive control passed vacuously in ScadaBridge.
+
+**DoD within this task:** temporarily comment out the two `RegisterReplicated` calls and confirm
+scenarios 1–3 go red (run locally, do not commit the red state); restore. Record the red/green
+evidence in the task notes. Commit `test(localdb): 2-node convergence harness + scenarios`.
+
+---
+
+### Task 13: docker-dev rig configuration
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 14
+
+**Files:**
+- Modify: `docker-dev/docker-compose.yml`
+
+**Steps** (adjust to the recon's findings on volumes/env style):
+1. Every driver node: a named volume (or existing data mount) backing `/app/data`, and
+   `LocalDb__Path=/app/data/otopcua-localdb.db`.
+2. **site-a pair only** (mirror the ScadaBridge default-OFF posture — site-b stays unreplicated as
+   the pin):
+   - site-a-1: `LocalDb__SyncListenPort=9001`, `LocalDb__Replication__PeerAddress=http://site-a-2:9001`,
+     `LocalDb__Replication__ApiKey=dev-site-a-localdb-sync-key`, `LocalDb__Replication__MaxBatchSize=16`
+   - site-a-2: `LocalDb__SyncListenPort=9001`, same `ApiKey`, same `MaxBatchSize`, **no PeerAddress**
+     (passive; the stream is bidirectional).
+   The key must be byte-identical on both — the interceptor fail-closes on any mismatch and the
+   pair silently stops converging.
+3. `MaxBatchSize=16` because batching is row-count-only against gRPC's 4 MB cap and artifact chunk
+   rows are ≈ 171 KB (16 × 171 KB ≈ 2.7 MB worst case).
+4. `docker compose config` to validate; do NOT bring the rig up in this task (the live gate task
+   owns rig runs).
+5. Commit `chore(localdb): rig config — consolidated path everywhere, replication on the site-a pair`.
+
+---
+
+### Task 14: Documentation
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 13
+
+**Files:**
+- Create: `docs/operations/2026-07-20-localdb-pair-replication.md` (runbook: enable/disable,
+  ApiKey handling via `${secret:}`, stop/start-together rule, tombstone-retention resurrection
+  window, MaxBatchSize row-count-vs-4MB, never host-`sqlite3` a live WAL DB / cp-triplet recipe)
+- Modify: `docs/Redundancy.md` (a short "pair-local config cache" section: what replicates, what
+  boot-from-cache does and does not cover)
+- Modify: `CLAUDE.md` (this repo): LocalDb adoption row/paragraph — state Phase 1 scope, default-OFF,
+  rig-only enablement
+- Modify: `docs/Configuration.md` if it documents config keys (add the `LocalDb` section)
+
+Commit `docs(localdb): phase-1 runbook + redundancy/config docs`.
+
+---
+
+### Task 15: DoD sweep (offline)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (final offline task)
+
+**Steps:**
+1. `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → **0 warnings** (warnings-as-errors).
+2. `dotnet test ZB.MOM.WW.OtOpcUa.slnx` → full suite green; record counts. Baseline any
+   pre-existing failures BEFORE this branch (`git stash` → run → unstash) rather than assuming;
+   report deltas only.
+3. Greps (phrase as "no references from code"; explanatory comments may remain):
+   `grep -rn "LiteDB\|ILocalConfigCache\|GenerationSealedCache" src/ tests/` → no code references.
+4. Confirm the positive-control evidence from Task 12 is recorded.
+5. Update `docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json` statuses.
+6. Commit `chore(localdb): phase-1 DoD sweep` and STOP. Report: branch name, commit list, test
+   counts, and that the live gate (Task 16) needs the rig + operator go-ahead.
+
+---
+
+### Task 16: Live gate on the docker-dev rig (run only with explicit user go-ahead)
+
+**Classification:** high-risk (rig)
+**Estimated implement time:** ~30 min wall-clock (mostly waiting)
+**Parallelizable with:** none
+
+**Preamble:** All DB inspection via `cp` of the `db`/`-wal`/`-shm` triplet out of the container
+(`docker cp`) and querying the copy — never host `sqlite3` on the live file (it poisons the WAL
+across virtiofs; root cause of the 2026-07-20 ScadaBridge incident). Metrics via
+`docker run --rm --network container:<node> curlimages/curl:latest -s localhost:<port>/metrics`.
+If an anomaly appears within seconds of one of your own measurements, suspect the measurement
+first — restart both nodes and re-run untouched before blaming the library.
+
+**Checks (all must PASS; record evidence in `docs/plans/2026-07-20-localdb-phase1-live-gate.md`):**
+1. Rig up, site-a pair healthy, `/metrics` on both shows `localdb_` series (proves the meter
+   allowlist work).
+2. Deploy a config through the normal deploy API → both site-a nodes' `deployment_pointer` +
+   `deployment_artifacts` rows are byte-identical with **identical HLC + origin node id** (one
+   node's row replicated, not two independent derivations — both nodes also locally store after
+   apply, so expect LWW-converged identical content either way; assert convergence, and note which
+   origin won).
+3. **Boot-from-cache:** stop the SQL container; restart site-a-2; it comes up serving the cached
+   config with the running-from-cache signal in its log; start SQL again; node recovers fresh
+   behavior.
+4. **Replicated-cache payoff:** wipe site-a-2's local DB file (container stopped), restart it with
+   SQL **up**, let replication repopulate the cache from site-a-1, then repeat check 3's SQL-down
+   restart — it boots from a cache it never wrote itself.
+5. Transport kill (pause site-a-2 container): store/deploy on a-1, oplog depth rises; unpause;
+   drains to 0; converged.
+6. Retention: deploy a 3rd config; pruned deployment's chunks gone on BOTH nodes with tombstone
+   rows present on both.
+7. Both-nodes-together restart: clean rejoin, identical counts, zero
+   `disk I/O error`/`SQLITE_IOERR`/`corrupt` in either log.
+8. site-b pair (default-OFF pin): LocalDb file exists and works locally, `ISyncStatus` health
+   Healthy, no sync connections, no listener on 9001.
+
+Record PASS/FAIL per check with the actual evidence (row counts, md5s, log lines). Commit the gate
+doc. Do not merge — report.
+
+---
+
+## Task persistence
+
+Tasks file: `docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json` (same directory).
+Update statuses as tasks complete; any deviation from this plan gets a `deviation` note on the task
+(the ScadaBridge tasks.json convention — the record of *why* is what makes the plan auditable).

docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json

@@ -0,0 +1,153 @@
+{
+  "planPath": "docs/plans/2026-07-20-localdb-adoption-phase1.md",
+  "tasks": [
+    {
+      "id": 0,
+      "subject": "Task 0: Preflight + recon (findings doc, STOP conditions)",
+      "status": "completed",
+      "notes": "Feed verified: LocalDb/.Contracts/.Replication all 0.1.1. All 3 STOP conditions clear. Findings: docs/plans/2026-07-20-localdb-phase1-recon.md. 8 deviations recorded (D-1..D-8); D-1/D-3/D-5/D-6 change downstream work materially.",
+      "deviation": "D-1 cache read cannot be keyed by ClusterId at the boot seam (unkeyed newest-pointer read instead); D-3 a third LiteDB test file must be deleted; D-5 WebApplicationFactory<Program> is deliberately unused in this repo (use TwoNodeClusterHarness); D-6 driver-only nodes have no ASPNETCORE_URLS so the Kestrel re-bind fallback is wrong. See recon doc \u00a79."
+    },
+    {
+      "id": 1,
+      "subject": "Task 1: Package references and pins (LocalDb 0.1.1, Grpc.AspNetCore, SQLitePCLRaw)",
+      "status": "completed",
+      "blockedBy": [
+        0
+      ]
+    },
+    {
+      "id": 2,
+      "subject": "Task 2: Schema + LocalDbSetup.OnReady + registration tests",
+      "status": "completed",
+      "blockedBy": [
+        1
+      ]
+    },
+    {
+      "id": 3,
+      "subject": "Task 3: Fail-closed sync auth interceptor + tests",
+      "status": "completed",
+      "blockedBy": [
+        1
+      ]
+    },
+    {
+      "id": 4,
+      "subject": "Task 4: LocalDbRegistration + Program.cs DI wiring + config defaults",
+      "status": "completed",
+      "blockedBy": [
+        2,
+        3
+      ]
+    },
+    {
+      "id": 5,
+      "subject": "Task 5: Dedicated h2c sync listener + MapZbLocalDbSync (Kestrel URLs-override risk)",
+      "status": "completed",
+      "blockedBy": [
+        4
+      ]
+    },
+    {
+      "id": 6,
+      "subject": "Task 6: IDeploymentArtifactCache + chunked LocalDb implementation + tests",
+      "status": "completed",
+      "blockedBy": [
+        2
+      ]
+    },
+    {
+      "id": 7,
+      "subject": "Task 7: Cache write path in DriverHostActor (Props expression-tree trap)",
+      "status": "completed",
+      "blockedBy": [
+        0,
+        6
+      ]
+    },
+    {
+      "id": 8,
+      "subject": "Task 8: Boot-from-cache read path + running-from-cache signal",
+      "status": "completed",
+      "blockedBy": [
+        7
+      ]
+    },
+    {
+      "id": 9,
+      "subject": "Task 9: Delete the dormant LiteDB LocalCache subsystem",
+      "status": "completed",
+      "blockedBy": [
+        0
+      ]
+    },
+    {
+      "id": 10,
+      "subject": "Task 10: Health check + telemetry meter allowlist",
+      "status": "completed",
+      "blockedBy": [
+        4
+      ]
+    },
+    {
+      "id": 11,
+      "subject": "Task 11: DI-pin integration tests over the real Program.cs",
+      "status": "completed",
+      "blockedBy": [
+        5,
+        10
+      ]
+    },
+    {
+      "id": 12,
+      "subject": "Task 12: Two-node convergence harness + scenarios (+ positive control)",
+      "status": "completed",
+      "blockedBy": [
+        5,
+        6
+      ]
+    },
+    {
+      "id": 13,
+      "subject": "Task 13: docker-dev rig configuration (site-a pair on, site-b pin off)",
+      "status": "completed",
+      "blockedBy": [
+        5
+      ]
+    },
+    {
+      "id": 14,
+      "subject": "Task 14: Documentation (runbook, Redundancy.md, CLAUDE.md)",
+      "status": "completed",
+      "blockedBy": [
+        8
+      ]
+    },
+    {
+      "id": 15,
+      "subject": "Task 15: DoD sweep (offline) \u2014 STOP and report after this",
+      "status": "completed",
+      "blockedBy": [
+        7,
+        8,
+        9,
+        11,
+        12,
+        13,
+        14
+      ],
+      "notes": "Offline DoD sweep: full-solution build 0 errors (824 pre-existing OTOPCUA0001 analyzer warnings in driver *test* projects; none reference any LocalDb/DeploymentCache file). Grep: no LiteDB/ILocalConfigCache/GenerationSealedCache code refs (2 explanatory doc-comment lines in ILdapGroupRoleMappingService remain, allowed). Runtime.Tests 407/0/31 twice (the previously-flagged intermittent did NOT reproduce). Host.IntegrationTests LocalDb subset 40/0/0. Full-solution `dotnet test` with stash-baseline NOT run: many suites are infra-gated (driver fixtures, full Akka mesh, LDAP real-bind, shared-SQL DB tests) and cannot run offline on macOS \u2014 deferred to Task 16 / CI. Task 12 positive-control evidence recorded in commit afa5be71. Did NOT merge to master \u2014 stopped here per plan."
+    },
+    {
+      "id": 16,
+      "subject": "Task 16: Live gate on the docker-dev rig (needs explicit user go-ahead)",
+      "status": "completed",
+      "blockedBy": [
+        15
+      ],
+      "notes": "Live gate RAN on the docker-dev rig with explicit user go-ahead. 8/8 checks pass (check 4 with a documented limitation). Caught + fixed FOUR real defects offline tests missed (4b2f0e6e NU1101 packageSourceMapping, ce9fa07f ASPNETCORE_HTTP_PORTS re-bind, 9137cb41 empty-address-space-on-cache-boot, c6a9f93a cache-not-repopulated-on-RestoreApplied), each with a regression test. Two documented limitations (follow-ups): no replication back-fill of a fully-wiped node; oplog growth on default-OFF nodes. Post-fix: full build 0 errors, Runtime.Tests 409/0, all LocalDb Host.IntegrationTests green; only consistent failure is the infra-gated AbCip_Green_AgainstSim (sim not up). Evidence: 2026-07-20-localdb-phase1-live-gate.md. NOT merged."
+    }
+  ],
+  "lastUpdated": "2026-07-20T00:00:00Z"
+}

docs/plans/2026-07-20-localdb-adoption-phase2.md

@@ -0,0 +1,316 @@
+# OtOpcUa LocalDb Adoption — Phase 2 Implementation Plan (alarm-historian store-and-forward)
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+>
+> **Execution model:** Optimized for **Claude Opus** agents (`claude --model opus`); dispatch
+> `high-risk` tasks on Opus. Branch **`feat/localdb-phase2`** in `~/Desktop/OtOpcUa` (remote
+> `lmxopcua`). **Prerequisite: Phase 1 (`docs/plans/2026-07-20-localdb-adoption-phase1.md`) is
+> merged (or this branch is stacked on it) and its live gate passed.** Do not merge as part of this
+> plan; stop at the DoD task.
+>
+> **Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md` §3.4,
+> D8, D9. Reference implementation for "replace a bespoke store" phasing: ScadaBridge Phase 2
+> (`~/Desktop/ScadaBridge/docs/plans/2026-07-19-localdb-adoption-phase2.md` + its live gate doc).
+
+**Goal:** Move the alarm-historian store-and-forward buffer (today the standalone
+`alarm-historian.db` owned by `SqliteStoreAndForwardSink`) into the consolidated LocalDb file as a
+replicated table with a primary-gated drain, so a redundant pair no longer loses buffered alarm
+history when a node dies — and delete the sink's bespoke file/connection management outright.
+
+**Architecture:** `alarm_sf_events` (TEXT GUID PK) registered in `LocalDbSetup.OnReady`; the sink
+rewired onto `ILocalDb` behind its unchanged public seam; drain gated on the delivered-snapshot
+Primary role via `PrimaryGatePolicy` (at-least-once across failover, accepted and documented); a
+one-time idempotent migrator from the legacy file, running **after** registration.
+
+**Risk framing (from ScadaBridge):** Phase 2 replaces a *working* mechanism — a harder risk class
+than Phase 1's "add where none existed." Cutover (delete + rewire) lands in **one commit**; there is
+no dual-mechanism period, and the cutover is the test.
+
+**Hard rules:** identical to Phase 1's list (OnReady ordering, no autoincrement/BLOB, fail-closed
+auth already in place, no in-memory SQLite in tests, cp-triplet-only rig inspection), plus:
+- **Legacy-copy column lists must INTERSECT** with what the legacy file actually has
+  (`pragma_table_info` probe) — a missing column throws and readers silently discard every row.
+- Absence assertions need a positive control.
+- DoD greps phrased as "no references from code" (explanatory comments may survive).
+
+---
+
+### Task 0: Recon (produces `docs/plans/2026-07-20-localdb-phase2-recon.md`)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Create: `docs/plans/2026-07-20-localdb-phase2-recon.md`
+- Read-only: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs` (and the
+  whole `Core.AlarmHistorian` project), `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/AlarmHistorianOptions.cs`,
+  the drain worker (whatever forwards batches to `GatewayHistorian`/`SendEvent`),
+  `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/PrimaryGatePolicy.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` (how `_localRole` +
+  driver-member-count reach the gate today)
+
+**Record, with `file:line` citations:**
+1. The sink's exact table schema: name, columns, PK type (**autoincrement?** — decides whether the
+   migrator needs deterministic `mig-{node}-{legacyId}` ids), indices, and any BLOB columns
+   (**STOP condition:** a BLOB payload column cannot be registered — it must become base64 TEXT in
+   the new schema, and the recon must size the largest realistic payload against the 171 KB-ish
+   chunk guidance; alarm events are small JSON, so expect this to be fine, but verify).
+
+   > **PRE-ANSWERED 2026-07-21 (verified against `master` `d218282c`) — the STOP condition does NOT
+   > fire.** Re-verify cheaply during recon, but do not expect a surprise:
+   >
+   > - **Executed DDL:** `SqliteStoreAndForwardSink.cs:657-667`. It matches the class doc-comment at
+   >   `:17-26` exactly — no drift between the documented and executed schema.
+   > - **No BLOB.** All 8 columns are TEXT/INTEGER; the payload column is **`PayloadJson TEXT NOT
+   >   NULL`**. So no base64 conversion is needed and the chunk-size sizing is moot.
+   > - **Payload is small and bounded by shape.** `PayloadJson` is a serialized
+   >   `AlarmHistorianEvent` — 10 scalar fields (`AlarmId`, `EquipmentPath`, `AlarmName`,
+   >   `AlarmTypeName`, `Severity`, `EventKind`, `Message`, `User`, `Comment?`, `TimestampUtc`).
+   >   No collections, no nesting. Realistic worst case is low single-digit KB (operator `Comment`
+   >   is the only unbounded-ish field), far under the 171 KB chunk guidance.
+   > - **PK IS autoincrement** — `RowId INTEGER PRIMARY KEY AUTOINCREMENT`. This confirms the
+   >   plan's own prediction: LocalDb cannot replicate an autoincrement key, so the migrator
+   >   **must** mint deterministic `mig-{node}-{legacyId}` ids, and the new table needs a TEXT
+   >   GUID PK as already specified in the Architecture note.
+   > - **One index to carry across:** `IX_Queue_Drain ON Queue (DeadLettered, RowId)` (`:667`) —
+   >   the drain's covering index. The `alarm_sf_events` equivalent wants the same shape over
+   >   (dead-lettered flag, insertion order) so the drain query stays index-covered.
+   > - **Legacy column list for the migrator's `pragma_table_info` intersection check:**
+   >   `RowId, AlarmId, EnqueuedUtc, PayloadJson, AttemptCount, LastAttemptUtc, LastError,
+   >   DeadLettered`.
+2. The public seam: the interface the drain worker and producers use (e.g. `IAlarmHistorianSink` /
+   enqueue+dequeue+markDelivered+deadLetter methods), so the rewire can keep it byte-compatible.
+3. Semantics to preserve: `Capacity` (1,000,000) enforcement, `MaxAttempts` (10), dead-letter
+   retention (30 d), `BatchSize` (100), `DrainIntervalSeconds` (5) — where each lives.
+4. The drain worker's lifecycle: hosted service or actor? Where a Primary-role check can be
+   injected, and how the delivered-snapshot role (`RedundancyStateChanged` cache) is accessible
+   from it (via `DriverHostActor`, a shared status service, or a message). If the role is only
+   available inside `DriverHostActor`, note the cleanest bridge (e.g. an `IRedundancyRoleView`
+   singleton the actor updates) — that becomes Task 4's shape.
+5. Whether the sink is constructed per-node config path (`AlarmHistorian:DatabasePath`) anywhere
+   else (tests, tooling).
+6. How `AlarmHistorian:Enabled=false` short-circuits (NullAlarmHistorianSink) — the rewire must
+   keep the disabled path allocating no LocalDb tables? No: tables are created unconditionally in
+   `OnReady` (cheap, empty); only the sink/drain stay Null. Note this in the doc.
+
+Commit: `docs(localdb): phase-2 recon findings`.
+
+---
+
+### Task 1: `alarm_sf_events` schema + registration (+ tests)
+
+**Classification:** standard
+**Estimated implement time:** ~4 min
+**Parallelizable with:** none (Task 2 depends on it)
+
+**Files:**
+- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmSfSchema.cs` (depends only on
+  `Microsoft.Data.Sqlite`)
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs`
+- Test: extend `LocalDbSetupTests`
+
+Schema shape (adjust column names to the recon's findings — preserve today's semantics):
+```sql
+CREATE TABLE IF NOT EXISTS alarm_sf_events (
+    id               TEXT    NOT NULL PRIMARY KEY,   -- app-minted GUID (never autoincrement)
+    payload_json     TEXT    NOT NULL,
+    enqueued_at_utc  TEXT    NOT NULL,
+    attempts         INTEGER NOT NULL DEFAULT 0,
+    status           TEXT    NOT NULL DEFAULT 'pending',  -- pending | delivered | dead
+    last_attempt_utc TEXT    NULL,
+    dead_at_utc      TEXT    NULL
+);
+CREATE INDEX IF NOT EXISTS ix_alarm_sf_events_status ON alarm_sf_events(status, enqueued_at_utc);
+```
+`OnReady` order becomes: Phase-1 DDL → `AlarmSfSchema.Apply` → the two Phase-1
+`RegisterReplicated` calls → `RegisterReplicated("alarm_sf_events")` → **migrator (Task 5) last**.
+(All DDL may run before all registrations; the invariant is registration-before-writes.)
+
+TDD: failing test first — exact replicated set becomes
+`["alarm_sf_events", "deployment_artifacts", "deployment_pointer"]` (ordinal-sorted; update the
+Phase-1 exact-set pins in the same commit — they are *supposed* to go red here, that's them
+working). Oplog-capture test for an `alarm_sf_events` insert. Commit
+`feat(localdb): alarm_sf_events replicated table`.
+
+---
+
+### Task 2: Rewire the sink onto `ILocalDb` + delete bespoke file management (the cutover commit, part 1 of 2 — see Task 3)
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs` (or replace
+  with `LocalDbStoreAndForwardSink.cs` — keep the public seam identical either way)
+- Modify: its registration (`AddAlarmHistorian`) to inject `ILocalDb`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/AlarmHistorianOptions.cs` — remove
+  `DatabasePath` (a breaking config key removal: note it in the runbook/CHANGELOG task)
+- Test: the sink's existing unit tests, rewired to a temp-file `ILocalDb` via `TestLocalDb`-style
+  helper (create `tests/.../TestSupport` helper if none exists — real DB, never a stub: a stubbed
+  bare `SqliteConnection` lacks `zb_hlc_next()` and fails closed on registered tables)
+
+**Steps:**
+1. Write/port failing tests for the seam's semantics: enqueue, drain batch of `BatchSize`,
+   `MaxAttempts` → dead-letter, capacity enforcement, dead-letter retention purge.
+2. Implement over `ILocalDb.ExecuteAsync/QueryAsync` (anonymous-object params). Delete the private
+   connection/pragma/file-open code and any `PRAGMA journal_mode` calls (LocalDb owns pragmas).
+   GUIDs minted at enqueue (`Guid.NewGuid().ToString("N")`).
+3. `Capacity` enforcement: count-based insert guard (preserve today's overflow behavior per recon).
+4. Core.AlarmHistorian gains a package ref on core `ZB.MOM.WW.LocalDb` (interface only).
+5. Build + project tests green. **Commit together with Task 3** if the drain gate can't compile
+   separately (the ScadaBridge tasks-14/15/16 circular-dependency landmine — check before assuming
+   they're independent commits).
+
+---
+
+### Task 3: Primary-gated drain
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:** (exact shape from recon item 4)
+- Modify: the drain worker
+- Create (if recon says so): `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/IRedundancyRoleView.cs`
+  — a singleton snapshot (`RedundancyRole? LocalRole`, `int DriverMemberCount`) updated by
+  `DriverHostActor` where it already caches `_localRole`
+- Test: drain-worker tests + an actor test pinning that `DriverHostActor` publishes role changes to
+  the view
+
+**Semantics:**
+- Drain runs only when `PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount)` is
+  true — same policy, same boot-window posture (unknown role drains only when the node is alone).
+- Delivered/dead-letter marks are row UPDATEs → they replicate, so the standby's copy tracks drain
+  progress and does not re-deliver already-marked rows after failover.
+- **At-least-once across failover is accepted:** rows delivered on the old primary whose
+  `delivered` mark hadn't replicated yet will be re-sent by the new primary. Document in the
+  runbook (Task 7); do NOT build dedup.
+- When replication is OFF (default), the gate still applies but `driverMemberCount` for a solo
+  node keeps today's behavior — verify with a test: single-node, role unknown → drains (no
+  regression for unpaired deployments).
+
+TDD: failing tests — secondary role does not drain; primary drains; unknown+alone drains;
+unknown+paired does not. Commit (with Task 2 if coupled):
+`feat(localdb): alarm S&F on LocalDb with primary-gated drain (cutover)`.
+
+---
+
+### Task 4: One-time legacy migrator (`alarm-historian.db` → consolidated)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 6
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/AlarmSfLegacyMigrator.cs`
+- Modify: `LocalDbSetup.OnReady` — call it **last**, after all `RegisterReplicated` calls
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/AlarmSfLegacyMigratorTests.cs`
+
+Mirror `~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs`:
+- Source path from the pre-removal `AlarmHistorian:DatabasePath` default (`alarm-historian.db`) —
+  read the raw config key even though the option property is gone.
+- `ShouldMigrate`: skip if file missing or `<file>.migrated` exists. `:memory:`/`file:` sources → no-op.
+- **If the legacy PK is autoincrement (recon):** deterministic ids `mig-{NodeName}-{legacyId}`
+  (node name from the recon-identified config key) — rerunnable without duplicates under
+  `INSERT OR IGNORE`, and no cross-node collision. If already GUIDs, copy as-is.
+- One transaction for the whole copy; `File.Move(path, path + ".migrated")` only after commit;
+  failure throws out of `OnReady` → boot fails, legacy untouched.
+- **Column intersection** via `pragma_table_info` on the legacy table; required-PK guard.
+- Both pair nodes migrate independently; their rows have distinct ids (node-prefixed), so the
+  merged buffer is the union — expected; note that the new primary will drain the standby's
+  migrated rows too.
+
+TDD: failing tests — happy path row counts; idempotent re-run; failure leaves legacy file
+untouched; migrated rows **enter the oplog** (assert `__localdb_oplog` — the registration-order
+pin); older legacy file missing a column still migrates (intersection). Commit
+`feat(localdb): one-time alarm-historian.db migrator`.
+
+---
+
+### Task 5: Convergence + failover scenarios in the pair harness
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 4
+
+**Files:**
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/AlarmSfConvergenceTests.cs`
+  (reuses the Phase-1 `LocalDbPairHarness`)
+
+**Scenarios:**
+1. `AlarmBurstOnA_ConvergesToB_AndOplogDrains` — enqueue N events on A; identical rowset on B;
+   oplog → 0.
+2. `DeliveredMarksReplicate` — mark rows delivered on A; B's copies show delivered (the
+   no-redeliver-after-failover property, asserted at the data layer).
+3. `WritesWhileTransportDown_SurviveRejoin`.
+4. Positive control: with `RegisterReplicated("alarm_sf_events")` commented out, scenarios 1–3 go
+   red (run locally, record, restore — do not commit red).
+
+Commit `test(localdb): alarm S&F convergence scenarios`.
+
+---
+
+### Task 6: Rig config + docs
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 4
+
+**Files:**
+- Modify: `docker-dev/docker-compose.yml` — enable `AlarmHistorian__Enabled=true` on the site-a
+  pair if not already (the gate needs a live buffer); remove any `AlarmHistorian__DatabasePath`
+  env vars (key deleted).
+- Modify: `docs/operations/2026-07-20-localdb-pair-replication.md` — add: alarm S&F replication
+  semantics, at-least-once-across-failover statement, migrator behavior (`.migrated` sidecar),
+  `DatabasePath` key removal.
+- Modify: `docs/AlarmHistorian.md` + `CLAUDE.md` — sink now lives in the consolidated LocalDb;
+  drain is primary-gated.
+
+Commit `docs+chore(localdb): phase-2 rig config + docs`.
+
+---
+
+### Task 7: DoD sweep (offline)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+1. Full solution build → 0 warnings; full test suite green (deltas vs pre-branch baseline only).
+2. Greps, phrased as "no references from **code**": the old bespoke connection management
+   (`AlarmHistorian:DatabasePath`, direct `new SqliteConnection` inside Core.AlarmHistorian except
+   via schema helpers/tests) — explanatory comments may remain.
+3. Positive-control evidence from Task 5 recorded.
+4. Exact-set replicated-tables pin = 3 tables, both directions.
+5. Update `…phase2.md.tasks.json`; commit `chore(localdb): phase-2 DoD sweep`; STOP and report.
+
+---
+
+### Task 8: Live gate on the docker-dev rig (run only with explicit user go-ahead)
+
+**Classification:** high-risk (rig)
+**Estimated implement time:** ~30 min wall-clock
+**Parallelizable with:** none
+
+Same inspection rules as Phase 1's gate (cp-triplet, curl sidecar, observer-suspicion rule).
+Record in `docs/plans/2026-07-20-localdb-phase2-live-gate.md`:
+1. Migration ran: `.migrated` sidecar present on both site-a nodes; row counts match legacy.
+2. Alarm burst (drive a real driver alarm or the historian-gateway-unreachable path) converges:
+   identical rowsets, oplog drains to 0, dead letters 0.
+3. Only the primary drains: stop the historian gateway egress, buffer builds on BOTH nodes'
+   tables (replicated), but only the primary's drain worker logs attempts.
+4. Failover: stop the primary; the standby (new primary) resumes draining the shared buffer;
+   count of double-delivered events observed and recorded (at-least-once evidence, not a failure).
+5. Both-nodes-together restart clean; zero `disk I/O error`/`SQLITE_IOERR` in logs.
+6. site-b (default-OFF pin): sink works locally, no sync traffic.
+
+Commit the gate doc; report; do not merge.
+
+---
+
+## Task persistence
+
+Tasks file: `docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json`. Record deviations per
+task — especially if Tasks 2/3 had to land as one commit (the expected outcome).

docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json

@@ -0,0 +1,86 @@
+{
+  "planPath": "docs/plans/2026-07-20-localdb-adoption-phase2.md",
+  "tasks": [
+    {
+      "id": 0,
+      "subject": "Task 0: Recon \u2014 sink schema, seam, drain lifecycle, role-view bridge (STOP conditions)",
+      "status": "completed",
+      "note": "STOP condition does NOT fire (no BLOB; PayloadJson TEXT). Recon doc: docs/plans/2026-07-20-localdb-phase2-recon.md. Key finding: the drain worker is an internal Timer inside the sink (not a hosted service/actor), so the gate is a Func<bool> ctor param; and HistorianAdapterActor already primary-gates ENQUEUE with a different policy (ShouldHistorize) - which is why today's ungated drain is safe and why replication breaks it. Deviations D-1..D-4 recorded."
+    },
+    {
+      "id": 1,
+      "subject": "Task 1: alarm_sf_events schema + registration (+ exact-set pin update)",
+      "status": "completed",
+      "blockedBy": [
+        0
+      ],
+      "note": "alarm_sf_events created in AlarmSfSchema (Core.AlarmHistorian) + registered third in LocalDbSetup.OnReady. Exact-set pin updated to 3 tables. DEVIATION D-5: kept the legacy delete-on-ack + dead_lettered flag + last_error rather than the plan's status column (no sweeper for 'delivered' rows; last_error is the only record of why a row died). Drain ORDER BY moves to (enqueued_at_utc, id)."
+    },
+    {
+      "id": 2,
+      "subject": "Task 2: Rewire sink onto ILocalDb + delete bespoke file management (cutover 1/2)",
+      "status": "completed",
+      "blockedBy": [
+        1
+      ],
+      "note": "Sink rewritten as LocalDbStoreAndForwardSink over ILocalDb; bespoke file/pragma/schema management deleted with the old class. AlarmHistorian:DatabasePath removed (breaking config key). Ids are a deterministic payload hash (D-1), not GUIDs."
+    },
+    {
+      "id": 3,
+      "subject": "Task 3: Primary-gated drain via PrimaryGatePolicy (cutover 2/2 \u2014 may co-commit with Task 2)",
+      "status": "completed",
+      "blockedBy": [
+        2
+      ],
+      "note": "Drain gated on IRedundancyRoleView, a singleton DriverHostActor publishes PrimaryGatePolicy's verdict to on every snapshot. Fails closed on a throwing gate; seeded OPEN so a non-redundant deployment is never silently stopped. New HistorianDrainState.NotPrimary + transition-logged (D-2). Landed with Task 2 as one commit (D-3)."
+    },
+    {
+      "id": 4,
+      "subject": "Task 4: One-time alarm-historian.db legacy migrator",
+      "status": "completed",
+      "blockedBy": [
+        3
+      ],
+      "note": "AlarmSfLegacyMigrator runs LAST in OnReady (which now takes IConfiguration - no skip-migration overload exists). DEVIATION D-6: ids are the payload hash (AlarmSfSchema.DeriveId, lifted out of the sink) rather than mig-{node}-{legacyId} - a warm pair's two legacy files OVERLAP, and node-prefixing would carry that duplication forward forever. DEVIATION: tests live in Host.IntegrationTests; the plan's Host.Tests project does not exist."
+    },
+    {
+      "id": 5,
+      "subject": "Task 5: Convergence + failover scenarios in the pair harness (+ positive control)",
+      "status": "completed",
+      "blockedBy": [
+        3
+      ],
+      "note": "4 scenarios green. POSITIVE CONTROL RUN (RegisterReplicated(alarm_sf_events) commented out): 3/4 went red immediately; the 4th (same-event-on-both-nodes) passed VACUOUSLY - with replication off each node trivially held its own single row. Strengthened by enqueuing a second distinct event on B and asserting BOTH nodes hold 2; it then went red under the control too. Control restored, all 4 green. Also fixed a SECOND exact-set pin the plan predicted: LocalDbWiringTests."
+    },
+    {
+      "id": 6,
+      "subject": "Task 6: Rig config + docs",
+      "status": "completed",
+      "blockedBy": [
+        3
+      ],
+      "note": "Rig: AlarmHistorian__Enabled=true on site-a-1/2 AND site-b-1/2, with MaxAttempts raised to 1e6 (D-4: the default 10 would dead-letter the queue ~5min into the rig's permanent gateway outage) and a deliberately unresolvable ServerHistorian__Endpoint. NOTE: the endpoint is REQUIRED - ServerHistorianOptionsValidator is consumer-gated on AlarmHistorian:Enabled, so an empty one fails host start. Docs: runbook, AlarmHistorian.md, AlarmTracking.md, Configuration.md, Redundancy.md, docs/README.md, CLAUDE.md."
+    },
+    {
+      "id": 7,
+      "subject": "Task 7: DoD sweep (offline) \u2014 STOP and report after this",
+      "status": "completed",
+      "blockedBy": [
+        4,
+        5,
+        6
+      ],
+      "note": "Build: 0 errors solution-wide; 0 warnings from every project this branch touches (the ~816 solution-wide warnings are pre-existing xUnit1051/OTOPCUA0001/CS86xx in untouched driver + client test projects). GREPS FOUND REAL DRIFT Task 6 missed: 8 live sites still named the deleted SqliteStoreAndForwardSink - CLAUDE.md, the AdminUI /alarms/historian panel text (user-visible), HistorianAdapterActor (a <see cref> that did NOT warn), Runtime + Host + 2 Driver.Historian.Gateway doc comments, 1 gateway test comment, docs/drivers/Historian.Wonderware.md; plus docs/AlarmTracking.md still claimed a Validate() startup warning for a relative DatabasePath (that branch is gone). All fixed. Code refs to AlarmHistorian:DatabasePath now reduce to exactly two intentional ones: AlarmSfLegacyMigrator.LegacyPathKey and its test. No 'new SqliteConnection' in Core.AlarmHistorian. Both exact-set pins assert the 3-table set, both directions. Guard-deletion evidence (both vacuous passes + the pin inventory) consolidated into the recon doc."
+    },
+    {
+      "id": 8,
+      "subject": "Task 8: Live gate on the docker-dev rig (needs explicit user go-ahead)",
+      "status": "completed",
+      "blockedBy": [
+        7
+      ],
+      "note": "Gate doc: docs/plans/2026-07-20-localdb-phase2-live-gate.md. Checks 1/2/5/6 PASS; checks 3+4 NOT SATISFIED (see below). FOUR production defects found + fixed, three of which crash-looped every driver node before check 1 could run: (1) empty ServerHistorian:ApiKey crashes the host - the validator had explicitly classified it as 'degrades', which is false because the gateway client validates its own options at construction; (2) UseTls vs endpoint scheme mismatch crashes likewise, both directions; (3) plaintext h2c was UNREACHABLE - the adapter forwarded TLS-only options unconditionally so every documented http:// deployment crashed; (4) THE BLOCKER - the drain gate deferred to a CLUSTER-WIDE elected Primary while the queue is PAIR-LOCAL. On the rig that Primary is central-1 (carries the driver Akka role, replicates nobody, runs no historian), so ALL FOUR driver nodes suspended their drains including unpaired site-b: silent permanent loss where pre-Phase-2 drained fine. Fixed in 3 layers (separate ShouldDrainAlarmHistory policy; peer-host matching in DriverHostActor; gate short-circuited entirely when replication is unconfigured, testing BOTH PeerAddress and SyncListenPort since only the dialer sets the former). Also fixed a THIRD vacuous test: AwaitAssert on the seeded-open value passed with the guard deleted; now asserts the sequence of PUBLISHED values via a recording view. Migration evidence: 11 legacy rows across two overlapping files -> exactly 9 identical rows on both nodes, proving D-6's payload-hash identity live. OPEN DESIGN FORK (in the gate doc): a pair cannot identify its own Primary, so both halves drain - safe (no loss, duplicates only) but the gate's de-dup benefit is unrealised."
+    }
+  ],
+  "lastUpdated": "2026-07-21T00:00:00Z"
+}

docs/plans/2026-07-20-localdb-phase1-live-gate.md

@@ -0,0 +1,93 @@
+# LocalDb Phase 1 — live gate (docker-dev rig)
+
+> Task 16 of `2026-07-20-localdb-adoption-phase1.md`. Evidence log for the 8 live-rig checks.
+> Rig: local `docker-dev/docker-compose.yml` (6 host nodes, one shared SQL). site-a pair replicates;
+> central + site-b are default-OFF.
+>
+> **Safety rules followed:** all DB inspection via `docker cp` of the `db`/`-wal`/`-shm` triplet out
+> of the container, querying the copy — never host `sqlite3` on the live WAL file. Metrics via a
+> `curlimages/curl` sidecar with `--network container:<node>` (`aspnet:10.0` has no curl).
+
+## Rig build/up
+
+Two real defects surfaced by the rebuild (both fixed on-branch, re-verified):
+
+1. **`NU1101` — packageSourceMapping gap** (`4b2f0e6e`). `ZB.MOM.WW.LocalDb*` was not mapped to the
+   `dohertj2-gitea` feed, so any *clean* restore (Docker image build, CI, fresh clone) failed. Local
+   dev builds masked it via the warm global NuGet cache. Fixed by adding the two patterns; verified
+   with a temp-packages-dir restore.
+2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`). The Kestrel re-bind read only
+   `urls`/`ASPNETCORE_URLS`; the aspnet:8.0+ base image sets `ASPNETCORE_HTTP_PORTS=8080` (all
+   interfaces) as the container default instead. A driver node that never sets `URLS` fell through to
+   `localhost:5000`, silently moving its health/metrics surface to loopback. Central nodes were fine
+   (they set `ASPNETCORE_URLS=9000` explicitly). Fixed to fall through URLS → HTTP_PORTS/HTTPS_PORTS →
+   5000; 4 new unit tests. After the fix, driver nodes bind `[::]:8080` + `[::]:9001` (site-a) /
+   `[::]:8080` only (site-b).
+
+Rig: 6 nodes rebuilt from branch `feat/localdb-phase1` @ `ce9fa07f`, all up.
+
+## Checks
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 1 | Rig up, site-a pair healthy, `/metrics` shows `localdb_` series | ✅ PASS | Both site-a nodes `/healthz` 200 Healthy; `/metrics` shows `localdb_oplog_depth` + `localdb_sync_reconnects_total` on meter `ZB.MOM.WW.LocalDb.Replication` (allowlist works); site-a-2 served `/localdb_sync.v1.LocalDbSync/Sync` with routing `match_status=success` (interceptor passed, session established); both oplogs depth 0 |
+| 2 | Deploy → both site-a nodes' pointer + artifacts byte-identical, same HLC + origin | ✅ PASS | Deploy `6e687451…` (rev `efb04c79…`) via `POST :9200/api/deployments` → both site-a-1 and site-a-2 `deployment_pointer` = `SITE-A / 6e687451… / efb04c79… / sha EFB04C79…` (identical); artifact 1 chunk each; **pointer `__localdb_row_version` identical on both: hlc `116955620225646592`, origin node_id `a0576df6-…`, tombstone 0** — one row replicated, not two derivations; both oplogs depth 0 |
+| 3 | Boot-from-cache: SQL down, restart site-a-2, serves cached config w/ signal | ✅ PASS (found + fixed a defect) | With SQL stopped, site-a-2 restart logged `RUNNING FROM CACHE — … booted deployment 6e687451…`; **address space materialised from the cached blob**: `AddressSpaceApplier: applied plan (added=18)` / `OpcUaPublish: applied rebuild (added=18)`. Browse confirms **16 nodes on site-a-2 = 16 on site-a-1**. **Defect the gate caught (`9137cb41`):** pre-fix the rebuild re-read the artifact from the down ConfigDb and no-op'd, so the cache-booted node served **1** node vs the healthy peer's 16 — it logged success but browsed empty. Fixed by passing the in-hand cached blob to `RebuildAddressSpace`; regression test added |
+| 4 | Replicated-cache payoff: wipe a-2 DB, repopulate, then SQL-down boot | ⚠️ PARTIAL — 1 defect fixed; the limitation is since **CLOSED** (LocalDb 0.1.2 — see Summary) | Wiped a-2's LocalDb volume, restarted with **SQL down**: sync session re-established (a-2 served a `Sync` call) but a-2's cache stayed **0/0/0** and a-1's oplog was **0** — the deploy's rows had been acked by the old a-2 and pruned, so there was no delta to send and **no snapshot-resync fired**. ⇒ **Limitation: pure replication does not back-fill a fully-wiped, already-converged node** (library delta-replication; snapshot-resync is gated on the oplog cap, which the running engine never hits). Separately found a **defect (`c6a9f93a`)**: recovery via `RestoreApplied` never wrote the cache (only fresh `ApplyAndAck` did), so a wiped node stayed cache-less. Fixed: `RestoreApplied` now re-caches. Verified — after the fix, wiped a-2 restarted with SQL up: `restored served state … on bootstrap` → cache repopulated `SITE-A / 6e687451`, chunks=1. **Net: a wiped node self-heals its cache from central on next boot (regaining boot-from-cache for future outages); it is not back-filled by its peer during the same outage.** |
+| 5 | Transport kill: pause a-2, oplog rises; unpause, drains to 0 | ✅ PASS | `docker pause` on a-2 severed the sync transport; during the window a-2 accumulated **oplog=2** unacked local writes (its re-cache) while a-1 held 0. `docker unpause` → a-2's oplog **drained to 0 within 5s**, both converged on the same pointer. Note: the rig's redeploys carry an identical revision, so a fresh cache write on a-1 short-circuits (no-op) — the store-during-partition variant is covered by the automated harness scenario `WritesWhileTransportDown_SurviveRejoin` (proven red without `RegisterReplicated`) |
+| 6 | Retention: 3rd deploy prunes oldest; chunks gone + tombstones on BOTH nodes | ✅ PASS | Cached 3 distinct deployment-ids (via deploy + restart re-cache). Both nodes retain exactly the newest 2 (`475df87a`, `962962dc`); the oldest `6e687451`'s chunks are **0 on both** (pruned); **1 artifact tombstone on both** — the prune replicated as a tombstone, not left as a live chunk |
+| 7 | Both-nodes-together restart: clean rejoin, identical counts, no I/O errors | ✅ PASS | Restarted site-a-1 + site-a-2 together: **0** `disk I/O error`/`SQLITE_IOERR`/`corrupt` lines in either log; both re-cached and converged to **identical** counts (ptr `962962dc…`, chunks=2, rowver=3). Byte-identical convergence preserved across a simultaneous restart |
+| 8 | site-b default-OFF pin: cache works locally, Healthy, no sync listener on 9001 | ✅ PASS (+1 finding, since **CLOSED** by the skip-if-unchanged cache write — see Summary) | site-b-1 cache works locally (`SITE-B / 6e687451`, chunks=1, from its own apply); listeners **`[::]:8080` only — no 9001**; port 9001 **not bound** (`/proc/net/tcp`); `/healthz` **200**; no sync/reconnect metric series. **Finding (follow-up, not a blocker):** `localdb_oplog_depth=5` — because `OnReady` registers replication unconditionally, a default-OFF node captures cache writes into its oplog but has no peer to ack/prune them, so the oplog grows slowly (≈2 rows/deploy, amplified by the re-cache-on-restore fix). Bounded by the library `MaxOplogRows` cap (1M); worth a periodic pruner or skip-if-unchanged in a follow-up |
+
+## Summary
+
+**8/8 checks pass** (check 4 with a documented limitation). The gate exercised the real image built
+from the branch, the real interceptor over a real loopback+cross-container h2c transport, and real
+`docker cp`-triplet DB inspection — and caught **four real defects that every offline test had
+missed**, each now fixed on-branch with a regression test:
+
+1. **`NU1101` packageSourceMapping gap** (`4b2f0e6e`) — `ZB.MOM.WW.LocalDb*` unmapped to the gitea
+   feed; every clean restore (Docker, CI, fresh clone) failed. Masked locally by the warm NuGet cache.
+2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`) — the sync-listener re-bind ignored the
+   aspnet:8.0+ container-default port var, silently moving driver health/metrics to loopback:5000.
+3. **Empty address space on boot-from-cache** (`9137cb41`) — the rebuild re-read the artifact from the
+   down ConfigDb and no-op'd, so a cache-booted node logged success but served clients **1 node vs the
+   healthy peer's 16**. The single most important find: boot-from-cache was half-working.
+4. **Cache not repopulated on `RestoreApplied`** (`c6a9f93a`) — a wiped/fresh-volume node recovered its
+   served state from central but stayed cache-less, unable to boot-from-cache on the next outage.
+
+**Two documented limitations (follow-ups, not blockers) — both since CLOSED:**
+
+- **No replication back-fill of a fully-wiped node** (check 4): a peer's already-acked cache rows are
+  pruned from its oplog and snapshot-resync is gated on the (never-hit) oplog cap, so a wiped node is
+  not healed by its peer — it self-heals from central on next boot instead.
+  → **FIXED in `ZB.MOM.WW.LocalDb` 0.1.2 + 0.1.3** (scadaproj `cad3bcb` + `3b7489a`), live-gated in
+  `2026-07-21-localdb-followups-live-gate.md`. The diagnosis in this doc was
+  slightly off: the cap was not the gate. `ComputeSnapshotRequiredAsync` measured the peer's gap
+  against the *oldest surviving oplog row* and read an empty oplog as "no gap possible" — and an
+  empty oplog is the **steady state of a converged pair**, since ack-pruning deletes everything the
+  peer has confirmed. So the healthy state was the one state that could not heal a wiped peer. With
+  an empty oplog the gap is now measured against `last_acked_seq`. Covered at both levels:
+  `SnapshotResyncTests.WipedPeer_AfterEverythingWasAckedAndPruned_StillGetsSnapshot` in the library,
+  and `LocalDbPairConvergenceTests.WipedNode_IsBackFilledByItsPeer_WithoutAnyNewDeploy` here —
+  the latter verified RED against the pinned 0.1.1 and green on 0.1.2. **0.1.3 closes a second,
+  deeper half that only became reachable once back-fill worked:** the rebuilt node's OWN writes were
+  silently dropped, because `last_applied_remote_seq` is a watermark in the *peer's* seq space and a
+  rebuilt peer numbers from 1 again. The follow-up live gate caught that one — offline tests could
+  not have, since the scenario had no test while it was still a documented limitation.
+- **Oplog growth on default-OFF nodes** (check 8): `OnReady` registers replication unconditionally, so
+  a node with no peer accumulates unacked/unpruned oplog rows (~2/deploy). Slow, bounded by the 1M cap;
+  a periodic pruner or a skip-if-unchanged cache write would close it.
+  → **FIXED at the source**: `LocalDbDeploymentArtifactCache.StoreAsync` now **skips entirely** when
+  the pointer already names this deployment/revision, the SHA matches the bytes, and the expected
+  chunk count is present — so re-caching an artifact the node already holds (every restart's
+  boot-from-cache, every `RestoreApplied`) writes nothing and mints no oplog rows. Two guard tests
+  pin what the skip must NOT do (stale bytes under an unchanged revision; a matching pointer over
+  missing chunks), both verified RED against a naive pointer-only skip.
+  Also correcting this row's bound: the growth was never headed for the 1M cap. `AddZbLocalDbReplication`
+  is registered unconditionally, so `MaintenanceBackgroundService` runs on default-OFF nodes too and
+  the **7-day `MaxOplogAge`** cap prunes — the true bound was ~7 days of writes, not 1M rows.
+
+Merged to `master` as `c957db52` (2026-07-21) and pushed. Both follow-ups closed on
+`fix/localdb-phase1-followups` (2026-07-21) and live-gated — see
+`2026-07-21-localdb-followups-live-gate.md`, which found a third defect in the process.

docs/plans/2026-07-20-localdb-phase1-progress.md

@@ -0,0 +1,72 @@
+# LocalDb Phase 1 — Execution Progress (resume state)
+
+**Branch:** `feat/localdb-phase1` (off `master`). Working tree CLEAN, all work committed, nothing pushed.
+**Plan:** `docs/plans/2026-07-20-localdb-adoption-phase1.md`
+**Recon:** `docs/plans/2026-07-20-localdb-phase1-recon.md` (READ THIS — it has the 8 deviations D-1..D-8 that override the plan text).
+**Skill in use:** `superpowers-extended-cc:executing-plans` — batch, verify, commit per task, report between batches.
+**Do NOT merge to master** — plan stops at Task 15 (DoD) and reports; Task 16 is a rig live-gate needing explicit user go-ahead.
+
+## Status: Tasks 0–9 DONE (10 commits). Tasks 10–16 REMAIN.
+
+| # | Task | State | Commit |
+|---|---|---|---|
+| 0 | Preflight + recon | ✅ | `42f85507` |
+| 1 | Package refs (LocalDb 0.1.1, Grpc.AspNetCore 2.76.0) | ✅ | `44255fcb` |
+| 2 | Schema + LocalDbSetup.OnReady + tests | ✅ | `3b65c24b` |
+| 3 | Fail-closed sync auth interceptor + tests | ✅ | `3d29be83` |
+| 4 | LocalDbRegistration + Program.cs DI + appsettings | ✅ | `6aff9a83` |
+| 5 | h2c sync listener + MapZbLocalDbSync (Kestrel) | ✅ | `b9ddf20e` |
+| (fix) | rename Runtime.Deployment → Runtime.DeploymentCache | ✅ | `e771a11a` |
+| 6 | IDeploymentArtifactCache + chunked impl + tests | ✅ | `a38a52b8` |
+| 9 | Delete LiteDB LocalCache | ✅ | `2bae1b4c` |
+| 7 | Cache write path in DriverHostActor | ✅ | `1becf591` |
+| 8 | Boot-from-cache + RunningFromCache signal | ✅ | `a27eff32` |
+| 10 | Health check + telemetry meter allowlist | ⬜ TODO | |
+| 11 | DI-pin integration tests over real host graph | ⬜ TODO | |
+| 12 | 2-node convergence harness + scenarios | ⬜ TODO | |
+| 13 | docker-dev rig config | ⬜ TODO | |
+| 14 | Documentation | ⬜ TODO | |
+| 15 | DoD sweep (offline) — STOP + report | ⬜ TODO | |
+| 16 | Live gate on docker-dev rig — NEEDS USER GO-AHEAD | ⬜ TODO | |
+
+Native task list (TaskCreate ids 1–17 map to plan tasks 0–16) tracks the same; tasks.json in plan dir mirrors it.
+
+## What exists now (key files)
+
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/DeploymentCacheSchema.cs` — DDL, namespace `Runtime.DeploymentCache`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactCache.cs` — `StoreAsync` / `GetCurrentAsync` / **`GetCurrentUnkeyedAsync`** (D-1 addition) + `CachedDeploymentArtifact` record.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/LocalDbDeploymentArtifactCache.cs` — chunked impl. NOTE the prune guard `AND deployment_id <> @DeploymentId` (bug I found: pointer-target could be pruned under same-tick GUIDs → permanent silent miss).
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs` — `OnReady` (DDL→RegisterReplicated). PUBLIC (Host has no InternalsVisibleTo).
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs` — `AddOtOpcUaLocalDb`, `SyncListenPort(config)`. Registers `IDeploymentArtifactCache → LocalDbDeploymentArtifactCache` singleton.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs` — fail-closed bearer.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/KestrelHttpBinding.cs` — URL parse/re-bind helper (D-6).
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` — LocalDb wired in `hasDriver` branch (~L176 after AddAlarmHistorian: `AddOtOpcUaLocalDb` + `AddGrpc(interceptor)`); Kestrel re-bind block before `builder.Build()`; `MapZbLocalDbSync()` gated `hasDriver && syncListenPort>0` before `MapOtOpcUaHealth()`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json` — `LocalDb` section (Path, SyncListenPort:0, Replication{}), between AlarmHistorian and Deployment.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` — cache dep threaded LAST in Props/ctor/forwarding; `CacheAppliedArtifact` (write, after PushDesiredSubscriptions, empty-blob skip, own try/catch); `TryBootFromCache` + `ApplyCachedArtifact` (in Bootstrap catch); `PushDesiredSubscriptionsFromArtifact` split out; `_isRunningFromCache` field; `SingleClusterCacheKey="__single"`; `ReconcileDrivers` now returns `byte[]?`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs` — resolves `IDeploymentArtifactCache` (optional, nullable) and passes into DriverHostActor.Props.
+- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Interfaces/NodeDiagnosticsSnapshot.cs` — added `bool RunningFromCache = false`.
+- Tests (all green): `Host.IntegrationTests/LocalDbSetupTests.cs`, `LocalDbSyncAuthInterceptorTests.cs`, `LocalDbSyncListenerTests.cs`; `Runtime.Tests/Deployment/LocalDbDeploymentArtifactCacheTests.cs`, `Runtime.Tests/Drivers/DriverHostActorArtifactCacheTests.cs`, `DriverHostActorBootFromCacheTests.cs`.
+
+## Load-bearing facts for the remaining tasks
+
+- **Namespace trap (recurs!):** any `Deployment` child namespace under `Runtime` OR `Runtime.Tests` shadows the `Deployment` EF entity → CS0118 across the assembly. Use `...DeploymentCache`. Only a FULL-SOLUTION build catches it (per-project builds pass).
+- **xUnit1051 is an error** in these test projects: every `ILocalDb`/DB call in a test must pass `TestContext.Current.CancellationToken` (xunit.v3 Host.IntegrationTests) — but `Runtime.Tests` is **xunit v2** (no `TestContext.Current`).
+- **Test project reality (D-5):** NO `Host.Tests` project; NO `WebApplicationFactory<Program>` (deliberately — `TwoNodeClusterHarness.cs:48` explains why). Host unit tests live in `Host.IntegrationTests` (xunit.v3). Actor tests in `Runtime.Tests` (xunit v2 + Akka.TestKit.Xunit2).
+- **Task 10 (D-7):** `AddOtOpcUaHealth()` takes NO args, called unconditionally at `Program.cs:365`. DON'T change its signature — register the LocalDb check unconditionally and return `Healthy` when LocalDb absent (ActiveNodeHealthCheck precedent). Health check classes: there are ZERO in-repo today; `Host/Health/` has only `HealthEndpoints.cs`. Meter allowlist is REAL and REQUIRED: `Host/Observability/ObservabilityExtensions.cs:30` `o.Meters = [OtOpcUaTelemetry.MeterName]` — add `LocalDbMetrics.MeterName` (from `ZB.MOM.WW.LocalDb.Replication`). Health check reads `ISyncStatus` + `IOptions<ReplicationOptions>`.
+- **Task 11:** copy the `TwoNodeClusterHarness` direct-host-build pattern, NOT WebApplicationFactory. Pins: `ILocalDb` singleton + `ReplicatedTables` == `["deployment_artifacts","deployment_pointer"]`; `IDeploymentArtifactCache`→concrete; `ISyncStatus` default-OFF (`Connected==false`); admin-only graph has NO `ILocalDb` + doesn't demand `LocalDb:Path`; health check registered. Set `OTOPCUA_ROLES` per case; `LocalDb:Path`→temp file, `ClearAllPools`+delete triplet in cleanup.
+- **Task 12:** harness copied from `~/Desktop/ScadaBridge/tests/.../LocalDbSitePairHarness.cs` + `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/tests/.../Convergence/ConvergenceFixture.cs`. Both nodes init via PRODUCTION `LocalDbSetup.OnReady`. Real loopback Kestrel h2c through the REAL interceptor. DBs as pre-constructed singletons. Positive control: comment out both `RegisterReplicated`, confirm scenarios 1–3 red, restore (don't commit red).
+- **Task 13 (D-8):** NO host node has a writable volume today — must ADD named volumes. `docker-dev/docker-compose.yml` services: `central-1/2` (admin,driver, MAIN), `site-a-1/2` (driver, SITE-A), `site-b-1/2` (driver, SITE-B). Env style `Section__Key`. `9001` free container-internal. Enable replication on **site-a pair only** (a-1 initiator w/ PeerAddress, a-2 passive, byte-identical ApiKey, MaxBatchSize=16). site-b stays OFF = default-OFF pin. Validate `docker compose config`; DON'T bring rig up.
+- **Task 15:** baseline pre-existing test failures FIRST (`git stash` → run full `dotnet test` → unstash), report deltas only. **KNOWN intermittent:** one `Runtime.Tests` failure seen ~3× across runs (mine + subagent's), never reproduced under trx logger, NOT one of the new tests — investigate/baseline here.
+- **Warnings-as-errors** across src + most test projects. Pre-existing CS0618/xUnit1051 warnings exist ONLY in `Client.Shared*` (which don't set TreatWarningsAsErrors) — not mine, ignore.
+- **SQLitePCLRaw:** no pin needed — LocalDb 0.1.1 nuspec floors `lib.e_sqlite3` at 2.1.12 (CVE-fixed). Confirmed.
+
+## Follow-ups logged (NOT in scope, don't action without asking)
+
+- `Polly.Core` in `Configuration.csproj` now orphaned (ResilientConfigReader was its only real consumer). Left in place — recorded in recon §7.3b.
+- Pre-existing: `TryRecoverFromStale` marks Applied w/o reconciling drivers (recon §3.5); `Install-Services.ps1` gives driver-only nodes no `ASPNETCORE_URLS` (recon §6.4, worked around by D-6).
+
+## Verify-anything-quick
+
+Full build: `dotnet build ZB.MOM.WW.OtOpcUa.slnx` (expect 0 errors). LocalDb tests:
+`dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LocalDb"` and
+`dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~ArtifactCache|FullyQualifiedName~BootFromCache"`.

docs/plans/2026-07-20-localdb-phase1-recon.md

@@ -0,0 +1,617 @@
+# LocalDb Phase 1 — Task 0 Recon Findings
+
+**Date:** 2026-07-20
+**Branch:** `feat/localdb-phase1` (based on `master`)
+**Plan:** `docs/plans/2026-07-20-localdb-adoption-phase1.md`
+**Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md`
+
+All `file:line` citations are against the branch base commit.
+
+---
+
+## Step 1 — Feed verification: PASS
+
+```
+$ dotnet package search ZB.MOM.WW.LocalDb --source dohertj2-gitea --format json
+ZB.MOM.WW.LocalDb              0.1.1
+ZB.MOM.WW.LocalDb.Contracts    0.1.1
+ZB.MOM.WW.LocalDb.Replication  0.1.1
+```
+
+All three present at **0.1.1**. No STOP. (Note: the local NuGet cache holds both `0.1.0` and `0.1.1`
+— pin exactly `0.1.1` so a stale cache cannot win.)
+
+---
+
+## Step 3 — STOP conditions: NONE TRIGGERED
+
+| STOP condition | Verdict |
+|---|---|
+| Artifact apply path cannot take a post-apply hook without restructuring the actor | **Clear** — `ApplyAndAck` has a clean insertion point (§2.2) |
+| `DeploymentId` / cluster identity are not stable strings/GUIDs | **Clear** — both are stable strings, but cluster identity is **not resolvable at boot**; see D-1 |
+| LiteDB LocalCache is referenced by live code | **Clear** — zero live-code references (§7) |
+
+Three STOP conditions clear. **Eight deviations** from the plan's stated assumptions are recorded in
+§9 — four of them (D-1, D-3, D-5, D-6) change the work materially.
+
+---
+
+## 1. Deploy fetch path
+
+### 1.1 Blob-load sites — there are **five** `CreateDbContext()` calls, not two
+
+| Line | Method | Purpose |
+|---|---|---|
+| `DriverHostActor.cs:513` | `Bootstrap()` | `NodeDeploymentStates` + `Deployments` at PreStart |
+| `DriverHostActor.cs:1472` | `ReconcileDrivers(DeploymentId)` | **loads `ArtifactBlob`** |
+| `DriverHostActor.cs:1543` | `PushDesiredSubscriptions(DeploymentId)` | **loads `ArtifactBlob` again** |
+| `DriverHostActor.cs:2000` | `TryRecoverFromStale()` | latest `Sealed` deployment (id + hash only, no blob) |
+| `DriverHostActor.cs:2029` | `UpsertNodeDeploymentState(...)` | writes `NodeDeploymentState` |
+
+`DriverHostActor.cs:1472-1476`:
+```csharp
+using var db = _dbFactory.CreateDbContext();
+blob = db.Deployments.AsNoTracking()
+    .Where(d => d.DeploymentId == deploymentId.Value)
+    .Select(d => d.ArtifactBlob)
+    .FirstOrDefault() ?? Array.Empty<byte>();
+```
+`:1543-1547` is byte-identical apart from the local. **The same blob is read twice per apply.**
+
+### 1.2 Artifact runtime type — there is no instance type
+
+`DeploymentArtifact` (`DeploymentArtifact.cs:48`) is a **static parser class** over
+`ReadOnlySpan<byte>`. The blob is never materialised into an object graph.
+
+- Storage: `byte[]` — `Deployment.cs:29` `public byte[] ArtifactBlob { get; init; } = Array.Empty<byte>();`
+- Format: **UTF-8 JSON**, `JsonDocument.Parse` (`DeploymentArtifact.cs:66`, `:490`). Not MessagePack.
+- Leniency contract: empty/malformed blob → empty result, **never throws** (`:62`, `:537-541`).
+
+**Consequence for the cache:** we cache raw `byte[]`. No serializer coupling. Good.
+
+### 1.3 Identity types
+
+| Type | Definition | Shape |
+|---|---|---|
+| `DeploymentId` | `Commons/Types/DeploymentId.cs:3` — `readonly record struct DeploymentId(Guid Value)` | `ToString()` → `Value.ToString("N")`, 32 hex chars, **no hyphens** (`:10`) |
+| `RevisionHash` | `Commons/Types/RevisionHash.cs:8` — `readonly record struct RevisionHash(string Value)` | SHA-256 hex, **lowercase 64 chars**, no `0x`; empty invalid (`:3-6`) |
+
+Both are stable, filesystem/SQL-safe strings. Use `DeploymentId.ToString()` (the "N" form) as the
+`deployment_id` TEXT column value.
+
+---
+
+## 2. Post-apply point
+
+### 2.1 `ApplyAndAck` control flow — `DriverHostActor.cs:1413-1458`
+
+```
+1415  _applyingDeploymentId = deploymentId;
+1416  Become(Applying);
+1425  UpsertNodeDeploymentState(..., Applying, null)
+1427  try {
+1429      ReconcileDrivers(deploymentId);            // blob fetch #1 — SWALLOWS DB errors
+1430      _currentRevision = revision;
+1431      UpsertNodeDeploymentState(..., Applied, null);
+1432      SendAck(..., ApplyAckOutcome.Applied, null, correlation);
+1436      _opcUaPublishActor?.Tell(RebuildAddressSpace(...));
+1439      PushDesiredSubscriptions(deploymentId);    // blob fetch #2 — SWALLOWS DB errors
+1440      OtOpcUaTelemetry.DeploymentApplied.Add(1, "outcome"="ack");
+1441      _log.Info("applied deployment ...");
+1443  }
+1444  catch (Exception ex) { ...Failed ACK... }
+1452  finally { _applyingDeploymentId = null; Become(Steady); }
+```
+
+### 2.2 Where the cache write belongs
+
+**Immediately after `DriverHostActor.cs:1439`**, before `:1440` — but **wrapped in its own
+try/catch**. Reason: `:1432` already sent an `Applied` ACK to the coordinator. An unguarded cache
+write that throws would fall into the `catch` at `:1444` and send a *second*, contradictory
+`Failed` ACK for a deployment the fleet already believes is applied.
+
+### 2.3 Paths that must NOT write to the cache
+
+1. **`catch` at `:1444`** — any apply failure.
+2. **Idempotent short-circuit at `:1402-1409`** — `HandleDispatchFromSteady` returns early with an
+   `Applied` ACK when `_currentRevision == msg.RevisionHash`. `ApplyAndAck` is never entered and no
+   blob is read.
+3. **⚠ THE SILENT-DEGRADATION TRAP.** `ReconcileDrivers:1478-1483` and
+   `PushDesiredSubscriptions:1549-1553` catch DB failures, log a **warning**, and `return` **without
+   rethrowing**. `ApplyAndAck` therefore reaches `:1440` and declares success having applied **zero
+   drivers** when the blob load failed. Caching "whatever we read" would **persist an empty artifact
+   as a successful apply** — poisoning the boot-from-cache path with a config that starts no drivers.
+
+   **Mitigation (load-bearing):** the cache write must be gated on the blob having actually loaded
+   non-empty. Have `ReconcileDrivers` surface the loaded blob upward (a field or an out-param)
+   rather than re-reading it a third time, and skip the cache write when it is
+   `Array.Empty<byte>()`. This also removes the duplicate read noted in §1.1.
+
+4. **`RestoreApplied` (`:1514-1528`)** — the bootstrap replay path. Restoring from central; caching
+   here is harmless and arguably desirable, but it is a distinct seam. **Phase 1 scope: skip it.**
+5. **`TryRecoverFromStale` (`:1996-2022`)** — never reads the artifact at all (see §3.4).
+
+---
+
+## 3. Cold-boot path
+
+### 3.1 PreStart — `DriverHostActor.cs:415-436`
+
+Ctor already did `Become(Steady)` at `:411`. PreStart subscribes to topics, spawns the virtual-tag
+and scripted-alarm hosts, then calls `Bootstrap()` at `:435`.
+
+### 3.2 `Bootstrap()` — `DriverHostActor.cs:506-565` — **yes, it queries central SQL at boot**
+
+Two queries inside one `try`: latest `NodeDeploymentState` for self (`:514-518`), then the matching
+`Deployment` row (`:527-532`). Four outcomes: no prior deployments → Steady; `Applied` →
+`RestoreApplied` (`:544`); `Applying` orphan → `ApplyAndAck` replay (`:549`); `Failed` → Steady.
+
+### 3.3 **THE SEAM** — `DriverHostActor.cs:560-564`
+
+```csharp
+catch (Exception ex)
+{
+    _log.Warning(ex, "DriverHost {Node}: ConfigDb unreachable on bootstrap; entering Stale", _localNode);
+    Become(Stale);
+}
+```
+
+**This is the exact point where "central fetch failed at boot" is known.** The cache fallback goes
+between `:562` and `:563`: attempt a local-cache load, and only `Become(Stale)` if the cache is also
+a miss.
+
+⚠ The catch also fires if the *second* query throws after the first succeeded, so `latest` is not in
+scope-valid state from the catch. **The cache must be self-describing** — it must carry its own
+`DeploymentId` + `RevisionHash`, because at this seam the actor has no idea what it should be
+restoring. The planned `deployment_pointer` shape already satisfies this.
+
+### 3.4 The `Stale` state — `DriverHostActor.cs:1343-1379`
+
+`ReconnectInterval = 30s` (`:55`), retry timer started at `:1378`.
+
+The dispatch-ignore at **`:1345-1348`** discards `DispatchDeployment` entirely — no ACK, no
+buffering (contrast `Applying:599` which does `Self.Forward(msg)`). A Stale node looks like a
+timeout to the coordinator. Phase 1 does **not** change this (per design §3.3: a *new* deployment
+still requires central).
+
+`RouteNodeWrite` fast-fails at `:1359-1360` with `"driver host stale (config DB unreachable)"`.
+
+### 3.5 Pre-existing gap worth recording (not Phase 1 scope)
+
+`TryRecoverFromStale` (`:1996-2022`) sets `_currentRevision` and marks `Applied` **without calling
+`ReconcileDrivers` or `PushDesiredSubscriptions`** — documented at `:1692-1695` and `:1706`. A
+Stale-recovered node reports Applied while running **zero drivers**, and
+`HandleDispatchFromSteady:1402` then short-circuits on revision match. Boot-from-cache largely
+neutralises this in practice, but the bookkeeping remains optimistic. **Logged as a follow-up, not
+fixed here.**
+
+### 3.6 `DbHealthProbeActor`
+
+**No interaction with `DriverHostActor` whatsoever.** Spawned as a sibling
+(`ServiceCollectionExtensions.cs:243-246`), consumed only by `OpcUaPublishActor`
+(`OpcUaPublishActor.cs:105`, `:256`, `:528`, `:543-544`). `DriverHostActor` runs its own independent
+30s `retry-db` timer. If running-from-cache should later influence OPC UA ServiceLevel,
+`OpcUaPublishActor`'s DB-health seam is the precedent to mirror.
+
+---
+
+## 4. Cluster identity — **the one real design problem (see D-1)**
+
+**There is no `ClusterId` field on `DriverHostActor` and no accessor for it.** The actor knows only
+`_localNode` (`DriverHostActor.cs:61`, type `Commons.Types.NodeId`, aliased at `:28`).
+
+`ClusterId` is resolved **per-artifact**, by `DeploymentArtifact.ResolveClusterScope(blob, nodeId)`
+(`DeploymentArtifact.cs:485-516`) — it scans the artifact's `Nodes[]` array for a matching `NodeId`
+and reads that element's `ClusterId`. Types: `ClusterScope` (`:46`), `ClusterFilterMode` (`:28`).
+
+`IClusterRoleInfo` (`Commons/Interfaces/IClusterRoleInfo.cs:12`) exposes `LocalNode`, `LocalRoles`,
+`HasRole`, `MembersWithRole`, `RoleLeader`, `RoleLeaderChanged` — **no `ClusterId`**.
+`docker-dev/docker-compose.yml` has `Cluster__Hostname`/`Port`/`Roles`/`SeedNodes` — **no
+`Cluster__ClusterId`**. The `ClusterNode` entity
+(`Configuration/Entities/ClusterNode.cs:7,10`) carries both, but lives in central SQL — exactly
+what is unreachable at the seam that needs it.
+
+> **The bind:** the design keys the cache by `ClusterId` so a replicated pair shares one cache
+> entry. But `ClusterId` is only discoverable *from a blob you already have* or *from central SQL*.
+> At the §3.3 boot-failure seam we have neither.
+
+**Resolution adopted (D-1):** keep `cluster_id` as the `deployment_pointer` PK — pair sharing is the
+whole point of replication and must not be given up. Resolve it at each end as follows:
+
+- **Write path** (`ApplyAndAck`, §2.2): call `DeploymentArtifact.ResolveClusterScope(blob, nodeId)`
+  on the blob just applied. `ClusterFilterMode.ScopeTo` → use its `ClusterId`. `None` (single-cluster
+  artifact) or `Suppress` → fall back to the literal `"__single"` sentinel, matching the actor's own
+  existing degenerate-case convention at `:1827-1830`.
+- **Read path** (boot seam, §3.3): we cannot compute the key, so **do not key the read**. Select the
+  single `deployment_pointer` row, ordered by `applied_at_utc DESC`, `LIMIT 1`. A node belongs to
+  exactly one cluster and its pair peer replicates the same cluster's row, so the table holds one
+  row in every real topology. If more than one row is present (a node was re-homed between
+  clusters), take the newest and **log a warning naming both cluster ids** — an operator-visible
+  signal rather than a silent wrong-config boot.
+- Optional escape hatch: honour a `Cluster:ClusterId` config key when present. **Deferred** — not
+  needed for the rig, and adding a required key contradicts the design's "storage ships
+  unconditionally" posture.
+
+This preserves the replication payoff (design §3.3) and the schema exactly as specified, and
+confines the change to the read query's `WHERE` clause.
+
+---
+
+## 5. `DriverHostActor` construction — the `Props` expression tree
+
+### 5.1 The factory — `DriverHostActor.cs:326-346`
+
+`static Props(...)` takes **16 parameters**, 14 optional, and forwards them **fully positionally**
+into `Akka.Actor.Props.Create(() => new DriverHostActor(...))` at **`:343-346`**. Ctor at
+`:375-412`, assignments `:393-408`.
+
+> ⚠ Six `IActorRef?` parameters and three interface-typed parameters mean many mis-bindings are
+> **type-compatible and therefore compile clean**. Callers use *named* arguments (safe); the
+> internal forwarding at `:343-346` is *positional* (unsafe).
+>
+> **Rule for Task 7: append the new parameter LAST in the `Props` signature, LAST in the ctor
+> signature, and LAST in the positional forwarding list at `:346`. Change nothing else.**
+
+### 5.2 Construction sites — **27 total** (1 production + 26 test)
+
+**Production (1):** `Runtime/ServiceCollectionExtensions.cs:332`, inside `WithOtOpcUaRuntimeActors`
+(`:193`); registered as `DriverHostActorKey` (`:344`). All args named except the first three. Does
+not pass `virtualTagHostOverride`, `scriptedAlarmHostOverride`, or `driverMemberCountProvider`.
+
+`WithOtOpcUaRuntimeActors` callers: `Host/Program.cs:331`,
+`Host.IntegrationTests/TwoNodeClusterHarness.cs:398`, `ServiceCollectionExtensionsTests.cs:42,144`.
+
+**Tests (26)**, all in `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/`:
+
+| File | Lines |
+|---|---|
+| `Drivers/DriverHostActorTests.cs` | 28, 60, 81, 118, 143, 189 |
+| `Drivers/DriverHostActorReconcileTests.cs` | 37, 59, 90, 119, 156, 193 |
+| `Drivers/DriverHostActorDiscoveryTests.cs` | 385, 671, 748, 1023 |
+| `Drivers/DriverHostActorWriteRoutingTests.cs` | 148, 208 |
+| `Drivers/DriverHostActorPrimaryGateTests.cs` | 228, 247 |
+| `Drivers/DriverHostActorVirtualTagTests.cs` | 44, 76 |
+| `Drivers/DriverHostActorNativeAlarmTests.cs` | 350 |
+| `Drivers/DriverHostActorNativeAlarmAckRoutingTests.cs` | 126 |
+| `Drivers/DriverHostActorProbeResultDropTests.cs` | 66 |
+| `Drivers/DriverHostActorHistoryWriterTests.cs` | 59 |
+| `Drivers/DiscoveryInjectionEndToEndTests.cs` | 219 |
+| `VirtualTags/DependencyMuxActorTests.cs` | 139 |
+
+No site calls `Props.Create<DriverHostActor>()` or `new DriverHostActor(...)` directly — the only
+`new DriverHostActor` in the tree is inside the factory at `:343`. Because the new parameter is
+optional and appended last, **no test site should need editing.**
+
+### 5.3 Status object for the running-from-cache signal
+
+`NodeDiagnosticsSnapshot` (`Commons/Interfaces/NodeDiagnosticsSnapshot.cs:17-22`) — built in
+`HandleGetDiagnostics` (`DriverHostActor.cs:1381-1398`), a positional record of
+`(NodeId, RevisionHash? CurrentRevision, IReadOnlyList<DriverInstanceDiagnostics> Drivers, DateTime AsOfUtc)`.
+`GetDiagnostics` is handled in **all three states** (`Steady:570`, `Applying:601`, `Stale:1349`), so
+a flag here is observable regardless of state. **This is the natural home for `RunningFromCache`**
+(append last to the record). Consumer: `IFleetDiagnosticsClient` → AdminUI.
+
+`IDriverHealthPublisher` (`:73`, defaulted `:402`) is **not** a fit — the host never calls it; it
+only passes it down to `DriverInstanceActor` children (`:1845`, `:1859`). Per-driver, not per-node.
+
+### 5.4 Async / side-effect IO convention
+
+The actor is **overwhelmingly synchronous-blocking on DB IO** — all five `CreateDbContext()` sites
+use `using var db = ...` with synchronous LINQ on the actor thread. No `async`/`await` in the file,
+no `Task.Run`.
+
+The single async pattern is `ContinueWith(..., TaskScheduler.Default).PipeTo(replyTo)` in
+`HandleRouteNodeWrite` (`:1188-1200`), with an explicit `Sender` capture before the continuation.
+
+**Recommendation for Task 7:** a **synchronous** cache write at the §2.2 point is *consistent with
+the file's existing style* and is correct there — the actor is mid-apply, holds `Applying`, and
+every other DB call on that path already blocks. A fire-and-forget `PipeTo(Self)` would need a
+handler registered in `Steady`, `Applying` **and** `Stale`, or it dead-letters after the
+`Become(Steady)` at `:1456`. **Prefer synchronous + try/catch.** (This deviates from the plan's
+"fire-and-forget"; see D-4.)
+
+`IWithTimers` is already implemented (`:50`, `:282`); `"retry-db"` key in use at `:1378`/`:2009`.
+
+---
+
+## 6. Host composition root
+
+### 6.1 Role flag — `Program.cs:47-50`
+
+```csharp
+var roles = RoleParser.Parse(Environment.GetEnvironmentVariable("OTOPCUA_ROLES"));
+var hasAdmin = roles.Contains("admin");
+var hasDriver = roles.Contains("driver");
+```
+Allowed roles: `admin`, `driver`, `dev` (`Cluster/RoleParser.cs:5-8`); unknown throws (`:25-27`).
+
+### 6.2 `hasDriver` branch — `Program.cs:131-310`
+
+Anchor for the new call: **`Program.cs:173-175`** `builder.Services.AddAlarmHistorian(...)`, inside
+the config-gated cluster at `:154-195`. Must land **before** `AddAkka` at `:314`.
+
+### 6.3 Pipeline — `Program.cs:368-404`
+
+`:383` `MapStaticAssets` · `:385-396` `if (hasAdmin) { ... }` · **`:398` `MapOtOpcUaHealth()`** ·
+`:399` `MapOtOpcUaMetrics()` · `:404` `RunAsync()`.
+
+⚠ **There is no `hasDriver` block in the pipeline half** — only `hasAdmin` is branched on. A
+driver-gated `app.Map*` is a new construct; it goes between `:396` and `:398`.
+
+`Program.cs:406-410` re-exports `public partial class Program`.
+
+### 6.4 Kestrel / URLs — **CONFIRMED: no Kestrel configuration exists**
+
+- `Program.cs:57` `builder.WebHost.UseStaticWebAssets();` — the **only** `builder.WebHost` call.
+- **Zero** `ConfigureKestrel` / `UseUrls` / `ListenAnyIP` / `UseKestrel` anywhere in `src/`.
+- Binding is exclusively via `ASPNETCORE_URLS`: `docker-compose.yml:173` (central-1) and `:240`
+  (central-2) set `http://+:9000`; `launchSettings.json:7` sets `http://localhost:9000`.
+
+Two traps for Task 5:
+
+1. ⚠ **`scripts/install/Install-Services.ps1:186`** (`$hostEnv += "ASPNETCORE_URLS=..."`) is inside
+   `if ($hasAdmin)` at `:185`. **A driver-only Windows-service node gets no `ASPNETCORE_URLS` at
+   all** and binds the ASP.NET default. The plan's parse-URLs-then-rebind snippet would re-bind
+   `http://+:9000` there — a port that node never listened on. The `?? "http://+:9000"` fallback in
+   the plan's snippet is therefore **wrong for driver-only nodes**; see D-6.
+2. ⚠ **`Host.IntegrationTests/TwoNodeClusterHarness.cs:331`** already calls
+   `builder.WebHost.UseKestrel(o => o.Listen(IPAddress.Parse(LoopbackHost), 0));`. Adding a
+   production `ConfigureKestrel` gives two competing explicit configurations in integration tests —
+   last-one-wins. The `syncPort > 0` gate keeps this dormant by default (harness sets no sync port),
+   but any harness that *does* set one will fight the port-0 binding.
+
+**Health route for smoke tests:** `/healthz` — liveness, runs no checks, always 200 while the
+process is up, `AllowAnonymous`. (`Health/HealthEndpoints.cs:48` → `MapZbHealth()`; routes from the
+`ZB.MOM.WW.Health` package.) Also `/health/ready`, `/health/active`, and `/metrics` (`Program.cs:399`).
+
+### 6.5 `SecretsRegistration` template — `Host/Configuration/SecretsRegistration.cs`
+
+Shape for `LocalDbRegistration` to mirror: `public static class` in
+`ZB.MOM.WW.OtOpcUa.Host.Configuration`; `public const string` section/key path constants; a
+`public static bool IsXEnabled(IConfiguration)` predicate using
+`GetValue(key, defaultValue: false)` (**default-deny**) that `Program.cs` can also call;
+`AddOtOpcUaX(this IServiceCollection, IConfiguration)` returning `IServiceCollection`, both args
+`ArgumentNullException.ThrowIfNull`-guarded; XML docs with `<remarks><para>` explaining *why* the
+gate is default-deny.
+
+Call-site precedent: `Program.cs:363` `AddOtOpcUaSecrets(...)`; predicate reused inside the `AddAkka`
+lambda at `Program.cs:323`.
+
+### 6.6 Telemetry meter allowlist — **EXISTS. Task 10 step 3 is REQUIRED, not conditional.**
+
+`Host/Observability/ObservabilityExtensions.cs:25-38`:
+```csharp
+return services.AddZbTelemetry(o =>
+{
+    o.ServiceName = "otopcua";
+    o.Meters = [OtOpcUaTelemetry.MeterName];          // <-- ObservabilityExtensions.cs:30
+    o.ActivitySources = [OtOpcUaTelemetry.ActivitySourceName];
+```
+
+`o.Meters` is a **strict allowlist** — `ZbTelemetryExtensions.cs:51-54` iterates it calling
+`metrics.AddMeter(name)`; **no wildcard support**. Anything unlisted is silently not exported.
+Single element today: `OtOpcUaTelemetry.MeterName` = `"ZB.MOM.WW.OtOpcUa"`
+(`Commons/Observability/OtOpcUaTelemetry.cs:19`).
+
+**Action:** add `LocalDbMetrics.MeterName` at `ObservabilityExtensions.cs:30`. This is exactly the
+omission the ScadaBridge live gate caught.
+
+### 6.7 Health registration — `Host/Health/HealthEndpoints.cs:19-41`
+
+`AddOtOpcUaHealth()` **takes no parameters** and is called **unconditionally** at `Program.cs:365`,
+outside both role blocks. Three `.AddTypeActivatedCheck<>` calls (`configdb`, `akka`,
+`admin-leader`); **no registration-time role gating exists** — `ActiveNodeHealthCheck` does runtime
+role scoping instead (returns `Healthy` when the node lacks the role).
+
+⚠ **There are zero `IHealthCheck` implementations in this repo** — all three come from the
+`ZB.MOM.WW.Health.*` packages (`Directory.Packages.props:124-126`, v0.1.0). A LocalDb check would be
+the **first health-check class in the tree**; `Host/Health/` holds only `HealthEndpoints.cs` today.
+
+**Consequence for Task 10:** driver-gating the new check requires changing
+`AddOtOpcUaHealth()`'s signature (add `bool hasDriver` or `IConfiguration`) and its `Program.cs:365`
+call site. **Preferred alternative:** follow the `ActiveNodeHealthCheck` precedent — register
+unconditionally and have the check return `Healthy` when LocalDb is not registered. That keeps the
+signature and matches the "default-OFF must not degrade a plain node" requirement for free. See D-7.
+
+### 6.8 Config files
+
+| File | Top-level sections |
+|---|---|
+| `appsettings.json` | `Serilog`, `Security`, `Secrets`, `ServerHistorian`, `ContinuousHistorization`, `AlarmHistorian`, `Deployment` |
+| `appsettings.driver.json` / `.admin.json` / `.admin-driver.json` / `.Development.json` | `Serilog`, `Security` only |
+
+Overlay selection: `Program.cs:67` sorts roles ordinally and joins with `-` → `admin,driver`
+becomes `appsettings.admin-driver.json`, added `optional: true` at `:70`; env vars (`:77`) and CLI
+args (`:78`) are re-appended after, so deployment overrides outrank the overlay.
+
+**`LocalDb` key search: CONFIRMED ABSENT.** Case-insensitive `localdb` across all `*.json`,
+`*.yml`, `*.ps1`, `*.csproj`, `*.props` matches only the two Phase-1/Phase-2 planning artifacts. No
+conflicting keys in any overlay or in `Install-Services.ps1`.
+
+**Placement convention:** each feature section opens with a `"_comment"` string then `"Enabled":
+false`; per-field caveats use sibling `"_<Field>Comment"` keys (`appsettings.json:34`, `:45`). Place
+`LocalDb` between `AlarmHistorian` (ends `:61`) and `Deployment` (`:62`).
+
+---
+
+## 7. LiteDB LocalCache — dormant, safe to delete
+
+### 7.1 Files (6) under `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/`
+
+`ILocalConfigCache.cs`, `GenerationSnapshot.cs`, `StaleConfigFlag.cs`, `LiteDbConfigCache.cs`
+(also declares `LocalConfigCacheCorruptException` at `:128`), `GenerationSealedCache.cs`,
+`ResilientConfigReader.cs`. Single namespace `...Configuration.LocalCache`. The only two
+`using LiteDB;` statements in `src/` are `LiteDbConfigCache.cs:1` and `GenerationSealedCache.cs:1`.
+
+### 7.2 Reference classification — **no live-code references. NO STOP.**
+
+- **(c) live code elsewhere: ZERO.** Also confirms full dormancy: `SealedBootstrap` (referenced by
+  `docs/v2/v2-release-readiness.md:46`) has **zero hits** in `src/` and `tests/` — it no longer exists.
+- Two benign near-misses:
+  - `Configuration/Services/ILdapGroupRoleMappingService.cs:14` — the string `ResilientConfigReader`
+    inside an **XML doc comment**. Prose only. Needs a comment edit or the doc goes stale.
+  - `Configuration/ZB.MOM.WW.OtOpcUa.Configuration.csproj:29` — `<PackageReference Include="LiteDB"/>`,
+    the only one in the repo.
+- Dozens of `LocalConfigCacheCorruptException` hits under `src/**/bin/**/*.xml` are **generated doc
+  build artifacts** (`GenerateDocumentationFile=true`), not source.
+
+### 7.3 ⚠ Test-file drift — the plan's delete list is incomplete
+
+Plan `…phase1.md:484-485` names only `GenerationSealedCacheTests.cs` and
+`ResilientConfigReaderTests.cs`. Actual test files in
+`tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/`:
+
+| File | Status |
+|---|---|
+| `GenerationSealedCacheTests.cs` | in plan ✅ |
+| `ResilientConfigReaderTests.cs` | in plan ✅ — note `StaleConfigFlagTests` is co-located here at `:323` |
+| **`LiteDbConfigCacheTests.cs`** | **MISSING from the plan** — will fail to compile if the source is deleted (`:125` calls `new LiteDB.LiteDatabase(...)` directly) |
+
+See D-3.
+
+### 7.3b Outcome (Task 9, executed 2026-07-20)
+
+Deleted: the 6 `LocalCache/` sources and **all three** test files (including
+`LiteDbConfigCacheTests.cs`, which the plan omitted — D-3). `LiteDB` removed from both
+`Configuration.csproj` and `Directory.Packages.props`. The stale XML-doc reference at
+`ILdapGroupRoleMappingService.cs:14` was rewritten to state the present truth (no fallback exists;
+reviving it means an admin-side cache on LocalDb, not restoring the old pipeline) rather than
+deleted, per the DoD's "explanatory prose may remain". Configuration tests: 92/92 green.
+
+**Follow-up found while executing:** `Polly.Core` in `Configuration.csproj` is now orphaned too —
+`ResilientConfigReader` was its only real consumer in that project (remaining `Polly` mentions are
+comments). Left in place deliberately: removing it is outside Task 9's scope and could break a
+consumer relying on the transitive flow. Worth a separate cleanup.
+
+### 7.4 Package removal
+
+- `PackageReference`: `Configuration.csproj:29` — only consumer.
+- `PackageVersion`: `Directory.Packages.props:35` — `LiteDB` `5.0.21`.
+
+Both safe to delete; no transitive consumer.
+
+---
+
+## 8. Test projects, rig, packages
+
+### 8.1 Test project layout
+
+| Question | Answer |
+|---|---|
+| `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/` | **DOES NOT EXIST** — there is no Host *unit*-test project at all |
+| Closest Host project | `Host.IntegrationTests` — despite the name it carries plenty of pure unit tests (`LdapOptionsValidatorTests`, `ServerHistorianOptionsValidatorTests`, `ResilienceInvokerFactoryRegistrationTests`, …) |
+| `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/` | **EXISTS** |
+| DriverHostActor tests | `Runtime.Tests/Drivers/` — 11 `DriverHostActor*Tests.cs` files. **xunit v2** (`xunit` 2.9.3), references **`Akka.TestKit.Xunit2`** ✅ |
+| `Host.IntegrationTests` xunit version | **xunit.v3** 1.1.0; SDK `Microsoft.NET.Sdk.Web` |
+
+`Directory.Packages.props` documents that AdminUI.Tests / ControlPlane.Tests / **Runtime.Tests** are
+the three xunit-v2 holdouts, held there by `Akka.TestKit.Xunit2` (verified 2026-07-13: no xunit.v3
+Akka TestKit exists). This matches the design's "actor tests live in an xunit2 TestKit project".
+
+### 8.2 ⚠ `WebApplicationFactory<Program>` is **deliberately not used** — Task 11 must be rewritten
+
+The only mention of the type in the entire `tests/` tree is a comment explaining why it is avoided:
+
+`Host.IntegrationTests/TwoNodeClusterHarness.cs:48`
+```
+/// <para>Why not <c>WebApplicationFactory&lt;Program&gt;</c>? Program.cs reads <c>OTOPCUA_ROLES</c> ...
+```
+
+The actual pattern builds the host directly — `TwoNodeClusterHarness.cs:329`:
+```csharp
+var builder = WebApplication.CreateBuilder(new WebApplicationOptions { Args = [] });
+```
+`Microsoft.AspNetCore.Mvc.Testing` is still referenced (csproj `:15`) but only for transitive
+test-host plumbing. **Copy `TwoNodeClusterHarness`, not `WebApplicationFactory`.** See D-5.
+
+`Host.IntegrationTests` also has its own `docker-compose.yml` (sql `14331`, ldap `3894`) gated by
+`DockerFixtureAvailability.cs`.
+
+### 8.3 docker-dev rig topology — plan's site-a/site-b assumption is **correct**
+
+Single Akka mesh (`otopcua`), seeded by `central-1`. Tenant separation is by `ServerCluster.ClusterId`
+rows (MAIN / SITE-A / SITE-B) inside **one shared ConfigDb** — not separate meshes.
+
+| Service | Roles | Cluster |
+|---|---|---|
+| `central-1`, `central-2` | `admin,driver` | MAIN |
+| `site-a-1`, `site-a-2` | `driver` | SITE-A |
+| `site-b-1`, `site-b-2` | `driver` | SITE-B |
+| `sql`, `migrator`, `cluster-seed`, `traefik` | — | infra |
+
+**There is no admin-only service** — the "admin" nodes are fused `admin,driver`. All six host nodes
+share the `&otopcua-host` YAML anchor.
+
+⚠ **NO HOST NODE HAS A WRITABLE VOLUME.** The only `volumes:` in the file are `sql` →
+`otopcua-mssql-data`, plus two read-only binds (`./seed:ro`, `./traefik-dynamic.yml:ro`). All six
+nodes write to the container's ephemeral layer, destroyed on recreate. This is already an accepted
+pattern (per-container `Secrets:SqlitePath=otopcua-secrets.db`, replicated over Akka rather than
+persisted). **Task 13 must add six named volumes** — there is nothing to piggyback on. See D-8.
+
+**Env style: confirmed double-underscore `Section__Key`**, with array indices and dictionary keys as
+a third segment (`Cluster__SeedNodes__0`, `Security__Ldap__GroupToRole__ReadOnly`). Exceptions that
+are plain SCREAMING_SNAKE (read directly, not config-bound): `OTOPCUA_ROLES`,
+`ZB_SECRETS_MASTER_KEY`, `GALAXY_MXGW_API_KEY`, `OTOPCUA_CONFIG_CONNECTION`, `ASPNETCORE_URLS`.
+
+**Ports.** Host-published: `14330`→sql, `4840`-`4845`→the six nodes' OPC UA, `9200`→traefik web,
+`8089`→traefik dashboard. Container-internal on every node: `4053` (Akka remoting), `9000`
+(Kestrel), `4840` (OPC UA). Also reserved on this machine: `14331`/`3894` (Host.IntegrationTests
+compose), `80`/`8080` (sibling scadabridge/scadalink stack), `10.100.0.35:3893` (shared GLAuth).
+
+**`9001` (the plan's choice) is free** container-internal and need not be published. ✅
+
+### 8.4 Package versions — `Directory.Packages.props`
+
+| Package | Version | Line |
+|---|---|---|
+| `Grpc.Core.Api` | 2.76.0 | 32 |
+| `Grpc.Net.Client` | 2.76.0 | 33 |
+| **`Grpc.AspNetCore`** | **NOT PRESENT** | — |
+| `Google.Protobuf` | 3.34.1 | 31 |
+| `Microsoft.Data.Sqlite` | 10.0.7 | ~55 |
+| `SQLitePCLRaw.bundle_e_sqlite3` | 2.1.12 | ~108 |
+| `LiteDB` | 5.0.21 | 35 |
+
+Both Grpc floors and the Protobuf floor are already satisfied — **no bumps needed**, only the new
+`Grpc.AspNetCore` entry (use 2.76.0 to match the train; watch for NU1605/NU1608 if it wants a
+Protobuf newer than the pinned 3.34.1).
+
+⚠ **`SQLitePCLRaw` is a surgical direct pin, not a transitive one.**
+`CentralPackageTransitivePinningEnabled` is deliberately **off** (it breaks the Roslyn version
+split). The 2.1.12 pin is promoted by a **direct `PackageReference` in `Core.AlarmHistorian`**
+(`Core.AlarmHistorian.csproj:15` + `:21`) — today the only Sqlite consumer in `src/`.
+
+> **Any new project that pulls `Microsoft.Data.Sqlite` MUST also take a direct
+> `<PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3"/>`,** or it silently resolves the
+> vulnerable 2.1.11 native bundle. This applies to **Host** and **Runtime** (both gain `ZB.MOM.WW.LocalDb`,
+> which depends on `Microsoft.Data.Sqlite`) and to every test project that builds a real `ILocalDb`.
+> Copy the `Core.AlarmHistorian.csproj:15,21` pair exactly. The plan's Task 1 mentions
+> `SQLitePCLRaw.lib.e_sqlite3` "if the audit flags it" — the correct package is
+> **`bundle_e_sqlite3`**, and it is **required, not conditional**.
+
+---
+
+## 9. Deviations from the plan
+
+Recorded per the tasks.json `deviation` convention. **D-1, D-3, D-5, D-6 change the work
+materially.**
+
+| # | Plan says | Reality | Resolution |
+|---|---|---|---|
+| **D-1** | Cache keyed by `ClusterId`; "cluster identity" is a recon item with a STOP condition | `ClusterId` is stable, but **not resolvable at the boot seam** — it lives inside the artifact or in the unreachable central DB (§4) | Keep `cluster_id` as PK (pair sharing is the point). **Write:** resolve via `DeploymentArtifact.ResolveClusterScope`, `"__single"` sentinel for the degenerate case. **Read:** unkeyed `ORDER BY applied_at_utc DESC LIMIT 1`, warn if >1 row. No STOP. |
+| **D-2** | Cache write "fire-and-forget (`PipeTo`-style or guarded `Task.Run`)" | The actor has **no async DB IO at all**; a `PipeTo(Self)` needs handlers in all three states or it dead-letters after `Become(Steady)` (§5.4) | Synchronous write in its own try/catch at `:1439`. Matches the file's style; failure still cannot fail the apply. |
+| **D-3** | Task 9 deletes 2 test files | **3** test files reference the subsystem — `LiteDbConfigCacheTests.cs` is missing from the list (§7.3) | Delete all three. Also fix the stale XML-doc mention at `ILdapGroupRoleMappingService.cs:14`. |
+| **D-4** | Cache the artifact after a successful apply | `ReconcileDrivers`/`PushDesiredSubscriptions` **swallow** DB errors, so a "successful" apply can have loaded an **empty blob** and started zero drivers (§2.3) | Gate the write on a non-empty blob; surface the blob from `ReconcileDrivers` instead of a third read. **Load-bearing — without it the cache can be poisoned with an empty config.** |
+| **D-5** | Task 11 uses `WebApplicationFactory<Program>` | Deliberately **not used** in this repo; `TwoNodeClusterHarness.cs:48` documents why (Program.cs reads `OTOPCUA_ROLES` from the environment) (§8.2) | Rewrite Task 11 against the `TwoNodeClusterHarness` direct-host-build pattern. |
+| **D-6** | Task 5 re-binds `ASPNETCORE_URLS ?? "http://+:9000"` | `Install-Services.ps1:185-186` sets `ASPNETCORE_URLS` **only when `hasAdmin`** — the fallback would bind a driver-only node to a port it never served (§6.4) | Re-bind only when a URL is actually configured; when absent, add **only** the sync listener. Also note `TwoNodeClusterHarness.cs:331` already calls `UseKestrel(Listen(…, 0))`. |
+| **D-7** | Task 10 driver-gates the health check | `AddOtOpcUaHealth()` takes no args and is called unconditionally at `Program.cs:365`; no registration-time gating exists anywhere (§6.7) | Register unconditionally; return `Healthy` when LocalDb is absent — the `ActiveNodeHealthCheck` precedent. Avoids a signature change and satisfies "default-OFF must not degrade a plain node". |
+| **D-8** | Task 13 "existing data mount" | **No host node has any writable volume** (§8.3) | Add six named volumes. Note: `9001` is free container-internal. Meter allowlist (§6.6) is **required**, not conditional. |
+
+### Follow-ups logged, not fixed here
+
+1. **`TryRecoverFromStale` marks `Applied` without reconciling drivers** (§3.5) — pre-existing,
+   documented in-code at `:1692-1695`. Out of Phase 1 scope.
+2. **Duplicate blob read per apply** (`:1472` and `:1543`, §1.1) — D-4's fix removes one of them as a
+   side effect.
+3. **`Install-Services.ps1` gives driver-only nodes no `ASPNETCORE_URLS`** (§6.4) — pre-existing;
+   D-6 works around it rather than fixing it.

docs/plans/2026-07-20-localdb-phase2-live-gate.md

@@ -0,0 +1,180 @@
+# LocalDb Phase 2 — live gate (docker-dev rig)
+
+> Task 8 of `2026-07-20-localdb-adoption-phase2.md`. Evidence log.
+> Rig: local `docker-dev/docker-compose.yml` — 2 central + site-a pair (replication ON) + site-b pair
+> (default-OFF), one Akka cluster, one shared SQL.
+>
+> **Safety rules followed** (same as the Phase 1 gate): all DB inspection via `docker cp` of the
+> `db`/`-wal`/`-shm` triplet out of the container, querying the copy — never host `sqlite3` on the
+> live WAL file.
+
+## Headline
+
+**The gate did not get past its own first boot before finding real defects.** Four production bugs,
+all fixed on-branch with regression tests, plus a fifth finding in the test suite itself. Three of
+the four crash-looped every driver node; the fourth silently stopped alarm history everywhere.
+
+Checks 1, 2, 5 and 6 pass. **Checks 3 and 4 are NOT satisfied** — see "The blocker" below; they
+cannot be run as written on a rig that has no per-pair redundancy roles, and the defect they were
+meant to confirm turned out to be the opposite of what the plan assumed.
+
+## Seeding the migration
+
+The rig had no pre-consolidation `alarm-historian.db`, so one was synthesised per node and `docker
+cp`-ed to `/app/alarm-historian.db` (the WORKDIR-relative default) into created-but-not-started
+containers — the exact upgrade shape.
+
+Deliberately overlapping content, to test D-6 in production: **6 rows on site-a-1, 5 on site-a-2,
+2 of them byte-identical payloads** (the warm-pair duplication `HistorianAdapterActor` produces in
+every boot window). 11 legacy rows, **9 distinct**.
+
+## Defects found and fixed
+
+### 1. Empty `ServerHistorian:ApiKey` crash-loops the host (was classified "degrades")
+
+`AlarmHistorian:Enabled=true` requires the gateway alarm writer, which requires a key.
+`HistorianGatewayClientOptions.Validate()` throws `The gateway API key must not be empty` inside the
+client factory during Akka startup, so every driver node crash-looped.
+
+`ServerHistorianOptionsValidator` exists precisely to convert that class of failure into a named
+`OptionsValidationException` — but its documented fail tier explicitly excluded `ApiKey`, reasoning
+that a keyless client "degrades rather than crashes (the gateway rejects calls)". **That premise is
+false**: the client validates its own options at construction, so the process dies before any call.
+Fixed; the key is never echoed in the message.
+
+### 2. `UseTls` / endpoint-scheme mismatch crash-loops the host
+
+Immediately after fixing #1, the rig crash-looped again: `UseTls` defaults to `true`, the rig
+endpoint is `http://`, and the client throws `UseTls requires an https gateway endpoint`. The
+inverse (`https://` + `UseTls=false`) throws too — both strings confirmed in the shipped client
+assembly. Setting an `http` endpoint without clearing `UseTls` is an ordinary migration slip that
+takes the host down. Now validated in both directions.
+
+### 3. Plaintext h2c was *unreachable* — every `http://` deployment crashed
+
+Third crash-loop, and a genuine product bug rather than operator config.
+`HistorianGatewayClientAdapter.Create` forwarded the TLS-only options unconditionally, and
+`AllowUntrustedServerCertificate` defaults to `false`, so it always sent
+`RequireCertificateValidation=true` — which the client rejects outright when `UseTls=false`
+(`RequireCertificateValidation is a TLS-only option and requires UseTls=true`).
+
+CLAUDE.md and `docs/Historian.md` both document `http://` as the supported way to select h2c, but
+**no h2c configuration could start**, and the only workaround was to claim
+`AllowUntrustedServerCertificate=true` — a certificate posture for a connection with no certificate.
+Fixed: TLS-only options stay at their defaults when `UseTls=false`.
+
+### 4. THE BLOCKER — the drain gate deferred to a Primary that cannot deliver
+
+With the rig finally up, **every driver node logged `Historian drain suspended`. Nothing drained
+anywhere** — including the two site-b nodes, which have no redundancy peer and no replication at all.
+Before Phase 2 this deployment drained fine, because the drain was ungated.
+
+Root cause, established from the rig rather than guessed:
+
+- `RedundancyStateActor` derives roles from Akka's **cluster-wide** `RoleLeader("driver")`.
+- `central-1`/`central-2` carry **both** `admin` and `driver` Akka roles (`OTOPCUA_ROLES=admin,driver`).
+- So the elected driver Primary is **central-1** — a node that replicates nobody's LocalDb and does
+  not even run the alarm historian.
+- Every site node was `Secondary`, saw that a Primary existed, and stood down in its favour.
+
+**The redundancy role is a cluster-wide election; the alarm queue is pair-local.** Deferring on
+"somebody is Primary" is therefore wrong in principle: in a multi-pair fleet the Primary is usually
+in another pair and holds none of this node's rows. The consequence is not a duplicate — it is the
+buffer growing on every node until the capacity wall evicts the oldest events, i.e. silent permanent
+loss of the audit trail the queue exists to protect.
+
+Fixed in three layers, each with a positive control:
+
+1. `PrimaryGatePolicy.ShouldDrainAlarmHistory(role, queueSharingPeerIsPrimary)` — a **separate**
+   policy from `ShouldServiceAsPrimary`. Unknown role ⇒ drain. A node stands down only for a Primary
+   that holds its rows. The two gates now deliberately disagree, and a test pins that disagreement so
+   nobody "harmonises" them back.
+2. `DriverHostActor` publishes that verdict, matching the snapshot's Primary against this node's
+   configured LocalDb replication peer host.
+3. `AddAlarmHistorian` short-circuits the gate entirely when replication is not configured — a node
+   whose rows exist nowhere else must never defer. The predicate tests **both** `Replication:PeerAddress`
+   and `SyncListenPort`, because only the dialing half sets the former and both halves share the queue.
+
+The asymmetry of the error costs is what drives every one of these: a false allow costs a duplicate
+history row, which the design already accepts (*at-least-once across a failover, by design*, and
+payload-hash ids collapse identical events); a false deny loses data silently.
+
+### 5. A third vacuous test — `AwaitAssert` on the seeded value
+
+Deleting the guard (publishing the old write-gate verdict) left `DriverHostActorRoleViewTests`
+**green**. `AwaitAssert` polls until an assertion passes, and the assertion was "the view reads
+open" — which is also the value the view is *seeded* with, so it succeeded at the first poll, before
+the actor had processed anything. It could not distinguish a correct publish from no publish at all.
+
+Fixed by asserting on the **sequence of published values** via a recording view, never on current
+state. Re-run under the control: red, for exactly the cases that matter. This is the same
+"absence-assertion passes by race" trap recorded for #485/#486, in a new disguise.
+
+## Checks
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 1 | Migration ran; `.migrated` sidecar on both site-a nodes; row counts match | ✅ PASS | `alarm-historian.db.migrated` present on both. 11 legacy rows across the two files → **exactly 9 rows on each node**, id sets byte-identical — the 2 shared payloads collapsed, proving D-6's payload-hash identity in production. Each node holds the *other's* rows (`legacy/a1-only-*` present on a-2 and vice versa). Legacy `attempt_count` distinction survived the copy (the 3-attempt row still leads the 0-attempt rows), `last_error` carried, 0 dead-lettered |
+| 2 | Alarm burst converges: identical rowsets, oplog drains to 0, dead letters 0 | ✅ PASS | Identical id sets on both nodes; `__localdb_oplog` depth **0** on both; **0** dead-lettered. Drain visibly live against the deliberately unresolvable gateway — `attempt_count` climbing 33 → 42 → 168 with `last_error='retry-please'`, which is exactly the buffering behaviour the queue exists for |
+| 3 | Only the primary drains | ❌ **NOT SATISFIED** — see defect 4 | As written the check assumes a per-pair Primary exists. On this rig the elected Primary is a central node outside both pairs, so the honest post-fix behaviour is that **all four nodes drain**. No node can currently identify a queue-sharing Primary: site-b is unpaired, and of the site-a pair only the dialing half knows its peer. Safe (no loss, duplicates only), but the check's intent is unproven |
+| 4 | Failover: standby resumes draining; count double-delivered events | ⛔ **NOT RUN** | Superseded by defect 4. A meaningful failover test needs a rig where the pair itself elects a Primary; measuring "double delivery" is meaningless while both halves drain unconditionally |
+| 5 | Both nodes restarted together: clean rejoin, identical counts, no I/O errors | ✅ PASS | `docker compose restart site-a-1 site-a-2` → **0** `disk I/O error` / `SQLITE_IOERR` / `corrupt` lines and 0 fatals in either log; both back to 9 rows / 0 dead / oplog 0, still identical |
+| 6 | site-b default-OFF pin: sink works locally, no sync traffic | ✅ PASS | `alarm_sf_events` created and registered on both site-b nodes; port **9001 not bound** (`/proc/net/tcp`), no sync listener, no replication config; both drain their own queue (defect 4's third fix) with 0 fatals |
+
+## The open design question
+
+The gate leaves one question that is a design fork, not a bug to patch:
+
+**A pair cannot currently identify its own Primary.** Redundancy roles are elected cluster-wide, but
+the queue is pair-local. Only the dialing half of a pair knows its peer (`Replication:PeerAddress`);
+the listening half sets `SyncListenPort` only. So the drain gate can never fully engage, and both
+halves of a replicated pair drain — safe, but duplicated.
+
+### Follow-up analysis — the root cause is wider than the drain gate
+
+Two things assumed while writing the section above turned out to be wrong, and correcting them
+changes the recommendation.
+
+**The rig is not misconfigured.** `central-1`/`central-2` carry `admin,driver` deliberately — the
+compose says so in as many words ("central is admin,driver"), because they are themselves a
+redundant pair that also runs the MAIN cluster's drivers. Stripping the driver role would change what
+the rig models, not fix anything.
+
+**A "pair" is not a missing concept — it already exists, as the application Cluster.** OtOpcUa's
+`Cluster`/`ClusterId` (the config-DB entity that owns drivers, devices and `ClusterNode` rows) is
+exactly the unit a redundant pair belongs to. Phase 1 already relies on this: the deployment-artifact
+cache is *keyed by ClusterId so a pair shares one entry*, and the rig's three application clusters
+(MAIN, SITE-A, SITE-B) are precisely its three pairs.
+
+So the real defect is that **`RedundancyStateActor` elects one Primary per *Akka* cluster, while
+every meaningful unit of redundancy is an *application* cluster.** The rig runs six driver nodes
+across three application clusters in one Akka cluster — a topology the product otherwise supports
+throughout — and exactly one of those six can ever be Primary.
+
+That mis-scoping is not confined to the alarm drain. Every consumer of the role inherits it:
+`ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate. On a multi-cluster fleet
+today, only one driver node cluster-wide will service Primary-gated work at all.
+
+### Revised recommendation
+
+**(C), as a separate change with its own live gate — and explicitly *not* (A).**
+
+- (C) is no longer "invent a pair concept"; it is "elect per `ClusterId` instead of per Akka cluster",
+  using a first-class entity that already exists. It is feasible where it needs to be:
+  `RedundancyStateActor` has no DB access today, but it is an admin-role singleton in the same
+  assembly as `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
+- **(A) is now actively undesirable.** Deriving pair identity from `LocalDb:Replication:PeerAddress`
+  would introduce a *second*, parallel notion of "who is my partner" that (C) immediately obsoletes —
+  and it cannot be done cleanly anyway: the library exposes no initiator/passive flag, so making both
+  halves declare a peer means both halves *dial*, opening two duplex sessions purely to obtain a
+  configuration value.
+- (C) touches the inbound device-write gate, which is safety-critical (the archreview 03/S4
+  dual-primary fix). It therefore belongs in its own change, with its own live gate, rather than
+  bolted onto Phase 2.
+
+Until then this branch is safe in every topology — no configuration loses data — so what remains is
+reclaiming the de-duplication benefit, not an outstanding hazard.
+
+## Not merged
+
+Per the plan, nothing on this branch is merged.

docs/plans/2026-07-20-localdb-phase2-recon.md

@@ -0,0 +1,371 @@
+# LocalDb Phase 2 — Recon findings
+
+**Date:** 2026-07-21 · **Branch:** `feat/localdb-phase2` · **Base:** `master` `2e46d054`
+
+Answers Task 0 of `2026-07-20-localdb-adoption-phase2.md`. Every claim carries a `file:line`
+citation against the base commit. Read the **Deviations** section last — two of them change the
+shape of Tasks 2/3 from what the plan assumed.
+
+---
+
+## 1. The sink's table schema
+
+**STOP condition does NOT fire** — confirmed by re-reading the executed DDL, as the plan's
+pre-answer instructed.
+
+Executed DDL: `SqliteStoreAndForwardSink.cs:652-670` (`InitializeSchema`). It matches the class
+doc-comment at `:17-26` exactly — no drift between documented and executed schema.
+
+```sql
+CREATE TABLE IF NOT EXISTS Queue (
+    RowId          INTEGER PRIMARY KEY AUTOINCREMENT,
+    AlarmId        TEXT    NOT NULL,
+    EnqueuedUtc    TEXT    NOT NULL,
+    PayloadJson    TEXT    NOT NULL,
+    AttemptCount   INTEGER NOT NULL DEFAULT 0,
+    LastAttemptUtc TEXT    NULL,
+    LastError      TEXT    NULL,
+    DeadLettered   INTEGER NOT NULL DEFAULT 0
+);
+CREATE INDEX IF NOT EXISTS IX_Queue_Drain ON Queue (DeadLettered, RowId);
+```
+
+| Question | Answer |
+|---|---|
+| BLOB columns? | **None.** All 8 columns are TEXT/INTEGER. Payload is `PayloadJson TEXT NOT NULL`. No base64 conversion needed; the 171 KB chunk sizing is moot. |
+| PK autoincrement? | **Yes** — `RowId INTEGER PRIMARY KEY AUTOINCREMENT`. The migrator must mint deterministic `mig-{node}-{legacyId}` ids and the new table needs a TEXT PK, as the plan specified. |
+| Indices to carry | One: `IX_Queue_Drain (DeadLettered, RowId)` — the drain's covering index. `alarm_sf_events` wants the same shape over (status, insertion order). |
+| Largest realistic payload | `PayloadJson` is a serialized `AlarmHistorianEvent` — 10 scalar fields (`AlarmHistorianEvent.cs`), no collections, no nesting. Low single-digit KB worst case; `Comment` is the only loosely-bounded field. |
+
+**Legacy column list** for the migrator's `pragma_table_info` intersection check:
+`RowId, AlarmId, EnqueuedUtc, PayloadJson, AttemptCount, LastAttemptUtc, LastError, DeadLettered`.
+
+`RegisterReplicated` will accept this table: no BLOB, and the PK will be app-minted TEXT.
+
+## 2. The public seam
+
+`IAlarmHistorianSink` (`IAlarmHistorianSink.cs:23-34`) is **two members**:
+
+```csharp
+Task EnqueueAsync(AlarmHistorianEvent evt, CancellationToken cancellationToken);
+HistorianSinkStatus GetStatus();
+```
+
+The concrete `SqliteStoreAndForwardSink` additionally exposes, and these are all consumed
+somewhere, so the rewire must keep them:
+
+| Member | Cited | Consumer |
+|---|---|---|
+| `StartDrainLoop(TimeSpan)` | `:189` | `ServiceCollectionExtensions.cs:105` (DI factory) |
+| `DrainOnceAsync(CancellationToken)` | `:320` | tests only — the deterministic drive |
+| `RetryDeadLettered()` | `:511` | operator action (AdminUI) |
+| `CurrentBackoff` | `:676` | tests |
+| `DebugCapacityProbeCount` | `:99` | tests |
+| `DefaultCapacity` / `DefaultMaxAttempts` consts | `:49`,`:53` | `AlarmHistorianOptions.cs:38,45` |
+| `IDisposable` | `:679` | DI + tests; **disposes the injected writer too** |
+
+There is **no separate enqueue/dequeue/markDelivered/deadLetter interface** — the plan's item 2
+guessed at one. Dequeue, delivered-marking and dead-lettering are all *private* to the sink
+(`ReadBatch :532`, `DeleteRow :564`, `DeadLetterRow :573`, `BumpAttempt :587`). That is good news:
+the rewire is entirely internal to one class and the public seam is untouched.
+
+## 3. Semantics to preserve
+
+All defaults live in **two** places that must stay in agreement:
+
+| Semantic | Sink default | Options default | Enforced at |
+|---|---|---|---|
+| `Capacity` 1,000,000 | `:49` | `AlarmHistorianOptions.cs:38` | `EnforceCapacityFastPathAsync :271` → `EnforceCapacityAsync :602` |
+| `MaxAttempts` 10 | `:53` | `:45` | drain loop `:429` |
+| Dead-letter retention 30 d | `:50` | `:41` | `PurgeAgedDeadLetters :638`, run once per drain tick |
+| `BatchSize` 100 | ctor arg `:116` | `:31` | `ReadBatch :542` |
+| `DrainIntervalSeconds` 5 | — | `:34` | `StartDrainLoop` via `ServiceCollectionExtensions.cs:105` |
+
+Behaviours that are **not** in the plan's list but are load-bearing, and which the cutover must
+not silently drop:
+
+- **Backoff ladder** 1s→2s→5s→15s→60s (`:57-64`), applied to the timer's *next due time*
+  (`RescheduleDrain :224`) so an outage genuinely slows the cadence.
+- **Overflow evicts oldest non-dead-lettered rows** with a WARN and a lifetime
+  `EvictedCount` counter (`:630-635`) — the durability guarantee is explicitly bounded.
+- **Corrupt-payload rows dead-letter immediately** for their own RowId so they cannot stall the
+  queue head (`:344-361`).
+- **Cardinality guard**: a writer returning ≠1 outcome per event is treated as a batch retry
+  (`:393-404`).
+- **In-memory row counter with periodic resync** every 10,000 enqueues (`:89-96`, `:271-290`) —
+  a perf optimisation that avoids `COUNT(*)` on the hot enqueue path.
+- **Post-dispose throwing contract**: `GetStatus`/`RetryDeadLettered`/`EnqueueAsync`/`StartDrainLoop`
+  throw `ObjectDisposedException`; `DrainOnceAsync` returns quietly (`:322`). Pinned by a
+  regression test (`SqliteStoreAndForwardSinkTests.cs:854-865`).
+
+## 4. Drain-worker lifecycle — **the finding that reshapes Task 3**
+
+> The plan asked "hosted service or actor?". **Neither.**
+
+The drain worker is a **self-rescheduling one-shot `System.Threading.Timer` owned by the sink
+itself** (`_drainTimer`, `:76`; started by `StartDrainLoop :189`; callback `DrainTimerCallback :199`;
+re-armed by `RescheduleDrain :224`). It is started from inside the DI singleton factory at
+`ServiceCollectionExtensions.cs:105`, so it begins ticking the moment the first consumer resolves
+`IAlarmHistorianSink`.
+
+Consequences for the Primary gate:
+
+1. **The gate has to live inside `DrainOnceAsync`**, not in a wrapper. There is no external
+   scheduler to gate.
+2. **`Core.AlarmHistorian` cannot see `PrimaryGatePolicy`.** The policy is in
+   `Runtime/Drivers/PrimaryGatePolicy.cs`, and Runtime → Core is the reference direction.
+   `Core.AlarmHistorian` references only `Core.Abstractions` (`.csproj`). So the sink takes a
+   **`Func<bool>` gate delegate** (default: always-drain) and Runtime supplies the lambda that
+   calls `PrimaryGatePolicy`. No new project reference, no new interface in Core.
+3. **A role view is still needed** because the timer fires outside any actor. See the next point
+   for who should own it.
+
+### There is already a Primary gate — on the *enqueue* side, with a different policy
+
+`HistorianAdapterActor` (`Historian/HistorianAdapterActor.cs`) already subscribes to
+`RedundancyStateChanged` (`:86`, handler `:145`), caches `_localRole` (`:42`), and gates the
+enqueue on `ShouldHistorize()` (`:97`):
+
+```csharp
+private bool ShouldHistorize() => _localRole is not (RedundancyRole.Secondary or RedundancyRole.Detached);
+```
+
+This is **not** `PrimaryGatePolicy` — it ignores driver-member count and defaults to *write* while
+the role is unknown. Its comment (`:68-71`) explains why it exists: DPS fans the Primary's single
+`alerts` publish to **every** node's adapter, so without the gate both nodes of a pair would
+double-write.
+
+**This is why today's ungated drain is safe, and why Phase 2 breaks that.** Today the Secondary
+never enqueues, so its queue is empty and an ungated drain has nothing to send. Once
+`alarm_sf_events` replicates, the Primary's rows land in the Secondary's table — and an ungated
+drain there would double-deliver *every* event, continuously, not just at failover. **The drain
+gate is not a nicety; it is required by the same commit that registers the table.**
+
+### Who owns the role view
+
+`HistorianAdapterActor`, not `DriverHostActor`. Both are spawned under
+`WithOtOpcUaRuntimeActors` (`ServiceCollectionExtensions.cs:353`) which `Program.cs:133` calls only
+under `hasDriver` — as are `AddAlarmHistorian` (`:175`) and `AddOtOpcUaLocalDb` (`:185`). So sink,
+LocalDb and adapter are co-located by construction. The adapter wins over `DriverHostActor`
+because it **already** holds the role for exactly this purpose; using it adds no new subscription
+and no new way for the view to go stale while the sink keeps draining.
+
+`DriverMemberCount` is read off Akka cluster state in `DriverHostActor.DriverMemberCount() :1446`
+(try/catch → 0 on a non-cluster provider ⇒ single-node posture). Task 3 will lift that into a
+small shared helper rather than copy it.
+
+## 5. Other `DatabasePath` construction sites
+
+- `ServiceCollectionExtensions.cs:98` — the only production site.
+- `appsettings.json:57` — `"DatabasePath": "alarm-historian.db"`.
+- `AlarmHistorianOptions.Validate() :54-55` — warns when the path is relative. **This warning
+  disappears with the key**; the validator loses one of its five checks.
+- `SqliteStoreAndForwardSinkTests.cs` — ~30 construction sites, all passing the per-test temp
+  `_dbPath` seeded at `:25` and deleted in `Dispose :32`. These are the tests Task 2 rewires onto
+  a temp-file `ILocalDb`.
+- **`docker-dev/docker-compose.yml` has no `AlarmHistorian` keys at all** — the rig runs with
+  `Enabled=false` today. Task 6 is therefore an *addition*, not an edit.
+
+No tooling or CLI constructs the sink.
+
+## 6. How `Enabled=false` short-circuits
+
+`AddAlarmHistorian` returns before registering anything when the section is absent or
+`Enabled != true` (`ServiceCollectionExtensions.cs:87`), leaving the `NullAlarmHistorianSink`
+seeded by `AddOtOpcUaRuntime :49`. The Null sink is a pure no-op (`IAlarmHistorianSink.cs:37-52`).
+
+Confirming the plan's own answer: **the tables are created unconditionally in `OnReady`** (cheap,
+empty) because `AddOtOpcUaLocalDb` is independent of the `AlarmHistorian` gate. Only the
+sink/drain stay Null. This is also required for correctness — `RegisterReplicated` must run on
+both pair members regardless of their historian config, or a node that later enables the sink
+would write rows that are never captured (see the CDC note below).
+
+## 7. Library constraints re-confirmed (README, `ZB.MOM.WW.LocalDb` 0.1.3)
+
+- **BLOB columns are rejected at registration.** Not a problem here (§1).
+- **CDC semantics** — rows that existed *before* `RegisterReplicated` are neither captured nor
+  snapshotted. This is the mechanical reason the migrator must run **after** registration, and
+  the reason `LocalDbSetup.OnReady`'s ordering comment is load-bearing (`LocalDbSetup.cs:23-31`).
+- **LWW keyed on the PK** — two nodes minting the same PK for *different* rows silently overwrite
+  each other; two nodes minting the same PK for the *same* row converge. Decision D-1 below turns
+  that second property into a feature.
+- `ILocalDb` surface in use: `ExecuteAsync` / `QueryAsync(sql, rowMapper, params, ct)` /
+  `BeginTransactionAsync` / `CreateConnection` / `RegisterReplicated`
+  (`Deployment/LocalDbDeploymentArtifactCache.cs:77,82,205`).
+
+---
+
+## Deviations and decisions
+
+Recorded here rather than raised as questions, per the plan's follow-up convention.
+
+### D-1 — Enqueue ids are a deterministic content hash, not `Guid.NewGuid()`
+
+**The plan says** GUIDs minted at enqueue (Task 2 step 2).
+
+**The problem.** The enqueue gate (`ShouldHistorize`, default-**write** on unknown role) and the
+drain gate the plan specifies (`PrimaryGatePolicy`, default-**deny** on unknown role when paired)
+disagree during the boot window. On a two-node pair before the first redundancy snapshot arrives,
+**both** adapters enqueue the same DPS-fanned transition. With random GUIDs those are two distinct
+rows that replicate into each other's tables — the buffer *duplicates* instead of converging, and
+whichever node becomes Primary later delivers both copies.
+
+**The decision.** Mint `id` as a hash over the serialized payload. Two nodes independently
+enqueuing the same fanned event produce the same PK, so LWW converges them to one row for free.
+Two genuinely distinct events cannot collide: `AlarmHistorianEvent` carries a full-precision
+`TimestampUtc` plus alarm id, kind, message and user, so an identical hash means an identical
+event.
+
+This costs one `SHA256.HashData` per enqueue and removes a whole duplicate-row class. It also
+slightly *improves* today's behaviour, where the same boot-window fan-out is already
+double-delivered because the drain is ungated on both nodes.
+
+`ShouldHistorize` itself is left unchanged — out of scope, and with D-1 in place the two gates
+disagreeing is no longer harmful.
+
+### D-2 — Gate denial must be observable
+
+`PrimaryGatePolicy` defaults to **deny** when the role is unknown on a multi-driver cluster. That
+is the correct safety posture, but it means a redundancy-snapshot identity mismatch — the
+documented 03/S5 shape that `DriverHostActor.cs:1485-1491` already warns about once — would
+silently stop alarm history on *both* nodes, degrading only as retry/eviction growth minutes later.
+
+Task 3 will therefore: log the skip once per gate-state transition (not per tick — the drain runs
+every 5 s), and surface it in `HistorianSinkStatus` so `/healthz` and the AdminUI can see
+"buffering, not draining, because not Primary" rather than an unexplained rising queue depth.
+Adding a `HistorianDrainState` member is additive; Task 3 will check every `switch` over it.
+
+### D-3 — Tasks 2 and 3 will land as one commit
+
+The plan anticipated this. Confirmed necessary: the gate delegate is a constructor parameter of
+the rewritten sink, so the sink cannot compile in its final shape without it, and an intermediate
+commit with a rewired-but-ungated sink would be a commit in which a replicated table is drained by
+both nodes. Not a state worth having in history.
+
+### D-5 — The table keeps the legacy delete-on-ack shape, not a `status` column
+
+**The plan proposes** `status TEXT NOT NULL DEFAULT 'pending'` with values `pending | delivered |
+dead`, and drops the legacy `LastError` column.
+
+**Two problems.** A `delivered` status means acknowledged rows accumulate in the table forever —
+the plan specifies no sweeper for them, and they would also have to be excluded from the capacity
+count, changing a semantic §3 says to preserve. And `LastError` is not dead weight: `DeadLetterRow`
+writes the failure reason into it and `RetryDeadLettered` clears it, so it is the only operator-
+facing record of *why* a row was dead-lettered.
+
+**The decision.** Keep the legacy shape: DELETE on acknowledgement, `dead_lettered` as the same 0/1
+flag, and retain `last_error`. Deleting keeps the table bounded with no second sweeper, and the
+replication engine carries the delete as a tombstone (pruned after `TombstoneRetention`, 7 d), so
+the peer drops its copy too — which is the property the plan wanted the `delivered` status for.
+Columns then map 1:1 onto the legacy table, making the Task-4 migrator a straight copy.
+
+One genuine change: the drain's `ORDER BY` moves from `RowId ASC` to `enqueued_at_utc, id`, because
+a hashed TEXT id carries no insertion order. Round-trip ("O") timestamps sort lexicographically in
+chronological order, and `id` makes the ordering total; the index covers both.
+
+### D-6 — The migrator keys on the payload hash, not `mig-{node}-{legacyId}`
+
+**The plan says** deterministic ids of the form `mig-{NodeName}-{legacyId}`, mirroring ScadaBridge.
+
+**Why that is not enough here.** Node-prefixing does solve the collision the legacy AUTOINCREMENT
+key would otherwise cause — node A's `RowId 7` and node B's `RowId 7` are different alarms — but it
+*preserves* a duplication that ought to be collapsed. A warm pair's two legacy files **overlap**:
+`HistorianAdapterActor` default-writes while its redundancy role is unknown, so both nodes accepted
+the same fanned transitions during every boot window they ever had. Prefixed ids would carry every
+one of those duplicates into the merged buffer permanently, for the eventual Primary to deliver
+twice.
+
+**The decision.** Reuse D-1's `AlarmSfSchema.DeriveId` (lifted out of the sink so both call sites
+share one identity rule). Distinct events cannot collide; identical ones converge. It also removes
+the migrator's only need for node identity from configuration, and makes a crash between commit and
+rename harmless under `INSERT OR IGNORE` — which the plan's own idempotency requirement wanted
+anyway.
+
+### D-7 — The migrator's tests live in `Host.IntegrationTests`
+
+The plan names `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests`. **That project does not exist.** The
+tests sit beside `LocalDbSetupTests` in `Host.IntegrationTests`, which is where every other
+LocalDb-facing test already lives. They are offline — a temp file and no external services.
+
+### D-4 — The live gate needs `MaxAttempts` raised on the rig
+
+Not actionable until Task 8, recorded now so it is not rediscovered there. The rig has no
+HistorianGateway, so every drain attempt fails. With `MaxAttempts: 10` and the backoff ladder
+capping at 60 s, buffered rows dead-letter roughly five minutes into the outage — which would make
+the gate's "dead letters 0" check (step 2) fail for reasons that have nothing to do with
+replication. Task 6 should set a high `MaxAttempts` on the rig, or Task 8 should assert on
+dead-letter *timing* instead.
+
+Related: `Program.cs:170-173` shows an `AlarmHistorian:Enabled=true` /
+`ServerHistorian:Enabled=false` deployment is explicitly supported and warned about — which is
+exactly the rig's shape. Task 6 must confirm `ServerHistorianOptionsValidator` does not fail host
+start for a disabled section with an empty `Endpoint`.
+
+---
+
+## Guard-deletion evidence (recorded at the Task 7 DoD sweep)
+
+Every guard this phase adds was proved by deleting it and watching the tests go red. Two of those
+runs found tests that would have passed over a broken implementation. Both are recorded here because
+the *shape* of the vacuity is reusable, not just the fix.
+
+### Control 1 — remove `IRedundancyRoleView.Publish` from `DriverHostActor`
+
+**3 of 6** `DriverHostActorRoleViewTests` went red. The other three assert
+`ShouldServiceAsPrimary == true`, which is exactly the value the view is **seeded** with — so they
+cannot distinguish "published true" from "never published at all". That split is expected and
+acceptable *given* the three that do go red pin the false cases, which only a real publish can
+produce. Worth knowing before someone reads a future 3/6 result as a regression.
+
+### Control 2 — remove `RegisterReplicated("alarm_sf_events")` from `LocalDbSetup.OnReady`
+
+**3 of 4** `AlarmSfConvergenceTests` went red immediately. The fourth,
+`TheSameEventOnBothNodes_ConvergesToOneRow`, **passed vacuously**: with replication switched off
+each node trivially held its own single row, and "count == 1 on both nodes" was satisfied by two
+completely unconverged databases. The assertion could not tell convergence from isolation.
+
+**Fix:** enqueue a second, *distinct* event on node B and require **both** nodes to hold two rows.
+Isolation now yields 1 and 2; only convergence yields 2 and 2. Re-run under the control: red.
+Control restored: all 4 green.
+
+The general trap — an assertion whose expected value is also what a *disabled* system produces —
+is the same one recorded in the `#485`/`#486` unreadable-artifact defect class.
+
+### Exact-set replicated-table pins
+
+Two, not one, and both assert set equality (ordered `ShouldBe` against the literal three-table list,
+so an added *or* dropped registration fails):
+
+- `LocalDbSetupTests` — drives the production `OnReady` callback directly.
+- `LocalDbWiringTests` — resolves `ILocalDb` from the full driver DI graph, which is what catches a
+  registration that never reaches a running host.
+
+## Test-suite evidence (Task 7 DoD sweep)
+
+Full solution run on the branch, compared against a **full solution run on a detached worktree at
+the pre-branch baseline `2e46d054`**. Comparing failure *sets*, not counts, because the suite has a
+standing set of environment- and load-dependent failures that a raw count would hide.
+
+**The two failure sets are identical — all 13 tests, same names, both runs. Zero new failures.**
+
+| Assembly | Baseline | Branch | Delta |
+|---|---|---|---|
+| `Core.AlarmHistorian.Tests` | 0 failed / 29 passed | 0 failed / **32** passed | **+3** (the drain-gate tests; 29 ported from the old sink's suite) |
+| `Runtime.Tests` | 1 failed / 418 passed | 1 failed / **424** passed | **+6** (`DriverHostActorRoleViewTests`) |
+| `Host.IntegrationTests` | 2 failed / 169 passed | 2 failed / **186** passed | **+17** (migrator + convergence + pin updates) |
+
+The 13 standing failures, and why each is not this branch's:
+
+- **3 × `DriverTypeNamesGuardTests`** — `ArgumentNullException (secretResolver)` out of
+  `GalaxyDriverFactoryExtensions.Register`, reached by reflection. A real pre-existing defect in a
+  guard test, reproduced exactly on the baseline worktree. Unrelated to this phase.
+- **4 × `AbLegacy.IntegrationTests`**, **3 × `OpcUaClient.IntegrationTests`**,
+  **1 × `DriverProbeHandshakeE2eTests.AbCip_Green_AgainstSim`** — driver fixtures on the
+  `10.100.0.35` docker host are not running.
+- **1 × `RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed`**
+  — times out under full-suite load; passes in isolation.
+- **1 × `ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks`** — fails
+  under full-suite load ("writer must have been called at least twice"), and `Runtime.Tests` passes
+  **425/425** when run alone. Also fails on the baseline. Worth noting for anyone reading a future
+  run: this one is *newly observed* here only because the previous sweep's log was truncated, not
+  because the branch introduced it.

docs/plans/2026-07-21-issue-485-live-gate.md

@@ -0,0 +1,76 @@
+# Issue #485 — live gate (docker-dev rig)
+
+> Verifies the two fixes for #485 — `OpcUaPublishActor` (master `c7f5b9cf`) and `DriverHostActor`
+> (master `5aad1e33`) — on the local `docker-dev` rig: 6 nodes, shared SQL, real Modbus fixture on
+> `10.100.0.35:5020`.
+>
+> **Safety rules followed**: all DB inspection + mutation via `sqlcmd` inside the SQL container; the
+> rig's Modbus fixture is read-only for this gate (no writes to the PLC sim).
+
+## Result
+
+**Both fixes verified, all three guards observed firing in production, with a live RED control that
+was not staged.** The gate also confirmed a **pre-existing** defect the fixes do not address — the node
+records `Applied` for a deployment it skipped (see "Finding" below).
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 1 | RED control — the bug in its natural habitat, unforced | ✅ | site-a-1 hit #485 during the *previous* gate's SQL flapping and had been serving an **empty address space for 34 minutes**: `failed to load artifact … rebuild becomes no-op` followed immediately by `applied plan (kind=PureRemove, added=0, removed=17, rebuild=True)`. Its 17 `ns=2` nodes were gone; a browse returned only the root `OtOpcUa` folder. Nothing was staged to produce this — it was already true on the pre-fix image. |
+| 2 | Rebuilt on the fixed image, the node recovers | ✅ | After `docker compose build` + `--force-recreate`, site-a-1 restored to `raw subtree materialised (containers=16, tags=1)` — 18 `ns=2` nodes. |
+| 3 | Address-space guard fires; nodes survive | ✅ | site-a-1, empty artifact: `OpcUaPublish: no usable artifact for deployment f4b9dd63… — KEEPING the currently served address space rather than tearing it down`. **No `PureRemove`.** Browse before vs after: 18 vs 18, `diff` **identical**. |
+| 4 | Reconcile guard fires; drivers survive | ✅ | `artifact for f4b9dd63… read back empty; skipping reconcile and keeping the 1 running driver(s)`, and the apply still reported `children=1`. Repeated on central-2 with a bigger blast radius: `keeping the 5 running driver(s)`, `children=5`. |
+| 5 | SubscribeBulk guard fires; **live values keep flowing** | ✅ | central-2, `artifact for 8c36a673… read back empty; skipping SubscribeBulk and keeping the live subscriptions`. The decisive proof is the value, not the log: `ns=2;s=pymodbus/plc/HR200X` read **56278 @ 06:59:51** before the induction, then **56336 @ 07:00:21** and **56354 @ 07:00:30** after it (induction at 07:00:07). The subscription was still delivering fresh Modbus data across the event. |
+| 6 | Positive control — a readable artifact still applies | ✅ | Immediately after the guards fired, a genuine deploy applied normally on both nodes: site-a-1 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 1 references`; central-2 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 5 references across 4 driver(s)`. The guards suppress the teardown, **not** deployment. |
+| 7 | Control — only the node that saw empty bytes took the guard path | ✅ | Of the six nodes, exactly **one** logged `read back empty` (the paused one); the other five read the real artifact before it was blanked and applied it normally. The induction window is what produced the condition, not a global change. |
+
+## How the failure was induced deterministically
+
+The original incident was a race (a transient ConfigDb blip landing between the apply and the artifact
+read). Racing it again would be unreliable, so the gate drives the **empty-bytes** branch instead — which
+is byte-for-byte what the code sees when the row cannot be read, and is the branch the driver-side guards
+key on (their *throw* paths already returned early before this work):
+
+1. `docker pause` the target node, freezing it mid-cluster.
+2. `POST /api/deployments` — the deployment seals and dispatches while the node cannot process it.
+3. `UPDATE Deployment SET ArtifactBlob = 0x WHERE DeploymentId = <new>`.
+4. `docker unpause` — the node now processes the dispatch and reads an empty artifact.
+
+The pause measured **<1s** on both runs, well inside `acceptable-heartbeat-pause = 10s` and SBR's
+`stable-after = 15s`, so the node was never flagged unreachable and no failover was triggered.
+
+## Finding — the node records `Applied` for a deployment it skipped (pre-existing, NOT fixed here)
+
+Both guards leave the node running its previous configuration, which is the intent. But `ApplyAndAck`
+advances `_currentRevision = revision` and writes `NodeDeploymentState = Applied` **before** it knows
+whether anything was applied — that assignment sits immediately after `ReconcileDrivers`, which returns
+`null` on both the (pre-existing) throw path and the new empty path:
+
+```
+[07:00:07 WRN] DriverHost central-2:4053: artifact … read back empty; skipping reconcile …
+[07:00:07 INF] DriverHost central-2:4053: applied deployment 8c36a673… (rev 03b33904…, children=5)
+```
+
+`NodeDeploymentState` for that deployment reads `Status = Applied` for central-2. The node now claims a
+revision whose configuration it never applied — and because `HandleDispatchFromSteady` short-circuits on
+a revision match (`dispatch matches current rev; immediate ACK`), **re-dispatching that same revision is
+a no-op**, so the node cannot self-heal. Only a later deployment with a *different* revision recovers it
+(check 6 confirms that path works).
+
+This is pre-existing — the throw path has always done it — and is strictly a reporting/convergence bug,
+not a data-plane one: the node keeps serving its last good configuration throughout. The honest fix is to
+ACK `Failed` and leave `_currentRevision` alone when the artifact could not be read, so the normal deploy
+retry heals it. Tracked separately rather than folded in here, because it changes deploy ACK semantics
+fleet-wide.
+
+## Addendum — issue #486 fix, live-gated (2026-07-21)
+
+The finding above is now fixed (`DriverHostActor.ApplyAndAck` fails the apply when `ReconcileDrivers`
+returns null) and gated on the same rig with the same induction.
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 8 | Regression — a normal deploy still ACKs Applied fleet-wide | ✅ | The change alters ACK semantics, so this is the risk that matters. A genuine deploy recorded `Status = Applied` on **all six** nodes. |
+| 9 | An unreadable artifact now ACKs Failed, with a reason | ✅ | central-2 recorded `Status = Failed` + `the deployment artifact could not be read (ConfigDb unreachable, or the row is missing/empty); nothing was applied`. The five nodes that read the real artifact recorded Applied — so the fleet view now distinguishes them. Pre-fix, all six claimed Applied. |
+| 10 | The revision is NOT advanced | ✅ | `apply of a855dd9e… FAILED — … Staying on revision 5bc0d9f3…` — the **previous** revision, not the dispatched `583122b7…`. This is what stops `HandleDispatchFromSteady` from short-circuiting a retry of that revision. |
+| 11 | The node keeps serving throughout (#485 still holds) | ✅ | The #485 guard fired first (`keeping the 5 running driver(s)`), and `ns=2;s=pymodbus/plc/HR200X` read **59603 @ 07:27:28**, live and current, after the Failed apply. Failing the apply reports the truth; it does not tear anything down. |
+| 12 | The node recovers on the next real deploy | ✅ | The following deployment applied on central-2 (`children=5`) and recorded Applied on all six nodes. |

docs/plans/2026-07-21-localdb-followups-live-gate.md

@@ -0,0 +1,86 @@
+# LocalDb Phase 1 follow-ups — live gate (docker-dev rig)
+
+> Closes the two limitations left open by `2026-07-20-localdb-phase1-live-gate.md` (checks 4 and 8).
+> Rig: local `docker-dev/docker-compose.yml`, 6 nodes rebuilt from `fix/localdb-phase1-followups`.
+> site-a pair replicates; central + site-b stay default-OFF.
+>
+> **Safety rules followed** (same as the Phase 1 gate): all DB inspection via `docker cp` of the
+> `db`/`-wal`/`-shm` triplet, querying the copy — never host `sqlite3` on the live WAL file.
+
+## Result
+
+**Both follow-ups closed, and the gate found a third defect** — one that was *unreachable* until the
+first fix landed. Library `ZB.MOM.WW.LocalDb` went `0.1.1` → `0.1.2` → `0.1.3` in the course of this
+gate; both consumers (OtOpcUa, ScadaBridge) are pinned to `0.1.3`.
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 8 | Oplog stops growing on a default-OFF node | ✅ PASS | site-b-1 restarted twice with an unchanged cached artifact: `oplog_depth` **7 → 7 → 7**, `deployment_pointer.applied_at_utc` frozen at `05:34:58` across both — the re-cache was skipped outright. Repeated on the `0.1.3` build after a real deploy: **13 → 13**. Pre-fix this grew ~2 rows per restart forever, because a default-OFF node has no peer to ack and prune them. |
+| 8b | …but a genuinely new deployment still writes (positive control) | ✅ PASS | A real config change (renamed tag `SITEA-tag`) produced rev `822bdf34`, then `002c16b3`. site-b-1's oplog **7 → 10 → 13** and its pointer advanced each time. The skip suppresses *identical* re-writes only; it is not a dead cache-write path. |
+| 4 | A wiped node is back-filled by its peer, no new deploy, central down | ✅ PASS (was ⚠️ PARTIAL) | Precondition asserted first — site-a-1's oplog **0**, `needs_snapshot 0`: the exact state that used to make back-fill impossible. Then SQL stopped, site-a-2's LocalDb **volume destroyed**, container recreated. a-1 logged `Snapshot sent (as_of_seq 0, 4 rows)` — a line that could not appear on `0.1.1`. a-2 came back with pointer `SITE-A / 475df87a @ 03:09:12` (a-1's **original** timestamp, not a re-derived one), chunks 2, and a `__localdb_row_version` dump **byte-identical to a-1's**, carrying origin node ids `6812ca9c` and `935659a5` — the rebuilt node's own id appears nowhere, so the rows were replicated, not recreated. |
+| 4b | …and the back-filled node then boots from cache | ✅ PASS | With SQL still down, site-a-2 logged `RUNNING FROM CACHE — … booted deployment 475df87a… cached at 2026-07-21T03:09:12`, `raw subtree materialised (containers=16, tags=1)`. OPC UA browse of both nodes: **17 ns=2 nodes each, diff-identical** — the rebuilt node serves exactly what its healthy peer does, from a configuration it never applied itself and could not have fetched. |
+| 4c | The rebuilt node's OWN writes reach its peer | ✅ PASS **after a third fix** (see below) | On `0.1.2` this failed: a-1 held `last_applied_remote = 10` while the rebuilt a-2's oplog was seqs **1,2,3** and its `last_acked` stayed **0** — never accepted. On `0.1.3`, a-1 logs `Replication peer identity changed (ebb10854 -> c85d8b1d): the peer was rebuilt … Resetting the inbound watermark`, and after the next deploy a-1 shows `last_applied_remote = 3` / a-2 shows `last_acked = 3` with a drained oplog and identical row_versions. |
+
+## The third defect (the reason this gate was worth running)
+
+`0.1.2` made back-fill work, which for the first time let a rebuilt node be observed *writing*. It
+turned out its writes went nowhere.
+
+`last_applied_remote_seq` is a watermark in the **peer's** seq space, and a rebuilt peer is a new
+database whose oplog numbers from 1. Two independent bugs stacked, and fixing either alone leaves the
+data stuck:
+
+1. The healthy node kept advertising the **old** peer's watermark. The *sending* side seeds its pump
+   directly from that number (`_sentThruSeq`), so the rebuilt node skipped its entire oplog.
+   → the watermark (and the observed peer clock) is now reset when the peer's node id changes.
+   `last_acked_seq` is deliberately **not** reset: it describes our own oplog's pruned horizon and is
+   exactly what `0.1.2` keys the snapshot decision off — clearing it would switch back-fill back off.
+2. A peer claiming to have applied more of our stream than we have ever produced can only be
+   remembering a previous incarnation of *us*. → that claim is now rejected in favour of pumping from
+   the start; LWW makes the re-send harmless. This mirrors the clamp `RecordPeerAckAsync` already
+   applies to acks.
+
+Both were reproduced offline as a RED test before fixing
+(`SnapshotResyncTests.RebuiltPeer_OwnWritesReachTheHealthyNode_DespiteTheOldPeersWatermark`).
+
+**Why offline tests could not have found it first:** the wiped-peer scenario had no test because the
+scenario was believed impossible to recover from — it was a *documented limitation*, not a bug. The
+gate is what turned the limitation into a reproducible case, and the second defect only existed
+behind the first.
+
+## Unrelated finding (NOT a follow-up of this work — pre-existing; `lmxopcua` issue #485, since FIXED)
+
+While SQL was being stopped and started to drive check 4, site-b-1 took a deploy whose artifact load
+hit a transient ConfigDb error, and **emptied its served address space**:
+
+```
+[06:11:20 ERR] An error occurred using the connection to database 'OtOpcUa' on server 'sql,1433'.
+[06:11:20 WRN] OpcUaPublish: failed to load artifact for deployment 43d54397…; rebuild becomes no-op
+[06:11:20 INF] AddressSpaceApplier: applied plan (kind=PureRemove, added=0, removed=16, rebuild=True)
+```
+
+The log says "rebuild becomes no-op" and the applier then removed all 16 nodes — those two disagree.
+The cache layer behaved correctly and refused to store the bad artifact ("not caching deployment … its
+artifact was empty or could not be loaded, so it is not a usable last-known-good configuration"), and
+the node recovered fully on restart (`containers=16`). But a transient ConfigDb blip emptying the
+served address space instead of holding last-known-good is the same failure *class* as the Phase 1
+gate's check-3 defect. Untouched by this work — site-b-1 has replication off entirely — and induced by
+this gate's own SQL flapping.
+
+**Fixed** (issue #485): `OpcUaPublishActor.HandleRebuild` now treats an artifact it could not obtain as
+*no answer* rather than *an empty configuration*, and abandons the rebuild with the served address space
+(and `_lastApplied`) intact. Nothing legitimate produces a zero-length blob — an operator who really
+deletes everything still deploys a JSON document with empty arrays — so "no bytes" can only mean the read
+failed. The log excerpt above is now impossible: the `PureRemove` never runs, and the loader's own
+"the rebuild is abandoned" line is finally true. Covered by
+`OpcUaPublishActorRebuildTests.Rebuild_keeps_the_last_known_good_address_space_when_the_artifact_load_fails`,
+with a positive control proving a *readable* artifact that genuinely drops an equipment still tears its
+subtree down.
+
+The **driver-side half** of the same mistake was fixed alongside it: `DriverHostActor.ReconcileDrivers`
+and `PushDesiredSubscriptionsFromArtifact` read the artifact the same way, so empty bytes meant zero
+driver specs — `DriverSpawnPlanner` planned every running child for `StopChild` — and then an empty
+desired set dropped each surviving driver's live subscription handle. A node lost its entire field I/O
+and still ACKed Applied. Both now skip on empty. Covered by `DriverHostActorUnreadableArtifactTests`,
+whose absence assertion is calibrated by a control that observes the very same unsubscribe *arriving*
+inside the settle window (without it the assertion passed on a race).

docs/plans/2026-07-21-per-cluster-mesh-design.md

@@ -0,0 +1,337 @@
+# Per-Cluster Akka Mesh — design
+
+> **Status:** design for review. **Phases 0a and 0b are DONE and merged** (2026-07-21); Phases 1–7
+> are not started and each needs its own plan.
+> **Decisions settled 2026-07-21** — see §6: one central DB with driver nodes disconnected from it,
+> the two-node keep-oldest gap closed upstream and now closed here too (§6.2), and their auth posture
+> matched for now.
+
+## 1. What this changes
+
+Today OtOpcUa runs **one Akka mesh containing every node of every application Cluster**. That was a
+deliberate choice — `docs/plans/2026-06-07-per-cluster-scoping.md`, "Per-ClusterId Scoping
+(hub-and-spoke **single mesh**)" — so the deploy channel could stay in-mesh: AdminUI →
+`admin-operations` singleton → `deployments` topic → every node filters to its own `ClusterId`.
+
+The proposal is to move to the sister project's shape: **one Akka mesh per application Cluster,
+two nodes maximum**, with central and clusters joined by explicit transports instead of cluster
+gossip.
+
+The motivation is that redundancy becomes correct *by construction*. Every Primary-gated decision
+(inbound device writes, native alarm acks, fleet alerts, ServiceLevel, and — since Phase 2 — the
+alarm-history drain) currently asks "am I the driver Primary?" of a **cluster-wide** election, while
+every resource being gated is **pair-local**. Splitting the mesh collapses the two scopes into one
+and the question stops being ambiguous.
+
+## 2. What ScadaBridge actually does
+
+Researched directly from `/Users/dohertj2/Desktop/ScadaBridge`. Five things differ from the
+summary in its own CLAUDE.md, and they matter:
+
+**Three transports cross the boundary, not two.**
+
+| Transport | Direction | Carries |
+|---|---|---|
+| Akka **ClusterClient** | central → site (+ replies) | command/control: deploy notify, lifecycle, queries, subscribe handshakes |
+| **gRPC** server-streaming | **central dials into the site** | real-time attribute + alarm events |
+| Plain **HTTP**, token-gated | site → central | the deployment config itself (notify-and-fetch) |
+
+**The gRPC direction is inverted from the obvious guess.** Data flows site→central, but *each site
+node hosts the gRPC server* (Kestrel h2c :8083) and *central is the client*. There is no gRPC server
+on central at all.
+
+**Active node = OLDEST Up member, explicitly not the cluster leader.**
+`ActiveNodeEvaluator.SelfIsOldestUp` is documented as *"THE single definition of 'active node'"*:
+
+> *"Cluster LEADERSHIP (lowest address) is an Akka-internal concept that diverges from singleton
+> placement permanently once the original first node restarts and rejoins; every product-level
+> active/standby decision must use this evaluator, never `cluster.State.Leader`."*
+
+The equivalence **oldest-Up == where `ClusterSingletonManager` places singletons** *is* the design.
+
+**All clusters share one ActorSystem name** (`"scadabridge"`, hardcoded). They are separate clusters
+only by seed-node partitioning — required, because Akka.Remote address matching means ClusterClient
+could not reach a differently-named system.
+
+**Site nodes carry two roles:** `"Site"` and `"site-{SiteId}"`. Singletons scope to the
+**site-specific** role.
+
+Other load-bearing details:
+
+- **Central discovers sites from the database** (a `Site` entity with `NodeAAddress`/`NodeBAddress`
+  + `GrpcNodeAAddress`/`GrpcNodeBAddress`), refreshed every 60 s and on admin change. Sites discover
+  central from **appsettings** — static, restart required. The asymmetry is deliberate.
+- **Exactly one actor is exposed per side** via `ClusterClientReceptionist.RegisterService` —
+  `/user/central-communication` and `/user/site-communication` — registered **per node, not as a
+  singleton**, so contact rotation reaches whichever node answers.
+- **No central buffering when a site is unreachable.** The send is dropped with a warning and the
+  caller's Ask times out. *"It keeps the central coordinator stateless with respect to site
+  availability."* A `ConnectionStateChanged` mechanism was built for this and **deleted as dead
+  code**.
+- **Every unhandled message replies with a typed failure rather than dropping**, so a central Ask
+  can never stall on a missing handler.
+- **Sender preservation is the whole Ask idiom**: `_client.Tell(new ClusterClient.Send(path, msg),
+  Sender)` routes the reply straight back to the waiting Ask, bypassing the comm actor.
+
+### The registered outage gap — read before copying
+
+> `Component-ClusterInfrastructure.md:113`: *"Only the FIRST seed listed in `Cluster:SeedNodes` may
+> self-join to form a *new* cluster… a lone restarted non-first-seed node (with the first seed still
+> down) loops on `InitJoin` forever — never `Up`, never routable. This is why the two-node
+> keep-oldest **oldest/active-node crash is a total-outage gap**: after the oldest dies the younger
+> survivor self-downs, and it cannot re-bootstrap alone. Recovery is operator-driven."*
+
+Their failover drill has a mode that exists *"to make the registered gap observable — not to pretend
+it is covered."* **A two-node keep-oldest mesh has an acknowledged total-outage hole.** If OtOpcUa
+adopts 2-node meshes, this must be decided deliberately, not inherited.
+
+> **CLOSED 2026-07-22.** The `InitJoin` half of this gap is fixed in both repos by **self-first seed
+> ordering**: every node that is one of its own seeds lists ITSELF as `seed-nodes[0]`, which is the
+> only ordering under which Akka runs `FirstSeedNodeProcess` and can form a cluster with no peer
+> answering. (An earlier `Cluster:SelfFormAfter` watchdog was tried here and retired — it raced the
+> join handshake.) See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Today's
+> driver-only site nodes are seeded by `central-1` only and are exempt; the per-cluster mesh removes
+> even that exception by making every pair node a seed of its own mesh.
+
+### Both inter-cluster transports are unauthenticated
+
+Akka remoting has no TLS, no secure cookie, no `trusted-selection-paths`. The gRPC listener is h2c
+with no auth covering `SiteStreamService` — including two Pull RPCs that return audit rows. The only
+authenticated pieces are the HTTP fetch token and `LocalDbSyncAuthInterceptor`, which gates *only*
+`/localdb_sync.v1.LocalDbSync/`. Their boundary assumes a trusted network.
+
+**OtOpcUa already ships that same fail-closed interceptor** (LocalDb Phase 1) — it is a ready-made
+template if we choose to authenticate more than they did.
+
+## 3. What OtOpcUa has today
+
+**Nine cross-node DPS topics plus a singleton**, all currently relying on one mesh:
+
+| Channel | Direction | Purpose |
+|---|---|---|
+| `deployments` / `deployment-acks` | central ↔ nodes | deploy dispatch + acks |
+| `driver-control` | central → nodes | AdminUI Reconnect/Restart |
+| `redundancy-state` | central → nodes | roles + peer probe results |
+| `alerts` | nodes → central | fleet alarms → `/alerts` |
+| `driver-health`, `driver-resilience-status`, `fleet-status`, `script-logs` | nodes → central | AdminUI live panels |
+| `admin-operations` singleton | central | operator ack/shelve into engines |
+
+Three facts make the split cheaper than that table suggests:
+
+1. **The deploy path is already notify-and-fetch.** `DispatchDeployment` carries only
+   `DeploymentId` + `RevisionHash` + `CorrelationId`; the node fetches the artifact itself. We
+   independently arrived at the pattern ScadaBridge had to retrofit, and we do **not** carry its
+   128 KB Akka frame exposure on this path.
+2. **Deploy state already has a DB substrate.** `ConfigPublishCoordinator` persists per-node ACKs to
+   `NodeDeploymentState` so a singleton failover recovers in-flight state from the DB.
+3. **`ClusterNode` rows already enumerate every node per application Cluster** — the same table
+   `AdminOperationsActor` groups by cluster. This replaces the coordinator's one genuinely
+   mesh-bound dependency: it derives its expected-ack set from `Akka.Cluster.State.Members` filtered
+   by role.
+
+So the deploy channel needs only a small notify + ack transport. **The nine live-telemetry channels
+are the real work**, and they are exactly what ScadaBridge built ClusterClient + gRPC for.
+
+## 4. A defect to fix regardless of this design
+
+OtOpcUa currently uses **two different node-selection rules at once**:
+
+- Split-brain resolution is **keep-oldest** (`Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49`).
+- `ClusterSingletonManager` places singletons on the **oldest** member.
+- But `RedundancyStateActor.BuildSnapshot` elects Primary from **`RoleLeader("driver")`** — lowest
+  address among driver members.
+
+Those select the same node only by coincidence. After a restart-and-rejoin the role leader and the
+oldest member diverge permanently, which means the node the SBR protects and hosts every singleton
+on need not be the node the data-plane gates consider Primary. This is precisely the rule ScadaBridge
+forbids by name, and for the reason it gives: *"both sides claim leadership during a partition"* —
+the dual-primary shape archreview 03/S4 exists to prevent.
+
+**Switching the role derivation to oldest-Up-with-role is small, is correct under either topology,
+and should not wait for this design.** It also happens to be exactly what the separate-mesh model
+needs, so it is not throwaway work.
+
+**FIXED 2026-07-21 (Phase 0b).** `RedundancyStateActor.BuildSnapshot` selects the oldest Up member
+carrying the `driver` role, matching singleton placement; `NodeRedundancyState.IsRoleLeaderForDriver`
+was renamed `IsDriverPrimary` so the contract stops asserting the old derivation. Pinned by
+`RedundancyPrimaryElectionTests`, which builds a real two-node cluster in which the oldest node holds
+the *higher* address, so the two orderings genuinely disagree.
+
+## 5. Target architecture
+
+Mirror ScadaBridge's shape, adapted to OtOpcUa's existing substrate:
+
+- **One Akka mesh per application `Cluster`, two nodes max.** Same ActorSystem name (`otopcua`)
+  everywhere; separation by seed-node partitioning only.
+- **Roles per node:** `driver` plus a cluster-specific `cluster-{ClusterId}`, with singletons scoped
+  to the cluster-specific role.
+- **Active node = oldest Up member with role** (§4), one definition, used by the data-plane gates,
+  ServiceLevel, and the alarm drain alike.
+- **Central ↔ cluster command/control over ClusterClient**, with exactly one receptionist-registered
+  actor per side and central discovering cluster node addresses from `ClusterNode` rows (extended
+  with Akka + gRPC addresses, mirroring their `Site` entity).
+- **Deploy stays notify-and-fetch, but the fetch changes source.** The notify crosses via
+  ClusterClient; the artifact then comes **from central**, not from the ConfigDb the driver node no
+  longer connects to (§6.1), and is cached in LocalDb. The coordinator's expected-ack set comes from
+  `ClusterNode` rows instead of cluster membership.
+- **Driver nodes hold no ConfigDb connection.** LocalDb becomes their steady-state configuration
+  store; five current DB consumers are re-homed or re-sourced per §6.1.
+- **Live telemetry over gRPC**, central dialling into each cluster node, replacing the seven
+  observability topics with one stream contract carrying a `oneof` event — additive-only field
+  evolution, contract-locked by test.
+
+**Copied:** their node-local configuration store fed by fetch-and-cache (§6.1), their oldest-Up
+active-node rule (§4), their two-node keep-oldest posture including its acknowledged outage gap
+(§6.2), and their unauthenticated inter-cluster transports (§6.3).
+
+**Deliberately not copied:** their *transient-only* central alarm cache. OtOpcUa persists alarm
+history centrally through the historian, and Phase 2's replicated store-and-forward buffer already
+guarantees delivery across a failover — so there is no reason to adopt a design whose stated
+trade-off is that a new active node re-seeds alarm state from scratch.
+
+## 6. Decisions (settled 2026-07-21)
+
+### 6.1 DECIDED — one central DB, and driver nodes never connect to it
+
+**Verified in ScadaBridge before adopting.** `Program.cs` calls
+`AddConfigurationDatabase(configDbConnectionString)` at line 264 — inside the Central branch
+(89–478). The Site branch begins at line 479. **Site nodes never register the central configuration
+database at all**; they receive config from central (notify → token-gated HTTP fetch) and persist it
+locally. There is still exactly one authoritative configuration database — it is simply
+central-only.
+
+OtOpcUa adopts the same shape: **the ConfigDb stays the single source of truth, and driver nodes stop
+connecting to it.** They obtain their configuration from the central nodes and cache it locally in
+LocalDb.
+
+**This promotes LocalDb from an outage cache to the driver node's steady-state configuration store.**
+That is a reframing of the Phase 1 work rather than a contradiction of it: boot-from-cache stops
+being the exceptional path and becomes the normal one, and "central SQL unreachable" stops being a
+degraded mode for driver nodes because they never held a connection to lose. The chunking,
+SHA-256 verification, newest-2 retention and pair replication all carry over unchanged.
+
+**Scope — five driver-side ConfigDb consumers must be re-homed or re-sourced.** Audited:
+
+| Consumer | Current use | Disposition |
+|---|---|---|
+| `DriverHostActor` | artifact fetch + ack writes (5 `CreateDbContext` sites) | artifact via fetch-and-cache; acks via the ack transport |
+| `EfAlarmConditionStateStore` | Part 9 alarm condition state | **move to LocalDb** — it is pair-local state, the same journey Phase 2 made for the alarm S&F buffer |
+| `DbHealthProbeActor` | DB reachability feeding ServiceLevel tiering | **semantics change** — a driver node has no DB to probe; decide what health input replaces it |
+| `OpcUaPublishActor` | (audit required) | TBD |
+| `ServiceCollectionExtensions` | registration | follows the above |
+
+`DbHealthProbeActor` deserves particular care: DB health is currently a ServiceLevel input, and
+under this model it stops existing for driver nodes. That is a behaviour change on a client-visible
+value, not just an internal refactor.
+
+### 6.2 CORRECTED — the gap was closed in ScadaBridge, and OtOpcUa still has it
+
+**An earlier revision of this section was wrong.** It reported the two-node keep-oldest outage gap as
+still open, citing `Component-ClusterInfrastructure.md:113` and Gitea PR
+[#12](https://gitea.dohertylan.com/dohertj2/ScadaBridge/pulls/12) (which made the failover drill
+*honest* about the gap rather than closing it). That doc line was stale: ScadaBridge closed the gap
+in `cf3bd52f`, *"feat(cluster): auto-down downing strategy — either-node crash now fails over (owner
+decision 2026-07-21: availability over partition-safety)"*.
+
+Its commit message carries the analysis, live-proven on their rig:
+
+> *"Two-node keep-oldest could NEVER survive a crash of the oldest/active node: Akka.NET 1.5.62
+> `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side with >= 2 members, so the 1-vs-1
+> survivor takes `DownReachable` and downs ITSELF — proven live on the rig ('SBR took decision …
+> including myself') before this change. `static-quorum(1)` is worse (`IsTooManyMembers` -> `DownAll`);
+> `keep-majority` just re-keys the fatal crash to the lowest address."*
+
+**OtOpcUa is running that exact configuration today** —
+`Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49-53`: `active-strategy = "keep-oldest"`,
+`keep-oldest { down-if-alone = on }`, `stable-after = 15s`, with
+`BuildClusterOptions` setting the matching typed `KeepOldestOption { DownIfAlone = true }`.
+
+So a two-node OtOpcUa pair **could not fail over when the oldest node crashed**: the survivor
+downed itself. That was a total-outage defect in the redundancy feature itself, independent of this
+design.
+
+**FIXED 2026-07-21 (Phase 0a).** `Cluster:SplitBrainResolverStrategy` now selects the downing
+provider and defaults to `auto-down` — Akka's `AutoDowning` with a 15s window — so the leader among
+the reachable members downs the unreachable peer and a crash of either node fails over. `keep-oldest`
+remains selectable for deployments that prefer partition-safety. Pinned by
+`SplitBrainResolverActivationTests`, which asserts the provider read back off a running ActorSystem
+rather than the configured one. See `docs/Redundancy.md`.
+
+**Its live gate is still outstanding**, and cannot run on today's rig: the failure only appears in a
+1-vs-1 split, and `docker-dev` is a single six-node mesh. The crash-the-oldest drill therefore lands
+with Phase 6/7, which makes every mesh exactly two nodes.
+
+**The bootstrap half of the gap is also closed (2026-07-22).** Downing was only ever one of the two
+outage modes: even with `auto-down`, a node cold-starting while its peer was dead looped on `InitJoin`
+forever, because Akka lets only `seed-nodes[0]` form a new cluster. The fix is **self-first seed
+ordering** — each seed node lists itself first, enforced at boot by `AkkaClusterOptionsValidator`.
+The `Cluster:SelfFormAfter` watchdog first shipped for this is deleted: a timer outside Akka's join
+handshake cannot tell "no seed answered" from "a seed answered and the join is in flight", and it
+islanded a manually-failed-over node live. Pinned by `SelfFirstSeedBootstrapTests`; see
+`docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Under this design each pair node is a
+seed of its own two-node mesh, so the ordering covers both sides and the current site-node exemption
+disappears.
+
+### 6.3 DECIDED — match ScadaBridge's auth posture for now
+
+Inter-cluster transports stay unauthenticated and unencrypted, as theirs are: the boundary assumes a
+trusted network. Recorded as an accepted risk rather than an oversight, with two notes for whenever
+it is revisited: `LocalDbSyncAuthInterceptor` is already in our tree as a fail-closed template, and
+the surface most worth gating first is any Pull-style RPC that returns historical rows.
+
+## 7. Sequencing sketch
+
+Deliberately not a task plan — per-phase plans follow, one at a time.
+
+| Phase | Content | Independent of the split? |
+|---|---|---|
+| 0a | Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2) | **Yes** — **DONE 2026-07-21**, live gate deferred to Phase 7 |
+| 0b | Oldest-Up role derivation (§4) | **Yes** — **DONE 2026-07-21** |
+| 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | **Yes** — **DONE 2026-07-22** (see below) |
+| 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No |
+| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |
+| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No |
+| 5 | gRPC stream contract; migrate the seven observability topics | No |
+| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No |
+| 7 | Failover drill (both directions, per §6.2); live gate | No |
+
+Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each
+deserves its own live gate.
+
+### Phase 1 as shipped (2026-07-22)
+
+Plan: `2026-07-21-per-cluster-mesh-phase1.md`. Two deviations from the sketch above, both decided
+before implementing:
+
+- **No cluster-scope filtering of the expected-ack set.** The plan called for it, but there is no
+  cluster-scoped deployment to filter on: `Deployment` has no `ClusterId`,
+  `ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and
+  `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The
+  expected set is every enabled `ClusterNode` row, which is what the membership rule produced too.
+- **No per-node role column.** The membership rule filtered on the `driver` role and the DB has none.
+  Rather than add one — a second declaration of node roles, free to drift from `Cluster:Roles` — every
+  `ClusterNode` row is now *defined* to be a driver node. An admin-only node must not be given one.
+
+Also shipped beyond the sketch: `ClusterNodeAddressReconciler` (Task 4), an admin singleton that
+catches `AkkaPort` drifting from the node's own `Cluster:Port`. **Phase 2 must revisit it** — once the
+meshes split, an admin node cannot see site members and every site row would report
+`EnabledRowNotInCluster` forever.
+
+## 8. Risks
+
+- **LocalDb becomes load-bearing for configuration, not just resilient.** Phase 1 built it as a
+  fallback; §6.1 makes it the driver node's only config store. A cache defect that was previously a
+  degraded-mode bug becomes a total-availability bug — the #485 unreadable-artifact class, with the
+  blast radius raised.
+- **`DbHealthProbeActor` feeds a client-visible value.** Removing the driver-side DB connection
+  changes what ServiceLevel means on those nodes; that is an interop-visible change, not internal.
+- **The rig currently models the topology that would be abolished.** Phase 6 rewrites
+  `docker-dev/docker-compose.yml` substantially, and every live gate that depends on it.
+- **Losing gossip loses free fleet-wide observability.** Seven AdminUI panels are fed by DPS today;
+  each needs an explicit stream and a reconnect story.
+- **Akka frame size.** We are clean on the deploy path, but anything new that carries payload over
+  ClusterClient inherits the 128 KB default with `log-frame-size-exceeding` off — a silent
+  single-message drop that leaves the association healthy. If we send payload at all, set both.
+- **Do not stack application-level last-write-wins on LocalDb's HLC.** Their `SiteStorageService`
+  carries this warning in code. Verified: our `deployment_pointer` upsert is unconditional and the
+  alarm sink is idempotent on a payload hash, so we are currently clean — keep it that way.

docs/plans/2026-07-21-per-cluster-mesh-phase1.md

@@ -0,0 +1,222 @@
+# Per-Cluster Mesh — Phase 1 plan
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+
+**Parent design:** [`2026-07-21-per-cluster-mesh-design.md`](2026-07-21-per-cluster-mesh-design.md).
+Phases 0a and 0b are done and merged; this is the next phase.
+
+**Goal:** Give `ClusterNode` the transport addresses central will need to reach each cluster
+directly, and cut `ConfigPublishCoordinator`'s last dependency on shared cluster membership by
+sourcing its expected-ack set from those rows.
+
+**Architecture:** Additive schema change plus one substitution inside the coordinator. Nothing else
+moves; the deploy channel stays on DistributedPubSub until Phase 2.
+
+**Status: DONE 2026-07-22** (branch `feat/mesh-phase1`). Live gate:
+[`2026-07-22-mesh-phase1-live-gate.md`](2026-07-22-mesh-phase1-live-gate.md) — **PASSED**, after gate
+step 4 found that the escape hatch this phase depends on did not exist. Three deviations from the
+plan below, all decided before implementing or forced by the gate:
+
+1. **Task 3's cluster-scope filtering was dropped** — there is no cluster-scoped deployment to filter
+   on. See the gate doc, step 2.
+2. **No per-node role column.** Every `ClusterNode` row is now *defined* to be a driver node.
+3. **Task 7 added:** `ClusterNode.MaintenanceMode`. `Enabled = 0` is rejected by
+   `DraftValidator.ValidateClusterTopology` on any 2-node pair, so it could never be the maintenance
+   hatch — which the plan called "the check that makes step 3 acceptable to ship".
+
+---
+
+## Why this is smaller than the design implies
+
+The design says "`ClusterNode` gains Akka + gRPC address columns; coordinator sources its
+expected-ack set from the DB", which reads as though the two are one change. They are not, and the
+second does not depend on the first.
+
+`ConfigPublishCoordinator.DiscoverDriverNodes` builds `NodeId.Parse($"{host}:{port}")` from
+`Cluster.State.Members`. `ClusterNode.NodeId` is **already in that identifier space** — the rig seeds
+`central-1:4053`, `site-a-1:4053` and so on (`docker-dev/seed/seed-clusters.sql:65`), and its comment
+says so explicitly, because `NodeDeploymentState.NodeId` is FK-bound to `ClusterNode.NodeId` and a
+mismatch throws FK 547 on deploy. The coordinator already writes `NodeDeploymentState` rows keyed by
+the membership-derived id and they already satisfy that FK — which is standing proof the two sets
+agree today.
+
+So the coordinator switch is a substitution inside one private method, and the address columns are
+purely forward-looking groundwork for Phase 2's ClusterClient and Phase 5's gRPC dial.
+
+## The decision this phase forces
+
+**Membership-derived and DB-derived expected-ack sets are not the same set, and the difference is a
+behaviour change on the deploy path.**
+
+| Situation | Today (membership) | After (DB rows) |
+|---|---|---|
+| Configured node is running | expected, acks, seals | same |
+| Configured node is **stopped** | not expected — deploy seals **successfully** without it | expected — deploy waits, then **fails at the apply deadline** |
+| Node running but not in `ClusterNode` | expected (and then FK 547 on the state row) | not expected; its ack is discarded |
+
+The middle row is the one that matters. Today a deployment can seal green while a node that should
+have received it was down and did not — the operator is told the fleet is deployed when it is not.
+Failing instead is arguably the correct behaviour and is the reason the design moves this to the DB
+(central cannot see cluster membership once the meshes split).
+
+But it is a behaviour change, and it needs an operator escape hatch or a node taken down for
+maintenance blocks every deployment. `ClusterNode.Enabled` already exists and is exactly that hatch,
+so the query filters on it: **`Enabled = 1` rows are expected to ack; disabled rows are not.**
+
+**Confirmed 2026-07-22 before executing Task 3** — DB-derived with the hatch. (The hatch turned out to be `MaintenanceMode`, not `Enabled`; see Task 7.) If the preference is to keep sealing green past a down
+node, the alternative is to intersect the DB set with current membership, which preserves today's
+behaviour but does not survive the mesh split — it would have to be revisited in Phase 2 anyway.
+
+---
+
+## Task 1: Address columns on `ClusterNode`
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** none (Task 2 migrates what this declares)
+
+**Files:**
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/ClusterNode.cs`
+- Modify: the `ClusterNode` entity configuration in `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/` (find the `IEntityTypeConfiguration<ClusterNode>` or the `OnModelCreating` block)
+- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/` (mirror an existing entity-mapping test)
+
+Add two nullable columns beside the existing `OpcUaPort` / `DashboardPort`:
+
+- `int AkkaPort` — default `4053`. The remoting port central will use to build the ClusterClient
+  contact point. Non-nullable with a default, because every node has one.
+- `int? GrpcPort` — nullable, no default. The Phase 5 telemetry-stream port. Nullable is deliberate:
+  it does not exist yet, and a non-null default would assert a port nothing listens on.
+
+Document on each property that these are **central's dial targets**, not the node's own binding
+configuration — the node binds from `Cluster:Port` in its own appsettings, and these columns
+duplicate that value for a reader that cannot see the node's config. That duplication is the reason
+Task 4 exists.
+
+**Step 1:** Write a mapping test asserting a round-tripped `ClusterNode` preserves both values and
+that `AkkaPort` defaults to 4053.
+**Step 2:** Run it; expect failure to compile.
+**Step 3:** Add the properties + mapping.
+**Step 4:** Run; expect pass.
+**Step 5:** Commit.
+
+## Task 2: EF migration
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+**Parallelizable with:** none
+
+**Files:**
+- Create: a new migration under the Configuration project's `Migrations/` folder
+
+```bash
+dotnet ef migrations add AddClusterNodeTransportPorts \
+  --project src/Core/ZB.MOM.WW.OtOpcUa.Configuration
+```
+
+Verify the generated `Up` adds `AkkaPort` with `defaultValue: 4053` and `GrpcPort` as nullable, and
+that `Down` drops both. **Check the migration is additive only** — an `AlterColumn` on anything else
+means the model had drifted from the last migration and that drift must be understood before this
+ships, not carried along by it.
+
+## Task 3: Coordinator sources the expected-ack set from `ClusterNode`
+
+**Classification:** high-risk — this is the deploy path
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs` (`DiscoverDriverNodes`, ~line 262, and the class doc comment at line 21 which states the membership rule)
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ConfigPublishCoordinatorTests.cs`
+
+Replace the `Cluster.State.Members` scan with a query over enabled `ClusterNode` rows. Keep
+`NodeIdComparer` — the case-insensitivity rationale in its comment applies unchanged, and SQL
+collation makes it more relevant, not less.
+
+Points to get right:
+
+- **Filter on `Enabled`.** See "The decision this phase forces".
+- **Scope to the deployment's cluster if the deployment is cluster-scoped.** `DeploymentArtifact.ResolveClusterScope`
+  already defines that scoping; a fleet-wide deployment expects every enabled node, a cluster-scoped
+  one expects only that cluster's. Getting this wrong makes every cluster-scoped deploy hang on nodes
+  that were never sent it. **Read `ResolveClusterScope` before writing the query.**
+- **The empty-set branch still matters.** `HandleDispatch` seals immediately when the set is empty,
+  and its log message says "no driver-role members in cluster" — reword it, because the reason is now
+  "no enabled ClusterNode rows", which is a configuration error rather than a topology observation.
+- The DB is read inside `HandleDispatch`, which already opens a context; reuse it rather than opening
+  a second.
+
+**Tests to add:** enabled-only filtering; cluster-scope filtering; empty set seals with the new
+reason; an ack from a node absent from `ClusterNode` is discarded. **Positive control on each:**
+delete the filter and confirm the test goes red — the existing suite passed throughout the
+membership era, so a test that also passes against the old derivation is testing nothing.
+
+## Task 4: Reconcile the duplicated port with the node's own config
+
+**Classification:** standard
+**Estimated implement time:** ~4 min
+**Parallelizable with:** none
+
+`ClusterNode.AkkaPort` and the node's own `Cluster:Port` are the same fact in two places, and nothing
+makes them agree. A node that binds 4054 while its row says 4053 is unreachable from central in Phase
+2, and the symptom will be a silent absence of acks rather than an error.
+
+Options, in preference order:
+
+1. **Have the node assert its row on startup** — on join, compare `Cluster:Port` /
+   `Cluster:PublicHostname` to its own `ClusterNode` row and log an Error (not a Warning) on
+   mismatch. Cheap, no schema change, no write path from driver nodes to the ConfigDb (which Phase 4
+   removes anyway — so put this check on an **admin** node, reading the membership it can already see).
+2. Deploy-time validation in `DraftValidator`, alongside the existing gates.
+
+Pick one and implement it. Do not skip this task: an unenforced duplicated address is the exact shape
+of the `Modbus`/`ModbusTcp` drift and the `TwinCat`/`Focas` drift already recorded in this repo.
+
+## Task 5: Rig seed + docs
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+
+**Files:**
+- Modify: `docker-dev/seed/seed-clusters.sql` — add `AkkaPort` to the six INSERTs (all 4053) so a
+  freshly-seeded rig matches a migrated one. Leave `GrpcPort` null.
+- Modify: `docs/Configuration.md` — document both columns.
+- Modify: `docs/plans/2026-07-21-per-cluster-mesh-design.md` — mark Phase 1 done in §7.
+
+## Task 6: Live gate
+
+**Classification:** high-risk
+**Estimated implement time:** n/a — operator-driven
+
+Offline tests cannot cover the substitution's real risk, which is that the DB set and the live set
+disagree on the rig. Run on `docker-dev`:
+
+1. Deploy with all six nodes up → seals green, six `NodeDeploymentState` rows, no FK 547.
+2. Deploy a **cluster-scoped** config → expected set is that cluster's nodes only.
+3. `docker stop` one node, deploy → **fails at the apply deadline** naming the missing node. This is
+   the new behaviour; confirm the failure is legible, because an operator will meet it.
+4. Set that node's `Enabled = 0`, deploy again → seals green without it. This proves the escape
+   hatch, and is the check that makes step 3 acceptable to ship.
+
+Record results in a gate doc beside this plan, per the Phase 1/Phase 2 precedent.
+
+## Task 7: `MaintenanceMode` — the escape hatch step 4 proved missing (ADDED 2026-07-22)
+
+**Classification:** high-risk — deploy path + schema
+**Why:** gate step 4 returned `422 ClusterEnabledNodeCountMismatch`. `Enabled = 0` on one node of a
+Warm/Hot pair fails `DraftValidator.ValidateClusterTopology` (enabled count must equal `NodeCount`),
+so the hatch Task 3 relies on cannot be used on any 2-node cluster — i.e. every cluster in the target
+topology. Task 3's behaviour change is only shippable with a hatch that works.
+
+**Decision:** split the meanings rather than weaken either rule.
+
+| Flag | Question | Enforced by |
+|---|---|---|
+| `Enabled` | part of the declared topology? | `DraftValidator` vs `ServerCluster.NodeCount` |
+| `MaintenanceMode` | expected to participate right now? | `ConfigPublishCoordinator`, `ClusterNodeAddressReconciler` |
+
+Rejected: counting configured rather than enabled nodes (removes the InvalidTopology guard);
+downgrading the rule to a warning (weakens a deploy gate for everyone); shipping without a hatch.
+
+**Done:** column + migration `AddClusterNodeMaintenanceMode`; both consumers filter on
+`Enabled && !MaintenanceMode`; deadline + reconciler messages name the right flag; docs; gate step 4
+re-run green.

docs/plans/2026-07-21-per-cluster-mesh-phase1.md.tasks.json

@@ -0,0 +1,76 @@
+{
+  "plan": "docs/plans/2026-07-21-per-cluster-mesh-phase1.md",
+  "branch": "feat/mesh-phase1",
+  "decisions": {
+    "seal-semantics": "DB-derived + Enabled hatch (confirmed 2026-07-22)",
+    "cluster-scope": "Dropped \u2014 no cluster-scoped deployment exists; expected set = all enabled ClusterNode rows",
+    "node-roles": "Every ClusterNode row is a driver node; documented + pinned by test, no Roles column",
+    "adminui-edit": "Deferred to Phase 2 (nothing reads the columns until then)",
+    "maintenance-hatch": "MaintenanceMode column, not Enabled=0 (blocked by DraftValidator on any 2-node pair)"
+  },
+  "tasks": [
+    {
+      "id": 1,
+      "subject": "Address columns on ClusterNode",
+      "status": "completed",
+      "commit": "2bbb0271",
+      "blockedBy": []
+    },
+    {
+      "id": 2,
+      "subject": "EF migration AddClusterNodeTransportPorts",
+      "status": "completed",
+      "commit": "da74ebd6",
+      "blockedBy": [
+        1
+      ]
+    },
+    {
+      "id": 3,
+      "subject": "Coordinator sources expected-ack set from ClusterNode",
+      "status": "completed",
+      "commit": "d88e2455",
+      "blockedBy": [
+        2
+      ]
+    },
+    {
+      "id": 4,
+      "subject": "Reconcile duplicated AkkaPort with the node's own Cluster:Port",
+      "status": "completed",
+      "blockedBy": [
+        2
+      ],
+      "commit": "90c1302f"
+    },
+    {
+      "id": 5,
+      "subject": "Rig seed + docs",
+      "status": "completed",
+      "blockedBy": [
+        3,
+        4
+      ],
+      "commit": "57e1d017"
+    },
+    {
+      "id": 6,
+      "subject": "Live gate on docker-dev",
+      "status": "completed",
+      "blockedBy": [
+        5
+      ],
+      "result": "live-gate PASSED (2026-07-22-mesh-phase1-live-gate.md)"
+    },
+    {
+      "id": 7,
+      "subject": "MaintenanceMode \u2014 escape hatch the live gate proved missing",
+      "status": "completed",
+      "commit": "ee69caa2",
+      "blockedBy": [
+        6
+      ],
+      "origin": "added from gate step 4 failure"
+    }
+  ]
+}

docs/plans/2026-07-22-mesh-phase1-live-gate.md

@@ -0,0 +1,134 @@
+# Per-cluster mesh Phase 1 — live gate record
+
+**Date:** 2026-07-22 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase1`
+**Plan:** [`2026-07-21-per-cluster-mesh-phase1.md`](2026-07-21-per-cluster-mesh-phase1.md)
+
+**Result: PASSED**, after the gate found a blocker that changed the design (step 4).
+
+Offline tests cannot cover the substitution's real risk — that the DB set and the live set disagree on
+a running fleet. This is that check.
+
+## Setup
+
+Migration applied by the rig's `migrator` service against the shared `OtOpcUa` database, then all six
+host nodes restarted on the rebuilt image. **All six pre-existing `ClusterNode` rows inherited
+`AkkaPort = 4053` from the migration's `DEFAULT 4053`** — the row-level evidence that the default does
+its job for rows predating the column:
+
+```
+central-1:4053|central-1|4053|NULL|1
+central-2:4053|central-2|4053|NULL|1
+site-a-1:4053 |site-a-1 |4053|NULL|1
+site-a-2:4053 |site-a-2 |4053|NULL|1
+site-b-1:4053 |site-b-1 |4053|NULL|1
+site-b-2:4053 |site-b-2 |4053|NULL|1
+   NodeId     |  Host   |Akka|Grpc|En
+```
+
+## Step 1 — deploy with all six nodes up → **PASS**
+
+`POST /api/deployments` → `202 Accepted`, deployment `38f83b37`.
+
+`Deployment.Status = 2` (Sealed). Six `NodeDeploymentState` rows, every one `Status = 1` (Applied).
+**No FK 547** — the DB-derived NodeIds satisfy the `NodeDeploymentState → ClusterNode` foreign key,
+which is the standing proof that the DB set and the membership set agree on this rig.
+
+## Step 2 — *dropped, not skipped*
+
+The plan called for a cluster-scoped deploy. **There is no such thing.** `Deployment` has no
+`ClusterId`, `ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and
+`DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. Removed
+from the plan rather than faked green.
+
+## Step 3 — deploy with one node stopped → **PASS** (the new behaviour)
+
+`docker stop otopcua-dev-site-b-2-1`, then deploy `37d72301`.
+
+`Deployment.Status = 4` (TimedOut) after the 2-minute apply deadline. Five nodes `Applied`,
+`site-b-2:4053` still `Applying`. Under the membership rule this deployment would have **sealed
+green** with five nodes and never mentioned the sixth.
+
+The operator-facing log, which the plan required to be legible:
+
+```
+[12:52:02 WRN] Deployment 37d723018e374c0eb04c7b6beccc3b3e timed out after 00:02:00 (5/6 acks
+landed). No ack from: site-b-2:4053. Each is an enabled ClusterNode row — start the node, or set
+ClusterNode.Enabled = 0 if it is down for maintenance, then redeploy
+```
+
+(The remedy in that message was corrected to `MaintenanceMode = 1` — see step 4.)
+
+## Step 4 — the escape hatch → **FAILED, then fixed and re-run → PASS**
+
+### First attempt: the hatch did not exist
+
+`UPDATE ClusterNode SET Enabled = 0 WHERE NodeId = 'site-b-2:4053'`, then deploy:
+
+```
+HTTP 422
+[ClusterEnabledNodeCountMismatch] Cluster 'SITE-B' declares NodeCount=2 but has 1 Enabled nodes.
+Toggle the missing node(s) back on or change RedundancyMode/NodeCount to match.
+```
+
+`DraftValidator.ValidateClusterTopology` rejects the deployment **before it dispatches**. So
+`Enabled = 0` cannot be used on a node of a 2-node cluster — and every cluster on this rig, and every
+cluster in the program plan's target topology, is a 2-node pair.
+
+**This was a blocker, not a wrinkle.** The plan called step 4 *"the check that makes step 3
+acceptable to ship"*. Without it, a node down for maintenance blocks every deployment to its cluster,
+with no remedy short of editing `NodeCount` / `RedundancyMode` — which changes the runtime redundancy
+posture to work around a deploy gate.
+
+The two rules were each reasonable and contradictory together: the validator reads `Enabled` as *"part
+of the declared topology"*, Phase 1 additionally read it as *"expect an ack"*.
+
+### Fix (Task 7): split the meanings
+
+New `ClusterNode.MaintenanceMode bit NOT NULL DEFAULT 0`. `Enabled` keeps its topology meaning and the
+validator is untouched; the coordinator's expected-ack set and the address reconciler both filter on
+`Enabled && !MaintenanceMode`.
+
+### Re-run: PASS
+
+`docker stop` `site-b-2` + `MaintenanceMode = 1`, then deploy `82c5e678`:
+
+- `202 Accepted` — the topology gate passes, because the node is still `Enabled` (`1|1`).
+- `Deployment.Status = 2` (Sealed).
+- **Five** `NodeDeploymentState` rows, all `Applied`. `site-b-2:4053` has no row at all — excluded
+  from the expected set rather than waited for and forgiven.
+
+## Task 4 (reconciler) — live behaviour, not a planned gate step
+
+Observed unprompted during the rolling restart, which is better evidence than a staged check:
+
+```
+[12:44:49 WRN] ClusterNode address check [EnabledRowNotInCluster] site-a-1:4053: ...
+[12:44:49 WRN] ... site-a-2:4053 ... site-b-1:4053 ... site-b-2:4053
+[12:44:56 INF] ClusterNode addresses reconcile with cluster membership (6 driver members)
+```
+
+It warned per row while the site nodes were still starting, then converged to a single clean line —
+and **logged nothing further**, confirming the change-detection guard suppresses repeats on the
+5-minute sweep.
+
+Maintenance interaction, from the step 4 re-run:
+
+```
+[13:06:02 WRN] ... site-b-2:4053 ...            ← node dropped, flag not yet set
+[13:06:05 INF] ... reconcile with cluster membership (5 driver members)   ← flag set: silent
+```
+
+The transient warning is honest — for those three seconds the row genuinely was enabled, present in
+the expected-ack set, and absent from the cluster.
+
+## What this gate did not cover
+
+- **Phase 2's premise.** `AkkaPort` / `GrpcPort` are groundwork; nothing dials them yet, so "the value
+  is correct" is only checked by the reconciler comparing it to gossip. The first real exercise is
+  Phase 2.
+- **The reconciler under split meshes.** On the current single mesh an admin node sees every driver
+  member. After Phase 6 it will not, and every site row would report `EnabledRowNotInCluster`
+  permanently. Recorded as a known limitation on the class and in the design doc.
+- **`Enabled = 0` on a single-node (`RedundancyMode.None`) cluster.** Untested — the rig has no such
+  cluster. It would fail the same topology rule (0 enabled vs `NodeCount = 1`), so `MaintenanceMode`
+  is the hatch there too.

docs/plans/2026-07-22-per-cluster-mesh-program.md

@@ -0,0 +1,208 @@
+# Per-Cluster Mesh Program — Align OtOpcUa's Akka Topology with ScadaBridge
+
+> **For Claude:** This is a PROGRAM plan (phase roadmap + gates), not a bite-sized task plan.
+> The authoritative design is `docs/plans/2026-07-21-per-cluster-mesh-design.md` (decisions settled
+> 2026-07-21; Phases 0a/0b DONE). Per that design's own instruction, **each phase gets its own
+> detailed plan, written when the phase starts** (superpowers writing-plans → executing-plans),
+> so plans are authored against current code, not against a forecast. This document sequences the
+> phases, fixes the deployment topology (co-location with ScadaBridge — NEW constraint 2026-07-22),
+> and defines each phase's entry/exit gates.
+
+**Goal:** OtOpcUa runs the same Akka.NET mesh shape as ScadaBridge — **one Akka mesh per
+application `Cluster`, two nodes max**, central ↔ cluster joined by explicit transports
+(ClusterClient + gRPC + fetch), driver nodes holding **no ConfigDb connection** — so that every
+Primary-gated decision and every gated resource share one pair-local scope, and both products
+present one operational model on the shared site hardware.
+
+**Why now (2026-07-22):** OtOpcUa cluster pairs will run **on the same two Windows VMs as the
+ScadaBridge site nodes**. That deployment makes the current single-fleet-mesh design actively
+wrong for sites (a site's OtOpcUa nodes would gossip across the WAN to central and elect ONE
+Primary fleet-wide), and makes the ScadaBridge shape the obviously correct one: each site's two
+VMs host two independent, identically-postured 2-node clusters (one per product), surviving alone
+on the site LAN. It also means driver nodes must not depend on reaching central SQL — **sites
+have no SQL Server**, so §6.1's "driver nodes never connect to the ConfigDb" stops being an
+architectural preference and becomes a deployment requirement.
+
+---
+
+## Deployment topology (the co-location constraint, NEW)
+
+Each **site**: 2 Windows VMs. Each VM runs one ScadaBridge site node AND one OtOpcUa driver node.
+Two independent 2-node Akka clusters per site — they share hardware, never a mesh. **Central**:
+2 Windows VMs, each running a ScadaBridge central node and an OtOpcUa central node
+(`admin,driver`), again as separate pairs.
+
+**Per-VM port allocation (no collisions — verify against actual deployment configs in Phase 6):**
+
+| Port | Owner | Purpose |
+|---|---|---|
+| 8081 / 8082 | ScadaBridge | Akka remoting (central / site) |
+| 8083 | ScadaBridge site | gRPC h2c (streams + LocalDb sync) |
+| 8084 | ScadaBridge site | metrics |
+| 5000 (+Traefik) | ScadaBridge central | UI + Inbound API |
+| **4053** | OtOpcUa | Akka remoting |
+| **4840** | OtOpcUa | OPC UA endpoint |
+| OtOpcUa AdminUI port | OtOpcUa central (admin) | AdminUI (driver-only site nodes host no UI) |
+| OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) |
+
+**Aligned HA posture (both products, per VM — already true or landing via the selfform plan):**
+auto-down downing (15 s window), oldest-Up active/primary election, self-first seed ordering
+(the `SelfFormAfter` self-form fallback that briefly stood in for it was retired 2026-07-22),
+termination-watchdog → process exit → `sc.exe failure` restart recovery.
+One failover story for operators regardless of product.
+
+**Prerequisite ordering:** the fallback/manual-failover plan
+(`docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`) executes **before** this
+program — it is small, independent, and Phase 6 depends on its semantics (under per-pair meshes
+every node lists itself + partner as seeds, so the fallback covers both nodes of every pair;
+the site-node island guard then simply never triggers).
+
+---
+
+## What changes, in one table (from the design doc)
+
+| Aspect | Today (single fleet mesh) | Target (ScadaBridge shape) |
+|---|---|---|
+| Mesh | one gossip ring, all nodes, seeded by central-1 | one 2-node mesh per application `Cluster`; same ActorSystem name; separation by seed-node partitioning |
+| Roles | `admin` / `driver` fleet-wide | `driver` + cluster-specific `cluster-{ClusterId}`; singletons scoped to the cluster role |
+| Primary election | mesh-wide oldest Up driver (Phase 0b) | pair-local by construction — same rule, correct scope |
+| Command/control | 9 DPS topics + singleton over gossip | ClusterClient (one receptionist actor per side); central discovers nodes from `ClusterNode` rows |
+| Deploy | notify (DPS) + node fetches from ConfigDb | notify via ClusterClient; artifact fetched **from central**, cached in LocalDb |
+| Driver ConfigDb connection | direct EF connection to central SQL | **none** — LocalDb is the steady-state config store |
+| Live telemetry | 7 observability DPS topics | one gRPC stream contract (`oneof` event), central dials each cluster node |
+| Rig | six-node single mesh (`docker-dev`) | per-cluster meshes; rig models the real topology |
+
+---
+
+## Phases
+
+Each phase below is one row of design-doc §7, expanded with entry/exit gates. **Execution recipe
+per phase:** (1) invoke writing-plans in this repo to produce
+`docs/plans/2026-07-2X-mesh-phase-N-<name>.md` from the scope notes here + the design doc §
+references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit
+gate; (4) update this file's status column and the design doc's §7 table.
+
+### Phase 1 — `ClusterNode` address columns + DB-sourced ack set — **DONE 2026-07-22**
+Shipped per `2026-07-21-per-cluster-mesh-phase1.md`; see that plan's Task 6 gate record and the
+design doc's "Phase 1 as shipped" note for the two scope deviations (no cluster-scope filtering — no
+such deployment exists; no per-node role column) and the one addition (`ClusterNodeAddressReconciler`).
+**AdminUI node edit was deferred to Phase 2** — nothing reads the columns until then, and the
+migration default plus the rig seed cover every node today.
+
+**Scope:** `ClusterNode` gains Akka + gRPC address columns (mirroring ScadaBridge's `Site`
+entity `NodeAAddress`/`GrpcNodeAAddress` pattern, but per-node rows); EF migration; AdminUI node
+edit surfaces the fields; `ConfigPublishCoordinator` derives its expected-ack set from
+`ClusterNode` rows instead of `Akka.Cluster.State.Members` filtered by role (design §3 fact 3 —
+this removes the coordinator's one genuinely mesh-bound dependency).
+**Independent of the split:** yes — safe on the current mesh.
+**Exit gate:** deploy on the unchanged docker-dev rig completes with the coordinator's expected-ack
+set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node
+missing; a node up but row absent → its ack is not expected).
+
+### Phase 2 — Comm actors + ClusterClient transport
+**Scope:** one receptionist-registered actor per side (`/user/central-communication`,
+`/user/cluster-communication`), registered **per node, not as a singleton** (contact rotation);
+ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge
+idioms copied verbatim: sender-preserving `Tell(new ClusterClient.Send(...), Sender)` Ask relay,
+typed-failure reply for every unhandled message, no central buffering toward unreachable clusters
+(drop + warn), central discovers contacts from Phase 1's `ClusterNode` rows (60 s refresh +
+admin-change refresh), clusters know central from appsettings (static, restart to change).
+**Frame-size guard:** anything carrying payload sets both the frame limit and
+`log-frame-size-exceeding` (design §8) — the deploy path stays payload-free by design.
+**Exit gate:** on the still-single-mesh rig, deploy notify + acks and AdminUI Reconnect/Restart
+flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly
+against a stopped node.
+
+### Phase 3 — Config fetch-and-cache from central
+**Scope:** the deploy artifact is served **by central** (transport per its own phase plan — the
+ScadaBridge analogue is token-gated HTTP; decide HTTP vs a gRPC fetch RPC when planning) and
+cached in LocalDb; driver nodes read config exclusively from LocalDb (boot-from-cache becomes the
+normal path — design §6.1); chunking, SHA-256 verify, newest-2 retention, pair replication carry
+over unchanged. **This phase changes a running data path — it gets its own live gate** (design §7
+note): deploy lands on a driver pair with central SQL stopped mid-fetch → retry lands; #485
+last-known-good semantics re-proven on the new path.
+**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable
+central boots into the current config; with central down it boots last-known-good.
+
+### Phase 4 — Cut the driver-side ConfigDb connection
+**Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local
+state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor`
+question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering,
+so define the replacement health input (candidate: central-reachability via the Phase 2/3
+transports) — this is a client-visible ServiceLevel semantics change and must be documented in
+`docs/Redundancy.md` + the interop playbook; audit `OpcUaPublishActor`'s ConfigDb use (TBD in the
+design) and re-source it; registration cleanup in `ServiceCollectionExtensions`; driver-role
+`Program.cs` branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only
+`AddConfigurationDatabase`).
+**Exit gate:** its own live gate — a driver pair runs a full deploy + alarm + historian cycle with
+**no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can
+resolve the ConfigDb context.
+
+### Phase 5 — gRPC stream contract for live telemetry
+**Scope:** one server-streaming contract carrying a `oneof` event, **cluster nodes host the gRPC
+server, central dials in** (the inverted direction is the load-bearing ScadaBridge finding —
+design §2); migrate the seven observability topics (`alerts`, `driver-health`,
+`driver-resilience-status`, `fleet-status`, `script-logs`, plus redundancy-state distribution and
+deployment-acks if Phase 2 left them on DPS); additive-only field evolution, contract locked by
+test; per-panel reconnect story for the AdminUI (design §8 — losing gossip loses free fleet
+observability).
+**Exit gate:** all AdminUI live panels green against a pair with DPS telemetry topics deleted;
+kill-and-reconnect of the central dialer recovers every stream.
+
+### Phase 6 — Mesh partition + co-location topology
+**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
+`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
+`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
+`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
+`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
+§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
+`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed
+of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
+central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
+including the co-location port table above (both products' compose files on shared per-site
+networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
+election is pair-local now) and the fallback's site-node island-guard docs note (moot — every
+node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair.
+**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair
+(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**);
+secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its
+topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot
+cross separate meshes).
+
+### Phase 7 — Failover drill + live gates
+**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
+directions; **close the two outstanding live gates**: (a) auto-down 1-vs-1 crash-the-oldest
+(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) self-first
+lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair;
+operator runbook for the co-located site (one page covering both products' failover on the same
+two VMs).
+**Exit gate:** drill green on every pair type (central, site); runbook merged; design doc §7
+table fully marked DONE. (Live gate (b) is now the **self-first cold-start-alone** drill, not
+`SelfFormAfter` — the watchdog it named was retired 2026-07-22.)
+
+---
+
+## Risks carried from the design (§8, unchanged — re-read before each phase plan)
+
+LocalDb becomes load-bearing for config (blast radius of the #485 class rises);
+`DbHealthProbeActor` feeds a client-visible value; the rig models the doomed topology until
+Phase 6; losing gossip loses free observability (7 panels); 128 KB ClusterClient frame drop is
+silent unless both knobs are set; never stack app-level LWW on LocalDb's HLC. **New (this
+program):** co-located VMs mean a VM loss now takes out one node of BOTH products at once — the
+drill in Phase 7 must include the shared-VM failure (both products fail over together), and
+resource sizing on the site VMs should be checked once both products run the full stack.
+
+## Tracking
+
+| Phase | Status |
+|---|---|
+| 0a downing strategy | DONE 2026-07-21 (live gate → Phase 7) |
+| 0b oldest-Up election | DONE 2026-07-21 |
+| Prereq: selfform-fallback + manual-failover plan | plan written 2026-07-22, not executed |
+| 1 ClusterNode columns + DB ack set | **detailed plan ALREADY WRITTEN** (`2026-07-21-per-cluster-mesh-phase1.md`, from the design session — do NOT write a new one), not executed. ⚠ Carries a decision gate: DB-derived expected-ack set changes deploy-seal semantics (a stopped-but-enabled node now FAILS the deploy at the apply deadline instead of sealing green without it; escape hatch = `ClusterNode.Enabled=0`) — confirm with the owner before its Task 3. |
+| 2 ClusterClient transport | not started |
+| 3 fetch-and-cache | not started |
+| 4 cut driver ConfigDb | not started |
+| 5 gRPC telemetry | not started |
+| 6 mesh partition + co-location | not started |
+| 7 drill + live gates | not started |

docs/plans/2026-07-22-per-cluster-mesh-program.md.tasks.json

@@ -0,0 +1,15 @@
+{
+  "planPath": "docs/plans/2026-07-22-per-cluster-mesh-program.md",
+  "note": "PROGRAM plan: each task = write that phase's detailed plan (writing-plans), execute it (executing-plans), run its exit gate, update the tracking tables. Prereq task 0 is the separate selfform-fallback plan in this repo.",
+  "tasks": [
+    {"id": 0, "subject": "Prereq: execute 2026-07-22-selfform-fallback-and-manual-failover.md (7 tasks)", "status": "pending"},
+    {"id": 1, "subject": "Phase 1: ClusterNode Akka+gRPC address columns; ConfigPublishCoordinator ack set from DB", "status": "pending"},
+    {"id": 2, "subject": "Phase 2: comm actors + receptionist + ClusterClient transport (deploy notify/acks, driver-control)", "status": "pending", "blockedBy": [1]},
+    {"id": 3, "subject": "Phase 3: config fetch-and-cache from central; LocalDb steady-state config store (live gate)", "status": "pending", "blockedBy": [2]},
+    {"id": 4, "subject": "Phase 4: cut driver-side ConfigDb — EfAlarmConditionStateStore to LocalDb, DbHealthProbeActor ServiceLevel input, OpcUaPublishActor audit (live gate)", "status": "pending", "blockedBy": [3]},
+    {"id": 5, "subject": "Phase 5: gRPC oneof stream contract; migrate 7 observability topics; AdminUI reconnect story", "status": "pending", "blockedBy": [2]},
+    {"id": 6, "subject": "Phase 6: mesh partition — per-pair seeds, cluster-scoped roles/singletons, co-location rig rewrite, secrets replication re-scope", "status": "pending", "blockedBy": [0, 4, 5]},
+    {"id": 7, "subject": "Phase 7: failover drill per pair (incl. shared-VM failure) + close auto-down 1v1 and self-first cold-start-alone live gates + operator runbook", "status": "pending", "blockedBy": [6]}
+  ],
+  "lastUpdated": "2026-07-22T00:00:00Z"
+}

docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md

@@ -0,0 +1,405 @@
+# OtOpcUa: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan
+
+> ## ⚠ SUPERSEDED IN PART (2026-07-22, same day)
+>
+> **Half (1) of this plan — the `Cluster:SelfFormAfter` self-form watchdog — was retired within hours
+> of landing.** `ClusterBootstrapFallback`, the `SelfFormAfter` option and `SelfFormBootstrapTests`
+> are **deleted**; the cold-start-alone goal is now met by **self-first seed ordering** (each seed
+> node lists ITSELF as `seed-nodes[0]`), enforced at boot by `AkkaClusterOptionsValidator` and pinned
+> by `SelfFirstSeedBootstrapTests`. Reason: a timer outside Akka's join handshake cannot distinguish
+> "no seed answered" from "a seed answered and the join is in flight", and `Cluster.Join(SelfAddress)`
+> is not ignored mid-handshake — it wins. Task 8 below caught exactly that live and patched the one
+> observed shape with a TCP reachability guard; the race remained. Mesh-program Phase 6 had this
+> convergence queued and it was pulled forward. See `docs/Redundancy.md` §"Bootstrap: self-first seed
+> ordering" and ScadaBridge `4a6341d8`.
+>
+> **Half (2) — manual failover — is unaffected and still current.** Tasks below are kept verbatim as
+> the execution record, including the Task 8 live-gate evidence, which is the whole reason the
+> watchdog was rejected. Do not execute the Task 1/2/3 code as written.
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+>
+> Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The ScadaBridge half lives in `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution.
+>
+> **⚠ This plan is expected to GROW** — see "Reserved for additions" at the end. Append new tasks there (and to the `.tasks.json`) rather than renumbering existing ones.
+
+**Goal:** (1) Either node of a 2-node OtOpcUa pair can cold-start alone and become operational, unattended — a configurable, fast (default 10 s) InitJoin timeout falls back to self-forming a cluster, with a hard guard so site nodes seeded only by central can never island themselves. (2) An admin-only "Trigger failover" control on the Cluster Redundancy page performs a graceful, audited swap of the driver Primary.
+
+**Architecture:** New `AkkaClusterOptions.SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables, bound from the `Cluster` section) arms `ClusterBootstrapFallback` via an Akka.Hosting startup task inside `WithOtOpcUaClusterBootstrap` — so production AND the Cluster.Tests hosts get it identically. Wait for membership via `RegisterOnMemberUp`; on expiry, `Cluster.Join(SelfAddress)` — only if this node's own address is in its own `SeedNodes`. Manual failover = graceful `Cluster.Leave(<oldest Up driver member>)` via a new `IManualFailoverService` in ControlPlane, surfaced on `/clusters/{id}/redundancy`.
+
+**Tech Stack:** .NET 10, Akka.NET 1.5.62, Akka.Hosting 1.5.62 (`AddStartup`), Blazor Server (AdminUI, InteractiveServer), bUnit, xunit. No new packages.
+
+**Branch:** `feat/selfform-fallback` off `master`.
+
+---
+
+## Design essentials (from the shared design doc)
+
+**The defect:** Akka only lets the FIRST listed seed self-join to form a *new* cluster; every other node loops on `InitJoin` forever — "auto-down removes the crash outage, not this one" (`docs/Redundancy.md:244-247`). A lone cold-starting `central-2` (or a pair node under the future per-cluster mesh) never comes Up.
+
+**Behavior spec:**
+
+| Scenario | Behavior with fallback |
+|---|---|
+| Peer alive (any boot order) | Normal seed join in ms — fallback never fires |
+| Lone cold-start, self IS in own `SeedNodes` | After `SelfFormAfter`: warn log + `Cluster.Join(SelfAddress)` → Up alone (`min-nr-of-members=1`) |
+| Lone cold-start, self NOT in own `SeedNodes` | Fallback inert (info log). **This is the docker-dev site-node topology** — site-a/b nodes list only `central-1`; self-forming there would island them permanently, so they must wait. |
+| Peer boots after survivor self-formed | Its InitJoin is answered → joins as youngest. No island. |
+| Both pair nodes cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) |
+| `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior |
+| ~~Window expires mid-join-handshake~~ | ~~Benign: Akka ignores `Join` once joined~~ **FALSE — proven live (Task 8): `Join(self)` during an in-flight join abandons it and islands the node. Fixed by the TCP reachability guard: never self-form while any seed peer accepts a connection; wait a window and re-arm. ScadaBridge rejected the watchdog entirely for self-first seed ordering; OtOpcUa converges at mesh-program Phase 6.** |
+
+**Manual failover rules:** graceful `Leave` of the **oldest Up `driver` member** (identical query to `RedundancyStateActor.SelectDriverPrimary` — `Member.AgeOrdering`, `Leaving` excluded via `Status == Up`), never `Down`. The leaving node hands over via the cluster-leave phases, `CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it, and it rejoins as youngest → the survivor is Primary (ServiceLevel 250 moves; OPC UA clients re-select). Admin-only; peer guard (<2 Up driver members → disabled); confirmation dialog; audited via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam before the Leave. **Mesh-scope caveat:** until the per-cluster mesh lands, the election is mesh-wide — the button acts on THE Primary of the whole Akka cluster, not per-`ServerCluster` row; the UI must say so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md:84-99`).
+
+**Multi-node TestKit:** assessed, NOT used — everything is deterministic in-process via real hosts through `WithOtOpcUaClusterBootstrap` (the `SplitBrainResolverActivationTests` pattern), and the Akka TestKit family is xunit-v2-only while this repo's integration tree is xunit.v3. Full verdict in the shared design doc.
+
+---
+
+### Task 1 (B1): `SelfFormAfter` on `AkkaClusterOptions`
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+**Parallelizable with:** none (first task)
+
+**Files:**
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs`
+
+**Step 1: Branch**
+
+```bash
+cd ~/Desktop/OtOpcUa && git checkout master && git checkout -b feat/selfform-fallback
+```
+
+**Step 2: Add the property** (after `SplitBrainResolverStrategy`, mirroring its XML-doc depth):
+
+```csharp
+    /// <summary>
+    /// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
+    /// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
+    /// while its peer is down loops on InitJoin forever ("auto-down removes the crash outage,
+    /// not this one" — docs/Redundancy.md). When this node has waited longer than this window
+    /// without cluster membership it joins itself — but ONLY if its own address appears in
+    /// <see cref="SeedNodes"/>; a non-seed node (e.g. a site node whose only seed is central-1)
+    /// stays waiting, because self-forming there creates a permanent island. Default 10s
+    /// (same-datacenter pair; a live peer answers InitJoin in milliseconds). <c>null</c> or a
+    /// non-positive value disables the fallback. Accepted trade: both pair nodes cold-starting
+    /// inside the window while mutually unreachable form two clusters — the same dual-active
+    /// class the auto-down strategy already accepts, same recovery (restart one side).
+    /// </summary>
+    public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);
+```
+
+**Step 3:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster` — 0 warnings. Commit:
+
+```bash
+git add -A && git commit -m "feat(cluster): SelfFormAfter option (bootstrap self-form fallback window)"
+```
+
+(The default-value pin test lands with Task 3's test file — the property is inert until Task 2.)
+
+---
+
+### Task 2 (B2): `ClusterBootstrapFallback` + `AddStartup` wiring
+
+**Classification:** high-risk (cluster formation behavior)
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterBootstrapFallback.cs`
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (`WithOtOpcUaClusterBootstrap`, after the `AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend)` call at line 93)
+
+**Step 1: Implement the fallback class** (deliberately duplicated from ScadaBridge's, like the termination watchdogs — see that repo's `ClusterBootstrapFallback.cs` for the full doc comment to adapt):
+
+```csharp
+using Akka.Actor;
+using Microsoft.Extensions.Logging;
+
+namespace ZB.MOM.WW.OtOpcUa.Cluster;
+
+/// <summary>
+/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1). Akka only
+/// lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin forever —
+/// "auto-down removes the crash outage, not this one" (docs/Redundancy.md). Waits
+/// <see cref="AkkaClusterOptions.SelfFormAfter"/> for membership, then joins itself.
+/// <para><b>Island safety:</b> fires ONLY when this node's own address is in its own
+/// <see cref="AkkaClusterOptions.SeedNodes"/> — a site node seeded only by central-1 must wait,
+/// not island. Sequential recovery is island-free: Akka's join protocol prefers an existing
+/// cluster (a booting node only self-forms when NO seed answers InitJoin).</para>
+/// <para><b>Benign race:</b> Akka ignores <c>Join</c> once the node has joined a cluster this
+/// incarnation. Residual risk: both pair nodes cold-starting inside the window while mutually
+/// unreachable self-form separately — the same dual-active class auto-down accepts.</para>
+/// </summary>
+public static class ClusterBootstrapFallback
+{
+    public static void Arm(ActorSystem system, AkkaClusterOptions options, ILogger logger)
+    {
+        if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero)
+        {
+            logger.LogInformation(
+                "Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting "
+                + "while its peer is down will wait on InitJoin indefinitely.");
+            return;
+        }
+
+        var cluster = Akka.Cluster.Cluster.Get(system);
+        var self = cluster.SelfAddress;
+        var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self));
+        if (!isSeed)
+        {
+            logger.LogInformation(
+                "Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list "
+                + "[{Seeds}] — self-forming here would island it from the real seeds.",
+                self, string.Join(", ", options.SeedNodes));
+            return;
+        }
+
+        var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
+        cluster.RegisterOnMemberUp(() => joined.TrySetResult());
+
+        _ = Task.Run(async () =>
+        {
+            var winner = await Task.WhenAny(joined.Task, Task.Delay(window));
+            if (winner == joined.Task || system.WhenTerminated.IsCompleted)
+                return;
+            logger.LogWarning(
+                "No cluster membership after {Window} — no seed answered InitJoin (peer down at boot). "
+                + "Self-forming a cluster at {Self} so this node becomes operational; if the peer was "
+                + "merely partitioned (not dead), the pair is now dual-active — restart one side after "
+                + "the partition heals (accepted availability-first trade, decision 2026-07-22).",
+                window, self);
+            cluster.Join(self);
+        });
+    }
+
+    private static bool TryParseAddress(string seed, out Address address)
+    {
+        try { address = Address.Parse(seed); return true; }
+        catch { address = default!; return false; }
+    }
+}
+```
+
+(If the Cluster project lacks a `Microsoft.Extensions.Logging.Abstractions` reference, add the `PackageReference` — check `Directory.Packages.props` for the pinned version first.)
+
+**Step 2: Wire via Akka.Hosting startup hook** — in `WithOtOpcUaClusterBootstrap`, after the downing HOCON line (~93):
+
+```csharp
+        // InitJoin self-form fallback (decision 2026-07-22): registered as an Akka.Hosting
+        // startup task so every host built through this bootstrap — production AND the test
+        // hosts in Cluster.Tests — arms it identically. Guarded inside Arm: disabled when
+        // SelfFormAfter is unset; inert when this node is not in its own seed list (site nodes
+        // seeded only by central-1 must wait, not island).
+        var fallbackLogger = (serviceProvider.GetService<ILoggerFactory>()
+                              ?? Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance)
+            .CreateLogger("ZB.MOM.WW.OtOpcUa.Cluster.ClusterBootstrapFallback");
+        builder.AddStartup((system, _) =>
+        {
+            ClusterBootstrapFallback.Arm(system, options, fallbackLogger);
+            return Task.CompletedTask;
+        });
+```
+
+(Akka.Hosting 1.5.62 has `AddStartup`. If the overload signature differs, alternative: register an `IHostedService` in `AddOtOpcUaCluster` mirroring `ActorSystemTerminationWatchdog`'s lazy-accessor pattern — but prefer `AddStartup` so test hosts arm it for free.)
+
+**Step 3:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster` — 0 warnings.
+
+**Step 4: Commit**
+
+```bash
+git add -A && git commit -m "feat(cluster): InitJoin self-form fallback armed via Akka.Hosting startup task"
+```
+
+---
+
+### Task 3 (B3): `SelfFormBootstrapTests` — 4 scenarios through the production bootstrap
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 4
+
+**Files:**
+- Create: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SelfFormBootstrapTests.cs`
+
+**Step 1: Write the tests** — follow `SplitBrainResolverActivationTests`' pattern (real `Host.CreateDefaultBuilder()` + `AddAkka` + `WithOtOpcUaClusterBootstrap`; that file's remarks explain why testing through the production bootstrap is mandatory — a test that re-creates the wiring pins the test's wiring, not the shipped one). Helper + scenarios:
+
+```csharp
+public sealed class SelfFormBootstrapTests
+{
+    private static int FreePort()
+    {
+        var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
+        l.Start();
+        var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port;
+        l.Stop();
+        return p;
+    }
+
+    private static async Task<IHost> StartNodeAsync(
+        string systemName, int selfPort, int peerPort, TimeSpan? selfFormAfter, bool selfInSeeds = true)
+    {
+        var options = new AkkaClusterOptions
+        {
+            SystemName = systemName, Hostname = "127.0.0.1", PublicHostname = "127.0.0.1",
+            Port = selfPort, Roles = new[] { "admin", "driver" },
+            SelfFormAfter = selfFormAfter,
+            SeedNodes = selfInSeeds
+                ? new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}",
+                          $"akka.tcp://{systemName}@127.0.0.1:{selfPort}" }   // self SECOND — only the fallback can form it
+                : new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}" },
+        };
+        var builder = Host.CreateDefaultBuilder();
+        builder.ConfigureServices(services =>
+        {
+            services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
+            services.AddAkka(systemName, (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
+        });
+        var host = builder.Build();
+        await host.StartAsync();
+        return host;
+    }
+
+    // 1) SelfFormAfter_defaults_to_ten_seconds
+    //      new AkkaClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10));
+    // 2) Lone_non_first_seed_self_forms_after_the_window
+    //      dead peer port, window 2s → poll Cluster.Get(system).State until 1 Up member ≤ 20s.
+    // 3) Disabled_fallback_keeps_waiting  (window null → State.Members empty after 6s;
+    //      POSITIVE CONTROL: cluster.Join(cluster.SelfAddress) → 1 Up ≤ 20s — proves the node
+    //      was formable and only the fallback was absent.)
+    // 4) Site_node_seeded_only_by_central_never_self_forms  (selfInSeeds:false, window 1s →
+    //      Members empty after 5s; positive control join. THE guard test: this is the exact
+    //      docker-dev topology — site nodes list only central-1.)
+}
+```
+
+Use a unique `systemName` per test (`otopcua-selfform-1` …) so concurrent hosts cannot cross-join; dispose each host in `finally` (`await host.StopAsync(); host.Dispose();`).
+
+**Step 2: Prove the key assertion has teeth** — before relying on scenario 2, temporarily run it with `selfFormAfter: null` and confirm it times out (red), then restore. Record that in the task notes.
+
+**Step 3: Run**
+
+```bash
+dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~SelfFormBootstrap"
+```
+Expected: 4 PASS.
+
+**Step 4: Commit**
+
+```bash
+git add -A && git commit -m "test(cluster): self-form fallback — lone-seed forms, disabled waits, site-node guard"
+```
+
+---
+
+### Task 4 (B4): Docs — Redundancy.md + mesh design note
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 3
+
+**Files:**
+- Modify: `docs/Redundancy.md` (~lines 162, 244–247) — replace the bootstrap-constraint text ("Auto-down removes the crash outage, not this one") with `SelfFormAfter` semantics, the 10 s default, the site-node island guard (why driver-only nodes seeded by central-1 deliberately do NOT self-form under the CURRENT one-mesh topology), and the note that under the future per-cluster mesh each pair node is a seed so the fallback covers both.
+- Modify: `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2/§7 — one-line note: the "registered outage gap" is now closed by `SelfFormAfter` (link Redundancy.md).
+- Modify: `CLAUDE.md` — brief note in the redundancy-corrections area.
+
+Commit: `docs(redundancy): SelfFormAfter closes the seed-node bootstrap gap`.
+
+---
+
+### Task 5 (B5): Full verification + optional docker-dev live check
+
+**Classification:** standard
+**Estimated implement time:** ~5 min (suite runtime dominates)
+**Parallelizable with:** none
+
+```bash
+cd ~/Desktop/OtOpcUa
+dotnet build ZB.MOM.WW.OtOpcUa.slnx     # 0 warnings
+dotnet test  ZB.MOM.WW.OtOpcUa.slnx     # green vs the pre-existing baseline
+```
+
+Optional live check (docker-dev six-node mesh): `docker compose stop central-1 central-2 && docker compose start central-2` → central-2 (a listed seed) self-forms in ~10 s; site nodes (non-seeds) keep waiting — the island guard on the real topology. Then `docker compose start central-1` → joins central-2's cluster, mesh re-forms. If the rig isn't running, record deferred-live.
+
+---
+
+### Task 6 (D4): Manual failover service + cluster-level test
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs`
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/ManualFailoverService.cs`
+- Modify: the ControlPlane/Host DI registration point (find where `RedundancyStateActor`-adjacent services register; admin-role branch of `Program.cs`)
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ManualFailoverServiceTests.cs`
+
+**Step 1: Failing tests** — reuse the two-node fixture pattern from `RedundancyPrimaryElectionTests` (node A older on the **higher** port so age ≠ address order; that fixture even has a guard test asserting the divergence — keep it honest here too):
+
+- `FailOver_targets_the_oldest_driver_member_not_the_role_leader`: service picks node A (oldest, higher port) even though B is `RoleLeader("driver")`; A's system terminates (graceful exit); B's next `RedundancyStateActor` snapshot names B Primary.
+- `FailOver_refuses_when_no_driver_peer_exists`: single driver member → returns null, no Leave, node still Up (positive assert).
+- `Parity_with_SelectDriverPrimary`: on the same fixture, the service's target equals the member `RedundancyStateActor` names Primary — pins the two 5-line queries together.
+
+**Step 2: Implement** — same shape as ScadaBridge's `FailOverCore` but role `"driver"`:
+
+```csharp
+public static Address? FailOverCore(ActorSystem system, bool dryRun = false)
+{
+    var cluster = Akka.Cluster.Cluster.Get(system);
+    // Intentionally identical to RedundancyStateActor.SelectDriverPrimary (oldest Up driver,
+    // Member.AgeOrdering) — the node acted on must be exactly the one the election names.
+    // Pinned by ManualFailoverServiceTests.Parity_with_SelectDriverPrimary.
+    var drivers = cluster.State.Members
+        .Where(m => m.Status == MemberStatus.Up && m.Roles.Contains("driver"))
+        .OrderBy(m => m, Member.AgeOrdering)
+        .ToList();
+    if (drivers.Count < 2) return null;
+    var primary = drivers[0];
+    if (!dryRun) cluster.Leave(primary.Address);
+    return primary.Address;
+}
+```
+
+Async wrapper `FailOverDriverPrimaryAsync(string actor)`: dry-run peer guard → audit via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam (action `cluster.manual-failover`, actor, target in `DetailsJson` — copy the call shape from an existing audited AdminUI admin action) → real Leave → warn log → return address string.
+
+**Step 3:** Tests PASS → commit `feat(redundancy): manual failover service — graceful Leave of the driver Primary`.
+
+---
+
+### Task 7 (D5): ClusterRedundancy page — live panel + failover button + bUnit tests
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Clusters/ClusterRedundancy.razor` (page is already `@rendermode RenderMode.InteractiveServer` — no rendermode work; and it lives in the AdminUI host, not an RCL, so the static-router wrapper concern from lmxopcua#483 does not apply)
+- Test: the AdminUI bUnit suite (place beside the existing Clusters page tests)
+- Modify: `docs/Redundancy.md` — "Manual failover" subsection (button semantics, mesh-wide scope caveat)
+
+**Step 1: Failing bUnit tests:** (a) button hidden under the read-only policy (`AdminUiPolicies.AuthenticatedRead`), visible under the admin/write policy — check `AdminUiPolicies` for the exact policy the mutating Clusters pages use (e.g. `ClusterEdit.razor`) and use the same one; (b) disabled with tooltip when the live snapshot shows <2 Up driver members; (c) confirm flow calls a fake `IManualFailoverService` exactly once.
+
+**Step 2: Implement.** Add a "Live redundancy" panel above the static config table: current driver Primary + Up driver members from a `GetSnapshot()` read-only companion on `IManualFailoverService` (sourced from live `Cluster.State`; do NOT subscribe to the DPS topic for v1), the mesh-scope notice ("in the current single-mesh topology this acts on the whole mesh's Primary"), and the admin-gated **Trigger failover** button with confirm dialog (states: the Primary node restarts, ServiceLevel 250 moves to the survivor, OPC UA clients re-select). On confirm: call the service with the authenticated user name; surface the returned target address.
+
+**Step 3:** Tests PASS → commit `feat(adminui): manual failover control on the cluster redundancy page`.
+
+**Live check (fold into the Task 5 gate when docker-dev is up):** press the button, ServiceLevel 250 moves (old Primary drops off, survivor publishes 250), denial meters stay quiet, audit event recorded.
+
+---
+
+## Reserved for additions (this plan is expected to grow)
+
+Append new tasks below as **Task 8+** (and extend `.tasks.json`) — do not renumber Tasks 1–7. Candidate items already flagged in the family docs, awaiting a decision on which to include:
+
+- **Configurable downing window** — promote `ServiceCollectionExtensions.DowningStableAfter` (hard-coded 15 s) to `AkkaClusterOptions` (e.g. `Cluster:DowningStableAfter`), keeping the pinned invariant window > `acceptable-heartbeat-pause` (`akka_failover.md` §6.2 calls this out; ScadaBridge's equivalent `Cluster:StableAfter` is already config).
+- **Failure-detector tuning for faster failover** — expose `heartbeat-interval` / `acceptable-heartbeat-pause` (currently fixed in `Resources/akka.conf`) as options, per the availability-first tuning table in `akka_failover.md` §2.
+- ~~Auto-down 1-vs-1 live gate~~ / ~~per-cluster mesh interplay~~ — **now owned by the per-cluster mesh PROGRAM plan** (`docs/plans/2026-07-22-per-cluster-mesh-program.md`, added 2026-07-22): Phases 1–7 align OtOpcUa's mesh with ScadaBridge's shape (one 2-node mesh per application Cluster, co-located on the ScadaBridge VMs). That program lists THIS plan as its prerequisite (execute this one first); its Phase 6 removes the ClusterRedundancy mesh-scope caveat + the site-node island-guard docs note, and Phase 7 closes the auto-down 1-vs-1 and `SelfFormAfter` live gates on the real per-pair topology.
+- *(add further items here)*
+
+---
+
+## Completion
+
+- Merge decision via the finishing-a-development-branch flow (family convention: ff-merge to `master` + push to gitea `lmxopcua`, or PR — ask the user).
+- After merge: update `scadaproj/akka_failover.md` §6.1 status, `scadaproj/CLAUDE.md` index row, and memory `ha-availability-over-partition-safety` (tracked in the scadaproj index plan).
+- Verification-before-completion applies throughout; live gates may be recorded deferred-live if docker-dev is down.

docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json

@@ -0,0 +1,14 @@
+{
+  "planPath": "docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md",
+  "tasks": [
+    {"id": 0, "subject": "Task 1: SelfFormAfter on AkkaClusterOptions", "status": "completed"},
+    {"id": 1, "subject": "Task 2: ClusterBootstrapFallback + AddStartup wiring", "status": "completed", "blockedBy": [0]},
+    {"id": 2, "subject": "Task 3: SelfFormBootstrapTests — 4 scenarios incl. site-node island guard", "status": "completed", "blockedBy": [1], "notes": "Teeth proven: scenario 2 run with selfFormAfter:null failed (red) before restore. A 5th test was added by Task 8."},
+    {"id": 3, "subject": "Task 4: Docs — Redundancy.md bootstrap section + mesh design note", "status": "completed", "blockedBy": [1]},
+    {"id": 4, "subject": "Task 5: Full verification + optional docker-dev live check", "status": "completed", "blockedBy": [2, 3], "notes": "Live gate PASSED for the cold-start case (defect observed on old image, fix on new, island guard on a real site node, one six-node cluster after recovery)."},
+    {"id": 5, "subject": "Task 6: Manual failover service + cluster-level test", "status": "completed", "blockedBy": [4], "notes": "4/4. Teeth proven: sabotaging FailOverCore to address order turned the target + parity tests red."},
+    {"id": 6, "subject": "Task 7: ClusterRedundancy live panel + failover button + tests", "status": "completed", "blockedBy": [5], "notes": "PLAN DEVIATION: plan said bUnit; repo has no bUnit (PageAuthorizationGuardTests documents itself as the substitute). Adapted to a pure ManualFailoverPageModel + 9 unit tests, razor as thin shell. Policy is FleetAdmin, not the ConfigEditor the neighbouring cluster pages use (this restarts a node; ConfigEditor admits Designer). Live-verified: panel, confirm dialog, execution, singleton handover, audit row."},
+    {"id": 7, "subject": "Task 8 (NEW, from live gate): reachability guard — don't self-form while a seed peer is reachable", "status": "completed", "blockedBy": [6], "notes": "DEFECT: the Task 7 live drill caught the self-form fallback islanding the node the failover bounced. central-1 restarted, got InitJoinAck from central-2 at 10:49:05 (join in flight, healthy) but no Welcome inside the window (peer's ring still held the old incarnation until 10:49:06); at 10:49:15 the fallback fired and formed a SECOND cluster. The plan's 'Window expires mid-join-handshake -> benign, Akka ignores Join once joined' row is FALSE — the node had not joined, and Join(self) wins. Manual failover deliberately produces that restart, so the two halves of the plan collided. FIX (user chose the reachability guard): TCP-probe the other seed addresses before self-forming; a reachable peer means wait another window and re-arm. Proven three ways: (1) new regression test whose 'peer' is a bare TcpListener speaking no Akka — RED without the guard; (2) live, a throwaway node seeded at sql:1433 (TCP-reachable, not Akka) logged the guard every 10s and never self-formed; (3) positive control, the same node seeded at a dead port self-formed at +10s. Failover re-drill on the rig: no island, all six nodes one cluster."}
+  ],
+  "lastUpdated": "2026-07-22T00:00:00Z"
+}

docs/v2/Cluster.md

@@ -64,7 +64,7 @@ The Cluster.Tests project verifies these key values stay correct (`HoconLoaderTe
 - `Hostname`: interface to bind. `0.0.0.0` listens on every interface.
 - `Port`: TCP port for cluster gossip. Default 4053.
 - `PublicHostname`: address advertised in cluster gossip. Must be reachable by every other node.
-- `SeedNodes`: where new nodes go to join. List one (or two) stable nodes. First node bootstraps the cluster from its own address.
+- `SeedNodes`: where new nodes go to join. List one (or two) stable nodes. **Order is load-bearing:** Akka only lets `seed-nodes[0]` form a *new* cluster (`FirstSeedNodeProcess`); every other node retries `InitJoin` forever. A node that is one of its own seeds must therefore list ITSELF first, or it can never cold-start while its peer is down — enforced at boot by `AkkaClusterOptionsValidator`, explained in [Redundancy.md § Bootstrap: self-first seed ordering](../Redundancy.md#bootstrap-self-first-seed-ordering). Nodes seeded only by someone else (driver-only site nodes) are exempt.
 - `Roles`: free-form tags Akka gossip propagates. v2 uses `admin` + `driver`; per-role wiring in `Program.cs` reads `OTOPCUA_ROLES` env var, not this list — these two should stay in sync.
 
 Per-role overlay files (`appsettings.admin.json`, `appsettings.driver.json`, `appsettings.admin-driver.json`) layer on top of base `appsettings.json` based on the parsed `OTOPCUA_ROLES` (alphabetical, joined by `-`). See [ServiceHosting.md § Per-role configuration overlays](../ServiceHosting.md#per-role-configuration-overlays).

docs/v2/config-db-schema.md

@@ -148,10 +148,13 @@ CREATE TABLE dbo.ClusterNode (
     Host                            nvarchar(255)   NOT NULL,
     OpcUaPort                       int             NOT NULL DEFAULT 4840,
     DashboardPort                   int             NOT NULL DEFAULT 8081,
+    AkkaPort                        int             NOT NULL DEFAULT 4053,
+    GrpcPort                        int             NULL,
     ApplicationUri                  nvarchar(256)   NOT NULL,
     ServiceLevelBase                tinyint         NOT NULL DEFAULT 200,
     DriverConfigOverridesJson       nvarchar(max)   NULL CHECK (DriverConfigOverridesJson IS NULL OR ISJSON(DriverConfigOverridesJson) = 1),
     Enabled                         bit             NOT NULL DEFAULT 1,
+    MaintenanceMode                 bit             NOT NULL DEFAULT 0,
     LastSeenAt                      datetime2(3)    NULL,
     CreatedAt                       datetime2(3)    NOT NULL DEFAULT SYSUTCDATETIME(),
     CreatedBy                       nvarchar(128)   NOT NULL
@@ -167,6 +170,49 @@ CREATE UNIQUE INDEX UX_ClusterNode_Primary_Per_Cluster
     WHERE RedundancyRole = 'Primary';
 ```
 
+#### `AkkaPort` / `GrpcPort` — central's dial targets (per-cluster mesh Phase 1)
+
+These are **the addresses central dials**, not the node's own binding configuration. The node binds
+from `Cluster:Port` in its own appsettings; these columns duplicate that value for a reader that
+cannot see the node's config. Today central shares a gossip ring with every node and does not need
+them — [per-cluster mesh Phase 2](../plans/2026-07-21-per-cluster-mesh-design.md) splits the fleet
+into one mesh per `Cluster`, after which `Host` + `AkkaPort` is how central builds its ClusterClient
+contact points, and `GrpcPort` is the Phase 5 telemetry stream.
+
+`AkkaPort` is `NOT NULL DEFAULT 4053` because every node listens on a remoting port — `0` is never a
+truthful value, and rows predating the column must migrate to something real. `GrpcPort` is nullable
+with **no** default because nothing listens on it until Phase 5, and a non-null default would assert
+a port that does not exist.
+
+**The duplication against `Cluster:Port` is not enforced by the schema.** A node that binds 4054 while
+its row says 4053 is unreachable from central in Phase 2, and the symptom there is a silent absence of
+acks rather than an error. `ClusterNodeAddressReconcilerActor` (an admin-role singleton) compares the
+rows against observed cluster membership and logs an Error on mismatch. It reads membership rather
+than having each driver node assert its own row, because Phase 4 removes the driver nodes' ConfigDb
+connection entirely.
+
+#### `Enabled` vs `MaintenanceMode` — two flags, two questions
+
+Phase 1 made the rows the deploy path's **expected-ack set**: `ConfigPublishCoordinator` no longer
+derives it from cluster membership, so a row whose node is down now fails the deployment at the apply
+deadline instead of letting it seal green without that node.
+
+The escape hatch is **`MaintenanceMode = 1`**, not `Enabled = 0`:
+
+| Flag | Question it answers | Checked by |
+|---|---|---|
+| `Enabled` | Is this node part of the cluster's declared topology? | `DraftValidator.ValidateClusterTopology` — the enabled-node count must equal `ServerCluster.NodeCount` |
+| `MaintenanceMode` | Should we expect it to participate right now? | `ConfigPublishCoordinator` (expected-ack set) and `ClusterNodeAddressReconciler` (absence warnings) |
+
+`Enabled = 0` on one node of a Warm/Hot pair is **rejected at the deploy gate** with
+`ClusterEnabledNodeCountMismatch` — the enabled count drops to 1 while `NodeCount` stays 2. Since
+every cluster in the target topology is a 2-node pair, `Enabled` cannot serve as the maintenance
+hatch at all. The Phase 1 live gate found this; the two rules were independently reasonable and
+contradictory together, so the meanings were split rather than either rule being weakened.
+
+A node in maintenance is still enabled, so the topology gate still passes, and the runtime redundancy
+posture (`NodeCount` / `RedundancyMode`) is unchanged.
+
 `DriverConfigOverridesJson` shape:
 
 ```jsonc

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs

@@ -20,7 +20,31 @@ public sealed class AkkaClusterOptions
     /// </summary>
     public string PublicHostname { get; set; } = "127.0.0.1";
 
-    /// <summary>Gets or sets the seed nodes for cluster bootstrapping.</summary>
+    /// <summary>
+    /// Gets or sets the seed nodes for cluster bootstrapping.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b>ORDER IS LOAD-BEARING (decision 2026-07-22): a node that is one of its own seeds
+    ///         must list ITSELF first.</b> Akka runs <c>FirstSeedNodeProcess</c> — the only bootstrap
+    ///         path that can form a NEW cluster when no peer answers <c>InitJoin</c> — exclusively
+    ///         when <c>seed-nodes[0]</c> is this node's own address; any other node runs
+    ///         <c>JoinSeedNodeProcess</c>, which retries <c>InitJoin</c> forever and can never form a
+    ///         cluster. So listing both peers does NOT mean either can cold-start alone: a node that
+    ///         lists its partner first never comes Up while that partner is down. Self-first ordering
+    ///         closes that gap inside Akka's own handshake — unlike the retired
+    ///         <c>Cluster:SelfFormAfter</c> watchdog, which sat outside it and could not tell "no
+    ///         seed answered" from "a seed answered and the join is in flight".
+    ///     </para>
+    ///     <para>
+    ///         The rule is <b>conditional</b>: it binds only when this node's own address appears in
+    ///         this list. A node seeded exclusively by someone else — today's docker-dev site nodes,
+    ///         which list only <c>central-1</c> — is legitimately not a seed and is exempt. Enforced
+    ///         at boot by <see cref="AkkaClusterOptionsValidator"/>; identity is
+    ///         <see cref="PublicHostname"/> + <see cref="Port"/> (what Akka puts in
+    ///         <c>SelfAddress</c>), never the <see cref="Hostname"/> bind address.
+    ///     </para>
+    /// </remarks>
     public string[] SeedNodes { get; set; } = Array.Empty<string>();
 
     /// <summary>
@@ -29,4 +53,30 @@ public sealed class AkkaClusterOptions
     /// <c>admin</c>, <c>driver</c>, <c>dev</c>.
     /// </summary>
     public string[] Roles { get; set; } = Array.Empty<string>();
+
+    /// <summary>
+    /// How the cluster decides to down a node it can no longer reach. One of <c>auto-down</c>
+    /// (default) or <c>keep-oldest</c>; any other value fails the host at startup.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b><c>auto-down</c> — availability.</b> The leader among the <i>reachable</i> members
+    ///         downs the unreachable peer after
+    ///         <see cref="ServiceCollectionExtensions.DowningStableAfter"/>. A hard crash of either
+    ///         node — including the oldest — fails over to the survivor with no operator action.
+    ///         The trade is that a genuine network partition (both nodes alive, link cut) leaves
+    ///         both sides running active until an operator restarts one.
+    ///     </para>
+    ///     <para>
+    ///         <b><c>keep-oldest</c> — partition-safety.</b> The SBR resolver sacrifices the younger
+    ///         side of a split, so a partition can never run dual-active. <b>The cost is severe for a
+    ///         two-node pair:</b> it cannot survive a crash of the oldest node at all. Akka.NET's
+    ///         <c>KeepOldest.OldestDecision</c> only lets <c>down-if-alone</c> rescue a side holding
+    ///         &gt;= 2 members, so the 1-vs-1 survivor downs <i>itself</i> and shuts down — the
+    ///         redundancy pair turns a single-node crash into a total outage. Choose this only for
+    ///         clusters of three or more nodes, or where dual-active is genuinely worse than an
+    ///         outage.
+    ///     </para>
+    /// </remarks>
+    public string SplitBrainResolverStrategy { get; set; } = "auto-down";
 }

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptionsValidator.cs

@@ -0,0 +1,96 @@
+using Akka.Actor;
+using ZB.MOM.WW.Configuration;
+
+namespace ZB.MOM.WW.OtOpcUa.Cluster;
+
+/// <summary>
+///     Fail-fast startup validator for <see cref="AkkaClusterOptions"/>, built on the shared
+///     <c>ZB.MOM.WW.Configuration</c> <see cref="OptionsValidatorBase{TOptions}"/> and wired through
+///     <c>AddValidatedOptions</c> in <see cref="ServiceCollectionExtensions.AddOtOpcUaCluster"/>.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>Self-first seed ordering (decision 2026-07-22).</b> Akka runs
+///         <c>FirstSeedNodeProcess</c> — the only bootstrap path that can form a NEW cluster when no
+///         peer answers <c>InitJoin</c> — exclusively when <c>seed-nodes[0]</c> is this node's own
+///         address. Every other node runs <c>JoinSeedNodeProcess</c> and retries <c>InitJoin</c>
+///         forever. A node that lists its partner first therefore cannot cold-start while that
+///         partner is down, and the failure is <i>silent</i>: the process runs, the port listens, the
+///         node simply never becomes a member. Enforced here so it is loud, at boot, on the node
+///         that has the problem.
+///     </para>
+///     <para>
+///         <b>The rule is conditional and must stay that way.</b> It binds only when this node's own
+///         address appears in its own seed list. A driver-only site node is seeded solely by
+///         <c>central-1</c> and is legitimately not a seed of anything; an unconditional "self must
+///         be first" rule would refuse to boot every one of them.
+///     </para>
+///     <para>
+///         <b>Identity is the advertised address, never the bind address.</b> In docker-dev
+///         <c>Cluster:Hostname</c> is <c>0.0.0.0</c> and <c>Cluster:PublicHostname</c> is the
+///         container DNS name — the value Akka puts in <c>SelfAddress</c> and the only one a seed URI
+///         can match. Comparing against <c>Hostname</c> would match nothing anywhere, silently exempt
+///         every node, and leave this rule inert. <c>PublicHostname</c> falling back to
+///         <c>Hostname</c> when blank mirrors Akka's own <c>public-hostname</c> semantics.
+///     </para>
+/// </remarks>
+public sealed class AkkaClusterOptionsValidator : OptionsValidatorBase<AkkaClusterOptions>
+{
+    /// <inheritdoc />
+    protected override void Validate(ValidationBuilder builder, AkkaClusterOptions options)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+
+        var seeds = options.SeedNodes ?? Array.Empty<string>();
+        if (seeds.Length == 0)
+        {
+            // Single-node / dev default: nothing is claimed about ordering, so there is nothing to
+            // enforce. (Akka bootstraps from an empty seed list by waiting for an explicit join.)
+            return;
+        }
+
+        var selfHost = string.IsNullOrWhiteSpace(options.PublicHostname)
+            ? options.Hostname
+            : options.PublicHostname;
+
+        var selfIndex = Array.FindIndex(seeds, s => IsSelf(s, selfHost, options.Port));
+        if (selfIndex <= 0)
+        {
+            // -1 = this node is not one of its own seeds (exempt); 0 = correctly ordered.
+            return;
+        }
+
+        builder.Add(
+            $"Cluster:SeedNodes must list this node itself first: entry {selfIndex} is this node "
+            + $"('{seeds[selfIndex]}') but entry 0 is '{seeds[0]}'. Akka only lets seed-nodes[0] form "
+            + "a new cluster (FirstSeedNodeProcess); every other node retries InitJoin forever, so "
+            + "with the partner listed first this node can never start while its peer is down. Swap "
+            + "the entries — see docs/Redundancy.md → 'Bootstrap: self-first seed ordering'.");
+    }
+
+    /// <summary>
+    /// True when <paramref name="seed"/> addresses this node itself — host AND port.
+    /// </summary>
+    /// <remarks>
+    /// Host comparison is case-insensitive because DNS names are, but is otherwise exact: Akka does
+    /// no DNS canonicalisation either, so <c>central-2</c> and <c>central-2.example.com</c> are
+    /// genuinely different seed identities to the cluster. A seed entry Akka itself cannot parse is
+    /// treated as "not this node" — reporting it is Akka's job, with Akka's wording, and this rule
+    /// must not turn a parse problem into a misleading ordering complaint.
+    /// </remarks>
+    private static bool IsSelf(string seed, string selfHost, int selfPort)
+    {
+        Address address;
+        try
+        {
+            address = Address.Parse(seed);
+        }
+        catch (Exception)
+        {
+            return false;
+        }
+
+        return address.Port == selfPort
+            && string.Equals(address.Host, selfHost, StringComparison.OrdinalIgnoreCase);
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs

@@ -7,7 +7,7 @@ public readonly record struct NodeHealthInputs(
     bool DbReachable,
     bool OpcUaProbeOk,
     bool Stale,
-    bool IsDriverRoleLeader);
+    bool IsDriverPrimary);
 
 /// <summary>
 /// Pure ServiceLevel computation per design §6. Output range 0–255, where higher = "more
@@ -18,7 +18,9 @@ public readonly record struct NodeHealthInputs(
 /// - Member not Up/Joining: 0 (cluster cannot trust this node).
 /// - DB reachable + OPC UA probe ok + not stale: 240 (full service).
 /// - Stale config (DB reachable or not, OPC UA probe state ignored): 100 or 200 depending on DB.
-/// - +10 bonus when this node holds the role-leader lease for the "driver" role.
+/// - +10 bonus when this node is the driver Primary (the oldest Up member with the "driver" role —
+///   where the cluster singletons live). Previously this keyed off Akka's role *leader*, the
+///   lowest-addressed member, which names a different node once any node has restarted.
 /// </summary>
 public static class ServiceLevelCalculator
 {
@@ -38,6 +40,6 @@ public static class ServiceLevelCalculator
             _ => 0,
         };
 
-        return (byte)Math.Clamp(basis + (h.IsDriverRoleLeader ? 10 : 0), 0, 255);
+        return (byte)Math.Clamp(basis + (h.IsDriverPrimary ? 10 : 0), 0, 255);
     }
 }

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf

@@ -37,14 +37,26 @@ akka {
         roles = []
         min-nr-of-members = 1
 
-        # Split-brain resolver (arch-review 03/S1). This block IS active: Akka.Cluster.Hosting's
-        # WithClustering enables an SBR downing provider by default (it applies
-        # SplitBrainResolverOption.Default when ClusterOptions.SplitBrainResolver is null), and that
-        # provider reads this block. ServiceCollectionExtensions.BuildClusterOptions additionally sets
-        # the typed KeepOldestOption { DownIfAlone = true } to make the strategy EXPLICIT in code —
-        # its active-strategy + keep-oldest.down-if-alone MUST match this block. (The cluster is NOT
-        # NoDowning; only an explicit downing-provider-class = "" would disable failover.)
-        # stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin.
+        # DOWNING. The provider is NOT selected here — this file is added with HoconAddMode.Append,
+        # which loses to what Akka.Hosting's WithClustering emits, so a downing-provider-class set
+        # here would be silently overridden. Selection lives in
+        # ServiceCollectionExtensions.BuildDowningHocon (Prepended, which outranks WithClustering),
+        # driven by Cluster:SplitBrainResolverStrategy.
+        #
+        # The DEFAULT strategy is "auto-down", under which this whole block is inert: Akka's
+        # AutoDowning provider is installed with auto-down-unreachable-after = 15s, and the leader
+        # among the reachable members downs the unreachable peer — so a hard crash of EITHER node,
+        # oldest included, fails over. The trade is dual-active during a real network partition.
+        #
+        # The block below applies only when an operator selects "keep-oldest". Note that keep-oldest
+        # is NOT safe for a two-node pair: Akka.NET's KeepOldest.OldestDecision only lets
+        # down-if-alone rescue a side holding >= 2 members, so a 1-vs-1 survivor downs ITSELF and
+        # (with run-coordinated-shutdown-when-down) exits — a crash of the oldest node becomes a
+        # total outage. See docs/Redundancy.md.
+        #
+        # stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin, and
+        # must equal ServiceCollectionExtensions.DowningStableAfter so both strategies fail over at
+        # the same latency. Both constraints are pinned by SplitBrainResolverActivationTests.
         split-brain-resolver {
             active-strategy = "keep-oldest"
             stable-after = 15s

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs

@@ -7,6 +7,7 @@ using Akka.Remote.Hosting;
 using Microsoft.Extensions.Configuration;
 using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.Options;
+using ZB.MOM.WW.Configuration;
 using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
 
 namespace ZB.MOM.WW.OtOpcUa.Cluster;
@@ -19,13 +20,19 @@ public static class ServiceCollectionExtensions
     /// configurator via <see cref="WithOtOpcUaClusterBootstrap"/> — keeping the entire Akka graph
     /// under Akka.Hosting's management so cluster singletons land on the same ActorSystem.
     /// </summary>
+    /// <remarks>
+    /// The binding is validated at startup (<c>ValidateOnStart</c>) by
+    /// <see cref="AkkaClusterOptionsValidator"/>, so a seed list that cannot bootstrap this node —
+    /// self listed behind its partner — fails the host loudly instead of leaving it running but
+    /// permanently outside the cluster.
+    /// </remarks>
     /// <param name="services">The service collection to configure.</param>
     /// <param name="configuration">The application configuration containing cluster options.</param>
     /// <returns>The same service collection, for chaining.</returns>
     public static IServiceCollection AddOtOpcUaCluster(this IServiceCollection services, IConfiguration configuration)
     {
-        services.AddOptions<AkkaClusterOptions>()
-            .Bind(configuration.GetSection(AkkaClusterOptions.SectionName));
+        services.AddValidatedOptions<AkkaClusterOptions, AkkaClusterOptionsValidator>(
+            configuration, AkkaClusterOptions.SectionName);
 
         services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
 
@@ -80,40 +87,133 @@ public static class ServiceCollectionExtensions
 
         builder.WithClustering(BuildClusterOptions(options));
 
+        // Must come AFTER WithClustering, which always emits a downing-provider-class of its own:
+        // Akka.Cluster.Hosting applies SplitBrainResolverOption.Default whenever the typed
+        // ClusterOptions.SplitBrainResolver is null, so there is always a competing value to beat.
+        // Prepend is the highest-precedence mode, which makes this block win irrespective of how
+        // many fragments are added or in what order.
+        //
+        // Do not take this comment's word for it. Which fragment actually wins is not obvious from
+        // the mode names — measured, Append also wins here, purely because it happens to be added
+        // last. SplitBrainResolverActivationTests reads the provider back off a *running*
+        // ActorSystem for exactly that reason: the effective value is the only one worth asserting.
+        builder.AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend);
+
+        // NOTE (2026-07-22): no bootstrap watchdog is armed here any more. The cold-start-alone gap
+        // it existed for — Akka lets only seed-nodes[0] form a NEW cluster — is now closed by
+        // self-first seed ordering (AkkaClusterOptions.SeedNodes), enforced at boot by
+        // AkkaClusterOptionsValidator. The retired ClusterBootstrapFallback timer sat OUTSIDE Akka's
+        // join handshake, so it could not distinguish "no seed answered" from "a seed answered and
+        // the join is in flight", and Cluster.Join(SelfAddress) is not ignored mid-handshake — it
+        // wins. See docs/Redundancy.md → "Bootstrap: self-first seed ordering".
+
         return builder;
     }
 
     /// <summary>
-    /// Builds the <see cref="ClusterOptions"/> for the fused-host cluster, setting the split-brain-resolver
-    /// strategy <b>explicitly in code</b>.
-    ///
-    /// <b>Activation note (arch-review 03/S1, corrected):</b> Akka.Cluster.Hosting's <c>WithClustering</c>
-    /// already enables an SBR downing provider <b>by default</b> — when <see cref="ClusterOptions.SplitBrainResolver"/>
-    /// is <c>null</c> it applies <c>SplitBrainResolverOption.Default</c>, which registers
-    /// <c>Akka.Cluster.SBR.SplitBrainResolverProvider</c> and reads the <c>split-brain-resolver</c> HOCON block in
-    /// <c>Resources/akka.conf</c>. So the cluster was <b>not</b> running <c>NoDowning</c> before this option was
-    /// set: the pre-existing akka.conf <c>keep-oldest</c> block was already active, and hard-crashed nodes already
-    /// failed over. Setting this typed option makes the strategy <b>explicit in code</b> (independent of the
-    /// framework default) rather than being the sole activator — it is reinforcing/belt-and-suspenders and
-    /// produces the same effective behavior. (Only an <i>explicit</i> <c>NoDowning</c>, e.g.
-    /// <c>downing-provider-class = ""</c>, would disable failover.)
-    ///
-    /// <see cref="KeepOldestOption"/> with <c>DownIfAlone=true</c> mirrors the HOCON intent and is the
-    /// correct strategy for a 2-node warm-redundancy pair: on an even split the oldest member (typically
-    /// the long-running primary) survives, and <c>down-if-alone</c> downs a singleton that loses its
-    /// peer. <c>keep-majority</c>/<c>static-quorum</c> are wrong for two nodes. The strategy + down-if-alone
-    /// here MUST stay consistent with the HOCON block; <c>stable-after</c> lives only in HOCON because the
-    /// typed option cannot express it (it must stay ≥ <c>failure-detector.acceptable-heartbeat-pause</c>).
+    /// How long a member must stay unreachable before it is downed. Shared by both strategies
+    /// (<c>auto-down-unreachable-after</c> and the SBR resolver's <c>stable-after</c>) so the two
+    /// have the same failover latency. Must stay above
+    /// <c>akka.cluster.failure-detector.acceptable-heartbeat-pause</c> or a merely-slow node gets
+    /// downed; both constraints are pinned by tests.
     /// </summary>
-    /// <param name="options">The bound cluster options carrying seed nodes and roles.</param>
-    /// <returns>The cluster options with seed nodes, roles, and the explicit split-brain-resolver strategy.</returns>
+    public static readonly TimeSpan DowningStableAfter = TimeSpan.FromSeconds(15);
+
+    /// <summary>
+    /// Builds the HOCON that selects the downing provider, per
+    /// <see cref="AkkaClusterOptions.SplitBrainResolverStrategy"/>.
+    /// </summary>
+    /// <param name="options">The bound cluster options carrying the strategy.</param>
+    /// <returns>
+    /// For <c>auto-down</c>, a block installing Akka's <c>AutoDowning</c> provider with a downing
+    /// window of <see cref="DowningStableAfter"/>. For <c>keep-oldest</c>, an empty string — the
+    /// typed <see cref="KeepOldestOption"/> from <see cref="BuildClusterOptions"/> already installs
+    /// the SBR provider, and the resolver's own settings live in <c>Resources/akka.conf</c>.
+    /// </returns>
+    /// <exception cref="ArgumentOutOfRangeException">The strategy is not a recognised value.</exception>
+    public static string BuildDowningHocon(AkkaClusterOptions options)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+
+        if (IsAutoDown(options))
+        {
+            // auto-down-unreachable-after is load-bearing: it defaults to `off`, under which
+            // AutoDowning is installed but never downs anything — failover silently disabled.
+            return $$"""
+                akka.cluster {
+                    downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"
+                    auto-down-unreachable-after = {{(int)DowningStableAfter.TotalMilliseconds}}ms
+                }
+                """;
+        }
+
+        if (IsKeepOldest(options))
+        {
+            return string.Empty;
+        }
+
+        throw new ArgumentOutOfRangeException(
+            nameof(options),
+            options.SplitBrainResolverStrategy,
+            $"Unknown {AkkaClusterOptions.SectionName}:{nameof(AkkaClusterOptions.SplitBrainResolverStrategy)}. "
+            + "Expected 'auto-down' (default; survives a crash of either node) or 'keep-oldest' "
+            + "(partition-safe, but a two-node pair cannot survive a crash of the oldest node).");
+    }
+
+    private static bool IsAutoDown(AkkaClusterOptions options) =>
+        string.Equals(options.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase);
+
+    private static bool IsKeepOldest(AkkaClusterOptions options) =>
+        string.Equals(options.SplitBrainResolverStrategy, "keep-oldest", StringComparison.OrdinalIgnoreCase);
+
+    /// <summary>
+    /// Builds the <see cref="ClusterOptions"/> for the fused-host cluster.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b>Activation note (arch-review 03/S1, corrected twice).</b> Akka.Cluster.Hosting's
+    ///         <c>WithClustering</c> always installs a downing provider: when
+    ///         <see cref="ClusterOptions.SplitBrainResolver"/> is <c>null</c> it applies
+    ///         <c>SplitBrainResolverOption.Default</c>, which registers
+    ///         <c>Akka.Cluster.SBR.SplitBrainResolverProvider</c> and reads the
+    ///         <c>split-brain-resolver</c> block in <c>Resources/akka.conf</c>. The cluster is
+    ///         therefore never <c>NoDowning</c>. What this method chooses is only <i>which</i>
+    ///         provider — and under the default <c>auto-down</c> strategy the choice is not made
+    ///         here at all but by the Prepended HOCON in
+    ///         <see cref="WithOtOpcUaClusterBootstrap"/>, which outranks whatever
+    ///         <c>WithClustering</c> emits.
+    ///     </para>
+    ///     <para>
+    ///         <b>Why keep-oldest is no longer the default (2026-07-21).</b> The previous
+    ///         documentation here asserted that <see cref="KeepOldestOption"/> with
+    ///         <c>DownIfAlone=true</c> was "the correct strategy for a 2-node warm-redundancy pair"
+    ///         because <c>down-if-alone</c> would down a node that lost its peer. That is the
+    ///         opposite of what Akka.NET does. In <c>KeepOldest.OldestDecision</c> the
+    ///         <c>down-if-alone</c> branch requires the surviving side to hold &gt;= 2 members, so
+    ///         in a 1-vs-1 split the survivor falls through to <c>DownReachable</c> and downs
+    ///         <i>itself</i>; with <c>run-coordinated-shutdown-when-down = on</c> it then exits. A
+    ///         two-node pair running keep-oldest converts a crash of the oldest node into a total
+    ///         outage — precisely the failure redundancy exists to absorb. Live-proven on the sister
+    ///         project's rig; see <c>docs/Redundancy.md</c> and
+    ///         <see cref="AkkaClusterOptions.SplitBrainResolverStrategy"/> for the trade.
+    ///     </para>
+    /// </remarks>
+    /// <param name="options">The bound cluster options carrying seed nodes, roles and the strategy.</param>
+    /// <returns>
+    /// The cluster options with seed nodes and roles. <see cref="ClusterOptions.SplitBrainResolver"/>
+    /// is set only under <c>keep-oldest</c>, where the SBR provider is the one wanted; under
+    /// <c>auto-down</c> it is left null because the Prepended HOCON replaces the provider anyway,
+    /// and naming a resolver that is not in force would be misleading.
+    /// </returns>
     public static ClusterOptions BuildClusterOptions(AkkaClusterOptions options)
     {
+        ArgumentNullException.ThrowIfNull(options);
+
         return new ClusterOptions
         {
             SeedNodes = options.SeedNodes,
             Roles = options.Roles,
-            SplitBrainResolver = new KeepOldestOption { DownIfAlone = true },
+            SplitBrainResolver = IsKeepOldest(options) ? new KeepOldestOption { DownIfAlone = true } : null,
         };
     }
 }

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ZB.MOM.WW.OtOpcUa.Cluster.csproj

@@ -14,6 +14,10 @@
         <PackageReference Include="Akka.Remote.Hosting"/>
         <PackageReference Include="Microsoft.Extensions.Hosting"/>
         <PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions"/>
+        <!-- Shared options-validation primitives (OptionsValidatorBase / AddValidatedOptions), the
+             same seam the Host's Ldap/OpcUa/Historian validators use. Pulled in here because the
+             self-first seed-ordering rule belongs with the options it validates. -->
+        <PackageReference Include="ZB.MOM.WW.Configuration"/>
     </ItemGroup>
 
     <ItemGroup>

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Interfaces/NodeDiagnosticsSnapshot.cs

@@ -14,8 +14,16 @@ public sealed record DriverInstanceDiagnostics(
 /// Per-node diagnostics returned by <c>IFleetDiagnosticsClient</c>. Populated by the node's
 /// local <c>DriverHostActor</c> via a request/response over Akka.
 /// </summary>
+/// <param name="RunningFromCache">
+///     True when the node booted its configuration from the node-local artifact cache because the
+///     central ConfigDb was unreachable. Such a node looks entirely healthy — full address space,
+///     live values — but its configuration is frozen and no deployment can reach it, so the state
+///     needs to be explicitly visible rather than inferred. Defaults to false, which is also what
+///     every node that booted normally reports.
+/// </param>
 public sealed record NodeDiagnosticsSnapshot(
     NodeId NodeId,
     RevisionHash? CurrentRevision,
     IReadOnlyList<DriverInstanceDiagnostics> Drivers,
-    DateTime AsOfUtc);
+    DateTime AsOfUtc,
+    bool RunningFromCache = false);

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Redundancy/NodeRedundancyState.cs

@@ -8,9 +8,20 @@ public enum RedundancyRole { Primary, Secondary, Detached }
 /// Snapshot of a single node's redundancy state. Aggregated by <c>RedundancyStateActor</c>
 /// to compute fleet-wide ServiceLevel.
 /// </summary>
+/// <param name="NodeId">Canonical <c>host:port</c> id of the node this entry describes.</param>
+/// <param name="Role">The node's redundancy role in the current snapshot.</param>
+/// <param name="IsClusterLeader">Whether the node is the Akka cluster leader.</param>
+/// <param name="IsDriverPrimary">
+/// Whether the node is the driver Primary — the <b>oldest</b> Up member carrying the <c>driver</c>
+/// role, which is where <c>ClusterSingletonManager</c> places singletons. Renamed from
+/// <c>IsRoleLeaderForDriver</c>: it was derived from <c>ClusterState.RoleLeader("driver")</c>, the
+/// lowest-<i>addressed</i> member, which diverges from the oldest after any node restart. The name
+/// now describes what the value means rather than how it used to be computed.
+/// </param>
+/// <param name="AsOfUtc">When the snapshot was computed.</param>
 public sealed record NodeRedundancyState(
     NodeId NodeId,
     RedundancyRole Role,
     bool IsClusterLeader,
-    bool IsRoleLeaderForDriver,
+    bool IsDriverPrimary,
     DateTime AsOfUtc);

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/ClusterNode.cs

@@ -18,6 +18,33 @@ public sealed class ClusterNode
     /// <summary>The dashboard HTTP port (default 8081).</summary>
     public int DashboardPort { get; set; } = 8081;
 
+    /// <summary>
+    ///     Akka remoting port (default 4053). <b>This is central's dial target, not the node's own
+    ///     binding configuration</b> — the node binds from <c>Cluster:Port</c> in its own appsettings,
+    ///     and this column duplicates that value for a reader that cannot see the node's config.
+    ///     Per-cluster mesh Phase 2 builds its ClusterClient contact points from
+    ///     <see cref="Host"/> + this port, at which point central and the site node no longer share a
+    ///     gossip ring and central has no other way to learn it.
+    /// </summary>
+    /// <remarks>
+    ///     The duplication is real and unenforced by the schema. <c>ClusterNodeAddressReconciler</c>
+    ///     on the admin node compares this row against observed cluster membership and logs an Error
+    ///     on mismatch — a node that binds 4054 while its row says 4053 is unreachable from central in
+    ///     Phase 2, and the symptom there is a silent absence of acks rather than an error.
+    /// </remarks>
+    public int AkkaPort { get; set; } = 4053;
+
+    /// <summary>
+    ///     gRPC port central dials for the Phase 5 telemetry stream, or <see langword="null"/> when the
+    ///     node exposes none. Also central's dial target rather than the node's binding config.
+    /// </summary>
+    /// <remarks>
+    ///     Nullable by intent: nothing listens on this port yet, and a non-null default would assert a
+    ///     port that does not exist. Phase 5 populates it; until then <see langword="null"/> is the
+    ///     honest value.
+    /// </remarks>
+    public int? GrpcPort { get; set; }
+
     /// <summary>
     ///     OPC UA <c>ApplicationUri</c> — MUST be unique per node per OPC UA spec. Clients pin trust here.
     ///     Fleet-wide unique index enforces no two nodes share a value.
@@ -36,8 +63,37 @@ public sealed class ClusterNode
     public string? DriverConfigOverridesJson { get; set; }
 
     /// <summary>Gets or sets a value indicating whether this node is enabled.</summary>
+    /// <remarks>
+    ///     <b>Part of the declared topology.</b> <c>DraftValidator.ValidateClusterTopology</c> requires
+    ///     the enabled-node count to equal <see cref="ServerCluster.NodeCount"/>, so disabling one node
+    ///     of a Warm/Hot pair is a deploy-blocking validation error by design — it would boot the
+    ///     runtime into the InvalidTopology band. To take a node out of service temporarily, use
+    ///     <see cref="MaintenanceMode"/> instead.
+    /// </remarks>
     public bool Enabled { get; set; } = true;
 
+    /// <summary>
+    ///     Node is temporarily out of service: still part of the cluster's declared topology, but not
+    ///     expected to participate. A deployment does not wait for its ack, and the address reconciler
+    ///     does not warn that it is absent.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         Separate from <see cref="Enabled"/> because the two answer different questions.
+    ///         <see cref="Enabled"/> means "this node is part of the declared topology" and is checked
+    ///         against <see cref="ServerCluster.NodeCount"/>; this flag means "do not expect it right
+    ///         now". Per-cluster mesh Phase 1 made an enabled row's ack mandatory, so without this a
+    ///         node down for maintenance would block every deployment to its cluster — and clearing
+    ///         <see cref="Enabled"/> to escape that is itself rejected by the topology gate on any
+    ///         2-node pair, which is every cluster in the target deployment.
+    ///     </para>
+    ///     <para>
+    ///         Found by the Phase 1 live gate: the two rules were independently reasonable and
+    ///         contradictory together.
+    ///     </para>
+    /// </remarks>
+    public bool MaintenanceMode { get; set; }
+
     /// <summary>Gets or sets the timestamp when this node was last seen.</summary>
     public DateTime? LastSeenAt { get; set; }
 

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/GenerationSealedCache.cs

@@ -1,197 +0,0 @@
-using LiteDB;
-
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Generation-sealed LiteDB cache per <c>docs/v2/plan.md</c> and Phase 6.1
-///     Stream D.1. Each published generation writes one <b>read-only</b> LiteDB file under
-///     <c>&lt;cache-root&gt;/&lt;clusterId&gt;/&lt;generationId&gt;.db</c>. A per-cluster
-///     <c>CURRENT</c> text file holds the currently-active generation id; it is updated
-///     atomically (temp file + <see cref="File.Replace(string, string, string?)"/>) only after
-///     the sealed file is fully written.
-/// </summary>
-/// <remarks>
-///     <para>Mixed-generation reads are impossible: any read opens the single file pointed to
-///     by <c>CURRENT</c>, which is a coherent snapshot. Corruption of the CURRENT file or the
-///     sealed file surfaces as <see cref="GenerationCacheUnavailableException"/> — the reader
-///     fails closed rather than silently falling back to an older generation. Recovery path
-///     is to re-fetch from the central DB (and the Phase 6.1 Stream C <c>UsingStaleConfig</c>
-///     flag goes true until that succeeds).</para>
-///
-///     <para>This cache is the read-path fallback when the central DB is unreachable. The
-///     write path (draft edits, publish) bypasses the cache and fails hard on DB outage per
-///     Stream D.2 — inconsistent writes are worse than a temporary inability to edit.</para>
-/// </remarks>
-public sealed class GenerationSealedCache
-{
-    private const string CollectionName = "generation";
-    private const string CurrentPointerFileName = "CURRENT";
-    private readonly string _cacheRoot;
-
-    // Private per-database BsonMapper with the entity pre-registered. LiteDB's default
-    // BsonMapper.Global is a process-wide singleton whose lazy per-type member registration is
-    // not thread-safe across concurrently-constructed LiteDatabase instances; a seal racing a
-    // read (or this cache racing LiteDbConfigCache) corrupts the global mapper, surfacing as
-    // "Member … not found on BsonMapper" or a bogus duplicate-_id insert.
-    private static BsonMapper BuildMapper()
-    {
-        var mapper = new BsonMapper();
-        mapper.Entity<GenerationSnapshot>();
-        return mapper;
-    }
-
-    /// <summary>Root directory for all clusters' sealed caches.</summary>
-    public string CacheRoot => _cacheRoot;
-
-    /// <summary>Initializes a new instance of the GenerationSealedCache class.</summary>
-    /// <param name="cacheRoot">The root directory for the cache.</param>
-    public GenerationSealedCache(string cacheRoot)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(cacheRoot);
-        _cacheRoot = cacheRoot;
-        Directory.CreateDirectory(_cacheRoot);
-    }
-
-    /// <summary>
-    ///     Seal a generation: write the snapshot to <c>&lt;cluster&gt;/&lt;generationId&gt;.db</c>,
-    ///     mark the file read-only, then atomically publish the <c>CURRENT</c> pointer. Existing
-    ///     sealed files for prior generations are preserved (prune separately).
-    /// </summary>
-    /// <param name="snapshot">The generation snapshot to seal.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task representing the asynchronous operation.</returns>
-    public async Task SealAsync(GenerationSnapshot snapshot, CancellationToken ct = default)
-    {
-        ArgumentNullException.ThrowIfNull(snapshot);
-        ct.ThrowIfCancellationRequested();
-
-        var clusterDir = Path.Combine(_cacheRoot, snapshot.ClusterId);
-        Directory.CreateDirectory(clusterDir);
-        var sealedPath = Path.Combine(clusterDir, $"{snapshot.GenerationId}.db");
-
-        if (File.Exists(sealedPath))
-        {
-            // Already sealed — idempotent. Treat as no-op + update pointer in case an earlier
-            // seal succeeded but the pointer update failed (crash recovery).
-            WritePointerAtomically(clusterDir, snapshot.GenerationId);
-            return;
-        }
-
-        var tmpPath = sealedPath + ".tmp";
-        try
-        {
-            using (var db = new LiteDatabase(new ConnectionString { Filename = tmpPath, Upgrade = false }, BuildMapper()))
-            {
-                var col = db.GetCollection<GenerationSnapshot>(CollectionName);
-                col.Insert(snapshot);
-            }
-
-            File.Move(tmpPath, sealedPath);
-            File.SetAttributes(sealedPath, File.GetAttributes(sealedPath) | FileAttributes.ReadOnly);
-            WritePointerAtomically(clusterDir, snapshot.GenerationId);
-        }
-        catch
-        {
-            try { if (File.Exists(tmpPath)) File.Delete(tmpPath); } catch { /* best-effort */ }
-            throw;
-        }
-
-        await Task.CompletedTask;
-    }
-
-    /// <summary>
-    ///     Read the current sealed snapshot for <paramref name="clusterId"/>. Throws
-    ///     <see cref="GenerationCacheUnavailableException"/> when the pointer is missing
-    ///     (first-boot-no-snapshot case) or when the sealed file is corrupt. Never silently
-    ///     falls back to a prior generation.
-    /// </summary>
-    /// <param name="clusterId">The cluster ID to read the snapshot for.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task representing the asynchronous operation containing the generation snapshot.</returns>
-    public Task<GenerationSnapshot> ReadCurrentAsync(string clusterId, CancellationToken ct = default)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
-        ct.ThrowIfCancellationRequested();
-
-        var clusterDir = Path.Combine(_cacheRoot, clusterId);
-        var pointerPath = Path.Combine(clusterDir, CurrentPointerFileName);
-        if (!File.Exists(pointerPath))
-            throw new GenerationCacheUnavailableException(
-                $"No sealed generation for cluster '{clusterId}' at '{clusterDir}'. First-boot case: the central DB must be reachable at least once before cache fallback is possible.");
-
-        long generationId;
-        try
-        {
-            var text = File.ReadAllText(pointerPath).Trim();
-            generationId = long.Parse(text, System.Globalization.CultureInfo.InvariantCulture);
-        }
-        catch (Exception ex)
-        {
-            throw new GenerationCacheUnavailableException(
-                $"CURRENT pointer at '{pointerPath}' is corrupt or unreadable.", ex);
-        }
-
-        var sealedPath = Path.Combine(clusterDir, $"{generationId}.db");
-        if (!File.Exists(sealedPath))
-            throw new GenerationCacheUnavailableException(
-                $"CURRENT points at generation {generationId} but '{sealedPath}' is missing — fails closed rather than serving an older generation.");
-
-        try
-        {
-            using var db = new LiteDatabase(new ConnectionString { Filename = sealedPath, ReadOnly = true }, BuildMapper());
-            var col = db.GetCollection<GenerationSnapshot>(CollectionName);
-            var snapshot = col.FindAll().FirstOrDefault()
-                ?? throw new GenerationCacheUnavailableException(
-                    $"Sealed file '{sealedPath}' contains no snapshot row — file is corrupt.");
-            return Task.FromResult(snapshot);
-        }
-        catch (GenerationCacheUnavailableException) { throw; }
-        catch (Exception ex) when (ex is LiteException or InvalidDataException or IOException
-                                      or NotSupportedException or FormatException)
-        {
-            throw new GenerationCacheUnavailableException(
-                $"Sealed file '{sealedPath}' is corrupt or unreadable — fails closed rather than falling back to an older generation.", ex);
-        }
-    }
-
-    /// <summary>Return the generation id the <c>CURRENT</c> pointer points at, or null if no pointer exists.</summary>
-    /// <param name="clusterId">The cluster ID to get the current generation ID for.</param>
-    /// <returns>The generation ID, or null if no pointer exists.</returns>
-    public long? TryGetCurrentGenerationId(string clusterId)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
-        var pointerPath = Path.Combine(_cacheRoot, clusterId, CurrentPointerFileName);
-        if (!File.Exists(pointerPath)) return null;
-        try
-        {
-            return long.Parse(File.ReadAllText(pointerPath).Trim(), System.Globalization.CultureInfo.InvariantCulture);
-        }
-        catch
-        {
-            return null;
-        }
-    }
-
-    private static void WritePointerAtomically(string clusterDir, long generationId)
-    {
-        var pointerPath = Path.Combine(clusterDir, CurrentPointerFileName);
-        var tmpPath = pointerPath + ".tmp";
-        File.WriteAllText(tmpPath, generationId.ToString(System.Globalization.CultureInfo.InvariantCulture));
-        if (File.Exists(pointerPath))
-            File.Replace(tmpPath, pointerPath, destinationBackupFileName: null);
-        else
-            File.Move(tmpPath, pointerPath);
-    }
-}
-
-/// <summary>Sealed cache is unreachable — caller must fail closed.</summary>
-public sealed class GenerationCacheUnavailableException : Exception
-{
-    /// <summary>Initializes a new instance of the GenerationCacheUnavailableException class.</summary>
-    /// <param name="message">The error message.</param>
-    public GenerationCacheUnavailableException(string message) : base(message) { }
-    /// <summary>Initializes a new instance of the GenerationCacheUnavailableException class with an inner exception.</summary>
-    /// <param name="message">The error message.</param>
-    /// <param name="inner">The inner exception.</param>
-    public GenerationCacheUnavailableException(string message, Exception inner) : base(message, inner) { }
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/GenerationSnapshot.cs

@@ -1,20 +0,0 @@
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     A self-contained snapshot of one generation — enough to rebuild the address space on a node
-///     that has lost DB connectivity. The payload is the JSON-serialized <c>sp_GetGenerationContent</c>
-///     result; the local cache doesn't inspect the shape, it just round-trips bytes.
-/// </summary>
-public sealed class GenerationSnapshot
-{
-    /// <summary>Gets or sets the auto-generated LiteDB ID.</summary>
-    public int Id { get; set; } // LiteDB auto-ID
-    /// <summary>Gets or sets the cluster identifier.</summary>
-    public required string ClusterId { get; set; }
-    /// <summary>Gets or sets the generation identifier.</summary>
-    public required long GenerationId { get; set; }
-    /// <summary>Gets or sets the time this snapshot was cached.</summary>
-    public required DateTime CachedAt { get; set; }
-    /// <summary>Gets or sets the JSON-serialized payload content.</summary>
-    public required string PayloadJson { get; set; }
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/ILocalConfigCache.cs

@@ -1,32 +0,0 @@
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Per-node local cache of the most-recently-applied generation(s). Used to bootstrap the
-///     address space when the central DB is unreachable (degraded-but-running).
-/// </summary>
-/// <remarks>
-///     <para><b>Concurrency contract:</b> implementations must serialize writes — specifically,
-///     <see cref="PutAsync"/> for the same <c>(ClusterId, GenerationId)</c> from concurrent
-///     callers must not produce duplicate rows. Reads may run concurrently with reads and writes.
-///     The <see cref="LiteDbConfigCache"/> implementation enforces this via an instance-level
-///     <see cref="SemaphoreSlim"/> around the find-then-insert/update window.</para>
-/// </remarks>
-public interface ILocalConfigCache
-{
-    /// <summary>Retrieves the most recent generation snapshot for the specified cluster.</summary>
-    /// <param name="clusterId">The cluster identifier.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>The most recent generation snapshot, or null if none exists.</returns>
-    Task<GenerationSnapshot?> GetMostRecentAsync(string clusterId, CancellationToken ct = default);
-    /// <summary>Stores a generation snapshot in the local cache.</summary>
-    /// <param name="snapshot">The generation snapshot to store.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task that represents the asynchronous operation.</returns>
-    Task PutAsync(GenerationSnapshot snapshot, CancellationToken ct = default);
-    /// <summary>Removes old generations, keeping only the most recent N.</summary>
-    /// <param name="clusterId">The cluster identifier.</param>
-    /// <param name="keepLatest">The number of latest generations to keep.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task that represents the asynchronous operation.</returns>
-    Task PruneOldGenerationsAsync(string clusterId, int keepLatest = 10, CancellationToken ct = default);
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/LiteDbConfigCache.cs

@@ -1,129 +0,0 @@
-using LiteDB;
-
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     LiteDB-backed <see cref="ILocalConfigCache"/>. One file per node (default
-///     <c>config_cache.db</c>), one collection per snapshot. Corruption surfaces as
-///     <see cref="LocalConfigCacheCorruptException"/> on construction or read — callers should
-///     delete and re-fetch from the central DB.
-/// </summary>
-public sealed class LiteDbConfigCache : ILocalConfigCache, IDisposable
-{
-    private const string CollectionName = "generations";
-
-    // LiteDB's default BsonMapper.Global is a process-wide singleton whose per-type member
-    // registration is lazy and NOT thread-safe across concurrently-constructed LiteDatabase
-    // instances. When several caches (this one + GenerationSealedCache) initialise in parallel
-    // the global mapper races, surfacing as "Member ClusterId not found on BsonMapper" or a
-    // bogus "duplicate key _id = 0" (the int auto-id mapping was lost so Insert writes a literal
-    // 0 twice). Give each database a private, pre-registered mapper so member
-    // resolution happens once, single-threaded, at construction and never touches the global.
-    private static BsonMapper BuildMapper()
-    {
-        var mapper = new BsonMapper();
-        mapper.Entity<GenerationSnapshot>();
-        return mapper;
-    }
-
-    private readonly LiteDatabase _db;
-    private readonly ILiteCollection<GenerationSnapshot> _col;
-    // PutAsync is a find-then-insert/update; without serialization, two concurrent puts for the
-    // same (ClusterId, GenerationId) can both observe `existing is null` and both Insert,
-    // producing duplicate rows. Serialize writes through this semaphore so
-    // the read-modify-write block is atomic for a given instance. LiteDB itself only locks the
-    // page-level write, not the find-then-insert window.
-    private readonly SemaphoreSlim _writeGate = new(initialCount: 1, maxCount: 1);
-
-    /// <summary>Initializes a new instance of the <see cref="LiteDbConfigCache"/> class.</summary>
-    /// <param name="dbPath">Path to the LiteDB database file.</param>
-    public LiteDbConfigCache(string dbPath)
-    {
-        // LiteDB can be tolerant of header-only corruption at construction time (it may overwrite
-        // the header and "recover"), so we force a write + read probe to fail fast on real corruption.
-        try
-        {
-            _db = new LiteDatabase(new ConnectionString { Filename = dbPath, Upgrade = true }, BuildMapper());
-            _col = _db.GetCollection<GenerationSnapshot>(CollectionName);
-            _col.EnsureIndex(s => s.ClusterId);
-            _col.EnsureIndex(s => s.GenerationId);
-            _ = _col.Count();
-        }
-        catch (Exception ex) when (ex is LiteException or InvalidDataException or IOException
-                                      or NotSupportedException or UnauthorizedAccessException
-                                      or ArgumentOutOfRangeException or FormatException)
-        {
-            _db?.Dispose();
-            throw new LocalConfigCacheCorruptException(
-                $"LiteDB cache at '{dbPath}' is corrupt or unreadable — delete the file and refetch from the central DB.",
-                ex);
-        }
-    }
-
-    /// <inheritdoc />
-    public Task<GenerationSnapshot?> GetMostRecentAsync(string clusterId, CancellationToken ct = default)
-    {
-        ct.ThrowIfCancellationRequested();
-        var snapshot = _col
-            .Find(s => s.ClusterId == clusterId)
-            .OrderByDescending(s => s.GenerationId)
-            .FirstOrDefault();
-        return Task.FromResult<GenerationSnapshot?>(snapshot);
-    }
-
-    /// <inheritdoc />
-    public async Task PutAsync(GenerationSnapshot snapshot, CancellationToken ct = default)
-    {
-        ct.ThrowIfCancellationRequested();
-        // Serialize the find-then-insert/update so concurrent callers do not observe a stale
-        // `existing is null` and both Insert. LiteDB's per-call lock is not enough — the
-        // read and the write are independent calls.
-        await _writeGate.WaitAsync(ct).ConfigureAwait(false);
-        try
-        {
-            // upsert by (ClusterId, GenerationId) — replace in place if already cached
-            var existing = _col
-                .Find(s => s.ClusterId == snapshot.ClusterId && s.GenerationId == snapshot.GenerationId)
-                .FirstOrDefault();
-
-            if (existing is null)
-                _col.Insert(snapshot);
-            else
-            {
-                snapshot.Id = existing.Id;
-                _col.Update(snapshot);
-            }
-        }
-        finally
-        {
-            _writeGate.Release();
-        }
-    }
-
-    /// <inheritdoc />
-    public Task PruneOldGenerationsAsync(string clusterId, int keepLatest = 10, CancellationToken ct = default)
-    {
-        ct.ThrowIfCancellationRequested();
-        var doomed = _col
-            .Find(s => s.ClusterId == clusterId)
-            .OrderByDescending(s => s.GenerationId)
-            .Skip(keepLatest)
-            .Select(s => s.Id)
-            .ToList();
-
-        foreach (var id in doomed)
-            _col.Delete(id);
-
-        return Task.CompletedTask;
-    }
-
-    /// <summary>Releases all resources used by the cache.</summary>
-    public void Dispose()
-    {
-        _writeGate.Dispose();
-        _db.Dispose();
-    }
-}
-
-public sealed class LocalConfigCacheCorruptException(string message, Exception inner)
-    : Exception(message, inner);

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/ResilientConfigReader.cs

@@ -1,140 +0,0 @@
-using System.Text.RegularExpressions;
-using Microsoft.Extensions.Logging;
-using Polly;
-using Polly.Retry;
-using Polly.Timeout;
-
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Wraps a central-DB fetch function with Phase 6.1 Stream D.2 resilience:
-///     <b>timeout 2 s → retry 3× jittered → fallback to sealed cache</b>. Maintains the
-///     <see cref="StaleConfigFlag"/> — fresh on central-DB success, stale on cache fallback.
-/// </summary>
-/// <remarks>
-///     <para>Read-path only per plan. The write path (draft save, publish) bypasses this
-///     wrapper entirely and fails hard on DB outage so inconsistent writes never land.</para>
-///
-///     <para>Fallback is triggered by <b>any exception</b> the fetch raises (central-DB
-///     unreachable, SqlException, timeout). If the sealed cache also fails (no pointer,
-///     corrupt file, etc.), <see cref="GenerationCacheUnavailableException"/> surfaces — caller
-///     must fail the current request (InitializeAsync for a driver, etc.).</para>
-/// </remarks>
-public sealed class ResilientConfigReader
-{
-    private readonly GenerationSealedCache _cache;
-    private readonly StaleConfigFlag _staleFlag;
-    private readonly ResiliencePipeline _pipeline;
-    private readonly ILogger<ResilientConfigReader> _logger;
-
-    /// <summary>Initializes a resilient config reader with the given cache and options.</summary>
-    /// <param name="cache">The sealed cache for fallback.</param>
-    /// <param name="staleFlag">The stale config flag to manage.</param>
-    /// <param name="logger">The logger instance.</param>
-    /// <param name="timeout">The timeout for central fetch (default 2s).</param>
-    /// <param name="retryCount">The number of retries (default 3).</param>
-    public ResilientConfigReader(
-        GenerationSealedCache cache,
-        StaleConfigFlag staleFlag,
-        ILogger<ResilientConfigReader> logger,
-        TimeSpan? timeout = null,
-        int retryCount = 3)
-    {
-        _cache = cache;
-        _staleFlag = staleFlag;
-        _logger = logger;
-        var builder = new ResiliencePipelineBuilder()
-            .AddTimeout(new TimeoutStrategyOptions { Timeout = timeout ?? TimeSpan.FromSeconds(2) });
-
-        if (retryCount > 0)
-        {
-            builder.AddRetry(new RetryStrategyOptions
-            {
-                MaxRetryAttempts = retryCount,
-                BackoffType = DelayBackoffType.Exponential,
-                UseJitter = true,
-                Delay = TimeSpan.FromMilliseconds(100),
-                MaxDelay = TimeSpan.FromSeconds(1),
-                // Handle ALL exceptions including OperationCanceledException. A SQL command-level
-                // timeout surfaces as TaskCanceledException (derives from OperationCanceledException)
-                // when the caller's token is NOT cancelled, and must be retried just like any other
-                // transient error. Polly itself checks the cancellation token between retries and
-                // stops with OperationCanceledException on genuine caller cancellation regardless of
-                // this predicate.
-                ShouldHandle = new PredicateBuilder().Handle<Exception>(),
-            });
-        }
-
-        _pipeline = builder.Build();
-    }
-
-    /// <summary>
-    ///     Redacts connection-string fragments (Password, User Id, Pwd, etc.)
-    ///     that a caller's exception message could carry. Conservative regex pass — anything
-    ///     matching <c>Key=Value</c> with a known credential key gets its value replaced.
-    /// </summary>
-    private static readonly Regex SecretsRegex = new(
-        @"(?ix)\b(Password|Pwd|User\s*Id|Uid|AccessToken|Authorization|Api[-_]?Key)\s*=\s*[^;,)\s]*",
-        RegexOptions.Compiled);
-
-    /// <summary>Redacts sensitive credential information from a message.</summary>
-    /// <param name="message">The message to scrub.</param>
-    /// <returns>The message with redacted credentials.</returns>
-    internal static string ScrubSecrets(string? message)
-    {
-        if (string.IsNullOrEmpty(message)) return message ?? string.Empty;
-        // Replace the entire matched fragment (key + value) with a redaction marker so the
-        // key name itself doesn't leak — log scrapers grep for "Password=" too.
-        return SecretsRegex.Replace(message, "[redacted credential]");
-    }
-
-    /// <summary>
-    ///     Executes a central fetch through the resilience pipeline. On full failure
-    ///     (post-retry), reads the sealed cache and extracts the requested shape.
-    /// </summary>
-    /// <typeparam name="T">The type of configuration to read.</typeparam>
-    /// <param name="clusterId">The cluster ID to fetch for.</param>
-    /// <param name="centralFetch">Function to fetch from central DB.</param>
-    /// <param name="fromSnapshot">Function to extract the config from a snapshot.</param>
-    /// <param name="cancellationToken">Cancellation token.</param>
-    /// <returns>The configuration of type T.</returns>
-    public async ValueTask<T> ReadAsync<T>(
-        string clusterId,
-        Func<CancellationToken, ValueTask<T>> centralFetch,
-        Func<GenerationSnapshot, T> fromSnapshot,
-        CancellationToken cancellationToken)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
-        ArgumentNullException.ThrowIfNull(centralFetch);
-        ArgumentNullException.ThrowIfNull(fromSnapshot);
-
-        try
-        {
-            var result = await _pipeline.ExecuteAsync(centralFetch, cancellationToken).ConfigureAwait(false);
-            _staleFlag.MarkFresh();
-            return result;
-        }
-        // Catch all exceptions that are NOT genuine caller cancellations. A SQL command-level
-        // timeout surfaces as TaskCanceledException (derives from OperationCanceledException)
-        // but the caller's token is NOT cancelled — we must fall back to the sealed cache for
-        // that case, not propagate. Only rethrow if the caller actually requested cancellation.
-        catch (Exception ex) when (ex is not OperationCanceledException || !cancellationToken.IsCancellationRequested)
-        {
-            // Do NOT pass the raw exception object — it carries the stack
-            // and inner-exception chain, and SqlException/wrapping delegates can surface
-            // connection-string fragments (Password=…, User Id=…) embedded in messages.
-            // Log only the exception type and a scrubbed message so secrets stay out of logs.
-            _logger.LogWarning(
-                "Central-DB read failed after retries ({ExceptionType}: {SanitizedMessage}); falling back to sealed cache for cluster {ClusterId}",
-                ex.GetType().Name,
-                ScrubSecrets(ex.Message),
-                clusterId);
-            // GenerationCacheUnavailableException surfaces intentionally — fails the caller's
-            // operation. StaleConfigFlag stays unchanged; the flag only flips when we actually
-            // served a cache snapshot.
-            var snapshot = await _cache.ReadCurrentAsync(clusterId, cancellationToken).ConfigureAwait(false);
-            _staleFlag.MarkStale();
-            return fromSnapshot(snapshot);
-        }
-    }
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/StaleConfigFlag.cs

@@ -1,20 +0,0 @@
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Thread-safe <c>UsingStaleConfig</c> signal per Phase 6.1 Stream D.3. Flips true whenever
-///     a read falls back to a sealed cache snapshot; flips false on the next successful central-DB
-///     round-trip. Surfaced on <c>/healthz</c> body and on the Admin <c>/hosts</c> page.
-/// </summary>
-public sealed class StaleConfigFlag
-{
-    private int _stale;
-
-    /// <summary>True when the last config read was served from the sealed cache, not the central DB.</summary>
-    public bool IsStale => Volatile.Read(ref _stale) != 0;
-
-    /// <summary>Mark the current config as stale (a read fell back to the cache).</summary>
-    public void MarkStale() => Volatile.Write(ref _stale, 1);
-
-    /// <summary>Mark the current config as fresh (a central-DB read succeeded).</summary>
-    public void MarkFresh() => Volatile.Write(ref _stale, 0);
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Migrations/20260722122301_AddClusterNodeTransportPorts.cs

@@ -0,0 +1,39 @@
+using Microsoft.EntityFrameworkCore.Migrations;
+
+#nullable disable
+
+namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
+{
+    /// <inheritdoc />
+    public partial class AddClusterNodeTransportPorts : Migration
+    {
+        /// <inheritdoc />
+        protected override void Up(MigrationBuilder migrationBuilder)
+        {
+            migrationBuilder.AddColumn<int>(
+                name: "AkkaPort",
+                table: "ClusterNode",
+                type: "int",
+                nullable: false,
+                defaultValue: 4053);
+
+            migrationBuilder.AddColumn<int>(
+                name: "GrpcPort",
+                table: "ClusterNode",
+                type: "int",
+                nullable: true);
+        }
+
+        /// <inheritdoc />
+        protected override void Down(MigrationBuilder migrationBuilder)
+        {
+            migrationBuilder.DropColumn(
+                name: "AkkaPort",
+                table: "ClusterNode");
+
+            migrationBuilder.DropColumn(
+                name: "GrpcPort",
+                table: "ClusterNode");
+        }
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Migrations/20260722125506_AddClusterNodeMaintenanceMode.cs

@@ -0,0 +1,29 @@
+using Microsoft.EntityFrameworkCore.Migrations;
+
+#nullable disable
+
+namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
+{
+    /// <inheritdoc />
+    public partial class AddClusterNodeMaintenanceMode : Migration
+    {
+        /// <inheritdoc />
+        protected override void Up(MigrationBuilder migrationBuilder)
+        {
+            migrationBuilder.AddColumn<bool>(
+                name: "MaintenanceMode",
+                table: "ClusterNode",
+                type: "bit",
+                nullable: false,
+                defaultValue: false);
+        }
+
+        /// <inheritdoc />
+        protected override void Down(MigrationBuilder migrationBuilder)
+        {
+            migrationBuilder.DropColumn(
+                name: "MaintenanceMode",
+                table: "ClusterNode");
+        }
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Migrations/OtOpcUaConfigDbContextModelSnapshot.cs

@@ -47,6 +47,11 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
                         .HasMaxLength(64)
                         .HasColumnType("nvarchar(64)");
 
+                    b.Property<int>("AkkaPort")
+                        .ValueGeneratedOnAdd()
+                        .HasColumnType("int")
+                        .HasDefaultValue(4053);
+
                     b.Property<string>("ApplicationUri")
                         .IsRequired()
                         .HasMaxLength(256)
@@ -76,6 +81,9 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
                     b.Property<bool>("Enabled")
                         .HasColumnType("bit");
 
+                    b.Property<int?>("GrpcPort")
+                        .HasColumnType("int");
+
                     b.Property<string>("Host")
                         .IsRequired()
                         .HasMaxLength(255)
@@ -84,6 +92,9 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
                     b.Property<DateTime?>("LastSeenAt")
                         .HasColumnType("datetime2(3)");
 
+                    b.Property<bool>("MaintenanceMode")
+                        .HasColumnType("bit");
+
                     b.Property<int>("OpcUaPort")
                         .HasColumnType("int");
 

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/OtOpcUaConfigDbContext.cs

@@ -142,6 +142,11 @@ public sealed class OtOpcUaConfigDbContext(DbContextOptions<OtOpcUaConfigDbConte
             e.Property(x => x.Host).HasMaxLength(255);
             e.Property(x => x.ApplicationUri).HasMaxLength(256);
             e.Property(x => x.DriverConfigOverridesJson).HasColumnType("nvarchar(max)");
+            // Central's dial targets (per-cluster mesh Phase 1). AkkaPort carries a DB-side default so
+            // rows migrated from before the column existed come out at 4053 rather than 0 — every node
+            // in the fleet does listen on a remoting port, so 0 is never a truthful value for it.
+            // GrpcPort has no default: nothing listens on it until Phase 5.
+            e.Property(x => x.AkkaPort).HasDefaultValue(4053);
             e.Property(x => x.LastSeenAt).HasColumnType("datetime2(3)");
             e.Property(x => x.CreatedAt).HasColumnType("datetime2(3)").HasDefaultValueSql("SYSUTCDATETIME()");
             e.Property(x => x.CreatedBy).HasMaxLength(128);

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Services/ILdapGroupRoleMappingService.cs

@@ -10,10 +10,18 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Services;
 ///     Phase 6.2 compliance check on control/data-plane separation).
 /// </summary>
 /// <remarks>
-///     Per Phase 6.2 Stream A.2 this service is expected to run behind the Phase 6.1
-///     <c>ResilientConfigReader</c> pipeline (timeout → retry → fallback-to-cache) so a
-///     transient DB outage during sign-in falls back to the sealed snapshot rather than
-///     denying every login.
+///     <para>
+///         This service has no local-cache fallback: a DB outage during sign-in denies logins.
+///     </para>
+///     <para>
+///         The Phase 6.1 <c>ResilientConfigReader</c> pipeline (timeout → retry →
+///         fallback-to-sealed-snapshot) this was once expected to run behind was never wired to
+///         anything and was deleted along with the rest of the dormant LiteDB local cache, which
+///         <c>ZB.MOM.WW.LocalDb</c> supersedes. LocalDb caches the deployed-configuration artifact
+///         for driver-role nodes; it does not currently cover admin-plane reads like this one.
+///         Reviving the fallback means adding an admin-side cache on LocalDb, not restoring the
+///         old pipeline.
+///     </para>
 /// </remarks>
 public interface ILdapGroupRoleMappingService
 {

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/ZB.MOM.WW.OtOpcUa.Configuration.csproj

@@ -24,9 +24,12 @@
             <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
         </PackageReference>
         <PackageReference Include="Microsoft.AspNetCore.DataProtection.EntityFrameworkCore"/>
+        <!-- Surgical transitive pin — see the System.Security.Cryptography.Xml block in
+             Directory.Packages.props. This is the one project where the DataProtection chain
+             enters the repo; every other consumer reaches it through a ProjectReference here. -->
+        <PackageReference Include="System.Security.Cryptography.Xml"/>
         <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions"/>
         <PackageReference Include="Microsoft.Extensions.Logging.Abstractions"/>
-        <PackageReference Include="LiteDB"/>
         <PackageReference Include="Polly.Core"/>
     </ItemGroup>
 

src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmSfSchema.cs

@@ -0,0 +1,105 @@
+using System.Security.Cryptography;
+using System.Text;
+using Microsoft.Data.Sqlite;
+
+namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
+
+/// <summary>
+///     DDL for the alarm store-and-forward buffer: one row per alarm event awaiting delivery to
+///     the historian gateway.
+/// </summary>
+/// <remarks>
+///     <para>
+///         Replaces the standalone <c>alarm-historian.db</c> the sink used to own outright. Living
+///         in the consolidated LocalDb file is what lets the buffer replicate to the redundant pair
+///         peer, so a node that dies with undelivered alarm history no longer takes it to the grave.
+///     </para>
+///     <para>
+///         Deliberately depends on nothing but <see cref="SqliteConnection"/> so it can be applied
+///         to any connection — the host's <c>LocalDbSetup.OnReady</c> in production, and a bare
+///         connection in a test — without dragging the DI graph along. Mirrors
+///         <c>DeploymentCacheSchema</c>.
+///     </para>
+///     <para>
+///         <b>Why <see cref="IdColumn"/> is TEXT and app-minted.</b> The legacy queue keyed on
+///         <c>RowId INTEGER PRIMARY KEY AUTOINCREMENT</c>. Convergence is last-writer-wins over the
+///         primary key, so two nodes independently allocating rowid 7 for different alarms would
+///         silently overwrite one another. The sink mints the id from a hash of the payload, which
+///         additionally makes the same event converge to one row when both nodes of a pair enqueue
+///         it — as they legitimately do in the window before the first redundancy snapshot arrives.
+///     </para>
+/// </remarks>
+public static class AlarmSfSchema
+{
+    /// <summary>Table holding queued alarm events awaiting delivery.</summary>
+    public const string EventsTable = "alarm_sf_events";
+
+    /// <summary>The single primary-key column.</summary>
+    public const string IdColumn = "id";
+
+    /// <summary>
+    ///     Derives a row's primary key from its serialized payload.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         Deterministic rather than a fresh GUID, so that the same event arriving twice
+    ///         converges to one row under last-writer-wins instead of duplicating. That happens for
+    ///         real in two places: <c>HistorianAdapterActor</c> default-writes while its redundancy
+    ///         role is unknown, so both nodes of a pair accept the same fanned transition in the
+    ///         window before the first snapshot; and both nodes independently migrate their own
+    ///         legacy queue file, which for a warm pair holds overlapping history.
+    ///     </para>
+    ///     <para>
+    ///         Two genuinely distinct events cannot collide. <c>AlarmHistorianEvent</c> carries a
+    ///         full-precision timestamp alongside the alarm id, transition kind, message and user,
+    ///         so an equal hash means an equal event.
+    ///     </para>
+    /// </remarks>
+    /// <param name="payloadJson">The serialized <c>AlarmHistorianEvent</c>.</param>
+    /// <returns>The row's primary key.</returns>
+    public static string DeriveId(string payloadJson)
+    {
+        ArgumentNullException.ThrowIfNull(payloadJson);
+        return Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(payloadJson)));
+    }
+
+    /// <summary>
+    ///     Creates the buffer table if it does not already exist. Idempotent.
+    /// </summary>
+    /// <param name="connection">
+    ///     An already-open connection. <c>ILocalDb.CreateConnection()</c> hands out open,
+    ///     pragma-configured connections — do not call <c>Open()</c> on one.
+    /// </param>
+    public static void Apply(SqliteConnection connection)
+    {
+        ArgumentNullException.ThrowIfNull(connection);
+
+        using var cmd = connection.CreateCommand();
+
+        // Columns map 1:1 onto the legacy Queue table so the one-time migrator is a straight copy
+        // and the drain keeps its existing semantics: dead_lettered is the same 0/1 flag, and an
+        // acknowledged row is DELETEd rather than marked. Deleting keeps the table bounded without
+        // a second sweeper, and the replication engine carries the delete as a tombstone, so the
+        // peer drops its copy too.
+        //
+        // The drain index orders by enqueued_at_utc because a hashed TEXT id carries no insertion
+        // order the way the legacy AUTOINCREMENT RowId did. Timestamps are round-trip ("O") format,
+        // so lexicographic ordering is chronological; id breaks ties so the order is total and the
+        // index covers the drain's read.
+        cmd.CommandText = """
+            CREATE TABLE IF NOT EXISTS alarm_sf_events (
+                id               TEXT    NOT NULL PRIMARY KEY,
+                alarm_id         TEXT    NOT NULL,
+                enqueued_at_utc  TEXT    NOT NULL,
+                payload_json     TEXT    NOT NULL,
+                attempt_count    INTEGER NOT NULL DEFAULT 0,
+                last_attempt_utc TEXT    NULL,
+                last_error       TEXT    NULL,
+                dead_lettered    INTEGER NOT NULL DEFAULT 0
+            );
+            CREATE INDEX IF NOT EXISTS ix_alarm_sf_events_drain
+                ON alarm_sf_events (dead_lettered, enqueued_at_utc, id);
+            """;
+        cmd.ExecuteNonQuery();
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/IAlarmHistorianSink.cs

@@ -4,13 +4,13 @@ namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
 ///     The historian sink contract — where qualifying alarm events land. Ingestion routes
 ///     through the HistorianGateway alarm writer (the gateway's <c>SendEvent</c> gRPC path)
 ///     behind the durable store-and-forward queue. Tests use an in-memory fake; production uses
-///     <see cref="SqliteStoreAndForwardSink"/>.
+///     <see cref="LocalDbStoreAndForwardSink"/>.
 /// </summary>
 /// <remarks>
 ///     <para>
 ///         <see cref="EnqueueAsync"/> is fire-and-forget from the engine's perspective —
 ///         the sink MUST NOT block the emitting thread. Production implementations
-///         (<see cref="SqliteStoreAndForwardSink"/>) persist to a local SQLite queue
+///         (<see cref="LocalDbStoreAndForwardSink"/>) persist to the node's local SQLite queue
 ///         first, then drain asynchronously to the actual historian. Per the Phase 7 plan,
 ///         failed downstream writes replay with exponential backoff;
 ///         operator actions are never blocked waiting on the historian.
@@ -79,6 +79,18 @@ public enum HistorianDrainState
     Idle,
     Draining,
     BackingOff,
+
+    /// <summary>
+    ///     Ticking, but not draining: this node does not hold the Primary role, so it leaves the
+    ///     (replicated) queue for the node that does.
+    /// </summary>
+    /// <remarks>
+    ///     Distinct from <see cref="Idle"/> so a rising queue depth on a Secondary reads as the
+    ///     designed behaviour rather than a stalled drain. It is also how a misconfigured pair
+    ///     becomes diagnosable: if BOTH nodes report this, no one is draining and alarm history is
+    ///     silently accumulating toward the capacity ceiling.
+    /// </remarks>
+    NotPrimary,
 }
 
 /// <summary>Returned by the historian alarm writer per event — drain worker uses this to decide retry cadence.</summary>

src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/LocalDbStoreAndForwardSink.cs

@@ -1,53 +1,54 @@
 using System.Text.Json;
-using Microsoft.Data.Sqlite;
 using Serilog;
+using ZB.MOM.WW.LocalDb;
 
 namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
 
 /// <summary>
-///     Durable SQLite queue on the node absorbs every qualifying alarm event, a drain
-///     worker batches rows to the Wonderware historian sidecar via
-///     <see cref="IAlarmHistorianWriter"/> on an exponential-backoff cadence, and
-///     operator acks never block on the historian being reachable.
+///     Durable queue in the node's consolidated LocalDb absorbs every qualifying alarm event, a
+///     drain worker batches rows to the historian gateway via <see cref="IAlarmHistorianWriter"/>
+///     on an exponential-backoff cadence, and operator acks never block on the historian being
+///     reachable.
 /// </summary>
 /// <remarks>
 ///     <para>
-///         Queue schema:
-///         <code>
-///         CREATE TABLE Queue (
-///             RowId INTEGER PRIMARY KEY AUTOINCREMENT,
-///             AlarmId TEXT NOT NULL,
-///             EnqueuedUtc TEXT NOT NULL,
-///             PayloadJson TEXT NOT NULL,
-///             AttemptCount INTEGER NOT NULL DEFAULT 0,
-///             LastAttemptUtc TEXT NULL,
-///             LastError TEXT NULL,
-///             DeadLettered INTEGER NOT NULL DEFAULT 0
-///         );
-///         </code>
-///         Dead-lettered rows stay in place for the configured retention window (default
-///         30 days) so operators can inspect + manually
-///         retry before the sweeper purges them. Regular queue capacity is bounded —
-///         overflow evicts the oldest non-dead-lettered rows with a WARN log. The
-///         durability guarantee is therefore bounded by <see cref="DefaultCapacity"/>:
-///         under a sustained historian outage, accepted events may be evicted before
-///         delivery. The <see cref="HistorianSinkStatus.EvictedCount"/> counter makes
-///         overflow visible to operators without requiring the WARN log to be scraped.
+///         Rows live in <c>alarm_sf_events</c> (see <see cref="AlarmSfSchema"/>), which the host
+///         registers for replication. That is the point of this type living on
+///         <see cref="ILocalDb"/> rather than owning its own file: the buffer mirrors to the
+///         redundant pair peer, so a node that dies holding undelivered alarm history no longer
+///         takes it to the grave.
+///     </para>
+///     <para>
+///         <b>Only one node of a pair may drain.</b> Replication puts the Primary's queued rows in
+///         the Secondary's table too; an ungated drain there would re-deliver every event,
+///         continuously. The <c>drainGate</c> constructor argument is how the caller scopes
+///         draining to the Primary. Enqueue is gated separately and upstream, by
+///         <c>HistorianAdapterActor</c>.
+///     </para>
+///     <para>
+///         Dead-lettered rows stay in place for the configured retention window (default 30 days)
+///         so operators can inspect + manually retry before the sweeper purges them. Regular queue
+///         capacity is bounded — overflow evicts the oldest non-dead-lettered rows with a WARN log.
+///         The durability guarantee is therefore bounded by <see cref="DefaultCapacity"/>: under a
+///         sustained historian outage, accepted events may be evicted before delivery. The
+///         <see cref="HistorianSinkStatus.EvictedCount"/> counter makes overflow visible to
+///         operators without requiring the WARN log to be scraped.
 ///     </para>
 ///     <para>
 ///         Drain runs on a self-rescheduling one-shot <see cref="System.Threading.Timer"/>.
 ///         Exponential backoff on <see cref="HistorianWriteOutcome.RetryPlease"/>:
-///         1s → 2s → 5s → 15s → 60s cap — the backoff is applied to the timer's next
-///         due-time, so a historian outage genuinely slows the drain cadence.
-///         <see cref="HistorianWriteOutcome.PermanentFail"/> rows flip
-///         the <c>DeadLettered</c> flag on the individual row; neighbors in the batch
-///         still retry on their own cadence.
+///         1s → 2s → 5s → 15s → 60s cap — the backoff is applied to the timer's next due-time, so a
+///         historian outage genuinely slows the drain cadence.
+///         <see cref="HistorianWriteOutcome.PermanentFail"/> rows flip the <c>dead_lettered</c> flag
+///         on the individual row; neighbors in the batch still retry on their own cadence.
 ///     </para>
 /// </remarks>
-public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
+public sealed class LocalDbStoreAndForwardSink : IAlarmHistorianSink, IDisposable
 {
     /// <summary>Default queue capacity — oldest non-dead-lettered rows evicted past this.</summary>
     public const long DefaultCapacity = 1_000_000;
+
+    /// <summary>Default window dead-lettered rows are retained for before the sweeper purges them.</summary>
     public static readonly TimeSpan DefaultDeadLetterRetention = TimeSpan.FromDays(30);
 
     /// <summary>Default max delivery attempts before a perpetually-retrying (poison) row is dead-lettered.</summary>
@@ -62,7 +63,7 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
         TimeSpan.FromSeconds(60),
     ];
 
-    private readonly string _connectionString;
+    private readonly ILocalDb _db;
     private readonly IAlarmHistorianWriter _writer;
     private readonly ILogger _logger;
     private readonly int _batchSize;
@@ -70,8 +71,9 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     private readonly TimeSpan _deadLetterRetention;
     private readonly int _maxAttempts;
     private readonly Func<DateTime> _clock;
+    private readonly Func<bool> _drainGate;
 
-    private readonly SemaphoreSlim _drainGate = new(1, 1);
+    private readonly SemaphoreSlim _drainGateLock = new(1, 1);
     private Timer? _drainTimer;
     private TimeSpan _tickInterval;
     private volatile int _backoffIndex;
@@ -85,13 +87,16 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     private string? _lastError;
     private HistorianDrainState _drainState = HistorianDrainState.Idle;
     private long _evictedCount;
+    // Tracks the last gate decision so a denial is logged on the transition rather than on every
+    // tick. Null until the first drain runs.
+    private bool? _lastGateDecision;
 
     private long _queuedRowCount;
     // Probe counter — incremented every time we actually issue a real COUNT(*) for
     // capacity enforcement. Public for test instrumentation only.
     private long _capacityProbeCount;
     // After every Nth enqueue we resync the in-memory counter from storage to defend
-    // against silent drift (e.g. an external process editing the DB).
+    // against silent drift (e.g. the peer replicating rows in underneath us).
     private const long ResyncEnqueueInterval = 10_000;
     private long _enqueuesSinceResync;
 
@@ -99,9 +104,13 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     public long DebugCapacityProbeCount => Interlocked.Read(ref _capacityProbeCount);
 
     /// <summary>
-    ///     Initializes a new instance of the <see cref="SqliteStoreAndForwardSink"/> class with the specified configuration.
+    ///     Initializes a new instance of the <see cref="LocalDbStoreAndForwardSink"/> class.
     /// </summary>
-    /// <param name="databasePath">The filesystem path to the SQLite database file.</param>
+    /// <param name="db">
+    ///     The node's local database. Its <c>alarm_sf_events</c> table must already exist and be
+    ///     registered for replication — the host does both in <c>LocalDbSetup.OnReady</c>, before
+    ///     any consumer can resolve this sink.
+    /// </param>
     /// <param name="writer">The alarm historian writer to handle batch forwarding.</param>
     /// <param name="logger">The logger for diagnostic output.</param>
     /// <param name="batchSize">The maximum number of rows to forward in a single batch. Defaults to 100.</param>
@@ -109,18 +118,24 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     /// <param name="deadLetterRetention">The timespan to retain dead-lettered rows before purging. Defaults to 30 days.</param>
     /// <param name="maxAttempts">The maximum number of delivery attempts before a perpetually-retrying (poison) row is dead-lettered. Defaults to 10.</param>
     /// <param name="clock">Optional clock function for testing; defaults to <see cref="DateTime.UtcNow"/>.</param>
-    public SqliteStoreAndForwardSink(
-        string databasePath,
+    /// <param name="drainGate">
+    ///     Consulted at the top of every drain tick; <c>false</c> skips the tick entirely, leaving
+    ///     the queue intact. Callers running a redundant pair MUST supply a gate that is true on at
+    ///     most one node, because the queue replicates — see the class remarks. The default
+    ///     (<c>null</c> ⇒ always drain) is the correct single-node and test posture.
+    /// </param>
+    public LocalDbStoreAndForwardSink(
+        ILocalDb db,
         IAlarmHistorianWriter writer,
         ILogger logger,
         int batchSize = 100,
         long capacity = DefaultCapacity,
         TimeSpan? deadLetterRetention = null,
         int maxAttempts = DefaultMaxAttempts,
-        Func<DateTime>? clock = null)
+        Func<DateTime>? clock = null,
+        Func<bool>? drainGate = null)
     {
-        if (string.IsNullOrWhiteSpace(databasePath))
-            throw new ArgumentException("Database path required.", nameof(databasePath));
+        _db = db ?? throw new ArgumentNullException(nameof(db));
         _writer = writer ?? throw new ArgumentNullException(nameof(writer));
         _logger = logger ?? throw new ArgumentNullException(nameof(logger));
         _batchSize = batchSize > 0 ? batchSize : throw new ArgumentOutOfRangeException(nameof(batchSize));
@@ -128,50 +143,11 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
         _deadLetterRetention = deadLetterRetention ?? DefaultDeadLetterRetention;
         _maxAttempts = maxAttempts > 0 ? maxAttempts : throw new ArgumentOutOfRangeException(nameof(maxAttempts));
         _clock = clock ?? (() => DateTime.UtcNow);
-        // DefaultTimeout gives ADO.NET command-level retry; the PRAGMA busy_timeout
-        // applied in OpenConnection backs it with SQLite's own busy-handler so an
-        // enqueue/drain collision waits out the file lock instead of throwing
-        // SQLITE_BUSY immediately.
-        _connectionString = new SqliteConnectionStringBuilder
-        {
-            DataSource = databasePath,
-            DefaultTimeout = 5,
-        }.ToString();
+        _drainGate = drainGate ?? (static () => true);
 
-        InitializeSchema();
         _queuedRowCount = ProbeQueuedRowCount();
     }
 
-    /// <summary>
-    ///     Open a connection with the busy timeout + WAL journal applied. SQLite
-    ///     serializes writers with a file lock; the busy_timeout lets a writer wait
-    ///     out a competing lock (default is 0 — fail fast), and WAL lets readers and
-    ///     the single writer proceed without blocking each other.
-    /// </summary>
-    private SqliteConnection OpenConnection()
-    {
-        var conn = new SqliteConnection(_connectionString);
-        conn.Open();
-        ApplyPragmas(conn);
-        return conn;
-    }
-
-    /// <summary>Apply busy_timeout + WAL pragmas to an already-open connection (sync).</summary>
-    private static void ApplyPragmas(SqliteConnection conn)
-    {
-        using var pragma = conn.CreateCommand();
-        pragma.CommandText = "PRAGMA busy_timeout=5000; PRAGMA journal_mode=WAL;";
-        pragma.ExecuteNonQuery();
-    }
-
-    /// <summary>Apply busy_timeout + WAL pragmas to an already-open connection (async).</summary>
-    private static async Task ApplyPragmasAsync(SqliteConnection conn, CancellationToken ct)
-    {
-        using var pragma = conn.CreateCommand();
-        pragma.CommandText = "PRAGMA busy_timeout=5000; PRAGMA journal_mode=WAL;";
-        await pragma.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
-    }
-
     /// <summary>
     ///     Start the background drain worker. Not started automatically so tests can
     ///     drive <see cref="DrainOnceAsync"/> deterministically.
@@ -188,7 +164,7 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     /// <param name="tickInterval">The base interval between drain attempts.</param>
     public void StartDrainLoop(TimeSpan tickInterval)
     {
-        if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
+        if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
         _tickInterval = tickInterval;
         _drainTimer?.Dispose();
         // One-shot: dueTime = tickInterval, period = Infinite. RescheduleDrain re-arms
@@ -239,25 +215,31 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     public async Task EnqueueAsync(AlarmHistorianEvent evt, CancellationToken cancellationToken)
     {
         if (evt is null) throw new ArgumentNullException(nameof(evt));
-        if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
+        if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
 
-        using var conn = new SqliteConnection(_connectionString);
-        await conn.OpenAsync(cancellationToken).ConfigureAwait(false);
-        await ApplyPragmasAsync(conn, cancellationToken).ConfigureAwait(false);
+        await EnforceCapacityFastPathAsync(cancellationToken).ConfigureAwait(false);
 
-        await EnforceCapacityFastPathAsync(conn, cancellationToken).ConfigureAwait(false);
+        var payload = JsonSerializer.Serialize(evt);
 
-        using var cmd = conn.CreateCommand();
-        cmd.CommandText = """
-            INSERT INTO Queue (AlarmId, EnqueuedUtc, PayloadJson, AttemptCount)
-            VALUES ($alarmId, $enqueued, $payload, 0);
-            """;
-        cmd.Parameters.AddWithValue("$alarmId", evt.AlarmId);
-        cmd.Parameters.AddWithValue("$enqueued", _clock().ToString("O"));
-        cmd.Parameters.AddWithValue("$payload", JsonSerializer.Serialize(evt));
-        await cmd.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
+        // ON CONFLICT DO NOTHING: re-enqueuing an identical event (the pair's boot-window
+        // double-accept) must not resurrect a row the drain already bumped or dead-lettered.
+        var inserted = await _db.ExecuteAsync(
+            """
+            INSERT INTO alarm_sf_events
+                (id, alarm_id, enqueued_at_utc, payload_json, attempt_count)
+            VALUES (@Id, @AlarmId, @EnqueuedAtUtc, @PayloadJson, 0)
+            ON CONFLICT(id) DO NOTHING
+            """,
+            new
+            {
+                Id = AlarmSfSchema.DeriveId(payload),
+                AlarmId = evt.AlarmId,
+                EnqueuedAtUtc = _clock().ToString("O"),
+                PayloadJson = payload,
+            },
+            cancellationToken).ConfigureAwait(false);
 
-        Interlocked.Increment(ref _queuedRowCount);
+        if (inserted > 0) Interlocked.Increment(ref _queuedRowCount);
     }
 
     /// <summary>
@@ -268,15 +250,17 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     ///     through <see cref="EnforceCapacityAsync"/> which still runs a precise
     ///     COUNT to compute the exact number of rows to evict.
     /// </summary>
-    private async Task EnforceCapacityFastPathAsync(SqliteConnection conn, CancellationToken ct)
+    private async Task EnforceCapacityFastPathAsync(CancellationToken ct)
     {
         var enqueuesSinceResync = Interlocked.Increment(ref _enqueuesSinceResync);
         var cached = Interlocked.Read(ref _queuedRowCount);
 
-        // Periodic resync — bounded amount of drift even under exotic conditions.
+        // Periodic resync — bounded amount of drift even under exotic conditions. Now doubly
+        // warranted: replication applies the peer's rows straight into the table, so the counter
+        // has a second way to fall behind that no local code path can observe.
         if (enqueuesSinceResync >= ResyncEnqueueInterval)
         {
-            await ResyncQueuedRowCountAsync(conn, ct).ConfigureAwait(false);
+            await ResyncQueuedRowCountAsync(ct).ConfigureAwait(false);
             cached = Interlocked.Read(ref _queuedRowCount);
             Interlocked.Exchange(ref _enqueuesSinceResync, 0);
         }
@@ -286,55 +270,70 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
 
         // Cached counter says we're at or above the capacity wall — fall back to the
         // precise path which probes COUNT(*) and evicts whatever's needed.
-        await EnforceCapacityAsync(conn, ct).ConfigureAwait(false);
+        await EnforceCapacityAsync(ct).ConfigureAwait(false);
     }
 
     /// <summary>Synchronously query <c>COUNT(*)</c> of non-dead-lettered rows. Used at startup.</summary>
     private long ProbeQueuedRowCount()
     {
         Interlocked.Increment(ref _capacityProbeCount);
-        using var conn = OpenConnection();
+        using var conn = _db.CreateConnection();
         using var cmd = conn.CreateCommand();
-        cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
+        cmd.CommandText = "SELECT COUNT(*) FROM alarm_sf_events WHERE dead_lettered = 0";
         return (long)(cmd.ExecuteScalar() ?? 0L);
     }
 
     /// <summary>Re-sync the in-memory counter from storage (async path).</summary>
-    private async Task ResyncQueuedRowCountAsync(SqliteConnection conn, CancellationToken ct)
+    private async Task ResyncQueuedRowCountAsync(CancellationToken ct)
     {
         Interlocked.Increment(ref _capacityProbeCount);
-        using var cmd = conn.CreateCommand();
-        cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
-        var live = (long)(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false) ?? 0L);
+        var live = await CountAsync("dead_lettered = 0", ct).ConfigureAwait(false);
         Interlocked.Exchange(ref _queuedRowCount, live);
     }
 
+    private async Task<long> CountAsync(string predicate, CancellationToken ct)
+    {
+        var rows = await _db.QueryAsync(
+            $"SELECT COUNT(*) FROM alarm_sf_events WHERE {predicate}",
+            r => r.GetInt64(0),
+            parameters: null,
+            ct).ConfigureAwait(false);
+        return rows.Count > 0 ? rows[0] : 0L;
+    }
+
     /// <summary>
     ///     Read up to <see cref="_batchSize"/> queued rows, forward through the writer,
     ///     remove Ack'd rows, dead-letter PermanentFail rows, and extend the backoff
     ///     on RetryPlease. Safe to call from multiple threads; the semaphore enforces
     ///     serial execution.
     /// </summary>
+    /// <remarks>
+    ///     Returns without touching the queue when the drain gate is closed. On a redundant pair
+    ///     that is the Secondary's steady state: its table fills by replication and stays put until
+    ///     it is promoted.
+    /// </remarks>
     /// <param name="ct">Cancellation token for the operation.</param>
     /// <returns>A task that represents the asynchronous operation.</returns>
     public async Task DrainOnceAsync(CancellationToken ct)
     {
         if (_disposed) return;
-        if (!await _drainGate.WaitAsync(0, ct).ConfigureAwait(false)) return;
+        if (!await _drainGateLock.WaitAsync(0, ct).ConfigureAwait(false)) return;
         try
         {
+            if (!EvaluateDrainGate())
+            {
+                lock (_statusLock) { _drainState = HistorianDrainState.NotPrimary; }
+                return;
+            }
+
             lock (_statusLock)
             {
                 _drainState = HistorianDrainState.Draining;
                 _lastDrainUtc = _clock();
             }
 
-            // One connection per drain tick — used by purge, read, corrupt-dead-letter,
-            // and the outcome-applying transaction.
-            using var conn = OpenConnection();
-
-            PurgeAgedDeadLetters(conn);
-            var batch = ReadBatch(conn);
+            await PurgeAgedDeadLettersAsync(ct).ConfigureAwait(false);
+            var batch = await ReadBatchAsync(ct).ConfigureAwait(false);
             if (batch.Count == 0)
             {
                 lock (_statusLock) { _drainState = HistorianDrainState.Idle; }
@@ -342,22 +341,24 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
             }
 
             // A null/un-deserializable payload can never succeed — dead-letter it
-            // immediately for its own RowId so it cannot stall the queue head, and
+            // immediately for its own id so it cannot stall the queue head, and
             // exclude it from the batch handed to the writer.
-            var corruptRowIds = batch.Where(r => r.Event is null).Select(r => r.RowId).ToList();
+            var corruptIds = batch.Where(r => r.Event is null).Select(r => r.Id).ToList();
             var liveRows = batch.Where(r => r.Event is not null).ToList();
             var events = liveRows.Select(r => r.Event!).ToList();
 
-            if (corruptRowIds.Count > 0)
+            if (corruptIds.Count > 0)
             {
-                using var corruptTx = conn.BeginTransaction();
-                foreach (var rowId in corruptRowIds)
-                    DeadLetterRow(conn, corruptTx, rowId, $"corrupt payload at {_clock():O}");
-                corruptTx.Commit();
-                Interlocked.Add(ref _queuedRowCount, -corruptRowIds.Count);
+                await using (var corruptTx = await _db.BeginTransactionAsync(ct).ConfigureAwait(false))
+                {
+                    foreach (var id in corruptIds)
+                        await DeadLetterRowAsync(corruptTx, id, $"corrupt payload at {_clock():O}", ct).ConfigureAwait(false);
+                    await corruptTx.CommitAsync(ct).ConfigureAwait(false);
+                }
+                Interlocked.Add(ref _queuedRowCount, -corruptIds.Count);
                 _logger.Warning(
                     "Dead-lettered {Count} historian queue row(s) with un-deserializable payload",
-                    corruptRowIds.Count);
+                    corruptIds.Count);
             }
 
             if (events.Count == 0)
@@ -404,41 +405,44 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
             }
 
             int rowsLeavingQueue = 0;
-            using (var tx = conn.BeginTransaction())
+            await using (var tx = await _db.BeginTransactionAsync(ct).ConfigureAwait(false))
             {
                 for (var i = 0; i < outcomes.Count; i++)
                 {
                     var outcome = outcomes[i];
-                    var rowId = liveRows[i].RowId;
+                    var id = liveRows[i].Id;
                     switch (outcome)
                     {
                         case HistorianWriteOutcome.Ack:
-                            DeleteRow(conn, tx, rowId);
+                            // Deleted, not marked delivered. The replication engine carries the
+                            // delete as a tombstone, so the peer drops its copy and cannot
+                            // re-deliver the row after a failover.
+                            await DeleteRowAsync(tx, id, ct).ConfigureAwait(false);
                             rowsLeavingQueue++;
                             break;
                         case HistorianWriteOutcome.PermanentFail:
-                            DeadLetterRow(conn, tx, rowId, $"permanent fail at {_clock():O}");
+                            await DeadLetterRowAsync(tx, id, $"permanent fail at {_clock():O}", ct).ConfigureAwait(false);
                             rowsLeavingQueue++;
                             break;
                         case HistorianWriteOutcome.RetryPlease:
                             // finding 002: cap retries so a perpetually-RetryPlease (poison)
                             // row cannot retry forever at the 60s backoff floor. The incoming
-                            // AttemptCount is the count BEFORE this attempt; +1 accounts for the
+                            // attempt count is the count BEFORE this attempt; +1 accounts for the
                             // bump this drain represents. At the cap, dead-letter instead of
                             // bumping — and count it as leaving the live queue like PermanentFail.
                             if (liveRows[i].AttemptCount + 1 >= _maxAttempts)
                             {
-                                DeadLetterRow(conn, tx, rowId, $"max attempts ({_maxAttempts}) exceeded");
+                                await DeadLetterRowAsync(tx, id, $"max attempts ({_maxAttempts}) exceeded", ct).ConfigureAwait(false);
                                 rowsLeavingQueue++;
                             }
                             else
                             {
-                                BumpAttempt(conn, tx, rowId, "retry-please");
+                                await BumpAttemptAsync(tx, id, "retry-please", ct).ConfigureAwait(false);
                             }
                             break;
                     }
                 }
-                tx.Commit();
+                await tx.CommitAsync(ct).ConfigureAwait(false);
             }
             // Ack-deleted + PermanentFail-dead-lettered rows both leave the
             // non-dead-lettered queue, as do RetryPlease rows that hit the max-attempts
@@ -464,22 +468,66 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
         }
         finally
         {
-            _drainGate.Release();
+            _drainGateLock.Release();
         }
     }
 
+    /// <summary>
+    ///     Consults the injected gate, logging only when the decision changes.
+    /// </summary>
+    /// <remarks>
+    ///     Logged at all because the closed state is indistinguishable from a healthy idle queue
+    ///     from the outside, and one way to reach it is a redundancy snapshot that never names this
+    ///     node — the documented identity-mismatch shape. Without this line, that misconfiguration
+    ///     would present as alarm history quietly ceasing on both nodes of a pair. Logged on the
+    ///     transition rather than per tick because the drain runs every few seconds.
+    /// </remarks>
+    /// <returns><c>true</c> when this tick may proceed.</returns>
+    private bool EvaluateDrainGate()
+    {
+        bool open;
+        try
+        {
+            open = _drainGate();
+        }
+        catch (Exception ex)
+        {
+            // A throwing gate must not wedge the queue silently, and must not be treated as
+            // permission either: skip this tick, say why, and re-ask on the next one.
+            _logger.Error(ex, "Historian drain gate threw; skipping this tick");
+            return false;
+        }
+
+        bool? previous;
+        lock (_statusLock)
+        {
+            previous = _lastGateDecision;
+            _lastGateDecision = open;
+        }
+
+        if (previous == open) return open;
+
+        if (open)
+            _logger.Information("Historian drain resumed — this node now services the alarm queue");
+        else
+            _logger.Information(
+                "Historian drain suspended — this node is not the Primary. Queued alarm events are retained and will drain from whichever node holds the Primary role");
+
+        return open;
+    }
+
     /// <inheritdoc />
     public HistorianSinkStatus GetStatus()
     {
-        if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
+        if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
         var queued = Interlocked.Read(ref _queuedRowCount);
         if (queued < 0) queued = 0;
 
         long deadlettered;
-        using (var conn = OpenConnection())
+        using (var conn = _db.CreateConnection())
         using (var cmd = conn.CreateCommand())
         {
-            cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 1";
+            cmd.CommandText = "SELECT COUNT(*) FROM alarm_sf_events WHERE dead_lettered = 1";
             deadlettered = (long)(cmd.ExecuteScalar() ?? 0L);
         }
 
@@ -510,10 +558,13 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     /// <returns>The number of rows revived from the dead-letter state.</returns>
     public int RetryDeadLettered()
     {
-        if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
-        using var conn = OpenConnection();
+        if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
+        // Through a LocalDb connection, not a private one: the capture triggers are what make this
+        // revival replicate to the peer, and they only exist on the registered table.
+        using var conn = _db.CreateConnection();
         using var cmd = conn.CreateCommand();
-        cmd.CommandText = "UPDATE Queue SET DeadLettered = 0, AttemptCount = 0, LastError = NULL WHERE DeadLettered = 1";
+        cmd.CommandText =
+            "UPDATE alarm_sf_events SET dead_lettered = 0, attempt_count = 0, last_error = NULL WHERE dead_lettered = 1";
         var revived = cmd.ExecuteNonQuery();
         // Dead-lettered rows rejoin the non-dead-lettered queue — keep the in-memory
         // counter aligned.
@@ -523,29 +574,31 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
 
     /// <summary>
     ///     One queued row paired with its deserialized event. <see cref="Event"/> is
-    ///     <c>null</c> when the row's <c>PayloadJson</c> is corrupt or un-deserializable —
-    ///     the <see cref="RowId"/> always stays bound to its own row so outcomes can
+    ///     <c>null</c> when the row's <c>payload_json</c> is corrupt or un-deserializable —
+    ///     the <see cref="Id"/> always stays bound to its own row so outcomes can
     ///     never be mapped to the wrong row.
     /// </summary>
-    private readonly record struct QueueRow(long RowId, AlarmHistorianEvent? Event, long AttemptCount);
+    private readonly record struct QueueRow(string Id, AlarmHistorianEvent? Event, long AttemptCount);
 
-    private List<QueueRow> ReadBatch(SqliteConnection conn)
+    private async Task<List<QueueRow>> ReadBatchAsync(CancellationToken ct)
     {
-        var rows = new List<QueueRow>();
-        using var cmd = conn.CreateCommand();
-        cmd.CommandText = """
-            SELECT RowId, PayloadJson, AttemptCount FROM Queue
-            WHERE DeadLettered = 0
-            ORDER BY RowId ASC
-            LIMIT $limit
-            """;
-        cmd.Parameters.AddWithValue("$limit", _batchSize);
-        using var reader = cmd.ExecuteReader();
-        while (reader.Read())
+        // Ordered by enqueue time rather than insertion order: a hashed TEXT primary key carries
+        // no sequence the way the legacy AUTOINCREMENT rowid did. Round-trip ("O") timestamps sort
+        // lexicographically in chronological order; id makes the ordering total.
+        var raw = await _db.QueryAsync(
+            """
+            SELECT id, payload_json, attempt_count FROM alarm_sf_events
+            WHERE dead_lettered = 0
+            ORDER BY enqueued_at_utc ASC, id ASC
+            LIMIT @Limit
+            """,
+            r => (Id: r.GetString(0), Payload: r.GetString(1), AttemptCount: r.GetInt64(2)),
+            new { Limit = _batchSize },
+            ct).ConfigureAwait(false);
+
+        var rows = new List<QueueRow>(raw.Count);
+        foreach (var (id, payload, attemptCount) in raw)
         {
-            var rowId = reader.GetInt64(0);
-            var payload = reader.GetString(1);
-            var attemptCount = reader.GetInt64(2);
             AlarmHistorianEvent? evt;
             try
             {
@@ -556,77 +609,58 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
                 // Malformed JSON — carry a null event so the caller dead-letters this row.
                 evt = null;
             }
-            rows.Add(new QueueRow(rowId, evt, attemptCount));
+            rows.Add(new QueueRow(id, evt, attemptCount));
         }
         return rows;
     }
 
-    private static void DeleteRow(SqliteConnection conn, SqliteTransaction tx, long rowId)
-    {
-        using var cmd = conn.CreateCommand();
-        cmd.Transaction = tx;
-        cmd.CommandText = "DELETE FROM Queue WHERE RowId = $id";
-        cmd.Parameters.AddWithValue("$id", rowId);
-        cmd.ExecuteNonQuery();
-    }
+    private static Task DeleteRowAsync(ILocalDbTransaction tx, string id, CancellationToken ct) =>
+        tx.ExecuteAsync("DELETE FROM alarm_sf_events WHERE id = @Id", new { Id = id }, ct);
 
-    private void DeadLetterRow(SqliteConnection conn, SqliteTransaction tx, long rowId, string reason)
-    {
-        using var cmd = conn.CreateCommand();
-        cmd.Transaction = tx;
-        cmd.CommandText = """
-            UPDATE Queue SET DeadLettered = 1, LastAttemptUtc = $now, LastError = $err, AttemptCount = AttemptCount + 1
-            WHERE RowId = $id
-            """;
-        cmd.Parameters.AddWithValue("$now", _clock().ToString("O"));
-        cmd.Parameters.AddWithValue("$err", reason);
-        cmd.Parameters.AddWithValue("$id", rowId);
-        cmd.ExecuteNonQuery();
-    }
+    private Task DeadLetterRowAsync(ILocalDbTransaction tx, string id, string reason, CancellationToken ct) =>
+        tx.ExecuteAsync(
+            """
+            UPDATE alarm_sf_events
+            SET dead_lettered = 1, last_attempt_utc = @Now, last_error = @Err,
+                attempt_count = attempt_count + 1
+            WHERE id = @Id
+            """,
+            new { Now = _clock().ToString("O"), Err = reason, Id = id },
+            ct);
 
-    private void BumpAttempt(SqliteConnection conn, SqliteTransaction tx, long rowId, string reason)
-    {
-        using var cmd = conn.CreateCommand();
-        cmd.Transaction = tx;
-        cmd.CommandText = """
-            UPDATE Queue SET LastAttemptUtc = $now, LastError = $err, AttemptCount = AttemptCount + 1
-            WHERE RowId = $id
-            """;
-        cmd.Parameters.AddWithValue("$now", _clock().ToString("O"));
-        cmd.Parameters.AddWithValue("$err", reason);
-        cmd.Parameters.AddWithValue("$id", rowId);
-        cmd.ExecuteNonQuery();
-    }
+    private Task BumpAttemptAsync(ILocalDbTransaction tx, string id, string reason, CancellationToken ct) =>
+        tx.ExecuteAsync(
+            """
+            UPDATE alarm_sf_events
+            SET last_attempt_utc = @Now, last_error = @Err, attempt_count = attempt_count + 1
+            WHERE id = @Id
+            """,
+            new { Now = _clock().ToString("O"), Err = reason, Id = id },
+            ct);
 
-    // Async variant used by EnqueueAsync.
-    private async Task EnforceCapacityAsync(SqliteConnection conn, CancellationToken ct)
+    private async Task EnforceCapacityAsync(CancellationToken ct)
     {
         Interlocked.Increment(ref _capacityProbeCount);
-        long count;
-        using (var cmd = conn.CreateCommand())
-        {
-            cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
-            count = (long)(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false) ?? 0L);
-        }
+        var count = await CountAsync("dead_lettered = 0", ct).ConfigureAwait(false);
+
         // Resync the in-memory counter while we have a fresh number.
         Interlocked.Exchange(ref _queuedRowCount, count);
         if (count < _capacity) return;
 
         var toEvict = count - _capacity + 1;
-        using (var cmd = conn.CreateCommand())
-        {
-            cmd.CommandText = """
-                DELETE FROM Queue
-                WHERE RowId IN (
-                    SELECT RowId FROM Queue
-                    WHERE DeadLettered = 0
-                    ORDER BY RowId ASC
-                    LIMIT $n
-                )
-                """;
-            cmd.Parameters.AddWithValue("$n", toEvict);
-            await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
-        }
+        await _db.ExecuteAsync(
+            """
+            DELETE FROM alarm_sf_events
+            WHERE id IN (
+                SELECT id FROM alarm_sf_events
+                WHERE dead_lettered = 0
+                ORDER BY enqueued_at_utc ASC, id ASC
+                LIMIT @N
+            )
+            """,
+            new { N = toEvict },
+            ct).ConfigureAwait(false);
+
         Interlocked.Add(ref _queuedRowCount, -toEvict);
         long lifetimeEvicted;
         lock (_statusLock) { _evictedCount += toEvict; lifetimeEvicted = _evictedCount; }
@@ -635,40 +669,21 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
             _capacity, toEvict, lifetimeEvicted);
     }
 
-    private void PurgeAgedDeadLetters(SqliteConnection conn)
+    private async Task PurgeAgedDeadLettersAsync(CancellationToken ct)
     {
         var cutoff = (_clock() - _deadLetterRetention).ToString("O");
-        using var cmd = conn.CreateCommand();
-        cmd.CommandText = """
-            DELETE FROM Queue
-            WHERE DeadLettered = 1 AND LastAttemptUtc IS NOT NULL AND LastAttemptUtc < $cutoff
-            """;
-        cmd.Parameters.AddWithValue("$cutoff", cutoff);
-        var purged = cmd.ExecuteNonQuery();
+        var purged = await _db.ExecuteAsync(
+            """
+            DELETE FROM alarm_sf_events
+            WHERE dead_lettered = 1 AND last_attempt_utc IS NOT NULL AND last_attempt_utc < @Cutoff
+            """,
+            new { Cutoff = cutoff },
+            ct).ConfigureAwait(false);
+
         if (purged > 0)
             _logger.Information("Purged {Count} dead-lettered row(s) past retention window", purged);
     }
 
-    private void InitializeSchema()
-    {
-        using var conn = OpenConnection();
-        using var cmd = conn.CreateCommand();
-        cmd.CommandText = """
-            CREATE TABLE IF NOT EXISTS Queue (
-                RowId INTEGER PRIMARY KEY AUTOINCREMENT,
-                AlarmId TEXT NOT NULL,
-                EnqueuedUtc TEXT NOT NULL,
-                PayloadJson TEXT NOT NULL,
-                AttemptCount INTEGER NOT NULL DEFAULT 0,
-                LastAttemptUtc TEXT NULL,
-                LastError TEXT NULL,
-                DeadLettered INTEGER NOT NULL DEFAULT 0
-            );
-            CREATE INDEX IF NOT EXISTS IX_Queue_Drain ON Queue (DeadLettered, RowId);
-            """;
-        cmd.ExecuteNonQuery();
-    }
-
     private void BumpBackoff() => _backoffIndex = Math.Min(_backoffIndex + 1, BackoffLadder.Length - 1);
     private void ResetBackoff() => _backoffIndex = 0;
 
@@ -676,12 +691,16 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     public TimeSpan CurrentBackoff => BackoffLadder[_backoffIndex];
 
     /// <summary>Disposes the sink and releases all held resources including the drain timer and the writer.</summary>
+    /// <remarks>
+    ///     The <see cref="ILocalDb"/> is NOT disposed here — it is a shared singleton owned by the
+    ///     host and used by the deployment cache too.
+    /// </remarks>
     public void Dispose()
     {
         if (_disposed) return;
         _disposed = true;
         _drainTimer?.Dispose();
-        _drainGate.Dispose();
+        _drainGateLock.Dispose();
         if (_writer is IDisposable writerDisposable) writerDisposable.Dispose();
     }
 }

src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.csproj

@@ -20,6 +20,12 @@
         -->
         <PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3"/>
         <PackageReference Include="Serilog"/>
+        <!--
+          The store-and-forward buffer lives in the node's consolidated LocalDb file so it
+          replicates to the redundant pair peer. Only the ILocalDb SQL surface is used here; the
+          Host owns the DI registration and the replication package.
+        -->
+        <PackageReference Include="ZB.MOM.WW.LocalDb"/>
     </ItemGroup>
 
     <ItemGroup>

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayAlarmHistorianWriter.cs

@@ -9,7 +9,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
 
 /// <summary>
 ///     <see cref="IAlarmHistorianWriter"/> backed by the HistorianGateway <c>SendEvent</c> path. The
-///     drain worker behind <c>SqliteStoreAndForwardSink</c> calls
+///     drain worker behind <c>LocalDbStoreAndForwardSink</c> calls
 ///     <see cref="WriteBatchAsync"/> and uses the returned per-event
 ///     <see cref="HistorianWriteOutcome"/> to decide retry vs. dead-letter, so this writer maps every
 ///     gateway result — success ack, the published client's typed exception hierarchy, raw

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianServiceCollectionExtensions.cs

@@ -47,7 +47,7 @@ public static class GatewayHistorian
     ///     <see cref="ServerHistorianOptions"/> — the <b>same single gateway</b> the read path
     ///     (<see cref="CreateDataSource"/>) targets. The Host's <c>AddAlarmHistorian</c> wiring supplies
     ///     this as the concrete <see cref="IAlarmHistorianWriter"/> the durable
-    ///     <c>SqliteStoreAndForwardSink</c> drain worker delegates to, sourcing the connection from the
+    ///     <c>LocalDbStoreAndForwardSink</c> drain worker delegates to, sourcing the connection from the
     ///     <c>ServerHistorian</c> section (endpoint/key/TLS) rather than the legacy Wonderware-shaped
     ///     <c>AlarmHistorian</c> host/port. Resolves an <see cref="ILoggerFactory"/> and the writer's
     ///     <see cref="ILogger{TCategoryName}"/> from <paramref name="services"/>, falling back to the null

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/HistorianGatewayClientAdapter.cs

@@ -57,16 +57,23 @@ public sealed class HistorianGatewayClientAdapter : IHistorianGatewayClient, IDi
                 $"ServerHistorian:Endpoint must be an absolute http(s) URI (e.g. https://host:5222); got '{options.Endpoint}'.");
         }
 
+        // TLS-only options must stay at their defaults on a plaintext h2c connection: the client rejects
+        // each of them outright when UseTls=false ("<name> is a TLS-only option and requires
+        // UseTls=true"). Forwarding them unconditionally made h2c UNREACHABLE — AllowUntrustedServer-
+        // Certificate defaults to false, so RequireCertificateValidation was always sent as true and
+        // every http:// deployment crashed at startup, even though the scheme is documented as the
+        // supported way to select h2c. There is no certificate to have a posture about here.
         var clientOptions = new HistorianGatewayClientOptions
         {
             Endpoint = endpointUri,
             ApiKey = options.ApiKey,
             UseTls = options.UseTls,
-            CaCertificatePath = options.CaCertificatePath,
-            // INVERTED mapping: ServerHistorianOptions.AllowUntrustedServerCertificate (opt-in to accept
-            // a self-signed cert) is the negation of the client's RequireCertificateValidation. Allowing
-            // an untrusted cert == not requiring validation; a pinned CaCertificatePath always verifies.
-            RequireCertificateValidation = !options.AllowUntrustedServerCertificate,
+            CaCertificatePath = options.UseTls ? options.CaCertificatePath : null,
+            // INVERTED mapping (TLS only): ServerHistorianOptions.AllowUntrustedServerCertificate
+            // (opt-in to accept a self-signed cert) is the negation of the client's
+            // RequireCertificateValidation. Allowing an untrusted cert == not requiring validation; a
+            // pinned CaCertificatePath always verifies.
+            RequireCertificateValidation = options.UseTls && !options.AllowUntrustedServerCertificate,
             DefaultCallTimeout = options.CallTimeout,
             LoggerFactory = loggerFactory,
         };

src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/AlarmsHistorian.razor

@@ -18,8 +18,9 @@
 <section class="panel notice rise" style="animation-delay:.02s">
     Snapshot from the local node's <span class="mono">HistorianAdapterActor</span>. Default sink
     is a no-op (<span class="mono">NullAlarmHistorianSink</span>); production wires
-    <span class="mono">SqliteStoreAndForwardSink</span> draining to the HistorianGateway
-    (<span class="mono">SendEvent</span>) behind it. Polling every @PollSeconds s.
+    <span class="mono">LocalDbStoreAndForwardSink</span> — buffering into this node's LocalDb, and
+    draining to the HistorianGateway (<span class="mono">SendEvent</span>) only while this node holds
+    the Primary role. Polling every @PollSeconds s.
 </section>
 
 @if (_status is null)

src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Clusters/ClusterRedundancy.razor

@@ -1,10 +1,16 @@
 @page "/clusters/{ClusterId}/redundancy"
 @attribute [Authorize(Policy = AdminUiPolicies.AuthenticatedRead)]
 @rendermode RenderMode.InteractiveServer
+@using Microsoft.AspNetCore.Authorization
 @using Microsoft.EntityFrameworkCore
+@using ZB.MOM.WW.OtOpcUa.AdminUI.Redundancy
 @using ZB.MOM.WW.OtOpcUa.Configuration
 @using ZB.MOM.WW.OtOpcUa.Configuration.Entities
+@using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy
 @inject IDbContextFactory<OtOpcUaConfigDbContext> DbFactory
+@inject IManualFailoverService FailoverService
+@inject AuthenticationStateProvider AuthState
+@inject IAuthorizationService AuthorizationService
 
 @if (!_loaded)
 {
@@ -45,6 +51,67 @@ else
         </div>
     </section>
 
+    <section class="panel rise mt-3" style="animation-delay:.11s">
+        <div class="panel-head">Live redundancy</div>
+        <div style="padding:1rem">
+            <div class="kv">
+                <span class="k">Driver Primary</span>
+                <span class="v mono">@(_failover.Snapshot?.PrimaryAddress ?? "—")</span>
+            </div>
+            <div class="kv">
+                <span class="k">Up driver members</span>
+                <span class="v mono">
+                    @(_failover.Snapshot is { DriverAddresses.Count: > 0 } s
+                        ? string.Join(", ", s.DriverAddresses)
+                        : "—")
+                </span>
+            </div>
+
+            <p class="text-muted small mt-2 mb-0">
+                Read live from cluster state on this node. <strong>Mesh-wide scope:</strong> the Primary is
+                elected once per Akka mesh, not per cluster row — in the current single-mesh topology this
+                acts on the whole mesh's Primary, which may be a node of another
+                <span class="mono">Cluster</span>. See <span class="mono">docs/Redundancy.md</span>.
+            </p>
+
+            <AuthorizeView Policy="@ManualFailoverPageModel.RequiredPolicy">
+                <Authorized>
+                    <div class="mt-3">
+                        <button class="btn btn-sm btn-outline-danger"
+                                disabled="@(!_failover.CanFailOver)"
+                                title="@(_failover.DisabledReason ?? "Gracefully move the driver Primary to its peer")"
+                                @onclick="() => _failover.RequestFailover()">
+                            Trigger failover
+                        </button>
+                        @if (_failover.DisabledReason is { } reason)
+                        {
+                            <span class="text-muted small ms-2">@reason</span>
+                        }
+                    </div>
+
+                    @if (_failover.ConfirmOpen)
+                    {
+                        <div class="panel notice mt-3">
+                            <strong>Fail over the driver Primary?</strong>
+                            <ul class="mb-2 mt-2">
+                                <li><span class="mono">@(_failover.Snapshot?.PrimaryAddress)</span> leaves the cluster and its process restarts.</li>
+                                <li>Its peer becomes Primary and advertises <span class="mono">ServiceLevel</span> 250.</li>
+                                <li>Connected OPC UA clients re-select the new Primary.</li>
+                            </ul>
+                            <button class="btn btn-sm btn-danger" @onclick="ConfirmFailoverAsync">Confirm failover</button>
+                            <button class="btn btn-sm btn-outline-secondary ms-2" @onclick="() => _failover.CancelFailover()">Cancel</button>
+                        </div>
+                    }
+                </Authorized>
+            </AuthorizeView>
+
+            @if (_failover.StatusMessage is { } msg)
+            {
+                <div class="mt-3 @(_failover.StatusIsError ? "text-danger" : "text-success")">@msg</div>
+            }
+        </div>
+    </section>
+
     <section class="panel rise mt-3" style="animation-delay:.14s">
         <div class="panel-head">Node service-level configuration</div>
         @if (_nodes is null || _nodes.Count == 0)
@@ -91,8 +158,16 @@ else
     private ServerCluster? _cluster;
     private List<ClusterNode>? _nodes;
 
+    // Everything consequential about the failover control (peer guard, confirm flow, outcome text)
+    // lives in this pure model rather than in the markup — the repo has no bUnit, so logic left in a
+    // .razor is verified only by driving the page. Covered by ManualFailoverPageModelTests.
+    private ManualFailoverPageModel _failover = default!;
+
     protected override async Task OnInitializedAsync()
     {
+        _failover = new ManualFailoverPageModel(FailoverService);
+        _failover.Refresh();
+
         await using var db = await DbFactory.CreateDbContextAsync();
         _cluster = await db.ServerClusters.AsNoTracking()
             .FirstOrDefaultAsync(c => c.ClusterId == ClusterId);
@@ -105,4 +180,22 @@ else
         }
         _loaded = true;
     }
+
+    /// <summary>
+    /// Defense-in-depth: the button is FleetAdmin-gated in markup, but this handler runs on the
+    /// server circuit — re-check the policy before bouncing a production node (the same pattern the
+    /// certificate-store actions use).
+    /// </summary>
+    private async Task ConfirmFailoverAsync()
+    {
+        var authState = await AuthState.GetAuthenticationStateAsync();
+        if (!(await AuthorizationService.AuthorizeAsync(
+                authState.User, null, ManualFailoverPageModel.RequiredPolicy)).Succeeded)
+        {
+            _failover.CancelFailover();
+            return;
+        }
+
+        await _failover.ConfirmFailoverAsync(authState.User.Identity?.Name ?? "system");
+    }
 }

src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Redundancy/ManualFailoverPageModel.cs

@@ -0,0 +1,134 @@
+using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
+using ZB.MOM.WW.OtOpcUa.Security.Auth;
+
+namespace ZB.MOM.WW.OtOpcUa.AdminUI.Redundancy;
+
+/// <summary>
+/// Presentation state for the Trigger-failover control on the cluster redundancy page.
+/// </summary>
+/// <remarks>
+///     <para>
+///         A pure model behind a thin razor shell — the same split the driver-typed tag editors use,
+///         and for the same reason: this repo has no bUnit (see
+///         <c>PageAuthorizationGuardTests</c>), so anything left inside the <c>.razor</c> is verified
+///         only by driving the page live. The peer guard, the confirm flow, and the
+///         refused-vs-succeeded outcome are consequential enough to be unit-testable, so they live
+///         here.
+///     </para>
+///     <para>
+///         What is deliberately NOT here: the markup authorization gate. It is expressed in the razor
+///         as <c>&lt;AuthorizeView Policy="@ManualFailoverPageModel.RequiredPolicy"&gt;</c> and
+///         re-checked server-side before the call, so <see cref="RequiredPolicy"/> is the single
+///         source of truth for both.
+///     </para>
+/// </remarks>
+public sealed class ManualFailoverPageModel
+{
+    /// <summary>
+    /// The policy gating the control, in markup and in the server-side re-check.
+    /// </summary>
+    /// <remarks>
+    /// <see cref="AdminUiPolicies.FleetAdmin"/> (Administrator only) rather than the
+    /// <see cref="AdminUiPolicies.ConfigEditor"/> the neighbouring cluster-authoring pages use:
+    /// this does not edit configuration, it restarts a production node. ConfigEditor also admits the
+    /// Designer role, which must not be able to bounce the Primary.
+    /// </remarks>
+    public const string RequiredPolicy = AdminUiPolicies.FleetAdmin;
+
+    private readonly IManualFailoverService _service;
+
+    /// <summary>Creates the model over the failover service.</summary>
+    /// <param name="service">The manual-failover seam; faked in tests.</param>
+    public ManualFailoverPageModel(IManualFailoverService service)
+        => _service = service ?? throw new ArgumentNullException(nameof(service));
+
+    /// <summary>The last read cluster snapshot, or <see langword="null"/> if it could not be read.</summary>
+    public ManualFailoverSnapshot? Snapshot { get; private set; }
+
+    /// <summary>Whether the confirmation dialog is open.</summary>
+    public bool ConfirmOpen { get; private set; }
+
+    /// <summary>Status text from the last completed action, if any.</summary>
+    public string? StatusMessage { get; private set; }
+
+    /// <summary>Whether <see cref="StatusMessage"/> reports a failure.</summary>
+    public bool StatusIsError { get; private set; }
+
+    /// <summary>Whether the button is enabled — a peer must exist to take over.</summary>
+    public bool CanFailOver => Snapshot?.CanFailOver == true;
+
+    /// <summary>
+    /// Why the button is disabled, or <see langword="null"/> when it is enabled. Rendered as the
+    /// button's tooltip so a disabled control explains itself.
+    /// </summary>
+    public string? DisabledReason => Snapshot is null
+        ? "Cluster state is unavailable on this node."
+        : Snapshot.CanFailOver
+            ? null
+            : $"Failover needs a driver peer to take over; this mesh has {Snapshot.DriverAddresses.Count} Up driver member(s).";
+
+    /// <summary>Re-reads live cluster state. Never throws — an unreadable cluster disables the button.</summary>
+    public void Refresh()
+    {
+        try
+        {
+            Snapshot = _service.GetSnapshot();
+        }
+        catch (Exception ex)
+        {
+            // A node whose ActorSystem is not yet up (or not a cluster provider) must render the page,
+            // not 500. The button stays disabled with DisabledReason explaining why.
+            Snapshot = null;
+            StatusIsError = true;
+            StatusMessage = $"Could not read cluster state: {ex.Message}";
+        }
+    }
+
+    /// <summary>Opens the confirmation dialog. No-op when the peer guard is closed.</summary>
+    public void RequestFailover()
+    {
+        if (!CanFailOver) return;
+        StatusMessage = null;
+        StatusIsError = false;
+        ConfirmOpen = true;
+    }
+
+    /// <summary>Closes the confirmation dialog without acting.</summary>
+    public void CancelFailover() => ConfirmOpen = false;
+
+    /// <summary>
+    /// Performs the failover the dialog is confirming.
+    /// </summary>
+    /// <param name="actor">The authenticated user name, recorded in the audit event.</param>
+    public async Task ConfirmFailoverAsync(string actor)
+    {
+        if (!ConfirmOpen) return;
+        ConfirmOpen = false;
+
+        try
+        {
+            var target = await _service.FailOverDriverPrimaryAsync(actor).ConfigureAwait(false);
+            if (target is null)
+            {
+                // The service re-evaluates the peer guard against live state, which may have changed
+                // since Refresh(). A refusal is not an error — but it must not read as a success.
+                StatusIsError = true;
+                StatusMessage = "Failover refused: no driver peer is available to take over.";
+            }
+            else
+            {
+                StatusIsError = false;
+                StatusMessage =
+                    $"Failover requested. {target} is leaving the cluster; its peer takes over as Primary "
+                    + "and the node restarts as the youngest member.";
+            }
+        }
+        catch (Exception ex)
+        {
+            StatusIsError = true;
+            StatusMessage = $"Failover failed: {ex.Message}";
+        }
+
+        Refresh();
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs

@@ -1,5 +1,4 @@
 using Akka.Actor;
-using Akka.Cluster;
 using Akka.Cluster.Tools.PublishSubscribe;
 using Akka.Event;
 using Microsoft.EntityFrameworkCore;
@@ -18,8 +17,40 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
 /// has acked Applied. Per-node ACKs are persisted in <c>NodeDeploymentState</c> so a failover of
 /// this singleton can recover in-flight state from the DB.
 ///
-/// Discovery of the "expected ACK set" comes from <c>Akka.Cluster.State.Members</c> filtered by
-/// the <c>driver</c> role — the DB does not own per-node role assignment.
+/// Discovery of the "expected ACK set" comes from the <b>enabled <c>ClusterNode</c> rows</b>, not
+/// from <c>Akka.Cluster.State.Members</c> (per-cluster mesh Phase 1). Central must be able to name
+/// the nodes a deployment is for without sharing a gossip ring with them — Phase 2 splits the fleet
+/// into one mesh per <c>Cluster</c>, after which central can no longer see a site node's membership
+/// at all. This was the coordinator's last genuinely mesh-bound dependency.
+///
+/// <para>
+/// <b>Behaviour change:</b> a configured node that is switched off is now <i>expected</i>, so a
+/// deployment dispatched while it is down fails at the apply deadline instead of sealing green
+/// without it. That is deliberate — under the membership rule the operator was told the fleet was
+/// deployed when it was not. <c>ClusterNode.MaintenanceMode = 1</c> is the escape hatch for a node
+/// taken down for maintenance.
+/// </para>
+/// <para>
+/// <b>Not <c>Enabled = 0</c>.</b> That was the intended hatch until the Phase 1 live gate found it
+/// unusable: <c>DraftValidator.ValidateClusterTopology</c> requires the enabled-node count to equal
+/// <c>ServerCluster.NodeCount</c>, so clearing <c>Enabled</c> on either node of a Warm/Hot pair
+/// rejects the deployment outright — and every cluster in the target topology is a pair. The two
+/// rules were independently reasonable and contradictory together, so the meanings were split:
+/// <c>Enabled</c> = part of the declared topology, <c>MaintenanceMode</c> = do not expect it now.
+/// </para>
+/// <para>
+/// <b>Every <c>ClusterNode</c> row is assumed to be a driver node.</b> The membership rule filtered
+/// on the <c>driver</c> role; the DB has no per-node role column, and deliberately does not gain one
+/// here (a second declaration of node roles would drift from <c>Cluster:Roles</c> in the node's own
+/// appsettings). The assumption holds because the entity's whole shape — <c>Host</c>,
+/// <c>OpcUaPort</c>, <c>ApplicationUri</c>, <c>ServiceLevelBase</c> — describes an OPC UA server
+/// node. An admin-only node must not be given a <c>ClusterNode</c> row: it would be expected to ack
+/// a deployment it has no <c>DriverHostActor</c> to apply, and every deploy would time out.
+/// </para>
+/// <para>
+/// <c>ServerCluster.Enabled</c> is <b>not</b> consulted — nothing else in the codebase consults it,
+/// so honouring it here would invent semantics. Disable the nodes, not the cluster row.
+/// </para>
 /// </summary>
 public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
 {
@@ -119,11 +150,15 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
     {
         _current = msg.DeploymentId;
         _acks.Clear();
-        _expectedAcks = DiscoverDriverNodes();
 
         // Seed NodeDeploymentState rows so a failover knows which nodes were expected to ack.
         using (var db = _dbFactory.CreateDbContext())
         {
+            // Discovery reuses this context rather than opening a second one — it reads the same
+            // ClusterNode table the NodeDeploymentState rows below are FK-bound to, so a single
+            // context keeps the expected set and the seeded rows consistent by construction.
+            _expectedAcks = DiscoverDriverNodes(db);
+
             foreach (var node in _expectedAcks)
             {
                 db.NodeDeploymentStates.Add(new NodeDeploymentState
@@ -142,9 +177,14 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
 
         if (_expectedAcks.Count == 0)
         {
-            // No driver-role members. Seal immediately — the alternative is hanging forever
-            // waiting for ACKs that will never come.
-            _log.Warning("DispatchDeployment {Id}: no driver-role members in cluster; sealing empty",
+            // No enabled ClusterNode rows. Seal immediately — the alternative is hanging forever
+            // waiting for ACKs that will never come. Note this is now a *configuration* error
+            // rather than a topology observation: under the old membership rule an empty set meant
+            // "no driver nodes have joined yet", which could resolve itself; an empty set here means
+            // the fleet has no enabled nodes registered, which cannot.
+            _log.Warning(
+                "DispatchDeployment {Id}: no enabled ClusterNode rows registered; sealing empty. " +
+                "Nothing will receive this deployment — check the fleet's node configuration",
                 msg.DeploymentId);
             SealDeployment();
         }
@@ -233,8 +273,15 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
         using var db = _dbFactory.CreateDbContext();
         UpdateDeploymentStatus(db, _current.Value, DeploymentStatus.TimedOut);
         db.SaveChanges();
-        _log.Warning("Deployment {Id} timed out after {Deadline} ({Acked}/{Total} acks landed)",
-            _current.Value, _applyDeadline, _acks.Count, _expectedAcks.Count);
+        // Name the silent nodes, not just the counts. Since the expected set became DB-derived this
+        // is the failure an operator meets after deploying while a configured node is switched off,
+        // so it has to say which node — "4/5 acks landed" leaves them reading logs on five machines.
+        var missing = _expectedAcks.Where(n => !_acks.ContainsKey(n)).Select(n => n.Value).Order().ToList();
+        _log.Warning(
+            "Deployment {Id} timed out after {Deadline} ({Acked}/{Total} acks landed). " +
+            "No ack from: {MissingNodes}. Each is an enabled, non-maintenance ClusterNode row — start " +
+            "the node, or set ClusterNode.MaintenanceMode = 1 if it is down for maintenance, then redeploy",
+            _current.Value, _applyDeadline, _acks.Count, _expectedAcks.Count, string.Join(", ", missing));
         ResetForNext();
     }
 
@@ -259,22 +306,29 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
         if (sealNow) d.SealedAtUtc = DateTime.UtcNow;
     }
 
-    private HashSet<NodeId> DiscoverDriverNodes()
-    {
-        var cluster = Akka.Cluster.Cluster.Get(Context.System);
-        var nodes = new HashSet<NodeId>(NodeIdComparer);
-        foreach (var member in cluster.State.Members)
-        {
-            if (member.Status is not (MemberStatus.Up or MemberStatus.Joining)) continue;
-            if (!member.Roles.Contains("driver")) continue;
-            var host = member.Address.Host;
-            if (string.IsNullOrWhiteSpace(host)) continue;
-            // Match ClusterRoleInfo's NodeId derivation (host:port) so DriverHostActor's
-            // self-identification and the coordinator's expected-ack set agree.
-            nodes.Add(NodeId.Parse($"{host}:{member.Address.Port ?? 0}"));
-        }
-        return nodes;
-    }
+    /// <summary>
+    ///     The set of nodes expected to ack this deployment: every <b>enabled</b> <c>ClusterNode</c>
+    ///     row. See the class remarks for why this is DB-derived rather than membership-derived, and
+    ///     for the two assumptions it rests on (every row is a driver node; <c>Enabled</c> is the
+    ///     maintenance hatch).
+    /// </summary>
+    /// <remarks>
+    ///     <c>ClusterNode.NodeId</c> is already in the <c>host:port</c> identifier space the
+    ///     membership rule derived — the rig seeds <c>central-1:4053</c> and so on, and
+    ///     <c>NodeDeploymentState.NodeId</c> is FK-bound to this column, so the fact that the
+    ///     membership-derived rows have always satisfied that FK is standing proof the two sets
+    ///     agree. It also has to stay that way: <c>DriverHostActor</c> self-identifies from
+    ///     <c>ClusterRoleInfo</c>'s <c>host:port</c>, so a row whose NodeId is anything else is a
+    ///     node whose acks will never match its expected-ack entry.
+    /// </remarks>
+    private static HashSet<NodeId> DiscoverDriverNodes(OtOpcUaConfigDbContext db) =>
+        db.ClusterNodes
+            .AsNoTracking()
+            .Where(n => n.Enabled && !n.MaintenanceMode)
+            .Select(n => n.NodeId)
+            .ToList()
+            .Select(NodeId.Parse)
+            .ToHashSet(NodeIdComparer);
 
     /// <summary>Case-insensitive <see cref="NodeId"/> equality (by <see cref="NodeId.Value"/>),
     /// matching the case-insensitive scoping in <c>DeploymentArtifact.ResolveClusterScope</c> so the

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconciler.cs

@@ -0,0 +1,138 @@
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
+
+/// <summary>What kind of disagreement was found between a node's real address and its row.</summary>
+public enum AddressMismatchKind
+{
+    /// <summary>
+    ///     The row was matched to a live member by <c>NodeId</c>, but its <c>Host</c> / <c>AkkaPort</c>
+    ///     columns name a different address. Phase 2 dials those columns, so central would dial
+    ///     somewhere the node is not — and the symptom there is a silent absence of acks.
+    /// </summary>
+    RowDialTargetDisagrees,
+
+    /// <summary>
+    ///     A driver-role member is in the cluster with no enabled <c>ClusterNode</c> row at its
+    ///     address. Its deployment acks are discarded (see <c>ConfigPublishCoordinator</c>), so it
+    ///     silently receives deployments it is never credited for.
+    /// </summary>
+    RunningNodeHasNoRow,
+
+    /// <summary>
+    ///     An enabled row names a node that is not currently a driver-role member. Either it is down
+    ///     — in which case the next deployment fails at the apply deadline — or it binds an address
+    ///     other than the one its row claims.
+    /// </summary>
+    EnabledRowNotInCluster,
+}
+
+/// <summary>One disagreement, with enough detail to act on without opening the DB.</summary>
+/// <param name="Kind">Which disagreement this is.</param>
+/// <param name="NodeId">The <c>host:port</c> identity the finding is about.</param>
+/// <param name="Detail">Human-readable specifics, including what to do about it.</param>
+public sealed record ClusterNodeAddressMismatch(AddressMismatchKind Kind, string NodeId, string Detail);
+
+/// <summary>A <c>ClusterNode</c> row reduced to the address facts this check cares about.</summary>
+/// <param name="NodeId">The row's primary key, in the <c>host:port</c> space.</param>
+/// <param name="Host">The host central will dial in Phase 2.</param>
+/// <param name="AkkaPort">The remoting port central will dial in Phase 2.</param>
+public sealed record ClusterNodeAddress(string NodeId, string Host, int AkkaPort);
+
+/// <summary>
+///     Compares the addresses driver nodes actually occupy against the <c>ClusterNode</c> rows that
+///     claim to describe them.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>Why this exists.</b> <c>ClusterNode.AkkaPort</c> and the node's own <c>Cluster:Port</c>
+///         are the same fact stored in two places, and nothing makes them agree. A node that binds
+///         4054 while its row says 4053 is unreachable from central once Phase 2 dials the row
+///         instead of gossiping — and an unenforced duplicated address is the exact shape of the
+///         <c>Modbus</c>/<c>ModbusTcp</c> and <c>TwinCat</c>/<c>Focas</c> drifts already recorded in
+///         this repo. Since Phase 1 also made the rows the deploy path's expected-ack set, a drifted
+///         row already costs a failed deployment today.
+///     </para>
+///     <para>
+///         <b>It runs on an admin node</b>, comparing rows against the membership an admin node can
+///         already see, rather than having each driver node assert its own row — Phase 4 removes the
+///         driver nodes' ConfigDb connection entirely, so a self-assertion written there would have
+///         to be deleted again.
+///     </para>
+///     <para>
+///         <b>Phase 2 must revisit this.</b> Once the fleet splits into one mesh per cluster, an
+///         admin node no longer sees site members at all, and every site row would report
+///         <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> forever. The check then has to
+///         move to whatever transport replaces gossip — or be scoped to the central mesh.
+///     </para>
+/// </remarks>
+public static class ClusterNodeAddressReconciler
+{
+    /// <summary>
+    ///     Reconciles observed driver-role member addresses against the configured rows.
+    /// </summary>
+    /// <param name="observedDriverMembers">
+    ///     <c>host</c>/<c>port</c> of every Up member carrying the <c>driver</c> role. Admin-only
+    ///     members must be excluded — they legitimately have no <c>ClusterNode</c> row, and including
+    ///     them would report a mismatch for correct configuration.
+    /// </param>
+    /// <param name="rows">Every <c>ClusterNode</c> row, enabled or not.</param>
+    /// <param name="expectedPresentNodeIds">
+    ///     The subset of <paramref name="rows"/> that should currently be in the cluster: enabled and
+    ///     not in maintenance. Excluded rows are deliberately still matched against membership (a
+    ///     disabled row for a running node is worth knowing) but never reported as
+    ///     <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> — being absent is the whole point
+    ///     of disabling one or putting it in maintenance.
+    /// </param>
+    /// <returns>Every disagreement found, ordered by node id for stable logging.</returns>
+    public static IReadOnlyList<ClusterNodeAddressMismatch> Reconcile(
+        IReadOnlyCollection<(string Host, int Port)> observedDriverMembers,
+        IReadOnlyCollection<ClusterNodeAddress> rows,
+        IReadOnlyCollection<string> expectedPresentNodeIds)
+    {
+        var byNodeId = new Dictionary<string, ClusterNodeAddress>(StringComparer.OrdinalIgnoreCase);
+        foreach (var row in rows) byNodeId[row.NodeId] = row;
+        var expectedPresent = new HashSet<string>(expectedPresentNodeIds, StringComparer.OrdinalIgnoreCase);
+
+        var findings = new List<ClusterNodeAddressMismatch>();
+        var seen = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
+
+        foreach (var (host, port) in observedDriverMembers)
+        {
+            var nodeId = $"{host}:{port}";
+            seen.Add(nodeId);
+
+            if (!byNodeId.TryGetValue(nodeId, out var row))
+            {
+                findings.Add(new ClusterNodeAddressMismatch(
+                    AddressMismatchKind.RunningNodeHasNoRow, nodeId,
+                    $"driver node is in the cluster at {nodeId} but no ClusterNode row has that NodeId; " +
+                    "its deployment acks are discarded. Either register the row or correct the node's " +
+                    "Cluster:PublicHostname / Cluster:Port to match an existing one"));
+                continue;
+            }
+
+            if (!string.Equals(row.Host, host, StringComparison.OrdinalIgnoreCase) || row.AkkaPort != port)
+            {
+                findings.Add(new ClusterNodeAddressMismatch(
+                    AddressMismatchKind.RowDialTargetDisagrees, nodeId,
+                    $"node is at {nodeId} but its row's dial target is {row.Host}:{row.AkkaPort}; " +
+                    "central will dial the wrong address once Phase 2 stops using gossip. " +
+                    "Correct ClusterNode.Host / ClusterNode.AkkaPort"));
+            }
+        }
+
+        foreach (var row in rows)
+        {
+            if (seen.Contains(row.NodeId) || !expectedPresent.Contains(row.NodeId)) continue;
+            findings.Add(new ClusterNodeAddressMismatch(
+                AddressMismatchKind.EnabledRowNotInCluster, row.NodeId,
+                $"enabled ClusterNode row {row.NodeId} has no matching driver member; the next " +
+                "deployment will fail at the apply deadline waiting for it. Start the node, or set " +
+                "ClusterNode.MaintenanceMode = 1 while it is down for maintenance"));
+        }
+
+        return findings
+            .OrderBy(f => f.NodeId, StringComparer.OrdinalIgnoreCase)
+            .ThenBy(f => f.Kind)
+            .ToList();
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconcilerActor.cs

@@ -0,0 +1,136 @@
+using Akka.Actor;
+using Akka.Cluster;
+using Akka.Event;
+using Microsoft.EntityFrameworkCore;
+using ZB.MOM.WW.OtOpcUa.Configuration;
+
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
+
+/// <summary>
+///     Admin-role cluster singleton that runs <see cref="ClusterNodeAddressReconciler"/> against live
+///     membership and logs what it finds. A singleton rather than a per-admin-node actor so a
+///     two-admin fleet reports each disagreement once.
+/// </summary>
+/// <remarks>
+///     Recomputes on membership change (debounced) and on a slow periodic sweep, and <b>logs only
+///     when the finding set changes</b>. A node legitimately down would otherwise reprint the same
+///     warning every sweep until someone silences it by ignoring the log, which is the failure mode
+///     this check exists to avoid.
+/// </remarks>
+public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimers
+{
+    /// <summary>Driver role name — only members carrying it are expected to have a ClusterNode row.</summary>
+    public const string DriverRole = "driver";
+
+    /// <summary>Collapses a burst of membership events (a rolling restart) into one reconcile.</summary>
+    public static readonly TimeSpan DebounceWindow = TimeSpan.FromSeconds(2);
+
+    /// <summary>Periodic sweep, so a row edited in the AdminUI is checked without a topology change.</summary>
+    public static readonly TimeSpan DefaultSweepInterval = TimeSpan.FromMinutes(5);
+
+    private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
+    private readonly Akka.Cluster.Cluster _cluster;
+    private readonly TimeSpan _sweepInterval;
+    private readonly ILoggingAdapter _log = Context.GetLogger();
+    private IReadOnlyList<ClusterNodeAddressMismatch>? _lastReported;
+
+    /// <summary>Gets the timer scheduler for this actor.</summary>
+    public ITimerScheduler Timers { get; set; } = null!;
+
+    /// <summary>Creates Props for the reconciler.</summary>
+    /// <param name="dbFactory">Factory for the config database context.</param>
+    /// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
+    /// <returns>Props for actor creation.</returns>
+    public static Props Props(
+        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, TimeSpan? sweepInterval = null) =>
+        Akka.Actor.Props.Create(() => new ClusterNodeAddressReconcilerActor(dbFactory, sweepInterval));
+
+    /// <summary>Initializes a new instance of the <see cref="ClusterNodeAddressReconcilerActor"/> class.</summary>
+    /// <param name="dbFactory">Factory for the config database context.</param>
+    /// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
+    public ClusterNodeAddressReconcilerActor(
+        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, TimeSpan? sweepInterval = null)
+    {
+        _dbFactory = dbFactory;
+        _cluster = Akka.Cluster.Cluster.Get(Context.System);
+        _sweepInterval = sweepInterval ?? DefaultSweepInterval;
+
+        Receive<ClusterEvent.IMemberEvent>(_ => ScheduleReconcile());
+        Receive<ClusterEvent.CurrentClusterState>(_ => ScheduleReconcile());
+        Receive<ReconcileNow>(_ => Reconcile());
+    }
+
+    /// <inheritdoc />
+    protected override void PreStart()
+    {
+        _cluster.Subscribe(Self, ClusterEvent.InitialStateAsEvents, typeof(ClusterEvent.IMemberEvent));
+        Timers.StartPeriodicTimer("sweep", ReconcileNow.Instance, _sweepInterval, _sweepInterval);
+    }
+
+    /// <inheritdoc />
+    protected override void PostStop() => _cluster.Unsubscribe(Self);
+
+    private void ScheduleReconcile() =>
+        Timers.StartSingleTimer("debounce", ReconcileNow.Instance, DebounceWindow);
+
+    private void Reconcile()
+    {
+        var observed = _cluster.State.Members
+            .Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(DriverRole))
+            .Select(m => (Host: m.Address.Host ?? string.Empty, Port: m.Address.Port ?? 0))
+            .Where(a => !string.IsNullOrWhiteSpace(a.Host) && a.Port > 0)
+            .ToList();
+
+        List<ClusterNodeAddress> rows;
+        List<string> enabled;
+        try
+        {
+            using var db = _dbFactory.CreateDbContext();
+            rows = db.ClusterNodes.AsNoTracking()
+                .Select(n => new ClusterNodeAddress(n.NodeId, n.Host, n.AkkaPort))
+                .ToList();
+            // Same predicate the coordinator's expected-ack set uses — a node the deploy path will
+            // not wait for must not be reported as missing either, or the maintenance hatch trades a
+            // failed deployment for a permanent warning.
+            enabled = db.ClusterNodes.AsNoTracking()
+                .Where(n => n.Enabled && !n.MaintenanceMode).Select(n => n.NodeId).ToList();
+        }
+        catch (Exception ex)
+        {
+            // A consistency check must never take the admin node down with it. Central SQL being
+            // briefly unreachable is an operational fact, not a reason to restart this singleton.
+            _log.Warning(ex, "ClusterNode address reconcile skipped — config database unreachable");
+            return;
+        }
+
+        var findings = ClusterNodeAddressReconciler.Reconcile(observed, rows, enabled);
+
+        if (_lastReported is not null && findings.SequenceEqual(_lastReported)) return;
+        _lastReported = findings;
+
+        if (findings.Count == 0)
+        {
+            _log.Info("ClusterNode addresses reconcile with cluster membership ({Count} driver members)",
+                observed.Count);
+            return;
+        }
+
+        foreach (var f in findings)
+        {
+            // Error for the two shapes that are always a misconfiguration; Warning for a row whose
+            // node is merely absent, which is a legitimate state during maintenance.
+            if (f.Kind == AddressMismatchKind.EnabledRowNotInCluster)
+                _log.Warning("ClusterNode address check [{Kind}] {NodeId}: {Detail}", f.Kind, f.NodeId, f.Detail);
+            else
+                _log.Error("ClusterNode address check [{Kind}] {NodeId}: {Detail}", f.Kind, f.NodeId, f.Detail);
+        }
+    }
+
+    /// <summary>Self-message: run a reconcile pass now.</summary>
+    public sealed class ReconcileNow
+    {
+        /// <summary>The singleton instance.</summary>
+        public static readonly ReconcileNow Instance = new();
+        private ReconcileNow() { }
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs

@@ -0,0 +1,54 @@
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
+
+/// <summary>
+/// A read-only view of the cluster's driver membership, as the manual-failover control sees it.
+/// </summary>
+/// <param name="PrimaryAddress">
+/// The current driver Primary — the oldest Up <c>driver</c> member, the same node
+/// <see cref="RedundancyStateActor.SelectDriverPrimary"/> names — or <see langword="null"/> when
+/// there is no Up driver member at all.
+/// </param>
+/// <param name="DriverAddresses">
+/// Every Up <c>driver</c> member, oldest first. A manual failover needs at least two: with one
+/// there is nothing to fail over to.
+/// </param>
+public sealed record ManualFailoverSnapshot(string? PrimaryAddress, IReadOnlyList<string> DriverAddresses)
+{
+    /// <summary>Whether a failover can be performed — i.e. a peer exists to take over.</summary>
+    public bool CanFailOver => DriverAddresses.Count >= 2;
+}
+
+/// <summary>
+/// Operator-initiated, graceful swap of the driver Primary.
+/// </summary>
+/// <remarks>
+///     <para>
+///         The mechanism is a cluster <b>Leave</b> of the current Primary, never a <c>Down</c>. The
+///         leaving node hands its singletons over through the normal cluster-leave phases,
+///         <c>CoordinatedShutdown</c> runs, <c>ActorSystemTerminationWatchdog</c> exits the process,
+///         the service supervisor restarts it, and it rejoins as the youngest member — so the
+///         survivor becomes the oldest Up driver member and therefore Primary. ServiceLevel 250
+///         moves with it and OPC UA clients re-select.
+///     </para>
+///     <para>
+///         <b>Mesh-scope caveat.</b> Until the per-cluster mesh work lands
+///         (<c>docs/plans/2026-07-21-per-cluster-mesh-design.md</c>) the Primary is elected once per
+///         Akka mesh, not per application <c>Cluster</c> row. On a fleet running several application
+///         clusters in one mesh this acts on the whole mesh's Primary. The UI says so.
+///     </para>
+/// </remarks>
+public interface IManualFailoverService
+{
+    /// <summary>Reads the current driver membership from live cluster state.</summary>
+    ManualFailoverSnapshot GetSnapshot();
+
+    /// <summary>
+    /// Gracefully removes the current driver Primary from the cluster so its peer takes over.
+    /// </summary>
+    /// <param name="actor">The authenticated identity performing the failover; recorded in the audit event.</param>
+    /// <returns>
+    /// The address of the node asked to leave, or <see langword="null"/> when the failover was
+    /// refused because no driver peer exists (in which case nothing was changed and nothing audited).
+    /// </returns>
+    Task<string?> FailOverDriverPrimaryAsync(string actor);
+}

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/ManualFailoverService.cs

@@ -0,0 +1,128 @@
+using System.Text.Json;
+using Akka.Actor;
+using Akka.Cluster;
+using Microsoft.Extensions.Logging;
+using ZB.MOM.WW.Audit;
+
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
+
+/// <summary>
+/// Default <see cref="IManualFailoverService"/> — a graceful cluster <c>Leave</c> of the oldest Up
+/// <c>driver</c> member. See the interface for the full behaviour and the mesh-scope caveat.
+/// </summary>
+public sealed class ManualFailoverService : IManualFailoverService
+{
+    /// <summary>The audit <see cref="AuditEvent.Action"/> stamped onto every manual failover.</summary>
+    public const string AuditAction = "cluster.manual-failover";
+
+    /// <summary>The audit <see cref="AuditEvent.Category"/> stamped onto every manual failover.</summary>
+    public const string AuditCategory = "Redundancy";
+
+    private readonly Func<ActorSystem> _system;
+    private readonly IAuditWriter _audit;
+    private readonly ILogger<ManualFailoverService> _log;
+
+    /// <summary>
+    /// Creates the service. The ActorSystem is taken as a factory, never as a constructed instance:
+    /// this is registered before Akka's hosted service has built the system, so resolving eagerly
+    /// would race startup (the same lazy-accessor pattern <c>ActorSystemTerminationWatchdog</c> uses).
+    /// </summary>
+    /// <param name="system">Lazy accessor for the node's ActorSystem.</param>
+    /// <param name="audit">The shared audit seam; a failover is recorded before it is performed.</param>
+    /// <param name="log">Logger for the failover decision.</param>
+    public ManualFailoverService(Func<ActorSystem> system, IAuditWriter audit, ILogger<ManualFailoverService> log)
+    {
+        _system = system ?? throw new ArgumentNullException(nameof(system));
+        _audit = audit ?? throw new ArgumentNullException(nameof(audit));
+        _log = log ?? throw new ArgumentNullException(nameof(log));
+    }
+
+    /// <inheritdoc/>
+    public ManualFailoverSnapshot GetSnapshot() => SnapshotCore(_system());
+
+    /// <inheritdoc/>
+    public async Task<string?> FailOverDriverPrimaryAsync(string actor)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(actor);
+
+        var system = _system();
+
+        // Dry run first so the peer guard is evaluated before anything is audited: a refused
+        // failover changed nothing and must not leave an audit row implying it did.
+        var target = FailOverCore(system, dryRun: true);
+        if (target is null)
+        {
+            _log.LogWarning(
+                "Manual failover refused for {Actor}: fewer than two Up driver members — there is no peer to take over.",
+                actor);
+            return null;
+        }
+
+        await _audit.WriteAsync(new AuditEvent
+        {
+            EventId = Guid.NewGuid(),
+            OccurredAtUtc = DateTimeOffset.UtcNow,
+            Actor = actor,
+            Action = AuditAction,
+            Category = AuditCategory,
+            SourceNode = target.ToString(),
+            Outcome = AuditOutcome.Success,
+            DetailsJson = JsonSerializer.Serialize(new
+            {
+                target = target.ToString(),
+                driverMembers = SnapshotCore(system).DriverAddresses,
+            }),
+        }).ConfigureAwait(false);
+
+        _log.LogWarning(
+            "Manual failover requested by {Actor}: asking the driver Primary {Target} to leave the cluster. "
+            + "It hands over its singletons, shuts down, and is restarted by its supervisor as the youngest "
+            + "member; its peer becomes Primary and advertises the primary ServiceLevel.",
+            actor,
+            target);
+
+        FailOverCore(system, dryRun: false);
+        return target.ToString();
+    }
+
+    /// <summary>
+    /// Selects — and, unless <paramref name="dryRun"/>, gracefully removes — the driver Primary.
+    /// </summary>
+    /// <param name="system">The ActorSystem whose cluster is acted on.</param>
+    /// <param name="dryRun">When <see langword="true"/>, selects the target without leaving.</param>
+    /// <returns>The target address, or <see langword="null"/> when fewer than two Up driver members exist.</returns>
+    public static Address? FailOverCore(ActorSystem system, bool dryRun = false)
+    {
+        ArgumentNullException.ThrowIfNull(system);
+
+        var cluster = Akka.Cluster.Cluster.Get(system);
+
+        // Intentionally identical to RedundancyStateActor.SelectDriverPrimary (oldest Up driver,
+        // Member.AgeOrdering) — the node acted on must be exactly the one the election names, or a
+        // failover would remove a node that was not Primary and change nothing. Pinned by
+        // ManualFailoverServiceTests.Parity_with_SelectDriverPrimary.
+        var drivers = OrderedDrivers(cluster.State.Members);
+
+        // A failover needs somewhere to fail over TO. With a single driver member, leaving would be
+        // a shutdown dressed up as a failover.
+        if (drivers.Count < 2) return null;
+
+        var primary = drivers[0];
+        if (!dryRun) cluster.Leave(primary.Address);
+        return primary.Address;
+    }
+
+    private static ManualFailoverSnapshot SnapshotCore(ActorSystem system)
+    {
+        var drivers = OrderedDrivers(Akka.Cluster.Cluster.Get(system).State.Members);
+
+        return new ManualFailoverSnapshot(
+            drivers.FirstOrDefault()?.Address.ToString(),
+            drivers.Select(m => m.Address.ToString()).ToList());
+    }
+
+    private static List<Member> OrderedDrivers(IEnumerable<Member> members) => members
+        .Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(RedundancyStateActor.DriverRole))
+        .OrderBy(m => m, Member.AgeOrdering)
+        .ToList();
+}

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/RedundancyStateActor.cs

@@ -111,9 +111,49 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
         _log.Debug("Published RedundancyStateChanged with {Count} nodes", snapshot.Count);
     }
 
+    /// <summary>The cluster role that marks a node as carrying the driver runtime.</summary>
+    public const string DriverRole = "driver";
+
+    /// <summary>
+    /// Selects the driver Primary: the <b>oldest</b> Up member carrying the <see cref="DriverRole"/>.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         This deliberately does <b>not</b> use <c>ClusterState.RoleLeader(role)</c>, which it
+    ///         previously did. Role leader is the <i>lowest-addressed</i> Up member with the role —
+    ///         address order (host, then port), with no relationship to time. Oldest is the member
+    ///         with the lowest up-number, and it is what <c>ClusterSingletonManager</c> uses to place
+    ///         singletons.
+    ///     </para>
+    ///     <para>
+    ///         On a freshly-formed cluster the two agree, which is why the discrepancy was invisible.
+    ///         They diverge after any restart: the restarted node re-joins as the youngest but keeps
+    ///         its address, so if it holds the lower address it becomes role leader while the
+    ///         singletons — and all the work they own — stay on the other node. Electing it Primary
+    ///         would enable the Primary-gated data plane (inbound writes, alarm acks, the fleet-wide
+    ///         alerts emit, the alarm-history drain) on the node that is not hosting the work.
+    ///     </para>
+    ///     <para>
+    ///         Only <see cref="MemberStatus.Up"/> members are eligible, matching singleton placement:
+    ///         a <c>Leaving</c> node is handing its singletons over and must not be named Primary.
+    ///     </para>
+    /// </remarks>
+    /// <param name="members">The current cluster members, in any order.</param>
+    /// <returns>The oldest Up driver member's address, or <c>null</c> when there is none.</returns>
+    public static Address? SelectDriverPrimary(IEnumerable<Member> members)
+    {
+        ArgumentNullException.ThrowIfNull(members);
+
+        return members
+            .Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(DriverRole))
+            .OrderBy(m => m, Member.AgeOrdering)
+            .FirstOrDefault()
+            ?.Address;
+    }
+
     private IReadOnlyList<NodeRedundancyState> BuildSnapshot()
     {
-        var driverLeader = _cluster.State.RoleLeader("driver");
+        var driverPrimary = SelectDriverPrimary(_cluster.State.Members);
         var clusterLeader = _cluster.State.Leader;
         var now = DateTime.UtcNow;
 
@@ -123,15 +163,16 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
             var host = member.Address.Host;
             if (string.IsNullOrWhiteSpace(host)) continue;
 
-            var role = member.Roles.Contains("driver")
-                ? (driverLeader == member.Address ? CommonsRedundancyRole.Primary : CommonsRedundancyRole.Secondary)
+            var isPrimary = driverPrimary is not null && driverPrimary == member.Address;
+            var role = member.Roles.Contains(DriverRole)
+                ? (isPrimary ? CommonsRedundancyRole.Primary : CommonsRedundancyRole.Secondary)
                 : CommonsRedundancyRole.Detached;
 
             list.Add(new NodeRedundancyState(
                 ToNodeId(member.Address),
                 role,
                 IsClusterLeader: clusterLeader == member.Address,
-                IsRoleLeaderForDriver: driverLeader == member.Address,
+                IsDriverPrimary: isPrimary,
                 AsOfUtc: now));
         }
         return list;

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs

@@ -23,9 +23,11 @@ public static class ServiceCollectionExtensions
     public const string AuditWriterSingletonName = "audit-writer";
     public const string FleetStatusSingletonName = "fleet-status";
     public const string RedundancyStateSingletonName = "redundancy-state";
+    public const string ClusterNodeAddressReconcilerSingletonName = "cluster-node-address-reconciler";
 
     /// <summary>
-    /// Registers all five admin-role cluster singletons + their proxies on the AkkaConfigurationBuilder.
+    /// Registers the admin-role cluster singletons on the AkkaConfigurationBuilder — five with
+    /// registry proxies, plus the address reconciler, which is proxy-less because nothing addresses it.
     /// Must be called against the same builder used by <c>AkkaHostedService</c> so the singletons
     /// share the host's ActorSystem.
     /// </summary>
@@ -90,6 +92,26 @@ public static class ServiceCollectionExtensions
             (system, registry, resolver) => RedundancyStateActor.Props(),
             singletonOptions);
 
+        // Per-cluster mesh Phase 1: ClusterNode.AkkaPort duplicates the node's own Cluster:Port and
+        // nothing makes them agree. Runs here rather than on each driver node because Phase 4 removes
+        // the driver nodes' ConfigDb connection entirely.
+        //
+        // createProxyToo: false — unlike the five singletons above, nothing ever resolves
+        // ClusterNodeAddressReconcilerKey from the registry. This actor only reads cluster state and
+        // logs; no one sends it messages. A proxy would be a second actor per node whose only job is
+        // to hunt for a singleton nobody addresses — visible as the "ClusterSingletonProxy failed to
+        // find an associated singleton after 30 seconds" startup warning, and as extra cluster
+        // chatter in every two-node test harness the integration suite builds.
+        builder.WithSingleton<ClusterNodeAddressReconcilerKey>(
+            ClusterNodeAddressReconcilerSingletonName,
+            (system, registry, resolver) =>
+            {
+                var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
+                return ClusterNodeAddressReconcilerActor.Props(dbFactory);
+            },
+            singletonOptions,
+            createProxyToo: false);
+
         return builder;
     }
 }
@@ -100,3 +122,4 @@ public sealed class AdminOperationsActorKey { }
 public sealed class AuditWriterActorKey { }
 public sealed class FleetStatusBroadcasterKey { }
 public sealed class RedundancyStateActorKey { }
+public sealed class ClusterNodeAddressReconcilerKey { }

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/AlarmSfLegacyMigrator.cs

@@ -0,0 +1,217 @@
+using Microsoft.Data.Sqlite;
+using Microsoft.Extensions.Configuration;
+using ZB.MOM.WW.LocalDb;
+using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     One-time copy of the pre-consolidation <c>alarm-historian.db</c> store-and-forward queue
+///     into the consolidated <c>ZB.MOM.WW.LocalDb</c> database.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>What is at stake.</b> Rows sit in that file precisely because the historian could not
+///         be reached. Discarding them on upgrade would throw away exactly the alarm audit trail
+///         the store-and-forward queue exists to protect — so the copy tolerates an older column
+///         set rather than bailing out, and refuses to rename a file it did not actually read.
+///     </para>
+///     <para>
+///         <b>Runs after <c>RegisterReplicated</c>, deliberately.</b> Capture is trigger-based, so
+///         rows inserted before registration never enter the oplog and never reach the peer —
+///         silently, and permanently, because nothing revisits history. Migrating after
+///         registration means the recovered backlog replicates like any other write.
+///     </para>
+///     <para>
+///         <b>Ids come from the payload, not from the legacy row id.</b> The legacy primary key was
+///         <c>RowId INTEGER PRIMARY KEY AUTOINCREMENT</c>, which cannot survive into a replicated
+///         table: each node allocated its own sequence, so node A's row 7 and node B's row 7 are
+///         different alarms that would silently overwrite one another under last-writer-wins.
+///         Hashing the payload (<see cref="AlarmSfSchema.DeriveId"/>) solves that and one more
+///         problem besides — a warm pair's two legacy files <i>overlap</i>, because
+///         <c>HistorianAdapterActor</c> default-writes while its redundancy role is unknown and
+///         both nodes therefore accepted the same transitions during every boot window. A
+///         node-prefixed id would carry that duplication into the merged buffer forever; an
+///         equal-payload id collapses it. It is also what makes a re-run harmless under
+///         <c>INSERT OR IGNORE</c>.
+///     </para>
+///     <para>
+///         <b>All-or-nothing.</b> The copy runs in one transaction and the legacy file is renamed
+///         to <c>&lt;name&gt;.migrated</c> only after the commit. A failure throws out of
+///         <c>OnReady</c>, failing host startup with the legacy file untouched — no half-migrated
+///         state to reason about, and the operator still holds the original. A later boot sees the
+///         renamed file and no-ops.
+///     </para>
+/// </remarks>
+public static class AlarmSfLegacyMigrator
+{
+    private const string MigratedSuffix = ".migrated";
+
+    /// <summary>The legacy queue table.</summary>
+    private const string LegacyTable = "Queue";
+
+    /// <summary>
+    ///     The configuration key that used to carry the standalone queue file's path. Read as a raw
+    ///     key because the corresponding <c>AlarmHistorianOptions</c> property was removed with the
+    ///     bespoke file management it configured.
+    /// </summary>
+    public const string LegacyPathKey = "AlarmHistorian:DatabasePath";
+
+    /// <summary>The pre-removal default for <see cref="LegacyPathKey"/>.</summary>
+    private const string DefaultLegacyPath = "alarm-historian.db";
+
+    /// <summary>
+    ///     The legacy column whose absence means this is not a queue file we can read. Without the
+    ///     payload there is no event and no id to derive from one.
+    /// </summary>
+    private const string RequiredColumn = "PayloadJson";
+
+    /// <summary>
+    ///     Every legacy column, mapped to its consolidated counterpart. Columns absent from an
+    ///     older file are dropped from the copy rather than failing it — see
+    ///     <see cref="PresentColumns"/>.
+    /// </summary>
+    private static readonly (string Legacy, string Current)[] ColumnMap =
+    [
+        ("AlarmId", "alarm_id"),
+        ("EnqueuedUtc", "enqueued_at_utc"),
+        ("PayloadJson", "payload_json"),
+        ("AttemptCount", "attempt_count"),
+        ("LastAttemptUtc", "last_attempt_utc"),
+        ("LastError", "last_error"),
+        ("DeadLettered", "dead_lettered"),
+    ];
+
+    /// <summary>
+    ///     Copies any legacy alarm queue into <paramref name="db"/>, then renames the legacy file.
+    /// </summary>
+    /// <param name="db">The consolidated database, with <c>alarm_sf_events</c> already registered.</param>
+    /// <param name="config">Configuration supplying the legacy queue's path.</param>
+    public static void Migrate(ILocalDb db, IConfiguration config)
+    {
+        ArgumentNullException.ThrowIfNull(db);
+        ArgumentNullException.ThrowIfNull(config);
+
+        var legacyPath = ResolveLegacyPath(config);
+        if (!ShouldMigrate(legacyPath)) return;
+
+        using (var legacy = OpenLegacyReadOnly(legacyPath))
+        {
+            var present = PresentColumns(legacy);
+
+            // An absent table probes as zero columns, so this one guard covers both "no queue
+            // table" and "a shape we do not recognise". Returning without renaming leaves the file
+            // for an operator to inspect.
+            if (!present.Contains(RequiredColumn)) return;
+
+            using var connection = db.CreateConnection();
+            using var transaction = connection.BeginTransaction();
+            CopyRows(legacy, connection, transaction, present);
+            transaction.Commit();
+        }
+
+        // Only after the commit. A crash between the two leaves the file in place and the next
+        // boot copies it again, which the payload-derived ids make harmless.
+        File.Move(legacyPath, legacyPath + MigratedSuffix, overwrite: true);
+    }
+
+    /// <summary>
+    ///     Resolves the legacy queue path from the removed configuration key, falling back to the
+    ///     code default it used to carry.
+    /// </summary>
+    /// <remarks>
+    ///     Relative paths resolve against the process working directory — not the LocalDb
+    ///     directory — because that is where the old code actually put them.
+    /// </remarks>
+    /// <param name="config">The application configuration.</param>
+    /// <returns>An absolute path, or empty when there is nothing durable to migrate from.</returns>
+    internal static string ResolveLegacyPath(IConfiguration config)
+    {
+        var path = config[LegacyPathKey] ?? DefaultLegacyPath;
+
+        // In-memory and URI-form sources (test / dev configurations) have nothing durable behind
+        // them, and Path.GetFullPath on them would produce nonsense.
+        if (string.IsNullOrWhiteSpace(path) ||
+            path.Equals(":memory:", StringComparison.OrdinalIgnoreCase) ||
+            path.StartsWith("file:", StringComparison.OrdinalIgnoreCase))
+        {
+            return string.Empty;
+        }
+
+        return Path.GetFullPath(path);
+    }
+
+    /// <summary>Whether there is a legacy file worth opening.</summary>
+    private static bool ShouldMigrate(string legacyPath) =>
+        !string.IsNullOrEmpty(legacyPath)
+        && File.Exists(legacyPath)
+        && !File.Exists(legacyPath + MigratedSuffix);
+
+    /// <summary>Opens the legacy file read-only, so a failed migration cannot damage it.</summary>
+    private static SqliteConnection OpenLegacyReadOnly(string path)
+    {
+        var connection = new SqliteConnection(new SqliteConnectionStringBuilder
+        {
+            DataSource = path,
+            Mode = SqliteOpenMode.ReadOnly,
+        }.ToString());
+        connection.Open();
+        return connection;
+    }
+
+    /// <summary>
+    ///     Which of the columns we know about the legacy table actually has.
+    /// </summary>
+    /// <remarks>
+    ///     A file written by an older build predates some columns, and naming a missing one in the
+    ///     SELECT throws "no such column" — which discards every row in the table rather than the
+    ///     one field. Intersecting first means an old file migrates its data and simply leaves the
+    ///     newer columns at their schema defaults.
+    /// </remarks>
+    private static HashSet<string> PresentColumns(SqliteConnection legacy)
+    {
+        var present = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
+
+        using var cmd = legacy.CreateCommand();
+        cmd.CommandText = $"SELECT name FROM pragma_table_info('{LegacyTable}')";
+        using var reader = cmd.ExecuteReader();
+        while (reader.Read())
+            present.Add(reader.GetString(0));
+
+        return present;
+    }
+
+    /// <summary>Copies every legacy row into the consolidated table, keyed by payload hash.</summary>
+    private static void CopyRows(
+        SqliteConnection legacy,
+        SqliteConnection target,
+        SqliteTransaction transaction,
+        HashSet<string> present)
+    {
+        var columns = ColumnMap.Where(c => present.Contains(c.Legacy)).ToArray();
+        var payloadOrdinal = Array.FindIndex(columns, c => c.Legacy == RequiredColumn);
+
+        using var read = legacy.CreateCommand();
+        read.CommandText =
+            $"SELECT {string.Join(", ", columns.Select(c => c.Legacy))} FROM {LegacyTable}";
+
+        using var reader = read.ExecuteReader();
+        while (reader.Read())
+        {
+            using var insert = target.CreateCommand();
+            insert.Transaction = transaction;
+            insert.CommandText = $"""
+                INSERT OR IGNORE INTO {AlarmSfSchema.EventsTable}
+                    ({AlarmSfSchema.IdColumn}, {string.Join(", ", columns.Select(c => c.Current))})
+                VALUES
+                    ($id, {string.Join(", ", columns.Select((_, i) => $"$c{i}"))})
+                """;
+
+            insert.Parameters.AddWithValue("$id", AlarmSfSchema.DeriveId(reader.GetString(payloadOrdinal)));
+            for (var i = 0; i < columns.Length; i++)
+                insert.Parameters.AddWithValue($"$c{i}", reader.GetValue(i) ?? DBNull.Value);
+
+            insert.ExecuteNonQuery();
+        }
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/KestrelHttpBinding.cs

@@ -0,0 +1,102 @@
+using System.Net;
+using Microsoft.AspNetCore.Server.Kestrel.Core;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     One HTTP endpoint the host was asked to serve, parsed out of the configured URL list so it
+///     can be re-bound explicitly.
+/// </summary>
+/// <param name="Host">The host component as written — <c>+</c>, <c>*</c>, a hostname, or an IP.</param>
+/// <param name="Port">The port.</param>
+/// <param name="IsSecure">True when the URL used the <c>https</c> scheme.</param>
+public sealed record KestrelHttpBinding(string Host, int Port, bool IsSecure)
+{
+    /// <summary>
+    ///     Parses a semicolon-separated URL list (the <c>ASPNETCORE_URLS</c> / <c>urls</c> format)
+    ///     into bindings. Unparseable entries are skipped rather than throwing — a malformed URL
+    ///     must not take the process down at startup.
+    /// </summary>
+    /// <param name="urls">The configured URL list, or null/empty when none is configured.</param>
+    /// <returns>The parsed bindings, in the order given; empty when nothing is configured.</returns>
+    public static IReadOnlyList<KestrelHttpBinding> Parse(string? urls)
+    {
+        if (string.IsNullOrWhiteSpace(urls))
+            return [];
+
+        var result = new List<KestrelHttpBinding>();
+        foreach (var raw in urls.Split(';', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries))
+        {
+            // Uri cannot parse the "+"/"*" wildcard hosts Kestrel accepts, so substitute a
+            // placeholder host purely to get scheme/port out, then keep the original host token.
+            var isWildcard = raw.Contains("://+", StringComparison.Ordinal)
+                          || raw.Contains("://*", StringComparison.Ordinal);
+            var probe = isWildcard
+                ? raw.Replace("://+", "://placeholder", StringComparison.Ordinal)
+                     .Replace("://*", "://placeholder", StringComparison.Ordinal)
+                : raw;
+
+            if (!Uri.TryCreate(probe, UriKind.Absolute, out var uri))
+                continue;
+
+            var host = isWildcard
+                ? raw.Contains("://+", StringComparison.Ordinal) ? "+" : "*"
+                : uri.Host;
+
+            result.Add(new KestrelHttpBinding(host, uri.Port, uri.Scheme == Uri.UriSchemeHttps));
+        }
+
+        return result;
+    }
+
+    /// <summary>
+    ///     Parses the bare-port list format of <c>ASPNETCORE_HTTP_PORTS</c> / <c>HTTP_PORTS</c> (and
+    ///     the <c>HTTPS</c> variants) into wildcard (<c>+</c>, all-interfaces) bindings — the shape
+    ///     Kestrel itself gives those variables. Modern .NET base images (aspnet:8.0+) set
+    ///     <c>ASPNETCORE_HTTP_PORTS=8080</c> as the container default <b>instead of</b>
+    ///     <c>ASPNETCORE_URLS</c>, so a node that never sets <c>URLS</c> explicitly still has a real
+    ///     configured surface here that must be re-bound, not replaced with Kestrel's localhost:5000.
+    /// </summary>
+    /// <param name="ports">A <c>;</c>- or <c>,</c>-separated list of bare ports, or null/empty.</param>
+    /// <param name="isSecure">True to mark the parsed bindings <c>https</c> (the <c>HTTPS_PORTS</c> vars).</param>
+    /// <returns>One all-interfaces binding per parseable port; empty when nothing is configured.</returns>
+    public static IReadOnlyList<KestrelHttpBinding> ParsePorts(string? ports, bool isSecure)
+    {
+        if (string.IsNullOrWhiteSpace(ports))
+            return [];
+
+        var result = new List<KestrelHttpBinding>();
+        foreach (var token in ports.Split([';', ','], StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries))
+        {
+            // Skip a malformed port rather than throwing — a bad env var must not down startup.
+            if (int.TryParse(token, out var port) && port is > 0 and <= 65535)
+                result.Add(new KestrelHttpBinding("+", port, isSecure));
+        }
+
+        return result;
+    }
+
+    /// <summary>
+    ///     Applies this binding to Kestrel, choosing the narrowest listen call the host component
+    ///     allows.
+    /// </summary>
+    /// <remarks>
+    ///     A hostname that is neither a literal IP nor <c>localhost</c> cannot be reliably mapped to
+    ///     a local interface, so it falls back to <see cref="KestrelServerOptions.ListenAnyIP"/> —
+    ///     the safe superset. Narrowing it and guessing wrong would silently stop serving.
+    /// </remarks>
+    /// <param name="options">The Kestrel options being configured.</param>
+    public void Apply(KestrelServerOptions options)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+
+        if (Host is "+" or "*")
+            options.ListenAnyIP(Port);
+        else if (IPAddress.TryParse(Host, out var address))
+            options.Listen(address, Port);
+        else if (string.Equals(Host, "localhost", StringComparison.OrdinalIgnoreCase))
+            options.ListenLocalhost(Port);
+        else
+            options.ListenAnyIP(Port);
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs

@@ -0,0 +1,101 @@
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.DependencyInjection;
+using ZB.MOM.WW.LocalDb;
+using ZB.MOM.WW.LocalDb.Replication;
+using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     Single registration point for the node-local LocalDb subsystem: the embedded SQLite store
+///     that caches the deployed-configuration artifact, plus the optional gRPC replication that
+///     mirrors it to the node's redundant pair peer.
+/// </summary>
+/// <remarks>
+///     <para>
+///         Exists as a named extension rather than inline <c>AddZbLocalDb</c> calls in
+///         <c>Program.cs</c> so the wiring is covered by a DI resolution test — <c>Program.cs</c> is
+///         top-level statements and cannot be exercised directly, which is exactly how a
+///         "registered but never resolvable" defect ships unnoticed. This family has shipped that
+///         defect three times (Secrets 0.2.0, Secrets 0.2.2, ScadaBridge #22).
+///     </para>
+///     <para>
+///         <b>Driver-role nodes only.</b> Admin-only nodes have no deployed configuration to cache,
+///         and registering here would impose the <c>LocalDb:Path</c>-required-or-no-boot constraint
+///         on them for nothing.
+///     </para>
+///     <para>
+///         <b>Storage is unconditional; replication is default-OFF.</b> A node with no
+///         <c>PeerAddress</c> and no <c>SyncListenPort</c> is simply a fast local SQLite file —
+///         <c>AddZbLocalDbReplication</c> registers the engine but it stays idle. Replication is
+///         opt-in per pair because there is no production distribution story for the sync
+///         <c>ApiKey</c> yet (the same open question the Secrets KEK has).
+///     </para>
+/// </remarks>
+public static class LocalDbRegistration
+{
+    /// <summary>Configuration section holding <c>LocalDbOptions</c>.</summary>
+    public const string LocalDbSectionPath = "LocalDb";
+
+    /// <summary>Configuration section holding <c>ReplicationOptions</c>.</summary>
+    public const string ReplicationSectionPath = "LocalDb:Replication";
+
+    /// <summary>
+    ///     Configuration key carrying the dedicated h2c sync listener's port. Zero or absent means
+    ///     no listener is bound and the passive sync endpoint is not mapped.
+    /// </summary>
+    public const string SyncListenPortKey = "LocalDb:SyncListenPort";
+
+    /// <summary>
+    ///     The port the dedicated cleartext-HTTP/2 sync listener should bind, or <c>0</c> when the
+    ///     node should not listen at all. Defaults to <c>0</c> — absent configuration must mean
+    ///     "off".
+    /// </summary>
+    /// <param name="configuration">The application configuration.</param>
+    /// <returns>The configured sync port, or <c>0</c>.</returns>
+    public static int SyncListenPort(IConfiguration configuration)
+    {
+        ArgumentNullException.ThrowIfNull(configuration);
+        return configuration.GetValue(SyncListenPortKey, defaultValue: 0);
+    }
+
+    /// <summary>
+    ///     Registers the local database (with the deployment-cache schema applied and registered for
+    ///     replication) and the replication engine.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <see cref="LocalDbSetup.OnReady"/> runs inside <c>AddZbLocalDb</c>'s singleton factory,
+    ///         before the first caller receives the <c>ILocalDb</c> — so every consumer is guaranteed
+    ///         to see the tables created and their capture triggers installed.
+    ///     </para>
+    ///     <para>
+    ///         <c>AddZbLocalDbReplication</c> is called unconditionally rather than behind an
+    ///         <c>Enabled</c> flag because the library already models "off" as "no
+    ///         <c>PeerAddress</c>": the initiator background service idles and the passive endpoint
+    ///         is only reachable if <c>Program.cs</c> maps it, which it does only when
+    ///         <see cref="SyncListenPortKey"/> is set. Adding a second gate on top would give two
+    ///         ways to express the same thing and a way for them to disagree.
+    ///     </para>
+    /// </remarks>
+    /// <param name="services">The service collection to add to.</param>
+    /// <param name="configuration">The application configuration.</param>
+    /// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
+    public static IServiceCollection AddOtOpcUaLocalDb(
+        this IServiceCollection services,
+        IConfiguration configuration)
+    {
+        ArgumentNullException.ThrowIfNull(services);
+        ArgumentNullException.ThrowIfNull(configuration);
+
+        services.AddZbLocalDb(configuration, db => LocalDbSetup.OnReady(db, configuration));
+        services.AddZbLocalDbReplication(configuration);
+
+        // The consumer-facing seam. WithOtOpcUaRuntimeActors resolves this optionally and threads it
+        // into DriverHostActor; registering the store without this leaves the cache tables present,
+        // replicating, and never written to.
+        services.AddSingleton<IDeploymentArtifactCache, LocalDbDeploymentArtifactCache>();
+
+        return services;
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs

@@ -0,0 +1,68 @@
+using Microsoft.Extensions.Configuration;
+using ZB.MOM.WW.LocalDb;
+using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
+using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     The <c>onReady</c> callback handed to <c>AddZbLocalDb</c>: creates the deployment-cache and
+///     alarm store-and-forward tables and opts them into replication.
+/// </summary>
+/// <remarks>
+///     <para>
+///         Public rather than internal only so <c>LocalDbSetupTests</c> can drive the production
+///         callback directly. Initialising a test database from a hand-written copy of this schema
+///         would prove only that the test agrees with itself — the whole value of those tests is
+///         that they exercise <b>this</b> method.
+///     </para>
+/// </remarks>
+public static class LocalDbSetup
+{
+    /// <summary>
+    ///     Initialises the local database: DDL first, then replication registration.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b>THE ORDER IS LOAD-BEARING: DDL → RegisterReplicated → writes.</b>
+    ///         <c>RegisterReplicated</c> is what installs the three AFTER triggers that capture
+    ///         changes into the oplog. Any row written before that call is never captured, so it
+    ///         never reaches the peer — silently, and permanently, because nothing ever revisits
+    ///         history. <see cref="AlarmSfLegacyMigrator"/> therefore runs <b>last</b>, after every
+    ///         registration — it writes rows, and they must be captured like any other write.
+    ///     </para>
+    ///     <para>
+    ///         The alarm buffer's tables are created unconditionally, regardless of whether this
+    ///         node has <c>AlarmHistorian:Enabled</c> set. An empty registered table costs three
+    ///         triggers and nothing else, whereas creating it lazily when the sink first appears
+    ///         would mean a node that enables the historian later writes rows before its triggers
+    ///         exist — which is precisely the silent-loss shape above.
+    ///     </para>
+    /// </remarks>
+    /// <param name="db">The freshly constructed local database.</param>
+    /// <param name="configuration">
+    ///     Application configuration, read by the legacy migrator for the pre-consolidation queue's
+    ///     path. Required rather than optional: an overload that silently skipped the migration
+    ///     would be one wiring mistake away from discarding a node's undelivered alarm history.
+    /// </param>
+    public static void OnReady(ILocalDb db, IConfiguration configuration)
+    {
+        ArgumentNullException.ThrowIfNull(db);
+        ArgumentNullException.ThrowIfNull(configuration);
+
+        // CreateConnection() hands back an already-open, pragma-configured connection carrying the
+        // zb_hlc_next() UDF the capture triggers need. Calling Open() on it would throw.
+        using (var connection = db.CreateConnection())
+        {
+            DeploymentCacheSchema.Apply(connection);
+            AlarmSfSchema.Apply(connection);
+        }
+
+        db.RegisterReplicated(DeploymentCacheSchema.ArtifactsTable);
+        db.RegisterReplicated(DeploymentCacheSchema.PointerTable);
+        db.RegisterReplicated(AlarmSfSchema.EventsTable);
+
+        // LAST, and only here. This is the one call in OnReady that writes rows.
+        AlarmSfLegacyMigrator.Migrate(db, configuration);
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs

@@ -0,0 +1,162 @@
+using System.Security.Cryptography;
+using System.Text;
+using Grpc.Core;
+using Grpc.Core.Interceptors;
+using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.LocalDb.Replication;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     Gates the LocalDb passive sync endpoint. The replication library deliberately leaves inbound
+///     authentication to the host — its <c>LocalDbSyncService</c> verifies nothing — so without this
+///     interceptor anything that can reach a driver node's sync port could stream arbitrary rows
+///     into that node's deployment-artifact cache.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>Why that matters here specifically.</b> The cached artifact is what a driver node
+///         boots from when central SQL is unreachable. An attacker able to write it could choose
+///         the configuration a node comes up on during exactly the outage nobody is watching.
+///     </para>
+///     <para>
+///         <b>Scoped by method path.</b> Only calls under <c>/localdb_sync.v1.LocalDbSync/</c> are
+///         gated; every other method on the shared gRPC pipeline passes through untouched.
+///     </para>
+///     <para>
+///         <b>Fail-closed.</b> With no <c>LocalDb:Replication:ApiKey</c> configured, NO sync stream
+///         is accepted, authenticated or not. That is the deliberate choice: the alternative —
+///         treating "no key" as "no auth required" — would silently expose the endpoint on exactly
+///         the default configuration every node ships with. An operator enabling replication must
+///         set the same key on both nodes of the pair, which is already required for the initiator
+///         to dial out (<c>SyncBackgroundService</c> sends <c>Authorization: Bearer &lt;key&gt;</c>).
+///         A key typo therefore stops convergence outright rather than degrading to unauthenticated.
+///     </para>
+///     <para>
+///         Comparison is <see cref="CryptographicOperations.FixedTimeEquals"/> over UTF-8 bytes, so
+///         a wrong key cannot be recovered byte-by-byte from response timing. Length differences are
+///         unavoidably observable and are not sensitive.
+///     </para>
+/// </remarks>
+public sealed class LocalDbSyncAuthInterceptor : Interceptor
+{
+    private const string ServicePrefix = "/localdb_sync.v1.LocalDbSync/";
+    private const string AuthorizationHeader = "authorization";
+    private const string BearerPrefix = "Bearer ";
+
+    private readonly IOptions<ReplicationOptions> _options;
+    private readonly ILogger<LocalDbSyncAuthInterceptor> _logger;
+
+    /// <summary>Creates the interceptor.</summary>
+    /// <param name="options">Replication options; <c>ApiKey</c> is the expected bearer token.</param>
+    /// <param name="logger">Logger for denial diagnostics.</param>
+    public LocalDbSyncAuthInterceptor(
+        IOptions<ReplicationOptions> options,
+        ILogger<LocalDbSyncAuthInterceptor> logger)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+        ArgumentNullException.ThrowIfNull(logger);
+
+        _options = options;
+        _logger = logger;
+    }
+
+    /// <inheritdoc />
+    public override Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
+        TRequest request,
+        ServerCallContext context,
+        UnaryServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(request, context);
+    }
+
+    /// <inheritdoc />
+    public override Task DuplexStreamingServerHandler<TRequest, TResponse>(
+        IAsyncStreamReader<TRequest> requestStream,
+        IServerStreamWriter<TResponse> responseStream,
+        ServerCallContext context,
+        DuplexStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(requestStream, responseStream, context);
+    }
+
+    /// <inheritdoc />
+    public override Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
+        IAsyncStreamReader<TRequest> requestStream,
+        ServerCallContext context,
+        ClientStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(requestStream, context);
+    }
+
+    /// <inheritdoc />
+    public override Task ServerStreamingServerHandler<TRequest, TResponse>(
+        TRequest request,
+        IServerStreamWriter<TResponse> responseStream,
+        ServerCallContext context,
+        ServerStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(request, responseStream, context);
+    }
+
+    /// <summary>
+    ///     Throws <see cref="RpcException"/> with <see cref="StatusCode.PermissionDenied"/> if this
+    ///     is a sync call that does not carry the configured bearer token. Non-sync calls return
+    ///     immediately.
+    /// </summary>
+    private void Authorize(ServerCallContext context)
+    {
+        if (!context.Method.StartsWith(ServicePrefix, StringComparison.Ordinal))
+            return;
+
+        var expected = _options.Value.ApiKey;
+        if (string.IsNullOrEmpty(expected))
+        {
+            _logger.LogWarning(
+                "Rejected a LocalDb sync call to {Method}: no LocalDb:Replication:ApiKey is configured, " +
+                "so the passive sync endpoint is closed. Configure the same key on both nodes of the pair.",
+                context.Method);
+            throw new RpcException(new Status(
+                StatusCode.PermissionDenied,
+                "LocalDb sync is not accepting connections: no API key is configured on this node."));
+        }
+
+        var presented = ExtractBearerToken(context.RequestHeaders);
+        if (presented is null || !FixedTimeEquals(presented, expected))
+        {
+            _logger.LogWarning(
+                "Rejected a LocalDb sync call to {Method}: {Reason}.",
+                context.Method,
+                presented is null ? "no bearer token presented" : "bearer token did not match");
+            throw new RpcException(new Status(
+                StatusCode.PermissionDenied,
+                "LocalDb sync authentication failed."));
+        }
+    }
+
+    private static string? ExtractBearerToken(Metadata headers)
+    {
+        // gRPC lowercases header keys on the wire; compare case-insensitively anyway so a
+        // hand-built Metadata in a test behaves the same as a real request.
+        foreach (var entry in headers)
+        {
+            if (!string.Equals(entry.Key, AuthorizationHeader, StringComparison.OrdinalIgnoreCase))
+                continue;
+
+            var value = entry.Value;
+            if (value is not null && value.StartsWith(BearerPrefix, StringComparison.OrdinalIgnoreCase))
+                return value[BearerPrefix.Length..];
+        }
+
+        return null;
+    }
+
+    private static bool FixedTimeEquals(string presented, string expected)
+        => CryptographicOperations.FixedTimeEquals(
+            Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ServerHistorianOptionsValidator.cs

@@ -24,12 +24,25 @@ namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
 ///         <see cref="ServerHistorianOptions.Enabled"/> in the Host, so <c>Enabled</c> covers it.
 ///     </para>
 ///     <para>
-///         <b>Fail tier = provably-crashing configs only.</b> Only an empty / non-absolute / non-http(s)
-///         <c>Endpoint</c> fails here (it throws in the factory). Empty <c>ApiKey</c> and non-positive
-///         <c>MaxTieClusterOverfetch</c> degrade rather than crash (the gateway rejects calls / the node
-///         manager surfaces a Bad read), so they stay operator warnings in
+///         <b>Fail tier = provably-crashing configs only.</b> Three settings qualify, and they all fail
+///         the same way — <c>HistorianGatewayClientOptions.Validate()</c> throws inside the client
+///         factory before any call is attempted, so the host dies at startup:
+///         an empty / non-absolute / non-http(s) <c>Endpoint</c>; an empty <c>ApiKey</c> ("The gateway
+///         API key must not be empty"); and a <c>UseTls</c> that disagrees with the endpoint scheme
+///         ("UseTls requires an https gateway endpoint" / "An https gateway endpoint requires UseTls").
+///         A non-positive <c>MaxTieClusterOverfetch</c> genuinely degrades rather than crashes (the node
+///         manager surfaces a Bad read), so it stays an operator warning in
 ///         <see cref="ServerHistorianOptions.Validate"/>. The <c>Endpoint</c> value is not a secret and
-///         is echoed to make the error actionable; the <c>ApiKey</c> is never surfaced.
+///         is echoed to make each error actionable; the <c>ApiKey</c> is never surfaced.
+///     </para>
+///     <para>
+///         <b>The <c>ApiKey</c> and <c>UseTls</c> checks were added after two consecutive live-rig
+///         crash-loops</b> (LocalDb Phase 2 gate). Both had been classified as degrading, on the
+///         assumption that a bad client configuration would connect and be rejected by the gateway — but
+///         the client validates its own options at construction, so the process dies during Akka startup
+///         instead, which is precisely the failure this validator exists to convert into a named,
+///         aggregated error. The lesson generalises: "degrades" is only true of settings the client
+///         does not itself validate.
 ///     </para>
 /// </remarks>
 public sealed class ServerHistorianOptionsValidator : OptionsValidatorBase<ServerHistorianOptions>
@@ -62,5 +75,26 @@ public sealed class ServerHistorianOptionsValidator : OptionsValidatorBase<Serve
         builder.RequireThat(
             endpointValid,
             $"ServerHistorian:Endpoint is empty or not an absolute http(s) URI ('{options.Endpoint}') — {reason}.");
+
+        // Deliberately not echoed: unlike the endpoint, the key is a secret.
+        builder.RequireThat(
+            !string.IsNullOrWhiteSpace(options.ApiKey),
+            $"ServerHistorian:ApiKey is empty — {reason}. The gateway client rejects a keyless "
+            + "configuration at construction, so the host would crash on startup rather than degrade. "
+            + "Supply it via the environment variable ServerHistorian__ApiKey.");
+
+        // The flag and the scheme must agree in BOTH directions — the client throws either way
+        // ("UseTls requires an https gateway endpoint" / "An https gateway endpoint requires UseTls").
+        // Only meaningful once the endpoint parsed; a malformed one already failed above.
+        if (endpointValid)
+        {
+            var https = uri!.Scheme == Uri.UriSchemeHttps;
+            builder.RequireThat(
+                options.UseTls == https,
+                $"ServerHistorian:UseTls is {options.UseTls.ToString().ToLowerInvariant()} but the "
+                + $"endpoint is '{options.Endpoint}' — {reason}. The flag must match the scheme "
+                + "(https ⇒ UseTls=true, http ⇒ UseTls=false); the gateway client rejects a mismatch "
+                + "at construction, crashing the host on startup.");
+        }
     }
 }

src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs

@@ -1,10 +1,15 @@
 using Microsoft.AspNetCore.Routing;
 using Microsoft.EntityFrameworkCore;
+using Microsoft.Extensions.Configuration;
 using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Diagnostics.HealthChecks;
+using Microsoft.Extensions.Options;
 using ZB.MOM.WW.Health;
 using ZB.MOM.WW.Health.Akka;
 using ZB.MOM.WW.Health.EntityFrameworkCore;
+using ZB.MOM.WW.LocalDb.Replication;
 using ZB.MOM.WW.OtOpcUa.Configuration;
+using ZB.MOM.WW.OtOpcUa.Host.Configuration;
 
 namespace ZB.MOM.WW.OtOpcUa.Host.Health;
 
@@ -36,7 +41,20 @@ public static class HealthEndpoints
                 "admin-leader",
                 failureStatus: null,
                 tags: new[] { ZbHealthTags.Active },
-                args: "admin");
+                args: "admin")
+            // Registered unconditionally, not driver-gated. AddOtOpcUaHealth takes no role argument
+            // and runs on every node; the check itself resolves ISyncStatus optionally and reports
+            // Healthy when LocalDb is absent (admin-only graphs) or replication is default-OFF, so a
+            // plain node is never degraded by it. A factory registration keeps this no-arg signature
+            // while still reading ISyncStatus + options + the sync port from the container.
+            .Add(new HealthCheckRegistration(
+                "localdb-replication",
+                sp => new LocalDbReplicationHealthCheck(
+                    sp.GetService<ISyncStatus>(),
+                    sp.GetRequiredService<IOptions<ReplicationOptions>>(),
+                    LocalDbRegistration.SyncListenPort(sp.GetRequiredService<IConfiguration>())),
+                failureStatus: null,
+                tags: new[] { ZbHealthTags.Active }));
         return services;
     }
 

src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/LocalDbReplicationHealthCheck.cs

@@ -0,0 +1,126 @@
+using Microsoft.Extensions.Diagnostics.HealthChecks;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.LocalDb.Replication;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Health;
+
+/// <summary>
+///     Pure decision function for the LocalDb replication probe, factored out of
+///     <see cref="LocalDbReplicationHealthCheck"/> so the whole matrix is table-testable without
+///     standing up a replication engine.
+/// </summary>
+public static class LocalDbReplicationDecision
+{
+    /// <summary>
+    ///     Maps the resolved replication facts to a health status.
+    /// </summary>
+    /// <param name="peerConfigured">
+    ///     True when this node participates in replication at all — it has a peer address to dial, or
+    ///     a sync listener bound. False means default-OFF, and default-OFF must never degrade a node.
+    /// </param>
+    /// <param name="connected">Whether a sync session is currently running.</param>
+    /// <param name="oplogBacklog">
+    ///     The unacked oplog backlog, or <see langword="null"/> when it could not be read. Null is
+    ///     deliberately not treated as zero: a failed poll is "unknown", not "caught up".
+    /// </param>
+    /// <param name="degradedThreshold">
+    ///     Backlog at or above which a connected pair is judged to be falling behind.
+    /// </param>
+    /// <returns>The status plus a human-readable reason.</returns>
+    public static (HealthStatus Status, string Description) Evaluate(
+        bool peerConfigured, bool connected, long? oplogBacklog, long degradedThreshold)
+    {
+        if (!peerConfigured)
+            return (HealthStatus.Healthy, "LocalDb replication is not configured on this node (default-OFF).");
+
+        if (!connected)
+            return (HealthStatus.Degraded, "LocalDb replication is configured but no sync session is connected.");
+
+        if (oplogBacklog is null)
+            return (HealthStatus.Degraded, "LocalDb replication is connected but its oplog backlog is unknown (the poll failed).");
+
+        if (oplogBacklog.Value >= degradedThreshold)
+            return (HealthStatus.Degraded,
+                $"LocalDb replication is connected but its oplog backlog ({oplogBacklog.Value}) is at or above the degraded threshold ({degradedThreshold}).");
+
+        return (HealthStatus.Healthy, $"LocalDb replication is connected; oplog backlog {oplogBacklog.Value}.");
+    }
+}
+
+/// <summary>
+///     Reports the health of this node's LocalDb replication link. Registered unconditionally with
+///     the shared health pipeline; when LocalDb is not present at all (admin-only graphs) it reports
+///     <see cref="HealthStatus.Healthy"/> rather than degrading — matching the
+///     <c>ActiveNodeHealthCheck</c> precedent of "not applicable ⇒ Healthy".
+/// </summary>
+/// <remarks>
+///     A node running LocalDb with replication switched off (no peer, no listener) is also Healthy:
+///     that is the default posture for most of the fleet, and it must not look degraded. Only a node
+///     that is <i>meant</i> to be replicating and is not — or is connected but cannot confirm it is
+///     draining — is surfaced as Degraded. It never reports Unhealthy: a replication problem does not
+///     stop the node serving its address space, so it must not fail a readiness gate.
+/// </remarks>
+public sealed class LocalDbReplicationHealthCheck : IHealthCheck
+{
+    /// <summary>
+    ///     Backlog at or above which a connected pair is reported Degraded. Well below the library's
+    ///     <c>MaxOplogRows</c> default (1,000,000, where a snapshot resync is forced), so the probe
+    ///     flags a pair that is falling behind long before the engine itself intervenes.
+    /// </summary>
+    public const long DefaultBacklogDegradedThreshold = 100_000;
+
+    private readonly ISyncStatus? _syncStatus;
+    private readonly ReplicationOptions _options;
+    private readonly int _syncListenPort;
+    private readonly long _degradedThreshold;
+
+    /// <summary>Creates the health check.</summary>
+    /// <param name="syncStatus">
+    ///     The replication status singleton, or <see langword="null"/> when the replication engine is
+    ///     not registered (admin-only nodes) — in which case the check is a Healthy no-op.
+    /// </param>
+    /// <param name="options">The bound replication options (peer address, etc.).</param>
+    /// <param name="syncListenPort">
+    ///     The configured sync listener port; a value greater than zero counts as "replication
+    ///     configured" even when this node is the passive side with no peer address.
+    /// </param>
+    /// <param name="degradedThreshold">Backlog degraded threshold; defaults to <see cref="DefaultBacklogDegradedThreshold"/>.</param>
+    public LocalDbReplicationHealthCheck(
+        ISyncStatus? syncStatus,
+        IOptions<ReplicationOptions> options,
+        int syncListenPort,
+        long degradedThreshold = DefaultBacklogDegradedThreshold)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+
+        _syncStatus = syncStatus;
+        _options = options.Value;
+        _syncListenPort = syncListenPort;
+        _degradedThreshold = degradedThreshold;
+    }
+
+    /// <inheritdoc />
+    public Task<HealthCheckResult> CheckHealthAsync(
+        HealthCheckContext context, CancellationToken cancellationToken = default)
+    {
+        // No replication engine registered at all → not applicable → Healthy. Admin-only nodes and
+        // any graph that did not call AddOtOpcUaLocalDb land here.
+        if (_syncStatus is null)
+            return Task.FromResult(HealthCheckResult.Healthy(
+                "LocalDb replication is not present on this node."));
+
+        var peerConfigured =
+            !string.IsNullOrWhiteSpace(_options.PeerAddress) || _syncListenPort > 0;
+
+        var (status, description) = LocalDbReplicationDecision.Evaluate(
+            peerConfigured, _syncStatus.Connected, _syncStatus.OplogBacklog, _degradedThreshold);
+
+        var result = status switch
+        {
+            HealthStatus.Healthy => HealthCheckResult.Healthy(description),
+            _ => HealthCheckResult.Degraded(description),
+        };
+
+        return Task.FromResult(result);
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Observability/ObservabilityExtensions.cs

@@ -1,4 +1,5 @@
 using Microsoft.Extensions.Configuration;
+using ZB.MOM.WW.LocalDb.Replication;
 using ZB.MOM.WW.OtOpcUa.Commons.Observability;
 using ZB.MOM.WW.Telemetry;
 
@@ -27,7 +28,11 @@ public static class ObservabilityExtensions
         return services.AddZbTelemetry(o =>
         {
             o.ServiceName = "otopcua";
-            o.Meters = [OtOpcUaTelemetry.MeterName];
+            // o.Meters is a STRICT allowlist — ZbTelemetry adds only the named meters, with no
+            // wildcard. The LocalDb replication engine publishes under its own meter name, so
+            // without this entry its localdb.sync.* / localdb.oplog.depth series are silently absent
+            // from /metrics on driver nodes (the exact omission a ScadaBridge live gate caught).
+            o.Meters = [OtOpcUaTelemetry.MeterName, LocalDbMetrics.MeterName];
             o.ActivitySources = [OtOpcUaTelemetry.ActivitySourceName];
             if (Enum.TryParse<ZbExporter>(configuration["OtOpcUa:Telemetry:Exporter"], ignoreCase: true, out var exporter))
                 o.Exporter = exporter;

src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs

@@ -1,7 +1,9 @@
 using Akka.Actor;
 using Akka.Hosting;
+using Microsoft.AspNetCore.Server.Kestrel.Core;
 using Microsoft.Extensions.DependencyInjection.Extensions;
 using Serilog;
+using ZB.MOM.WW.LocalDb.Replication;
 using Serilog.Events;
 using ZB.MOM.WW.OtOpcUa.AdminUI;
 using ZB.MOM.WW.OtOpcUa.AdminUI.Clients;
@@ -11,6 +13,7 @@ using ZB.MOM.WW.OtOpcUa.AdminUI.ScriptAnalysis;
 using ZB.MOM.WW.OtOpcUa.Cluster;
 using ZB.MOM.WW.OtOpcUa.Configuration;
 using ZB.MOM.WW.OtOpcUa.ControlPlane;
+using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
 using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
 using ZB.MOM.WW.OtOpcUa.Commons.Engines;
 using ZB.MOM.WW.OtOpcUa.Core.Scripting;
@@ -134,7 +137,7 @@ if (hasDriver)
 
     // Config-gated durable alarm-historian sink. When the AlarmHistorian section is enabled this
     // overrides the NullAlarmHistorianSink default from AddOtOpcUaRuntime (last registration wins)
-    // with a SqliteStoreAndForwardSink draining to the gateway SendEvent writer. The alarm-write path
+    // with a LocalDbStoreAndForwardSink draining to the gateway SendEvent writer. The alarm-write path
     // targets the SAME single gateway as the read path, so its connection (endpoint/key/TLS) is sourced
     // from the ServerHistorian section. AlarmHistorianOptions supplies only the Enabled gate + the
     // SQLite store-and-forward knobs (consumed inside AddAlarmHistorian) — it carries no connection fields.
@@ -174,6 +177,20 @@ if (hasDriver)
         builder.Configuration,
         (_, sp) => GatewayHistorian.CreateAlarmWriter(serverHistorianOptions, sp));
 
+    // Node-local LocalDb: caches the deployed-configuration artifact so this node can boot from
+    // its last-known-good config when central SQL is unreachable, and optionally replicates that
+    // cache to its redundant pair peer. Driver-role only — admin-only nodes have nothing to cache,
+    // and registering here would impose LocalDb:Path-required-or-no-boot on them for nothing.
+    // Storage ships unconditionally; replication stays inert until LocalDb:Replication:PeerAddress
+    // (initiator) / LocalDb:SyncListenPort (listener) are set. See LocalDbRegistration.
+    builder.Services.AddOtOpcUaLocalDb(builder.Configuration);
+
+    // gRPC server plumbing for the passive sync endpoint mapped below. Registered only under
+    // hasDriver so admin-only nodes expose no sync surface at all. The interceptor is the ONLY
+    // inbound auth on that endpoint — the replication library's LocalDbSyncService verifies
+    // nothing — and it fail-closes when no ApiKey is configured.
+    builder.Services.AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>());
+
     // Config-gated server-side HistoryRead backend. When the ServerHistorian section is enabled this
     // overrides the NullHistorianDataSource default from AddOtOpcUaRuntime (last registration wins) with
     // a read-only HistorianGateway-backed data source the node manager's HistoryRead overrides
@@ -356,6 +373,13 @@ if (hasAdmin)
     builder.Services.AddCascadingAuthenticationState();
     builder.Services.AddSignalR();
     builder.Services.AddOtOpcUaAdminClients();
+    // Manual failover backs the Trigger-failover control on the cluster redundancy page. Admin-only:
+    // it is driven from the UI, and the node it acts on is chosen from live cluster state, so any
+    // admin node can drive it regardless of whether that node is the Primary. The ActorSystem comes
+    // through the Func<> accessor below (never resolved at construction — this runs before Akka's
+    // hosted service has built the system).
+    builder.Services.TryAddSingleton<Func<ActorSystem>>(sp => () => sp.GetRequiredService<ActorSystem>());
+    builder.Services.AddSingleton<IManualFailoverService, ManualFailoverService>();
 }
 
 // Registered unconditionally: driver-role nodes resolve Layer-B DriverConfig secrets and have no auth/DP/AdminUI.
@@ -365,6 +389,92 @@ builder.Services.AddOtOpcUaSecrets(builder.Configuration);
 builder.Services.AddOtOpcUaHealth();
 builder.Services.AddOtOpcUaObservability(builder.Configuration);
 
+// ---------------------------------------------------------------------------------------------
+// LocalDb sync listener (default-OFF).
+//
+// DANGER: any explicit Kestrel Listen* call makes Kestrel IGNORE ASPNETCORE_URLS/urls ENTIRELY
+// (it logs "Overriding address(es)"). The host has no ConfigureKestrel today and binds solely via
+// that configuration, so adding a listener naively would silently unbind the AdminUI + deploy API
+// behind Traefik. Everything the host was already asked to serve is therefore re-bound explicitly
+// in the same block.
+//
+// The listener is HTTP/2-ONLY on purpose: the sync client speaks prior-knowledge h2c, which a
+// cleartext Http1AndHttp2 endpoint cannot negotiate (there is no ALPN without TLS). Hence a
+// dedicated port rather than multiplexing onto the main one.
+//
+// When LocalDb:SyncListenPort is 0 (the default) none of this runs and URL binding is untouched.
+// ---------------------------------------------------------------------------------------------
+var syncListenPort = LocalDbRegistration.SyncListenPort(builder.Configuration);
+if (hasDriver && syncListenPort > 0)
+{
+    // Mirror Kestrel's own source precedence: URLS wins; else the HTTP_PORTS/HTTPS_PORTS bare-port
+    // vars; else Kestrel's localhost:5000 default. Missing the HTTP_PORTS leg is not academic — the
+    // aspnet:8.0+ base images set ASPNETCORE_HTTP_PORTS=8080 as the CONTAINER default in place of
+    // ASPNETCORE_URLS, so a driver node that never sets URLS is really serving :8080 on all
+    // interfaces. Falling straight through to localhost:5000 would silently move its health/metrics
+    // surface to loopback (found on the docker-dev live gate).
+    var configuredUrls = builder.Configuration["urls"]
+                      ?? builder.Configuration["ASPNETCORE_URLS"];
+    var existingBindings = KestrelHttpBinding.Parse(configuredUrls);
+
+    if (existingBindings.Count == 0)
+    {
+        var httpPorts = builder.Configuration["ASPNETCORE_HTTP_PORTS"] ?? builder.Configuration["HTTP_PORTS"];
+        var httpsPorts = builder.Configuration["ASPNETCORE_HTTPS_PORTS"] ?? builder.Configuration["HTTPS_PORTS"];
+        existingBindings =
+        [
+            .. KestrelHttpBinding.ParsePorts(httpPorts, isSecure: false),
+            .. KestrelHttpBinding.ParsePorts(httpsPorts, isSecure: true),
+        ];
+        if (existingBindings.Count > 0)
+            Log.Information(
+                "LocalDb sync listener enabled; re-binding the HTTP_PORTS surface ({HttpPorts}/{HttpsPorts}) " +
+                "alongside the sync port.", httpPorts, httpsPorts);
+    }
+
+    // A driver-only Windows-service node gets NO URLS and NO *_PORTS (Install-Services.ps1 sets URLS
+    // only under $hasAdmin), so "nothing configured" is a real, supported state — not an error.
+    // Kestrel's own default is localhost:5000; re-state it explicitly, because taking over
+    // configuration means taking over the default too. Health probes hit this port.
+    if (existingBindings.Count == 0)
+    {
+        existingBindings = [new KestrelHttpBinding("localhost", 5000, IsSecure: false)];
+        Log.Information(
+            "LocalDb sync listener enabled with no URLs configured; re-binding Kestrel's default " +
+            "http://localhost:5000 alongside the sync port.");
+    }
+
+    // Re-binding an https endpoint would need its certificate configuration replayed too, which
+    // this host does not model (TLS is terminated at Traefik). Rather than silently serve it
+    // without TLS — or drop it — refuse to take over Kestrel at all and leave the existing
+    // surface exactly as it was. The operator gets a loud, actionable error instead of an
+    // AdminUI that stopped answering.
+    if (existingBindings.Any(b => b.IsSecure))
+    {
+        Log.Error(
+            "LocalDb:SyncListenPort is set to {Port} but this host serves HTTPS endpoint(s) ({Urls}). " +
+            "Binding the sync listener requires re-binding every existing endpoint explicitly, and the " +
+            "HTTPS certificate configuration cannot be replayed safely. The sync listener is DISABLED; " +
+            "terminate TLS upstream (as the docker-dev rig does) or leave replication off on this node.",
+            syncListenPort, configuredUrls);
+        syncListenPort = 0;
+    }
+    else
+    {
+        builder.WebHost.ConfigureKestrel(kestrel =>
+        {
+            foreach (var binding in existingBindings)
+                binding.Apply(kestrel);
+
+            kestrel.ListenAnyIP(syncListenPort, o => o.Protocols = HttpProtocols.Http2);
+        });
+
+        Log.Information(
+            "LocalDb sync listener bound on :{SyncPort} (h2c); re-bound existing endpoint(s) {Urls}.",
+            syncListenPort, configuredUrls ?? "http://localhost:5000 (Kestrel default)");
+    }
+}
+
 var app = builder.Build();
 
 // AddZbSerilog registers Serilog as the MEL logging provider but does NOT assign the static
@@ -395,6 +505,15 @@ if (hasAdmin)
     app.MapOtOpcUaDeployApi(app.Configuration);
 }
 
+// Passive LocalDb sync endpoint. Gated on the same port check that bound the listener, so a
+// default-OFF or admin-only graph carries no sync surface whatsoever. Mapping it would be safe
+// even unauthenticated — LocalDbSyncAuthInterceptor fail-closes with no ApiKey configured — but
+// not mapping it at all is the stronger default.
+if (hasDriver && syncListenPort > 0)
+{
+    app.MapZbLocalDbSync();
+}
+
 app.MapOtOpcUaHealth();
 app.MapOtOpcUaMetrics();
 

src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj

@@ -33,6 +33,13 @@
         <PackageReference Include="ZB.MOM.WW.Telemetry" />
         <PackageReference Include="ZB.MOM.WW.Telemetry.Serilog" />
         <PackageReference Include="ZB.MOM.WW.Configuration" />
+        <!-- LocalDb: node-local SQLite cache (AddZbLocalDb) plus the replication sync engine and
+             its passive gRPC endpoint (AddZbLocalDbReplication / MapZbLocalDbSync). Grpc.AspNetCore
+             is required because the sync endpoint is served by this host's Kestrel, not dialled by
+             it — Grpc.Net.Client (already pinned) only covers the initiator side. -->
+        <PackageReference Include="Grpc.AspNetCore" />
+        <PackageReference Include="ZB.MOM.WW.LocalDb" />
+        <PackageReference Include="ZB.MOM.WW.LocalDb.Replication" />
         <PackageReference Include="ZB.MOM.WW.Secrets" />
         <PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" />
         <PackageReference Include="ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" />

src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json

@@ -52,13 +52,22 @@
     "MaxBackoffSeconds": 30
   },
   "AlarmHistorian": {
-    "_comment": "Durable SQLite store-and-forward alarm sink. Drains alarm events to the ServerHistorian gateway's SendEvent path; the downstream connection (endpoint/key/TLS) is sourced from the ServerHistorian section.",
+    "_comment": "Durable store-and-forward alarm sink. Buffers into the node's consolidated LocalDb (see the LocalDb section) so the queue replicates to the redundant pair peer, and drains alarm events to the ServerHistorian gateway's SendEvent path from whichever node holds the Primary role. The downstream connection (endpoint/key/TLS) is sourced from the ServerHistorian section.",
     "Enabled": false,
-    "DatabasePath": "alarm-historian.db",
     "DrainIntervalSeconds": 5,
     "Capacity": 1000000,
     "DeadLetterRetentionDays": 30
   },
+  "LocalDb": {
+    "_comment": "Node-local SQLite store caching the deployed-configuration artifact, so a driver node can boot from its last-known-good config when central SQL is unreachable. Registered on driver-role nodes only (see LocalDbRegistration). Storage is unconditional; replication to the redundant pair peer is default-OFF.",
+    "Path": "./data/otopcua-localdb.db",
+    "_SyncListenPortComment": "Port for the dedicated cleartext-HTTP/2 (h2c) listener serving the passive sync endpoint. 0 = no listener bound and the endpoint is not mapped. Must be a port distinct from the main HTTP listener: prior-knowledge h2c cannot share a cleartext Http1AndHttp2 port.",
+    "SyncListenPort": 0,
+    "Replication": {
+      "_comment": "Empty = passive/off. Set PeerAddress on exactly ONE node of the pair (the stream is bidirectional, so the other side still pushes its own deltas back). ApiKey must be BYTE-IDENTICAL on both nodes — the host interceptor fail-closes, so a typo silently stops convergence rather than degrading to unauthenticated. Supply it via ${secret:...} in production, never committed.",
+      "_MaxBatchSizeComment": "Row count, NOT bytes, against gRPC's 4 MB default message cap. Artifact chunk rows are ~171 KB base64, so keep this at 16 on the pair (16 x 171 KB ~= 2.7 MB worst case)."
+    }
+  },
   "Deployment": {
     "_comment": "R2-11 (05/CONV-2): deploy-gate TagConfig strictness. Warn (default) = non-blocking warnings logged + appended to the deployment result message; Error = a config with a typo'd enum or unparseable TagConfig is rejected at the draft gate. Running servers are untouched; the gate only sees re-deploys.",
     "TagConfigValidationMode": "Warn"

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/DeploymentCacheSchema.cs

@@ -0,0 +1,70 @@
+using Microsoft.Data.Sqlite;
+
+namespace ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
+
+/// <summary>
+///     DDL for the node-local deployment-artifact cache: the chunked artifact table plus the
+///     per-cluster current-deployment pointer.
+/// </summary>
+/// <remarks>
+///     <para>
+///         Deliberately depends on nothing but <see cref="SqliteConnection"/> so it can be applied
+///         to any connection — the host's <c>LocalDbSetup.OnReady</c> in production, and a bare
+///         connection in a test — without dragging the DI graph along.
+///     </para>
+///     <para>
+///         <b>Why the artifact is chunked base64 TEXT and not one BLOB row.</b> Two independent
+///         constraints force it. <c>RegisterReplicated</c> rejects BLOB columns outright; and the
+///         replication engine batches by <i>row count</i> against gRPC's 4 MB default message cap,
+///         so a single multi-megabyte artifact row would simply never be deliverable — at any
+///         batch size. A 128 KiB raw chunk is ≈ 171 KB once base64-encoded, which keeps a
+///         worst-case batch comfortably inside the cap.
+///     </para>
+///     <para>
+///         <b>Why no autoincrement PKs.</b> Convergence is last-writer-wins keyed on the primary
+///         key. Two nodes independently allocating rowid 7 for different rows would silently
+///         overwrite one another. Every key here is TEXT or a composite of caller-supplied values.
+///     </para>
+/// </remarks>
+public static class DeploymentCacheSchema
+{
+    /// <summary>Table holding the artifact bytes, split into base64 chunks.</summary>
+    public const string ArtifactsTable = "deployment_artifacts";
+
+    /// <summary>Table holding one current-deployment pointer per cluster.</summary>
+    public const string PointerTable = "deployment_pointer";
+
+    /// <summary>
+    ///     Creates both cache tables if they do not already exist. Idempotent.
+    /// </summary>
+    /// <param name="connection">
+    ///     An already-open connection. <c>ILocalDb.CreateConnection()</c> hands out open,
+    ///     pragma-configured connections — do not call <c>Open()</c> on one.
+    /// </param>
+    public static void Apply(SqliteConnection connection)
+    {
+        ArgumentNullException.ThrowIfNull(connection);
+
+        using var cmd = connection.CreateCommand();
+        cmd.CommandText = """
+            CREATE TABLE IF NOT EXISTS deployment_artifacts (
+                deployment_id   TEXT    NOT NULL,
+                chunk_index     INTEGER NOT NULL,
+                cluster_id      TEXT    NOT NULL,
+                revision_hash   TEXT    NOT NULL,
+                chunk_count     INTEGER NOT NULL,
+                chunk_base64    TEXT    NOT NULL,
+                cached_at_utc   TEXT    NOT NULL,
+                PRIMARY KEY (deployment_id, chunk_index)
+            );
+            CREATE TABLE IF NOT EXISTS deployment_pointer (
+                cluster_id        TEXT NOT NULL PRIMARY KEY,
+                deployment_id     TEXT NOT NULL,
+                revision_hash     TEXT NOT NULL,
+                artifact_sha256   TEXT NOT NULL,
+                applied_at_utc    TEXT NOT NULL
+            );
+            """;
+        cmd.ExecuteNonQuery();
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactCache.cs

@@ -0,0 +1,68 @@
+namespace ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
+
+/// <summary>
+///     Node-local durable cache of the deployment artifact this node last applied.
+/// </summary>
+/// <remarks>
+///     <para>
+///         Exists so a driver node can boot into its last-known-good address space when the central
+///         configuration database is unreachable. Without it, a control-plane outage that outlives a
+///         node restart leaves that node with no address space at all — the plant loses visibility
+///         for a reason that has nothing to do with the plant.
+///     </para>
+/// </remarks>
+public interface IDeploymentArtifactCache
+{
+    /// <summary>
+    ///     Persists <paramref name="artifact"/> as the cluster's current deployment.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         Idempotent per <paramref name="deploymentId"/>: re-storing the same deployment
+    ///         replaces its chunks rather than accumulating orphans. Retention is bounded to the two
+    ///         newest deployments per cluster, so a long-lived node cannot grow its local database
+    ///         without limit.
+    ///     </para>
+    /// </remarks>
+    Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
+                    byte[] artifact, CancellationToken ct = default);
+
+    /// <summary>
+    ///     Reads the cached artifact for <paramref name="clusterId"/>, or <see langword="null"/>
+    ///     when nothing is cached or the cached bytes fail their integrity check.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         A failed integrity check is deliberately indistinguishable from a miss. Booting a
+    ///         plant from a truncated address space is strictly worse than booting from none: the
+    ///         missing half looks like a deliberate configuration rather than an error.
+    ///     </para>
+    /// </remarks>
+    Task<CachedDeploymentArtifact?> GetCurrentAsync(string clusterId, CancellationToken ct = default);
+
+    /// <summary>
+    ///     Reads the single cached pointer without knowing the cluster id.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         The boot seam needs this. A node's cluster id is only derivable from an artifact it
+    ///         has already loaded, or from the central database — which is precisely what is
+    ///         unreachable in the scenario this cache exists to survive. So the cold path reads the
+    ///         newest pointer unkeyed.
+    ///     </para>
+    ///     <para>
+    ///         More than one pointer row means the node was re-homed between clusters. The newest
+    ///         wins, but the choice is logged as a warning naming both clusters — a node silently
+    ///         booting a neighbouring cluster's configuration is exactly the failure that must never
+    ///         be quiet.
+    ///     </para>
+    /// </remarks>
+    Task<CachedDeploymentArtifact?> GetCurrentUnkeyedAsync(CancellationToken ct = default);
+}
+
+/// <summary>
+///     A deployment artifact recovered from the node-local cache, with the metadata needed to decide
+///     whether it is still current once the control plane is reachable again.
+/// </summary>
+public sealed record CachedDeploymentArtifact(
+    string DeploymentId, string RevisionHash, byte[] Artifact, DateTimeOffset AppliedAtUtc);

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/LocalDbDeploymentArtifactCache.cs

@@ -0,0 +1,400 @@
+using System.Globalization;
+using System.Security.Cryptography;
+using Microsoft.Extensions.Logging;
+using ZB.MOM.WW.LocalDb;
+
+namespace ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
+
+/// <summary>
+///     <see cref="IDeploymentArtifactCache"/> backed by the replicated LocalDb deployment-cache
+///     tables.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>Why the artifact is stored as base64 chunks.</b> See
+///         <see cref="DeploymentCacheSchema"/> — replication rejects BLOB columns and batches by row
+///         count against gRPC's message cap, so a whole-artifact row could never be delivered.
+///         Everything in this class that looks like ceremony (chunk indices, a chunk count, a
+///         separate SHA over the raw bytes) exists to make that split safely reversible.
+///     </para>
+/// </remarks>
+public sealed class LocalDbDeploymentArtifactCache : IDeploymentArtifactCache
+{
+    /// <summary>
+    ///     Raw bytes per chunk, before base64 expansion.
+    /// </summary>
+    /// <remarks>
+    ///     128 KiB raw is ≈ 171 KB encoded, which keeps a worst-case replication batch inside gRPC's
+    ///     4 MB default cap with room to spare.
+    /// </remarks>
+    private const int ChunkSize = 128 * 1024;
+
+    /// <summary>
+    ///     How many deployments per cluster survive a store.
+    /// </summary>
+    /// <remarks>
+    ///     Two, not one: the previous artifact is what a rollback would need, and keeping it costs
+    ///     one artifact's worth of disk. Three would only add a generation nobody rolls back to.
+    /// </remarks>
+    private const int RetainedDeployments = 2;
+
+    private readonly ILocalDb _db;
+    private readonly ILogger<LocalDbDeploymentArtifactCache> _logger;
+
+    public LocalDbDeploymentArtifactCache(ILocalDb db, ILogger<LocalDbDeploymentArtifactCache> logger)
+    {
+        ArgumentNullException.ThrowIfNull(db);
+        ArgumentNullException.ThrowIfNull(logger);
+
+        _db = db;
+        _logger = logger;
+    }
+
+    /// <inheritdoc/>
+    public async Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
+                                 byte[] artifact, CancellationToken ct = default)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
+        ArgumentException.ThrowIfNullOrWhiteSpace(deploymentId);
+        ArgumentException.ThrowIfNullOrWhiteSpace(revisionHash);
+        ArgumentNullException.ThrowIfNull(artifact);
+
+        var sha = Convert.ToHexString(SHA256.HashData(artifact));
+        var cachedAtUtc = FormatTimestamp(DateTimeOffset.UtcNow);
+        var chunkCount = (artifact.Length + ChunkSize - 1) / ChunkSize;
+
+        if (await IsAlreadyCachedAsync(clusterId, deploymentId, revisionHash, sha, chunkCount, ct))
+        {
+            _logger.LogDebug(
+                "Deployment {DeploymentId} (revision {RevisionHash}) is already cached intact for cluster {ClusterId}; skipping the re-store.",
+                deploymentId, revisionHash, clusterId);
+            return;
+        }
+
+        // One transaction for the whole store. A pointer that commits without its chunks — or
+        // chunks that commit without the pointer — is a cache that reads back as a corrupt hit
+        // rather than a clean miss, which is the one outcome this type exists to prevent.
+        await using var tx = await _db.BeginTransactionAsync(ct);
+
+        // Delete-then-insert rather than upsert-per-chunk: a re-store with FEWER chunks than last
+        // time would otherwise leave the old tail behind, and those orphans read back as a
+        // chunk-count mismatch forever.
+        await tx.ExecuteAsync(
+            "DELETE FROM deployment_artifacts WHERE deployment_id = @DeploymentId",
+            new { DeploymentId = deploymentId },
+            ct);
+
+        for (var index = 0; index < chunkCount; index++)
+        {
+            var offset = index * ChunkSize;
+            var length = Math.Min(ChunkSize, artifact.Length - offset);
+
+            await tx.ExecuteAsync(
+                """
+                INSERT INTO deployment_artifacts
+                    (deployment_id, chunk_index, cluster_id, revision_hash, chunk_count,
+                     chunk_base64, cached_at_utc)
+                VALUES
+                    (@DeploymentId, @ChunkIndex, @ClusterId, @RevisionHash, @ChunkCount,
+                     @ChunkBase64, @CachedAtUtc)
+                """,
+                new
+                {
+                    DeploymentId = deploymentId,
+                    ChunkIndex = index,
+                    ClusterId = clusterId,
+                    RevisionHash = revisionHash,
+                    ChunkCount = chunkCount,
+                    ChunkBase64 = Convert.ToBase64String(artifact, offset, length),
+                    CachedAtUtc = cachedAtUtc,
+                },
+                ct);
+        }
+
+        await tx.ExecuteAsync(
+            """
+            INSERT INTO deployment_pointer
+                (cluster_id, deployment_id, revision_hash, artifact_sha256, applied_at_utc)
+            VALUES
+                (@ClusterId, @DeploymentId, @RevisionHash, @Sha, @AppliedAtUtc)
+            ON CONFLICT(cluster_id) DO UPDATE SET
+                deployment_id   = excluded.deployment_id,
+                revision_hash   = excluded.revision_hash,
+                artifact_sha256 = excluded.artifact_sha256,
+                applied_at_utc  = excluded.applied_at_utc
+            """,
+            new
+            {
+                ClusterId = clusterId,
+                DeploymentId = deploymentId,
+                RevisionHash = revisionHash,
+                Sha = sha,
+                AppliedAtUtc = cachedAtUtc,
+            },
+            ct);
+
+        // Prune inside the same transaction so the cache is never briefly unbounded.
+        //
+        // The `deployment_id <> @DeploymentId` clause is load-bearing, not belt-and-braces: it makes
+        // "the deployment the pointer names is always present" a structural invariant instead of a
+        // consequence of clock resolution. Without it, three stores landing inside one timestamp
+        // tick fall through to the `deployment_id DESC` tiebreak — and since real deployment ids are
+        // GUIDs, that ordering is effectively random, so the row just written can lose. The pointer
+        // would then name chunks that no longer exist, which reads back as a cache miss on every
+        // subsequent boot until the next deploy overwrites it. Silent and permanent: exactly the
+        // failure this cache exists to prevent.
+        await tx.ExecuteAsync(
+            $"""
+             DELETE FROM deployment_artifacts
+             WHERE cluster_id = @ClusterId
+               AND deployment_id <> @DeploymentId
+               AND deployment_id NOT IN (
+                   SELECT deployment_id
+                   FROM deployment_artifacts
+                   WHERE cluster_id = @ClusterId
+                   GROUP BY deployment_id
+                   ORDER BY MAX(cached_at_utc) DESC, deployment_id DESC
+                   LIMIT {RetainedDeployments}
+               )
+             """,
+            new { ClusterId = clusterId, DeploymentId = deploymentId },
+            ct);
+
+        await tx.CommitAsync(ct);
+    }
+
+    /// <summary>
+    ///     Whether this exact artifact is already cached <b>and readable</b> for the cluster, so
+    ///     storing it again would rewrite every row to its current value.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b>Why this exists.</b> Every write to a replicated table fires a capture trigger and
+    ///         mints an oplog row. Re-caching an artifact the node already holds — which is what a
+    ///         restart's boot-from-cache and every <c>RestoreApplied</c> recovery does — would
+    ///         therefore cost a delete plus an insert per chunk plus a pointer update, all carrying
+    ///         values identical to what is already there. On a replicating pair that is pure churn;
+    ///         on an unpaired (default-OFF) node there is no peer to ack those rows, so they
+    ///         accumulate in the oplog until the library's age cap prunes them. The Phase 1 live gate
+    ///         saw exactly that (check 8).
+    ///     </para>
+    ///     <para>
+    ///         <b>Identity is over the bytes, not the ids.</b> The pointer must match on
+    ///         <c>artifact_sha256</c> as well as the deployment id and revision hash: a re-composed
+    ///         artifact can legitimately carry the same ids with different bytes, and keeping the
+    ///         stale copy would serve a boot-from-cache node the wrong address space while every
+    ///         id-shaped signal claimed it was right.
+    ///     </para>
+    ///     <para>
+    ///         <b>The chunk count is load-bearing.</b> A pointer can name a deployment whose chunks
+    ///         are missing or partial — a half-replicated artifact, or a prune that raced the
+    ///         pointer. Skipping on the pointer alone would make that state permanent, because the
+    ///         re-store that repairs it is the call being skipped.
+    ///     </para>
+    ///     <para>
+    ///         <b>Accepted consequence:</b> a skipped store leaves <c>applied_at_utc</c> at the
+    ///         moment the artifact was <i>first</i> cached rather than last confirmed. Refreshing it
+    ///         would mint the very oplog row this check exists to avoid. Nothing reads it as a
+    ///         liveness signal; retention ranks distinct deployments, which are unaffected.
+    ///     </para>
+    /// </remarks>
+    private async Task<bool> IsAlreadyCachedAsync(
+        string clusterId, string deploymentId, string revisionHash, string sha, int chunkCount,
+        CancellationToken ct)
+    {
+        var matches = await _db.QueryAsync(
+            """
+            SELECT (
+                SELECT COUNT(*)
+                FROM deployment_artifacts
+                WHERE cluster_id = @ClusterId
+                  AND deployment_id = @DeploymentId
+                  AND chunk_count = @ChunkCount
+            )
+            FROM deployment_pointer
+            WHERE cluster_id = @ClusterId
+              AND deployment_id = @DeploymentId
+              AND revision_hash = @RevisionHash
+              AND artifact_sha256 = @Sha
+            """,
+            r => r.GetInt32(0),
+            new
+            {
+                ClusterId = clusterId,
+                DeploymentId = deploymentId,
+                RevisionHash = revisionHash,
+                Sha = sha,
+                ChunkCount = chunkCount,
+            },
+            ct);
+
+        return matches.Count == 1 && matches[0] == chunkCount;
+    }
+
+    /// <inheritdoc/>
+    public async Task<CachedDeploymentArtifact?> GetCurrentAsync(
+        string clusterId, CancellationToken ct = default)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
+
+        var pointers = await _db.QueryAsync(
+            """
+            SELECT deployment_id, revision_hash, artifact_sha256, applied_at_utc
+            FROM deployment_pointer
+            WHERE cluster_id = @ClusterId
+            """,
+            ReadPointer,
+            new { ClusterId = clusterId },
+            ct);
+
+        if (pointers.Count == 0)
+        {
+            _logger.LogDebug("No cached deployment pointer for cluster {ClusterId}.", clusterId);
+            return null;
+        }
+
+        return await ReassembleAsync(clusterId, pointers[0], ct);
+    }
+
+    /// <inheritdoc/>
+    public async Task<CachedDeploymentArtifact?> GetCurrentUnkeyedAsync(CancellationToken ct = default)
+    {
+        // applied_at_utc is round-trip ISO-8601 UTC, so ordering it as TEXT is chronological —
+        // that is the reason the format is pinned rather than left to the current culture.
+        var pointers = await _db.QueryAsync(
+            """
+            SELECT cluster_id, deployment_id, revision_hash, artifact_sha256, applied_at_utc
+            FROM deployment_pointer
+            ORDER BY applied_at_utc DESC
+            """,
+            r => (ClusterId: r.GetString(0), Pointer: new PointerRow(
+                DeploymentId: r.GetString(1),
+                RevisionHash: r.GetString(2),
+                ArtifactSha256: r.GetString(3),
+                AppliedAtUtc: r.GetString(4))),
+            parameters: null,
+            ct);
+
+        if (pointers.Count == 0)
+        {
+            _logger.LogDebug("No cached deployment pointer of any cluster.");
+            return null;
+        }
+
+        if (pointers.Count > 1)
+        {
+            // The node was re-homed between clusters. Booting the newest is the only defensible
+            // choice, but it must never be a silent one — a wrong-cluster address space presents as
+            // a plausible configuration, so the operator needs the cluster names to spot it.
+            _logger.LogWarning(
+                "Local deployment cache holds pointers for {PointerCount} clusters ({ClusterIds}); " +
+                "booting the newest ({ChosenClusterId}). This node appears to have been re-homed — " +
+                "clear the stale cache if that is unexpected.",
+                pointers.Count,
+                string.Join(", ", pointers.Select(p => p.ClusterId)),
+                pointers[0].ClusterId);
+        }
+
+        return await ReassembleAsync(pointers[0].ClusterId, pointers[0].Pointer, ct);
+    }
+
+    /// <summary>
+    ///     Rebuilds the artifact the pointer names, returning <see langword="null"/> unless every
+    ///     integrity check passes.
+    /// </summary>
+    private async Task<CachedDeploymentArtifact?> ReassembleAsync(
+        string clusterId, PointerRow pointer, CancellationToken ct)
+    {
+        var chunks = await _db.QueryAsync(
+            """
+            SELECT chunk_count, chunk_base64
+            FROM deployment_artifacts
+            WHERE deployment_id = @DeploymentId
+            ORDER BY chunk_index
+            """,
+            r => (ChunkCount: r.GetInt32(0), Base64: r.GetString(1)),
+            new { pointer.DeploymentId },
+            ct);
+
+        // The chunk_count carried on every row is what makes a partial replica detectable. Without
+        // it a missing tail chunk is indistinguishable from a shorter artifact.
+        var expectedChunkCount = chunks.Count > 0 ? chunks[0].ChunkCount : 0;
+
+        if (chunks.Count != expectedChunkCount)
+        {
+            _logger.LogWarning(
+                "Cached deployment {DeploymentId} for cluster {ClusterId} is incomplete: found " +
+                "{FoundChunks} of {ExpectedChunks} chunks. Treating as a cache miss.",
+                pointer.DeploymentId, clusterId, chunks.Count, expectedChunkCount);
+            return null;
+        }
+
+        byte[] artifact;
+        try
+        {
+            artifact = Decode(chunks.Select(c => c.Base64));
+        }
+        catch (FormatException ex)
+        {
+            _logger.LogWarning(
+                ex,
+                "Cached deployment {DeploymentId} for cluster {ClusterId} has a chunk that is not " +
+                "valid base64. Treating as a cache miss.",
+                pointer.DeploymentId, clusterId);
+            return null;
+        }
+
+        var actualSha = Convert.ToHexString(SHA256.HashData(artifact));
+        if (!string.Equals(actualSha, pointer.ArtifactSha256, StringComparison.OrdinalIgnoreCase))
+        {
+            _logger.LogWarning(
+                "Cached deployment {DeploymentId} for cluster {ClusterId} failed its SHA-256 check " +
+                "(expected {ExpectedSha}, computed {ActualSha}). Treating as a cache miss.",
+                pointer.DeploymentId, clusterId, pointer.ArtifactSha256, actualSha);
+            return null;
+        }
+
+        if (!DateTimeOffset.TryParse(pointer.AppliedAtUtc, CultureInfo.InvariantCulture,
+                DateTimeStyles.RoundtripKind, out var appliedAtUtc))
+        {
+            _logger.LogWarning(
+                "Cached deployment {DeploymentId} for cluster {ClusterId} has an unparseable " +
+                "applied_at_utc ({AppliedAtUtc}). Treating as a cache miss.",
+                pointer.DeploymentId, clusterId, pointer.AppliedAtUtc);
+            return null;
+        }
+
+        return new CachedDeploymentArtifact(
+            pointer.DeploymentId, pointer.RevisionHash, artifact, appliedAtUtc);
+    }
+
+    private static byte[] Decode(IEnumerable<string> chunksInOrder)
+    {
+        using var buffer = new MemoryStream();
+
+        foreach (var chunk in chunksInOrder)
+        {
+            var bytes = Convert.FromBase64String(chunk);
+            buffer.Write(bytes, 0, bytes.Length);
+        }
+
+        return buffer.ToArray();
+    }
+
+    private static PointerRow ReadPointer(Microsoft.Data.Sqlite.SqliteDataReader reader)
+        => new(
+            DeploymentId: reader.GetString(0),
+            RevisionHash: reader.GetString(1),
+            ArtifactSha256: reader.GetString(2),
+            AppliedAtUtc: reader.GetString(3));
+
+    /// <summary>
+    ///     Round-trip ("O") UTC, so lexicographic ordering of the stored TEXT is chronological
+    ///     ordering — the retention query sorts on it directly.
+    /// </summary>
+    private static string FormatTimestamp(DateTimeOffset value)
+        => value.ToUniversalTime().ToString("O", CultureInfo.InvariantCulture);
+
+    private sealed record PointerRow(
+        string DeploymentId, string RevisionHash, string ArtifactSha256, string AppliedAtUtc);
+}

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs

@@ -23,6 +23,8 @@ using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
 using ZB.MOM.WW.OtOpcUa.Core.Scripting;
 using ZB.MOM.WW.OtOpcUa.Core.VirtualTags;
 using ZB.MOM.WW.OtOpcUa.OpcUaServer;
+using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
+using ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
 using ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms;
 using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
 using CommonsNodeId = ZB.MOM.WW.OtOpcUa.Commons.Types.NodeId;
@@ -57,6 +59,31 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
     /// <summary>Publishing interval handed to each driver's SubscribeBulk pass after an apply.</summary>
     private static readonly TimeSpan SubscriptionPublishingInterval = TimeSpan.FromSeconds(1);
 
+    /// <summary>
+    ///     Cache key used when an artifact carries no cluster scoping (single-cluster or unscoped
+    ///     deployments). A literal rather than an empty string so the row is visibly deliberate when
+    ///     someone reads the table during an incident.
+    /// </summary>
+    private const string SingleClusterCacheKey = "__single";
+
+    /// <summary>
+    ///     Node-local cache of applied deployment artifacts, or null when this node has none
+    ///     (admin-only graphs, and tests that do not exercise it).
+    /// </summary>
+    private readonly IDeploymentArtifactCache? _deploymentArtifactCache;
+
+    /// <summary>
+    ///     True once this node has booted a cached artifact because central SQL was unreachable.
+    /// </summary>
+    /// <remarks>
+    ///     Surfaced on <see cref="NodeDiagnosticsSnapshot"/> because a node running from cache looks
+    ///     completely healthy from the outside — it serves a full address space with live values.
+    ///     The difference is that its configuration is frozen at whatever the cache held, and no
+    ///     deployment can reach it. Without an explicit signal that is invisible until someone
+    ///     wonders why a deploy "succeeded" everywhere but did not take effect here.
+    /// </remarks>
+    private bool _isRunningFromCache;
+
     private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
     private readonly CommonsNodeId _localNode;
     private readonly IActorRef? _coordinatorOverride;
@@ -226,6 +253,17 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
     /// non-cluster ActorRefProvider). See <see cref="DriverMemberCount"/>.</summary>
     private readonly Func<int>? _driverMemberCountProvider;
 
+    /// <summary>Where the Primary-gate decision is published for consumers that live outside a mailbox —
+    /// today, the alarm store-and-forward drain. Null on nodes that do not wire one.</summary>
+    private readonly IRedundancyRoleView? _redundancyRoleView;
+
+    /// <summary>
+    ///     Host of this node's LocalDb replication partner — the only node that holds a copy of this
+    ///     node's alarm queue, and therefore the only node it may ever stand down in favour of.
+    ///     <c>null</c> when this node dials nobody (unpaired, or the listening half of a pair).
+    /// </summary>
+    private readonly string? _replicationPeerHost;
+
     /// <summary>Debounces the S5 "snapshot omitted this node" Warning to once per process — a persistent
     /// identity mismatch (03/S5) would otherwise log on every snapshot.</summary>
     private bool _warnedSnapshotMissingLocalNode;
@@ -339,11 +377,20 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         ScriptRootLogger? scriptRootLogger = null,
         IActorRef? scriptedAlarmHostOverride = null,
         IDriverCapabilityInvokerFactory? invokerFactory = null,
-        Func<int>? driverMemberCountProvider = null) =>
+        Func<int>? driverMemberCountProvider = null,
+        IDeploymentArtifactCache? deploymentArtifactCache = null,
+        IRedundancyRoleView? redundancyRoleView = null,
+        string? replicationPeerHost = null) =>
+        // WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
+        // expression tree. Six IActorRef? parameters and several interface-typed ones mean a
+        // mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
+        // the wrong dependency at runtime. New parameters go LAST in all three places — this
+        // signature, the constructor's, and this call — and nothing else moves.
         Akka.Actor.Props.Create(() => new DriverHostActor(
             dbFactory, localNode, coordinator, driverFactory, localRoles, dependencyMux, opcUaPublishActor,
             healthPublisher, virtualTagEvaluator, historyWriter, virtualTagHostOverride,
-            loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider));
+            loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider,
+            deploymentArtifactCache, redundancyRoleView, replicationPeerHost));
 
     /// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
     /// <param name="dbFactory">Database context factory for configuration database access.</param>
@@ -372,6 +419,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
     /// <param name="driverMemberCountProvider">Test seam (archreview 03/S4): overrides the count of Up
     /// <c>driver</c>-role cluster members the Primary gate reads while the role is unknown. When null the
     /// default reads <c>Cluster.Get(Context.System).State.Members</c> (0 on a non-cluster ActorRefProvider).</param>
+    /// <param name="deploymentArtifactCache">Optional node-local cache of applied deployment artifacts.
+    /// When supplied, each successful apply stores its artifact so the node can boot from its
+    /// last-known-good configuration while central SQL is unreachable. Null on admin-only nodes and in
+    /// tests that do not exercise the cache — caching is then simply skipped.</param>
     public DriverHostActor(
         IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
         CommonsNodeId localNode,
@@ -388,8 +439,14 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         ScriptRootLogger? scriptRootLogger = null,
         IActorRef? scriptedAlarmHostOverride = null,
         IDriverCapabilityInvokerFactory? invokerFactory = null,
-        Func<int>? driverMemberCountProvider = null)
+        Func<int>? driverMemberCountProvider = null,
+        IDeploymentArtifactCache? deploymentArtifactCache = null,
+        IRedundancyRoleView? redundancyRoleView = null,
+        string? replicationPeerHost = null)
     {
+        _deploymentArtifactCache = deploymentArtifactCache;
+        _redundancyRoleView = redundancyRoleView;
+        _replicationPeerHost = replicationPeerHost;
         _dbFactory = dbFactory;
         _localNode = localNode;
         _coordinatorOverride = coordinator;
@@ -541,7 +598,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
                     // space were lost on restart. Re-spawn + re-materialise + re-subscribe from the
                     // applied deployment so a restarted/rebuilt node restores its served state instead
                     // of waiting for a config change (whose identical-config revision would no-op).
-                    RestoreApplied(new DeploymentId(latest.DeploymentId));
+                    RestoreApplied(new DeploymentId(latest.DeploymentId), revision);
                     break;
                 case NodeDeploymentStatus.Applying:
                     _log.Warning("DriverHost {Node}: found orphan Applying row for deployment {Id}; replaying",
@@ -560,10 +617,122 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         catch (Exception ex)
         {
             _log.Warning(ex, "DriverHost {Node}: ConfigDb unreachable on bootstrap; entering Stale", _localNode);
-            Become(Stale);
+
+            // Central is unreachable, so try the node-local cache before giving up. On a hit this
+            // node serves its last-known-good configuration through the outage instead of coming up
+            // with an empty address space. On a miss, behaviour is exactly what it always was.
+            if (!TryBootFromCache())
+                Become(Stale);
         }
     }
 
+    /// <summary>
+    ///     Last-resort boot path: apply the artifact cached by a previous successful deploy (or
+    ///     replicated from this node's pair peer) when central SQL cannot be reached.
+    /// </summary>
+    /// <returns>
+    ///     <see langword="true"/> when a cached artifact was applied and the actor is now Steady;
+    ///     <see langword="false"/> when the caller should fall through to <c>Stale</c>.
+    /// </returns>
+    /// <remarks>
+    ///     <para>
+    ///         <b>The read is unkeyed.</b> The cache is keyed by ClusterId so a pair shares one
+    ///         entry, but ClusterId is only derivable from an artifact you already hold or from the
+    ///         central DB — and at this seam we have neither. So the newest pointer row wins, which
+    ///         is correct in every real topology because a node belongs to one cluster and its peer
+    ///         replicates that same cluster's row. A node re-homed between clusters is the only
+    ///         ambiguous case, and the cache logs a warning naming both.
+    ///     </para>
+    ///     <para>
+    ///         <b>This does not make the node current.</b> <c>_currentRevision</c> is set from the
+    ///         cached artifact, so a subsequent dispatch of that same revision correctly no-ops,
+    ///         while any NEW revision still requires central — the cache holds the past, not the
+    ///         future. Dispatch handling is unchanged.
+    ///     </para>
+    ///     <para>
+    ///         Never throws: a fault here must degrade to Stale, which is exactly where the node
+    ///         would have been without a cache at all.
+    ///     </para>
+    /// </remarks>
+    private bool TryBootFromCache()
+    {
+        if (_deploymentArtifactCache is null)
+            return false;
+
+        try
+        {
+            var cached = _deploymentArtifactCache.GetCurrentUnkeyedAsync().GetAwaiter().GetResult();
+            if (cached is null)
+            {
+                _log.Info(
+                    "DriverHost {Node}: no cached deployment available; entering Stale with no configuration.",
+                    _localNode);
+                return false;
+            }
+
+            var deploymentId = DeploymentId.Parse(cached.DeploymentId);
+            var revision = RevisionHash.Parse(cached.RevisionHash);
+
+            // Steady, not Stale: this node is serving a real configuration. The retry-db timer that
+            // Stale would have started does not run here, so recovery rides on the next dispatch —
+            // matching how a normally-booted node behaves.
+            _currentRevision = revision;
+            Become(Steady);
+
+            ApplyCachedArtifact(deploymentId, cached.Artifact);
+
+            _isRunningFromCache = true;
+            _log.Warning(
+                "DriverHost {Node}: RUNNING FROM CACHE — central ConfigDb is unreachable, so this node " +
+                "booted deployment {Id} (rev {Rev}) cached at {CachedAtUtc:o} from its local database. " +
+                "Configuration changes cannot be applied until the ConfigDb is reachable again.",
+                _localNode, deploymentId, revision, cached.AppliedAtUtc);
+
+            return true;
+        }
+        catch (Exception ex)
+        {
+            _log.Error(ex,
+                "DriverHost {Node}: failed to boot from the local deployment cache; falling back to Stale.",
+                _localNode);
+            return false;
+        }
+    }
+
+    /// <summary>
+    ///     Applies an artifact already in hand — no ConfigDb read, no ACK.
+    /// </summary>
+    /// <remarks>
+    ///     Mirrors <see cref="RestoreApplied"/>, but sources the artifact from the cache rather than
+    ///     re-reading it from a database that is by definition unreachable here. It also skips
+    ///     <c>UpsertNodeDeploymentState</c> and <c>SendAck</c> for the same reason: both write to or
+    ///     depend on central.
+    /// </remarks>
+    private void ApplyCachedArtifact(DeploymentId deploymentId, byte[] blob)
+    {
+        var correlation = CorrelationId.NewId();
+
+        var specs = DeploymentArtifact.ParseDriverInstances(blob, _localNode.Value);
+        var snapshots = _children.ToDictionary(
+            kv => kv.Key,
+            kv => new DriverChildSnapshot(kv.Value.DriverType, kv.Value.LastConfigJson, kv.Value.ResilienceConfig),
+            StringComparer.Ordinal);
+        var plan = DriverSpawnPlanner.Compute(snapshots, specs);
+
+        foreach (var id in plan.ToStop) StopChild(id);
+        foreach (var spec in plan.ToApplyDelta) ApplyChildDelta(spec);
+        foreach (var spec in plan.ToSpawn) SpawnChild(spec);
+
+        // Hand the cached blob to the rebuild so it materialises the client-facing address space from
+        // it directly. Passing only the deploymentId would drive a ConfigDb read — unreachable here by
+        // definition — and the rebuild would no-op, leaving OPC UA clients browsing an empty server
+        // while the drivers below poll happily. (Found on the docker-dev live gate.)
+        _opcUaPublishActor?.Tell(
+            new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RebuildAddressSpace(correlation, deploymentId, blob));
+
+        PushDesiredSubscriptionsFromArtifact(deploymentId, blob);
+    }
+
     private void Steady()
     {
         Receive<DispatchDeployment>(HandleDispatchFromSteady);
@@ -1306,6 +1475,14 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         }
     }
 
+    /// <summary>Host part of a <c>host:port</c> node id.</summary>
+    private static string HostOf(NodeId nodeId)
+    {
+        var text = nodeId.ToString();
+        var colon = text.LastIndexOf(':');
+        return colon < 0 ? text : text[..colon];
+    }
+
     /// <summary>The Primary-gate deny reason tag for the denial meter (<c>secondary|detached|role-unknown</c>).</summary>
     private string PrimaryGateDenyReason() => _localRole switch
     {
@@ -1325,19 +1502,35 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         if (local is not null)
         {
             _localRole = local.Role;
-            return;
         }
-
-        // S5 diagnostic: the snapshot never mentioned this node. On a multi-driver cluster that means the
-        // Primary gate stays default-DENY indefinitely for an unknown role — surface it once so a silent
-        // identity mismatch (e.g. an ApplicationUri/NodeId skew) is diagnosable.
-        if (!_warnedSnapshotMissingLocalNode && DriverMemberCount() > 1)
+        else if (!_warnedSnapshotMissingLocalNode && DriverMemberCount() > 1)
         {
+            // S5 diagnostic: the snapshot never mentioned this node. On a multi-driver cluster that means the
+            // Primary gate stays default-DENY indefinitely for an unknown role — surface it once so a silent
+            // identity mismatch (e.g. an ApplicationUri/NodeId skew) is diagnosable.
             _warnedSnapshotMissingLocalNode = true;
             _log.Warning(
                 "DriverHost {Node}: redundancy snapshot omitted this node (snapshotNodes={SnapshotNodes}); Primary data-plane gate stays default-DENY while role is unknown on a multi-driver cluster",
                 _localNode, string.Join(",", msg.Nodes.Select(n => n.NodeId)));
         }
+
+        // Publish on EVERY snapshot, including ones that leave the cached role untouched, so a role
+        // that changes without this node being named still reaches the drain within one heartbeat.
+        //
+        // NOTE the different policy: the drain gate keys on the role ALONE and opens when it is
+        // unknown, where the data-plane gate above resolves an unknown role by member count and
+        // denies. Publishing ShouldServiceAsPrimary() here suspended the drain on every node of a
+        // multi-driver cluster that had no redundancy configured — see ShouldDrainAlarmHistory.
+        // Defer ONLY to the node that holds a copy of our rows. The redundancy election is
+        // cluster-wide and the queue is pair-local, so "somebody is Primary" is not a licence to stop
+        // draining — that Primary is usually in another pair entirely, and cannot deliver our events.
+        var peerIsPrimary =
+            _replicationPeerHost is not null
+            && msg.Nodes.Any(n => n.Role == RedundancyRole.Primary
+                                  && HostOf(n.NodeId) == _replicationPeerHost);
+
+        _redundancyRoleView?.Publish(
+            PrimaryGatePolicy.ShouldDrainAlarmHistory(_localRole, queueSharingPeerIsPrimary: peerIsPrimary));
     }
 
     private void Stale()
@@ -1393,7 +1586,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
             NodeId: _localNode,
             CurrentRevision: _currentRevision,
             Drivers: drivers,
-            AsOfUtc: DateTime.UtcNow);
+            AsOfUtc: DateTime.UtcNow,
+            RunningFromCache: _isRunningFromCache);
         Sender.Tell(snapshot);
     }
 
@@ -1426,7 +1620,30 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
 
         try
         {
-            ReconcileDrivers(deploymentId);
+            var appliedBlob = ReconcileDrivers(deploymentId);
+
+            // Issue #486 — a null blob means the artifact could not be read (unreachable ConfigDb, or the
+            // empty bytes a missing row yields), so NOTHING was applied. Reporting Applied here used to
+            // advance _currentRevision too, and HandleDispatchFromSteady short-circuits on a revision
+            // match — so the node claimed a configuration it never applied AND could never be healed by
+            // re-dispatching that revision. Failing the apply keeps the revision where it was, which is
+            // what makes the retry land instead of being waved through. The node keeps serving its
+            // last-known-good address space, drivers and subscriptions throughout (issue #485) — this is
+            // about telling the truth, not about tearing anything down.
+            if (appliedBlob is null)
+            {
+                const string reason =
+                    "the deployment artifact could not be read (ConfigDb unreachable, or the row is missing/empty); nothing was applied";
+                UpsertNodeDeploymentState(deploymentId, NodeDeploymentStatus.Failed, reason);
+                SendAck(deploymentId, ApplyAckOutcome.Failed, reason, correlation);
+                OtOpcUaTelemetry.DeploymentApplied.Add(1, new KeyValuePair<string, object?>("outcome", "reject"));
+                span?.SetStatus(ActivityStatusCode.Error, reason);
+                _log.Error(
+                    "DriverHost {Node}: apply of {Id} FAILED — {Reason}. Staying on revision {Rev} and continuing to serve it; re-dispatch once the ConfigDb is readable",
+                    _localNode, deploymentId, reason, _currentRevision);
+                return;
+            }
+
             _currentRevision = revision;
             UpsertNodeDeploymentState(deploymentId, NodeDeploymentStatus.Applied, failureReason: null);
             SendAck(deploymentId, ApplyAckOutcome.Applied, failureReason: null, correlation);
@@ -1437,6 +1654,12 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
             // SubscribeBulk pass: hand each driver its desired tag references so live values flow into
             // the just-rebuilt address space instead of staying BadWaitingForInitialData.
             PushDesiredSubscriptions(deploymentId);
+            CacheAppliedArtifact(deploymentId, revision, appliedBlob);
+            // Reaching here means the artifact came from central, so the node is no longer serving a
+            // cache-sourced configuration. Note this only clears on a real apply: a dispatch of the
+            // revision already booted from cache short-circuits in HandleDispatchFromSteady without
+            // touching the ConfigDb, and the flag correctly stays set.
+            _isRunningFromCache = false;
             OtOpcUaTelemetry.DeploymentApplied.Add(1, new KeyValuePair<string, object?>("outcome", "ack"));
             _log.Info("DriverHost {Node}: applied deployment {Id} (rev {Rev}, children={Count})",
                 _localNode, deploymentId, revision, _children.Count);
@@ -1460,11 +1683,24 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
     /// <summary>
     ///     Read the deployment artifact + reconcile the set of running <see cref="DriverInstanceActor"/>
     ///     children. Spawn missing, ApplyDelta on config change, stop removed/disabled drivers.
-    ///     When the artifact blob is empty (legacy ControlPlane tests, smoke fixtures) or the
-    ///     configured <see cref="IDriverFactory"/> can't materialise any of the requested
-    ///     types, this is effectively a no-op.
+    ///     When the artifact blob is empty (legacy ControlPlane tests, smoke fixtures) the reconcile is
+    ///     SKIPPED outright — see the issue #485 guard below — and when the configured
+    ///     <see cref="IDriverFactory"/> can't materialise any of the requested types this is a no-op.
     /// </summary>
-    private void ReconcileDrivers(DeploymentId deploymentId)
+    /// <returns>
+    ///     The artifact blob that was reconciled, or <see langword="null"/> when it could not be
+    ///     loaded at all.
+    /// </returns>
+    /// <remarks>
+    ///     Returning the blob rather than swallowing it is load-bearing for the artifact cache. This
+    ///     method catches its own DB failures, logs a warning and returns WITHOUT rethrowing — so
+    ///     <see cref="ApplyAndAck"/> proceeds to its success path, ACKs Applied and logs success
+    ///     having spawned zero drivers. A cache write that trusted "the apply succeeded" would then
+    ///     persist an empty artifact as this node's last-known-good configuration, and the node would
+    ///     later boot from it into an empty address space. The caller distinguishes the two cases by
+    ///     this return value.
+    /// </remarks>
+    private byte[]? ReconcileDrivers(DeploymentId deploymentId)
     {
         byte[] blob;
         try
@@ -1479,7 +1715,21 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         {
             _log.Warning(ex, "DriverHost {Node}: failed to load artifact for {Id}; skipping reconcile",
                 _localNode, deploymentId);
-            return;
+            return null;
+        }
+
+        // Issue #485 — no bytes is no answer, not "a configuration with no drivers". Parsing zero bytes
+        // yields zero specs, and DriverSpawnPlanner then plans EVERY running child for StopChild: a missing
+        // deployment row (or a half-written one) silently takes this node's entire field I/O down while the
+        // apply still ACKs Applied. Nothing legitimate produces a zero-length blob — deleting the last
+        // driver still deploys a JSON document with empty arrays — so treat it exactly like the load
+        // failure above and leave the running children alone.
+        if (blob.Length == 0)
+        {
+            _log.Warning(
+                "DriverHost {Node}: artifact for {Id} read back empty; skipping reconcile and keeping the {Count} running driver(s)",
+                _localNode, deploymentId, _children.Count);
+            return null;
         }
 
         var specs = DeploymentArtifact.ParseDriverInstances(blob, _localNode.Value);
@@ -1500,6 +1750,74 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
         // sequenced AFTER ReinitializeAsync rebuilds DependencyRefs — a synchronous re-register HERE would
         // re-read the STALE ref set (ApplyDelta runs asynchronously on the child), so it is deliberately NOT
         // done inline.
+
+        return blob;
+    }
+
+    /// <summary>
+    ///     Store a successfully applied artifact in the node-local cache, so this node can boot from
+    ///     it while central SQL is unreachable.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b>Never throws.</b> By the time this runs the deployment is already recorded Applied
+    ///         in central SQL and an Applied ACK has been sent to the coordinator. An exception
+    ///         escaping here would unwind into <see cref="ApplyAndAck"/>'s catch and send a second,
+    ///         contradictory Failed ACK for a deployment the fleet already believes is live.
+    ///     </para>
+    ///     <para>
+    ///         <b>An empty or unloadable blob is skipped, not cached.</b> See
+    ///         <see cref="ReconcileDrivers"/>: a DB failure there degrades silently, so "the apply
+    ///         succeeded" does not imply "a real configuration was applied". Caching an empty
+    ///         artifact would make it this node's last-known-good and boot it into an empty address
+    ///         space during the next outage — strictly worse than having no cache at all.
+    ///     </para>
+    ///     <para>
+    ///         Runs synchronously on the actor thread. That matches every other DB call on this path
+    ///         (all of which already block), and the alternative — piping the result back as a
+    ///         self-message — would need a handler registered in Steady, Applying AND Stale or it
+    ///         dead-letters after the <c>Become(Steady)</c> in the enclosing finally.
+    ///     </para>
+    /// </remarks>
+    private void CacheAppliedArtifact(DeploymentId deploymentId, RevisionHash revision, byte[]? blob)
+    {
+        if (_deploymentArtifactCache is null)
+            return;
+
+        if (blob is null || blob.Length == 0)
+        {
+            _log.Warning(
+                "DriverHost {Node}: not caching deployment {Id} — its artifact was empty or could not " +
+                "be loaded, so it is not a usable last-known-good configuration.",
+                _localNode, deploymentId);
+            return;
+        }
+
+        try
+        {
+            // The node's ClusterId is carried inside the artifact (ClusterNode rows), not in config
+            // or on IClusterRoleInfo. A single-cluster or unscoped artifact has none, so it shares
+            // one sentinel key — the pair still converges on a single pointer row either way.
+            var scope = DeploymentArtifact.ResolveClusterScope(blob, _localNode.Value);
+            var clusterId = scope.Mode == ClusterFilterMode.ScopeTo && !string.IsNullOrWhiteSpace(scope.ClusterId)
+                ? scope.ClusterId
+                : SingleClusterCacheKey;
+
+            _deploymentArtifactCache
+                .StoreAsync(clusterId, deploymentId.ToString(), revision.Value, blob)
+                .GetAwaiter()
+                .GetResult();
+
+            _log.Debug("DriverHost {Node}: cached deployment {Id} (rev {Rev}, cluster {ClusterId}, {Bytes} bytes)",
+                _localNode, deploymentId, revision, clusterId, blob.Length);
+        }
+        catch (Exception ex)
+        {
+            _log.Error(ex,
+                "DriverHost {Node}: failed to cache applied deployment {Id}. The apply itself succeeded " +
+                "and is unaffected; this node simply has no local fallback for that revision.",
+                _localNode, deploymentId);
+        }
     }
 
     /// <summary>
@@ -1511,14 +1829,24 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
     ///     drivers, rebuilds the address space from the applied artifact, and re-pushes SubscribeBulk.
     ///     No re-ack: the deployment is already Applied.
     /// </summary>
-    private void RestoreApplied(DeploymentId deploymentId)
+    private void RestoreApplied(DeploymentId deploymentId, RevisionHash? revision)
     {
         var correlation = CorrelationId.NewId();
         try
         {
-            ReconcileDrivers(deploymentId);
-            _opcUaPublishActor?.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RebuildAddressSpace(correlation, deploymentId));
+            var appliedBlob = ReconcileDrivers(deploymentId);
+            _opcUaPublishActor?.Tell(new ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RebuildAddressSpace(correlation, deploymentId, appliedBlob));
             PushDesiredSubscriptions(deploymentId);
+            // Populate the node-local cache from the artifact this restored node is now serving. The
+            // cache invariant is "holds what the node currently serves"; without this only a FRESH
+            // apply (ApplyAndAck) ever wrote it, so a node whose cache was lost — a wiped/fresh volume,
+            // a disk failure — would recover its served state here yet stay cache-less, unable to
+            // boot-from-cache on the NEXT central outage until some future new deploy happened to land.
+            // Replication does not heal that gap either: a peer's already-acked rows are pruned from its
+            // oplog, so a fully-wiped node is never back-filled. Re-caching on restore is what closes
+            // it. (Found on the docker-dev live gate.)
+            if (revision is { } rev)
+                CacheAppliedArtifact(deploymentId, rev, appliedBlob);
             _log.Info("DriverHost {Node}: restored served state for applied deployment {Id} on bootstrap", _localNode, deploymentId);
         }
         catch (Exception ex)
@@ -1552,6 +1880,31 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
             return;
         }
 
+        PushDesiredSubscriptionsFromArtifact(deploymentId, blob);
+    }
+
+    /// <summary>
+    ///     The artifact-processing half of <see cref="PushDesiredSubscriptions"/>, split out so the
+    ///     boot-from-cache path can drive it with bytes already in hand.
+    /// </summary>
+    /// <remarks>
+    ///     Separated because the cache path runs precisely when the ConfigDb read above cannot
+    ///     succeed — re-reading the artifact there would fail by definition.
+    /// </remarks>
+    private void PushDesiredSubscriptionsFromArtifact(DeploymentId deploymentId, byte[] blob)
+    {
+        // Issue #485 — the same "no bytes is no answer" rule as ReconcileDrivers. An empty artifact parses
+        // to a composition with no raw tags, which hands every child an EMPTY desired set — and an empty set
+        // drops the live subscription handle rather than re-subscribing. Reachable independently of the
+        // reconcile guard: this method does its OWN ConfigDb read, so the row can go missing between the two.
+        if (blob.Length == 0)
+        {
+            _log.Warning(
+                "DriverHost {Node}: artifact for {Id} read back empty; skipping SubscribeBulk and keeping the live subscriptions",
+                _localNode, deploymentId);
+            return;
+        }
+
         AddressSpaceComposition composition;
         try
         {

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/PrimaryGatePolicy.cs

@@ -25,4 +25,50 @@ public static class PrimaryGatePolicy
             RedundancyRole.Secondary or RedundancyRole.Detached => false,
             _ => driverMemberCount <= 1,   // unknown: allow only when no driver peer exists
         };
+
+    /// <summary>
+    ///     Decide whether this node should drain the replicated alarm store-and-forward queue.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b>Deliberately not <see cref="ShouldServiceAsPrimary"/>.</b> That gate protects a shared
+    ///         field device, where two nodes acting at once is dangerous and irreversible, so an unknown
+    ///         role with a peer present must deny. This gate protects an append-only historian, and the
+    ///         delivery contract is already <i>at-least-once across a failover, by design</i> — identical
+    ///         events even collapse to a single row, because ids hash the payload.
+    ///     </para>
+    ///     <para>
+    ///         The two error costs are therefore asymmetric: a false allow costs a duplicate history row;
+    ///         a false deny silently stops the alarm audit trail and eventually evicts it at the capacity
+    ///         wall. So this decision keys on the role <b>alone</b> and opens when the role is unknown —
+    ///         no membership tie-break, because cluster-wide driver count says nothing about whether
+    ///         <i>this</i> node has a redundant partner.
+    ///     </para>
+    ///     <para>
+    ///         Found by the LocalDb Phase 2 live gate: on a rig of four driver nodes in one cluster with
+    ///         no <c>Redundancy</c> section, borrowing the write gate's verdict suspended the drain on
+    ///         every node — including unpaired ones — so nothing drained anywhere.
+    ///     </para>
+    /// </remarks>
+    /// <param name="localRole">This node's last-known redundancy role, or <c>null</c> when unknown.</param>
+    /// <param name="queueSharingPeerIsPrimary">
+    ///     Whether the node holding <see cref="RedundancyRole.Primary"/> in the same snapshot is a node
+    ///     that <b>shares this node's queue</b> — i.e. its LocalDb replication partner.
+    ///     <para>
+    ///         Merely knowing that <i>a</i> Primary exists is not enough, because the redundancy role is
+    ///         a CLUSTER-WIDE election while the alarm queue is PAIR-LOCAL. A fleet runs many pairs in
+    ///         one cluster, so the elected driver Primary is usually in some other pair — and on the
+    ///         docker-dev rig it is a central node, which carries the driver Akka role, replicates
+    ///         nobody's LocalDb and does not even run the alarm historian. Deferring to it means this
+    ///         node's events are delivered by no one, ever.
+    ///     </para>
+    /// </param>
+    /// <returns><c>true</c> to drain; <c>false</c> only when a node holding these same rows is doing it.</returns>
+    public static bool ShouldDrainAlarmHistory(RedundancyRole? localRole, bool queueSharingPeerIsPrimary) =>
+        localRole switch
+        {
+            RedundancyRole.Primary => true,
+            RedundancyRole.Secondary or RedundancyRole.Detached => !queueSharingPeerIsPrimary,
+            _ => true,   // unknown role ⇒ drain; a duplicate row beats a silent gap
+        };
 }

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/AlarmHistorianOptions.cs

@@ -1,5 +1,4 @@
 using System.Collections.Generic;
-using System.IO;
 using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
 
 namespace ZB.MOM.WW.OtOpcUa.Runtime.Historian;
@@ -7,11 +6,12 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Historian;
 /// <summary>
 ///     Binds the <c>AlarmHistorian</c> configuration section that gates the durable
 ///     store-and-forward alarm sink. When <see cref="Enabled"/> is <c>true</c>,
-///     <c>AddAlarmHistorian</c> registers a <c>SqliteStoreAndForwardSink</c> (draining to the
+///     <c>AddAlarmHistorian</c> registers a <c>LocalDbStoreAndForwardSink</c> (draining to the
 ///     gateway alarm writer supplied by the Host) in place of the
 ///     <c>NullAlarmHistorianSink</c> default; otherwise the Null default survives. This section
-///     supplies only the <see cref="Enabled"/> gate and the SQLite store-and-forward knobs — the
-///     downstream connection (endpoint/key/TLS) is sourced from the <c>ServerHistorian</c> section.
+///     supplies only the <see cref="Enabled"/> gate and the store-and-forward knobs — the
+///     downstream connection (endpoint/key/TLS) is sourced from the <c>ServerHistorian</c> section,
+///     and the queue's storage location is the node's consolidated <c>LocalDb:Path</c> database.
 /// </summary>
 public sealed class AlarmHistorianOptions
 {
@@ -24,9 +24,6 @@ public sealed class AlarmHistorianOptions
     /// </summary>
     public bool Enabled { get; init; }
 
-    /// <summary>Filesystem path to the local SQLite store-and-forward queue database.</summary>
-    public string DatabasePath { get; init; } = "alarm-historian.db";
-
     /// <summary>Maximum number of queued rows the drain worker forwards in a single batch.</summary>
     public int BatchSize { get; init; } = 100;
 
@@ -34,15 +31,15 @@ public sealed class AlarmHistorianOptions
     public int DrainIntervalSeconds { get; init; } = 5;
 
     /// <summary>Maximum queued rows before the sink evicts the oldest. Defaults to 1,000,000
-    /// (matches <c>SqliteStoreAndForwardSink</c>'s <c>DefaultCapacity</c>).</summary>
-    public long Capacity { get; init; } = SqliteStoreAndForwardSink.DefaultCapacity;
+    /// (matches <c>LocalDbStoreAndForwardSink</c>'s <c>DefaultCapacity</c>).</summary>
+    public long Capacity { get; init; } = LocalDbStoreAndForwardSink.DefaultCapacity;
 
     /// <summary>Days to retain dead-lettered rows before purge. Defaults to 30.</summary>
     public int DeadLetterRetentionDays { get; init; } = 30;
 
     /// <summary>Maximum delivery attempts before a perpetually-retrying (poison) row is dead-lettered.
-    /// Defaults to 10 (matches <c>SqliteStoreAndForwardSink</c>'s <c>DefaultMaxAttempts</c>).</summary>
-    public int MaxAttempts { get; init; } = SqliteStoreAndForwardSink.DefaultMaxAttempts;
+    /// Defaults to 10 (matches <c>LocalDbStoreAndForwardSink</c>'s <c>DefaultMaxAttempts</c>).</summary>
+    public int MaxAttempts { get; init; } = LocalDbStoreAndForwardSink.DefaultMaxAttempts;
 
     /// <summary>Returns operator-facing misconfiguration warnings for an <c>Enabled</c> historian
     /// (empty when disabled or correctly configured). Pure — the registration logs each entry.</summary>
@@ -51,8 +48,6 @@ public sealed class AlarmHistorianOptions
     {
         var warnings = new List<string>();
         if (!Enabled) return warnings;
-        if (!Path.IsPathRooted(DatabasePath))
-            warnings.Add($"AlarmHistorian:DatabasePath '{DatabasePath}' is relative — it resolves against the process working directory (e.g. System32 for a Windows service). Set an absolute path.");
         if (DrainIntervalSeconds <= 0)
             warnings.Add($"AlarmHistorian:DrainIntervalSeconds is {DrainIntervalSeconds} — must be > 0; the drain timer will throw or spin at startup.");
         if (Capacity <= 0)

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs

@@ -15,7 +15,7 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Historian;
 /// Thin actor wrapper around <see cref="IAlarmHistorianSink"/>. Engine code (ScriptedAlarmActor,
 /// Galaxy native alarm bridge, AB CIP ALMD reader) tells <see cref="AlarmHistorianEvent"/>s to this
 /// actor; the actor enqueues them on the sink fire-and-forget. Production deployments register
-/// <see cref="SqliteStoreAndForwardSink"/> against <c>IAlarmHistorianSink</c>; the sink owns the
+/// <see cref="LocalDbStoreAndForwardSink"/> against <c>IAlarmHistorianSink</c>; the sink owns the
 /// durable queue + drain-to-HistorianGateway-SendEvent loop. The actor here owns nothing operational beyond
 /// the message contract — its job is to keep the engine actors on Akka's mailbox without blocking
 /// them on disk I/O or gateway round-trips.

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/OpcUa/OpcUaPublishActor.cs

@@ -73,7 +73,17 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
     ///     applied config + the SubscribeBulk pass. It is null only for legacy/dev callers, which
     ///     fall back to the latest sealed deployment (lags a not-yet-sealed apply by one revision).
     /// </summary>
-    public sealed record RebuildAddressSpace(CorrelationId Correlation, DeploymentId? DeploymentId = null);
+    /// <param name="Correlation">Correlation id for tracing this rebuild.</param>
+    /// <param name="DeploymentId">The applied deployment whose artifact to materialise, or null for the latest-sealed fallback.</param>
+    /// <param name="Artifact">
+    ///     The artifact bytes already in hand, used INSTEAD of loading them from the ConfigDb. This is
+    ///     what makes boot-from-cache actually serve its address space: on a central-SQL outage the
+    ///     host has the cached blob but <see cref="DeploymentId"/> alone would drive a ConfigDb read
+    ///     that cannot succeed, leaving the rebuild a no-op and clients browsing an empty server. Null
+    ///     on the normal path, where loading from the ConfigDb by id is correct.
+    /// </param>
+    public sealed record RebuildAddressSpace(
+        CorrelationId Correlation, DeploymentId? DeploymentId = null, byte[]? Artifact = null);
 
     /// <summary>Inject driver-discovered nodes (FixedTree) under an equipment at runtime (post-connect).</summary>
     /// <param name="EquipmentRootNodeId">The OPC UA NodeId of the equipment root folder to inject the
@@ -102,6 +112,10 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
     private int _writes;
     private byte _lastServiceLevel;
     private bool _publishedAtLeastOnce;
+    /// <summary>True once a real composition has been applied, i.e. there is a served address space worth
+    /// protecting. Distinguishes "the artifact went missing under a running server" (issue #485 — warn, and
+    /// hold what is materialised) from "nothing has been deployed here yet" (a quiet no-op at boot).</summary>
+    private bool _hasAppliedComposition;
     private DbHealthProbeActor.DbHealthStatus? _lastDbHealth;
     private RedundancyStateChanged? _lastSnapshot;
     private (bool Ok, DateTime At)? _probeAboutMe;
@@ -345,13 +359,37 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
 
         try
         {
-            // Prefer the artifact of the deployment the host just applied — at apply time it is not
-            // yet Sealed, so LoadLatestArtifact would return the PREVIOUS revision and materialise a
-            // stale composition (variables that don't match the SubscribeBulk refs). Fall back to
-            // latest-sealed only for legacy callers that don't carry a DeploymentId.
-            var artifact = msg.DeploymentId is { } depId
-                ? LoadArtifact(depId)
-                : LoadLatestArtifact();
+            // An in-hand artifact (boot-from-cache) wins: the host already has the cached bytes, and
+            // the ConfigDb read the DeploymentId path would do is exactly what is unreachable during
+            // the outage this exists to survive. Otherwise prefer the artifact of the deployment the
+            // host just applied — at apply time it is not yet Sealed, so LoadLatestArtifact would
+            // return the PREVIOUS revision and materialise a stale composition (variables that don't
+            // match the SubscribeBulk refs). Fall back to latest-sealed only for legacy callers that
+            // carry neither.
+            var artifact = msg.Artifact
+                ?? (msg.DeploymentId is { } depId ? LoadArtifact(depId) : LoadLatestArtifact());
+
+            // Issue #485 — an artifact we could not obtain is NOT an empty configuration. Parsing zero
+            // bytes yields an empty composition, which the planner diffs against the live one as a
+            // PureRemove and the applier then faithfully tears down: a transient ConfigDb blip empties the
+            // served address space. Nothing legitimate produces a zero-length blob (an operator who really
+            // deletes everything still deploys a JSON document with empty arrays), so the only honest
+            // reading of "no bytes" is "no answer" — hold the last-known-good address space and let the
+            // next deploy (or the next boot) supply a real one. Returning here also leaves _lastApplied
+            // intact, so the retry diffs against what is actually materialised.
+            if (artifact is null or { Length: 0 })
+            {
+                if (_hasAppliedComposition)
+                    _log.Warning(
+                        "OpcUaPublish: no usable artifact for deployment {Id} (correlation={Correlation}) — KEEPING the currently served address space rather than tearing it down",
+                        msg.DeploymentId, msg.Correlation);
+                else
+                    _log.Debug(
+                        "OpcUaPublish: no artifact to materialise yet (deployment={Id}, correlation={Correlation})",
+                        msg.DeploymentId, msg.Correlation);
+                return;
+            }
+
             var composition = _localNode is { } ln
                 ? DeploymentArtifact.ParseComposition(artifact, ln.Value,
                     inconsistency => _log.Warning("OpcUaPublish {Node}: cross-cluster binding — {Message}", ln, inconsistency))
@@ -367,6 +405,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
 
             var outcome = _applier.Apply(plan);
             _lastApplied = composition;
+            _hasAppliedComposition = true;
 
             // Sum swallowed per-node materialise failures across every pass together with Apply's own
             // removal-pass tally (archreview 01/S-1). A degraded apply used to vanish into per-node
@@ -431,7 +470,8 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
     }
 
     /// <summary>Read a specific deployment's artifact blob from ConfigDb (the one just applied,
-    /// which may not be Sealed yet). Empty array on any failure — parser treats it as "no composition".</summary>
+    /// which may not be Sealed yet). Empty array on any failure — <see cref="HandleRebuild"/> treats that as
+    /// "no answer" and abandons the rebuild, leaving the served address space untouched (issue #485).</summary>
     private byte[] LoadArtifact(DeploymentId deploymentId)
     {
         try
@@ -444,13 +484,13 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
         }
         catch (Exception ex)
         {
-            _log.Warning(ex, "OpcUaPublish: failed to load artifact for deployment {Id}; rebuild becomes no-op", deploymentId);
+            _log.Warning(ex, "OpcUaPublish: failed to load artifact for deployment {Id}; the rebuild is abandoned", deploymentId);
             return Array.Empty<byte>();
         }
     }
 
     /// <summary>Read the most recent <c>Sealed</c> deployment's artifact blob from ConfigDb.
-    /// Empty array on any failure — the parser treats empty blob as "no composition".</summary>
+    /// Empty array on any failure — see <see cref="LoadArtifact"/> for how that is handled.</summary>
     private byte[] LoadLatestArtifact()
     {
         try
@@ -464,7 +504,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
         }
         catch (Exception ex)
         {
-            _log.Warning(ex, "OpcUaPublish: failed to load latest deployment artifact; rebuild becomes no-op");
+            _log.Warning(ex, "OpcUaPublish: failed to load latest deployment artifact; the rebuild is abandoned");
             return Array.Empty<byte>();
         }
     }
@@ -612,7 +652,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
             Stale: !_lastDbHealth.Reachable
                    || (now - _lastDbHealth.AsOfUtc) > _staleWindow
                    || (now - entry.AsOfUtc) > _staleWindow,
-            IsDriverRoleLeader: entry.IsRoleLeaderForDriver);
+            IsDriverPrimary: entry.IsDriverPrimary);
 
         Self.Tell(new ServiceLevelChanged(ServiceLevelCalculator.Compute(inputs)));
     }
@@ -621,7 +661,7 @@ public sealed class OpcUaPublishActor : ReceiveActor, IWithTimers
     /// secondary → 100, _ → 0). Preserved as the back-compat / bootstrap seam.</summary>
     private static byte LegacyRoleOnly(NodeRedundancyState entry) => entry.Role switch
     {
-        RedundancyRole.Primary when entry.IsRoleLeaderForDriver => 240,
+        RedundancyRole.Primary when entry.IsDriverPrimary => 240,
         RedundancyRole.Primary => 200,
         RedundancyRole.Secondary => 100,
         _ => 0,

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/IRedundancyRoleView.cs

@@ -0,0 +1,57 @@
+using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
+
+namespace ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
+
+/// <summary>
+///     A singleton snapshot of the alarm-history drain decision, so code that lives outside an actor
+///     can ask "should this node be draining the alarm queue right now?".
+/// </summary>
+/// <remarks>
+///     <para>
+///         Exists for the alarm store-and-forward drain. That drain runs on a timer owned by the
+///         sink, not on an actor mailbox, so it cannot receive <c>RedundancyStateChanged</c>
+///         directly — but it must be role-scoped, because the queue it drains replicates to the
+///         pair peer and two draining nodes would deliver every event twice.
+///     </para>
+///     <para>
+///         Deliberately publishes the <i>decision</i> rather than the role that produced it, so
+///         <see cref="PrimaryGatePolicy"/> stays the single place that decides.
+///     </para>
+///     <para>
+///         <b>This is not the device-write gate.</b> It is fed by
+///         <see cref="PrimaryGatePolicy.ShouldDrainAlarmHistory"/>, which opens on an unknown role
+///         where <see cref="PrimaryGatePolicy.ShouldServiceAsPrimary"/> closes. The naming is
+///         explicit about the consumer for that reason: an earlier revision published the
+///         write-gate verdict here, and on a cluster with several driver nodes but no configured
+///         redundancy it suspended the drain everywhere, so alarm history accumulated on every node
+///         and left none.
+///     </para>
+/// </remarks>
+public interface IRedundancyRoleView
+{
+    /// <summary>Whether this node should be draining the alarm store-and-forward queue right now.</summary>
+    bool ShouldDrainAlarmHistory { get; }
+
+    /// <summary>Records a fresh decision. Called by <c>DriverHostActor</c> on every redundancy snapshot.</summary>
+    /// <param name="shouldDrainAlarmHistory">
+    ///     The decision <see cref="PrimaryGatePolicy.ShouldDrainAlarmHistory"/> produced.
+    /// </param>
+    void Publish(bool shouldDrainAlarmHistory);
+}
+
+/// <summary>Thread-safe <see cref="IRedundancyRoleView"/> backed by a volatile field.</summary>
+public sealed class RedundancyRoleView : IRedundancyRoleView
+{
+    // Seeded with the decision for "role unknown" rather than a bare false. An unpublished view and a
+    // node whose role nothing ever reports are genuinely the same situation, and they must behave the
+    // same: a deployment that runs no redundancy at all never publishes here, and defaulting closed
+    // would silently stop its alarm history forever.
+    private volatile bool _shouldDrainAlarmHistory =
+        PrimaryGatePolicy.ShouldDrainAlarmHistory(localRole: null, queueSharingPeerIsPrimary: false);
+
+    /// <inheritdoc/>
+    public bool ShouldDrainAlarmHistory => _shouldDrainAlarmHistory;
+
+    /// <inheritdoc/>
+    public void Publish(bool shouldDrainAlarmHistory) => _shouldDrainAlarmHistory = shouldDrainAlarmHistory;
+}

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs

@@ -15,11 +15,14 @@ using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
 using ZB.MOM.WW.OtOpcUa.Core.Scripting;
 using ZB.MOM.WW.OtOpcUa.Core.VirtualTags;
 using ZB.MOM.WW.OtOpcUa.OpcUaServer;
+using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
 using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
 using ZB.MOM.WW.OtOpcUa.Runtime.Health;
 using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
 using ZB.MOM.WW.OtOpcUa.Runtime.OpcUa;
+using ZB.MOM.WW.OtOpcUa.Runtime.Redundancy;
 using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
+using ZB.MOM.WW.LocalDb;
 
 namespace ZB.MOM.WW.OtOpcUa.Runtime;
 
@@ -38,7 +41,7 @@ public static class ServiceCollectionExtensions
     /// <summary>
     /// Registers shared runtime services. Currently binds <see cref="IAlarmHistorianSink"/>
     /// to <see cref="NullAlarmHistorianSink"/> as the default; production deployments
-    /// override this with <c>SqliteStoreAndForwardSink</c> wrapping the HistorianGateway alarm writer.
+    /// override this with <c>LocalDbStoreAndForwardSink</c> wrapping the HistorianGateway alarm writer.
     /// Call this BEFORE <c>AddAkka</c>.
     /// </summary>
     /// <param name="services">The service collection to register with.</param>
@@ -64,12 +67,23 @@ public static class ServiceCollectionExtensions
 
     /// <summary>
     ///     Config-gated durable alarm-historian sink. When the <c>AlarmHistorian</c> section has
-    ///     <c>Enabled=true</c>, registers a <see cref="SqliteStoreAndForwardSink"/> (draining via the
+    ///     <c>Enabled=true</c>, registers a <see cref="LocalDbStoreAndForwardSink"/> (draining via the
     ///     <paramref name="writerFactory"/>-supplied writer) as the <see cref="IAlarmHistorianSink"/>,
     ///     overriding the <see cref="NullAlarmHistorianSink"/> default. Otherwise a no-op (Null stays).
     ///     The writer is injected so the durable downstream (the HistorianGateway alarm writer) can be
     ///     supplied by the Host, which is the only project that references it.
     /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         The queue lives in the node's consolidated <see cref="ILocalDb"/>, which replicates it
+    ///         to the redundant pair peer — so undelivered alarm history survives losing the node
+    ///         holding it. That is also why the drain is gated on <see cref="IRedundancyRoleView"/>:
+    ///         the Secondary holds a full replica of the Primary's queue, and an ungated drain there
+    ///         would re-deliver every event. Both services are resolved from the provider, so a
+    ///         deployment that enables this section without registering LocalDb fails loudly at
+    ///         resolution rather than silently buffering to nowhere.
+    ///     </para>
+    /// </remarks>
     /// <param name="services">The service collection to register with.</param>
     /// <param name="configuration">The configuration carrying the <c>AlarmHistorian</c> section.</param>
     /// <param name="writerFactory">
@@ -86,21 +100,56 @@ public static class ServiceCollectionExtensions
         if (opts is not { Enabled: true }) return services; // leave the Null default from AddOtOpcUaRuntime
 
         foreach (var warning in opts.Validate())
-            Serilog.Log.Logger.ForContext<SqliteStoreAndForwardSink>().Warning("Historian config: {HistorianConfigWarning}", warning);
+            Serilog.Log.Logger.ForContext<LocalDbStoreAndForwardSink>().Warning("Historian config: {HistorianConfigWarning}", warning);
+
+        // The view is a plain singleton so it exists even on nodes whose DriverHostActor never
+        // publishes to it — an unpublished view reads as "no peer, drain", which is the correct
+        // posture for a deployment that runs no redundancy at all.
+        services.TryAddSingleton<IRedundancyRoleView, RedundancyRoleView>();
+
+        // THE GATE ONLY APPLIES TO A REPLICATED QUEUE. Standing down is only ever safe because some
+        // other node holds the same rows and will send them instead — and the only node that does is
+        // this node's LocalDb replication peer. Without replication configured, these rows exist here
+        // and nowhere else, so deferring to anyone means they are never delivered by anyone.
+        //
+        // This is not hypothetical. The redundancy role is a CLUSTER-WIDE election
+        // (RedundancyStateActor keys on Akka's RoleLeader("driver")), while the queue is PAIR-LOCAL.
+        // On the docker-dev rig the elected driver Primary is a central node — which carries the
+        // driver Akka role, replicates nobody's LocalDb, and does not even run the alarm historian —
+        // so every site node dutifully suspended its drain in favour of a node that could not
+        // possibly deliver its events. Scoping the gate to "is my queue actually shared?" is what
+        // keeps the two scopes from disagreeing.
+        // BOTH halves of a pair share the queue, but only one of them dials: the initiator sets
+        // Replication:PeerAddress, its partner only sets SyncListenPort and waits. Testing the dial
+        // side alone would leave the listening half permanently ungated — one drainer per pair by
+        // accident rather than by role, and the wrong one whenever the roles swap.
+        var replicated =
+            !string.IsNullOrWhiteSpace(configuration["LocalDb:Replication:PeerAddress"])
+            || !string.IsNullOrWhiteSpace(configuration["LocalDb:SyncListenPort"]);
+
+        if (!replicated)
+        {
+            Serilog.Log.Logger.ForContext<LocalDbStoreAndForwardSink>().Information(
+                "Alarm historian: LocalDb replication is not configured, so this node's queue is not "
+                + "shared with any peer and the Primary drain gate does not apply — this node always "
+                + "drains its own alarm queue.");
+        }
 
         services.AddSingleton<IAlarmHistorianSink>(sp =>
         {
-            // SqliteStoreAndForwardSink takes a Serilog ILogger (not Microsoft.Extensions.Logging).
+            // LocalDbStoreAndForwardSink takes a Serilog ILogger (not Microsoft.Extensions.Logging).
             // Resolve it off the host's configured static logger so the drain worker's WARN/INFO
             // lines land in the same sinks as the rest of the process.
-            var sink = new SqliteStoreAndForwardSink(
-                opts.DatabasePath,
+            var roleView = sp.GetRequiredService<IRedundancyRoleView>();
+            var sink = new LocalDbStoreAndForwardSink(
+                sp.GetRequiredService<ILocalDb>(),
                 writerFactory(opts, sp),
-                Serilog.Log.Logger.ForContext<SqliteStoreAndForwardSink>(),
+                Serilog.Log.Logger.ForContext<LocalDbStoreAndForwardSink>(),
                 batchSize: opts.BatchSize,
                 capacity: opts.Capacity,
                 deadLetterRetention: TimeSpan.FromDays(opts.DeadLetterRetentionDays),
-                maxAttempts: opts.MaxAttempts);
+                maxAttempts: opts.MaxAttempts,
+                drainGate: () => !replicated || roleView.ShouldDrainAlarmHistory);
             sink.StartDrainLoop(TimeSpan.FromSeconds(opts.DrainIntervalSeconds));
             return sink;
         });
@@ -221,6 +270,26 @@ public static class ServiceCollectionExtensions
             var serviceLevel = resolver.GetService<IServiceLevelPublisher>() ?? NullServiceLevelPublisher.Instance;
             var loggerFactory = resolver.GetService<ILoggerFactory>() ?? NullLoggerFactory.Instance;
             var healthPublisher = resolver.GetService<IDriverHealthPublisher>() ?? NullDriverHealthPublisher.Instance;
+            // Node-local deployment-artifact cache. Registered by the Host's AddOtOpcUaLocalDb on
+            // driver-role nodes only; deliberately left null elsewhere (admin-only graphs, test
+            // harnesses) rather than given a null-object, so DriverHostActor skips caching outright
+            // instead of pretending to cache into a sink that drops everything.
+            var deploymentArtifactCache = resolver.GetService<IDeploymentArtifactCache>();
+            // Where this actor publishes its Primary-gate verdict for the alarm store-and-forward
+            // drain, which runs on a timer and so cannot read RedundancyStateChanged itself.
+            // Registered by AddAlarmHistorian; absent when no durable sink is configured, in which
+            // case there is nothing downstream to inform.
+            var redundancyRoleView = resolver.GetService<IRedundancyRoleView>();
+            // Host of this node's LocalDb replication partner: the ONLY node that holds a copy of this
+            // node's alarm queue, and so the only node it may stand down in favour of. Null when this
+            // node dials nobody, which correctly means "never stand down".
+            var replicationPeerHost =
+                Uri.TryCreate(
+                    resolver.GetService<IConfiguration>()?["LocalDb:Replication:PeerAddress"],
+                    UriKind.Absolute,
+                    out var peerUri)
+                    ? peerUri.Host
+                    : null;
             // Root script logger backs the ScriptedAlarm host's engine + script logging. Registered in
             // Host DI inside the hasDriver block; may be absent in some role configs / test harnesses,
             // in which case the DriverHostActor gracefully skips spawning the ScriptedAlarm host.
@@ -338,7 +407,10 @@ public static class ServiceCollectionExtensions
                     historyWriter: historyWriter,
                     loggerFactory: loggerFactory,
                     scriptRootLogger: scriptRootLogger,
-                    invokerFactory: invokerFactory),
+                    invokerFactory: invokerFactory,
+                    deploymentArtifactCache: deploymentArtifactCache,
+                    redundancyRoleView: redundancyRoleView,
+                    replicationPeerHost: replicationPeerHost),
                 DriverHostActorName);
             registry.Register<DriverHostActorKey>(driverHost);
 

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ZB.MOM.WW.OtOpcUa.Runtime.csproj

@@ -8,6 +8,9 @@
     <ItemGroup>
         <PackageReference Include="Akka.Hosting"/>
         <PackageReference Include="Akka.Cluster.Tools"/>
+        <!-- Core LocalDb only (ILocalDb + the connection/transaction seam) — the deployment-artifact
+             cache lives here, but the sync engine and its gRPC endpoint are the Host's concern. -->
+        <PackageReference Include="ZB.MOM.WW.LocalDb" />
     </ItemGroup>
 
     <ItemGroup>

tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/AkkaClusterOptionsValidatorTests.cs

@@ -0,0 +1,177 @@
+using Shouldly;
+using Xunit;
+
+namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
+
+/// <summary>
+/// Guards the self-first seed-ordering rule at the only moment it can still be fixed cheaply: boot.
+/// </summary>
+/// <remarks>
+///     <para>
+///         The invariant (decision 2026-07-22) is <b>conditional</b>: <i>if</i> a node's own address
+///         appears in its own <see cref="AkkaClusterOptions.SeedNodes"/>, it must be entry 0. Akka
+///         runs <c>FirstSeedNodeProcess</c> — the only path that can form a NEW cluster when no peer
+///         answers <c>InitJoin</c> — exclusively for <c>seed-nodes[0]</c>; every other node retries
+///         <c>InitJoin</c> forever. A node listed second therefore cannot cold-start while its peer
+///         is down, and it fails <i>silently</i>: the process is healthy, the port is open, the node
+///         simply never becomes a cluster member. That is why the rule is enforced loudly here.
+///     </para>
+///     <para>
+///         The conditional form is required, not incidental: a flat "self must be first" rule would
+///         reject every driver-only site node, which is seeded solely by <c>central-1</c> and is
+///         legitimately not a seed of anything —
+///         see <see cref="Node_absent_from_its_own_seed_list_is_exempt"/>.
+///     </para>
+/// </remarks>
+public sealed class AkkaClusterOptionsValidatorTests
+{
+    private static AkkaClusterOptions CentralNode(string publicHostname, params string[] seeds) =>
+        new()
+        {
+            // The docker-dev shape: bind to every interface, advertise the container DNS name.
+            Hostname = "0.0.0.0",
+            PublicHostname = publicHostname,
+            Port = 4053,
+            Roles = new[] { "admin", "driver" },
+            SeedNodes = seeds,
+        };
+
+    private static string Seed(string host, int port = 4053) => $"akka.tcp://otopcua@{host}:{port}";
+
+    /// <summary>The shipped central-1 / central-2 shape: own address first, partner second.</summary>
+    [Fact]
+    public void Self_first_seed_order_passes()
+    {
+        var options = CentralNode("central-2", Seed("central-2"), Seed("central-1"));
+
+        var result = new AkkaClusterOptionsValidator().Validate(null, options);
+
+        result.Failed.ShouldBeFalse(result.FailureMessage);
+    }
+
+    /// <summary>
+    /// The pre-2026-07-22 ordering — partner first — is the boot-alone outage gap, and must not
+    /// start. This is the shape <c>docker-dev</c>'s <c>central-2</c> carried.
+    /// </summary>
+    [Fact]
+    public void Peer_first_seed_order_fails()
+    {
+        var options = CentralNode("central-2", Seed("central-1"), Seed("central-2"));
+
+        var result = new AkkaClusterOptionsValidator().Validate(null, options);
+
+        result.Failed.ShouldBeTrue();
+        result.FailureMessage.ShouldNotBeNull().ShouldContain("SeedNodes");
+        result.FailureMessage.ShouldContain("must list this node itself first");
+    }
+
+    /// <summary>
+    /// THE conditional exemption. A driver-only site node lists only <c>central-1</c> — it is not a
+    /// seed at all, is never legitimately first, and must pass. An unconditional "self must be
+    /// first" rule would refuse to boot every site node in the fleet.
+    /// </summary>
+    [Fact]
+    public void Node_absent_from_its_own_seed_list_is_exempt()
+    {
+        var options = CentralNode("site-a-1", Seed("central-1"));
+
+        var result = new AkkaClusterOptionsValidator().Validate(null, options);
+
+        result.Failed.ShouldBeFalse(result.FailureMessage);
+    }
+
+    /// <summary>
+    /// The bind address is NOT the node's identity. In docker-dev <c>Cluster:Hostname</c> is
+    /// <c>0.0.0.0</c> and <c>Cluster:PublicHostname</c> is the container name — the value Akka puts
+    /// in <c>SelfAddress</c> and therefore the only one a seed entry can match. A validator that
+    /// compared against <c>Hostname</c> would find no match anywhere, silently exempt every node,
+    /// and pass this misordered config.
+    /// </summary>
+    [Fact]
+    public void Identity_is_the_public_hostname_not_the_bind_address()
+    {
+        var options = CentralNode("central-2", Seed("central-1"), Seed("central-2"));
+        options.Hostname.ShouldBe("0.0.0.0", "the trap only exists when the bind address is a wildcard");
+
+        var result = new AkkaClusterOptionsValidator().Validate(null, options);
+
+        result.Failed.ShouldBeTrue("matching on the 0.0.0.0 bind address would make this rule vacuous");
+    }
+
+    /// <summary>
+    /// With <c>PublicHostname</c> blank Akka advertises the bind hostname, so that is the identity to
+    /// match — otherwise a loopback/bare-metal pair configured that way would be silently exempt.
+    /// </summary>
+    [Fact]
+    public void Identity_falls_back_to_the_bind_hostname_when_no_public_hostname_is_set()
+    {
+        var options = CentralNode("central-2", Seed("node-a"), Seed("node-b"));
+        options.Hostname = "node-b";
+        options.PublicHostname = string.Empty;
+
+        var result = new AkkaClusterOptionsValidator().Validate(null, options);
+
+        result.Failed.ShouldBeTrue();
+        result.FailureMessage.ShouldNotBeNull().ShouldContain("must list this node itself first");
+    }
+
+    /// <summary>
+    /// Host AND port: a two-node install sharing one hostname and differing only by port (a
+    /// loopback/dev pair) is a real topology, and matching on host alone would clear the misordered
+    /// node.
+    /// </summary>
+    [Fact]
+    public void Self_match_compares_host_and_port()
+    {
+        var options = CentralNode("127.0.0.1", Seed("127.0.0.1", 4053), Seed("127.0.0.1", 4054));
+        options.Port = 4054;
+
+        var result = new AkkaClusterOptionsValidator().Validate(null, options);
+
+        result.Failed.ShouldBeTrue("this node is 127.0.0.1:4054, which is the SECOND seed");
+    }
+
+    /// <summary>
+    /// Positive control for the pair above: the same host on the same port really is a match, so the
+    /// port comparison cannot be passing merely by never matching anything.
+    /// </summary>
+    [Fact]
+    public void Self_match_on_the_same_host_and_port_passes()
+    {
+        var options = CentralNode("127.0.0.1", Seed("127.0.0.1", 4054), Seed("127.0.0.1", 4053));
+        options.Port = 4054;
+
+        var result = new AkkaClusterOptionsValidator().Validate(null, options);
+
+        result.Failed.ShouldBeFalse(result.FailureMessage);
+    }
+
+    /// <summary>
+    /// An empty seed list is the single-node/dev default and says nothing about ordering — the rule
+    /// has nothing to bind to and must not invent a failure.
+    /// </summary>
+    [Fact]
+    public void Empty_seed_list_passes()
+    {
+        var options = CentralNode("central-1");
+
+        var result = new AkkaClusterOptionsValidator().Validate(null, options);
+
+        result.Failed.ShouldBeFalse(result.FailureMessage);
+    }
+
+    /// <summary>
+    /// A malformed seed URI is Akka's error to report, with Akka's wording. This rule must not
+    /// crash on it, and must not turn a parse problem into a misleading "ordering" complaint —
+    /// here the ordering is correct and only a later entry is unparseable.
+    /// </summary>
+    [Fact]
+    public void Malformed_seed_entry_does_not_throw()
+    {
+        var options = CentralNode("central-1", Seed("central-1"), "not-an-akka-address");
+
+        var result = new AkkaClusterOptionsValidator().Validate(null, options);
+
+        result.Failed.ShouldBeFalse(result.FailureMessage);
+    }
+}