Merge branch 'feat/mesh-phase3' — per-cluster mesh Phase 3 (config fetch-and-cache)

Driver nodes can obtain config via a gRPC fetch from central + LocalDb cache instead of
reading central SQL, behind the ConfigSource:Mode=FetchAndCache dark switch (Direct
default). Central serves the artifact over a dedicated h2c listener gated by a shared node
key; the node SHA-verifies against the dispatched RevisionHash, applies from the bytes in
hand, and boots from the LocalDb pointer with no SQL read. Live gate PASSED on docker-dev
(caught + fixed a Status==Sealed serve-gate deadlock). Phase boundary: config READS only.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

.dockerignore

@@ -0,0 +1,29 @@
+# Build context exclusions for docker-dev/Dockerfile, whose `COPY . .` takes the whole repo.
+#
+# Without this file the context also carries every project's bin/obj AND .claude/worktrees — agent
+# worktrees that are themselves full checkouts WITH their own build outputs. That reached ~19 GB and
+# exhausted the Docker VM's disk mid-COPY, failing the rig build with "no space left on device"
+# rather than anything a build log would attribute to the repo.
+#
+# Nothing excluded here is needed: the build restores and publishes inside the image, and the
+# migrator stage runs `dotnet ef` against the same restored source.
+
+# Agent worktrees — full checkouts, build outputs and all.
+.claude/
+
+# Local build outputs; the image builds its own.
+**/bin/
+**/obj/
+artifacts/
+
+# VCS + editor state.
+.git/
+.gitignore
+.vs/
+.vscode/
+.idea/
+
+# Docker's own state, and anything the image would never read.
+**/.DS_Store
+**/*.user
+docker-dev/**/*.log

CLAUDE.md

@@ -219,6 +219,142 @@ The server supports configurable OPC UA transport security via the `OpcUa:Enable
 
 The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide.
 
+**Two corrections landed 2026-07-21 — both were total-outage or wrong-node defects, and both are worth knowing before touching this area.**
+
+- **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is still open** — the pathology only appears 1-vs-1 and docker-dev is a single six-node mesh, so the crash-the-oldest drill is deferred to the per-cluster mesh work.
+- **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**.
+
+**A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node’s own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering".
+
+Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
+
+**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2 code-complete behind a dark switch; 3–7 not started).
+
+## Mesh command transport (`MeshTransport`)
+
+Per-cluster mesh **Phase 2** added a second transport for the three central→node command channels
+(`deployments`, `driver-control`, `alarm-commands`) and the `deployment-acks` reply, selected by
+`MeshTransport:Mode`: `Dps` (the DistributedPubSub path this repo has always used, **the default**)
+or `ClusterClient`, which does **not** require central and the node to share a gossip ring — the
+prerequisite for splitting the fleet into one mesh per application `Cluster`. Both `CentralCommunicationActor`
+(`/user/central-communication`) and `NodeCommunicationActor` (`/user/node-communication`) are spawned
+and registered with the `ClusterClientReceptionist` in **both** modes, so flipping the flag is a config
+change, not a redeploy. Things worth knowing before touching it:
+
+- **`SendToAll`, never `Send`.** Today's DPS publish reaches every `DriverHostActor`; there is no
+  ClusterId or node filter on the node side at all (scoping happens later, inside the artifact via
+  `DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered
+  actor — it would deploy to a single node while every other node silently kept its old configuration,
+  and the deployment could still seal green.
+- **Exactly ONE fleet-wide ClusterClient while the fleet is one mesh.** A receptionist serves its whole
+  cluster, so `SendToAll` reaches every registered node-comm actor regardless of which node's address was
+  dialled. One client per application `Cluster` — the obviously-right-looking shape, and what Phase 6
+  wants — would fan each command out once per cluster (N× duplicate dispatch *and* N× duplicate ack).
+  Marked `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient`. **Corollary:** the contact set does
+  not scope delivery; excluding a `MaintenanceMode` node from the contacts does not stop it receiving.
+- **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and
+  central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber.
+  Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both.
+- **Comm actors are per-node, NOT singletons.** A client rotates across contact points and must find a live
+  comm actor at whichever node answers; `akka.cluster.client.receptionist.role` is empty for the same reason.
+- **`buffer-size = 0` is a behaviour decision, not a tuning knob.** Akka's default buffers 1000 and replays
+  on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not raise
+  it to quiet a flaky test. **But do not over-claim what it buys:** the Phase 2 live gate observed a node
+  applying a deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()`
+  replaying the orphan `Applying` row on restart. Zero buffering removes the silent replay, not every replay.
+
+Contact points are node **addresses only** (`akka.tcp://<system>@<host>:<port>`); `/system/receptionist` is
+appended at construction and `MeshTransportOptionsValidator` refuses to start a node whose contact carries an
+actor-path suffix. `redundancy-state` stays on DPS in both modes (bidirectional, and built from
+`Cluster.State`). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md` §"Command transport",
+and `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`.
+
+**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
+
+- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag.
+- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out.
+- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
+- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever.
+
+## Config source (`ConfigSource` / `ConfigServe`)
+
+Per-cluster mesh **Phase 3** (config fetch-and-cache) added a second way a **driver** node obtains its
+deployed configuration, selected by `ConfigSource:Mode`: `Direct` (read `Deployment.ArtifactBlob` from
+central SQL — **the default**, byte-for-byte today's behaviour) or `FetchAndCache` (fetch the artifact
+bytes from central over a dedicated-h2c **gRPC stream**, verify `SHA-256 == RevisionHash`, cache in
+LocalDb, and read **only** the cache — no central-SQL config read). Central (admin role) serves via
+`DeploymentArtifactService` behind `ConfigServeAuthInterceptor` on `ConfigServe:GrpcListenPort`, gated by
+a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`,
+fail-closed). Things worth knowing before touching it:
+
+- **The serve side is wired in both modes** (mapped whenever `ConfigServe:GrpcListenPort > 0` on an
+  admin node), so flipping a node to `FetchAndCache` is a config change, not a redeploy — the same
+  dark-switch discipline as [MeshTransport](#mesh-command-transport-meshtransport).
+- **`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the revision
+  hash as the lowercase-hex SHA-256 of the exact blob, so the node verifies a fetch against the
+  `RevisionHash` the dispatch already carries — **no hash rides in the proto, and `DispatchDeployment`
+  stays payload-free.** A future change to how the revision hash is derived silently breaks every fetch.
+- **A null fetch is "no answer," never an empty config (#485).** All endpoints down / `NotFound` / SHA
+  mismatch / zero-length ⇒ the fetcher returns null ⇒ the apply FAILS, the revision does not advance,
+  and the node keeps serving last-known-good. It never tears down to an empty address space. At boot a
+  `FetchAndCache` node restores served state from the LocalDb pointer with **no** central-SQL read; an
+  empty/unreadable cache lands Steady-with-no-revision (the first dispatch fetches), **never Stale**.
+- **Both pair nodes fetch independently** — no Primary gating of the fetch (same shape as both reading
+  SQL today; `StoreAsync` is idempotent and the cache replicates).
+- **The dedicated h2c listener coexists with the LocalDb-sync listener.** A fused admin+driver node with
+  both ports set re-binds its existing HTTP surface exactly **once** and adds both dedicated HTTP/2-only
+  ports — getting this wrong either double-binds (crash) or silently unbinds the AdminUI (the Kestrel
+  `Listen*`-discards-`ASPNETCORE_URLS` trap the LocalDb block already warned about).
+- **Phase boundary: config *reads* only.** A `FetchAndCache` node still writes its `NodeDeploymentState`
+  ack row to central SQL; `DbHealthProbe` and `EfAlarmConditionStateStore` still touch SQL. Cutting those
+  is Phase 4 — resist folding it in, or the "no SQL" scope becomes unprovable-in-parts.
+
+On the docker-dev rig the central pair carries `ConfigServe` (`:4055`, committed dev key) and stays
+`Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`.
+Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up`. See
+`docs/Configuration.md` §`ConfigSource` / `ConfigServe`, `docs/Redundancy.md` §"Command transport", and
+`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`.
+
+## LocalDb pair-local store (Phases 1 + 2)
+
+Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
+LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**:
+`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2).
+
+**Phase 1** caches the
+**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a
+driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes
+the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`,
+in `Runtime/Deployment/`) and reads it as a boot fallback, flagging running-from-cache. Wired by
+`AddOtOpcUaLocalDb` in the `hasDriver` branch only (`Host/Configuration/LocalDbRegistration.cs` +
+`LocalDbSetup.OnReady`, whose DDL→`RegisterReplicated` order is load-bearing); **admin-only nodes
+never register it.** The cache can optionally replicate to the node's redundant pair peer over the
+library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAuthInterceptor`), gated
+on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the
+docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is
+the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret
+exception).
+
+**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db`
+and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm
+history no longer takes it to the grave. `SqliteStoreAndForwardSink` became
+`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key
+`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely
+to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy
+commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the
+sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that
+`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new
+`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by
+design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the
+same event accepted on both nodes — which happens in every boot window, since
+`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the
+docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a
+deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.
+
+See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
+cache), `docs/AlarmHistorian.md`, and the runbook
+`docs/operations/2026-07-20-localdb-pair-replication.md`.
+
 ## LDAP Authentication
 
 The server uses LDAP-based user authentication via the `Security:Ldap` section in `appsettings.json`. When enabled, credentials are validated by LDAP bind against a GLAuth server, and LDAP group membership maps to OPC UA permissions: `ReadOnly` (browse/read), `WriteOperate` (write FreeAccess/Operate attributes), `WriteTune` (write Tune attributes), `WriteConfigure` (write Configure attributes), `AlarmAck` (alarm acknowledgment). `LdapOpcUaUserAuthenticator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`) implements `IOpcUaUserAuthenticator`, delegating the LDAP bind + group lookup to `OtOpcUaLdapAuthService` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs`, an `ILdapAuthService`). See `docs/security.md` for the full guide.
@@ -321,11 +457,13 @@ authorization uses the standard `AccessLevels.HistoryRead` bit set at materializ
 ### Alarm-history path (`AlarmHistorian` section)
 
 Alarm events are written through `GatewayAlarmHistorianWriter` (the gateway **`SendEvent`** path) behind
-the durable **`SqliteStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink`
-default for the SQLite store-and-forward queue, whose drain worker forwards batches to the gateway and uses
-per-event outcomes to decide retry vs. dead-letter (never throws). The `AlarmHistorian` section carries
-only the `Enabled` gate + the SQLite knobs (`DatabasePath`, `DrainIntervalSeconds`, `Capacity`,
-`DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection
+the durable **`LocalDbStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink`
+default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the
+replicated `alarm_sf_events` table (see the LocalDb section) and whose drain worker forwards batches to
+the gateway and uses per-event outcomes to decide retry vs. dead-letter (never throws). **Only the node
+holding the Primary role drains** (a gated node reports `HistorianDrainState.NotPrimary`). The
+`AlarmHistorian` section carries only the `Enabled` gate + the queue knobs (`DrainIntervalSeconds`,
+`Capacity`, `DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection
 (endpoint/key/TLS) is sourced from the `ServerHistorian` section. **Alarm-history `ReadEvents` requires the
 target gateway deployed with `RuntimeDb:EventReadsEnabled=true`** (the C2 SQL event-read workaround).
 
@@ -397,11 +535,16 @@ The `AlarmHistorian` section's old Wonderware connection keys (`Host`/`Port`/`Us
 were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from
 `ServerHistorian`. See `docs/Historian.md` for the full guide.
 
-### KNOWN LIMITATION 1 — live-validation gate (do before merging/trusting the cutover)
+### Live-validation gate — PASSED (was "KNOWN LIMITATION 1")
 
-The cutover is code-complete but **must be live-validated against a real gateway** (VPN to
-`wonder-sql-vd03`, gateway running the prerequisites above) before it is merged or trusted. Run the
-env-gated suite:
+**No longer a limitation.** The cutover was live-validated against the real gateway on
+`wonder-sql-vd03`: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a *confirmed
+protocol limitation* (the captured `CM_EVENT` wire never carries `SourceName`, so the gateway cannot
+populate `Source_Object` — not a fixable bug). Evidence:
+`docs/plans/2026-06-27-otopcua-historian-followups.md`.
+
+Kept here because it is the recipe to **re-run** the gate after any gateway or backend change (VPN to
+`wonder-sql-vd03`, gateway running the prerequisites above):
 
 ```bash
 export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222   # absolute gateway URI; absent ⇒ all live tests skip
@@ -416,7 +559,7 @@ dotnet test --filter "Category=LiveIntegration"
 The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the
 gate the operator runs on the VPN before trusting the cutover.
 
-### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending
+### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending (issue #491)
 
 The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer +
 meters). It is spawned with an **empty *initial* ref set** (`Array.Empty<string>()` in
@@ -430,4 +573,4 @@ deployed set of historized value tags from the first deploy onward (the feed is
 path is code-complete.** What remains is the **live gate**: an end-to-end verification against a real gateway
 (deploy → dependency-mux value delta → outbox append → `WriteLiveValues`) plus a restart-convergence test
 proving the recorder re-registers the deployed set after a process restart. Until that live verification runs,
-treat continuous value-capture as unproven-in-production rather than absent.
+treat continuous value-capture as unproven-in-production rather than absent. Tracked as **issue #491**.

Directory.Build.rsp

@@ -0,0 +1,12 @@
+# MSBuild auto-response file, applied to every build run from this repo root.
+#
+# Disable the MSBuild node pool. By default MSBuild leaves worker nodes resident
+# after a build so the next one starts warm. Across a session of repeated
+# solution-wide builds and test runs this repo accumulated 55 idle nodes holding
+# ~6 GB, which is not merely untidy: the integration suites assert on timeouts,
+# and memory pressure turns those into failures indistinguishable from real
+# regressions (see the Workstation-GC note in Host.IntegrationTests.csproj for
+# the same failure mode from a different cause).
+#
+# The cost is a small startup penalty per build. That is a good trade here.
+-nodeReuse:false

Directory.Packages.props

@@ -29,10 +29,11 @@
     <PackageVersion Include="coverlet.collector" Version="6.0.4" />
     <PackageVersion Include="FluentAssertions" Version="8.3.0" />
     <PackageVersion Include="Google.Protobuf" Version="3.34.1" />
+    <PackageVersion Include="Grpc.AspNetCore" Version="2.76.0" />
     <PackageVersion Include="Grpc.Core.Api" Version="2.76.0" />
     <PackageVersion Include="Grpc.Net.Client" Version="2.76.0" />
+    <PackageVersion Include="Grpc.Tools" Version="2.76.0" />
     <PackageVersion Include="libplctag" Version="1.5.2" />
-    <PackageVersion Include="LiteDB" Version="5.0.21" />
     <PackageVersion Include="MessagePack" Version="2.5.301" />
     <PackageVersion Include="Microsoft.AspNetCore.Authorization" Version="10.0.7" />
     <PackageVersion Include="Microsoft.AspNetCore.DataProtection" Version="10.0.7" />
@@ -57,6 +58,7 @@
     <PackageVersion Include="Microsoft.EntityFrameworkCore.InMemory" Version="10.0.7" />
     <PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="10.0.7" />
     <PackageVersion Include="Microsoft.FASTER.Core" Version="2.6.5" />
+    <PackageVersion Include="Microsoft.Extensions.Configuration" Version="10.0.7" />
     <PackageVersion Include="Microsoft.Extensions.Configuration.Abstractions" Version="10.0.7" />
     <PackageVersion Include="Microsoft.Extensions.Configuration.Json" Version="10.0.7" />
     <PackageVersion Include="Microsoft.Extensions.DependencyInjection" Version="10.0.7" />
@@ -103,6 +105,23 @@
       clearing the advisory without CentralPackageTransitivePinningEnabled (which breaks the Roslyn split).
     -->
     <PackageVersion Include="SQLitePCLRaw.bundle_e_sqlite3" Version="2.1.12" />
+    <!--
+      Surgical transitive pin #2. Four NU1903 high-severity advisories (GHSA-23rf-6693-g89p,
+      GHSA-8q5v-6pqq-x66h, GHSA-cvvh-rhrc-wg4q, GHSA-g8r8-53c2-pm3f) landed in the NuGet audit
+      data against System.Security.Cryptography.Xml 10.0.7, which Microsoft.AspNetCore.DataProtection
+      10.0.7 pulls in transitively for its key storage. Under TreatWarningsAsErrors every FRESH
+      restore goes red — 204 NU1903 errors across the solution — while machines with cached audit
+      data keep building, so this surfaces first on a clean clone or a docker image build, not
+      locally. Verified 2026-07-21 by restoring a detached worktree.
+
+      10.0.10 is the patched version, pinned by direct PackageReference in Core.Configuration (the
+      one project where the DataProtection chain enters). Bumping the DataProtection parent to
+      10.0.10 instead was rejected in the sister repo for a reason that applies here too: it floors
+      Microsoft.Extensions.* and, through the EFCore adapter, Microsoft.EntityFrameworkCore at
+      10.0.10, forcing a family-wide servicing bump (NU1605 downgrade cascade) that deserves its own
+      reviewed change.
+    -->
+    <PackageVersion Include="System.Security.Cryptography.Xml" Version="10.0.10" />
     <PackageVersion Include="System.CommandLine" Version="2.0.5" />
     <PackageVersion Include="System.Data.SqlClient" Version="4.9.0" />
     <PackageVersion Include="System.IdentityModel.Tokens.Jwt" Version="8.11.0" />
@@ -122,6 +141,13 @@
     <PackageVersion Include="xunit.runner.visualstudio" Version="3.0.2" />
     <PackageVersion Include="xunit.v3" Version="1.1.0" />
     <PackageVersion Include="ZB.MOM.WW.Health" Version="0.1.0" />
+    <!--
+      LocalDb: embedded SQLite node-local cache + optional 2-node gRPC replication. The core
+      package floors the native SQLitePCLRaw.lib.e_sqlite3 at 2.1.12, so consumers inherit the
+      CVE-2025-6965-fixed native binary without needing the surgical pin below.
+    -->
+    <PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.3" />
+    <PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.3" />
     <PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.1.0" />
     <PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.1.0" />
     <PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />

NuGet.config

@@ -27,6 +27,8 @@
       <package pattern="ZB.MOM.WW.HistorianGateway.Client" />
       <package pattern="ZB.MOM.WW.Secrets" />
       <package pattern="ZB.MOM.WW.Secrets.*" />
+      <package pattern="ZB.MOM.WW.LocalDb" />
+      <package pattern="ZB.MOM.WW.LocalDb.*" />
     </packageSource>
   </packageSourceMapping>
 </configuration>

docker-dev/Dockerfile

@@ -3,7 +3,12 @@
 # Compose drives OTOPCUA_ROLES + Cluster:* env per container to differentiate them.
 # A separate `migrator` stage (below) applies EF migrations once on bring-up.
 
-FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
+# Build stage is PINNED to linux/amd64. Per-cluster mesh Phase 3 added an in-repo .proto compiled by
+# Grpc.Tools, whose bundled linux_arm64 `protoc` segfaults (exit 139) on Apple-Silicon Docker — so a
+# native-arm64 build stage cannot generate the gRPC stubs. `dotnet publish` here is framework-dependent
+# (portable IL + every RID's native assets), so the amd64-built output runs unchanged on the native
+# runtime image below; only this one-time build is emulated, runtime stays native/fast.
+FROM --platform=linux/amd64 mcr.microsoft.com/dotnet/sdk:10.0 AS build
 WORKDIR /src
 COPY . .
 RUN dotnet restore ZB.MOM.WW.OtOpcUa.slnx

docker-dev/docker-compose.yml

@@ -13,7 +13,9 @@
 #   central-1, central-2   OTOPCUA_ROLES=admin,driver — the ONLY UI + deploy
 #                          singleton, plus the MAIN cluster's OPC UA publishers.
 #                          Reachable at http://localhost:9200 (via Traefik).
-#                          central-1 is the Akka seed node; central-2 joins it.
+#                          Both are Akka seed nodes, each listing ITSELF first
+#                          (2026-07-22) so either can cold-start while the other
+#                          is down; the site nodes seed off central-1 only.
 #   site-a-1, site-a-2     OTOPCUA_ROLES=driver — driver-only members of the same
 #   site-b-1, site-b-2     mesh, scoped to SITE-A / SITE-B by ClusterId. They
 #                          serve no UI and authenticate no users; the central
@@ -136,7 +138,8 @@ services:
 
   # ── Central cluster (2-node fused admin+driver) ─────────────────────────────
   # The only UI + deploy singleton; also the MAIN cluster's OPC UA publishers.
-  # central-1 seeds the single Akka mesh that every other node joins.
+  # central-1 seeds the single Akka mesh that every other node joins; central-2 is a seed too
+  # (listing itself first), so it can form the mesh alone if central-1 is down.
 
   central-1: &otopcua-host
     build:
@@ -175,13 +178,39 @@ services:
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "central-1"
+      # Per-cluster mesh Phase 3 (config fetch-and-cache). Central SERVES the deployed-configuration
+      # artifact to FetchAndCache site nodes over a dedicated h2c gRPC listener (:4055), gated by the
+      # shared node key. Central itself is ALWAYS Direct — it IS the SQL source and must never fetch.
+      # ConfigServe:ApiKey must be byte-identical to the site nodes' ConfigSource:ApiKey (fail-closed).
+      # Binding :4055 activates the dedicated-h2c Kestrel takeover on central (it re-binds :9000 too).
+      ConfigServe__GrpcListenPort: "4055"
+      ConfigServe__ApiKey: "configserve-docker-dev-key"
+      ConfigSource__Mode: "Direct"
       # Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
-      # re-join the mesh via EITHER peer, not only central-1. With restart supervision this is
-      # the 2-node keep-oldest exit-and-rejoin recovery path.
+      # re-join the mesh via EITHER peer, not only central-1.
+      #
+      # ORDER IS LOAD-BEARING (2026-07-22): each node lists ITSELF first. Akka runs
+      # FirstSeedNodeProcess — the only path that can form a NEW cluster when no peer answers
+      # InitJoin — exclusively for seed-nodes[0]; every other node retries InitJoin forever. So a
+      # node listing its partner first cannot cold-start while that partner is down. Enforced at
+      # boot by AkkaClusterOptionsValidator; see docs/Redundancy.md.
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
       Cluster__Roles__0: "admin"
       Cluster__Roles__1: "driver"
+      # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
+      # the transport this repo has always used, and every node keeps its ClusterClient wiring built
+      # and its comm actor registered with the receptionist either way. Flip the whole rig with
+      # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
+      #
+      # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
+      # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
+      # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
+      # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
+      # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
+      MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
+      MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
+      MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
       Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
       Security__Jwt__Issuer: "otopcua-dev"
       Security__Jwt__Audience: "otopcua-dev"
@@ -224,8 +253,15 @@ services:
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # Consolidated pair-local config cache (ZB.MOM.WW.LocalDb). Every driver-role node
+      # (central is admin,driver) gets one, on a per-node named volume so the cache survives
+      # container recreates. The central pair leaves replication OFF (default) — only the
+      # site-a pair replicates, as the enablement demo; site-b is the default-OFF pin.
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4840:4840"
+    volumes:
+      - otopcua-localdb-central-1:/app/data
 
   central-2:
     <<: *otopcua-host
@@ -241,11 +277,31 @@ services:
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "central-2"
-      # Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1.
-      Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
-      Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
+      # Per-cluster mesh Phase 3 — serves the artifact to FetchAndCache site nodes (see central-1).
+      # Always Direct; ConfigServe:ApiKey must match the site nodes' ConfigSource:ApiKey.
+      ConfigServe__GrpcListenPort: "4055"
+      ConfigServe__ApiKey: "configserve-docker-dev-key"
+      ConfigSource__Mode: "Direct"
+      # Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST:
+      # central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1
+      # is down (it previously listed central-1 first and simply never came Up in that case).
+      Cluster__SeedNodes__0: "akka.tcp://otopcua@central-2:4053"
+      Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "admin"
       Cluster__Roles__1: "driver"
+      # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
+      # the transport this repo has always used, and every node keeps its ClusterClient wiring built
+      # and its comm actor registered with the receptionist either way. Flip the whole rig with
+      # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
+      #
+      # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
+      # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
+      # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
+      # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
+      # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
+      MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
+      MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
+      MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
       Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
       Security__Jwt__Issuer: "otopcua-dev"
       Security__Jwt__Audience: "otopcua-dev"
@@ -283,8 +339,12 @@ services:
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # Pair-local config cache; central pair leaves replication OFF (see central-1).
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4841:4840"
+    volumes:
+      - otopcua-localdb-central-2:/app/data
 
   # ── Site A cluster (2-node driver-only) ─────────────────────────────────────
   # Driver-only members of the single mesh, scoped to SITE-A by ClusterId. No UI,
@@ -303,8 +363,33 @@ services:
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "site-a-1"
+      # Per-cluster mesh Phase 3 dark switch. Default Direct (reads central SQL). Flip the site nodes to
+      # FetchAndCache with `OTOPCUA_CONFIG_MODE=FetchAndCache docker compose up`: they then fetch the
+      # deployed artifact from central over gRPC (endpoints below) and read ONLY the LocalDb cache — no
+      # central SQL config read. Central stays Direct, so this env var moves ONLY the four site nodes.
+      # ConfigSource:ApiKey must equal central's ConfigServe:ApiKey (the interceptor is fail-closed).
+      ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
+      ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
+      ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
+      ConfigSource__ApiKey: "configserve-docker-dev-key"
+      # Site nodes are deliberately NOT seeds — they join the central pair's mesh. The self-first
+      # seed rule is conditional for exactly this reason (it binds only when a node's own address
+      # is in its own seed list), so these configs are exempt rather than broken.
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
+      # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
+      # the transport this repo has always used, and every node keeps its ClusterClient wiring built
+      # and its comm actor registered with the receptionist either way. Flip the whole rig with
+      # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
+      #
+      # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
+      # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
+      # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
+      # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
+      # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
+      MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
+      MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
+      MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
       <<: *secrets-env
       # Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
       # mem_reservation are inherited from the *otopcua-host anchor.
@@ -313,8 +398,49 @@ services:
       # Resolved at runtime by GalaxyDriver.ResolveApiKey when a DriverInstance's
       # Gateway.ApiKeySecretRef = "env:GALAXY_MXGW_API_KEY".
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # Pair-local config cache + REPLICATION ON — the site-a pair is the enablement demo.
+      # site-a-1 is the initiator: it binds the h2c sync listener on 9001 AND dials the peer.
+      # Setting SyncListenPort makes Program.cs add a dedicated Http2-only Kestrel listener and
+      # re-bind the primary HTTP port (an explicit Listen* otherwise discards ASPNETCORE_URLS).
+      # Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
+      # buffer is a live, replicated table rather than an empty one — the live gate needs a real
+      # queue to watch converge, and needs to see that only the Primary drains it.
+      #
+      # There is no HistorianGateway on this rig, so the endpoint below is deliberately
+      # unresolvable: every drain attempt fails, which is exactly the historian-outage state the
+      # buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
+      # URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
+      # AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
+      #
+      # MaxAttempts is raised far above the production default of 10. With a permanently
+      # unreachable gateway the default would dead-letter the whole queue about five minutes into
+      # the outage, turning a buffering test into a dead-letter test.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
+      # Required whenever the section is consumed: the gateway client validates its own options at
+      # construction and throws on an empty key, so a keyless node crash-loops during Akka startup
+      # rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
+      # above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
+      # ServerHistorian__ApiKey from the environment.
+      ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
+      # Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
+      # direction, so an http endpoint with the default UseTls=true crash-loops the host.
+      ServerHistorian__UseTls: "false"
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
+      LocalDb__SyncListenPort: "9001"
+      LocalDb__Replication__PeerAddress: "http://site-a-2:9001"
+      # DEV-ONLY committed key (rig philosophy — like the SQL password + secrets KEK above). It
+      # MUST be byte-identical on both nodes: the interceptor is fail-closed, so any mismatch
+      # silently stops the pair converging. NEVER reuse this key outside this local dev rig.
+      LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
+      # Row-count batching against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so
+      # 16 × 171 KB ≈ 2.7 MB stays under the cap with headroom.
+      LocalDb__Replication__MaxBatchSize: "16"
     ports:
       - "4842:4840"
+    volumes:
+      - otopcua-localdb-site-a-1:/app/data
 
   site-a-2:
     <<: *otopcua-host
@@ -328,14 +454,68 @@ services:
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "site-a-2"
+      # Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache
+      # flips it to fetch the artifact from central over gRPC and read only the LocalDb cache.
+      ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
+      ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
+      ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
+      ConfigSource__ApiKey: "configserve-docker-dev-key"
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
+      # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
+      # the transport this repo has always used, and every node keeps its ClusterClient wiring built
+      # and its comm actor registered with the receptionist either way. Flip the whole rig with
+      # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
+      #
+      # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
+      # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
+      # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
+      # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
+      # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
+      MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
+      MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
+      MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
       <<: *secrets-env
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # site-a-2 is the passive half of the replicating pair: it binds the sync listener on 9001
+      # but does NOT dial (no PeerAddress). The replication stream is bidirectional, so a-1's
+      # single dial carries both directions. Same key + MaxBatchSize as a-1 (byte-identical key
+      # is mandatory — the interceptor fail-closes on a mismatch).
+      # Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
+      # buffer is a live, replicated table rather than an empty one — the live gate needs a real
+      # queue to watch converge, and needs to see that only the Primary drains it.
+      #
+      # There is no HistorianGateway on this rig, so the endpoint below is deliberately
+      # unresolvable: every drain attempt fails, which is exactly the historian-outage state the
+      # buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
+      # URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
+      # AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
+      #
+      # MaxAttempts is raised far above the production default of 10. With a permanently
+      # unreachable gateway the default would dead-letter the whole queue about five minutes into
+      # the outage, turning a buffering test into a dead-letter test.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
+      # Required whenever the section is consumed: the gateway client validates its own options at
+      # construction and throws on an empty key, so a keyless node crash-loops during Akka startup
+      # rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
+      # above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
+      # ServerHistorian__ApiKey from the environment.
+      ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
+      # Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
+      # direction, so an http endpoint with the default UseTls=true crash-loops the host.
+      ServerHistorian__UseTls: "false"
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
+      LocalDb__SyncListenPort: "9001"
+      LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
+      LocalDb__Replication__MaxBatchSize: "16"
     ports:
       - "4843:4840"
+    volumes:
+      - otopcua-localdb-site-a-2:/app/data
 
   # ── Site B cluster (2-node driver-only) ─────────────────────────────────────
 
@@ -351,14 +531,53 @@ services:
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "site-b-1"
+      # Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache
+      # flips it to fetch the artifact from central over gRPC and read only the LocalDb cache.
+      ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
+      ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
+      ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
+      ConfigSource__ApiKey: "configserve-docker-dev-key"
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
+      # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
+      # the transport this repo has always used, and every node keeps its ClusterClient wiring built
+      # and its comm actor registered with the receptionist either way. Flip the whole rig with
+      # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
+      #
+      # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
+      # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
+      # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
+      # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
+      # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
+      MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
+      MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
+      MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
       <<: *secrets-env
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # site-b is the default-OFF pin: it gets the pair-local cache but NO SyncListenPort and NO
+      # replication config, so no sync listener binds and ISyncStatus stays disconnected/Healthy.
+      # This is what the live gate's check 8 verifies — the cache works locally with replication off.
+      # Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
+      # must work as a plain node-local buffer here, with no sync listener and no peer traffic.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
+      # Required whenever the section is consumed: the gateway client validates its own options at
+      # construction and throws on an empty key, so a keyless node crash-loops during Akka startup
+      # rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
+      # above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
+      # ServerHistorian__ApiKey from the environment.
+      ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
+      # Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
+      # direction, so an http endpoint with the default UseTls=true crash-loops the host.
+      ServerHistorian__UseTls: "false"
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4844:4840"
+    volumes:
+      - otopcua-localdb-site-b-1:/app/data
 
   site-b-2:
     <<: *otopcua-host
@@ -372,14 +591,51 @@ services:
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "site-b-2"
+      # Per-cluster mesh Phase 3 dark switch (see site-a-1). Default Direct; OTOPCUA_CONFIG_MODE=FetchAndCache
+      # flips it to fetch the artifact from central over gRPC and read only the LocalDb cache.
+      ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"
+      ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
+      ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
+      ConfigSource__ApiKey: "configserve-docker-dev-key"
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
+      # Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
+      # the transport this repo has always used, and every node keeps its ClusterClient wiring built
+      # and its comm actor registered with the receptionist either way. Flip the whole rig with
+      # OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
+      #
+      # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
+      # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
+      # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
+      # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
+      # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
+      MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
+      MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
+      MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
       <<: *secrets-env
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # site-b default-OFF pin (see site-b-1): cache on, replication off.
+      # Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
+      # must work as a plain node-local buffer here, with no sync listener and no peer traffic.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
+      # Required whenever the section is consumed: the gateway client validates its own options at
+      # construction and throws on an empty key, so a keyless node crash-loops during Akka startup
+      # rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
+      # above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
+      # ServerHistorian__ApiKey from the environment.
+      ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
+      # Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
+      # direction, so an http endpoint with the default UseTls=true crash-loops the host.
+      ServerHistorian__UseTls: "false"
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4845:4840"
+    volumes:
+      - otopcua-localdb-site-b-2:/app/data
 
   traefik:
     image: traefik:v3.1
@@ -400,3 +656,13 @@ services:
 volumes:
   # SQL Server data dir — persists the OtOpcUa ConfigDb across container recreates.
   otopcua-mssql-data:
+  # Per-node pair-local config-cache (ZB.MOM.WW.LocalDb) files. One volume per node — the
+  # cache is node-local, not shared; it persists the cached deployment artifact (and, on the
+  # site-a pair, the replicated oplog state) across container recreates so a node can boot
+  # from cache after a restart even with central SQL down.
+  otopcua-localdb-central-1:
+  otopcua-localdb-central-2:
+  otopcua-localdb-site-a-1:
+  otopcua-localdb-site-a-2:
+  otopcua-localdb-site-b-1:
+  otopcua-localdb-site-b-2:

docker-dev/seed/seed-clusters.sql

@@ -64,13 +64,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ServerCluster WHERE ClusterId = 'SITE-B')
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-1:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('central-1:4053', 'MAIN', 'central-1', 4840, 8081, 'urn:OtOpcUa:central-1', 200, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('central-1:4053', 'MAIN', 'central-1', 4840, 8081, 4053, 'urn:OtOpcUa:central-1', 200, 1, 'docker-dev-seed');
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('central-2:4053', 'MAIN', 'central-2', 4840, 8081, 'urn:OtOpcUa:central-2', 150, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('central-2:4053', 'MAIN', 'central-2', 4840, 8081, 4053, 'urn:OtOpcUa:central-2', 150, 1, 'docker-dev-seed');
 
 ------------------------------------------------------------------------------
 -- ClusterNode — site A
@@ -78,13 +78,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053')
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-1:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('site-a-1:4053', 'SITE-A', 'site-a-1', 4840, 8081, 'urn:OtOpcUa:site-a-1', 200, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('site-a-1:4053', 'SITE-A', 'site-a-1', 4840, 8081, 4053, 'urn:OtOpcUa:site-a-1', 200, 1, 'docker-dev-seed');
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('site-a-2:4053', 'SITE-A', 'site-a-2', 4840, 8081, 'urn:OtOpcUa:site-a-2', 150, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('site-a-2:4053', 'SITE-A', 'site-a-2', 4840, 8081, 4053, 'urn:OtOpcUa:site-a-2', 150, 1, 'docker-dev-seed');
 
 ------------------------------------------------------------------------------
 -- ClusterNode — site B
@@ -92,13 +92,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053')
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-1:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('site-b-1:4053', 'SITE-B', 'site-b-1', 4840, 8081, 'urn:OtOpcUa:site-b-1', 200, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('site-b-1:4053', 'SITE-B', 'site-b-1', 4840, 8081, 4053, 'urn:OtOpcUa:site-b-1', 200, 1, 'docker-dev-seed');
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-2:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 4053, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed');
 
 ------------------------------------------------------------------------------
 -- Galaxy MxAccess gateway — INTENTIONALLY NOT SEEDED (retired 2026-06-15)

docs/AlarmHistorian.md

@@ -42,7 +42,7 @@ unless noted.
 - **`NullAlarmHistorianSink`** — the no-op default for tests and deployments
   that don't historize alarms. It is the default DI binding (registered in the
   Runtime's `AddOtOpcUaRuntime`); production overrides it with
-  `SqliteStoreAndForwardSink`.
+  `LocalDbStoreAndForwardSink`.
 - **`AlarmHistorianEvent`**
   ([`AlarmHistorianEvent.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmHistorianEvent.cs))
   — the source-agnostic event record: `AlarmId`, `EquipmentPath` (UNS path,
@@ -61,42 +61,84 @@ unless noted.
 - **`HistorianSinkStatus`** — diagnostic snapshot surfaced to the AdminUI and
   `/healthz`: `QueueDepth`, `DeadLetterDepth`, `LastDrainUtc`, `LastSuccessUtc`,
   `LastError`, `DrainState`, and `EvictedCount`.
-- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff`.
+- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff` /
+  **`NotPrimary`** (ticking, but leaving the replicated queue for the node that
+  holds the Primary role).
 
 ---
 
-## SqliteStoreAndForwardSink
+## LocalDbStoreAndForwardSink
 
-[`SqliteStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs)
-is the production `IAlarmHistorianSink`. Construction takes a SQLite database
-path, an `IAlarmHistorianWriter`, a logger, and optional `batchSize` (default
-100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30 days),
-and a test clock.
+[`LocalDbStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/LocalDbStoreAndForwardSink.cs)
+is the production `IAlarmHistorianSink`. Construction takes the node's
+`ILocalDb`, an `IAlarmHistorianWriter`, a logger, and optional `batchSize`
+(default 100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30
+days), a test clock, and a **`drainGate`**.
+
+### The drain gate
+
+`drainGate` is a `Func<bool>` consulted at the top of every tick; `false` skips
+the tick entirely, leaving the queue untouched and reporting
+`DrainState = NotPrimary`.
+
+It exists because the queue **replicates**. The Secondary holds a full copy of
+the Primary's rows, and an ungated drain there would re-deliver every event,
+continuously — not just at a failover. Runtime supplies the gate from
+`IRedundancyRoleView`, a singleton `DriverHostActor` publishes its
+`PrimaryGatePolicy` verdict to on every redundancy snapshot, so the drain agrees
+with the inbound-write and native-ack gates by construction.
+
+Two postures are deliberate:
+
+- **Unset (the default) means drain.** So does an unpublished view. A deployment
+  that runs no redundancy never publishes a role, and defaulting closed would
+  silently stop its alarm history forever.
+- **A gate that throws means "not now", never permission.** Draining on a gate
+  fault would put a pair into dual delivery exactly when its redundancy signal is
+  sick.
 
 ### Queue table
 
-The sink owns one SQLite table (created on construction, WAL journal mode):
+The sink owns one table in the node's consolidated LocalDb — created and
+registered for replication by `LocalDbSetup.OnReady`, before any consumer can
+resolve the sink:
 
 ```sql
-CREATE TABLE Queue (
-    RowId          INTEGER PRIMARY KEY AUTOINCREMENT,
-    AlarmId        TEXT    NOT NULL,
-    EnqueuedUtc    TEXT    NOT NULL,
-    PayloadJson    TEXT    NOT NULL,   -- JSON-serialized AlarmHistorianEvent
-    AttemptCount   INTEGER NOT NULL DEFAULT 0,
-    LastAttemptUtc TEXT    NULL,
-    LastError      TEXT    NULL,
-    DeadLettered   INTEGER NOT NULL DEFAULT 0
+CREATE TABLE alarm_sf_events (
+    id               TEXT    NOT NULL PRIMARY KEY,   -- SHA-256 of payload_json
+    alarm_id         TEXT    NOT NULL,
+    enqueued_at_utc  TEXT    NOT NULL,
+    payload_json     TEXT    NOT NULL,   -- JSON-serialized AlarmHistorianEvent
+    attempt_count    INTEGER NOT NULL DEFAULT 0,
+    last_attempt_utc TEXT    NULL,
+    last_error       TEXT    NULL,
+    dead_lettered    INTEGER NOT NULL DEFAULT 0
 );
-CREATE INDEX IX_Queue_Drain ON Queue (DeadLettered, RowId);
+CREATE INDEX ix_alarm_sf_events_drain
+    ON alarm_sf_events (dead_lettered, enqueued_at_utc, id);
 ```
 
-`EnqueueAsync` does a single `INSERT` on the hot path. To avoid a
-`SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory non-dead-lettered
-row counter (seeded at startup, kept current by every mutation, and re-synced
-from storage every 10,000 enqueues to defend against drift). SQLite writer
-contention is handled via `PRAGMA busy_timeout=5000` + WAL so an enqueue/drain
-collision waits out the file lock instead of failing fast.
+**The primary key is a hash of the payload, not an autoincrement rowid.** Two
+reasons. Replication converges last-writer-wins *per primary key*, so two nodes
+independently allocating rowid 7 to different alarms would silently overwrite one
+another. And an equal-payload key makes the same event arriving twice collapse
+into one row — which happens for real, because `HistorianAdapterActor`
+default-writes while its redundancy role is unknown and both nodes of a pair
+therefore accept the same fanned transition in every boot window. Two genuinely
+distinct events cannot collide: `AlarmHistorianEvent` carries a full-precision
+timestamp alongside the alarm id, kind, message and user.
+
+Ordering follows from that: the drain reads in `enqueued_at_utc` order (with `id`
+as a tiebreak so the ordering is total), because a hashed key carries no
+insertion sequence.
+
+`EnqueueAsync` does a single `INSERT … ON CONFLICT DO NOTHING` on the hot path.
+To avoid a `SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory
+non-dead-lettered row counter (seeded at startup, kept current by every mutation,
+and re-synced from storage every 10,000 enqueues to defend against drift — now
+doubly warranted, since replication applies the peer's rows straight into the
+table where no local code path can observe them). Pragmas and connection
+lifetime belong to LocalDb; the sink no longer manages either.
 
 ### Drain worker
 
@@ -104,13 +146,17 @@ collision waits out the file lock instead of failing fast.
 `System.Threading.Timer`** (not started automatically — tests drive
 `DrainOnceAsync` deterministically). Each tick:
 
+0. Consults the drain gate. A closed gate returns immediately, leaving the queue
+   untouched and `DrainState = NotPrimary`.
 1. Purges aged dead-lettered rows past the retention window.
-2. Reads up to `batchSize` non-dead-lettered rows in `RowId` order.
+2. Reads up to `batchSize` non-dead-lettered rows in `enqueued_at_utc, id` order.
 3. Rows with un-deserializable payloads are dead-lettered immediately (by their
-   own `RowId`) so they can't stall the queue head.
+   own `id`) so they can't stall the queue head.
 4. The remaining batch is handed to `IAlarmHistorianWriter.WriteBatchAsync`, and
-   each outcome is applied in one transaction: `Ack` deletes the row,
-   `PermanentFail` flips its `DeadLettered` flag, `RetryPlease` bumps its attempt
+   each outcome is applied in one transaction: `Ack` deletes the row (the delete
+   replicates as a tombstone, so the peer drops its copy and a promoted standby
+   cannot re-send it), `PermanentFail` flips its `dead_lettered` flag,
+   `RetryPlease` bumps its attempt
    count and leaves it queued. A row whose `AttemptCount` has reached the configured
    **`MaxAttempts`** cap (default 10) is dead-lettered automatically on the next drain
    tick rather than retried — this breaks infinite retry loops for poison events whose
@@ -131,7 +177,7 @@ an unobserved task exception.
 
 **The durability guarantee is bounded by `capacity` (default 1,000,000 rows).**
 When the non-dead-lettered queue reaches capacity, `EnqueueAsync` evicts the
-oldest non-dead-lettered rows (oldest `RowId` first) to make room, logs a WARN,
+oldest non-dead-lettered rows (oldest `enqueued_at_utc` first) to make room, logs a WARN,
 and increments `HistorianSinkStatus.EvictedCount`. Under a sustained historian
 outage, accepted alarm events can therefore be dropped before delivery. A
 non-zero `EvictedCount` is a data-loss signal that requires operator attention —
@@ -140,7 +186,7 @@ it surfaces silent loss without log scraping.
 ### Dead-letter + operator recovery
 
 `PermanentFail` and corrupt-payload rows are retained in-place with
-`DeadLettered = 1` for the retention window (default 30 days) so operators can
+`dead_lettered = 1` for the retention window (default 30 days) so operators can
 inspect them before the sweeper purges them. `RetryDeadLettered()` is the
 operator action (from the AdminUI) that clears the dead-letter flag and attempt
 count on every dead-lettered row, returning them to the regular queue with a
@@ -173,17 +219,28 @@ the `alerts` topic (future).
 The real sink is opt-in via the `AlarmHistorian` section of `appsettings.json`.
 When `Enabled` is `false` (the default), `AddAlarmHistorian` registers
 `NullAlarmHistorianSink` and the feature is dormant. When `Enabled` is `true`,
-`AddAlarmHistorian` constructs `SqliteStoreAndForwardSink` and registers
+`AddAlarmHistorian` constructs `LocalDbStoreAndForwardSink` and registers
 `GatewayAlarmHistorianWriter` as the `IAlarmHistorianWriter`. This section carries
-**only** the `Enabled` gate + the SQLite store-and-forward knobs — the downstream
+**only** the `Enabled` gate + the store-and-forward knobs — the downstream
 gateway connection (endpoint / key / TLS) is sourced from the `ServerHistorian`
-section (see [Historian.md](Historian.md)).
+section (see [Historian.md](Historian.md)), and the queue's storage location is the
+node's consolidated `LocalDb:Path` database (see
+[Configuration.md](Configuration.md)).
+
+> **Where the queue lives (changed).** The buffer is a table — `alarm_sf_events` — in the node's
+> consolidated LocalDb, not a standalone `alarm-historian.db`. That is what lets it **replicate to
+> the redundant pair peer**, so undelivered alarm history survives losing the node holding it.
+> Two consequences follow, both covered in the
+> [pair-replication runbook](operations/2026-07-20-localdb-pair-replication.md):
+> **only the Primary drains** (the Secondary holds a full replica and would otherwise re-send
+> everything), and **delivery is at-least-once across a failover** by design. On first boot after
+> the upgrade, `AlarmSfLegacyMigrator` copies any existing `alarm-historian.db` across and renames
+> it `.migrated`.
 
 ```json
 {
   "AlarmHistorian": {
     "Enabled": true,
-    "DatabasePath": "C:\\ProgramData\\OtOpcUa\\alarmhistorian.db",
     "BatchSize": 100,
     "DrainIntervalSeconds": 5,
     "Capacity": 1000000,
@@ -194,8 +251,7 @@ section (see [Historian.md](Historian.md)).
 
 | Key | Type | Default | Description |
 |---|---|---|---|
-| `Enabled` | bool | `false` | Enable the SQLite store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. |
-| `DatabasePath` | string | `alarm-historian.db` | Path to the SQLite queue file. Created on first use (WAL mode). Set an **absolute** path in production. |
+| `Enabled` | bool | `false` | Enable the durable store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. Requires LocalDb to be registered — an enabled sink with no `ILocalDb` fails at resolution rather than silently discarding events. |
 | `BatchSize` | int | `100` | Max rows per drain cycle handed to `IAlarmHistorianWriter.WriteBatchAsync`. |
 | `DrainIntervalSeconds` | int | `5` | Seconds between drain-worker ticks. |
 | `Capacity` | long | `1000000` | Max queued rows before the sink evicts the oldest (data-loss signal via `EvictedCount`). |
@@ -207,6 +263,15 @@ section (see [Historian.md](Historian.md)).
 > `RuntimeDb:EventReadsEnabled=true`. The old Wonderware connection keys (`SharedSecret` /
 > `AlarmHistorian:Host`/`Port`/`UseTls`/`ServerCertThumbprint`) were pruned.
 
+> **`DatabasePath` was removed.** The queue no longer owns a file. The key is still *read*, by
+> `AlarmSfLegacyMigrator`, purely to locate a pre-consolidation `alarm-historian.db` to migrate —
+> so leave it in place through the upgrade, then delete it once every node has an
+> `alarm-historian.db.migrated` sidecar. A leftover value configures nothing.
+
+> **`DrainState` gained `NotPrimary`.** A gated-off Secondary reports it instead of `Idle`, so a
+> rising queue depth there is distinguishable from a stalled drain. Both nodes reporting it means
+> nobody is draining — see the runbook.
+
 > Dev and docker-dev deployments leave `Enabled` unset (defaults to `false`) so alarm transitions historize to nowhere unless a HistorianGateway is configured.
 
 ---
@@ -221,3 +286,5 @@ section (see [Historian.md](Historian.md)).
 - [ScriptedAlarms.md](ScriptedAlarms.md) — the scripted-alarm engine that emits
   most events into this sink.
 - [ServiceHosting.md](ServiceHosting.md) — the external HistorianGateway backend.
+- [operations/2026-07-20-localdb-pair-replication.md](operations/2026-07-20-localdb-pair-replication.md)
+  — where the queue lives, the Primary-only drain, and the one-time legacy migration.

docs/AlarmTracking.md

@@ -365,12 +365,16 @@ AB CIP ALMD) route to AVEVA Historian via the HistorianGateway:
 - `IAlarmHistorianSink` is the DI-registered intake contract. The
   default binding is `NullAlarmHistorianSink` (registered in
   `ServiceCollectionExtensions.AddOtOpcUaRuntime`). Production
-  deployments override it with `SqliteStoreAndForwardSink` wrapping
+  deployments override it with `LocalDbStoreAndForwardSink` wrapping
   `GatewayAlarmHistorianWriter` (the HistorianGateway `SendEvent` path)
   — see [ServiceHosting.md](ServiceHosting.md) for the HistorianGateway setup.
-- `SqliteStoreAndForwardSink` queues each transition to a local
-  SQLite database and drains in the background via an
-  `IAlarmHistorianWriter`. **The durability guarantee is bounded**: the
+- `LocalDbStoreAndForwardSink` queues each transition into the node's
+  consolidated LocalDb — where it **replicates to the redundant pair peer**,
+  so undelivered alarm history survives losing the node holding it — and
+  drains in the background via an `IAlarmHistorianWriter`. **Only the node
+  holding the Primary role drains**, since the peer holds a full replica;
+  delivery is consequently at-least-once across a failover, by design. See
+  [AlarmHistorian.md](AlarmHistorian.md). **The durability guarantee is bounded**: the
   queue capacity defaults to 1,000,000 rows; under a sustained
   historian outage, older non-dead-lettered rows are evicted (oldest
   first) to make room for new events. The `HistorianSinkStatus.EvictedCount`
@@ -379,8 +383,9 @@ AB CIP ALMD) route to AVEVA Historian via the HistorianGateway:
   capacity, and dead-letter retention are tunable via the `AlarmHistorian`
   config section (`DrainIntervalSeconds`, `Capacity`,
   `DeadLetterRetentionDays`); `AlarmHistorianOptions.Validate()` logs a
-  startup warning for an empty `SharedSecret`, a relative `DatabasePath`,
-  or a non-positive knob.
+  startup warning for a non-positive knob. (It no longer warns about
+  `SharedSecret` or `DatabasePath` — both keys are gone: the connection
+  comes from `ServerHistorian`, and the queue lives in the LocalDb.)
 - `HistorianAdapterActor`
   (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs`)
   subscribes to the cluster `alerts` DPS topic, translates each

docs/Configuration.md

@@ -101,13 +101,67 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
 |---|---|---|---|
 | `SystemName` | string | `otopcua` | Akka actor-system name. |
 | `Hostname` | string | `0.0.0.0` | Bind hostname. |
-| `Port` | int | `4053` | Cluster transport port. |
-| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. |
-| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. |
+| `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. |
+| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. |
+| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). |
 | `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`. |
 
 > The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.
 
+> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
+
+### `MeshTransport` (central↔node command transport)
+
+- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split.
+- **Options class:** `MeshTransportOptions` (`SectionName = "MeshTransport"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`. Bound and validated by `AddOtOpcUaCluster(config)`; `MeshTransportOptionsValidator` runs at `ValidateOnStart`.
+
+| Key | Type | Default | Meaning |
+|---|---|---|---|
+| `Mode` | string | `Dps` | `Dps` = DistributedPubSub (the transport this repo has always used). `ClusterClient` = the mesh boundary. Any other value **refuses to start**. |
+| `CentralContactPoints` | string[] | `[]` | Node-side only: central node **addresses** (`akka.tcp://<system>@<host>:<port>`). Required under `ClusterClient`, ignored under `Dps`. |
+| `ContactRefreshSeconds` | int | `60` | Central-side only: how often central rebuilds its ClusterClient contact set from the `ClusterNode` rows. |
+
+**Addresses only — no actor-path suffix.** `/system/receptionist` is appended at construction time; a contact carrying `/user/` or `/system/` is rejected at startup, because a doubled suffix produces a path that resolves to nothing and drops every command in silence.
+
+**It is a dark switch, and both sides are always wired.** The comm actors are spawned and registered with the `ClusterClientReceptionist` on every node in *both* modes, so flipping `Mode` is a config change rather than a redeploy. Under `Dps` the registration is simply idle.
+
+**Central's contact set comes from the enabled, non-maintenance `ClusterNode` rows** — the same source Phase 1 gave the deploy ack set. A node in `MaintenanceMode` is not dialled, but note that **the contact set does not scope delivery**: a receptionist serves its whole cluster, so `SendToAll` still reaches every registered node exactly as today's DPS broadcast does. The filter is about which receptionists are worth dialling, not about who receives.
+
+**Buffering is off by design** (`akka.cluster.client.buffer-size = 0`, set in the shipped `akka.conf`). If no receptionist is known, the command is dropped rather than queued. Akka's default of 1000 would replay a `DispatchDeployment` on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not "fix" it back to quiet a flaky test.
+
+> **What this does *not* buy you.** It does not guarantee a node never applies a deployment the DB records as `TimedOut`. A node that missed a dispatch keeps the orphan `Applying` row the coordinator pre-created, and `DriverHostActor.Bootstrap()` replays that row on the node's next restart — live-observed applying a deployment 44 s after it was sealed `TimedOut` (see the [Phase 2 live gate](plans/2026-07-22-mesh-phase2-live-gate.md), finding B). The boot replay is the better-behaved of the two — deliberate, Warning-logged, and bounded to a restart — but it is still a replay. `buffer-size = 0` removes the *silent, timer-driven* one only.
+
+On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`.
+
+### `ConfigSource` / `ConfigServe` (config fetch-and-cache)
+
+- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 3**. `ConfigSource:Mode` selects where a **driver** node reads its deployed configuration: `Direct` (read `Deployment.ArtifactBlob` from central SQL, today's behaviour) or `FetchAndCache` (fetch the artifact bytes from central over a gRPC stream, verify SHA-256, cache in LocalDb, and read **only** the cache — no central-SQL config read). `ConfigServe` is the **central** (admin-role) serve side that a `FetchAndCache` node dials.
+- **Options classes:** `ConfigSourceOptions` (`SectionName = "ConfigSource"`) + `ConfigServeOptions` (`SectionName = "ConfigServe"`), both in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`. Bound by `AddOtOpcUaCluster(config)`; `ConfigSourceOptionsValidator` runs at `ValidateOnStart`.
+
+**`ConfigSource` (driver-side fetch):**
+
+| Key | Type | Default | Meaning |
+|---|---|---|---|
+| `Mode` | string | `Direct` | `Direct` = read central SQL (unchanged). `FetchAndCache` = fetch from central over gRPC, read the LocalDb cache. Any other value **refuses to start**. |
+| `CentralFetchEndpoints` | string[] | `[]` | Central artifact-gRPC base addresses, e.g. `http://central-1:4055` (h2c ⇒ the `http` scheme). Tried in order for failover. **Required** under `FetchAndCache`. |
+| `ApiKey` | string | `""` | Shared bearer key; must equal central's `ConfigServe:ApiKey`. **Supply via env `ConfigSource__ApiKey` — never commit.** Required under `FetchAndCache`. |
+| `FetchTimeoutSeconds` | int | `30` | Per-fetch deadline. Must be positive under `FetchAndCache`. |
+
+**`ConfigServe` (central serve):**
+
+| Key | Type | Default | Meaning |
+|---|---|---|---|
+| `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the artifact gRPC service. `0` = disabled (nothing bound). Must differ from the main HTTP port and from `LocalDb:SyncListenPort` (h2c cannot share a cleartext HTTP/1 port). Mapped only on admin-role nodes. |
+| `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `ConfigServe__ApiKey` — never commit.** |
+
+**It is a dark switch; the serve side is always wired.** Central maps the `DeploymentArtifactService` behind the `ConfigServeAuthInterceptor` (path-scoped, `FixedTimeEquals`, fail-closed) whenever `ConfigServe:GrpcListenPort > 0`, in both modes — so flipping a node to `FetchAndCache` is a config change, not a redeploy. A `FetchAndCache` node that cannot fetch (all endpoints down, `NotFound`, SHA mismatch) **fails the apply and keeps last-known-good** — a null fetch is "no answer," never an empty configuration (the [#485](plans/2026-07-15-v3-batch4-address-space-plan.md) contract). At boot it restores served state from the LocalDb pointer with no SQL read; an empty or unreadable cache lands Steady-with-no-revision (the first dispatch fetches), never Stale.
+
+**The listener is dedicated h2c and coexists with the LocalDb-sync listener.** A fused admin+driver node with **both** `ConfigServe:GrpcListenPort` and `LocalDb:SyncListenPort` set re-binds its existing HTTP surface exactly once and adds both dedicated HTTP/2-only ports on top — see the same Kestrel-takeover caveat under [`LocalDb`](#localdb-pair-local-config-cache--driver-role-only).
+
+**`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the deployment's revision hash as the lowercase-hex SHA-256 of the exact artifact bytes, so the node verifies a fetch against the `RevisionHash` the dispatch already carries — no hash rides in the proto. If a future change makes the revision hash anything else, every fetch fails verification.
+
+On the docker-dev rig the central pair carries `ConfigServe` (port `4055`, dev key) and stays `Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`. Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up` — central keeps SQL and models the target topology.
+
 ### `ConnectionStrings` → `ConfigDb`
 
 - **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally.
@@ -136,8 +190,9 @@ The historian backend is the external **`ZB.MOM.WW.HistorianGateway`** sidecar,
 `ZB.MOM.WW.HistorianGateway.Client` gRPC package (the retired Wonderware TCP sidecar is documented at
 [`docs/drivers/Historian.Wonderware.md`](drivers/Historian.Wonderware.md)). The OtOpcUa host reads three
 appsettings sections — `ServerHistorian` (read path + gateway connection), `ContinuousHistorization`
-(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (SQLite store-and-forward
-alarm sink draining to `SendEvent`). The gateway connection (endpoint / key / TLS) lives **only** in
+(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (the store-and-forward
+alarm sink draining to `SendEvent`; its buffer lives in the node's `LocalDb:Path` database, not a
+file of its own). The gateway connection (endpoint / key / TLS) lives **only** in
 `ServerHistorian`; the other two sections source it from there.
 
 The gateway API key is supplied via the environment variable **`ServerHistorian__ApiKey`** — never committed
@@ -146,6 +201,28 @@ key must carry the scopes `historian:read`, `historian:write`, `historian:tags:w
 [`docs/Historian.md`](Historian.md) for the full key reference, the migration note (old Wonderware keys →
 gateway keys), and the deployment prerequisites.
 
+### `LocalDb` (pair-local config cache — driver role only)
+
+Driver-role nodes keep a consolidated **`ZB.MOM.WW.LocalDb`** SQLite database that caches the deployed
+configuration artifact (chunked) so the node can **boot from cache when central SQL is unreachable**. The
+section is bound by `AddOtOpcUaLocalDb` (`Host/Configuration/LocalDbRegistration.cs`) inside the `hasDriver`
+branch — **admin-only nodes never register it, and never require `LocalDb:Path`**. Storage is unconditional;
+replication stays inert until the sync port + peer are set.
+
+| Key | Default | Meaning |
+|---|---|---|
+| `LocalDb:Path` | `./data/otopcua-localdb.db` | SQLite file path. `ValidateOnStart`-required once `AddZbLocalDb` runs (driver nodes only). In containers put it on a durable volume. |
+| `LocalDb:SyncListenPort` | `0` | `0` = replication off (no sync listener). A non-zero port binds a dedicated **h2c** sync listener; set the **same** port on both pair nodes. Setting it makes the Host add an explicit Kestrel `Listen*` and re-bind the primary HTTP port (an explicit `Listen*` otherwise discards `ASPNETCORE_URLS`). |
+| `LocalDb:Replication:PeerAddress` | `""` | The peer this node **dials** (`http://<peer>:<port>`). Set on the initiator only; leave empty on the passive node. The stream is bidirectional. |
+| `LocalDb:Replication:ApiKey` | `""` | Bearer token for the sync stream. **Fail-closed:** no key ⇒ the passive endpoint refuses everything; a mismatch ⇒ the pair silently stops converging. **Must be byte-identical on both nodes.** Supply via `${secret:...}` / env — never a cleartext literal in production. |
+| `LocalDb:Replication:MaxBatchSize` | (library default) | Rows per replication batch — **row-count-only** against gRPC's 4 MB cap. `16` for the artifact cache (chunk rows ≈171 KB, so 16 × 171 KB ≈ 2.7 MB). |
+
+Replication is **default-OFF** across the fleet; it is enabled per-pair as an opt-in. See
+[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
+for the enablement runbook (ApiKey handling, stop/start-together rule, tombstone-retention window, and the
+never-`sqlite3`-a-live-WAL-DB inspection rules) and the "pair-local config cache" section of
+[`docs/Redundancy.md`](Redundancy.md) for what boot-from-cache does and does not cover.
+
 ---
 
 ## Environment variables

docs/README.md

@@ -30,7 +30,7 @@ The project was originally called **LmxOpcUa** (a single-driver Galaxy/MXAccess
 | [Subscriptions.md](v1/Subscriptions.md) | Monitored items → `ISubscribable` + per-driver subscription refcount (v1 archive) |
 | [AlarmTracking.md](AlarmTracking.md) | `IAlarmSource` + `AlarmSurfaceInvoker` + OPC UA alarm conditions — native Galaxy alarms end-to-end (live) |
 | [AlarmTracking.md](v1/AlarmTracking.md) | Original alarm-tracking write-up (v1 archive) |
-| [AlarmHistorian.md](AlarmHistorian.md) | `Core.AlarmHistorian` store-and-forward SQLite sink — `SqliteStoreAndForwardSink`, `IAlarmHistorianWriter`, dead-letter/retry/eviction |
+| [AlarmHistorian.md](AlarmHistorian.md) | `Core.AlarmHistorian` store-and-forward sink — `LocalDbStoreAndForwardSink`, the replicated `alarm_sf_events` buffer + Primary-only drain, `IAlarmHistorianWriter`, dead-letter/retry/eviction |
 | [DataTypeMapping.md](v1/DataTypeMapping.md) | Per-driver `DriverAttributeInfo` → OPC UA variable types (v1 archive — live mapping is in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DataTypeMap.cs`) |
 | [IncrementalSync.md](IncrementalSync.md) | Address-space rebuild on redeploy + `sp_ComputeGenerationDiff` |
 | [HistoricalDataAccess.md](v1/HistoricalDataAccess.md) | `IHistoryProvider` as a per-driver optional capability (v1 archive) |

docs/Redundancy.md

@@ -6,7 +6,7 @@ OtOpcUa supports OPC UA **non-transparent** warm/hot redundancy. Two or more `Ot
 
 > **Discovery surface.** The `ServerArray` path on the `Server` object is what each node populates with self + peer `ApplicationUri`s — see `OpcUaApplicationHost.PopulateServerArray` and the per-node `PeerApplicationUris` option below. The redundancy-object-type `ServerUriArray` proper (a child of `Server.ServerRedundancy`) remains deferred pending an SDK object-type upgrade; clients should read `Server.ServerArray` for peer discovery today.
 
-> **v2 change.** v1's operator-managed `ClusterNode.RedundancyRole` column + `RedundancyCoordinator` / `ApplyLeaseRegistry` / `PeerHttpProbeLoop` are gone. Primary/secondary is now derived from **Akka cluster role-leader** for the `driver` role. The operator no longer writes a role into the DB; cluster topology (specifically the `driver` role-leader) drives ServiceLevel automatically.
+> **v2 change.** v1's operator-managed `ClusterNode.RedundancyRole` column + `RedundancyCoordinator` / `ApplyLeaseRegistry` / `PeerHttpProbeLoop` are gone. Primary/secondary is now derived from **Akka cluster membership** — the oldest Up member carrying the `driver` role. The operator no longer writes a role into the DB; cluster topology drives ServiceLevel automatically.
 
 The runtime pieces live in:
 
@@ -19,7 +19,7 @@ The runtime pieces live in:
 | `DbHealthProbeActor` | `OtOpcUa.Runtime.Health` | Per-node; runs `SELECT 1` against ConfigDb every 5s. Read by the health endpoint AND by `OpcUaPublishActor` (the `DbReachable` ServiceLevel input). |
 | `PeerProbeSupervisor` | `OtOpcUa.Runtime.Health` | Per-node; subscribes to the `redundancy-state` topic and maintains one `PeerOpcUaProbeActor` child per OTHER driver-role peer (spawn on join, stop on departure), so every node is continuously probed by its peers. |
 | `PeerOpcUaProbeActor` | `OtOpcUa.Runtime.Health` | Spawned by `PeerProbeSupervisor`; pings a peer `opc.tcp://peer:4840` with a TCP connect (2s timeout) and publishes `OpcUaProbeResult` on the `redundancy-state` topic. A full secure-channel Hello handshake is a possible future upgrade; the TCP connect is the current real probe. |
-| `ClusterRoleInfo` | `OtOpcUa.Cluster` | Live view of cluster membership + role-leader; exposes `IClusterRoleInfo` to the rest of the host. |
+| `ClusterRoleInfo` | `OtOpcUa.Cluster` | Live view of cluster membership + role-leader; exposes `IClusterRoleInfo` to the rest of the host. (Akka's role-leader is *not* what elects the redundancy Primary — see below.) |
 
 ## ServiceLevel tiers
 
@@ -38,7 +38,7 @@ The four inputs are sourced locally per driver node:
 | `DbReachable` | Local `DbHealthProbeActor` — `OpcUaPublishActor` Asks it on each `HealthTick`; an Ask timeout is treated as `Reachable=false`. |
 | `OpcUaProbeOk` | Result of a peer probing THIS node's OPC UA endpoint: `PeerProbeSupervisor` spawns one `PeerOpcUaProbeActor` per OTHER driver-role peer; each probe publishes `OpcUaProbeResult(probed-node, ok)` on the `redundancy-state` topic; the publish actor consumes only results whose target is itself. Freshness-debounced: absent or stale (>30 s) → `true` (benefit of the doubt — single-node clusters and a departed peer never demote); only an actively-observed RECENT `false` demotes. |
 | `Stale` (derived) | `!DbReachable \|\| (now − lastDbHealth.AsOfUtc) > 30 s \|\| (now − snapshotEntry.AsOfUtc) > 30 s`. |
-| `IsDriverRoleLeader` | The local node's entry in the `RedundancyStateChanged` snapshot from `RedundancyStateActor`. |
+| `IsDriverPrimary` | The local node's entry in the `RedundancyStateChanged` snapshot from `RedundancyStateActor`. |
 
 The resulting truth table (all tiers are now reachable at runtime):
 
@@ -47,8 +47,8 @@ The resulting truth table (all tiers are now reachable at runtime):
 | Down / Detached | 0 | Member status is not `Up` or `Joining` (leaving, removed, exiting), OR node has no `driver` role (Detached). Published immediately — a starting or detached node never leaves the SDK default 255. |
 | Critically degraded | 100 | ConfigDb unreachable AND data is stale. |
 | Stale | 200 | Data stale but ConfigDb reachable. |
-| Healthy follower | 240 | DB reachable + OPC UA probe ok + not stale + not role-leader. |
-| Healthy leader | 250 | Same as healthy follower + this node is the `driver` role-leader (+10 bonus). |
+| Healthy follower | 240 | DB reachable + OPC UA probe ok + not stale + not the driver Primary. |
+| Healthy primary | 250 | Same as healthy follower + this node is the driver Primary (+10 bonus). |
 
 > **Secondary 100 → 240 (behavior change).** Previously a healthy Secondary
 > published 100 (coarse role-only mapping). It now publishes **240** — both
@@ -66,9 +66,38 @@ ServiceLevel (even 0) is always published so no node lingers at the SDK default
 255.
 
 Roles come from `RedundancyStateActor.BuildSnapshot`: a node with the `driver`
-role is `Primary` when it holds the `driver` role-leader lease, otherwise
+role is `Primary` when it is the **oldest Up member carrying the `driver` role**, otherwise
 `Secondary`; a node without the `driver` role is `Detached`.
 
+**Oldest, not role leader (changed 2026-07-21).** The election previously used
+`ClusterState.RoleLeader("driver")` — the *lowest-addressed* Up member with the role. Akka offers
+both, and they name different nodes: role leader is ordered by address (host, then port), oldest by
+up-number. They agree on a freshly-formed cluster, and diverge after any restart — a restarted node
+re-joins as the *youngest* while keeping its address, so if it holds the lower address it becomes
+role leader. Meanwhile `ClusterSingletonManager` places singletons on the **oldest**, so the old
+derivation could name a Primary that was not hosting the singletons, enabling the Primary-gated data
+plane (writes, alarm acks, the alerts emit, the alarm-history drain) on the wrong node. Pinned by
+`RedundancyPrimaryElectionTests`, which forms a real two-node cluster where the oldest node holds the
+*higher* address so the two orderings genuinely disagree. `NodeRedundancyState.IsRoleLeaderForDriver`
+was renamed `IsDriverPrimary` to stop the name asserting the old derivation.
+
+> **KNOWN LIMITATION — the election is per *Akka* cluster, not per application `Cluster`.**
+> The election yields exactly **one** Primary across the whole Akka cluster. A fleet that
+> runs several application clusters (each with its own redundant pair) inside one Akka cluster —
+> which is what `docker-dev/docker-compose.yml` models, with MAIN, SITE-A and SITE-B — therefore has
+> one Primary among *all* its driver nodes, not one per pair. Every consumer of the role inherits
+> this: `ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate below.
+>
+> The unit of redundancy everywhere else in the product is the application `Cluster` (`ClusterId`) —
+> it owns drivers, devices and `ClusterNode` rows, and the LocalDb deployment-artifact cache is
+> already keyed by it *so that a pair shares one entry*. Scoping the election the same way is the
+> fix; `RedundancyStateActor` is an admin-role singleton in the same assembly as
+> `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
+>
+> Surfaced by the LocalDb Phase 2 live gate, where it stopped the alarm-history drain on every node
+> in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`. The alarm drain no longer
+> depends on this (it defers only to a node that shares its queue), but the other three gates still do.
+
 ## Data flow
 
 ```
@@ -114,13 +143,18 @@ The admin singleton is the cluster's only `RedundancyStateActor`. If the admin l
 
 Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var:
 
-```json
+```jsonc
 {
   "Cluster": {
     "Hostname": "0.0.0.0",
     "Port": 4053,
     "PublicHostname": "node-a.lan",
-    "SeedNodes": ["akka.tcp://otopcua@node-a.lan:4053"],
+    // Self FIRST, partner second — see "Bootstrap: self-first seed ordering".
+    // node-b.lan's own config lists node-b.lan first and node-a.lan second.
+    "SeedNodes": [
+      "akka.tcp://otopcua@node-a.lan:4053",
+      "akka.tcp://otopcua@node-b.lan:4053"
+    ],
     "Roles": ["admin", "driver"]
   }
 }
@@ -130,9 +164,9 @@ Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var:
 OTOPCUA_ROLES=admin,driver
 ```
 
-Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list.
+Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. List **both** peers in `SeedNodes` on **both** nodes so either can cold-start alone — **and list each node ITSELF first**: the order decides which Akka bootstrap process runs, and a node that lists its partner first can never form the cluster while that partner is down. See [Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering).
 
-There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster role-leadership). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).
+There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).
 
 > **`RedundancyStateActor` NodeId consistency (fixed).** `RedundancyStateActor` now keys each node's `NodeRedundancyState` entry by the canonical `host:port` node id (via a `ToNodeId(Address)` helper mirroring `ClusterRoleInfo.ToNodeId`). Previously it keyed by `member.Address.Host` (host-only, e.g. `central-2`); since every subscriber matches by the canonical `host:port` form, the mismatch silently meant no node ever matched its own entry — all nodes stayed at the default ServiceLevel 255 and never learned their role. This fix makes `RedundancyStateActor` consistent with the stated contract above. Additionally, `RedundancyStateActor` now **re-publishes the current snapshot on a periodic heartbeat (default 10 s)** so any node that subscribes after the last topology-change publish converges within the interval (DistributedPubSub does not replay to late subscribers).
 
@@ -152,55 +186,201 @@ Each node advertises its partner via `OpcUaApplicationHostOptions.PeerApplicatio
 
 Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpointTests` in `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/` — boots two `OpcUaApplicationHost` instances on loopback, asserts a real OPCFoundation client `Session` reading `Server.ServerArray` from Node A sees both URIs.
 
-## Split-brain
+## Split-brain / downing
 
-The split-brain resolver is **active by default**: Akka.Cluster.Hosting's `WithClustering` enables an SBR
-downing provider whenever `ClusterOptions.SplitBrainResolver` is null (it applies
-`SplitBrainResolverOption.Default`, which registers `Akka.Cluster.SBR.SplitBrainResolverProvider`), and that
-provider reads the `split-brain-resolver` HOCON block in `akka.conf`. On top of that,
-`ServiceCollectionExtensions.BuildClusterOptions` sets the typed `ClusterOptions.SplitBrainResolver`
-(`KeepOldestOption { DownIfAlone = true }`) to make the strategy **explicit in code** rather than relying on
-the framework default — it is reinforcing, not the sole activator, and yields the same effective behavior. So
-the cluster is **not** running `NoDowning`; hard-crashed nodes are downed and the cluster recovers (how it
-recovers depends on *which* node was lost — see the table below). (Only an *explicit* `NoDowning`, e.g.
-`akka.cluster.downing-provider-class = ""`, would leave both redundancy sides at ServiceLevel 240
-indefinitely.) The HOCON block carries the tuning: `active-strategy = keep-oldest`, `stable-after = 15s`,
-`keep-oldest.down-if-alone = on`, `failure-detector.threshold = 10.0` (its `active-strategy` + `down-if-alone`
-must stay consistent with the typed option; `stable-after` lives only in HOCON because the typed option can't
-express it).
+**The default downing strategy is `auto-down` (changed 2026-07-21).** It is selected by
+`Cluster:SplitBrainResolverStrategy`, and `ServiceCollectionExtensions.BuildDowningHocon` turns that
+into the `akka.cluster.downing-provider-class` actually installed:
 
-`keep-oldest` is the correct strategy for a 2-node warm-redundancy pair (`keep-majority`/`static-quorum`
-are wrong for two nodes — no majority in a 1-1 split), but its two loss cases recover **differently**, and
-this is inherent to any 2-node cluster:
-
-| Node lost | What keep-oldest does | Recovery |
+| `Cluster:SplitBrainResolverStrategy` | Provider installed | Posture |
 |---|---|---|
-| **Younger** node (crash or partition) | The oldest is on the surviving side → it stays up and downs the younger side (`DownUnreachable`). | **In-place, fast.** The oldest keeps its singletons + `driver` role-leadership; `RedundancyStateActor` re-computes from the post-loss `Cluster.State`. |
-| **Oldest** node (crash or partition) | The lone survivor is "the side **without** the oldest" → keep-oldest downs the **survivor itself** (`DownReachable "including myself"`), and `run-coordinated-shutdown-when-down = on` terminates it. `down-if-alone` does **not** rescue a lone survivor (its rescue branch needs ≥ 2 surviving members, so it is a 3+-node feature). | **Exit-and-rejoin.** The self-downed survivor **and** the crashed oldest are both restarted by the service supervisor and re-form / rejoin. There is a brief restart-window outage; there is **no** in-place takeover. |
+| `auto-down` (**default**) | `Akka.Cluster.AutoDowning` with `auto-down-unreachable-after = 15s` | **Availability.** The leader among the *reachable* members downs the unreachable peer, so a crash of **either** node — oldest included — fails over in place. |
+| `keep-oldest` | `Akka.Cluster.SBR.SplitBrainResolverProvider` reading the `split-brain-resolver` block in `akka.conf` | **Partition-safety.** A partition can never run dual-active. **Not safe for a 2-node pair** — see below. |
 
-**Recovery therefore depends on three things being in place** (the [ScadaBridge](../../ScadaBridge/CLAUDE.md)
-sister project runs the same pattern):
+Any other value **fails the host at startup** rather than falling through to a default.
+
+### Why keep-oldest is no longer the default
+
+The previous documentation in this section — and the code comments behind it — asserted that
+`keep-oldest` with `down-if-alone = on` was "the correct strategy for a 2-node warm-redundancy pair."
+**That was wrong, and the error was not conservative: it described a total-outage configuration as the
+recommended one.**
+
+In Akka.NET 1.5.62's `KeepOldest.OldestDecision`, the `down-if-alone` rescue branch requires the
+*surviving* side to hold **≥ 2 members**. In a two-node cluster a lost peer is always a 1-vs-1 split, so
+the branch never fires and the lone survivor falls through to `DownReachable` — downing **itself** — and
+`run-coordinated-shutdown-when-down = on` then terminates it. `down-if-alone` is a 3+-node feature; it
+does not do what its name suggests here. Live-proven on the sister project's rig, whose survivor logged
+`SBR took decision …DownReachable and is downing [self] including myself, [1] unreachable of [2] members`.
+See [`../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md`](../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md)
+for the upstream decision record, which OtOpcUa follows.
+
+The alternatives do not help a pair: `static-quorum` with quorum 1 hits Akka's `IsTooManyMembers` guard
+and returns `DownAll`; with quorum 2 the survivor downs itself; `keep-majority` keeps the lowest-address
+side, which just moves the fatal crash from "the oldest node" to "the other node"; `lease-majority` needs
+a shared lease store both nodes can reach.
+
+### The accepted trade
+
+Under `auto-down`, a **genuine network partition** (both nodes alive, link cut) leaves each side downing
+the other and continuing alone: **both run active**, both hold singletons, and both advertise the primary
+`ServiceLevel`. Recovery is operator-driven — after the link heals, restart **one** side; it rejoins as a
+fresh incarnation and settles as secondary. The two sides do not merge on their own, because the mutual
+downing quarantines the association.
+
+This is a deliberate choice of availability over partition-safety, matching ScadaBridge: the pairs run one
+node per VM with no external arbiter available, so a crash is a far more likely event than a partition, and
+a crash under `keep-oldest` was unrecoverable without operator action anyway.
+
+Deployments that would rather take an outage than ever run dual-active can set
+`Cluster:SplitBrainResolverStrategy: "keep-oldest"` — but should read the section above first, because for a
+two-node pair that choice means *any* crash of the oldest node is a full outage.
+
+### Recovery still depends on supervision
 
 1. **A service supervisor that restarts the process** on exit — production `Install-Services.ps1` sets
    `sc.exe failure OtOpcUaHost … actions= restart/5000/restart/30000/restart/60000`; the docker-dev rig sets
-   `restart: unless-stopped`. Without it a downed node stays down and an oldest-crash looks like a total outage.
-2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) — it
-   watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls
-   `IHostApplicationLifetime.StopApplication()` so the process exits (and the supervisor restarts it) instead of
-   idling forever with a dead actor system.
-3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join the mesh via either peer.
+   `restart: unless-stopped`.
+2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) —
+   it watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls
+   `IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead
+   of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this
+   (it is never the one downed); the **downed** node still does.
+3. **Both peers listed in `SeedNodes`** on every node, **each node listing itself first**, so a restarted
+   node can re-join via either peer and a node cold-starting while its peer is dead still forms the
+   cluster. Without the self-first ordering it waits in `InitJoin` forever (auto-down removes the crash
+   outage, not that one) — see [Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering).
 
 > **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
-> sole `driver` role-leader, and it passes — but it simulates the crash with `provider.Transport.Shutdown()`,
-> which leaves node A's `ActorSystem` **alive** (a transport partition with the oldest still running), not a
-> real process death. It is therefore **not** representative of an oldest-process crash; a real 2-container
-> `docker kill` of the oldest downs the survivor (verified 2026-07-15, see
-> `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`). Treat the recovery guarantee for
-> the oldest as "exit-and-rejoin under supervision," not "in-place failover."
+> sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with
+> `provider.Transport.Shutdown()`, which leaves node A's `ActorSystem` **alive** (a transport partition with
+> the oldest still running), not a real process death. A real 2-container `docker kill` of the oldest downed
+> the survivor (verified 2026-07-15, `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`).
+> **It is a standing example of a green test over a fatal defect**: keep the distinction between a transport
+> partition and a process death in mind when reading it.
 
-There is no operator-driven role swap during a partition. Failover / recovery is what the cluster + supervisor
-do automatically. **Instant in-place takeover on *any* single-node loss requires 3+ members** (an odd cluster
-or a lightweight witness) — a deliberate future option, not the current 2-node posture.
+`SplitBrainResolverActivationTests` pins the strategy by starting a real host through the production
+bootstrap and reading `akka.cluster.downing-provider-class` back off the running `ActorSystem` — asserting
+the typed option or the akka.conf text is not sufficient, because several HOCON fragments compete and the
+losing one still looks correct in the file.
+
+Failover during a partition is what the cluster and supervisor do automatically. For a *planned* swap —
+patching, draining a node — see [Manual failover](#manual-failover) below.
+
+## Manual failover
+
+The cluster redundancy page (`/clusters/{id}/redundancy`) carries a **Live redundancy** panel — the current
+driver Primary and every Up `driver` member, read from live cluster state — and an admin-gated **Trigger
+failover** button.
+
+The mechanism is a graceful cluster **`Leave`** of the current Primary, never a `Down`:
+`IManualFailoverService.FailOverDriverPrimaryAsync` selects the **oldest Up `driver` member** — the identical
+query to `RedundancyStateActor.SelectDriverPrimary`, pinned together by
+`ManualFailoverServiceTests.Parity_with_SelectDriverPrimary` so the node acted on is exactly the node the
+election names. The leaving node hands its singletons over through the cluster-leave phases,
+`CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it,
+and it rejoins as the **youngest** member — so the survivor is now the oldest, becomes Primary, and
+advertises `ServiceLevel` 250. OPC UA clients re-select.
+
+| Rule | Behaviour |
+|---|---|
+| Authorization | `AdminUiPolicies.FleetAdmin` (**Administrator only**) — gated in markup *and* re-checked server-side before the call. Deliberately not `ConfigEditor`, which the neighbouring cluster pages use: this restarts a production node rather than editing config, and ConfigEditor also admits Designer. |
+| Peer guard | Disabled, with the reason as its tooltip, when fewer than two Up driver members exist — with one node a "failover" is a shutdown. Re-evaluated inside the service against live state, so a stale page cannot bypass it. |
+| Confirmation | A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select). |
+| Audit | Written through the shared `ZB.MOM.WW.Audit.IAuditWriter` seam **before** the Leave — action `cluster.manual-failover`, actor, target. A *refused* failover changes nothing and writes nothing. |
+
+> **Mesh-scope caveat.** Until the per-cluster mesh work lands
+> (`docs/plans/2026-07-21-per-cluster-mesh-design.md`), the Primary is elected once per Akka mesh rather than
+> per application `Cluster` row — so on a fleet running several clusters in one mesh this button acts on the
+> whole mesh's Primary, which may belong to a different cluster than the page you are on. The panel says so.
+> Phase 6 of the mesh program removes both the caveat and this notice.
+
+> **Live gate outstanding.** The auto-down change is verified by unit tests against the effective
+> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has
+> **not** yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where
+> the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster;
+> assert the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) should run
+> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
+> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
+
+## Bootstrap: self-first seed ordering
+
+Listing both peers in `SeedNodes` does **not** mean either node can cold-start alone — and neither does any
+downing strategy: **auto-down removes the crash outage, not this one.** Akka runs a different bootstrap
+process depending on whether `seed-nodes[0]` is the node's own address:
+
+| `seed-nodes[0]` is… | Process Akka runs | Can it form a NEW cluster? |
+|---|---|---|
+| this node's own address | `FirstSeedNodeProcess` | **Yes** — it `InitJoin`s the other seeds and self-joins once `seed-node-timeout` (5 s) passes with nobody answering. |
+| some other node | `JoinSeedNodeProcess` | **No** — it retries `InitJoin` forever, however long the peer stays down. |
+
+So the ordering *is* the mechanism. **Every node that is a seed of its own mesh lists ITSELF first and its
+partner second** (decision 2026-07-22):
+
+```jsonc
+// central-1                                     // central-2
+"SeedNodes": [                                   "SeedNodes": [
+  "akka.tcp://otopcua@central-1:4053",             "akka.tcp://otopcua@central-2:4053",
+  "akka.tcp://otopcua@central-2:4053"              "akka.tcp://otopcua@central-1:4053"
+]                                                ]
+```
+
+The rule is **conditional**: it binds only when a node's own address appears in its own seed list. A
+driver-only site node is seeded solely by `central-1` (today's docker-dev topology) — it is legitimately
+not a seed of anything and is exempt. An unconditional "self must be first" rule would refuse to boot every
+site node in the fleet.
+
+`AkkaClusterOptionsValidator` enforces it at startup (`AddValidatedOptions` → `ValidateOnStart` in
+`AddOtOpcUaCluster`), because the invariant otherwise fails **silently**: the process runs, the port
+listens, the node simply never becomes a member. Identity is compared on `Cluster:PublicHostname` (falling
+back to `Cluster:Hostname` when blank) **and** `Cluster:Port` — the address Akka puts in `SelfAddress`.
+Comparing against `Cluster:Hostname` would be worse than wrong: in docker-dev it is `0.0.0.0`, which matches
+no seed URI anywhere, so the rule would silently exempt the entire rig.
+
+Sequential recovery is island-free by construction: a `FirstSeedNodeProcess` node self-joins only when **no**
+seed answered, so a peer booting after the survivor formed simply joins it as the youngest member. Both
+nodes cold-starting *simultaneously* converge on one cluster while they are mutually reachable — the
+`InitJoin` handshake resolves it before either self-join deadline expires. The residual risk is a boot-time
+**partition** (both cold-starting, mutually unreachable): both form, which is the same dual-active class the
+`auto-down` strategy already accepts, with the same recovery (restart one side).
+
+Pinned by `SelfFirstSeedBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real
+in-process clusters through the production bootstrap at production failure-detection timings: a lone
+cold-start with a dead peer forms alone; a restarted node rejoins its live peer; two nodes cold-starting
+together converge on one cluster. Its **falsifiability control** — the old peer-first ordering, same
+scenario, never comes Up — is what keeps the other three from going vacuous, and carries a positive-control
+self-join proving the node was formable all along. `AkkaClusterOptionsValidatorTests` covers the validator,
+including the site-node exemption and the `0.0.0.0` identity trap.
+
+### Why the self-form watchdog was retired (2026-07-22)
+
+The first fix for this gap was `Cluster:SelfFormAfter` + `ClusterBootstrapFallback`: wait 10 s for
+membership, then call `Cluster.Join(SelfAddress)`. **Both are deleted.** A timer sits *outside* Akka's join
+handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight" —
+and `Cluster.Join(SelfAddress)` is **not** ignored mid-handshake: it wins.
+
+> **Live-gate finding (docker-dev, 2026-07-22 — preserved because it is the evidence).** The watchdog
+> islanded a node that manual failover had bounced. `central-1` restarted, received `InitJoinAck` from
+> `central-2` at 10:49:05 — the join was in flight and healthy — but no Welcome arrived inside the window,
+> because the peer's ring still held its previous incarnation (`Exiting` → `Down` → `Removed`, retired at
+> 10:49:06). At 10:49:15 the watchdog fired and the node formed a **second cluster**, islanded until an
+> operator restarted it. Manual failover deliberately produces exactly that restart, so the two halves of
+> that plan collided. A TCP reachability guard (`ea45ace1`) patched that one shape — never self-form while
+> a seed peer accepts a connection — and the earlier live gate for the watchdog itself had genuinely
+> passed: `central-2` cold-started alone in 10 s and a non-seed `site-a-2` correctly stayed out.
+
+The reachability guard closed the observed failure, not the class: a peer can be TCP-reachable and not
+answering, or answering and slow, and the timer cannot tell. Self-first ordering has no such race because
+the decision is made *inside* the handshake, by Akka, on the actual `InitJoin` replies. It was also inert
+for every non-seed node, so after this change it could never usefully fire. `SelfFormBootstrapTests` is
+replaced by `SelfFirstSeedBootstrapTests`, whose `Restarting_node_rejoins_its_live_peer_instead_of_islanding`
+is the drilled scenario above.
+
+> **Live gate outstanding for the new mechanism.** Self-first ordering is verified here by in-process
+> clusters at production timings, and on the sister project (ScadaBridge `4a6341d8`) on real clusters *and*
+> live docker containers. It has **not** been re-drilled on the OtOpcUa docker-dev rig since the swap — the
+> `central-2` cold-start-alone drill should be repeated on the rebuilt image. Owned by
+> `docs/plans/2026-07-22-per-cluster-mesh-program.md` Phase 7 alongside the auto-down 1-vs-1 gate.
 
 ## Primary data-plane gate (writes, acks, alerts emit)
 
@@ -221,11 +401,88 @@ Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and ev
 
 - **`alerts` topic emission** — `ScriptedAlarmHostActor` and `DriverHostActor.ForwardNativeAlarm` subscribe to the `redundancy-state` DPS topic and cache the local node's `RedundancyRole`, then gate the cluster `alerts` publish through `PrimaryGatePolicy` (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain **ungated** on both nodes so the secondary is always ready to serve clients after a failover.
 - **`HistorianAdapterActor` historization** — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the `alerts` DPS topic and translates each `AlarmTransitionEvent` → `AlarmHistorianEvent` before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
+- **The alarm sink's DRAIN** — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. `LocalDbStoreAndForwardSink` takes a `drainGate` fed from `IRedundancyRoleView` — a singleton `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, because the drain runs on a timer and cannot receive `RedundancyStateChanged` itself. A gated-off node reports `HistorianDrainState.NotPrimary`. Delivery is **at-least-once across a failover** by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See [AlarmHistorian.md](AlarmHistorian.md).
 
 Net effect: each alarm transition appears **once** on `/alerts` and would historize once, not once per node.
 
 See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals.
 
+## Command transport (central↔node)
+
+Everything above rides one Akka mesh: central publishes on DistributedPubSub, every node
+subscribes, and the two ends must be members of the same cluster. Per-cluster mesh **Phase 2**
+adds a second transport under `MeshTransport:Mode` (see
+[Configuration.md § `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport))
+that does not need shared membership — the prerequisite for splitting the fleet into one mesh per
+application `Cluster`.
+
+| | `Dps` (default) | `ClusterClient` |
+|---|---|---|
+| Central → node commands | `Publish` on `deployments` / `driver-control` / `alarm-commands` | `ClusterClient.SendToAll` to `/user/node-communication`, re-emitted on the node's **local** `EventStream` |
+| Node → central acks | `Publish` on `deployment-acks` | `ClusterClient.Send` to `/user/central-communication`, forwarded to the deploy-coordinator singleton |
+| Requires shared cluster membership | **yes** | no |
+
+Two properties are load-bearing and easy to break:
+
+- **Both comm actors are registered per node, NOT as cluster singletons.** A `ClusterClient` rotates
+  across its contact points and must find a live comm actor at whichever node answers. As a
+  singleton the boundary would be reachable through exactly one node, and rotation would fail
+  silently against the others — acks landing or not depending on where the client happened to
+  settle. `akka.cluster.client.receptionist.role` is deliberately empty for the same reason.
+- **Inbound commands go to the node-local `EventStream`, never back onto a DPS topic.** DPS is
+  mesh-wide and central `SendToAll`s to *every* node, so a DPS re-publish would deliver N copies of
+  every command to every subscriber.
+
+Phase 2 changes no redundancy semantics: the Primary gate, ServiceLevel, and the singleton layout
+are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional and built from
+`Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the
+meshes can split).
+
+**Config bytes travel out-of-band (Phase 3).** Commands stay on one of the two transports above, but
+the *configuration artifact itself* does not ride any Akka message — `DispatchDeployment` is
+payload-free (`DeploymentId` + `RevisionHash`). Under `ConfigSource:Mode = FetchAndCache` (see
+[Configuration.md § `ConfigSource`](Configuration.md#configsource--configserve-config-fetch-and-cache))
+a driver node fetches the bytes from central over a dedicated h2c **gRPC stream**, verifies
+`SHA-256 == RevisionHash`, and reads only its LocalDb cache — no central-SQL config read. **Both pair
+nodes fetch independently**, exactly as both read SQL today: there is no Primary gating of the fetch.
+That is safe because `StoreAsync` is idempotent and the cache replicates (pair replication carries the
+artifact to a node that missed its own fetch), and it keeps config delivery decoupled from redundancy
+timing. Central serves the bytes from its SQL row; a `FetchAndCache` node that cannot reach central
+fails the apply and keeps last-known-good (it never tears down to an empty address space), and it boots
+its served state from the LocalDb pointer with no central dependency at all.
+
+## Pair-local store (LocalDb — Phases 1 + 2)
+
+Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated
+[`ZB.MOM.WW.LocalDb`](../CLAUDE.md) SQLite database. Phase 2 added the **alarm store-and-forward
+buffer** (`alarm_sf_events`) to it alongside the config cache, so a node that dies holding
+undelivered alarm history no longer takes it to the grave — see the Primary-only drain note above
+and [AlarmHistorian.md](AlarmHistorian.md). Phase 1 caches the **deployed-configuration
+artifact** (chunked). Its job is a single failure mode redundancy does not otherwise cover: a driver
+node restarting into a **central-SQL-Server outage** would come up with no configuration at all. With
+the cache, it **boots from the last artifact it applied** instead, and logs a running-from-cache
+signal.
+
+- **What replicates.** The cache can *optionally* replicate to the node's redundant pair peer over
+  the LocalDb library's gRPC sync (HLC-stamped, last-writer-wins). Two tables replicate:
+  `deployment_artifacts` and `deployment_pointer`. Replication is **default-OFF and fail-closed** —
+  it stays inert until `LocalDb:SyncListenPort` + a peer are configured, and a missing/mismatched
+  `LocalDb:Replication:ApiKey` refuses or silently halts sync. It is enabled per-pair as an opt-in.
+- **The replicated-cache payoff.** In a pair with replication on, *either* node can be the one that
+  applied the latest deploy; the peer holds a byte-identical copy. So a node that restarts during a
+  central outage can boot the current config **even if it never applied that deploy itself** — its
+  peer did, and the row replicated.
+- **What it does NOT cover.** The cache is a **boot fallback for central-SQL outages only**, not a
+  standby data path and not a replacement for central. A **new deployment still requires central
+  SQL**; the cache is read only when the central fetch fails at startup, and the node resumes
+  fresh-config behavior once central returns. It does not replicate live tag values, alarms, or
+  historian data — only the config artifact. It is orthogonal to the ServiceLevel/primary-gate
+  logic: a node running from cache still participates in redundancy normally.
+
+Full detail: the enablement runbook at
+[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
+and the `LocalDb` section of [`docs/Configuration.md`](Configuration.md).
+
 ## Client-side failover
 
 The OtOpcUa Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` supports `-F` / `--failover-urls` for automatic client-side failover; for long-running subscriptions the CLI monitors session KeepAlive and reconnects to the next available server, recreating the subscription on the new endpoint. See [`Client.CLI.md`](Client.CLI.md).

docs/ServiceHosting.md

@@ -56,6 +56,7 @@ The host joins an Akka.NET cluster bound to the address in `appsettings.json::Cl
 }
 ```
 
+- `SeedNodes` is **ordered**: a node listed in its own seed list must be entry 0, since Akka only lets `seed-nodes[0]` form a new cluster. `AkkaClusterOptionsValidator` fails startup otherwise — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering).
 - `WithOtOpcUaClusterBootstrap` (in `OtOpcUa.Cluster`) loads the embedded HOCON (split-brain resolver, pinned dispatcher, failure detector tuning) and overlays remote endpoint + cluster options.
 - All cluster singletons + per-node actors live on this single ActorSystem — there is no second Akka instance.
 

docs/drivers/Historian.Wonderware.md

@@ -13,7 +13,7 @@ OtOpcUa now consumes the **`ZB.MOM.WW.HistorianGateway`** sidecar through the Gi
 
 - **HistoryRead** → `GatewayHistorianDataSource` over the `ServerHistorian` appsettings section.
 - **Alarm history** → `GatewayAlarmHistorianWriter` (the gateway `SendEvent` path) behind the durable
-  `SqliteStoreAndForwardSink`; alarm-history `ReadEvents` needs the gateway running
+  `LocalDbStoreAndForwardSink`; alarm-history `ReadEvents` needs the gateway running
   `RuntimeDb:EventReadsEnabled=true`.
 - **Continuous historization** → a crash-safe FasterLog outbox + `ContinuousHistorizationRecorder`
   draining to the gateway's `WriteLiveValues` (`ContinuousHistorization` section); needs the gateway

docs/operations/2026-07-20-localdb-pair-replication.md

@@ -0,0 +1,191 @@
+# LocalDb pair replication — operations runbook
+
+> **Scope.** Every driver-role OtOpcUa node keeps a consolidated
+> [`ZB.MOM.WW.LocalDb`](../../CLAUDE.md) SQLite database. It holds two things: the
+> **deployed-configuration artifact** (chunked, Phase 1) so a node can **boot from cache when
+> central SQL Server is unreachable**, and the **alarm store-and-forward buffer** (Phase 2) so a
+> node that dies holding undelivered alarm history does not take it to the grave. Both can
+> *optionally* replicate to the node's redundant pair peer over the library's gRPC sync.
+> **Replication is default-OFF and fail-closed.** This runbook covers enabling it, the operational
+> rules that keep a pair converging, and the DB-inspection safety rules.
+
+## What replicates, and what it buys you
+
+- **Replicated tables:** `deployment_artifacts` (the artifact, split into base64 chunks),
+  `deployment_pointer` (one current-deployment pointer per cluster), and `alarm_sf_events` (the
+  alarm store-and-forward buffer). Registered — in this order, after the DDL — by
+  `LocalDbSetup.OnReady`.
+- **The payoff:** in a redundant driver pair, *either* node can be the one that applied the latest
+  deploy and wrote the cache. With replication on, the peer holds a byte-identical copy. So a node
+  that restarts into a central-SQL outage can boot the current config **even if it never applied
+  that deploy itself** — its peer did, and the row replicated.
+- **Convergence model:** last-writer-wins per primary key, HLC-stamped. Both nodes also store
+  locally after every apply, so the pair converges to identical content regardless of which node
+  deployed. Retention (newest-2 deployments per cluster) prunes on one node and the deletes
+  replicate as tombstones.
+
+## Alarm store-and-forward — what replication changes
+
+Phase 2 moved the alarm buffer out of its own `alarm-historian.db` and into this database. Three
+consequences an operator needs to know:
+
+- **Only the Primary drains.** The buffer replicates, so the Secondary holds a full copy of the
+  Primary's queue. Its drain worker is gated on the same Primary decision the inbound-write and
+  native-ack gates use, and reports `DrainState = NotPrimary`. **A rising queue depth on a
+  Secondary is correct.** A rising queue depth on *both* nodes is not — see below.
+- **Delivery is at-least-once across a failover, by design.** Rows the old Primary delivered
+  moments before it died may not have had their deletes replicated yet, so the new Primary will
+  re-send them. This is accepted, not a defect: a duplicate alarm-history row is recoverable, a
+  missing one is not. There is deliberately no dedup layer.
+- **The buffer is bounded.** `AlarmHistorian:Capacity` (1,000,000 by default) still evicts the
+  oldest undelivered rows when the historian has been unreachable long enough. Watch
+  `EvictedCount` — non-zero means accepted alarm events were dropped before reaching the historian.
+
+### If BOTH nodes report `NotPrimary`
+
+Nobody is draining, and the queue is filling toward the capacity ceiling on both. The gate
+default-DENIES while the redundancy role is unknown *and* a driver peer exists, so this is what a
+redundancy snapshot that never names either node looks like — the identity-mismatch shape
+`DriverHostActor` also warns about once ("redundancy snapshot omitted this node"). Check node
+identity (`Cluster:PublicHostname` / `Cluster:Port` skew) before suspecting the sink.
+
+### One-time migration from `alarm-historian.db`
+
+On first boot after the upgrade, `AlarmSfLegacyMigrator` copies any pre-existing
+`alarm-historian.db` into `alarm_sf_events` and renames the file to `alarm-historian.db.migrated`.
+
+- **Both nodes of a pair migrate independently**, and their files overlap — row ids are derived
+  from the event payload, so the same event on both nodes converges to one row rather than
+  duplicating. The new Primary drains the union.
+- **The rename happens only after the copy commits.** A failure fails host startup with the legacy
+  file untouched; the `.migrated` sidecar is what makes a later boot a no-op.
+- **A file whose shape is unrecognised is left alone** — not renamed, not copied — so it stays
+  available for inspection.
+- The `AlarmHistorian:DatabasePath` key is **removed**. It is still *read* by the migrator, to
+  find the legacy file; it no longer configures anything. Delete it from appsettings once the
+  `.migrated` sidecar exists on every node.
+
+## Enabling replication on a pair
+
+Replication has two knobs, both under `LocalDb`:
+
+| Key | Role | Notes |
+|---|---|---|
+| `LocalDb:SyncListenPort` | binds the dedicated **h2c** sync listener | `0` (default) = no listener, replication off. Set the **same** non-zero port on **both** nodes of the pair. |
+| `LocalDb:Replication:PeerAddress` | the address this node **dials** | Set on **one** node (the initiator); leave empty on the other (passive). The stream is bidirectional — one dial carries both directions. |
+| `LocalDb:Replication:ApiKey` | bearer token for the sync stream | **Must be byte-identical on both nodes.** See fail-closed rule below. Supply via `${secret:...}` / env — never a cleartext literal in production config. |
+| `LocalDb:Replication:MaxBatchSize` | rows per replication batch | `16` for the artifact cache. Batching is **row-count-only** against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so 16 × 171 KB ≈ 2.7 MB stays under the cap. Raising it risks tripping the cap. |
+
+Example (site-a-1 initiator, site-a-2 passive):
+
+```
+site-a-1:  LocalDb__SyncListenPort=9001
+           LocalDb__Replication__PeerAddress=http://site-a-2:9001
+           LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key}
+           LocalDb__Replication__MaxBatchSize=16
+site-a-2:  LocalDb__SyncListenPort=9001
+           LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key}   # no PeerAddress
+           LocalDb__Replication__MaxBatchSize=16
+```
+
+> **Kestrel note.** Setting `SyncListenPort` makes the Host add an explicit `Listen*` for the h2c
+> sync listener. An explicit `Listen*` makes Kestrel **ignore `ASPNETCORE_URLS` entirely**, so the
+> Host also re-binds the primary HTTP port in the same block. If you change the primary port,
+> verify both `:<httpPort>` and `:<syncPort>` appear in the startup "Now listening on" lines.
+
+### The ApiKey rule (fail-closed)
+
+The replication library's passive endpoint verifies **no** authentication — the host
+`LocalDbSyncAuthInterceptor` is the only gate, and it is **fail-closed**:
+
+- **No key configured ⇒ every sync call is refused** (`PermissionDenied`). "No key" is never
+  "no auth required".
+- **A key mismatch ⇒ the pair silently stops converging.** The initiator's stream is rejected at
+  the peer; nothing errors loudly. A typo in one node's key looks exactly like "replication is
+  broken". If a pair is not converging, **check the keys match first.**
+
+## Operational rules
+
+- **Stop / start the pair together where you can.** Each node keeps working (and caching locally)
+  while its peer is down; the outage is not a data-loss event — the surviving node accumulates
+  writes and the peer catches up on rejoin. But a long-lived solo node drifts further from its
+  peer, so avoid leaving a pair split for extended periods.
+- **Tombstone-retention resurrection window.** Retention prunes to the newest 2 deployments and
+  replicates the prune as tombstones. Tombstones are themselves retained only for a bounded window.
+  If a node is offline **longer than the tombstone-retention window**, a delete that happened during
+  its outage may no longer be expressible as a tombstone on rejoin — a pruned deployment could
+  briefly reappear until the next deploy re-prunes it. Keep pair outages well inside that window.
+- **A rebuilt node is back-filled by its peer.** If a node loses its LocalDb file — a wiped volume,
+  a re-imaged host, a fresh container — it rejoins with an empty database and its peer snapshots the
+  cached configuration back to it, with no new deploy and no central SQL. Two things are worth
+  knowing: this needs **`ZB.MOM.WW.LocalDb` ≥ 0.1.3**, and it heals the *cache*, so the node regains
+  boot-from-cache for future outages. (On `0.1.1` a converged pair has pruned every oplog row on ack
+  and the wiped node was never snapshotted — it stayed empty until the next deploy. On `0.1.2` it is
+  back-filled but its OWN writes are silently dropped until its restarted seq counter climbs past the
+  peer's stale watermark, so the pair looks converged right up until the moment only the rebuilt node
+  applies a deploy.) A node
+  wiped **during** a central outage is still repopulated by its peer under 0.1.2; a node with **no**
+  peer replication configured self-heals only from central on its next successful apply.
+
+- **What boot-from-cache does NOT cover.** The cache is a *fallback for central-SQL outages at
+  boot*, not a replacement for central. A **new deployment still requires central SQL** — the cache
+  is only read when the central fetch fails at startup. When central returns, the node resumes
+  fresh-config behavior. Boot-from-cache logs a running-from-cache signal; treat a node that stays
+  on it as a central-connectivity incident, not steady state.
+
+## Inspecting the LocalDb file — safety rules
+
+The database runs in **WAL mode**. These rules exist because violating them corrupted a live DB in
+the 2026-07-20 ScadaBridge incident:
+
+- **Never run `sqlite3` on the live file** (host-side, against a bind-mounted or container path).
+  Opening a live WAL DB from a second process across virtiofs poisons the WAL. **Always copy the
+  triplet first** and query the copy:
+
+  ```bash
+  docker cp <node>:/app/data/otopcua-localdb.db      /tmp/localdb.db
+  docker cp <node>:/app/data/otopcua-localdb.db-wal  /tmp/localdb.db-wal
+  docker cp <node>:/app/data/otopcua-localdb.db-shm  /tmp/localdb.db-shm
+  sqlite3 /tmp/localdb.db 'SELECT cluster_id, deployment_id FROM deployment_pointer;'
+  ```
+
+- **Metrics** come from the container, not a host curl (`aspnet:10.0` has no `curl`):
+
+  ```bash
+  docker run --rm --network container:<node> curlimages/curl:latest -s localhost:<httpPort>/metrics | grep localdb_
+  ```
+
+- **If an anomaly appears within seconds of your own measurement, suspect the measurement.**
+  Restart both nodes and re-observe untouched before blaming the library.
+
+## Useful queries (on a copied triplet)
+
+```sql
+-- What each node thinks the current deployment is
+SELECT cluster_id, deployment_id, revision_hash, applied_at_utc FROM deployment_pointer;
+
+-- Convergence check: the pointer's origin stamp should match on both nodes
+SELECT pk_json, hlc, node_id, is_tombstone
+FROM __localdb_row_version WHERE table_name = 'deployment_pointer' ORDER BY pk_json;
+
+-- Replication backlog (should drain to 0 when a pair is caught up)
+SELECT COUNT(*) FROM __localdb_oplog;
+
+-- Undelivered alarm history, and how much of it has given up
+SELECT dead_lettered, COUNT(*) FROM alarm_sf_events GROUP BY dead_lettered;
+
+-- Why the dead-lettered rows died
+SELECT alarm_id, attempt_count, last_attempt_utc, last_error
+FROM alarm_sf_events WHERE dead_lettered = 1 ORDER BY last_attempt_utc DESC LIMIT 20;
+
+-- Convergence check for the buffer: identical dumps mean the peer holds the ORIGIN-stamped rows
+SELECT pk_json, hlc, node_id, is_tombstone
+FROM __localdb_row_version WHERE table_name = 'alarm_sf_events' ORDER BY pk_json;
+```
+
+## See also
+
+- [`docs/Redundancy.md`](../Redundancy.md) — the pair-local config cache section.
+- [`docs/Configuration.md`](../Configuration.md) — the `LocalDb` appsettings section.
+- [`docs/AlarmHistorian.md`](../AlarmHistorian.md) — the alarm sink, its knobs, and the drain.
+- The two-node convergence harness: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/`.

docs/plans/2026-07-16-v3-batch4-PR.md

@@ -94,7 +94,7 @@ dual-namespace + event-delivery), Runtime 377 (+ VT reassert regression), AdminU
 
 ## Documented follow-ups (non-blocking)
 
-- **Scripted-alarm redeploy recovery (CONFIRMED pre-existing bug, deferred — deserves its own careful fix).**
+- **Scripted-alarm redeploy recovery (CONFIRMED pre-existing bug, deferred — deserves its own careful fix). Filed as issue #487.**
   The scripted-alarm Part 9 condition node has the same redeploy-reset race the VT `ReassertValue` fix closed:
   `RebuildAddressSpace` clears `_alarmConditions` and `MaterialiseScriptedAlarms` recreates each condition
   fresh/normal, but `ScriptedAlarmEngine.LoadAsync` reloads from the persisted state and yields
@@ -106,10 +106,11 @@ dual-namespace + event-delivery), Runtime 377 (+ VT reassert regression), AdminU
   historian issue). Proposed shape: a host-driven, node-only `AlarmStateUpdate` re-assert in `OnAlarmsLoaded`
   (same safe post-materialise ordering as the VT fix), reusing `_engine.GetState(alarmId)` + `LoadedAlarmIds`,
   **never touching the `alerts` topic**.
-- Raw-tag rename hot-rebind of the renamed tag's live value without a recreate (today: deploy-THEN-recreate).
-- Absolute-path Monaco completion → RawPath (from Batch 3; the `{{equip}}/` completion works).
+- Raw-tag rename hot-rebind of the renamed tag's live value without a recreate (today: deploy-THEN-recreate). Filed as issue #489.
+- Absolute-path Monaco completion → RawPath (from Batch 3; the `{{equip}}/` completion works). Filed as issue #490.
 - Cross-repo (post-merge): ScadaBridge Data-Connection-Layer NodeId/namespace cutover (raw `s=<RawPath>` /
-  UNS `s=<Area>/<Line>/<Equipment>/<EffectiveName>`, retired `EquipmentNodeIds`) + the umbrella index
-  `../scadaproj/CLAUDE.md` OtOpcUa entry.
+  UNS `s=<Area>/<Line>/<Equipment>/<EffectiveName>`, retired `EquipmentNodeIds`) — tracked in the
+  ScadaBridge repo as its issue #14 (its #20 landed the `nsu=` binding + native-alarm routing). The umbrella
+  index `../scadaproj/CLAUDE.md` OtOpcUa entry is **DONE** (scadaproj `ede5275`).
 
 https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox

docs/plans/2026-07-20-localdb-adoption-phase1.md

@@ -0,0 +1,706 @@
+# OtOpcUa LocalDb Adoption — Phase 1 Implementation Plan
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+>
+> **Execution model:** This plan is optimized for **Claude Opus** agents (`claude --model opus`).
+> Dispatch implementer subagents on Opus for every task classified `high-risk`; `small`/`trivial`
+> tasks may use Sonnet. Work on branch **`feat/localdb-phase1`** in this repo (`~/Desktop/OtOpcUa`,
+> remote `lmxopcua`). Do NOT merge to `master` as part of this plan — stop at the DoD task and
+> report.
+>
+> **Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md`.
+> Read it before Task 0. The reference adoption is ScadaBridge (merged PR #23) — when in doubt,
+> mirror `~/Desktop/ScadaBridge` (files named per task below).
+
+**Goal:** Give every driver-role OtOpcUa node a consolidated `ZB.MOM.WW.LocalDb` database that caches
+the deployed-configuration artifact (chunked), boots from that cache when central SQL Server is
+unreachable, and optionally replicates it to the node's redundant pair peer over the library's gRPC
+sync (default-OFF, fail-closed bearer auth).
+
+**Architecture:** `AddZbLocalDb`/`AddZbLocalDbReplication` wired in a new `LocalDbRegistration`
+(mirroring `SecretsRegistration`), DDL + `RegisterReplicated` in `LocalDbSetup.OnReady`, a dedicated
+Kestrel h2c listener for `MapZbLocalDbSync` gated on `LocalDb:SyncListenPort`, a consumer-supplied
+fail-closed bearer interceptor, and an `IDeploymentArtifactCache` seam written by `DriverHostActor`
+after each successful apply and read as a boot fallback. The dormant LiteDB LocalCache subsystem is
+deleted as superseded.
+
+**Tech stack:** .NET 10, `ZB.MOM.WW.LocalDb`/`.Replication`/`.Contracts` **0.1.1** (Gitea feed),
+`Grpc.AspNetCore` 2.76.0, Akka.NET (existing), xunit.v3 (Host.IntegrationTests) + xunit2 TestKit
+(actor tests).
+
+**Hard rules (from the ScadaBridge adoption — violating any of these is a defect):**
+1. Order inside `OnReady` is load-bearing: **DDL → `RegisterReplicated` → writes**. Rows written
+   before registration are never captured or snapshotted — silently, forever.
+2. No autoincrement PKs, no BLOB columns in replicated tables.
+3. The library's passive sync endpoint verifies **no** ApiKey — the host interceptor is the only
+   auth, and it must be fail-closed (no key configured ⇒ deny everything).
+4. `ILocalDb.CreateConnection()` returns an **already-open**, pragma-configured connection with the
+   `zb_hlc_next()` UDF. Never call `Open()` on it. There is **no in-memory mode** — tests use temp
+   files + `SqliteConnection.ClearAllPools()` before delete.
+5. `Grpc.Core.Testing` does not exist on grpc-dotnet — hand-roll fake `ServerCallContext`s.
+6. Explicit Kestrel `Listen*` calls make Kestrel **ignore `ASPNETCORE_URLS`** — when adding the sync
+   listener you must re-bind the existing HTTP port in the same block, and verify it on the rig.
+7. Never run host `sqlite3` against a live bind-mounted WAL DB (copy the `db`/`-wal`/`-shm` triplet
+   with `cp` instead). `aspnet:10.0` containers have no `curl` — use a
+   `curlimages/curl` sidecar with `--network container:<name>`.
+
+---
+
+### Task 0: Preflight + recon (produces `docs/plans/2026-07-20-localdb-phase1-recon.md`)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (everything downstream consumes its findings)
+
+**Files:**
+- Create: `docs/plans/2026-07-20-localdb-phase1-recon.md`
+- Read-only: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs`,
+  `Directory.Packages.props`, `docker-dev/docker-compose.yml`,
+  the observability registration (`AddOtOpcUaObservability` implementation),
+  `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/*`
+
+**Step 1: Verify the feed packages restore.**
+
+Run:
+```bash
+cd ~/Desktop/OtOpcUa && dotnet package search ZB.MOM.WW.LocalDb --source dohertj2-gitea --format json | head -50
+```
+Expected: `ZB.MOM.WW.LocalDb`, `.Replication`, `.Contracts` at `0.1.1`. If absent → STOP, report.
+
+**Step 2: Map and record, with `file:line` citations, each of:**
+1. **Deploy fetch path:** where `DriverHostActor` loads `Deployment.ArtifactBlob` (the
+   `_dbFactory.CreateDbContext()` sites, ~L1472/1543), the artifact's runtime type
+   (`DeploymentArtifact` — how it's deserialized from the blob), the type/format of
+   `DeploymentId` and `RevisionHash`, and where a successful apply completes (the point after
+   which the cache write belongs).
+2. **Cold-boot path:** what the actor does at startup before any `DispatchDeployment` arrives —
+   does it query central SQL for the current deployment? Where does the failure path land
+   (the `Stale` state, `DbHealthProbeActor` interaction, L1345 dispatch-ignore)? Identify the
+   exact seam where "central fetch failed at boot" is known — that is where the cache fallback goes.
+3. **Cluster identity:** how a driver node knows its `ClusterId` (config key / `ClusterNode` row /
+   `IClusterRoleInfo`) — the cache is keyed by it.
+4. **DriverHostActor construction:** how its dependencies are injected
+   (`WithOtOpcUaRuntimeActors`, `Props` wiring) so `IDeploymentArtifactCache` can be threaded in.
+   ⚠ `Props.Create` builds an expression tree — adding a parameter silently rebinds
+   out-of-position named args at call sites; list every construction site.
+5. **Telemetry allowlist:** does `AddOtOpcUaObservability` restrict meters
+   (`ZbTelemetryOptions.Meters` or equivalent)? If yes, record where the list lives.
+6. **Kestrel/URLs:** confirm the Host binds via `ASPNETCORE_URLS=http://+:9000` with no
+   `ConfigureKestrel` calls; record any existing `builder.WebHost` usage.
+7. **LiteDB LocalCache:** confirm (grep) that nothing outside
+   `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/` and its tests references
+   `ILocalConfigCache`/`GenerationSealedCache`/`ResilientConfigReader`/`LiteDB`.
+8. **docker-dev volumes:** whether nodes have a writable data volume; record what `LocalDb:Path`
+   should be per node and how env vars are passed (`Cluster__*` style double-underscore).
+
+**Step 3: STOP conditions — halt the plan and report instead of improvising if:**
+- the artifact apply path cannot be given a post-apply hook without restructuring the actor;
+- `DeploymentId`/cluster identity are not stable strings/GUIDs;
+- LiteDB LocalCache turns out to be referenced by live code.
+
+**Step 4: Commit.**
+```bash
+git add docs/plans/2026-07-20-localdb-phase1-recon.md
+git commit -m "docs(localdb): phase-1 recon findings"
+```
+
+---
+
+### Task 1: Package references and pins
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+**Parallelizable with:** none (all code tasks build on it)
+
+**Files:**
+- Modify: `Directory.Packages.props`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ZB.MOM.WW.OtOpcUa.Runtime.csproj`
+
+**Step 1:** Add to `Directory.Packages.props` (versions exact):
+```xml
+<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.1" />
+<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.1" />
+<PackageVersion Include="Grpc.AspNetCore" Version="2.76.0" />
+```
+Check the existing `Grpc.Net.Client`/`Grpc.Core.Api` are already ≥ 2.76.0 and
+`Google.Protobuf` ≥ 3.34.1 (the LocalDb.Replication nuspec floors); raise if not. Confirm a
+`SQLitePCLRaw` pin ≥ 2.1.12 exists for the transitive `Microsoft.Data.Sqlite` chain (the family
+advisory GHSA-2m69-gcr7-jv3q); Core.AlarmHistorian already pins `bundle_e_sqlite3` 2.1.12 — add
+`SQLitePCLRaw.lib.e_sqlite3` 2.1.12 as an explicit reference in the Host if the audit flags it.
+
+**Step 2:** Host csproj: `ZB.MOM.WW.LocalDb`, `ZB.MOM.WW.LocalDb.Replication`, `Grpc.AspNetCore`.
+Runtime csproj: `ZB.MOM.WW.LocalDb` only (it needs `ILocalDb`, not the sync engine).
+
+**Step 3:** `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → 0 warnings/errors (repo builds warnings-as-errors).
+
+**Step 4: Commit.** `git commit -m "build(localdb): reference ZB.MOM.WW.LocalDb 0.1.1 + Grpc.AspNetCore"`
+
+---
+
+### Task 2: Schema + `LocalDbSetup.OnReady` + registration tests
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 3
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/DeploymentCacheSchema.cs`
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/LocalDbSetupTests.cs` (create the test project
+  reference layout to match neighboring Host unit tests; if Host has no unit-test project, place in
+  the closest existing one and note the deviation)
+
+**Step 1: Write the failing tests first** — using a real temp-file `ILocalDb` (build it via
+`new ServiceCollection().AddZbLocalDb(config, LocalDbSetup.OnReady)` with `LocalDb:Path` pointed at
+a temp file; dispose + `SqliteConnection.ClearAllPools()` in cleanup):
+- `OnReady_RegistersExactlyTheTwoDeploymentTables` — `db.ReplicatedTables.Keys` ordinal-sorted
+  equals `["deployment_artifacts", "deployment_pointer"]` (assert BOTH directions: no fewer, no more).
+- `DeploymentArtifacts_PkIsDeploymentIdPlusChunkIndex` and `DeploymentPointer_PkIsClusterId`
+  (pin `ReplicatedTable.PkColumns`).
+- `RowsWrittenAfterOnReady_EnterTheOplog` — insert a row, assert `SELECT COUNT(*) FROM __localdb_oplog` ≥ 1
+  (pins the DDL→register ordering; this is the assertion that catches a silently-broken CDC).
+
+**Step 2: Run tests, watch them fail** (types don't exist yet).
+
+**Step 3: Implement.** `DeploymentCacheSchema` depends only on `Microsoft.Data.Sqlite` (the
+ScadaBridge `*Schema.Apply` pattern — self-sufficient for direct construction in tests):
+
+```csharp
+namespace ZB.MOM.WW.OtOpcUa.Runtime.Deployment;
+
+public static class DeploymentCacheSchema
+{
+    public static void Apply(SqliteConnection connection)
+    {
+        using var cmd = connection.CreateCommand();
+        cmd.CommandText = """
+            CREATE TABLE IF NOT EXISTS deployment_artifacts (
+                deployment_id   TEXT    NOT NULL,
+                chunk_index     INTEGER NOT NULL,
+                cluster_id      TEXT    NOT NULL,
+                revision_hash   TEXT    NOT NULL,
+                chunk_count     INTEGER NOT NULL,
+                chunk_base64    TEXT    NOT NULL,
+                cached_at_utc   TEXT    NOT NULL,
+                PRIMARY KEY (deployment_id, chunk_index)
+            );
+            CREATE TABLE IF NOT EXISTS deployment_pointer (
+                cluster_id        TEXT NOT NULL PRIMARY KEY,
+                deployment_id     TEXT NOT NULL,
+                revision_hash     TEXT NOT NULL,
+                artifact_sha256   TEXT NOT NULL,
+                applied_at_utc    TEXT NOT NULL
+            );
+            """;
+        cmd.ExecuteNonQuery();
+    }
+}
+```
+
+`LocalDbSetup` (Host):
+```csharp
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+internal static class LocalDbSetup
+{
+    // ORDER IS LOAD-BEARING: DDL, then RegisterReplicated, then (nothing else in Phase 1).
+    // Rows written before RegisterReplicated are invisible to the peer forever, silently.
+    public static void OnReady(ILocalDb db)
+    {
+        using var connection = db.CreateConnection(); // already open — do NOT call Open()
+        DeploymentCacheSchema.Apply(connection);
+        db.RegisterReplicated("deployment_artifacts");
+        db.RegisterReplicated("deployment_pointer");
+    }
+}
+```
+
+**Step 4:** `dotnet test --filter "FullyQualifiedName~LocalDbSetupTests"` → PASS.
+
+**Step 5: Commit.** `git commit -m "feat(localdb): deployment-cache schema + OnReady registration"`
+
+---
+
+### Task 3: Fail-closed sync auth interceptor + tests
+
+**Classification:** standard
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 2
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/LocalDbSyncAuthInterceptorTests.cs`
+
+Port ScadaBridge's `src/ZB.MOM.WW.ScadaBridge.Host/LocalDbSyncAuthInterceptor.cs` (read it first;
+keep behavior identical, adjust namespace only). Semantics to preserve exactly:
+- `ServicePrefix = "/localdb_sync.v1.LocalDbSync/"`; any other method path passes through untouched.
+- Expected token from `IOptions<ReplicationOptions>.Value.ApiKey`.
+- **Fail-closed:** null/empty configured key ⇒ `RpcException(new Status(StatusCode.PermissionDenied, ...))`
+  for every sync call. Missing/mismatched bearer ⇒ same.
+- `CryptographicOperations.FixedTimeEquals` over UTF-8 bytes; header `authorization`,
+  case-insensitive, `"Bearer "` prefix. Override all four server handler kinds.
+
+**Tests** (hand-rolled fake `ServerCallContext` — `Grpc.Core.Testing` does not exist on grpc-dotnet;
+copy ScadaBridge's `LocalDbSyncAuthInterceptorTests.cs` fake): non-sync method passes with no key;
+sync + no configured key → PermissionDenied; wrong bearer → PermissionDenied; correct bearer → passes.
+
+Write tests first (fail), implement, `dotnet test --filter "FullyQualifiedName~LocalDbSyncAuthInterceptor"`,
+commit `feat(localdb): fail-closed bearer interceptor for the sync endpoint`.
+
+---
+
+### Task 4: `LocalDbRegistration` + Program.cs DI wiring + config defaults
+
+**Classification:** high-risk (touches Program.cs / role gating)
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (Task 5 edits the same file)
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (inside the `hasDriver` branch)
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json`
+
+**Step 1:** `LocalDbRegistration`, mirroring `SecretsRegistration`'s shape:
+```csharp
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+internal static class LocalDbRegistration
+{
+    /// Driver-role nodes only. Storage is unconditional; replication stays inert until
+    /// LocalDb:Replication:PeerAddress (initiator) / LocalDb:SyncListenPort (listener) are set.
+    public static IServiceCollection AddOtOpcUaLocalDb(
+        this IServiceCollection services, IConfiguration configuration)
+    {
+        services.AddZbLocalDb(configuration, LocalDbSetup.OnReady);
+        services.AddZbLocalDbReplication(configuration);
+        return services;
+    }
+
+    public static int SyncListenPort(IConfiguration configuration) =>
+        configuration.GetValue<int>("LocalDb:SyncListenPort");
+}
+```
+
+**Step 2:** Program.cs, in the `hasDriver` block (near `AddAlarmHistorian`):
+`builder.Services.AddOtOpcUaLocalDb(builder.Configuration);` and
+`builder.Services.AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>());`
+(AddGrpc only under `hasDriver` — admin-only nodes expose no sync surface).
+
+**Step 3:** appsettings.json:
+```json
+"LocalDb": {
+  "Path": "./data/otopcua-localdb.db",
+  "SyncListenPort": 0,
+  "Replication": {}
+}
+```
+`LocalDb:Path` is `ValidateOnStart`-required once `AddZbLocalDb` runs — since registration is
+driver-gated, admin-only configs need nothing. Check the role-overlay files
+(`appsettings.driver.json` / `appsettings.admin-driver.json`) and any deploy templates for
+conflicting `LocalDb` keys.
+
+**Step 4:** Build + full Host unit tests. Verify an admin-only graph doesn't require the key: this
+is pinned properly in Task 10's integration tests, but do a quick
+`OTOPCUA_ROLES=admin dotnet run` smoke only if cheap; otherwise rely on Task 10.
+
+**Step 5: Commit.** `feat(localdb): wire AddOtOpcUaLocalDb into the driver role`
+
+---
+
+### Task 5: Dedicated h2c sync listener + endpoint mapping (THE Kestrel task)
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (Program.cs)
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`
+
+**Why high-risk:** the Host today binds exclusively via `ASPNETCORE_URLS=http://+:9000`. Any
+explicit `ConfigureKestrel(... Listen*)` makes Kestrel **ignore URLs entirely** (it logs
+"Overriding address(es)"), which would silently kill the AdminUI/deploy API behind Traefik.
+
+**Step 1:** Add, before `builder.Build()`:
+```csharp
+var syncPort = LocalDbRegistration.SyncListenPort(builder.Configuration);
+if (hasDriver && syncPort > 0)
+{
+    // Explicit Listen* replaces ASPNETCORE_URLS wholesale, so re-bind the primary HTTP
+    // endpoint here too. Parse it from the configured URLs rather than hard-coding 9000.
+    var urls = builder.Configuration["ASPNETCORE_URLS"] ?? builder.Configuration["urls"] ?? "http://+:9000";
+    var httpPort = new Uri(urls.Split(';')[0].Replace("+", "localhost").Replace("*", "localhost")).Port;
+    builder.WebHost.ConfigureKestrel(k =>
+    {
+        k.ListenAnyIP(httpPort);                                            // HTTP/1.1 (existing surface)
+        k.ListenAnyIP(syncPort, o => o.Protocols = HttpProtocols.Http2);    // h2c, prior-knowledge gRPC
+    });
+}
+```
+When `syncPort == 0` (the default) nothing changes — URLs binding stays untouched. Cleartext
+`Http1AndHttp2` cannot serve prior-knowledge h2c, hence the dedicated `Http2`-only listener.
+
+**Step 2:** After the app pipeline's other `Map*` calls, driver-gated:
+```csharp
+if (hasDriver && syncPort > 0)
+{
+    app.MapZbLocalDbSync();
+}
+```
+(Mapping is harmless when unauthenticated — the interceptor fail-closes — but gating on the port
+keeps admin-only and default-OFF graphs entirely free of the endpoint.)
+
+**Step 3: Verify both surfaces.**
+```bash
+dotnet build ZB.MOM.WW.OtOpcUa.slnx
+```
+Then a local smoke with the port on:
+`ASPNETCORE_URLS=http://+:9000 OTOPCUA_ROLES=driver LocalDb__SyncListenPort=9001 dotnet run --project src/Server/ZB.MOM.WW.OtOpcUa.Host ...`
+— expect startup logs to show BOTH `:9000` and `:9001` bound, and `curl -s localhost:9000/healthz`
+(or the mapped health route) still answering. If the app needs SQL to boot, defer the smoke to the
+Task 12 rig check but say so explicitly in the task notes.
+
+**Step 4: Commit.** `feat(localdb): dedicated h2c listener + MapZbLocalDbSync gated on LocalDb:SyncListenPort`
+
+---
+
+### Task 6: `IDeploymentArtifactCache` + chunked LocalDb implementation + tests
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 3, Task 5
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactCache.cs`
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/LocalDbDeploymentArtifactCache.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Deployment/LocalDbDeploymentArtifactCacheTests.cs`
+
+**Step 1: Failing tests** (real temp-file `ILocalDb` with `LocalDbSetup.OnReady` applied):
+- round-trip: store a 300 KiB random artifact → `GetCurrentAsync` returns byte-identical payload
+  (forces ≥ 3 chunks at 128 KiB);
+- retention: store 3 deployments for one cluster → only the newest 2 remain in
+  `deployment_artifacts`; pointer names the newest;
+- integrity: corrupt one chunk row via SQL → `GetCurrentAsync` returns null (miss), never a
+  truncated artifact;
+- missing pointer → null.
+
+**Step 2: Implement:**
+```csharp
+public interface IDeploymentArtifactCache
+{
+    Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
+                    byte[] artifact, CancellationToken ct = default);
+    Task<CachedDeploymentArtifact?> GetCurrentAsync(string clusterId, CancellationToken ct = default);
+}
+
+public sealed record CachedDeploymentArtifact(
+    string DeploymentId, string RevisionHash, byte[] Artifact, DateTimeOffset AppliedAtUtc);
+```
+`LocalDbDeploymentArtifactCache(ILocalDb db)`:
+- `StoreAsync`: one `ILocalDbTransaction` — delete any existing chunks for this `deployment_id`
+  (idempotent re-store), insert chunks (raw 128 * 1024 bytes per chunk, `Convert.ToBase64String`),
+  upsert the pointer (`INSERT ... ON CONFLICT(cluster_id) DO UPDATE`), then prune: delete
+  `deployment_artifacts` rows whose `deployment_id` is not among the newest 2 `cached_at_utc` for
+  this `cluster_id`. SHA-256 over the raw artifact into the pointer. ISO-8601 UTC timestamps.
+- `GetCurrentAsync`: read pointer; read chunks `ORDER BY chunk_index`; verify count ==
+  `chunk_count` and SHA-256 matches; return null on any mismatch (log a warning naming which check
+  failed).
+- Use `db.ExecuteAsync`/`QueryAsync` with anonymous-object parameters (`@Name` markers). Remember
+  a `Dictionary<string,string>` parameter **throws** — use anonymous objects.
+
+**Step 3:** run tests → PASS. **Step 4: Commit.**
+`feat(localdb): chunked deployment-artifact cache over ILocalDb`
+
+---
+
+### Task 7: Cache write path in `DriverHostActor`
+
+**Classification:** high-risk (actor model, `Props` expression-tree trap)
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:** (exact edit sites come from the Task 0 recon doc — cite it in the commit)
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
+- Modify: the actor-registration site (`WithOtOpcUaRuntimeActors` / DI extension) to provide
+  `IDeploymentArtifactCache`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` or `LocalDbRegistration` — register
+  `IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache` singleton (driver branch)
+- Test: the **xunit2** TestKit project used for existing DriverHostActor tests (recon names it)
+
+**Steps:**
+1. **Failing test:** after a successful artifact apply, the cache received `StoreAsync` with the
+   applied deployment's id/hash/bytes (inject a recording fake `IDeploymentArtifactCache`).
+   Also: a cache that throws on `StoreAsync` does NOT fail the apply (apply result unchanged, error
+   logged).
+2. Thread the dependency through the actor's constructor. ⚠ `Props.Create` is an expression tree:
+   after adding the parameter, re-check **every** construction site for positional/named-arg
+   rebinding (the recon lists them) — do not rely on the compiler.
+3. Invoke `StoreAsync` fire-and-forget (`PipeTo`-style or a guarded `Task.Run` per the actor's
+   existing async conventions — match whatever pattern the actor already uses for side-effect IO)
+   at the post-apply point identified in recon. Cluster id from the recon-identified source.
+4. Run the actor test suite for this project. Commit:
+   `feat(localdb): DriverHostActor stores applied artifacts in the pair-local cache`
+
+---
+
+### Task 8: Boot-from-cache read path + running-from-cache signal
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
+- Test: same xunit2 TestKit project as Task 7
+
+**Steps:**
+1. **Failing tests:**
+   - central fetch fails at startup AND cache has an artifact → the actor applies the cached
+     artifact (assert via whatever observable the apply already exposes) and logs/flags
+     running-from-cache;
+   - central fetch fails AND cache empty → today's behavior exactly (Stale, no apply);
+   - central fetch **succeeds** → cache is NOT consulted (fresh config wins; assert the fake
+     cache's `GetCurrentAsync` was never called on the happy path).
+2. Implement at the recon-identified boot-failure seam. The signal: a log warning at minimum plus,
+   if the actor already publishes health/status (recon says how), a `RunningFromCache` marker on it.
+   Do NOT touch `DispatchDeployment` handling — a new deployment still requires central.
+3. Run tests; commit `feat(localdb): boot from the pair-local artifact cache when central SQL is unreachable`.
+
+---
+
+### Task 9: Delete the dormant LiteDB LocalCache subsystem
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+**Parallelizable with:** Task 6, Task 10, Task 11
+
+**Files:**
+- Delete: `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/` (entire directory:
+  `ILocalConfigCache`, `LiteDbConfigCache`, `GenerationSealedCache`, `ResilientConfigReader`,
+  `GenerationSnapshot`, `StaleConfigFlag`, exceptions)
+- Delete: `tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/GenerationSealedCacheTests.cs`,
+  `ResilientConfigReaderTests.cs`
+- Modify: `Directory.Packages.props` + the Configuration csproj — remove the `LiteDB` package if
+  nothing else references it (grep first)
+
+**Steps:** Confirm the Task 0 recon's "nothing references it" finding still holds
+(`grep -rn "ILocalConfigCache\|GenerationSealedCache\|ResilientConfigReader\|LiteDB" src/ tests/`),
+delete, build the full solution, run the Configuration test project. DoD phrasing: **no references
+from code** — leave any explanatory prose/comments that point readers at LocalDb instead. Add one
+line to the recon doc noting LiteDB's removal. Commit:
+`refactor(localdb): delete the dormant LiteDB LocalCache (superseded by ZB.MOM.WW.LocalDb)`
+
+---
+
+### Task 10: Health check + telemetry meter allowlist
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 9, Task 11
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/LocalDbReplicationHealthCheck.cs` (or the
+  directory `AddOtOpcUaHealth` uses — recon names it)
+- Modify: the `AddOtOpcUaHealth` registration (driver branch)
+- Modify: the observability meter list IF Task 0 found an allowlist
+
+**Steps:**
+1. Failing unit tests: replication unconfigured (no peer, no listener) → `Healthy` (default-OFF
+   must not degrade a plain node); peer configured + `ISyncStatus.Connected == false` → `Degraded`;
+   connected + backlog `null` → `Degraded` (unknown backlog is not healthy); connected + backlog
+   small → `Healthy`.
+2. Implement against `ISyncStatus` + `IOptions<ReplicationOptions>` (peer-configured = non-empty
+   `PeerAddress` OR `SyncListenPort > 0` — pass the latter in via options/config).
+3. **Meter allowlist:** if `AddOtOpcUaObservability` restricts meters, add
+   `"ZB.MOM.WW.LocalDb.Replication"` (use `LocalDbMetrics.MeterName`) — this is a silent allowlist;
+   the ScadaBridge live gate is the proof it bites. If there is no allowlist, record that in the
+   recon doc and skip.
+4. Commit `feat(localdb): replication health check + meter export`.
+
+---
+
+### Task 11: DI-pin integration tests over the real Program.cs
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 9, Task 10
+
+**Files:**
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbWiringTests.cs` (xunit.v3,
+  `WebApplicationFactory<Program>` — the project already builds the real Program.cs; set
+  `OTOPCUA_ROLES` per test case the way neighboring tests do)
+
+**Pins (each its own test):**
+1. Driver graph: `ILocalDb` resolves, is a singleton, `ReplicatedTables` ordinal-sorted ==
+   `["deployment_artifacts", "deployment_pointer"]` — exact set, both directions load-bearing.
+2. Driver graph: `IDeploymentArtifactCache` resolves to `LocalDbDeploymentArtifactCache`.
+3. Default-OFF pin: `ISyncStatus` resolves with `Connected == false`, `PeerNodeId == null`.
+4. Admin-only graph: `ILocalDb` is NOT registered (`GetService<ILocalDb>()` is null) and boot does
+   not demand `LocalDb:Path`.
+5. Health: the LocalDb health check is registered in the driver graph.
+
+Point `LocalDb:Path` at a per-test temp file via the factory's config overrides;
+`ClearAllPools` + delete in cleanup. These tests exist because DI extensions without a
+container-built test have shipped inert three times in this family (Secrets 0.2.0/0.2.2,
+ScadaBridge#22). Commit `test(localdb): DI pins over the real host graph`.
+
+---
+
+### Task 12: Two-node convergence harness + tests
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min (harness) + ~4 min (scenarios) — split the commit if needed
+**Parallelizable with:** none (depends on Tasks 2, 3, 5, 6)
+
+**Files:**
+- Create: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairHarness.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairConvergenceTests.cs`
+
+**Harness:** copy the pattern from
+`~/Desktop/ScadaBridge/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs` and
+the library's `ConvergenceFixture` (`~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/tests/.../Convergence/ConvergenceFixture.cs`):
+- Two full stacks over a real loopback Kestrel h2c socket (port 0), **through the real
+  `LocalDbSyncAuthInterceptor`** with a shared test key; node A initiator, node B passive.
+- Both initialized via the **production `LocalDbSetup.OnReady`** — a hand-written schema proves
+  only that the test agrees with itself.
+- DBs owned by the harness, registered as pre-constructed singletons (so MS.DI won't dispose them
+  on host teardown); `KillTransportAsync`/`RestartTransportAsync`; tight `FlushInterval` (50 ms),
+  bounded backoff (2 s); temp files, `ClearAllPools` cleanup; a collection definition to serialize
+  these tests.
+
+**Scenarios:**
+1. `ArtifactStoredOnA_ConvergesToB_ByteIdentical` — store a multi-chunk artifact via
+   `LocalDbDeploymentArtifactCache` on A; B's cache returns byte-identical bytes; both nodes'
+   `__localdb_row_version` rows for the pointer carry the **same HLC and origin node id** (B holds
+   A's row, not a re-derived one).
+2. `RetentionPruneOnA_TombstonesReachB` — third deployment on A prunes the first; B's chunk rows
+   for the pruned deployment disappear and `__localdb_row_version WHERE is_tombstone=1` rows exist
+   on B.
+3. `WritesWhileTransportDown_SurviveRejoin` — kill transport, store on A, restart, converge.
+4. `WrongApiKey_NeverConverges` — harness with mismatched keys: assert **no** convergence after a
+   bounded wait AND (positive control) that the same scenario with matching keys converges — an
+   absence assertion without a positive control passed vacuously in ScadaBridge.
+
+**DoD within this task:** temporarily comment out the two `RegisterReplicated` calls and confirm
+scenarios 1–3 go red (run locally, do not commit the red state); restore. Record the red/green
+evidence in the task notes. Commit `test(localdb): 2-node convergence harness + scenarios`.
+
+---
+
+### Task 13: docker-dev rig configuration
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 14
+
+**Files:**
+- Modify: `docker-dev/docker-compose.yml`
+
+**Steps** (adjust to the recon's findings on volumes/env style):
+1. Every driver node: a named volume (or existing data mount) backing `/app/data`, and
+   `LocalDb__Path=/app/data/otopcua-localdb.db`.
+2. **site-a pair only** (mirror the ScadaBridge default-OFF posture — site-b stays unreplicated as
+   the pin):
+   - site-a-1: `LocalDb__SyncListenPort=9001`, `LocalDb__Replication__PeerAddress=http://site-a-2:9001`,
+     `LocalDb__Replication__ApiKey=dev-site-a-localdb-sync-key`, `LocalDb__Replication__MaxBatchSize=16`
+   - site-a-2: `LocalDb__SyncListenPort=9001`, same `ApiKey`, same `MaxBatchSize`, **no PeerAddress**
+     (passive; the stream is bidirectional).
+   The key must be byte-identical on both — the interceptor fail-closes on any mismatch and the
+   pair silently stops converging.
+3. `MaxBatchSize=16` because batching is row-count-only against gRPC's 4 MB cap and artifact chunk
+   rows are ≈ 171 KB (16 × 171 KB ≈ 2.7 MB worst case).
+4. `docker compose config` to validate; do NOT bring the rig up in this task (the live gate task
+   owns rig runs).
+5. Commit `chore(localdb): rig config — consolidated path everywhere, replication on the site-a pair`.
+
+---
+
+### Task 14: Documentation
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 13
+
+**Files:**
+- Create: `docs/operations/2026-07-20-localdb-pair-replication.md` (runbook: enable/disable,
+  ApiKey handling via `${secret:}`, stop/start-together rule, tombstone-retention resurrection
+  window, MaxBatchSize row-count-vs-4MB, never host-`sqlite3` a live WAL DB / cp-triplet recipe)
+- Modify: `docs/Redundancy.md` (a short "pair-local config cache" section: what replicates, what
+  boot-from-cache does and does not cover)
+- Modify: `CLAUDE.md` (this repo): LocalDb adoption row/paragraph — state Phase 1 scope, default-OFF,
+  rig-only enablement
+- Modify: `docs/Configuration.md` if it documents config keys (add the `LocalDb` section)
+
+Commit `docs(localdb): phase-1 runbook + redundancy/config docs`.
+
+---
+
+### Task 15: DoD sweep (offline)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (final offline task)
+
+**Steps:**
+1. `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → **0 warnings** (warnings-as-errors).
+2. `dotnet test ZB.MOM.WW.OtOpcUa.slnx` → full suite green; record counts. Baseline any
+   pre-existing failures BEFORE this branch (`git stash` → run → unstash) rather than assuming;
+   report deltas only.
+3. Greps (phrase as "no references from code"; explanatory comments may remain):
+   `grep -rn "LiteDB\|ILocalConfigCache\|GenerationSealedCache" src/ tests/` → no code references.
+4. Confirm the positive-control evidence from Task 12 is recorded.
+5. Update `docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json` statuses.
+6. Commit `chore(localdb): phase-1 DoD sweep` and STOP. Report: branch name, commit list, test
+   counts, and that the live gate (Task 16) needs the rig + operator go-ahead.
+
+---
+
+### Task 16: Live gate on the docker-dev rig (run only with explicit user go-ahead)
+
+**Classification:** high-risk (rig)
+**Estimated implement time:** ~30 min wall-clock (mostly waiting)
+**Parallelizable with:** none
+
+**Preamble:** All DB inspection via `cp` of the `db`/`-wal`/`-shm` triplet out of the container
+(`docker cp`) and querying the copy — never host `sqlite3` on the live file (it poisons the WAL
+across virtiofs; root cause of the 2026-07-20 ScadaBridge incident). Metrics via
+`docker run --rm --network container:<node> curlimages/curl:latest -s localhost:<port>/metrics`.
+If an anomaly appears within seconds of one of your own measurements, suspect the measurement
+first — restart both nodes and re-run untouched before blaming the library.
+
+**Checks (all must PASS; record evidence in `docs/plans/2026-07-20-localdb-phase1-live-gate.md`):**
+1. Rig up, site-a pair healthy, `/metrics` on both shows `localdb_` series (proves the meter
+   allowlist work).
+2. Deploy a config through the normal deploy API → both site-a nodes' `deployment_pointer` +
+   `deployment_artifacts` rows are byte-identical with **identical HLC + origin node id** (one
+   node's row replicated, not two independent derivations — both nodes also locally store after
+   apply, so expect LWW-converged identical content either way; assert convergence, and note which
+   origin won).
+3. **Boot-from-cache:** stop the SQL container; restart site-a-2; it comes up serving the cached
+   config with the running-from-cache signal in its log; start SQL again; node recovers fresh
+   behavior.
+4. **Replicated-cache payoff:** wipe site-a-2's local DB file (container stopped), restart it with
+   SQL **up**, let replication repopulate the cache from site-a-1, then repeat check 3's SQL-down
+   restart — it boots from a cache it never wrote itself.
+5. Transport kill (pause site-a-2 container): store/deploy on a-1, oplog depth rises; unpause;
+   drains to 0; converged.
+6. Retention: deploy a 3rd config; pruned deployment's chunks gone on BOTH nodes with tombstone
+   rows present on both.
+7. Both-nodes-together restart: clean rejoin, identical counts, zero
+   `disk I/O error`/`SQLITE_IOERR`/`corrupt` in either log.
+8. site-b pair (default-OFF pin): LocalDb file exists and works locally, `ISyncStatus` health
+   Healthy, no sync connections, no listener on 9001.
+
+Record PASS/FAIL per check with the actual evidence (row counts, md5s, log lines). Commit the gate
+doc. Do not merge — report.
+
+---
+
+## Task persistence
+
+Tasks file: `docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json` (same directory).
+Update statuses as tasks complete; any deviation from this plan gets a `deviation` note on the task
+(the ScadaBridge tasks.json convention — the record of *why* is what makes the plan auditable).

docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json

@@ -0,0 +1,153 @@
+{
+  "planPath": "docs/plans/2026-07-20-localdb-adoption-phase1.md",
+  "tasks": [
+    {
+      "id": 0,
+      "subject": "Task 0: Preflight + recon (findings doc, STOP conditions)",
+      "status": "completed",
+      "notes": "Feed verified: LocalDb/.Contracts/.Replication all 0.1.1. All 3 STOP conditions clear. Findings: docs/plans/2026-07-20-localdb-phase1-recon.md. 8 deviations recorded (D-1..D-8); D-1/D-3/D-5/D-6 change downstream work materially.",
+      "deviation": "D-1 cache read cannot be keyed by ClusterId at the boot seam (unkeyed newest-pointer read instead); D-3 a third LiteDB test file must be deleted; D-5 WebApplicationFactory<Program> is deliberately unused in this repo (use TwoNodeClusterHarness); D-6 driver-only nodes have no ASPNETCORE_URLS so the Kestrel re-bind fallback is wrong. See recon doc \u00a79."
+    },
+    {
+      "id": 1,
+      "subject": "Task 1: Package references and pins (LocalDb 0.1.1, Grpc.AspNetCore, SQLitePCLRaw)",
+      "status": "completed",
+      "blockedBy": [
+        0
+      ]
+    },
+    {
+      "id": 2,
+      "subject": "Task 2: Schema + LocalDbSetup.OnReady + registration tests",
+      "status": "completed",
+      "blockedBy": [
+        1
+      ]
+    },
+    {
+      "id": 3,
+      "subject": "Task 3: Fail-closed sync auth interceptor + tests",
+      "status": "completed",
+      "blockedBy": [
+        1
+      ]
+    },
+    {
+      "id": 4,
+      "subject": "Task 4: LocalDbRegistration + Program.cs DI wiring + config defaults",
+      "status": "completed",
+      "blockedBy": [
+        2,
+        3
+      ]
+    },
+    {
+      "id": 5,
+      "subject": "Task 5: Dedicated h2c sync listener + MapZbLocalDbSync (Kestrel URLs-override risk)",
+      "status": "completed",
+      "blockedBy": [
+        4
+      ]
+    },
+    {
+      "id": 6,
+      "subject": "Task 6: IDeploymentArtifactCache + chunked LocalDb implementation + tests",
+      "status": "completed",
+      "blockedBy": [
+        2
+      ]
+    },
+    {
+      "id": 7,
+      "subject": "Task 7: Cache write path in DriverHostActor (Props expression-tree trap)",
+      "status": "completed",
+      "blockedBy": [
+        0,
+        6
+      ]
+    },
+    {
+      "id": 8,
+      "subject": "Task 8: Boot-from-cache read path + running-from-cache signal",
+      "status": "completed",
+      "blockedBy": [
+        7
+      ]
+    },
+    {
+      "id": 9,
+      "subject": "Task 9: Delete the dormant LiteDB LocalCache subsystem",
+      "status": "completed",
+      "blockedBy": [
+        0
+      ]
+    },
+    {
+      "id": 10,
+      "subject": "Task 10: Health check + telemetry meter allowlist",
+      "status": "completed",
+      "blockedBy": [
+        4
+      ]
+    },
+    {
+      "id": 11,
+      "subject": "Task 11: DI-pin integration tests over the real Program.cs",
+      "status": "completed",
+      "blockedBy": [
+        5,
+        10
+      ]
+    },
+    {
+      "id": 12,
+      "subject": "Task 12: Two-node convergence harness + scenarios (+ positive control)",
+      "status": "completed",
+      "blockedBy": [
+        5,
+        6
+      ]
+    },
+    {
+      "id": 13,
+      "subject": "Task 13: docker-dev rig configuration (site-a pair on, site-b pin off)",
+      "status": "completed",
+      "blockedBy": [
+        5
+      ]
+    },
+    {
+      "id": 14,
+      "subject": "Task 14: Documentation (runbook, Redundancy.md, CLAUDE.md)",
+      "status": "completed",
+      "blockedBy": [
+        8
+      ]
+    },
+    {
+      "id": 15,
+      "subject": "Task 15: DoD sweep (offline) \u2014 STOP and report after this",
+      "status": "completed",
+      "blockedBy": [
+        7,
+        8,
+        9,
+        11,
+        12,
+        13,
+        14
+      ],
+      "notes": "Offline DoD sweep: full-solution build 0 errors (824 pre-existing OTOPCUA0001 analyzer warnings in driver *test* projects; none reference any LocalDb/DeploymentCache file). Grep: no LiteDB/ILocalConfigCache/GenerationSealedCache code refs (2 explanatory doc-comment lines in ILdapGroupRoleMappingService remain, allowed). Runtime.Tests 407/0/31 twice (the previously-flagged intermittent did NOT reproduce). Host.IntegrationTests LocalDb subset 40/0/0. Full-solution `dotnet test` with stash-baseline NOT run: many suites are infra-gated (driver fixtures, full Akka mesh, LDAP real-bind, shared-SQL DB tests) and cannot run offline on macOS \u2014 deferred to Task 16 / CI. Task 12 positive-control evidence recorded in commit afa5be71. Did NOT merge to master \u2014 stopped here per plan."
+    },
+    {
+      "id": 16,
+      "subject": "Task 16: Live gate on the docker-dev rig (needs explicit user go-ahead)",
+      "status": "completed",
+      "blockedBy": [
+        15
+      ],
+      "notes": "Live gate RAN on the docker-dev rig with explicit user go-ahead. 8/8 checks pass (check 4 with a documented limitation). Caught + fixed FOUR real defects offline tests missed (4b2f0e6e NU1101 packageSourceMapping, ce9fa07f ASPNETCORE_HTTP_PORTS re-bind, 9137cb41 empty-address-space-on-cache-boot, c6a9f93a cache-not-repopulated-on-RestoreApplied), each with a regression test. Two documented limitations (follow-ups): no replication back-fill of a fully-wiped node; oplog growth on default-OFF nodes. Post-fix: full build 0 errors, Runtime.Tests 409/0, all LocalDb Host.IntegrationTests green; only consistent failure is the infra-gated AbCip_Green_AgainstSim (sim not up). Evidence: 2026-07-20-localdb-phase1-live-gate.md. NOT merged."
+    }
+  ],
+  "lastUpdated": "2026-07-20T00:00:00Z"
+}

docs/plans/2026-07-20-localdb-adoption-phase2.md

@@ -0,0 +1,316 @@
+# OtOpcUa LocalDb Adoption — Phase 2 Implementation Plan (alarm-historian store-and-forward)
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+>
+> **Execution model:** Optimized for **Claude Opus** agents (`claude --model opus`); dispatch
+> `high-risk` tasks on Opus. Branch **`feat/localdb-phase2`** in `~/Desktop/OtOpcUa` (remote
+> `lmxopcua`). **Prerequisite: Phase 1 (`docs/plans/2026-07-20-localdb-adoption-phase1.md`) is
+> merged (or this branch is stacked on it) and its live gate passed.** Do not merge as part of this
+> plan; stop at the DoD task.
+>
+> **Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md` §3.4,
+> D8, D9. Reference implementation for "replace a bespoke store" phasing: ScadaBridge Phase 2
+> (`~/Desktop/ScadaBridge/docs/plans/2026-07-19-localdb-adoption-phase2.md` + its live gate doc).
+
+**Goal:** Move the alarm-historian store-and-forward buffer (today the standalone
+`alarm-historian.db` owned by `SqliteStoreAndForwardSink`) into the consolidated LocalDb file as a
+replicated table with a primary-gated drain, so a redundant pair no longer loses buffered alarm
+history when a node dies — and delete the sink's bespoke file/connection management outright.
+
+**Architecture:** `alarm_sf_events` (TEXT GUID PK) registered in `LocalDbSetup.OnReady`; the sink
+rewired onto `ILocalDb` behind its unchanged public seam; drain gated on the delivered-snapshot
+Primary role via `PrimaryGatePolicy` (at-least-once across failover, accepted and documented); a
+one-time idempotent migrator from the legacy file, running **after** registration.
+
+**Risk framing (from ScadaBridge):** Phase 2 replaces a *working* mechanism — a harder risk class
+than Phase 1's "add where none existed." Cutover (delete + rewire) lands in **one commit**; there is
+no dual-mechanism period, and the cutover is the test.
+
+**Hard rules:** identical to Phase 1's list (OnReady ordering, no autoincrement/BLOB, fail-closed
+auth already in place, no in-memory SQLite in tests, cp-triplet-only rig inspection), plus:
+- **Legacy-copy column lists must INTERSECT** with what the legacy file actually has
+  (`pragma_table_info` probe) — a missing column throws and readers silently discard every row.
+- Absence assertions need a positive control.
+- DoD greps phrased as "no references from code" (explanatory comments may survive).
+
+---
+
+### Task 0: Recon (produces `docs/plans/2026-07-20-localdb-phase2-recon.md`)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Create: `docs/plans/2026-07-20-localdb-phase2-recon.md`
+- Read-only: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs` (and the
+  whole `Core.AlarmHistorian` project), `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/AlarmHistorianOptions.cs`,
+  the drain worker (whatever forwards batches to `GatewayHistorian`/`SendEvent`),
+  `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/PrimaryGatePolicy.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` (how `_localRole` +
+  driver-member-count reach the gate today)
+
+**Record, with `file:line` citations:**
+1. The sink's exact table schema: name, columns, PK type (**autoincrement?** — decides whether the
+   migrator needs deterministic `mig-{node}-{legacyId}` ids), indices, and any BLOB columns
+   (**STOP condition:** a BLOB payload column cannot be registered — it must become base64 TEXT in
+   the new schema, and the recon must size the largest realistic payload against the 171 KB-ish
+   chunk guidance; alarm events are small JSON, so expect this to be fine, but verify).
+
+   > **PRE-ANSWERED 2026-07-21 (verified against `master` `d218282c`) — the STOP condition does NOT
+   > fire.** Re-verify cheaply during recon, but do not expect a surprise:
+   >
+   > - **Executed DDL:** `SqliteStoreAndForwardSink.cs:657-667`. It matches the class doc-comment at
+   >   `:17-26` exactly — no drift between the documented and executed schema.
+   > - **No BLOB.** All 8 columns are TEXT/INTEGER; the payload column is **`PayloadJson TEXT NOT
+   >   NULL`**. So no base64 conversion is needed and the chunk-size sizing is moot.
+   > - **Payload is small and bounded by shape.** `PayloadJson` is a serialized
+   >   `AlarmHistorianEvent` — 10 scalar fields (`AlarmId`, `EquipmentPath`, `AlarmName`,
+   >   `AlarmTypeName`, `Severity`, `EventKind`, `Message`, `User`, `Comment?`, `TimestampUtc`).
+   >   No collections, no nesting. Realistic worst case is low single-digit KB (operator `Comment`
+   >   is the only unbounded-ish field), far under the 171 KB chunk guidance.
+   > - **PK IS autoincrement** — `RowId INTEGER PRIMARY KEY AUTOINCREMENT`. This confirms the
+   >   plan's own prediction: LocalDb cannot replicate an autoincrement key, so the migrator
+   >   **must** mint deterministic `mig-{node}-{legacyId}` ids, and the new table needs a TEXT
+   >   GUID PK as already specified in the Architecture note.
+   > - **One index to carry across:** `IX_Queue_Drain ON Queue (DeadLettered, RowId)` (`:667`) —
+   >   the drain's covering index. The `alarm_sf_events` equivalent wants the same shape over
+   >   (dead-lettered flag, insertion order) so the drain query stays index-covered.
+   > - **Legacy column list for the migrator's `pragma_table_info` intersection check:**
+   >   `RowId, AlarmId, EnqueuedUtc, PayloadJson, AttemptCount, LastAttemptUtc, LastError,
+   >   DeadLettered`.
+2. The public seam: the interface the drain worker and producers use (e.g. `IAlarmHistorianSink` /
+   enqueue+dequeue+markDelivered+deadLetter methods), so the rewire can keep it byte-compatible.
+3. Semantics to preserve: `Capacity` (1,000,000) enforcement, `MaxAttempts` (10), dead-letter
+   retention (30 d), `BatchSize` (100), `DrainIntervalSeconds` (5) — where each lives.
+4. The drain worker's lifecycle: hosted service or actor? Where a Primary-role check can be
+   injected, and how the delivered-snapshot role (`RedundancyStateChanged` cache) is accessible
+   from it (via `DriverHostActor`, a shared status service, or a message). If the role is only
+   available inside `DriverHostActor`, note the cleanest bridge (e.g. an `IRedundancyRoleView`
+   singleton the actor updates) — that becomes Task 4's shape.
+5. Whether the sink is constructed per-node config path (`AlarmHistorian:DatabasePath`) anywhere
+   else (tests, tooling).
+6. How `AlarmHistorian:Enabled=false` short-circuits (NullAlarmHistorianSink) — the rewire must
+   keep the disabled path allocating no LocalDb tables? No: tables are created unconditionally in
+   `OnReady` (cheap, empty); only the sink/drain stay Null. Note this in the doc.
+
+Commit: `docs(localdb): phase-2 recon findings`.
+
+---
+
+### Task 1: `alarm_sf_events` schema + registration (+ tests)
+
+**Classification:** standard
+**Estimated implement time:** ~4 min
+**Parallelizable with:** none (Task 2 depends on it)
+
+**Files:**
+- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmSfSchema.cs` (depends only on
+  `Microsoft.Data.Sqlite`)
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs`
+- Test: extend `LocalDbSetupTests`
+
+Schema shape (adjust column names to the recon's findings — preserve today's semantics):
+```sql
+CREATE TABLE IF NOT EXISTS alarm_sf_events (
+    id               TEXT    NOT NULL PRIMARY KEY,   -- app-minted GUID (never autoincrement)
+    payload_json     TEXT    NOT NULL,
+    enqueued_at_utc  TEXT    NOT NULL,
+    attempts         INTEGER NOT NULL DEFAULT 0,
+    status           TEXT    NOT NULL DEFAULT 'pending',  -- pending | delivered | dead
+    last_attempt_utc TEXT    NULL,
+    dead_at_utc      TEXT    NULL
+);
+CREATE INDEX IF NOT EXISTS ix_alarm_sf_events_status ON alarm_sf_events(status, enqueued_at_utc);
+```
+`OnReady` order becomes: Phase-1 DDL → `AlarmSfSchema.Apply` → the two Phase-1
+`RegisterReplicated` calls → `RegisterReplicated("alarm_sf_events")` → **migrator (Task 5) last**.
+(All DDL may run before all registrations; the invariant is registration-before-writes.)
+
+TDD: failing test first — exact replicated set becomes
+`["alarm_sf_events", "deployment_artifacts", "deployment_pointer"]` (ordinal-sorted; update the
+Phase-1 exact-set pins in the same commit — they are *supposed* to go red here, that's them
+working). Oplog-capture test for an `alarm_sf_events` insert. Commit
+`feat(localdb): alarm_sf_events replicated table`.
+
+---
+
+### Task 2: Rewire the sink onto `ILocalDb` + delete bespoke file management (the cutover commit, part 1 of 2 — see Task 3)
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs` (or replace
+  with `LocalDbStoreAndForwardSink.cs` — keep the public seam identical either way)
+- Modify: its registration (`AddAlarmHistorian`) to inject `ILocalDb`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/AlarmHistorianOptions.cs` — remove
+  `DatabasePath` (a breaking config key removal: note it in the runbook/CHANGELOG task)
+- Test: the sink's existing unit tests, rewired to a temp-file `ILocalDb` via `TestLocalDb`-style
+  helper (create `tests/.../TestSupport` helper if none exists — real DB, never a stub: a stubbed
+  bare `SqliteConnection` lacks `zb_hlc_next()` and fails closed on registered tables)
+
+**Steps:**
+1. Write/port failing tests for the seam's semantics: enqueue, drain batch of `BatchSize`,
+   `MaxAttempts` → dead-letter, capacity enforcement, dead-letter retention purge.
+2. Implement over `ILocalDb.ExecuteAsync/QueryAsync` (anonymous-object params). Delete the private
+   connection/pragma/file-open code and any `PRAGMA journal_mode` calls (LocalDb owns pragmas).
+   GUIDs minted at enqueue (`Guid.NewGuid().ToString("N")`).
+3. `Capacity` enforcement: count-based insert guard (preserve today's overflow behavior per recon).
+4. Core.AlarmHistorian gains a package ref on core `ZB.MOM.WW.LocalDb` (interface only).
+5. Build + project tests green. **Commit together with Task 3** if the drain gate can't compile
+   separately (the ScadaBridge tasks-14/15/16 circular-dependency landmine — check before assuming
+   they're independent commits).
+
+---
+
+### Task 3: Primary-gated drain
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:** (exact shape from recon item 4)
+- Modify: the drain worker
+- Create (if recon says so): `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/IRedundancyRoleView.cs`
+  — a singleton snapshot (`RedundancyRole? LocalRole`, `int DriverMemberCount`) updated by
+  `DriverHostActor` where it already caches `_localRole`
+- Test: drain-worker tests + an actor test pinning that `DriverHostActor` publishes role changes to
+  the view
+
+**Semantics:**
+- Drain runs only when `PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount)` is
+  true — same policy, same boot-window posture (unknown role drains only when the node is alone).
+- Delivered/dead-letter marks are row UPDATEs → they replicate, so the standby's copy tracks drain
+  progress and does not re-deliver already-marked rows after failover.
+- **At-least-once across failover is accepted:** rows delivered on the old primary whose
+  `delivered` mark hadn't replicated yet will be re-sent by the new primary. Document in the
+  runbook (Task 7); do NOT build dedup.
+- When replication is OFF (default), the gate still applies but `driverMemberCount` for a solo
+  node keeps today's behavior — verify with a test: single-node, role unknown → drains (no
+  regression for unpaired deployments).
+
+TDD: failing tests — secondary role does not drain; primary drains; unknown+alone drains;
+unknown+paired does not. Commit (with Task 2 if coupled):
+`feat(localdb): alarm S&F on LocalDb with primary-gated drain (cutover)`.
+
+---
+
+### Task 4: One-time legacy migrator (`alarm-historian.db` → consolidated)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 6
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/AlarmSfLegacyMigrator.cs`
+- Modify: `LocalDbSetup.OnReady` — call it **last**, after all `RegisterReplicated` calls
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/AlarmSfLegacyMigratorTests.cs`
+
+Mirror `~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs`:
+- Source path from the pre-removal `AlarmHistorian:DatabasePath` default (`alarm-historian.db`) —
+  read the raw config key even though the option property is gone.
+- `ShouldMigrate`: skip if file missing or `<file>.migrated` exists. `:memory:`/`file:` sources → no-op.
+- **If the legacy PK is autoincrement (recon):** deterministic ids `mig-{NodeName}-{legacyId}`
+  (node name from the recon-identified config key) — rerunnable without duplicates under
+  `INSERT OR IGNORE`, and no cross-node collision. If already GUIDs, copy as-is.
+- One transaction for the whole copy; `File.Move(path, path + ".migrated")` only after commit;
+  failure throws out of `OnReady` → boot fails, legacy untouched.
+- **Column intersection** via `pragma_table_info` on the legacy table; required-PK guard.
+- Both pair nodes migrate independently; their rows have distinct ids (node-prefixed), so the
+  merged buffer is the union — expected; note that the new primary will drain the standby's
+  migrated rows too.
+
+TDD: failing tests — happy path row counts; idempotent re-run; failure leaves legacy file
+untouched; migrated rows **enter the oplog** (assert `__localdb_oplog` — the registration-order
+pin); older legacy file missing a column still migrates (intersection). Commit
+`feat(localdb): one-time alarm-historian.db migrator`.
+
+---
+
+### Task 5: Convergence + failover scenarios in the pair harness
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 4
+
+**Files:**
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/AlarmSfConvergenceTests.cs`
+  (reuses the Phase-1 `LocalDbPairHarness`)
+
+**Scenarios:**
+1. `AlarmBurstOnA_ConvergesToB_AndOplogDrains` — enqueue N events on A; identical rowset on B;
+   oplog → 0.
+2. `DeliveredMarksReplicate` — mark rows delivered on A; B's copies show delivered (the
+   no-redeliver-after-failover property, asserted at the data layer).
+3. `WritesWhileTransportDown_SurviveRejoin`.
+4. Positive control: with `RegisterReplicated("alarm_sf_events")` commented out, scenarios 1–3 go
+   red (run locally, record, restore — do not commit red).
+
+Commit `test(localdb): alarm S&F convergence scenarios`.
+
+---
+
+### Task 6: Rig config + docs
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 4
+
+**Files:**
+- Modify: `docker-dev/docker-compose.yml` — enable `AlarmHistorian__Enabled=true` on the site-a
+  pair if not already (the gate needs a live buffer); remove any `AlarmHistorian__DatabasePath`
+  env vars (key deleted).
+- Modify: `docs/operations/2026-07-20-localdb-pair-replication.md` — add: alarm S&F replication
+  semantics, at-least-once-across-failover statement, migrator behavior (`.migrated` sidecar),
+  `DatabasePath` key removal.
+- Modify: `docs/AlarmHistorian.md` + `CLAUDE.md` — sink now lives in the consolidated LocalDb;
+  drain is primary-gated.
+
+Commit `docs+chore(localdb): phase-2 rig config + docs`.
+
+---
+
+### Task 7: DoD sweep (offline)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+1. Full solution build → 0 warnings; full test suite green (deltas vs pre-branch baseline only).
+2. Greps, phrased as "no references from **code**": the old bespoke connection management
+   (`AlarmHistorian:DatabasePath`, direct `new SqliteConnection` inside Core.AlarmHistorian except
+   via schema helpers/tests) — explanatory comments may remain.
+3. Positive-control evidence from Task 5 recorded.
+4. Exact-set replicated-tables pin = 3 tables, both directions.
+5. Update `…phase2.md.tasks.json`; commit `chore(localdb): phase-2 DoD sweep`; STOP and report.
+
+---
+
+### Task 8: Live gate on the docker-dev rig (run only with explicit user go-ahead)
+
+**Classification:** high-risk (rig)
+**Estimated implement time:** ~30 min wall-clock
+**Parallelizable with:** none
+
+Same inspection rules as Phase 1's gate (cp-triplet, curl sidecar, observer-suspicion rule).
+Record in `docs/plans/2026-07-20-localdb-phase2-live-gate.md`:
+1. Migration ran: `.migrated` sidecar present on both site-a nodes; row counts match legacy.
+2. Alarm burst (drive a real driver alarm or the historian-gateway-unreachable path) converges:
+   identical rowsets, oplog drains to 0, dead letters 0.
+3. Only the primary drains: stop the historian gateway egress, buffer builds on BOTH nodes'
+   tables (replicated), but only the primary's drain worker logs attempts.
+4. Failover: stop the primary; the standby (new primary) resumes draining the shared buffer;
+   count of double-delivered events observed and recorded (at-least-once evidence, not a failure).
+5. Both-nodes-together restart clean; zero `disk I/O error`/`SQLITE_IOERR` in logs.
+6. site-b (default-OFF pin): sink works locally, no sync traffic.
+
+Commit the gate doc; report; do not merge.
+
+---
+
+## Task persistence
+
+Tasks file: `docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json`. Record deviations per
+task — especially if Tasks 2/3 had to land as one commit (the expected outcome).

docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json

@@ -0,0 +1,86 @@
+{
+  "planPath": "docs/plans/2026-07-20-localdb-adoption-phase2.md",
+  "tasks": [
+    {
+      "id": 0,
+      "subject": "Task 0: Recon \u2014 sink schema, seam, drain lifecycle, role-view bridge (STOP conditions)",
+      "status": "completed",
+      "note": "STOP condition does NOT fire (no BLOB; PayloadJson TEXT). Recon doc: docs/plans/2026-07-20-localdb-phase2-recon.md. Key finding: the drain worker is an internal Timer inside the sink (not a hosted service/actor), so the gate is a Func<bool> ctor param; and HistorianAdapterActor already primary-gates ENQUEUE with a different policy (ShouldHistorize) - which is why today's ungated drain is safe and why replication breaks it. Deviations D-1..D-4 recorded."
+    },
+    {
+      "id": 1,
+      "subject": "Task 1: alarm_sf_events schema + registration (+ exact-set pin update)",
+      "status": "completed",
+      "blockedBy": [
+        0
+      ],
+      "note": "alarm_sf_events created in AlarmSfSchema (Core.AlarmHistorian) + registered third in LocalDbSetup.OnReady. Exact-set pin updated to 3 tables. DEVIATION D-5: kept the legacy delete-on-ack + dead_lettered flag + last_error rather than the plan's status column (no sweeper for 'delivered' rows; last_error is the only record of why a row died). Drain ORDER BY moves to (enqueued_at_utc, id)."
+    },
+    {
+      "id": 2,
+      "subject": "Task 2: Rewire sink onto ILocalDb + delete bespoke file management (cutover 1/2)",
+      "status": "completed",
+      "blockedBy": [
+        1
+      ],
+      "note": "Sink rewritten as LocalDbStoreAndForwardSink over ILocalDb; bespoke file/pragma/schema management deleted with the old class. AlarmHistorian:DatabasePath removed (breaking config key). Ids are a deterministic payload hash (D-1), not GUIDs."
+    },
+    {
+      "id": 3,
+      "subject": "Task 3: Primary-gated drain via PrimaryGatePolicy (cutover 2/2 \u2014 may co-commit with Task 2)",
+      "status": "completed",
+      "blockedBy": [
+        2
+      ],
+      "note": "Drain gated on IRedundancyRoleView, a singleton DriverHostActor publishes PrimaryGatePolicy's verdict to on every snapshot. Fails closed on a throwing gate; seeded OPEN so a non-redundant deployment is never silently stopped. New HistorianDrainState.NotPrimary + transition-logged (D-2). Landed with Task 2 as one commit (D-3)."
+    },
+    {
+      "id": 4,
+      "subject": "Task 4: One-time alarm-historian.db legacy migrator",
+      "status": "completed",
+      "blockedBy": [
+        3
+      ],
+      "note": "AlarmSfLegacyMigrator runs LAST in OnReady (which now takes IConfiguration - no skip-migration overload exists). DEVIATION D-6: ids are the payload hash (AlarmSfSchema.DeriveId, lifted out of the sink) rather than mig-{node}-{legacyId} - a warm pair's two legacy files OVERLAP, and node-prefixing would carry that duplication forward forever. DEVIATION: tests live in Host.IntegrationTests; the plan's Host.Tests project does not exist."
+    },
+    {
+      "id": 5,
+      "subject": "Task 5: Convergence + failover scenarios in the pair harness (+ positive control)",
+      "status": "completed",
+      "blockedBy": [
+        3
+      ],
+      "note": "4 scenarios green. POSITIVE CONTROL RUN (RegisterReplicated(alarm_sf_events) commented out): 3/4 went red immediately; the 4th (same-event-on-both-nodes) passed VACUOUSLY - with replication off each node trivially held its own single row. Strengthened by enqueuing a second distinct event on B and asserting BOTH nodes hold 2; it then went red under the control too. Control restored, all 4 green. Also fixed a SECOND exact-set pin the plan predicted: LocalDbWiringTests."
+    },
+    {
+      "id": 6,
+      "subject": "Task 6: Rig config + docs",
+      "status": "completed",
+      "blockedBy": [
+        3
+      ],
+      "note": "Rig: AlarmHistorian__Enabled=true on site-a-1/2 AND site-b-1/2, with MaxAttempts raised to 1e6 (D-4: the default 10 would dead-letter the queue ~5min into the rig's permanent gateway outage) and a deliberately unresolvable ServerHistorian__Endpoint. NOTE: the endpoint is REQUIRED - ServerHistorianOptionsValidator is consumer-gated on AlarmHistorian:Enabled, so an empty one fails host start. Docs: runbook, AlarmHistorian.md, AlarmTracking.md, Configuration.md, Redundancy.md, docs/README.md, CLAUDE.md."
+    },
+    {
+      "id": 7,
+      "subject": "Task 7: DoD sweep (offline) \u2014 STOP and report after this",
+      "status": "completed",
+      "blockedBy": [
+        4,
+        5,
+        6
+      ],
+      "note": "Build: 0 errors solution-wide; 0 warnings from every project this branch touches (the ~816 solution-wide warnings are pre-existing xUnit1051/OTOPCUA0001/CS86xx in untouched driver + client test projects). GREPS FOUND REAL DRIFT Task 6 missed: 8 live sites still named the deleted SqliteStoreAndForwardSink - CLAUDE.md, the AdminUI /alarms/historian panel text (user-visible), HistorianAdapterActor (a <see cref> that did NOT warn), Runtime + Host + 2 Driver.Historian.Gateway doc comments, 1 gateway test comment, docs/drivers/Historian.Wonderware.md; plus docs/AlarmTracking.md still claimed a Validate() startup warning for a relative DatabasePath (that branch is gone). All fixed. Code refs to AlarmHistorian:DatabasePath now reduce to exactly two intentional ones: AlarmSfLegacyMigrator.LegacyPathKey and its test. No 'new SqliteConnection' in Core.AlarmHistorian. Both exact-set pins assert the 3-table set, both directions. Guard-deletion evidence (both vacuous passes + the pin inventory) consolidated into the recon doc."
+    },
+    {
+      "id": 8,
+      "subject": "Task 8: Live gate on the docker-dev rig (needs explicit user go-ahead)",
+      "status": "completed",
+      "blockedBy": [
+        7
+      ],
+      "note": "Gate doc: docs/plans/2026-07-20-localdb-phase2-live-gate.md. Checks 1/2/5/6 PASS; checks 3+4 NOT SATISFIED (see below). FOUR production defects found + fixed, three of which crash-looped every driver node before check 1 could run: (1) empty ServerHistorian:ApiKey crashes the host - the validator had explicitly classified it as 'degrades', which is false because the gateway client validates its own options at construction; (2) UseTls vs endpoint scheme mismatch crashes likewise, both directions; (3) plaintext h2c was UNREACHABLE - the adapter forwarded TLS-only options unconditionally so every documented http:// deployment crashed; (4) THE BLOCKER - the drain gate deferred to a CLUSTER-WIDE elected Primary while the queue is PAIR-LOCAL. On the rig that Primary is central-1 (carries the driver Akka role, replicates nobody, runs no historian), so ALL FOUR driver nodes suspended their drains including unpaired site-b: silent permanent loss where pre-Phase-2 drained fine. Fixed in 3 layers (separate ShouldDrainAlarmHistory policy; peer-host matching in DriverHostActor; gate short-circuited entirely when replication is unconfigured, testing BOTH PeerAddress and SyncListenPort since only the dialer sets the former). Also fixed a THIRD vacuous test: AwaitAssert on the seeded-open value passed with the guard deleted; now asserts the sequence of PUBLISHED values via a recording view. Migration evidence: 11 legacy rows across two overlapping files -> exactly 9 identical rows on both nodes, proving D-6's payload-hash identity live. OPEN DESIGN FORK (in the gate doc): a pair cannot identify its own Primary, so both halves drain - safe (no loss, duplicates only) but the gate's de-dup benefit is unrealised."
+    }
+  ],
+  "lastUpdated": "2026-07-21T00:00:00Z"
+}

docs/plans/2026-07-20-localdb-phase1-live-gate.md

@@ -0,0 +1,93 @@
+# LocalDb Phase 1 — live gate (docker-dev rig)
+
+> Task 16 of `2026-07-20-localdb-adoption-phase1.md`. Evidence log for the 8 live-rig checks.
+> Rig: local `docker-dev/docker-compose.yml` (6 host nodes, one shared SQL). site-a pair replicates;
+> central + site-b are default-OFF.
+>
+> **Safety rules followed:** all DB inspection via `docker cp` of the `db`/`-wal`/`-shm` triplet out
+> of the container, querying the copy — never host `sqlite3` on the live WAL file. Metrics via a
+> `curlimages/curl` sidecar with `--network container:<node>` (`aspnet:10.0` has no curl).
+
+## Rig build/up
+
+Two real defects surfaced by the rebuild (both fixed on-branch, re-verified):
+
+1. **`NU1101` — packageSourceMapping gap** (`4b2f0e6e`). `ZB.MOM.WW.LocalDb*` was not mapped to the
+   `dohertj2-gitea` feed, so any *clean* restore (Docker image build, CI, fresh clone) failed. Local
+   dev builds masked it via the warm global NuGet cache. Fixed by adding the two patterns; verified
+   with a temp-packages-dir restore.
+2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`). The Kestrel re-bind read only
+   `urls`/`ASPNETCORE_URLS`; the aspnet:8.0+ base image sets `ASPNETCORE_HTTP_PORTS=8080` (all
+   interfaces) as the container default instead. A driver node that never sets `URLS` fell through to
+   `localhost:5000`, silently moving its health/metrics surface to loopback. Central nodes were fine
+   (they set `ASPNETCORE_URLS=9000` explicitly). Fixed to fall through URLS → HTTP_PORTS/HTTPS_PORTS →
+   5000; 4 new unit tests. After the fix, driver nodes bind `[::]:8080` + `[::]:9001` (site-a) /
+   `[::]:8080` only (site-b).
+
+Rig: 6 nodes rebuilt from branch `feat/localdb-phase1` @ `ce9fa07f`, all up.
+
+## Checks
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 1 | Rig up, site-a pair healthy, `/metrics` shows `localdb_` series | ✅ PASS | Both site-a nodes `/healthz` 200 Healthy; `/metrics` shows `localdb_oplog_depth` + `localdb_sync_reconnects_total` on meter `ZB.MOM.WW.LocalDb.Replication` (allowlist works); site-a-2 served `/localdb_sync.v1.LocalDbSync/Sync` with routing `match_status=success` (interceptor passed, session established); both oplogs depth 0 |
+| 2 | Deploy → both site-a nodes' pointer + artifacts byte-identical, same HLC + origin | ✅ PASS | Deploy `6e687451…` (rev `efb04c79…`) via `POST :9200/api/deployments` → both site-a-1 and site-a-2 `deployment_pointer` = `SITE-A / 6e687451… / efb04c79… / sha EFB04C79…` (identical); artifact 1 chunk each; **pointer `__localdb_row_version` identical on both: hlc `116955620225646592`, origin node_id `a0576df6-…`, tombstone 0** — one row replicated, not two derivations; both oplogs depth 0 |
+| 3 | Boot-from-cache: SQL down, restart site-a-2, serves cached config w/ signal | ✅ PASS (found + fixed a defect) | With SQL stopped, site-a-2 restart logged `RUNNING FROM CACHE — … booted deployment 6e687451…`; **address space materialised from the cached blob**: `AddressSpaceApplier: applied plan (added=18)` / `OpcUaPublish: applied rebuild (added=18)`. Browse confirms **16 nodes on site-a-2 = 16 on site-a-1**. **Defect the gate caught (`9137cb41`):** pre-fix the rebuild re-read the artifact from the down ConfigDb and no-op'd, so the cache-booted node served **1** node vs the healthy peer's 16 — it logged success but browsed empty. Fixed by passing the in-hand cached blob to `RebuildAddressSpace`; regression test added |
+| 4 | Replicated-cache payoff: wipe a-2 DB, repopulate, then SQL-down boot | ⚠️ PARTIAL — 1 defect fixed; the limitation is since **CLOSED** (LocalDb 0.1.2 — see Summary) | Wiped a-2's LocalDb volume, restarted with **SQL down**: sync session re-established (a-2 served a `Sync` call) but a-2's cache stayed **0/0/0** and a-1's oplog was **0** — the deploy's rows had been acked by the old a-2 and pruned, so there was no delta to send and **no snapshot-resync fired**. ⇒ **Limitation: pure replication does not back-fill a fully-wiped, already-converged node** (library delta-replication; snapshot-resync is gated on the oplog cap, which the running engine never hits). Separately found a **defect (`c6a9f93a`)**: recovery via `RestoreApplied` never wrote the cache (only fresh `ApplyAndAck` did), so a wiped node stayed cache-less. Fixed: `RestoreApplied` now re-caches. Verified — after the fix, wiped a-2 restarted with SQL up: `restored served state … on bootstrap` → cache repopulated `SITE-A / 6e687451`, chunks=1. **Net: a wiped node self-heals its cache from central on next boot (regaining boot-from-cache for future outages); it is not back-filled by its peer during the same outage.** |
+| 5 | Transport kill: pause a-2, oplog rises; unpause, drains to 0 | ✅ PASS | `docker pause` on a-2 severed the sync transport; during the window a-2 accumulated **oplog=2** unacked local writes (its re-cache) while a-1 held 0. `docker unpause` → a-2's oplog **drained to 0 within 5s**, both converged on the same pointer. Note: the rig's redeploys carry an identical revision, so a fresh cache write on a-1 short-circuits (no-op) — the store-during-partition variant is covered by the automated harness scenario `WritesWhileTransportDown_SurviveRejoin` (proven red without `RegisterReplicated`) |
+| 6 | Retention: 3rd deploy prunes oldest; chunks gone + tombstones on BOTH nodes | ✅ PASS | Cached 3 distinct deployment-ids (via deploy + restart re-cache). Both nodes retain exactly the newest 2 (`475df87a`, `962962dc`); the oldest `6e687451`'s chunks are **0 on both** (pruned); **1 artifact tombstone on both** — the prune replicated as a tombstone, not left as a live chunk |
+| 7 | Both-nodes-together restart: clean rejoin, identical counts, no I/O errors | ✅ PASS | Restarted site-a-1 + site-a-2 together: **0** `disk I/O error`/`SQLITE_IOERR`/`corrupt` lines in either log; both re-cached and converged to **identical** counts (ptr `962962dc…`, chunks=2, rowver=3). Byte-identical convergence preserved across a simultaneous restart |
+| 8 | site-b default-OFF pin: cache works locally, Healthy, no sync listener on 9001 | ✅ PASS (+1 finding, since **CLOSED** by the skip-if-unchanged cache write — see Summary) | site-b-1 cache works locally (`SITE-B / 6e687451`, chunks=1, from its own apply); listeners **`[::]:8080` only — no 9001**; port 9001 **not bound** (`/proc/net/tcp`); `/healthz` **200**; no sync/reconnect metric series. **Finding (follow-up, not a blocker):** `localdb_oplog_depth=5` — because `OnReady` registers replication unconditionally, a default-OFF node captures cache writes into its oplog but has no peer to ack/prune them, so the oplog grows slowly (≈2 rows/deploy, amplified by the re-cache-on-restore fix). Bounded by the library `MaxOplogRows` cap (1M); worth a periodic pruner or skip-if-unchanged in a follow-up |
+
+## Summary
+
+**8/8 checks pass** (check 4 with a documented limitation). The gate exercised the real image built
+from the branch, the real interceptor over a real loopback+cross-container h2c transport, and real
+`docker cp`-triplet DB inspection — and caught **four real defects that every offline test had
+missed**, each now fixed on-branch with a regression test:
+
+1. **`NU1101` packageSourceMapping gap** (`4b2f0e6e`) — `ZB.MOM.WW.LocalDb*` unmapped to the gitea
+   feed; every clean restore (Docker, CI, fresh clone) failed. Masked locally by the warm NuGet cache.
+2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`) — the sync-listener re-bind ignored the
+   aspnet:8.0+ container-default port var, silently moving driver health/metrics to loopback:5000.
+3. **Empty address space on boot-from-cache** (`9137cb41`) — the rebuild re-read the artifact from the
+   down ConfigDb and no-op'd, so a cache-booted node logged success but served clients **1 node vs the
+   healthy peer's 16**. The single most important find: boot-from-cache was half-working.
+4. **Cache not repopulated on `RestoreApplied`** (`c6a9f93a`) — a wiped/fresh-volume node recovered its
+   served state from central but stayed cache-less, unable to boot-from-cache on the next outage.
+
+**Two documented limitations (follow-ups, not blockers) — both since CLOSED:**
+
+- **No replication back-fill of a fully-wiped node** (check 4): a peer's already-acked cache rows are
+  pruned from its oplog and snapshot-resync is gated on the (never-hit) oplog cap, so a wiped node is
+  not healed by its peer — it self-heals from central on next boot instead.
+  → **FIXED in `ZB.MOM.WW.LocalDb` 0.1.2 + 0.1.3** (scadaproj `cad3bcb` + `3b7489a`), live-gated in
+  `2026-07-21-localdb-followups-live-gate.md`. The diagnosis in this doc was
+  slightly off: the cap was not the gate. `ComputeSnapshotRequiredAsync` measured the peer's gap
+  against the *oldest surviving oplog row* and read an empty oplog as "no gap possible" — and an
+  empty oplog is the **steady state of a converged pair**, since ack-pruning deletes everything the
+  peer has confirmed. So the healthy state was the one state that could not heal a wiped peer. With
+  an empty oplog the gap is now measured against `last_acked_seq`. Covered at both levels:
+  `SnapshotResyncTests.WipedPeer_AfterEverythingWasAckedAndPruned_StillGetsSnapshot` in the library,
+  and `LocalDbPairConvergenceTests.WipedNode_IsBackFilledByItsPeer_WithoutAnyNewDeploy` here —
+  the latter verified RED against the pinned 0.1.1 and green on 0.1.2. **0.1.3 closes a second,
+  deeper half that only became reachable once back-fill worked:** the rebuilt node's OWN writes were
+  silently dropped, because `last_applied_remote_seq` is a watermark in the *peer's* seq space and a
+  rebuilt peer numbers from 1 again. The follow-up live gate caught that one — offline tests could
+  not have, since the scenario had no test while it was still a documented limitation.
+- **Oplog growth on default-OFF nodes** (check 8): `OnReady` registers replication unconditionally, so
+  a node with no peer accumulates unacked/unpruned oplog rows (~2/deploy). Slow, bounded by the 1M cap;
+  a periodic pruner or a skip-if-unchanged cache write would close it.
+  → **FIXED at the source**: `LocalDbDeploymentArtifactCache.StoreAsync` now **skips entirely** when
+  the pointer already names this deployment/revision, the SHA matches the bytes, and the expected
+  chunk count is present — so re-caching an artifact the node already holds (every restart's
+  boot-from-cache, every `RestoreApplied`) writes nothing and mints no oplog rows. Two guard tests
+  pin what the skip must NOT do (stale bytes under an unchanged revision; a matching pointer over
+  missing chunks), both verified RED against a naive pointer-only skip.
+  Also correcting this row's bound: the growth was never headed for the 1M cap. `AddZbLocalDbReplication`
+  is registered unconditionally, so `MaintenanceBackgroundService` runs on default-OFF nodes too and
+  the **7-day `MaxOplogAge`** cap prunes — the true bound was ~7 days of writes, not 1M rows.
+
+Merged to `master` as `c957db52` (2026-07-21) and pushed. Both follow-ups closed on
+`fix/localdb-phase1-followups` (2026-07-21) and live-gated — see
+`2026-07-21-localdb-followups-live-gate.md`, which found a third defect in the process.

docs/plans/2026-07-20-localdb-phase1-progress.md

@@ -0,0 +1,72 @@
+# LocalDb Phase 1 — Execution Progress (resume state)
+
+**Branch:** `feat/localdb-phase1` (off `master`). Working tree CLEAN, all work committed, nothing pushed.
+**Plan:** `docs/plans/2026-07-20-localdb-adoption-phase1.md`
+**Recon:** `docs/plans/2026-07-20-localdb-phase1-recon.md` (READ THIS — it has the 8 deviations D-1..D-8 that override the plan text).
+**Skill in use:** `superpowers-extended-cc:executing-plans` — batch, verify, commit per task, report between batches.
+**Do NOT merge to master** — plan stops at Task 15 (DoD) and reports; Task 16 is a rig live-gate needing explicit user go-ahead.
+
+## Status: Tasks 0–9 DONE (10 commits). Tasks 10–16 REMAIN.
+
+| # | Task | State | Commit |
+|---|---|---|---|
+| 0 | Preflight + recon | ✅ | `42f85507` |
+| 1 | Package refs (LocalDb 0.1.1, Grpc.AspNetCore 2.76.0) | ✅ | `44255fcb` |
+| 2 | Schema + LocalDbSetup.OnReady + tests | ✅ | `3b65c24b` |
+| 3 | Fail-closed sync auth interceptor + tests | ✅ | `3d29be83` |
+| 4 | LocalDbRegistration + Program.cs DI + appsettings | ✅ | `6aff9a83` |
+| 5 | h2c sync listener + MapZbLocalDbSync (Kestrel) | ✅ | `b9ddf20e` |
+| (fix) | rename Runtime.Deployment → Runtime.DeploymentCache | ✅ | `e771a11a` |
+| 6 | IDeploymentArtifactCache + chunked impl + tests | ✅ | `a38a52b8` |
+| 9 | Delete LiteDB LocalCache | ✅ | `2bae1b4c` |
+| 7 | Cache write path in DriverHostActor | ✅ | `1becf591` |
+| 8 | Boot-from-cache + RunningFromCache signal | ✅ | `a27eff32` |
+| 10 | Health check + telemetry meter allowlist | ⬜ TODO | |
+| 11 | DI-pin integration tests over real host graph | ⬜ TODO | |
+| 12 | 2-node convergence harness + scenarios | ⬜ TODO | |
+| 13 | docker-dev rig config | ⬜ TODO | |
+| 14 | Documentation | ⬜ TODO | |
+| 15 | DoD sweep (offline) — STOP + report | ⬜ TODO | |
+| 16 | Live gate on docker-dev rig — NEEDS USER GO-AHEAD | ⬜ TODO | |
+
+Native task list (TaskCreate ids 1–17 map to plan tasks 0–16) tracks the same; tasks.json in plan dir mirrors it.
+
+## What exists now (key files)
+
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/DeploymentCacheSchema.cs` — DDL, namespace `Runtime.DeploymentCache`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactCache.cs` — `StoreAsync` / `GetCurrentAsync` / **`GetCurrentUnkeyedAsync`** (D-1 addition) + `CachedDeploymentArtifact` record.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/LocalDbDeploymentArtifactCache.cs` — chunked impl. NOTE the prune guard `AND deployment_id <> @DeploymentId` (bug I found: pointer-target could be pruned under same-tick GUIDs → permanent silent miss).
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs` — `OnReady` (DDL→RegisterReplicated). PUBLIC (Host has no InternalsVisibleTo).
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs` — `AddOtOpcUaLocalDb`, `SyncListenPort(config)`. Registers `IDeploymentArtifactCache → LocalDbDeploymentArtifactCache` singleton.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs` — fail-closed bearer.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/KestrelHttpBinding.cs` — URL parse/re-bind helper (D-6).
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` — LocalDb wired in `hasDriver` branch (~L176 after AddAlarmHistorian: `AddOtOpcUaLocalDb` + `AddGrpc(interceptor)`); Kestrel re-bind block before `builder.Build()`; `MapZbLocalDbSync()` gated `hasDriver && syncListenPort>0` before `MapOtOpcUaHealth()`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json` — `LocalDb` section (Path, SyncListenPort:0, Replication{}), between AlarmHistorian and Deployment.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` — cache dep threaded LAST in Props/ctor/forwarding; `CacheAppliedArtifact` (write, after PushDesiredSubscriptions, empty-blob skip, own try/catch); `TryBootFromCache` + `ApplyCachedArtifact` (in Bootstrap catch); `PushDesiredSubscriptionsFromArtifact` split out; `_isRunningFromCache` field; `SingleClusterCacheKey="__single"`; `ReconcileDrivers` now returns `byte[]?`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs` — resolves `IDeploymentArtifactCache` (optional, nullable) and passes into DriverHostActor.Props.
+- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Interfaces/NodeDiagnosticsSnapshot.cs` — added `bool RunningFromCache = false`.
+- Tests (all green): `Host.IntegrationTests/LocalDbSetupTests.cs`, `LocalDbSyncAuthInterceptorTests.cs`, `LocalDbSyncListenerTests.cs`; `Runtime.Tests/Deployment/LocalDbDeploymentArtifactCacheTests.cs`, `Runtime.Tests/Drivers/DriverHostActorArtifactCacheTests.cs`, `DriverHostActorBootFromCacheTests.cs`.
+
+## Load-bearing facts for the remaining tasks
+
+- **Namespace trap (recurs!):** any `Deployment` child namespace under `Runtime` OR `Runtime.Tests` shadows the `Deployment` EF entity → CS0118 across the assembly. Use `...DeploymentCache`. Only a FULL-SOLUTION build catches it (per-project builds pass).
+- **xUnit1051 is an error** in these test projects: every `ILocalDb`/DB call in a test must pass `TestContext.Current.CancellationToken` (xunit.v3 Host.IntegrationTests) — but `Runtime.Tests` is **xunit v2** (no `TestContext.Current`).
+- **Test project reality (D-5):** NO `Host.Tests` project; NO `WebApplicationFactory<Program>` (deliberately — `TwoNodeClusterHarness.cs:48` explains why). Host unit tests live in `Host.IntegrationTests` (xunit.v3). Actor tests in `Runtime.Tests` (xunit v2 + Akka.TestKit.Xunit2).
+- **Task 10 (D-7):** `AddOtOpcUaHealth()` takes NO args, called unconditionally at `Program.cs:365`. DON'T change its signature — register the LocalDb check unconditionally and return `Healthy` when LocalDb absent (ActiveNodeHealthCheck precedent). Health check classes: there are ZERO in-repo today; `Host/Health/` has only `HealthEndpoints.cs`. Meter allowlist is REAL and REQUIRED: `Host/Observability/ObservabilityExtensions.cs:30` `o.Meters = [OtOpcUaTelemetry.MeterName]` — add `LocalDbMetrics.MeterName` (from `ZB.MOM.WW.LocalDb.Replication`). Health check reads `ISyncStatus` + `IOptions<ReplicationOptions>`.
+- **Task 11:** copy the `TwoNodeClusterHarness` direct-host-build pattern, NOT WebApplicationFactory. Pins: `ILocalDb` singleton + `ReplicatedTables` == `["deployment_artifacts","deployment_pointer"]`; `IDeploymentArtifactCache`→concrete; `ISyncStatus` default-OFF (`Connected==false`); admin-only graph has NO `ILocalDb` + doesn't demand `LocalDb:Path`; health check registered. Set `OTOPCUA_ROLES` per case; `LocalDb:Path`→temp file, `ClearAllPools`+delete triplet in cleanup.
+- **Task 12:** harness copied from `~/Desktop/ScadaBridge/tests/.../LocalDbSitePairHarness.cs` + `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/tests/.../Convergence/ConvergenceFixture.cs`. Both nodes init via PRODUCTION `LocalDbSetup.OnReady`. Real loopback Kestrel h2c through the REAL interceptor. DBs as pre-constructed singletons. Positive control: comment out both `RegisterReplicated`, confirm scenarios 1–3 red, restore (don't commit red).
+- **Task 13 (D-8):** NO host node has a writable volume today — must ADD named volumes. `docker-dev/docker-compose.yml` services: `central-1/2` (admin,driver, MAIN), `site-a-1/2` (driver, SITE-A), `site-b-1/2` (driver, SITE-B). Env style `Section__Key`. `9001` free container-internal. Enable replication on **site-a pair only** (a-1 initiator w/ PeerAddress, a-2 passive, byte-identical ApiKey, MaxBatchSize=16). site-b stays OFF = default-OFF pin. Validate `docker compose config`; DON'T bring rig up.
+- **Task 15:** baseline pre-existing test failures FIRST (`git stash` → run full `dotnet test` → unstash), report deltas only. **KNOWN intermittent:** one `Runtime.Tests` failure seen ~3× across runs (mine + subagent's), never reproduced under trx logger, NOT one of the new tests — investigate/baseline here.
+- **Warnings-as-errors** across src + most test projects. Pre-existing CS0618/xUnit1051 warnings exist ONLY in `Client.Shared*` (which don't set TreatWarningsAsErrors) — not mine, ignore.
+- **SQLitePCLRaw:** no pin needed — LocalDb 0.1.1 nuspec floors `lib.e_sqlite3` at 2.1.12 (CVE-fixed). Confirmed.
+
+## Follow-ups logged (NOT in scope, don't action without asking)
+
+- `Polly.Core` in `Configuration.csproj` now orphaned (ResilientConfigReader was its only real consumer). Left in place — recorded in recon §7.3b.
+- Pre-existing: `TryRecoverFromStale` marks Applied w/o reconciling drivers (recon §3.5); `Install-Services.ps1` gives driver-only nodes no `ASPNETCORE_URLS` (recon §6.4, worked around by D-6).
+
+## Verify-anything-quick
+
+Full build: `dotnet build ZB.MOM.WW.OtOpcUa.slnx` (expect 0 errors). LocalDb tests:
+`dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LocalDb"` and
+`dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~ArtifactCache|FullyQualifiedName~BootFromCache"`.

docs/plans/2026-07-20-localdb-phase1-recon.md

@@ -0,0 +1,617 @@
+# LocalDb Phase 1 — Task 0 Recon Findings
+
+**Date:** 2026-07-20
+**Branch:** `feat/localdb-phase1` (based on `master`)
+**Plan:** `docs/plans/2026-07-20-localdb-adoption-phase1.md`
+**Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md`
+
+All `file:line` citations are against the branch base commit.
+
+---
+
+## Step 1 — Feed verification: PASS
+
+```
+$ dotnet package search ZB.MOM.WW.LocalDb --source dohertj2-gitea --format json
+ZB.MOM.WW.LocalDb              0.1.1
+ZB.MOM.WW.LocalDb.Contracts    0.1.1
+ZB.MOM.WW.LocalDb.Replication  0.1.1
+```
+
+All three present at **0.1.1**. No STOP. (Note: the local NuGet cache holds both `0.1.0` and `0.1.1`
+— pin exactly `0.1.1` so a stale cache cannot win.)
+
+---
+
+## Step 3 — STOP conditions: NONE TRIGGERED
+
+| STOP condition | Verdict |
+|---|---|
+| Artifact apply path cannot take a post-apply hook without restructuring the actor | **Clear** — `ApplyAndAck` has a clean insertion point (§2.2) |
+| `DeploymentId` / cluster identity are not stable strings/GUIDs | **Clear** — both are stable strings, but cluster identity is **not resolvable at boot**; see D-1 |
+| LiteDB LocalCache is referenced by live code | **Clear** — zero live-code references (§7) |
+
+Three STOP conditions clear. **Eight deviations** from the plan's stated assumptions are recorded in
+§9 — four of them (D-1, D-3, D-5, D-6) change the work materially.
+
+---
+
+## 1. Deploy fetch path
+
+### 1.1 Blob-load sites — there are **five** `CreateDbContext()` calls, not two
+
+| Line | Method | Purpose |
+|---|---|---|
+| `DriverHostActor.cs:513` | `Bootstrap()` | `NodeDeploymentStates` + `Deployments` at PreStart |
+| `DriverHostActor.cs:1472` | `ReconcileDrivers(DeploymentId)` | **loads `ArtifactBlob`** |
+| `DriverHostActor.cs:1543` | `PushDesiredSubscriptions(DeploymentId)` | **loads `ArtifactBlob` again** |
+| `DriverHostActor.cs:2000` | `TryRecoverFromStale()` | latest `Sealed` deployment (id + hash only, no blob) |
+| `DriverHostActor.cs:2029` | `UpsertNodeDeploymentState(...)` | writes `NodeDeploymentState` |
+
+`DriverHostActor.cs:1472-1476`:
+```csharp
+using var db = _dbFactory.CreateDbContext();
+blob = db.Deployments.AsNoTracking()
+    .Where(d => d.DeploymentId == deploymentId.Value)
+    .Select(d => d.ArtifactBlob)
+    .FirstOrDefault() ?? Array.Empty<byte>();
+```
+`:1543-1547` is byte-identical apart from the local. **The same blob is read twice per apply.**
+
+### 1.2 Artifact runtime type — there is no instance type
+
+`DeploymentArtifact` (`DeploymentArtifact.cs:48`) is a **static parser class** over
+`ReadOnlySpan<byte>`. The blob is never materialised into an object graph.
+
+- Storage: `byte[]` — `Deployment.cs:29` `public byte[] ArtifactBlob { get; init; } = Array.Empty<byte>();`
+- Format: **UTF-8 JSON**, `JsonDocument.Parse` (`DeploymentArtifact.cs:66`, `:490`). Not MessagePack.
+- Leniency contract: empty/malformed blob → empty result, **never throws** (`:62`, `:537-541`).
+
+**Consequence for the cache:** we cache raw `byte[]`. No serializer coupling. Good.
+
+### 1.3 Identity types
+
+| Type | Definition | Shape |
+|---|---|---|
+| `DeploymentId` | `Commons/Types/DeploymentId.cs:3` — `readonly record struct DeploymentId(Guid Value)` | `ToString()` → `Value.ToString("N")`, 32 hex chars, **no hyphens** (`:10`) |
+| `RevisionHash` | `Commons/Types/RevisionHash.cs:8` — `readonly record struct RevisionHash(string Value)` | SHA-256 hex, **lowercase 64 chars**, no `0x`; empty invalid (`:3-6`) |
+
+Both are stable, filesystem/SQL-safe strings. Use `DeploymentId.ToString()` (the "N" form) as the
+`deployment_id` TEXT column value.
+
+---
+
+## 2. Post-apply point
+
+### 2.1 `ApplyAndAck` control flow — `DriverHostActor.cs:1413-1458`
+
+```
+1415  _applyingDeploymentId = deploymentId;
+1416  Become(Applying);
+1425  UpsertNodeDeploymentState(..., Applying, null)
+1427  try {
+1429      ReconcileDrivers(deploymentId);            // blob fetch #1 — SWALLOWS DB errors
+1430      _currentRevision = revision;
+1431      UpsertNodeDeploymentState(..., Applied, null);
+1432      SendAck(..., ApplyAckOutcome.Applied, null, correlation);
+1436      _opcUaPublishActor?.Tell(RebuildAddressSpace(...));
+1439      PushDesiredSubscriptions(deploymentId);    // blob fetch #2 — SWALLOWS DB errors
+1440      OtOpcUaTelemetry.DeploymentApplied.Add(1, "outcome"="ack");
+1441      _log.Info("applied deployment ...");
+1443  }
+1444  catch (Exception ex) { ...Failed ACK... }
+1452  finally { _applyingDeploymentId = null; Become(Steady); }
+```
+
+### 2.2 Where the cache write belongs
+
+**Immediately after `DriverHostActor.cs:1439`**, before `:1440` — but **wrapped in its own
+try/catch**. Reason: `:1432` already sent an `Applied` ACK to the coordinator. An unguarded cache
+write that throws would fall into the `catch` at `:1444` and send a *second*, contradictory
+`Failed` ACK for a deployment the fleet already believes is applied.
+
+### 2.3 Paths that must NOT write to the cache
+
+1. **`catch` at `:1444`** — any apply failure.
+2. **Idempotent short-circuit at `:1402-1409`** — `HandleDispatchFromSteady` returns early with an
+   `Applied` ACK when `_currentRevision == msg.RevisionHash`. `ApplyAndAck` is never entered and no
+   blob is read.
+3. **⚠ THE SILENT-DEGRADATION TRAP.** `ReconcileDrivers:1478-1483` and
+   `PushDesiredSubscriptions:1549-1553` catch DB failures, log a **warning**, and `return` **without
+   rethrowing**. `ApplyAndAck` therefore reaches `:1440` and declares success having applied **zero
+   drivers** when the blob load failed. Caching "whatever we read" would **persist an empty artifact
+   as a successful apply** — poisoning the boot-from-cache path with a config that starts no drivers.
+
+   **Mitigation (load-bearing):** the cache write must be gated on the blob having actually loaded
+   non-empty. Have `ReconcileDrivers` surface the loaded blob upward (a field or an out-param)
+   rather than re-reading it a third time, and skip the cache write when it is
+   `Array.Empty<byte>()`. This also removes the duplicate read noted in §1.1.
+
+4. **`RestoreApplied` (`:1514-1528`)** — the bootstrap replay path. Restoring from central; caching
+   here is harmless and arguably desirable, but it is a distinct seam. **Phase 1 scope: skip it.**
+5. **`TryRecoverFromStale` (`:1996-2022`)** — never reads the artifact at all (see §3.4).
+
+---
+
+## 3. Cold-boot path
+
+### 3.1 PreStart — `DriverHostActor.cs:415-436`
+
+Ctor already did `Become(Steady)` at `:411`. PreStart subscribes to topics, spawns the virtual-tag
+and scripted-alarm hosts, then calls `Bootstrap()` at `:435`.
+
+### 3.2 `Bootstrap()` — `DriverHostActor.cs:506-565` — **yes, it queries central SQL at boot**
+
+Two queries inside one `try`: latest `NodeDeploymentState` for self (`:514-518`), then the matching
+`Deployment` row (`:527-532`). Four outcomes: no prior deployments → Steady; `Applied` →
+`RestoreApplied` (`:544`); `Applying` orphan → `ApplyAndAck` replay (`:549`); `Failed` → Steady.
+
+### 3.3 **THE SEAM** — `DriverHostActor.cs:560-564`
+
+```csharp
+catch (Exception ex)
+{
+    _log.Warning(ex, "DriverHost {Node}: ConfigDb unreachable on bootstrap; entering Stale", _localNode);
+    Become(Stale);
+}
+```
+
+**This is the exact point where "central fetch failed at boot" is known.** The cache fallback goes
+between `:562` and `:563`: attempt a local-cache load, and only `Become(Stale)` if the cache is also
+a miss.
+
+⚠ The catch also fires if the *second* query throws after the first succeeded, so `latest` is not in
+scope-valid state from the catch. **The cache must be self-describing** — it must carry its own
+`DeploymentId` + `RevisionHash`, because at this seam the actor has no idea what it should be
+restoring. The planned `deployment_pointer` shape already satisfies this.
+
+### 3.4 The `Stale` state — `DriverHostActor.cs:1343-1379`
+
+`ReconnectInterval = 30s` (`:55`), retry timer started at `:1378`.
+
+The dispatch-ignore at **`:1345-1348`** discards `DispatchDeployment` entirely — no ACK, no
+buffering (contrast `Applying:599` which does `Self.Forward(msg)`). A Stale node looks like a
+timeout to the coordinator. Phase 1 does **not** change this (per design §3.3: a *new* deployment
+still requires central).
+
+`RouteNodeWrite` fast-fails at `:1359-1360` with `"driver host stale (config DB unreachable)"`.
+
+### 3.5 Pre-existing gap worth recording (not Phase 1 scope)
+
+`TryRecoverFromStale` (`:1996-2022`) sets `_currentRevision` and marks `Applied` **without calling
+`ReconcileDrivers` or `PushDesiredSubscriptions`** — documented at `:1692-1695` and `:1706`. A
+Stale-recovered node reports Applied while running **zero drivers**, and
+`HandleDispatchFromSteady:1402` then short-circuits on revision match. Boot-from-cache largely
+neutralises this in practice, but the bookkeeping remains optimistic. **Logged as a follow-up, not
+fixed here.**
+
+### 3.6 `DbHealthProbeActor`
+
+**No interaction with `DriverHostActor` whatsoever.** Spawned as a sibling
+(`ServiceCollectionExtensions.cs:243-246`), consumed only by `OpcUaPublishActor`
+(`OpcUaPublishActor.cs:105`, `:256`, `:528`, `:543-544`). `DriverHostActor` runs its own independent
+30s `retry-db` timer. If running-from-cache should later influence OPC UA ServiceLevel,
+`OpcUaPublishActor`'s DB-health seam is the precedent to mirror.
+
+---
+
+## 4. Cluster identity — **the one real design problem (see D-1)**
+
+**There is no `ClusterId` field on `DriverHostActor` and no accessor for it.** The actor knows only
+`_localNode` (`DriverHostActor.cs:61`, type `Commons.Types.NodeId`, aliased at `:28`).
+
+`ClusterId` is resolved **per-artifact**, by `DeploymentArtifact.ResolveClusterScope(blob, nodeId)`
+(`DeploymentArtifact.cs:485-516`) — it scans the artifact's `Nodes[]` array for a matching `NodeId`
+and reads that element's `ClusterId`. Types: `ClusterScope` (`:46`), `ClusterFilterMode` (`:28`).
+
+`IClusterRoleInfo` (`Commons/Interfaces/IClusterRoleInfo.cs:12`) exposes `LocalNode`, `LocalRoles`,
+`HasRole`, `MembersWithRole`, `RoleLeader`, `RoleLeaderChanged` — **no `ClusterId`**.
+`docker-dev/docker-compose.yml` has `Cluster__Hostname`/`Port`/`Roles`/`SeedNodes` — **no
+`Cluster__ClusterId`**. The `ClusterNode` entity
+(`Configuration/Entities/ClusterNode.cs:7,10`) carries both, but lives in central SQL — exactly
+what is unreachable at the seam that needs it.
+
+> **The bind:** the design keys the cache by `ClusterId` so a replicated pair shares one cache
+> entry. But `ClusterId` is only discoverable *from a blob you already have* or *from central SQL*.
+> At the §3.3 boot-failure seam we have neither.
+
+**Resolution adopted (D-1):** keep `cluster_id` as the `deployment_pointer` PK — pair sharing is the
+whole point of replication and must not be given up. Resolve it at each end as follows:
+
+- **Write path** (`ApplyAndAck`, §2.2): call `DeploymentArtifact.ResolveClusterScope(blob, nodeId)`
+  on the blob just applied. `ClusterFilterMode.ScopeTo` → use its `ClusterId`. `None` (single-cluster
+  artifact) or `Suppress` → fall back to the literal `"__single"` sentinel, matching the actor's own
+  existing degenerate-case convention at `:1827-1830`.
+- **Read path** (boot seam, §3.3): we cannot compute the key, so **do not key the read**. Select the
+  single `deployment_pointer` row, ordered by `applied_at_utc DESC`, `LIMIT 1`. A node belongs to
+  exactly one cluster and its pair peer replicates the same cluster's row, so the table holds one
+  row in every real topology. If more than one row is present (a node was re-homed between
+  clusters), take the newest and **log a warning naming both cluster ids** — an operator-visible
+  signal rather than a silent wrong-config boot.
+- Optional escape hatch: honour a `Cluster:ClusterId` config key when present. **Deferred** — not
+  needed for the rig, and adding a required key contradicts the design's "storage ships
+  unconditionally" posture.
+
+This preserves the replication payoff (design §3.3) and the schema exactly as specified, and
+confines the change to the read query's `WHERE` clause.
+
+---
+
+## 5. `DriverHostActor` construction — the `Props` expression tree
+
+### 5.1 The factory — `DriverHostActor.cs:326-346`
+
+`static Props(...)` takes **16 parameters**, 14 optional, and forwards them **fully positionally**
+into `Akka.Actor.Props.Create(() => new DriverHostActor(...))` at **`:343-346`**. Ctor at
+`:375-412`, assignments `:393-408`.
+
+> ⚠ Six `IActorRef?` parameters and three interface-typed parameters mean many mis-bindings are
+> **type-compatible and therefore compile clean**. Callers use *named* arguments (safe); the
+> internal forwarding at `:343-346` is *positional* (unsafe).
+>
+> **Rule for Task 7: append the new parameter LAST in the `Props` signature, LAST in the ctor
+> signature, and LAST in the positional forwarding list at `:346`. Change nothing else.**
+
+### 5.2 Construction sites — **27 total** (1 production + 26 test)
+
+**Production (1):** `Runtime/ServiceCollectionExtensions.cs:332`, inside `WithOtOpcUaRuntimeActors`
+(`:193`); registered as `DriverHostActorKey` (`:344`). All args named except the first three. Does
+not pass `virtualTagHostOverride`, `scriptedAlarmHostOverride`, or `driverMemberCountProvider`.
+
+`WithOtOpcUaRuntimeActors` callers: `Host/Program.cs:331`,
+`Host.IntegrationTests/TwoNodeClusterHarness.cs:398`, `ServiceCollectionExtensionsTests.cs:42,144`.
+
+**Tests (26)**, all in `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/`:
+
+| File | Lines |
+|---|---|
+| `Drivers/DriverHostActorTests.cs` | 28, 60, 81, 118, 143, 189 |
+| `Drivers/DriverHostActorReconcileTests.cs` | 37, 59, 90, 119, 156, 193 |
+| `Drivers/DriverHostActorDiscoveryTests.cs` | 385, 671, 748, 1023 |
+| `Drivers/DriverHostActorWriteRoutingTests.cs` | 148, 208 |
+| `Drivers/DriverHostActorPrimaryGateTests.cs` | 228, 247 |
+| `Drivers/DriverHostActorVirtualTagTests.cs` | 44, 76 |
+| `Drivers/DriverHostActorNativeAlarmTests.cs` | 350 |
+| `Drivers/DriverHostActorNativeAlarmAckRoutingTests.cs` | 126 |
+| `Drivers/DriverHostActorProbeResultDropTests.cs` | 66 |
+| `Drivers/DriverHostActorHistoryWriterTests.cs` | 59 |
+| `Drivers/DiscoveryInjectionEndToEndTests.cs` | 219 |
+| `VirtualTags/DependencyMuxActorTests.cs` | 139 |
+
+No site calls `Props.Create<DriverHostActor>()` or `new DriverHostActor(...)` directly — the only
+`new DriverHostActor` in the tree is inside the factory at `:343`. Because the new parameter is
+optional and appended last, **no test site should need editing.**
+
+### 5.3 Status object for the running-from-cache signal
+
+`NodeDiagnosticsSnapshot` (`Commons/Interfaces/NodeDiagnosticsSnapshot.cs:17-22`) — built in
+`HandleGetDiagnostics` (`DriverHostActor.cs:1381-1398`), a positional record of
+`(NodeId, RevisionHash? CurrentRevision, IReadOnlyList<DriverInstanceDiagnostics> Drivers, DateTime AsOfUtc)`.
+`GetDiagnostics` is handled in **all three states** (`Steady:570`, `Applying:601`, `Stale:1349`), so
+a flag here is observable regardless of state. **This is the natural home for `RunningFromCache`**
+(append last to the record). Consumer: `IFleetDiagnosticsClient` → AdminUI.
+
+`IDriverHealthPublisher` (`:73`, defaulted `:402`) is **not** a fit — the host never calls it; it
+only passes it down to `DriverInstanceActor` children (`:1845`, `:1859`). Per-driver, not per-node.
+
+### 5.4 Async / side-effect IO convention
+
+The actor is **overwhelmingly synchronous-blocking on DB IO** — all five `CreateDbContext()` sites
+use `using var db = ...` with synchronous LINQ on the actor thread. No `async`/`await` in the file,
+no `Task.Run`.
+
+The single async pattern is `ContinueWith(..., TaskScheduler.Default).PipeTo(replyTo)` in
+`HandleRouteNodeWrite` (`:1188-1200`), with an explicit `Sender` capture before the continuation.
+
+**Recommendation for Task 7:** a **synchronous** cache write at the §2.2 point is *consistent with
+the file's existing style* and is correct there — the actor is mid-apply, holds `Applying`, and
+every other DB call on that path already blocks. A fire-and-forget `PipeTo(Self)` would need a
+handler registered in `Steady`, `Applying` **and** `Stale`, or it dead-letters after the
+`Become(Steady)` at `:1456`. **Prefer synchronous + try/catch.** (This deviates from the plan's
+"fire-and-forget"; see D-4.)
+
+`IWithTimers` is already implemented (`:50`, `:282`); `"retry-db"` key in use at `:1378`/`:2009`.
+
+---
+
+## 6. Host composition root
+
+### 6.1 Role flag — `Program.cs:47-50`
+
+```csharp
+var roles = RoleParser.Parse(Environment.GetEnvironmentVariable("OTOPCUA_ROLES"));
+var hasAdmin = roles.Contains("admin");
+var hasDriver = roles.Contains("driver");
+```
+Allowed roles: `admin`, `driver`, `dev` (`Cluster/RoleParser.cs:5-8`); unknown throws (`:25-27`).
+
+### 6.2 `hasDriver` branch — `Program.cs:131-310`
+
+Anchor for the new call: **`Program.cs:173-175`** `builder.Services.AddAlarmHistorian(...)`, inside
+the config-gated cluster at `:154-195`. Must land **before** `AddAkka` at `:314`.
+
+### 6.3 Pipeline — `Program.cs:368-404`
+
+`:383` `MapStaticAssets` · `:385-396` `if (hasAdmin) { ... }` · **`:398` `MapOtOpcUaHealth()`** ·
+`:399` `MapOtOpcUaMetrics()` · `:404` `RunAsync()`.
+
+⚠ **There is no `hasDriver` block in the pipeline half** — only `hasAdmin` is branched on. A
+driver-gated `app.Map*` is a new construct; it goes between `:396` and `:398`.
+
+`Program.cs:406-410` re-exports `public partial class Program`.
+
+### 6.4 Kestrel / URLs — **CONFIRMED: no Kestrel configuration exists**
+
+- `Program.cs:57` `builder.WebHost.UseStaticWebAssets();` — the **only** `builder.WebHost` call.
+- **Zero** `ConfigureKestrel` / `UseUrls` / `ListenAnyIP` / `UseKestrel` anywhere in `src/`.
+- Binding is exclusively via `ASPNETCORE_URLS`: `docker-compose.yml:173` (central-1) and `:240`
+  (central-2) set `http://+:9000`; `launchSettings.json:7` sets `http://localhost:9000`.
+
+Two traps for Task 5:
+
+1. ⚠ **`scripts/install/Install-Services.ps1:186`** (`$hostEnv += "ASPNETCORE_URLS=..."`) is inside
+   `if ($hasAdmin)` at `:185`. **A driver-only Windows-service node gets no `ASPNETCORE_URLS` at
+   all** and binds the ASP.NET default. The plan's parse-URLs-then-rebind snippet would re-bind
+   `http://+:9000` there — a port that node never listened on. The `?? "http://+:9000"` fallback in
+   the plan's snippet is therefore **wrong for driver-only nodes**; see D-6.
+2. ⚠ **`Host.IntegrationTests/TwoNodeClusterHarness.cs:331`** already calls
+   `builder.WebHost.UseKestrel(o => o.Listen(IPAddress.Parse(LoopbackHost), 0));`. Adding a
+   production `ConfigureKestrel` gives two competing explicit configurations in integration tests —
+   last-one-wins. The `syncPort > 0` gate keeps this dormant by default (harness sets no sync port),
+   but any harness that *does* set one will fight the port-0 binding.
+
+**Health route for smoke tests:** `/healthz` — liveness, runs no checks, always 200 while the
+process is up, `AllowAnonymous`. (`Health/HealthEndpoints.cs:48` → `MapZbHealth()`; routes from the
+`ZB.MOM.WW.Health` package.) Also `/health/ready`, `/health/active`, and `/metrics` (`Program.cs:399`).
+
+### 6.5 `SecretsRegistration` template — `Host/Configuration/SecretsRegistration.cs`
+
+Shape for `LocalDbRegistration` to mirror: `public static class` in
+`ZB.MOM.WW.OtOpcUa.Host.Configuration`; `public const string` section/key path constants; a
+`public static bool IsXEnabled(IConfiguration)` predicate using
+`GetValue(key, defaultValue: false)` (**default-deny**) that `Program.cs` can also call;
+`AddOtOpcUaX(this IServiceCollection, IConfiguration)` returning `IServiceCollection`, both args
+`ArgumentNullException.ThrowIfNull`-guarded; XML docs with `<remarks><para>` explaining *why* the
+gate is default-deny.
+
+Call-site precedent: `Program.cs:363` `AddOtOpcUaSecrets(...)`; predicate reused inside the `AddAkka`
+lambda at `Program.cs:323`.
+
+### 6.6 Telemetry meter allowlist — **EXISTS. Task 10 step 3 is REQUIRED, not conditional.**
+
+`Host/Observability/ObservabilityExtensions.cs:25-38`:
+```csharp
+return services.AddZbTelemetry(o =>
+{
+    o.ServiceName = "otopcua";
+    o.Meters = [OtOpcUaTelemetry.MeterName];          // <-- ObservabilityExtensions.cs:30
+    o.ActivitySources = [OtOpcUaTelemetry.ActivitySourceName];
+```
+
+`o.Meters` is a **strict allowlist** — `ZbTelemetryExtensions.cs:51-54` iterates it calling
+`metrics.AddMeter(name)`; **no wildcard support**. Anything unlisted is silently not exported.
+Single element today: `OtOpcUaTelemetry.MeterName` = `"ZB.MOM.WW.OtOpcUa"`
+(`Commons/Observability/OtOpcUaTelemetry.cs:19`).
+
+**Action:** add `LocalDbMetrics.MeterName` at `ObservabilityExtensions.cs:30`. This is exactly the
+omission the ScadaBridge live gate caught.
+
+### 6.7 Health registration — `Host/Health/HealthEndpoints.cs:19-41`
+
+`AddOtOpcUaHealth()` **takes no parameters** and is called **unconditionally** at `Program.cs:365`,
+outside both role blocks. Three `.AddTypeActivatedCheck<>` calls (`configdb`, `akka`,
+`admin-leader`); **no registration-time role gating exists** — `ActiveNodeHealthCheck` does runtime
+role scoping instead (returns `Healthy` when the node lacks the role).
+
+⚠ **There are zero `IHealthCheck` implementations in this repo** — all three come from the
+`ZB.MOM.WW.Health.*` packages (`Directory.Packages.props:124-126`, v0.1.0). A LocalDb check would be
+the **first health-check class in the tree**; `Host/Health/` holds only `HealthEndpoints.cs` today.
+
+**Consequence for Task 10:** driver-gating the new check requires changing
+`AddOtOpcUaHealth()`'s signature (add `bool hasDriver` or `IConfiguration`) and its `Program.cs:365`
+call site. **Preferred alternative:** follow the `ActiveNodeHealthCheck` precedent — register
+unconditionally and have the check return `Healthy` when LocalDb is not registered. That keeps the
+signature and matches the "default-OFF must not degrade a plain node" requirement for free. See D-7.
+
+### 6.8 Config files
+
+| File | Top-level sections |
+|---|---|
+| `appsettings.json` | `Serilog`, `Security`, `Secrets`, `ServerHistorian`, `ContinuousHistorization`, `AlarmHistorian`, `Deployment` |
+| `appsettings.driver.json` / `.admin.json` / `.admin-driver.json` / `.Development.json` | `Serilog`, `Security` only |
+
+Overlay selection: `Program.cs:67` sorts roles ordinally and joins with `-` → `admin,driver`
+becomes `appsettings.admin-driver.json`, added `optional: true` at `:70`; env vars (`:77`) and CLI
+args (`:78`) are re-appended after, so deployment overrides outrank the overlay.
+
+**`LocalDb` key search: CONFIRMED ABSENT.** Case-insensitive `localdb` across all `*.json`,
+`*.yml`, `*.ps1`, `*.csproj`, `*.props` matches only the two Phase-1/Phase-2 planning artifacts. No
+conflicting keys in any overlay or in `Install-Services.ps1`.
+
+**Placement convention:** each feature section opens with a `"_comment"` string then `"Enabled":
+false`; per-field caveats use sibling `"_<Field>Comment"` keys (`appsettings.json:34`, `:45`). Place
+`LocalDb` between `AlarmHistorian` (ends `:61`) and `Deployment` (`:62`).
+
+---
+
+## 7. LiteDB LocalCache — dormant, safe to delete
+
+### 7.1 Files (6) under `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/`
+
+`ILocalConfigCache.cs`, `GenerationSnapshot.cs`, `StaleConfigFlag.cs`, `LiteDbConfigCache.cs`
+(also declares `LocalConfigCacheCorruptException` at `:128`), `GenerationSealedCache.cs`,
+`ResilientConfigReader.cs`. Single namespace `...Configuration.LocalCache`. The only two
+`using LiteDB;` statements in `src/` are `LiteDbConfigCache.cs:1` and `GenerationSealedCache.cs:1`.
+
+### 7.2 Reference classification — **no live-code references. NO STOP.**
+
+- **(c) live code elsewhere: ZERO.** Also confirms full dormancy: `SealedBootstrap` (referenced by
+  `docs/v2/v2-release-readiness.md:46`) has **zero hits** in `src/` and `tests/` — it no longer exists.
+- Two benign near-misses:
+  - `Configuration/Services/ILdapGroupRoleMappingService.cs:14` — the string `ResilientConfigReader`
+    inside an **XML doc comment**. Prose only. Needs a comment edit or the doc goes stale.
+  - `Configuration/ZB.MOM.WW.OtOpcUa.Configuration.csproj:29` — `<PackageReference Include="LiteDB"/>`,
+    the only one in the repo.
+- Dozens of `LocalConfigCacheCorruptException` hits under `src/**/bin/**/*.xml` are **generated doc
+  build artifacts** (`GenerateDocumentationFile=true`), not source.
+
+### 7.3 ⚠ Test-file drift — the plan's delete list is incomplete
+
+Plan `…phase1.md:484-485` names only `GenerationSealedCacheTests.cs` and
+`ResilientConfigReaderTests.cs`. Actual test files in
+`tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/`:
+
+| File | Status |
+|---|---|
+| `GenerationSealedCacheTests.cs` | in plan ✅ |
+| `ResilientConfigReaderTests.cs` | in plan ✅ — note `StaleConfigFlagTests` is co-located here at `:323` |
+| **`LiteDbConfigCacheTests.cs`** | **MISSING from the plan** — will fail to compile if the source is deleted (`:125` calls `new LiteDB.LiteDatabase(...)` directly) |
+
+See D-3.
+
+### 7.3b Outcome (Task 9, executed 2026-07-20)
+
+Deleted: the 6 `LocalCache/` sources and **all three** test files (including
+`LiteDbConfigCacheTests.cs`, which the plan omitted — D-3). `LiteDB` removed from both
+`Configuration.csproj` and `Directory.Packages.props`. The stale XML-doc reference at
+`ILdapGroupRoleMappingService.cs:14` was rewritten to state the present truth (no fallback exists;
+reviving it means an admin-side cache on LocalDb, not restoring the old pipeline) rather than
+deleted, per the DoD's "explanatory prose may remain". Configuration tests: 92/92 green.
+
+**Follow-up found while executing:** `Polly.Core` in `Configuration.csproj` is now orphaned too —
+`ResilientConfigReader` was its only real consumer in that project (remaining `Polly` mentions are
+comments). Left in place deliberately: removing it is outside Task 9's scope and could break a
+consumer relying on the transitive flow. Worth a separate cleanup.
+
+### 7.4 Package removal
+
+- `PackageReference`: `Configuration.csproj:29` — only consumer.
+- `PackageVersion`: `Directory.Packages.props:35` — `LiteDB` `5.0.21`.
+
+Both safe to delete; no transitive consumer.
+
+---
+
+## 8. Test projects, rig, packages
+
+### 8.1 Test project layout
+
+| Question | Answer |
+|---|---|
+| `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/` | **DOES NOT EXIST** — there is no Host *unit*-test project at all |
+| Closest Host project | `Host.IntegrationTests` — despite the name it carries plenty of pure unit tests (`LdapOptionsValidatorTests`, `ServerHistorianOptionsValidatorTests`, `ResilienceInvokerFactoryRegistrationTests`, …) |
+| `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/` | **EXISTS** |
+| DriverHostActor tests | `Runtime.Tests/Drivers/` — 11 `DriverHostActor*Tests.cs` files. **xunit v2** (`xunit` 2.9.3), references **`Akka.TestKit.Xunit2`** ✅ |
+| `Host.IntegrationTests` xunit version | **xunit.v3** 1.1.0; SDK `Microsoft.NET.Sdk.Web` |
+
+`Directory.Packages.props` documents that AdminUI.Tests / ControlPlane.Tests / **Runtime.Tests** are
+the three xunit-v2 holdouts, held there by `Akka.TestKit.Xunit2` (verified 2026-07-13: no xunit.v3
+Akka TestKit exists). This matches the design's "actor tests live in an xunit2 TestKit project".
+
+### 8.2 ⚠ `WebApplicationFactory<Program>` is **deliberately not used** — Task 11 must be rewritten
+
+The only mention of the type in the entire `tests/` tree is a comment explaining why it is avoided:
+
+`Host.IntegrationTests/TwoNodeClusterHarness.cs:48`
+```
+/// <para>Why not <c>WebApplicationFactory&lt;Program&gt;</c>? Program.cs reads <c>OTOPCUA_ROLES</c> ...
+```
+
+The actual pattern builds the host directly — `TwoNodeClusterHarness.cs:329`:
+```csharp
+var builder = WebApplication.CreateBuilder(new WebApplicationOptions { Args = [] });
+```
+`Microsoft.AspNetCore.Mvc.Testing` is still referenced (csproj `:15`) but only for transitive
+test-host plumbing. **Copy `TwoNodeClusterHarness`, not `WebApplicationFactory`.** See D-5.
+
+`Host.IntegrationTests` also has its own `docker-compose.yml` (sql `14331`, ldap `3894`) gated by
+`DockerFixtureAvailability.cs`.
+
+### 8.3 docker-dev rig topology — plan's site-a/site-b assumption is **correct**
+
+Single Akka mesh (`otopcua`), seeded by `central-1`. Tenant separation is by `ServerCluster.ClusterId`
+rows (MAIN / SITE-A / SITE-B) inside **one shared ConfigDb** — not separate meshes.
+
+| Service | Roles | Cluster |
+|---|---|---|
+| `central-1`, `central-2` | `admin,driver` | MAIN |
+| `site-a-1`, `site-a-2` | `driver` | SITE-A |
+| `site-b-1`, `site-b-2` | `driver` | SITE-B |
+| `sql`, `migrator`, `cluster-seed`, `traefik` | — | infra |
+
+**There is no admin-only service** — the "admin" nodes are fused `admin,driver`. All six host nodes
+share the `&otopcua-host` YAML anchor.
+
+⚠ **NO HOST NODE HAS A WRITABLE VOLUME.** The only `volumes:` in the file are `sql` →
+`otopcua-mssql-data`, plus two read-only binds (`./seed:ro`, `./traefik-dynamic.yml:ro`). All six
+nodes write to the container's ephemeral layer, destroyed on recreate. This is already an accepted
+pattern (per-container `Secrets:SqlitePath=otopcua-secrets.db`, replicated over Akka rather than
+persisted). **Task 13 must add six named volumes** — there is nothing to piggyback on. See D-8.
+
+**Env style: confirmed double-underscore `Section__Key`**, with array indices and dictionary keys as
+a third segment (`Cluster__SeedNodes__0`, `Security__Ldap__GroupToRole__ReadOnly`). Exceptions that
+are plain SCREAMING_SNAKE (read directly, not config-bound): `OTOPCUA_ROLES`,
+`ZB_SECRETS_MASTER_KEY`, `GALAXY_MXGW_API_KEY`, `OTOPCUA_CONFIG_CONNECTION`, `ASPNETCORE_URLS`.
+
+**Ports.** Host-published: `14330`→sql, `4840`-`4845`→the six nodes' OPC UA, `9200`→traefik web,
+`8089`→traefik dashboard. Container-internal on every node: `4053` (Akka remoting), `9000`
+(Kestrel), `4840` (OPC UA). Also reserved on this machine: `14331`/`3894` (Host.IntegrationTests
+compose), `80`/`8080` (sibling scadabridge/scadalink stack), `10.100.0.35:3893` (shared GLAuth).
+
+**`9001` (the plan's choice) is free** container-internal and need not be published. ✅
+
+### 8.4 Package versions — `Directory.Packages.props`
+
+| Package | Version | Line |
+|---|---|---|
+| `Grpc.Core.Api` | 2.76.0 | 32 |
+| `Grpc.Net.Client` | 2.76.0 | 33 |
+| **`Grpc.AspNetCore`** | **NOT PRESENT** | — |
+| `Google.Protobuf` | 3.34.1 | 31 |
+| `Microsoft.Data.Sqlite` | 10.0.7 | ~55 |
+| `SQLitePCLRaw.bundle_e_sqlite3` | 2.1.12 | ~108 |
+| `LiteDB` | 5.0.21 | 35 |
+
+Both Grpc floors and the Protobuf floor are already satisfied — **no bumps needed**, only the new
+`Grpc.AspNetCore` entry (use 2.76.0 to match the train; watch for NU1605/NU1608 if it wants a
+Protobuf newer than the pinned 3.34.1).
+
+⚠ **`SQLitePCLRaw` is a surgical direct pin, not a transitive one.**
+`CentralPackageTransitivePinningEnabled` is deliberately **off** (it breaks the Roslyn version
+split). The 2.1.12 pin is promoted by a **direct `PackageReference` in `Core.AlarmHistorian`**
+(`Core.AlarmHistorian.csproj:15` + `:21`) — today the only Sqlite consumer in `src/`.
+
+> **Any new project that pulls `Microsoft.Data.Sqlite` MUST also take a direct
+> `<PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3"/>`,** or it silently resolves the
+> vulnerable 2.1.11 native bundle. This applies to **Host** and **Runtime** (both gain `ZB.MOM.WW.LocalDb`,
+> which depends on `Microsoft.Data.Sqlite`) and to every test project that builds a real `ILocalDb`.
+> Copy the `Core.AlarmHistorian.csproj:15,21` pair exactly. The plan's Task 1 mentions
+> `SQLitePCLRaw.lib.e_sqlite3` "if the audit flags it" — the correct package is
+> **`bundle_e_sqlite3`**, and it is **required, not conditional**.
+
+---
+
+## 9. Deviations from the plan
+
+Recorded per the tasks.json `deviation` convention. **D-1, D-3, D-5, D-6 change the work
+materially.**
+
+| # | Plan says | Reality | Resolution |
+|---|---|---|---|
+| **D-1** | Cache keyed by `ClusterId`; "cluster identity" is a recon item with a STOP condition | `ClusterId` is stable, but **not resolvable at the boot seam** — it lives inside the artifact or in the unreachable central DB (§4) | Keep `cluster_id` as PK (pair sharing is the point). **Write:** resolve via `DeploymentArtifact.ResolveClusterScope`, `"__single"` sentinel for the degenerate case. **Read:** unkeyed `ORDER BY applied_at_utc DESC LIMIT 1`, warn if >1 row. No STOP. |
+| **D-2** | Cache write "fire-and-forget (`PipeTo`-style or guarded `Task.Run`)" | The actor has **no async DB IO at all**; a `PipeTo(Self)` needs handlers in all three states or it dead-letters after `Become(Steady)` (§5.4) | Synchronous write in its own try/catch at `:1439`. Matches the file's style; failure still cannot fail the apply. |
+| **D-3** | Task 9 deletes 2 test files | **3** test files reference the subsystem — `LiteDbConfigCacheTests.cs` is missing from the list (§7.3) | Delete all three. Also fix the stale XML-doc mention at `ILdapGroupRoleMappingService.cs:14`. |
+| **D-4** | Cache the artifact after a successful apply | `ReconcileDrivers`/`PushDesiredSubscriptions` **swallow** DB errors, so a "successful" apply can have loaded an **empty blob** and started zero drivers (§2.3) | Gate the write on a non-empty blob; surface the blob from `ReconcileDrivers` instead of a third read. **Load-bearing — without it the cache can be poisoned with an empty config.** |
+| **D-5** | Task 11 uses `WebApplicationFactory<Program>` | Deliberately **not used** in this repo; `TwoNodeClusterHarness.cs:48` documents why (Program.cs reads `OTOPCUA_ROLES` from the environment) (§8.2) | Rewrite Task 11 against the `TwoNodeClusterHarness` direct-host-build pattern. |
+| **D-6** | Task 5 re-binds `ASPNETCORE_URLS ?? "http://+:9000"` | `Install-Services.ps1:185-186` sets `ASPNETCORE_URLS` **only when `hasAdmin`** — the fallback would bind a driver-only node to a port it never served (§6.4) | Re-bind only when a URL is actually configured; when absent, add **only** the sync listener. Also note `TwoNodeClusterHarness.cs:331` already calls `UseKestrel(Listen(…, 0))`. |
+| **D-7** | Task 10 driver-gates the health check | `AddOtOpcUaHealth()` takes no args and is called unconditionally at `Program.cs:365`; no registration-time gating exists anywhere (§6.7) | Register unconditionally; return `Healthy` when LocalDb is absent — the `ActiveNodeHealthCheck` precedent. Avoids a signature change and satisfies "default-OFF must not degrade a plain node". |
+| **D-8** | Task 13 "existing data mount" | **No host node has any writable volume** (§8.3) | Add six named volumes. Note: `9001` is free container-internal. Meter allowlist (§6.6) is **required**, not conditional. |
+
+### Follow-ups logged, not fixed here
+
+1. **`TryRecoverFromStale` marks `Applied` without reconciling drivers** (§3.5) — pre-existing,
+   documented in-code at `:1692-1695`. Out of Phase 1 scope.
+2. **Duplicate blob read per apply** (`:1472` and `:1543`, §1.1) — D-4's fix removes one of them as a
+   side effect.
+3. **`Install-Services.ps1` gives driver-only nodes no `ASPNETCORE_URLS`** (§6.4) — pre-existing;
+   D-6 works around it rather than fixing it.

docs/plans/2026-07-20-localdb-phase2-live-gate.md

@@ -0,0 +1,180 @@
+# LocalDb Phase 2 — live gate (docker-dev rig)
+
+> Task 8 of `2026-07-20-localdb-adoption-phase2.md`. Evidence log.
+> Rig: local `docker-dev/docker-compose.yml` — 2 central + site-a pair (replication ON) + site-b pair
+> (default-OFF), one Akka cluster, one shared SQL.
+>
+> **Safety rules followed** (same as the Phase 1 gate): all DB inspection via `docker cp` of the
+> `db`/`-wal`/`-shm` triplet out of the container, querying the copy — never host `sqlite3` on the
+> live WAL file.
+
+## Headline
+
+**The gate did not get past its own first boot before finding real defects.** Four production bugs,
+all fixed on-branch with regression tests, plus a fifth finding in the test suite itself. Three of
+the four crash-looped every driver node; the fourth silently stopped alarm history everywhere.
+
+Checks 1, 2, 5 and 6 pass. **Checks 3 and 4 are NOT satisfied** — see "The blocker" below; they
+cannot be run as written on a rig that has no per-pair redundancy roles, and the defect they were
+meant to confirm turned out to be the opposite of what the plan assumed.
+
+## Seeding the migration
+
+The rig had no pre-consolidation `alarm-historian.db`, so one was synthesised per node and `docker
+cp`-ed to `/app/alarm-historian.db` (the WORKDIR-relative default) into created-but-not-started
+containers — the exact upgrade shape.
+
+Deliberately overlapping content, to test D-6 in production: **6 rows on site-a-1, 5 on site-a-2,
+2 of them byte-identical payloads** (the warm-pair duplication `HistorianAdapterActor` produces in
+every boot window). 11 legacy rows, **9 distinct**.
+
+## Defects found and fixed
+
+### 1. Empty `ServerHistorian:ApiKey` crash-loops the host (was classified "degrades")
+
+`AlarmHistorian:Enabled=true` requires the gateway alarm writer, which requires a key.
+`HistorianGatewayClientOptions.Validate()` throws `The gateway API key must not be empty` inside the
+client factory during Akka startup, so every driver node crash-looped.
+
+`ServerHistorianOptionsValidator` exists precisely to convert that class of failure into a named
+`OptionsValidationException` — but its documented fail tier explicitly excluded `ApiKey`, reasoning
+that a keyless client "degrades rather than crashes (the gateway rejects calls)". **That premise is
+false**: the client validates its own options at construction, so the process dies before any call.
+Fixed; the key is never echoed in the message.
+
+### 2. `UseTls` / endpoint-scheme mismatch crash-loops the host
+
+Immediately after fixing #1, the rig crash-looped again: `UseTls` defaults to `true`, the rig
+endpoint is `http://`, and the client throws `UseTls requires an https gateway endpoint`. The
+inverse (`https://` + `UseTls=false`) throws too — both strings confirmed in the shipped client
+assembly. Setting an `http` endpoint without clearing `UseTls` is an ordinary migration slip that
+takes the host down. Now validated in both directions.
+
+### 3. Plaintext h2c was *unreachable* — every `http://` deployment crashed
+
+Third crash-loop, and a genuine product bug rather than operator config.
+`HistorianGatewayClientAdapter.Create` forwarded the TLS-only options unconditionally, and
+`AllowUntrustedServerCertificate` defaults to `false`, so it always sent
+`RequireCertificateValidation=true` — which the client rejects outright when `UseTls=false`
+(`RequireCertificateValidation is a TLS-only option and requires UseTls=true`).
+
+CLAUDE.md and `docs/Historian.md` both document `http://` as the supported way to select h2c, but
+**no h2c configuration could start**, and the only workaround was to claim
+`AllowUntrustedServerCertificate=true` — a certificate posture for a connection with no certificate.
+Fixed: TLS-only options stay at their defaults when `UseTls=false`.
+
+### 4. THE BLOCKER — the drain gate deferred to a Primary that cannot deliver
+
+With the rig finally up, **every driver node logged `Historian drain suspended`. Nothing drained
+anywhere** — including the two site-b nodes, which have no redundancy peer and no replication at all.
+Before Phase 2 this deployment drained fine, because the drain was ungated.
+
+Root cause, established from the rig rather than guessed:
+
+- `RedundancyStateActor` derives roles from Akka's **cluster-wide** `RoleLeader("driver")`.
+- `central-1`/`central-2` carry **both** `admin` and `driver` Akka roles (`OTOPCUA_ROLES=admin,driver`).
+- So the elected driver Primary is **central-1** — a node that replicates nobody's LocalDb and does
+  not even run the alarm historian.
+- Every site node was `Secondary`, saw that a Primary existed, and stood down in its favour.
+
+**The redundancy role is a cluster-wide election; the alarm queue is pair-local.** Deferring on
+"somebody is Primary" is therefore wrong in principle: in a multi-pair fleet the Primary is usually
+in another pair and holds none of this node's rows. The consequence is not a duplicate — it is the
+buffer growing on every node until the capacity wall evicts the oldest events, i.e. silent permanent
+loss of the audit trail the queue exists to protect.
+
+Fixed in three layers, each with a positive control:
+
+1. `PrimaryGatePolicy.ShouldDrainAlarmHistory(role, queueSharingPeerIsPrimary)` — a **separate**
+   policy from `ShouldServiceAsPrimary`. Unknown role ⇒ drain. A node stands down only for a Primary
+   that holds its rows. The two gates now deliberately disagree, and a test pins that disagreement so
+   nobody "harmonises" them back.
+2. `DriverHostActor` publishes that verdict, matching the snapshot's Primary against this node's
+   configured LocalDb replication peer host.
+3. `AddAlarmHistorian` short-circuits the gate entirely when replication is not configured — a node
+   whose rows exist nowhere else must never defer. The predicate tests **both** `Replication:PeerAddress`
+   and `SyncListenPort`, because only the dialing half sets the former and both halves share the queue.
+
+The asymmetry of the error costs is what drives every one of these: a false allow costs a duplicate
+history row, which the design already accepts (*at-least-once across a failover, by design*, and
+payload-hash ids collapse identical events); a false deny loses data silently.
+
+### 5. A third vacuous test — `AwaitAssert` on the seeded value
+
+Deleting the guard (publishing the old write-gate verdict) left `DriverHostActorRoleViewTests`
+**green**. `AwaitAssert` polls until an assertion passes, and the assertion was "the view reads
+open" — which is also the value the view is *seeded* with, so it succeeded at the first poll, before
+the actor had processed anything. It could not distinguish a correct publish from no publish at all.
+
+Fixed by asserting on the **sequence of published values** via a recording view, never on current
+state. Re-run under the control: red, for exactly the cases that matter. This is the same
+"absence-assertion passes by race" trap recorded for #485/#486, in a new disguise.
+
+## Checks
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 1 | Migration ran; `.migrated` sidecar on both site-a nodes; row counts match | ✅ PASS | `alarm-historian.db.migrated` present on both. 11 legacy rows across the two files → **exactly 9 rows on each node**, id sets byte-identical — the 2 shared payloads collapsed, proving D-6's payload-hash identity in production. Each node holds the *other's* rows (`legacy/a1-only-*` present on a-2 and vice versa). Legacy `attempt_count` distinction survived the copy (the 3-attempt row still leads the 0-attempt rows), `last_error` carried, 0 dead-lettered |
+| 2 | Alarm burst converges: identical rowsets, oplog drains to 0, dead letters 0 | ✅ PASS | Identical id sets on both nodes; `__localdb_oplog` depth **0** on both; **0** dead-lettered. Drain visibly live against the deliberately unresolvable gateway — `attempt_count` climbing 33 → 42 → 168 with `last_error='retry-please'`, which is exactly the buffering behaviour the queue exists for |
+| 3 | Only the primary drains | ❌ **NOT SATISFIED** — see defect 4 | As written the check assumes a per-pair Primary exists. On this rig the elected Primary is a central node outside both pairs, so the honest post-fix behaviour is that **all four nodes drain**. No node can currently identify a queue-sharing Primary: site-b is unpaired, and of the site-a pair only the dialing half knows its peer. Safe (no loss, duplicates only), but the check's intent is unproven |
+| 4 | Failover: standby resumes draining; count double-delivered events | ⛔ **NOT RUN** | Superseded by defect 4. A meaningful failover test needs a rig where the pair itself elects a Primary; measuring "double delivery" is meaningless while both halves drain unconditionally |
+| 5 | Both nodes restarted together: clean rejoin, identical counts, no I/O errors | ✅ PASS | `docker compose restart site-a-1 site-a-2` → **0** `disk I/O error` / `SQLITE_IOERR` / `corrupt` lines and 0 fatals in either log; both back to 9 rows / 0 dead / oplog 0, still identical |
+| 6 | site-b default-OFF pin: sink works locally, no sync traffic | ✅ PASS | `alarm_sf_events` created and registered on both site-b nodes; port **9001 not bound** (`/proc/net/tcp`), no sync listener, no replication config; both drain their own queue (defect 4's third fix) with 0 fatals |
+
+## The open design question
+
+The gate leaves one question that is a design fork, not a bug to patch:
+
+**A pair cannot currently identify its own Primary.** Redundancy roles are elected cluster-wide, but
+the queue is pair-local. Only the dialing half of a pair knows its peer (`Replication:PeerAddress`);
+the listening half sets `SyncListenPort` only. So the drain gate can never fully engage, and both
+halves of a replicated pair drain — safe, but duplicated.
+
+### Follow-up analysis — the root cause is wider than the drain gate
+
+Two things assumed while writing the section above turned out to be wrong, and correcting them
+changes the recommendation.
+
+**The rig is not misconfigured.** `central-1`/`central-2` carry `admin,driver` deliberately — the
+compose says so in as many words ("central is admin,driver"), because they are themselves a
+redundant pair that also runs the MAIN cluster's drivers. Stripping the driver role would change what
+the rig models, not fix anything.
+
+**A "pair" is not a missing concept — it already exists, as the application Cluster.** OtOpcUa's
+`Cluster`/`ClusterId` (the config-DB entity that owns drivers, devices and `ClusterNode` rows) is
+exactly the unit a redundant pair belongs to. Phase 1 already relies on this: the deployment-artifact
+cache is *keyed by ClusterId so a pair shares one entry*, and the rig's three application clusters
+(MAIN, SITE-A, SITE-B) are precisely its three pairs.
+
+So the real defect is that **`RedundancyStateActor` elects one Primary per *Akka* cluster, while
+every meaningful unit of redundancy is an *application* cluster.** The rig runs six driver nodes
+across three application clusters in one Akka cluster — a topology the product otherwise supports
+throughout — and exactly one of those six can ever be Primary.
+
+That mis-scoping is not confined to the alarm drain. Every consumer of the role inherits it:
+`ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate. On a multi-cluster fleet
+today, only one driver node cluster-wide will service Primary-gated work at all.
+
+### Revised recommendation
+
+**(C), as a separate change with its own live gate — and explicitly *not* (A).**
+
+- (C) is no longer "invent a pair concept"; it is "elect per `ClusterId` instead of per Akka cluster",
+  using a first-class entity that already exists. It is feasible where it needs to be:
+  `RedundancyStateActor` has no DB access today, but it is an admin-role singleton in the same
+  assembly as `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
+- **(A) is now actively undesirable.** Deriving pair identity from `LocalDb:Replication:PeerAddress`
+  would introduce a *second*, parallel notion of "who is my partner" that (C) immediately obsoletes —
+  and it cannot be done cleanly anyway: the library exposes no initiator/passive flag, so making both
+  halves declare a peer means both halves *dial*, opening two duplex sessions purely to obtain a
+  configuration value.
+- (C) touches the inbound device-write gate, which is safety-critical (the archreview 03/S4
+  dual-primary fix). It therefore belongs in its own change, with its own live gate, rather than
+  bolted onto Phase 2.
+
+Until then this branch is safe in every topology — no configuration loses data — so what remains is
+reclaiming the de-duplication benefit, not an outstanding hazard.
+
+## Not merged
+
+Per the plan, nothing on this branch is merged.

docs/plans/2026-07-20-localdb-phase2-recon.md

@@ -0,0 +1,371 @@
+# LocalDb Phase 2 — Recon findings
+
+**Date:** 2026-07-21 · **Branch:** `feat/localdb-phase2` · **Base:** `master` `2e46d054`
+
+Answers Task 0 of `2026-07-20-localdb-adoption-phase2.md`. Every claim carries a `file:line`
+citation against the base commit. Read the **Deviations** section last — two of them change the
+shape of Tasks 2/3 from what the plan assumed.
+
+---
+
+## 1. The sink's table schema
+
+**STOP condition does NOT fire** — confirmed by re-reading the executed DDL, as the plan's
+pre-answer instructed.
+
+Executed DDL: `SqliteStoreAndForwardSink.cs:652-670` (`InitializeSchema`). It matches the class
+doc-comment at `:17-26` exactly — no drift between documented and executed schema.
+
+```sql
+CREATE TABLE IF NOT EXISTS Queue (
+    RowId          INTEGER PRIMARY KEY AUTOINCREMENT,
+    AlarmId        TEXT    NOT NULL,
+    EnqueuedUtc    TEXT    NOT NULL,
+    PayloadJson    TEXT    NOT NULL,
+    AttemptCount   INTEGER NOT NULL DEFAULT 0,
+    LastAttemptUtc TEXT    NULL,
+    LastError      TEXT    NULL,
+    DeadLettered   INTEGER NOT NULL DEFAULT 0
+);
+CREATE INDEX IF NOT EXISTS IX_Queue_Drain ON Queue (DeadLettered, RowId);
+```
+
+| Question | Answer |
+|---|---|
+| BLOB columns? | **None.** All 8 columns are TEXT/INTEGER. Payload is `PayloadJson TEXT NOT NULL`. No base64 conversion needed; the 171 KB chunk sizing is moot. |
+| PK autoincrement? | **Yes** — `RowId INTEGER PRIMARY KEY AUTOINCREMENT`. The migrator must mint deterministic `mig-{node}-{legacyId}` ids and the new table needs a TEXT PK, as the plan specified. |
+| Indices to carry | One: `IX_Queue_Drain (DeadLettered, RowId)` — the drain's covering index. `alarm_sf_events` wants the same shape over (status, insertion order). |
+| Largest realistic payload | `PayloadJson` is a serialized `AlarmHistorianEvent` — 10 scalar fields (`AlarmHistorianEvent.cs`), no collections, no nesting. Low single-digit KB worst case; `Comment` is the only loosely-bounded field. |
+
+**Legacy column list** for the migrator's `pragma_table_info` intersection check:
+`RowId, AlarmId, EnqueuedUtc, PayloadJson, AttemptCount, LastAttemptUtc, LastError, DeadLettered`.
+
+`RegisterReplicated` will accept this table: no BLOB, and the PK will be app-minted TEXT.
+
+## 2. The public seam
+
+`IAlarmHistorianSink` (`IAlarmHistorianSink.cs:23-34`) is **two members**:
+
+```csharp
+Task EnqueueAsync(AlarmHistorianEvent evt, CancellationToken cancellationToken);
+HistorianSinkStatus GetStatus();
+```
+
+The concrete `SqliteStoreAndForwardSink` additionally exposes, and these are all consumed
+somewhere, so the rewire must keep them:
+
+| Member | Cited | Consumer |
+|---|---|---|
+| `StartDrainLoop(TimeSpan)` | `:189` | `ServiceCollectionExtensions.cs:105` (DI factory) |
+| `DrainOnceAsync(CancellationToken)` | `:320` | tests only — the deterministic drive |
+| `RetryDeadLettered()` | `:511` | operator action (AdminUI) |
+| `CurrentBackoff` | `:676` | tests |
+| `DebugCapacityProbeCount` | `:99` | tests |
+| `DefaultCapacity` / `DefaultMaxAttempts` consts | `:49`,`:53` | `AlarmHistorianOptions.cs:38,45` |
+| `IDisposable` | `:679` | DI + tests; **disposes the injected writer too** |
+
+There is **no separate enqueue/dequeue/markDelivered/deadLetter interface** — the plan's item 2
+guessed at one. Dequeue, delivered-marking and dead-lettering are all *private* to the sink
+(`ReadBatch :532`, `DeleteRow :564`, `DeadLetterRow :573`, `BumpAttempt :587`). That is good news:
+the rewire is entirely internal to one class and the public seam is untouched.
+
+## 3. Semantics to preserve
+
+All defaults live in **two** places that must stay in agreement:
+
+| Semantic | Sink default | Options default | Enforced at |
+|---|---|---|---|
+| `Capacity` 1,000,000 | `:49` | `AlarmHistorianOptions.cs:38` | `EnforceCapacityFastPathAsync :271` → `EnforceCapacityAsync :602` |
+| `MaxAttempts` 10 | `:53` | `:45` | drain loop `:429` |
+| Dead-letter retention 30 d | `:50` | `:41` | `PurgeAgedDeadLetters :638`, run once per drain tick |
+| `BatchSize` 100 | ctor arg `:116` | `:31` | `ReadBatch :542` |
+| `DrainIntervalSeconds` 5 | — | `:34` | `StartDrainLoop` via `ServiceCollectionExtensions.cs:105` |
+
+Behaviours that are **not** in the plan's list but are load-bearing, and which the cutover must
+not silently drop:
+
+- **Backoff ladder** 1s→2s→5s→15s→60s (`:57-64`), applied to the timer's *next due time*
+  (`RescheduleDrain :224`) so an outage genuinely slows the cadence.
+- **Overflow evicts oldest non-dead-lettered rows** with a WARN and a lifetime
+  `EvictedCount` counter (`:630-635`) — the durability guarantee is explicitly bounded.
+- **Corrupt-payload rows dead-letter immediately** for their own RowId so they cannot stall the
+  queue head (`:344-361`).
+- **Cardinality guard**: a writer returning ≠1 outcome per event is treated as a batch retry
+  (`:393-404`).
+- **In-memory row counter with periodic resync** every 10,000 enqueues (`:89-96`, `:271-290`) —
+  a perf optimisation that avoids `COUNT(*)` on the hot enqueue path.
+- **Post-dispose throwing contract**: `GetStatus`/`RetryDeadLettered`/`EnqueueAsync`/`StartDrainLoop`
+  throw `ObjectDisposedException`; `DrainOnceAsync` returns quietly (`:322`). Pinned by a
+  regression test (`SqliteStoreAndForwardSinkTests.cs:854-865`).
+
+## 4. Drain-worker lifecycle — **the finding that reshapes Task 3**
+
+> The plan asked "hosted service or actor?". **Neither.**
+
+The drain worker is a **self-rescheduling one-shot `System.Threading.Timer` owned by the sink
+itself** (`_drainTimer`, `:76`; started by `StartDrainLoop :189`; callback `DrainTimerCallback :199`;
+re-armed by `RescheduleDrain :224`). It is started from inside the DI singleton factory at
+`ServiceCollectionExtensions.cs:105`, so it begins ticking the moment the first consumer resolves
+`IAlarmHistorianSink`.
+
+Consequences for the Primary gate:
+
+1. **The gate has to live inside `DrainOnceAsync`**, not in a wrapper. There is no external
+   scheduler to gate.
+2. **`Core.AlarmHistorian` cannot see `PrimaryGatePolicy`.** The policy is in
+   `Runtime/Drivers/PrimaryGatePolicy.cs`, and Runtime → Core is the reference direction.
+   `Core.AlarmHistorian` references only `Core.Abstractions` (`.csproj`). So the sink takes a
+   **`Func<bool>` gate delegate** (default: always-drain) and Runtime supplies the lambda that
+   calls `PrimaryGatePolicy`. No new project reference, no new interface in Core.
+3. **A role view is still needed** because the timer fires outside any actor. See the next point
+   for who should own it.
+
+### There is already a Primary gate — on the *enqueue* side, with a different policy
+
+`HistorianAdapterActor` (`Historian/HistorianAdapterActor.cs`) already subscribes to
+`RedundancyStateChanged` (`:86`, handler `:145`), caches `_localRole` (`:42`), and gates the
+enqueue on `ShouldHistorize()` (`:97`):
+
+```csharp
+private bool ShouldHistorize() => _localRole is not (RedundancyRole.Secondary or RedundancyRole.Detached);
+```
+
+This is **not** `PrimaryGatePolicy` — it ignores driver-member count and defaults to *write* while
+the role is unknown. Its comment (`:68-71`) explains why it exists: DPS fans the Primary's single
+`alerts` publish to **every** node's adapter, so without the gate both nodes of a pair would
+double-write.
+
+**This is why today's ungated drain is safe, and why Phase 2 breaks that.** Today the Secondary
+never enqueues, so its queue is empty and an ungated drain has nothing to send. Once
+`alarm_sf_events` replicates, the Primary's rows land in the Secondary's table — and an ungated
+drain there would double-deliver *every* event, continuously, not just at failover. **The drain
+gate is not a nicety; it is required by the same commit that registers the table.**
+
+### Who owns the role view
+
+`HistorianAdapterActor`, not `DriverHostActor`. Both are spawned under
+`WithOtOpcUaRuntimeActors` (`ServiceCollectionExtensions.cs:353`) which `Program.cs:133` calls only
+under `hasDriver` — as are `AddAlarmHistorian` (`:175`) and `AddOtOpcUaLocalDb` (`:185`). So sink,
+LocalDb and adapter are co-located by construction. The adapter wins over `DriverHostActor`
+because it **already** holds the role for exactly this purpose; using it adds no new subscription
+and no new way for the view to go stale while the sink keeps draining.
+
+`DriverMemberCount` is read off Akka cluster state in `DriverHostActor.DriverMemberCount() :1446`
+(try/catch → 0 on a non-cluster provider ⇒ single-node posture). Task 3 will lift that into a
+small shared helper rather than copy it.
+
+## 5. Other `DatabasePath` construction sites
+
+- `ServiceCollectionExtensions.cs:98` — the only production site.
+- `appsettings.json:57` — `"DatabasePath": "alarm-historian.db"`.
+- `AlarmHistorianOptions.Validate() :54-55` — warns when the path is relative. **This warning
+  disappears with the key**; the validator loses one of its five checks.
+- `SqliteStoreAndForwardSinkTests.cs` — ~30 construction sites, all passing the per-test temp
+  `_dbPath` seeded at `:25` and deleted in `Dispose :32`. These are the tests Task 2 rewires onto
+  a temp-file `ILocalDb`.
+- **`docker-dev/docker-compose.yml` has no `AlarmHistorian` keys at all** — the rig runs with
+  `Enabled=false` today. Task 6 is therefore an *addition*, not an edit.
+
+No tooling or CLI constructs the sink.
+
+## 6. How `Enabled=false` short-circuits
+
+`AddAlarmHistorian` returns before registering anything when the section is absent or
+`Enabled != true` (`ServiceCollectionExtensions.cs:87`), leaving the `NullAlarmHistorianSink`
+seeded by `AddOtOpcUaRuntime :49`. The Null sink is a pure no-op (`IAlarmHistorianSink.cs:37-52`).
+
+Confirming the plan's own answer: **the tables are created unconditionally in `OnReady`** (cheap,
+empty) because `AddOtOpcUaLocalDb` is independent of the `AlarmHistorian` gate. Only the
+sink/drain stay Null. This is also required for correctness — `RegisterReplicated` must run on
+both pair members regardless of their historian config, or a node that later enables the sink
+would write rows that are never captured (see the CDC note below).
+
+## 7. Library constraints re-confirmed (README, `ZB.MOM.WW.LocalDb` 0.1.3)
+
+- **BLOB columns are rejected at registration.** Not a problem here (§1).
+- **CDC semantics** — rows that existed *before* `RegisterReplicated` are neither captured nor
+  snapshotted. This is the mechanical reason the migrator must run **after** registration, and
+  the reason `LocalDbSetup.OnReady`'s ordering comment is load-bearing (`LocalDbSetup.cs:23-31`).
+- **LWW keyed on the PK** — two nodes minting the same PK for *different* rows silently overwrite
+  each other; two nodes minting the same PK for the *same* row converge. Decision D-1 below turns
+  that second property into a feature.
+- `ILocalDb` surface in use: `ExecuteAsync` / `QueryAsync(sql, rowMapper, params, ct)` /
+  `BeginTransactionAsync` / `CreateConnection` / `RegisterReplicated`
+  (`Deployment/LocalDbDeploymentArtifactCache.cs:77,82,205`).
+
+---
+
+## Deviations and decisions
+
+Recorded here rather than raised as questions, per the plan's follow-up convention.
+
+### D-1 — Enqueue ids are a deterministic content hash, not `Guid.NewGuid()`
+
+**The plan says** GUIDs minted at enqueue (Task 2 step 2).
+
+**The problem.** The enqueue gate (`ShouldHistorize`, default-**write** on unknown role) and the
+drain gate the plan specifies (`PrimaryGatePolicy`, default-**deny** on unknown role when paired)
+disagree during the boot window. On a two-node pair before the first redundancy snapshot arrives,
+**both** adapters enqueue the same DPS-fanned transition. With random GUIDs those are two distinct
+rows that replicate into each other's tables — the buffer *duplicates* instead of converging, and
+whichever node becomes Primary later delivers both copies.
+
+**The decision.** Mint `id` as a hash over the serialized payload. Two nodes independently
+enqueuing the same fanned event produce the same PK, so LWW converges them to one row for free.
+Two genuinely distinct events cannot collide: `AlarmHistorianEvent` carries a full-precision
+`TimestampUtc` plus alarm id, kind, message and user, so an identical hash means an identical
+event.
+
+This costs one `SHA256.HashData` per enqueue and removes a whole duplicate-row class. It also
+slightly *improves* today's behaviour, where the same boot-window fan-out is already
+double-delivered because the drain is ungated on both nodes.
+
+`ShouldHistorize` itself is left unchanged — out of scope, and with D-1 in place the two gates
+disagreeing is no longer harmful.
+
+### D-2 — Gate denial must be observable
+
+`PrimaryGatePolicy` defaults to **deny** when the role is unknown on a multi-driver cluster. That
+is the correct safety posture, but it means a redundancy-snapshot identity mismatch — the
+documented 03/S5 shape that `DriverHostActor.cs:1485-1491` already warns about once — would
+silently stop alarm history on *both* nodes, degrading only as retry/eviction growth minutes later.
+
+Task 3 will therefore: log the skip once per gate-state transition (not per tick — the drain runs
+every 5 s), and surface it in `HistorianSinkStatus` so `/healthz` and the AdminUI can see
+"buffering, not draining, because not Primary" rather than an unexplained rising queue depth.
+Adding a `HistorianDrainState` member is additive; Task 3 will check every `switch` over it.
+
+### D-3 — Tasks 2 and 3 will land as one commit
+
+The plan anticipated this. Confirmed necessary: the gate delegate is a constructor parameter of
+the rewritten sink, so the sink cannot compile in its final shape without it, and an intermediate
+commit with a rewired-but-ungated sink would be a commit in which a replicated table is drained by
+both nodes. Not a state worth having in history.
+
+### D-5 — The table keeps the legacy delete-on-ack shape, not a `status` column
+
+**The plan proposes** `status TEXT NOT NULL DEFAULT 'pending'` with values `pending | delivered |
+dead`, and drops the legacy `LastError` column.
+
+**Two problems.** A `delivered` status means acknowledged rows accumulate in the table forever —
+the plan specifies no sweeper for them, and they would also have to be excluded from the capacity
+count, changing a semantic §3 says to preserve. And `LastError` is not dead weight: `DeadLetterRow`
+writes the failure reason into it and `RetryDeadLettered` clears it, so it is the only operator-
+facing record of *why* a row was dead-lettered.
+
+**The decision.** Keep the legacy shape: DELETE on acknowledgement, `dead_lettered` as the same 0/1
+flag, and retain `last_error`. Deleting keeps the table bounded with no second sweeper, and the
+replication engine carries the delete as a tombstone (pruned after `TombstoneRetention`, 7 d), so
+the peer drops its copy too — which is the property the plan wanted the `delivered` status for.
+Columns then map 1:1 onto the legacy table, making the Task-4 migrator a straight copy.
+
+One genuine change: the drain's `ORDER BY` moves from `RowId ASC` to `enqueued_at_utc, id`, because
+a hashed TEXT id carries no insertion order. Round-trip ("O") timestamps sort lexicographically in
+chronological order, and `id` makes the ordering total; the index covers both.
+
+### D-6 — The migrator keys on the payload hash, not `mig-{node}-{legacyId}`
+
+**The plan says** deterministic ids of the form `mig-{NodeName}-{legacyId}`, mirroring ScadaBridge.
+
+**Why that is not enough here.** Node-prefixing does solve the collision the legacy AUTOINCREMENT
+key would otherwise cause — node A's `RowId 7` and node B's `RowId 7` are different alarms — but it
+*preserves* a duplication that ought to be collapsed. A warm pair's two legacy files **overlap**:
+`HistorianAdapterActor` default-writes while its redundancy role is unknown, so both nodes accepted
+the same fanned transitions during every boot window they ever had. Prefixed ids would carry every
+one of those duplicates into the merged buffer permanently, for the eventual Primary to deliver
+twice.
+
+**The decision.** Reuse D-1's `AlarmSfSchema.DeriveId` (lifted out of the sink so both call sites
+share one identity rule). Distinct events cannot collide; identical ones converge. It also removes
+the migrator's only need for node identity from configuration, and makes a crash between commit and
+rename harmless under `INSERT OR IGNORE` — which the plan's own idempotency requirement wanted
+anyway.
+
+### D-7 — The migrator's tests live in `Host.IntegrationTests`
+
+The plan names `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests`. **That project does not exist.** The
+tests sit beside `LocalDbSetupTests` in `Host.IntegrationTests`, which is where every other
+LocalDb-facing test already lives. They are offline — a temp file and no external services.
+
+### D-4 — The live gate needs `MaxAttempts` raised on the rig
+
+Not actionable until Task 8, recorded now so it is not rediscovered there. The rig has no
+HistorianGateway, so every drain attempt fails. With `MaxAttempts: 10` and the backoff ladder
+capping at 60 s, buffered rows dead-letter roughly five minutes into the outage — which would make
+the gate's "dead letters 0" check (step 2) fail for reasons that have nothing to do with
+replication. Task 6 should set a high `MaxAttempts` on the rig, or Task 8 should assert on
+dead-letter *timing* instead.
+
+Related: `Program.cs:170-173` shows an `AlarmHistorian:Enabled=true` /
+`ServerHistorian:Enabled=false` deployment is explicitly supported and warned about — which is
+exactly the rig's shape. Task 6 must confirm `ServerHistorianOptionsValidator` does not fail host
+start for a disabled section with an empty `Endpoint`.
+
+---
+
+## Guard-deletion evidence (recorded at the Task 7 DoD sweep)
+
+Every guard this phase adds was proved by deleting it and watching the tests go red. Two of those
+runs found tests that would have passed over a broken implementation. Both are recorded here because
+the *shape* of the vacuity is reusable, not just the fix.
+
+### Control 1 — remove `IRedundancyRoleView.Publish` from `DriverHostActor`
+
+**3 of 6** `DriverHostActorRoleViewTests` went red. The other three assert
+`ShouldServiceAsPrimary == true`, which is exactly the value the view is **seeded** with — so they
+cannot distinguish "published true" from "never published at all". That split is expected and
+acceptable *given* the three that do go red pin the false cases, which only a real publish can
+produce. Worth knowing before someone reads a future 3/6 result as a regression.
+
+### Control 2 — remove `RegisterReplicated("alarm_sf_events")` from `LocalDbSetup.OnReady`
+
+**3 of 4** `AlarmSfConvergenceTests` went red immediately. The fourth,
+`TheSameEventOnBothNodes_ConvergesToOneRow`, **passed vacuously**: with replication switched off
+each node trivially held its own single row, and "count == 1 on both nodes" was satisfied by two
+completely unconverged databases. The assertion could not tell convergence from isolation.
+
+**Fix:** enqueue a second, *distinct* event on node B and require **both** nodes to hold two rows.
+Isolation now yields 1 and 2; only convergence yields 2 and 2. Re-run under the control: red.
+Control restored: all 4 green.
+
+The general trap — an assertion whose expected value is also what a *disabled* system produces —
+is the same one recorded in the `#485`/`#486` unreadable-artifact defect class.
+
+### Exact-set replicated-table pins
+
+Two, not one, and both assert set equality (ordered `ShouldBe` against the literal three-table list,
+so an added *or* dropped registration fails):
+
+- `LocalDbSetupTests` — drives the production `OnReady` callback directly.
+- `LocalDbWiringTests` — resolves `ILocalDb` from the full driver DI graph, which is what catches a
+  registration that never reaches a running host.
+
+## Test-suite evidence (Task 7 DoD sweep)
+
+Full solution run on the branch, compared against a **full solution run on a detached worktree at
+the pre-branch baseline `2e46d054`**. Comparing failure *sets*, not counts, because the suite has a
+standing set of environment- and load-dependent failures that a raw count would hide.
+
+**The two failure sets are identical — all 13 tests, same names, both runs. Zero new failures.**
+
+| Assembly | Baseline | Branch | Delta |
+|---|---|---|---|
+| `Core.AlarmHistorian.Tests` | 0 failed / 29 passed | 0 failed / **32** passed | **+3** (the drain-gate tests; 29 ported from the old sink's suite) |
+| `Runtime.Tests` | 1 failed / 418 passed | 1 failed / **424** passed | **+6** (`DriverHostActorRoleViewTests`) |
+| `Host.IntegrationTests` | 2 failed / 169 passed | 2 failed / **186** passed | **+17** (migrator + convergence + pin updates) |
+
+The 13 standing failures, and why each is not this branch's:
+
+- **3 × `DriverTypeNamesGuardTests`** — `ArgumentNullException (secretResolver)` out of
+  `GalaxyDriverFactoryExtensions.Register`, reached by reflection. A real pre-existing defect in a
+  guard test, reproduced exactly on the baseline worktree. Unrelated to this phase.
+- **4 × `AbLegacy.IntegrationTests`**, **3 × `OpcUaClient.IntegrationTests`**,
+  **1 × `DriverProbeHandshakeE2eTests.AbCip_Green_AgainstSim`** — driver fixtures on the
+  `10.100.0.35` docker host are not running.
+- **1 × `RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed`**
+  — times out under full-suite load; passes in isolation.
+- **1 × `ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks`** — fails
+  under full-suite load ("writer must have been called at least twice"), and `Runtime.Tests` passes
+  **425/425** when run alone. Also fails on the baseline. Worth noting for anyone reading a future
+  run: this one is *newly observed* here only because the previous sweep's log was truncated, not
+  because the branch introduced it.

docs/plans/2026-07-21-issue-485-live-gate.md

@@ -0,0 +1,76 @@
+# Issue #485 — live gate (docker-dev rig)
+
+> Verifies the two fixes for #485 — `OpcUaPublishActor` (master `c7f5b9cf`) and `DriverHostActor`
+> (master `5aad1e33`) — on the local `docker-dev` rig: 6 nodes, shared SQL, real Modbus fixture on
+> `10.100.0.35:5020`.
+>
+> **Safety rules followed**: all DB inspection + mutation via `sqlcmd` inside the SQL container; the
+> rig's Modbus fixture is read-only for this gate (no writes to the PLC sim).
+
+## Result
+
+**Both fixes verified, all three guards observed firing in production, with a live RED control that
+was not staged.** The gate also confirmed a **pre-existing** defect the fixes do not address — the node
+records `Applied` for a deployment it skipped (see "Finding" below).
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 1 | RED control — the bug in its natural habitat, unforced | ✅ | site-a-1 hit #485 during the *previous* gate's SQL flapping and had been serving an **empty address space for 34 minutes**: `failed to load artifact … rebuild becomes no-op` followed immediately by `applied plan (kind=PureRemove, added=0, removed=17, rebuild=True)`. Its 17 `ns=2` nodes were gone; a browse returned only the root `OtOpcUa` folder. Nothing was staged to produce this — it was already true on the pre-fix image. |
+| 2 | Rebuilt on the fixed image, the node recovers | ✅ | After `docker compose build` + `--force-recreate`, site-a-1 restored to `raw subtree materialised (containers=16, tags=1)` — 18 `ns=2` nodes. |
+| 3 | Address-space guard fires; nodes survive | ✅ | site-a-1, empty artifact: `OpcUaPublish: no usable artifact for deployment f4b9dd63… — KEEPING the currently served address space rather than tearing it down`. **No `PureRemove`.** Browse before vs after: 18 vs 18, `diff` **identical**. |
+| 4 | Reconcile guard fires; drivers survive | ✅ | `artifact for f4b9dd63… read back empty; skipping reconcile and keeping the 1 running driver(s)`, and the apply still reported `children=1`. Repeated on central-2 with a bigger blast radius: `keeping the 5 running driver(s)`, `children=5`. |
+| 5 | SubscribeBulk guard fires; **live values keep flowing** | ✅ | central-2, `artifact for 8c36a673… read back empty; skipping SubscribeBulk and keeping the live subscriptions`. The decisive proof is the value, not the log: `ns=2;s=pymodbus/plc/HR200X` read **56278 @ 06:59:51** before the induction, then **56336 @ 07:00:21** and **56354 @ 07:00:30** after it (induction at 07:00:07). The subscription was still delivering fresh Modbus data across the event. |
+| 6 | Positive control — a readable artifact still applies | ✅ | Immediately after the guards fired, a genuine deploy applied normally on both nodes: site-a-1 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 1 references`; central-2 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 5 references across 4 driver(s)`. The guards suppress the teardown, **not** deployment. |
+| 7 | Control — only the node that saw empty bytes took the guard path | ✅ | Of the six nodes, exactly **one** logged `read back empty` (the paused one); the other five read the real artifact before it was blanked and applied it normally. The induction window is what produced the condition, not a global change. |
+
+## How the failure was induced deterministically
+
+The original incident was a race (a transient ConfigDb blip landing between the apply and the artifact
+read). Racing it again would be unreliable, so the gate drives the **empty-bytes** branch instead — which
+is byte-for-byte what the code sees when the row cannot be read, and is the branch the driver-side guards
+key on (their *throw* paths already returned early before this work):
+
+1. `docker pause` the target node, freezing it mid-cluster.
+2. `POST /api/deployments` — the deployment seals and dispatches while the node cannot process it.
+3. `UPDATE Deployment SET ArtifactBlob = 0x WHERE DeploymentId = <new>`.
+4. `docker unpause` — the node now processes the dispatch and reads an empty artifact.
+
+The pause measured **<1s** on both runs, well inside `acceptable-heartbeat-pause = 10s` and SBR's
+`stable-after = 15s`, so the node was never flagged unreachable and no failover was triggered.
+
+## Finding — the node records `Applied` for a deployment it skipped (pre-existing, NOT fixed here)
+
+Both guards leave the node running its previous configuration, which is the intent. But `ApplyAndAck`
+advances `_currentRevision = revision` and writes `NodeDeploymentState = Applied` **before** it knows
+whether anything was applied — that assignment sits immediately after `ReconcileDrivers`, which returns
+`null` on both the (pre-existing) throw path and the new empty path:
+
+```
+[07:00:07 WRN] DriverHost central-2:4053: artifact … read back empty; skipping reconcile …
+[07:00:07 INF] DriverHost central-2:4053: applied deployment 8c36a673… (rev 03b33904…, children=5)
+```
+
+`NodeDeploymentState` for that deployment reads `Status = Applied` for central-2. The node now claims a
+revision whose configuration it never applied — and because `HandleDispatchFromSteady` short-circuits on
+a revision match (`dispatch matches current rev; immediate ACK`), **re-dispatching that same revision is
+a no-op**, so the node cannot self-heal. Only a later deployment with a *different* revision recovers it
+(check 6 confirms that path works).
+
+This is pre-existing — the throw path has always done it — and is strictly a reporting/convergence bug,
+not a data-plane one: the node keeps serving its last good configuration throughout. The honest fix is to
+ACK `Failed` and leave `_currentRevision` alone when the artifact could not be read, so the normal deploy
+retry heals it. Tracked separately rather than folded in here, because it changes deploy ACK semantics
+fleet-wide.
+
+## Addendum — issue #486 fix, live-gated (2026-07-21)
+
+The finding above is now fixed (`DriverHostActor.ApplyAndAck` fails the apply when `ReconcileDrivers`
+returns null) and gated on the same rig with the same induction.
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 8 | Regression — a normal deploy still ACKs Applied fleet-wide | ✅ | The change alters ACK semantics, so this is the risk that matters. A genuine deploy recorded `Status = Applied` on **all six** nodes. |
+| 9 | An unreadable artifact now ACKs Failed, with a reason | ✅ | central-2 recorded `Status = Failed` + `the deployment artifact could not be read (ConfigDb unreachable, or the row is missing/empty); nothing was applied`. The five nodes that read the real artifact recorded Applied — so the fleet view now distinguishes them. Pre-fix, all six claimed Applied. |
+| 10 | The revision is NOT advanced | ✅ | `apply of a855dd9e… FAILED — … Staying on revision 5bc0d9f3…` — the **previous** revision, not the dispatched `583122b7…`. This is what stops `HandleDispatchFromSteady` from short-circuiting a retry of that revision. |
+| 11 | The node keeps serving throughout (#485 still holds) | ✅ | The #485 guard fired first (`keeping the 5 running driver(s)`), and `ns=2;s=pymodbus/plc/HR200X` read **59603 @ 07:27:28**, live and current, after the Failed apply. Failing the apply reports the truth; it does not tear anything down. |
+| 12 | The node recovers on the next real deploy | ✅ | The following deployment applied on central-2 (`children=5`) and recorded Applied on all six nodes. |

docs/plans/2026-07-21-localdb-followups-live-gate.md

@@ -0,0 +1,86 @@
+# LocalDb Phase 1 follow-ups — live gate (docker-dev rig)
+
+> Closes the two limitations left open by `2026-07-20-localdb-phase1-live-gate.md` (checks 4 and 8).
+> Rig: local `docker-dev/docker-compose.yml`, 6 nodes rebuilt from `fix/localdb-phase1-followups`.
+> site-a pair replicates; central + site-b stay default-OFF.
+>
+> **Safety rules followed** (same as the Phase 1 gate): all DB inspection via `docker cp` of the
+> `db`/`-wal`/`-shm` triplet, querying the copy — never host `sqlite3` on the live WAL file.
+
+## Result
+
+**Both follow-ups closed, and the gate found a third defect** — one that was *unreachable* until the
+first fix landed. Library `ZB.MOM.WW.LocalDb` went `0.1.1` → `0.1.2` → `0.1.3` in the course of this
+gate; both consumers (OtOpcUa, ScadaBridge) are pinned to `0.1.3`.
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 8 | Oplog stops growing on a default-OFF node | ✅ PASS | site-b-1 restarted twice with an unchanged cached artifact: `oplog_depth` **7 → 7 → 7**, `deployment_pointer.applied_at_utc` frozen at `05:34:58` across both — the re-cache was skipped outright. Repeated on the `0.1.3` build after a real deploy: **13 → 13**. Pre-fix this grew ~2 rows per restart forever, because a default-OFF node has no peer to ack and prune them. |
+| 8b | …but a genuinely new deployment still writes (positive control) | ✅ PASS | A real config change (renamed tag `SITEA-tag`) produced rev `822bdf34`, then `002c16b3`. site-b-1's oplog **7 → 10 → 13** and its pointer advanced each time. The skip suppresses *identical* re-writes only; it is not a dead cache-write path. |
+| 4 | A wiped node is back-filled by its peer, no new deploy, central down | ✅ PASS (was ⚠️ PARTIAL) | Precondition asserted first — site-a-1's oplog **0**, `needs_snapshot 0`: the exact state that used to make back-fill impossible. Then SQL stopped, site-a-2's LocalDb **volume destroyed**, container recreated. a-1 logged `Snapshot sent (as_of_seq 0, 4 rows)` — a line that could not appear on `0.1.1`. a-2 came back with pointer `SITE-A / 475df87a @ 03:09:12` (a-1's **original** timestamp, not a re-derived one), chunks 2, and a `__localdb_row_version` dump **byte-identical to a-1's**, carrying origin node ids `6812ca9c` and `935659a5` — the rebuilt node's own id appears nowhere, so the rows were replicated, not recreated. |
+| 4b | …and the back-filled node then boots from cache | ✅ PASS | With SQL still down, site-a-2 logged `RUNNING FROM CACHE — … booted deployment 475df87a… cached at 2026-07-21T03:09:12`, `raw subtree materialised (containers=16, tags=1)`. OPC UA browse of both nodes: **17 ns=2 nodes each, diff-identical** — the rebuilt node serves exactly what its healthy peer does, from a configuration it never applied itself and could not have fetched. |
+| 4c | The rebuilt node's OWN writes reach its peer | ✅ PASS **after a third fix** (see below) | On `0.1.2` this failed: a-1 held `last_applied_remote = 10` while the rebuilt a-2's oplog was seqs **1,2,3** and its `last_acked` stayed **0** — never accepted. On `0.1.3`, a-1 logs `Replication peer identity changed (ebb10854 -> c85d8b1d): the peer was rebuilt … Resetting the inbound watermark`, and after the next deploy a-1 shows `last_applied_remote = 3` / a-2 shows `last_acked = 3` with a drained oplog and identical row_versions. |
+
+## The third defect (the reason this gate was worth running)
+
+`0.1.2` made back-fill work, which for the first time let a rebuilt node be observed *writing*. It
+turned out its writes went nowhere.
+
+`last_applied_remote_seq` is a watermark in the **peer's** seq space, and a rebuilt peer is a new
+database whose oplog numbers from 1. Two independent bugs stacked, and fixing either alone leaves the
+data stuck:
+
+1. The healthy node kept advertising the **old** peer's watermark. The *sending* side seeds its pump
+   directly from that number (`_sentThruSeq`), so the rebuilt node skipped its entire oplog.
+   → the watermark (and the observed peer clock) is now reset when the peer's node id changes.
+   `last_acked_seq` is deliberately **not** reset: it describes our own oplog's pruned horizon and is
+   exactly what `0.1.2` keys the snapshot decision off — clearing it would switch back-fill back off.
+2. A peer claiming to have applied more of our stream than we have ever produced can only be
+   remembering a previous incarnation of *us*. → that claim is now rejected in favour of pumping from
+   the start; LWW makes the re-send harmless. This mirrors the clamp `RecordPeerAckAsync` already
+   applies to acks.
+
+Both were reproduced offline as a RED test before fixing
+(`SnapshotResyncTests.RebuiltPeer_OwnWritesReachTheHealthyNode_DespiteTheOldPeersWatermark`).
+
+**Why offline tests could not have found it first:** the wiped-peer scenario had no test because the
+scenario was believed impossible to recover from — it was a *documented limitation*, not a bug. The
+gate is what turned the limitation into a reproducible case, and the second defect only existed
+behind the first.
+
+## Unrelated finding (NOT a follow-up of this work — pre-existing; `lmxopcua` issue #485, since FIXED)
+
+While SQL was being stopped and started to drive check 4, site-b-1 took a deploy whose artifact load
+hit a transient ConfigDb error, and **emptied its served address space**:
+
+```
+[06:11:20 ERR] An error occurred using the connection to database 'OtOpcUa' on server 'sql,1433'.
+[06:11:20 WRN] OpcUaPublish: failed to load artifact for deployment 43d54397…; rebuild becomes no-op
+[06:11:20 INF] AddressSpaceApplier: applied plan (kind=PureRemove, added=0, removed=16, rebuild=True)
+```
+
+The log says "rebuild becomes no-op" and the applier then removed all 16 nodes — those two disagree.
+The cache layer behaved correctly and refused to store the bad artifact ("not caching deployment … its
+artifact was empty or could not be loaded, so it is not a usable last-known-good configuration"), and
+the node recovered fully on restart (`containers=16`). But a transient ConfigDb blip emptying the
+served address space instead of holding last-known-good is the same failure *class* as the Phase 1
+gate's check-3 defect. Untouched by this work — site-b-1 has replication off entirely — and induced by
+this gate's own SQL flapping.
+
+**Fixed** (issue #485): `OpcUaPublishActor.HandleRebuild` now treats an artifact it could not obtain as
+*no answer* rather than *an empty configuration*, and abandons the rebuild with the served address space
+(and `_lastApplied`) intact. Nothing legitimate produces a zero-length blob — an operator who really
+deletes everything still deploys a JSON document with empty arrays — so "no bytes" can only mean the read
+failed. The log excerpt above is now impossible: the `PureRemove` never runs, and the loader's own
+"the rebuild is abandoned" line is finally true. Covered by
+`OpcUaPublishActorRebuildTests.Rebuild_keeps_the_last_known_good_address_space_when_the_artifact_load_fails`,
+with a positive control proving a *readable* artifact that genuinely drops an equipment still tears its
+subtree down.
+
+The **driver-side half** of the same mistake was fixed alongside it: `DriverHostActor.ReconcileDrivers`
+and `PushDesiredSubscriptionsFromArtifact` read the artifact the same way, so empty bytes meant zero
+driver specs — `DriverSpawnPlanner` planned every running child for `StopChild` — and then an empty
+desired set dropped each surviving driver's live subscription handle. A node lost its entire field I/O
+and still ACKed Applied. Both now skip on empty. Covered by `DriverHostActorUnreadableArtifactTests`,
+whose absence assertion is calibrated by a control that observes the very same unsubscribe *arriving*
+inside the settle window (without it the assertion passed on a race).

docs/plans/2026-07-21-per-cluster-mesh-design.md

@@ -0,0 +1,343 @@
+# Per-Cluster Akka Mesh — design
+
+> **Status:** design for review. **Phases 0a and 0b are DONE and merged** (2026-07-21); Phases 1–7
+> are not started and each needs its own plan.
+> **Decisions settled 2026-07-21** — see §6: one central DB with driver nodes disconnected from it,
+> the two-node keep-oldest gap closed upstream and now closed here too (§6.2), and their auth posture
+> matched for now.
+
+## 1. What this changes
+
+Today OtOpcUa runs **one Akka mesh containing every node of every application Cluster**. That was a
+deliberate choice — `docs/plans/2026-06-07-per-cluster-scoping.md`, "Per-ClusterId Scoping
+(hub-and-spoke **single mesh**)" — so the deploy channel could stay in-mesh: AdminUI →
+`admin-operations` singleton → `deployments` topic → every node filters to its own `ClusterId`.
+
+The proposal is to move to the sister project's shape: **one Akka mesh per application Cluster,
+two nodes maximum**, with central and clusters joined by explicit transports instead of cluster
+gossip.
+
+The motivation is that redundancy becomes correct *by construction*. Every Primary-gated decision
+(inbound device writes, native alarm acks, fleet alerts, ServiceLevel, and — since Phase 2 — the
+alarm-history drain) currently asks "am I the driver Primary?" of a **cluster-wide** election, while
+every resource being gated is **pair-local**. Splitting the mesh collapses the two scopes into one
+and the question stops being ambiguous.
+
+## 2. What ScadaBridge actually does
+
+Researched directly from `/Users/dohertj2/Desktop/ScadaBridge`. Five things differ from the
+summary in its own CLAUDE.md, and they matter:
+
+**Three transports cross the boundary, not two.**
+
+| Transport | Direction | Carries |
+|---|---|---|
+| Akka **ClusterClient** | central → site (+ replies) | command/control: deploy notify, lifecycle, queries, subscribe handshakes |
+| **gRPC** server-streaming | **central dials into the site** | real-time attribute + alarm events |
+| Plain **HTTP**, token-gated | site → central | the deployment config itself (notify-and-fetch) |
+
+**The gRPC direction is inverted from the obvious guess.** Data flows site→central, but *each site
+node hosts the gRPC server* (Kestrel h2c :8083) and *central is the client*. There is no gRPC server
+on central at all.
+
+**Active node = OLDEST Up member, explicitly not the cluster leader.**
+`ActiveNodeEvaluator.SelfIsOldestUp` is documented as *"THE single definition of 'active node'"*:
+
+> *"Cluster LEADERSHIP (lowest address) is an Akka-internal concept that diverges from singleton
+> placement permanently once the original first node restarts and rejoins; every product-level
+> active/standby decision must use this evaluator, never `cluster.State.Leader`."*
+
+The equivalence **oldest-Up == where `ClusterSingletonManager` places singletons** *is* the design.
+
+**All clusters share one ActorSystem name** (`"scadabridge"`, hardcoded). They are separate clusters
+only by seed-node partitioning — required, because Akka.Remote address matching means ClusterClient
+could not reach a differently-named system.
+
+**Site nodes carry two roles:** `"Site"` and `"site-{SiteId}"`. Singletons scope to the
+**site-specific** role.
+
+Other load-bearing details:
+
+- **Central discovers sites from the database** (a `Site` entity with `NodeAAddress`/`NodeBAddress`
+  + `GrpcNodeAAddress`/`GrpcNodeBAddress`), refreshed every 60 s and on admin change. Sites discover
+  central from **appsettings** — static, restart required. The asymmetry is deliberate.
+- **Exactly one actor is exposed per side** via `ClusterClientReceptionist.RegisterService` —
+  `/user/central-communication` and `/user/site-communication` — registered **per node, not as a
+  singleton**, so contact rotation reaches whichever node answers.
+- **No central buffering when a site is unreachable.** The send is dropped with a warning and the
+  caller's Ask times out. *"It keeps the central coordinator stateless with respect to site
+  availability."* A `ConnectionStateChanged` mechanism was built for this and **deleted as dead
+  code**.
+- ~~**Every unhandled message replies with a typed failure rather than dropping**, so a central Ask
+  can never stall on a missing handler.~~ **CORRECTED 2026-07-22 (Phase 2 recon).** This overstates
+  what ScadaBridge does. Neither of their comm actors has a `ReceiveAny` or an `Unhandled` override:
+  the typed-failure idiom is **per message type**, and it fires only when a *registration* is missing
+  (no ClusterClient for the site yet, no handler registered for that type). A genuinely unknown
+  message type still dead-letters exactly as it would anywhere else. Do not build a catch-all failure
+  reply believing you are copying them. OtOpcUa's Phase 2 comm actors list each type explicitly and
+  deliberately let an unknown type dead-letter loudly rather than republish it blind.
+- **Sender preservation is the whole Ask idiom**: `_client.Tell(new ClusterClient.Send(path, msg),
+  Sender)` routes the reply straight back to the waiting Ask, bypassing the comm actor.
+
+### The registered outage gap — read before copying
+
+> `Component-ClusterInfrastructure.md:113`: *"Only the FIRST seed listed in `Cluster:SeedNodes` may
+> self-join to form a *new* cluster… a lone restarted non-first-seed node (with the first seed still
+> down) loops on `InitJoin` forever — never `Up`, never routable. This is why the two-node
+> keep-oldest **oldest/active-node crash is a total-outage gap**: after the oldest dies the younger
+> survivor self-downs, and it cannot re-bootstrap alone. Recovery is operator-driven."*
+
+Their failover drill has a mode that exists *"to make the registered gap observable — not to pretend
+it is covered."* **A two-node keep-oldest mesh has an acknowledged total-outage hole.** If OtOpcUa
+adopts 2-node meshes, this must be decided deliberately, not inherited.
+
+> **CLOSED 2026-07-22.** The `InitJoin` half of this gap is fixed in both repos by **self-first seed
+> ordering**: every node that is one of its own seeds lists ITSELF as `seed-nodes[0]`, which is the
+> only ordering under which Akka runs `FirstSeedNodeProcess` and can form a cluster with no peer
+> answering. (An earlier `Cluster:SelfFormAfter` watchdog was tried here and retired — it raced the
+> join handshake.) See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Today's
+> driver-only site nodes are seeded by `central-1` only and are exempt; the per-cluster mesh removes
+> even that exception by making every pair node a seed of its own mesh.
+
+### Both inter-cluster transports are unauthenticated
+
+Akka remoting has no TLS, no secure cookie, no `trusted-selection-paths`. The gRPC listener is h2c
+with no auth covering `SiteStreamService` — including two Pull RPCs that return audit rows. The only
+authenticated pieces are the HTTP fetch token and `LocalDbSyncAuthInterceptor`, which gates *only*
+`/localdb_sync.v1.LocalDbSync/`. Their boundary assumes a trusted network.
+
+**OtOpcUa already ships that same fail-closed interceptor** (LocalDb Phase 1) — it is a ready-made
+template if we choose to authenticate more than they did.
+
+## 3. What OtOpcUa has today
+
+**Nine cross-node DPS topics plus a singleton**, all currently relying on one mesh:
+
+| Channel | Direction | Purpose |
+|---|---|---|
+| `deployments` / `deployment-acks` | central ↔ nodes | deploy dispatch + acks |
+| `driver-control` | central → nodes | AdminUI Reconnect/Restart |
+| `redundancy-state` | central → nodes | roles + peer probe results |
+| `alerts` | nodes → central | fleet alarms → `/alerts` |
+| `driver-health`, `driver-resilience-status`, `fleet-status`, `script-logs` | nodes → central | AdminUI live panels |
+| `admin-operations` singleton | central | operator ack/shelve into engines |
+
+Three facts make the split cheaper than that table suggests:
+
+1. **The deploy path is already notify-and-fetch.** `DispatchDeployment` carries only
+   `DeploymentId` + `RevisionHash` + `CorrelationId`; the node fetches the artifact itself. We
+   independently arrived at the pattern ScadaBridge had to retrofit, and we do **not** carry its
+   128 KB Akka frame exposure on this path.
+2. **Deploy state already has a DB substrate.** `ConfigPublishCoordinator` persists per-node ACKs to
+   `NodeDeploymentState` so a singleton failover recovers in-flight state from the DB.
+3. **`ClusterNode` rows already enumerate every node per application Cluster** — the same table
+   `AdminOperationsActor` groups by cluster. This replaces the coordinator's one genuinely
+   mesh-bound dependency: it derives its expected-ack set from `Akka.Cluster.State.Members` filtered
+   by role.
+
+So the deploy channel needs only a small notify + ack transport. **The nine live-telemetry channels
+are the real work**, and they are exactly what ScadaBridge built ClusterClient + gRPC for.
+
+## 4. A defect to fix regardless of this design
+
+OtOpcUa currently uses **two different node-selection rules at once**:
+
+- Split-brain resolution is **keep-oldest** (`Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49`).
+- `ClusterSingletonManager` places singletons on the **oldest** member.
+- But `RedundancyStateActor.BuildSnapshot` elects Primary from **`RoleLeader("driver")`** — lowest
+  address among driver members.
+
+Those select the same node only by coincidence. After a restart-and-rejoin the role leader and the
+oldest member diverge permanently, which means the node the SBR protects and hosts every singleton
+on need not be the node the data-plane gates consider Primary. This is precisely the rule ScadaBridge
+forbids by name, and for the reason it gives: *"both sides claim leadership during a partition"* —
+the dual-primary shape archreview 03/S4 exists to prevent.
+
+**Switching the role derivation to oldest-Up-with-role is small, is correct under either topology,
+and should not wait for this design.** It also happens to be exactly what the separate-mesh model
+needs, so it is not throwaway work.
+
+**FIXED 2026-07-21 (Phase 0b).** `RedundancyStateActor.BuildSnapshot` selects the oldest Up member
+carrying the `driver` role, matching singleton placement; `NodeRedundancyState.IsRoleLeaderForDriver`
+was renamed `IsDriverPrimary` so the contract stops asserting the old derivation. Pinned by
+`RedundancyPrimaryElectionTests`, which builds a real two-node cluster in which the oldest node holds
+the *higher* address, so the two orderings genuinely disagree.
+
+## 5. Target architecture
+
+Mirror ScadaBridge's shape, adapted to OtOpcUa's existing substrate:
+
+- **One Akka mesh per application `Cluster`, two nodes max.** Same ActorSystem name (`otopcua`)
+  everywhere; separation by seed-node partitioning only.
+- **Roles per node:** `driver` plus a cluster-specific `cluster-{ClusterId}`, with singletons scoped
+  to the cluster-specific role.
+- **Active node = oldest Up member with role** (§4), one definition, used by the data-plane gates,
+  ServiceLevel, and the alarm drain alike.
+- **Central ↔ cluster command/control over ClusterClient**, with exactly one receptionist-registered
+  actor per side and central discovering cluster node addresses from `ClusterNode` rows (extended
+  with Akka + gRPC addresses, mirroring their `Site` entity).
+- **Deploy stays notify-and-fetch, but the fetch changes source.** The notify crosses via
+  ClusterClient; the artifact then comes **from central**, not from the ConfigDb the driver node no
+  longer connects to (§6.1), and is cached in LocalDb. The coordinator's expected-ack set comes from
+  `ClusterNode` rows instead of cluster membership.
+- **Driver nodes hold no ConfigDb connection.** LocalDb becomes their steady-state configuration
+  store; five current DB consumers are re-homed or re-sourced per §6.1.
+- **Live telemetry over gRPC**, central dialling into each cluster node, replacing the seven
+  observability topics with one stream contract carrying a `oneof` event — additive-only field
+  evolution, contract-locked by test.
+
+**Copied:** their node-local configuration store fed by fetch-and-cache (§6.1), their oldest-Up
+active-node rule (§4), their two-node keep-oldest posture including its acknowledged outage gap
+(§6.2), and their unauthenticated inter-cluster transports (§6.3).
+
+**Deliberately not copied:** their *transient-only* central alarm cache. OtOpcUa persists alarm
+history centrally through the historian, and Phase 2's replicated store-and-forward buffer already
+guarantees delivery across a failover — so there is no reason to adopt a design whose stated
+trade-off is that a new active node re-seeds alarm state from scratch.
+
+## 6. Decisions (settled 2026-07-21)
+
+### 6.1 DECIDED — one central DB, and driver nodes never connect to it
+
+**Verified in ScadaBridge before adopting.** `Program.cs` calls
+`AddConfigurationDatabase(configDbConnectionString)` at line 264 — inside the Central branch
+(89–478). The Site branch begins at line 479. **Site nodes never register the central configuration
+database at all**; they receive config from central (notify → token-gated HTTP fetch) and persist it
+locally. There is still exactly one authoritative configuration database — it is simply
+central-only.
+
+OtOpcUa adopts the same shape: **the ConfigDb stays the single source of truth, and driver nodes stop
+connecting to it.** They obtain their configuration from the central nodes and cache it locally in
+LocalDb.
+
+**This promotes LocalDb from an outage cache to the driver node's steady-state configuration store.**
+That is a reframing of the Phase 1 work rather than a contradiction of it: boot-from-cache stops
+being the exceptional path and becomes the normal one, and "central SQL unreachable" stops being a
+degraded mode for driver nodes because they never held a connection to lose. The chunking,
+SHA-256 verification, newest-2 retention and pair replication all carry over unchanged.
+
+**Scope — five driver-side ConfigDb consumers must be re-homed or re-sourced.** Audited:
+
+| Consumer | Current use | Disposition |
+|---|---|---|
+| `DriverHostActor` | artifact fetch + ack writes (5 `CreateDbContext` sites) | artifact via fetch-and-cache; acks via the ack transport |
+| `EfAlarmConditionStateStore` | Part 9 alarm condition state | **move to LocalDb** — it is pair-local state, the same journey Phase 2 made for the alarm S&F buffer |
+| `DbHealthProbeActor` | DB reachability feeding ServiceLevel tiering | **semantics change** — a driver node has no DB to probe; decide what health input replaces it |
+| `OpcUaPublishActor` | (audit required) | TBD |
+| `ServiceCollectionExtensions` | registration | follows the above |
+
+`DbHealthProbeActor` deserves particular care: DB health is currently a ServiceLevel input, and
+under this model it stops existing for driver nodes. That is a behaviour change on a client-visible
+value, not just an internal refactor.
+
+### 6.2 CORRECTED — the gap was closed in ScadaBridge, and OtOpcUa still has it
+
+**An earlier revision of this section was wrong.** It reported the two-node keep-oldest outage gap as
+still open, citing `Component-ClusterInfrastructure.md:113` and Gitea PR
+[#12](https://gitea.dohertylan.com/dohertj2/ScadaBridge/pulls/12) (which made the failover drill
+*honest* about the gap rather than closing it). That doc line was stale: ScadaBridge closed the gap
+in `cf3bd52f`, *"feat(cluster): auto-down downing strategy — either-node crash now fails over (owner
+decision 2026-07-21: availability over partition-safety)"*.
+
+Its commit message carries the analysis, live-proven on their rig:
+
+> *"Two-node keep-oldest could NEVER survive a crash of the oldest/active node: Akka.NET 1.5.62
+> `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side with >= 2 members, so the 1-vs-1
+> survivor takes `DownReachable` and downs ITSELF — proven live on the rig ('SBR took decision …
+> including myself') before this change. `static-quorum(1)` is worse (`IsTooManyMembers` -> `DownAll`);
+> `keep-majority` just re-keys the fatal crash to the lowest address."*
+
+**OtOpcUa is running that exact configuration today** —
+`Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49-53`: `active-strategy = "keep-oldest"`,
+`keep-oldest { down-if-alone = on }`, `stable-after = 15s`, with
+`BuildClusterOptions` setting the matching typed `KeepOldestOption { DownIfAlone = true }`.
+
+So a two-node OtOpcUa pair **could not fail over when the oldest node crashed**: the survivor
+downed itself. That was a total-outage defect in the redundancy feature itself, independent of this
+design.
+
+**FIXED 2026-07-21 (Phase 0a).** `Cluster:SplitBrainResolverStrategy` now selects the downing
+provider and defaults to `auto-down` — Akka's `AutoDowning` with a 15s window — so the leader among
+the reachable members downs the unreachable peer and a crash of either node fails over. `keep-oldest`
+remains selectable for deployments that prefer partition-safety. Pinned by
+`SplitBrainResolverActivationTests`, which asserts the provider read back off a running ActorSystem
+rather than the configured one. See `docs/Redundancy.md`.
+
+**Its live gate is still outstanding**, and cannot run on today's rig: the failure only appears in a
+1-vs-1 split, and `docker-dev` is a single six-node mesh. The crash-the-oldest drill therefore lands
+with Phase 6/7, which makes every mesh exactly two nodes.
+
+**The bootstrap half of the gap is also closed (2026-07-22).** Downing was only ever one of the two
+outage modes: even with `auto-down`, a node cold-starting while its peer was dead looped on `InitJoin`
+forever, because Akka lets only `seed-nodes[0]` form a new cluster. The fix is **self-first seed
+ordering** — each seed node lists itself first, enforced at boot by `AkkaClusterOptionsValidator`.
+The `Cluster:SelfFormAfter` watchdog first shipped for this is deleted: a timer outside Akka's join
+handshake cannot tell "no seed answered" from "a seed answered and the join is in flight", and it
+islanded a manually-failed-over node live. Pinned by `SelfFirstSeedBootstrapTests`; see
+`docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Under this design each pair node is a
+seed of its own two-node mesh, so the ordering covers both sides and the current site-node exemption
+disappears.
+
+### 6.3 DECIDED — match ScadaBridge's auth posture for now
+
+Inter-cluster transports stay unauthenticated and unencrypted, as theirs are: the boundary assumes a
+trusted network. Recorded as an accepted risk rather than an oversight, with two notes for whenever
+it is revisited: `LocalDbSyncAuthInterceptor` is already in our tree as a fail-closed template, and
+the surface most worth gating first is any Pull-style RPC that returns historical rows.
+
+## 7. Sequencing sketch
+
+Deliberately not a task plan — per-phase plans follow, one at a time.
+
+| Phase | Content | Independent of the split? |
+|---|---|---|
+| 0a | Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2) | **Yes** — **DONE 2026-07-21**, live gate deferred to Phase 7 |
+| 0b | Oldest-Up role derivation (§4) | **Yes** — **DONE 2026-07-21** |
+| 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | **Yes** — **DONE 2026-07-22** (see below) |
+| 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No |
+| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |
+| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No |
+| 5 | gRPC stream contract; migrate the seven observability topics | No |
+| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No |
+| 7 | Failover drill (both directions, per §6.2); live gate | No |
+
+Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each
+deserves its own live gate.
+
+### Phase 1 as shipped (2026-07-22)
+
+Plan: `2026-07-21-per-cluster-mesh-phase1.md`. Two deviations from the sketch above, both decided
+before implementing:
+
+- **No cluster-scope filtering of the expected-ack set.** The plan called for it, but there is no
+  cluster-scoped deployment to filter on: `Deployment` has no `ClusterId`,
+  `ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and
+  `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The
+  expected set is every enabled `ClusterNode` row, which is what the membership rule produced too.
+- **No per-node role column.** The membership rule filtered on the `driver` role and the DB has none.
+  Rather than add one — a second declaration of node roles, free to drift from `Cluster:Roles` — every
+  `ClusterNode` row is now *defined* to be a driver node. An admin-only node must not be given one.
+
+Also shipped beyond the sketch: `ClusterNodeAddressReconciler` (Task 4), an admin singleton that
+catches `AkkaPort` drifting from the node's own `Cluster:Port`. **Phase 2 must revisit it** — once the
+meshes split, an admin node cannot see site members and every site row would report
+`EnabledRowNotInCluster` forever.
+
+## 8. Risks
+
+- **LocalDb becomes load-bearing for configuration, not just resilient.** Phase 1 built it as a
+  fallback; §6.1 makes it the driver node's only config store. A cache defect that was previously a
+  degraded-mode bug becomes a total-availability bug — the #485 unreadable-artifact class, with the
+  blast radius raised.
+- **`DbHealthProbeActor` feeds a client-visible value.** Removing the driver-side DB connection
+  changes what ServiceLevel means on those nodes; that is an interop-visible change, not internal.
+- **The rig currently models the topology that would be abolished.** Phase 6 rewrites
+  `docker-dev/docker-compose.yml` substantially, and every live gate that depends on it.
+- **Losing gossip loses free fleet-wide observability.** Seven AdminUI panels are fed by DPS today;
+  each needs an explicit stream and a reconnect story.
+- **Akka frame size.** We are clean on the deploy path, but anything new that carries payload over
+  ClusterClient inherits the 128 KB default with `log-frame-size-exceeding` off — a silent
+  single-message drop that leaves the association healthy. If we send payload at all, set both.
+- **Do not stack application-level last-write-wins on LocalDb's HLC.** Their `SiteStorageService`
+  carries this warning in code. Verified: our `deployment_pointer` upsert is unconditional and the
+  alarm sink is idempotent on a payload hash, so we are currently clean — keep it that way.

docs/plans/2026-07-21-per-cluster-mesh-phase1.md

@@ -0,0 +1,222 @@
+# Per-Cluster Mesh — Phase 1 plan
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+
+**Parent design:** [`2026-07-21-per-cluster-mesh-design.md`](2026-07-21-per-cluster-mesh-design.md).
+Phases 0a and 0b are done and merged; this is the next phase.
+
+**Goal:** Give `ClusterNode` the transport addresses central will need to reach each cluster
+directly, and cut `ConfigPublishCoordinator`'s last dependency on shared cluster membership by
+sourcing its expected-ack set from those rows.
+
+**Architecture:** Additive schema change plus one substitution inside the coordinator. Nothing else
+moves; the deploy channel stays on DistributedPubSub until Phase 2.
+
+**Status: DONE 2026-07-22** (branch `feat/mesh-phase1`). Live gate:
+[`2026-07-22-mesh-phase1-live-gate.md`](2026-07-22-mesh-phase1-live-gate.md) — **PASSED**, after gate
+step 4 found that the escape hatch this phase depends on did not exist. Three deviations from the
+plan below, all decided before implementing or forced by the gate:
+
+1. **Task 3's cluster-scope filtering was dropped** — there is no cluster-scoped deployment to filter
+   on. See the gate doc, step 2.
+2. **No per-node role column.** Every `ClusterNode` row is now *defined* to be a driver node.
+3. **Task 7 added:** `ClusterNode.MaintenanceMode`. `Enabled = 0` is rejected by
+   `DraftValidator.ValidateClusterTopology` on any 2-node pair, so it could never be the maintenance
+   hatch — which the plan called "the check that makes step 3 acceptable to ship".
+
+---
+
+## Why this is smaller than the design implies
+
+The design says "`ClusterNode` gains Akka + gRPC address columns; coordinator sources its
+expected-ack set from the DB", which reads as though the two are one change. They are not, and the
+second does not depend on the first.
+
+`ConfigPublishCoordinator.DiscoverDriverNodes` builds `NodeId.Parse($"{host}:{port}")` from
+`Cluster.State.Members`. `ClusterNode.NodeId` is **already in that identifier space** — the rig seeds
+`central-1:4053`, `site-a-1:4053` and so on (`docker-dev/seed/seed-clusters.sql:65`), and its comment
+says so explicitly, because `NodeDeploymentState.NodeId` is FK-bound to `ClusterNode.NodeId` and a
+mismatch throws FK 547 on deploy. The coordinator already writes `NodeDeploymentState` rows keyed by
+the membership-derived id and they already satisfy that FK — which is standing proof the two sets
+agree today.
+
+So the coordinator switch is a substitution inside one private method, and the address columns are
+purely forward-looking groundwork for Phase 2's ClusterClient and Phase 5's gRPC dial.
+
+## The decision this phase forces
+
+**Membership-derived and DB-derived expected-ack sets are not the same set, and the difference is a
+behaviour change on the deploy path.**
+
+| Situation | Today (membership) | After (DB rows) |
+|---|---|---|
+| Configured node is running | expected, acks, seals | same |
+| Configured node is **stopped** | not expected — deploy seals **successfully** without it | expected — deploy waits, then **fails at the apply deadline** |
+| Node running but not in `ClusterNode` | expected (and then FK 547 on the state row) | not expected; its ack is discarded |
+
+The middle row is the one that matters. Today a deployment can seal green while a node that should
+have received it was down and did not — the operator is told the fleet is deployed when it is not.
+Failing instead is arguably the correct behaviour and is the reason the design moves this to the DB
+(central cannot see cluster membership once the meshes split).
+
+But it is a behaviour change, and it needs an operator escape hatch or a node taken down for
+maintenance blocks every deployment. `ClusterNode.Enabled` already exists and is exactly that hatch,
+so the query filters on it: **`Enabled = 1` rows are expected to ack; disabled rows are not.**
+
+**Confirmed 2026-07-22 before executing Task 3** — DB-derived with the hatch. (The hatch turned out to be `MaintenanceMode`, not `Enabled`; see Task 7.) If the preference is to keep sealing green past a down
+node, the alternative is to intersect the DB set with current membership, which preserves today's
+behaviour but does not survive the mesh split — it would have to be revisited in Phase 2 anyway.
+
+---
+
+## Task 1: Address columns on `ClusterNode`
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** none (Task 2 migrates what this declares)
+
+**Files:**
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/ClusterNode.cs`
+- Modify: the `ClusterNode` entity configuration in `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/` (find the `IEntityTypeConfiguration<ClusterNode>` or the `OnModelCreating` block)
+- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/` (mirror an existing entity-mapping test)
+
+Add two nullable columns beside the existing `OpcUaPort` / `DashboardPort`:
+
+- `int AkkaPort` — default `4053`. The remoting port central will use to build the ClusterClient
+  contact point. Non-nullable with a default, because every node has one.
+- `int? GrpcPort` — nullable, no default. The Phase 5 telemetry-stream port. Nullable is deliberate:
+  it does not exist yet, and a non-null default would assert a port nothing listens on.
+
+Document on each property that these are **central's dial targets**, not the node's own binding
+configuration — the node binds from `Cluster:Port` in its own appsettings, and these columns
+duplicate that value for a reader that cannot see the node's config. That duplication is the reason
+Task 4 exists.
+
+**Step 1:** Write a mapping test asserting a round-tripped `ClusterNode` preserves both values and
+that `AkkaPort` defaults to 4053.
+**Step 2:** Run it; expect failure to compile.
+**Step 3:** Add the properties + mapping.
+**Step 4:** Run; expect pass.
+**Step 5:** Commit.
+
+## Task 2: EF migration
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+**Parallelizable with:** none
+
+**Files:**
+- Create: a new migration under the Configuration project's `Migrations/` folder
+
+```bash
+dotnet ef migrations add AddClusterNodeTransportPorts \
+  --project src/Core/ZB.MOM.WW.OtOpcUa.Configuration
+```
+
+Verify the generated `Up` adds `AkkaPort` with `defaultValue: 4053` and `GrpcPort` as nullable, and
+that `Down` drops both. **Check the migration is additive only** — an `AlterColumn` on anything else
+means the model had drifted from the last migration and that drift must be understood before this
+ships, not carried along by it.
+
+## Task 3: Coordinator sources the expected-ack set from `ClusterNode`
+
+**Classification:** high-risk — this is the deploy path
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs` (`DiscoverDriverNodes`, ~line 262, and the class doc comment at line 21 which states the membership rule)
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ConfigPublishCoordinatorTests.cs`
+
+Replace the `Cluster.State.Members` scan with a query over enabled `ClusterNode` rows. Keep
+`NodeIdComparer` — the case-insensitivity rationale in its comment applies unchanged, and SQL
+collation makes it more relevant, not less.
+
+Points to get right:
+
+- **Filter on `Enabled`.** See "The decision this phase forces".
+- **Scope to the deployment's cluster if the deployment is cluster-scoped.** `DeploymentArtifact.ResolveClusterScope`
+  already defines that scoping; a fleet-wide deployment expects every enabled node, a cluster-scoped
+  one expects only that cluster's. Getting this wrong makes every cluster-scoped deploy hang on nodes
+  that were never sent it. **Read `ResolveClusterScope` before writing the query.**
+- **The empty-set branch still matters.** `HandleDispatch` seals immediately when the set is empty,
+  and its log message says "no driver-role members in cluster" — reword it, because the reason is now
+  "no enabled ClusterNode rows", which is a configuration error rather than a topology observation.
+- The DB is read inside `HandleDispatch`, which already opens a context; reuse it rather than opening
+  a second.
+
+**Tests to add:** enabled-only filtering; cluster-scope filtering; empty set seals with the new
+reason; an ack from a node absent from `ClusterNode` is discarded. **Positive control on each:**
+delete the filter and confirm the test goes red — the existing suite passed throughout the
+membership era, so a test that also passes against the old derivation is testing nothing.
+
+## Task 4: Reconcile the duplicated port with the node's own config
+
+**Classification:** standard
+**Estimated implement time:** ~4 min
+**Parallelizable with:** none
+
+`ClusterNode.AkkaPort` and the node's own `Cluster:Port` are the same fact in two places, and nothing
+makes them agree. A node that binds 4054 while its row says 4053 is unreachable from central in Phase
+2, and the symptom will be a silent absence of acks rather than an error.
+
+Options, in preference order:
+
+1. **Have the node assert its row on startup** — on join, compare `Cluster:Port` /
+   `Cluster:PublicHostname` to its own `ClusterNode` row and log an Error (not a Warning) on
+   mismatch. Cheap, no schema change, no write path from driver nodes to the ConfigDb (which Phase 4
+   removes anyway — so put this check on an **admin** node, reading the membership it can already see).
+2. Deploy-time validation in `DraftValidator`, alongside the existing gates.
+
+Pick one and implement it. Do not skip this task: an unenforced duplicated address is the exact shape
+of the `Modbus`/`ModbusTcp` drift and the `TwinCat`/`Focas` drift already recorded in this repo.
+
+## Task 5: Rig seed + docs
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+
+**Files:**
+- Modify: `docker-dev/seed/seed-clusters.sql` — add `AkkaPort` to the six INSERTs (all 4053) so a
+  freshly-seeded rig matches a migrated one. Leave `GrpcPort` null.
+- Modify: `docs/Configuration.md` — document both columns.
+- Modify: `docs/plans/2026-07-21-per-cluster-mesh-design.md` — mark Phase 1 done in §7.
+
+## Task 6: Live gate
+
+**Classification:** high-risk
+**Estimated implement time:** n/a — operator-driven
+
+Offline tests cannot cover the substitution's real risk, which is that the DB set and the live set
+disagree on the rig. Run on `docker-dev`:
+
+1. Deploy with all six nodes up → seals green, six `NodeDeploymentState` rows, no FK 547.
+2. Deploy a **cluster-scoped** config → expected set is that cluster's nodes only.
+3. `docker stop` one node, deploy → **fails at the apply deadline** naming the missing node. This is
+   the new behaviour; confirm the failure is legible, because an operator will meet it.
+4. Set that node's `Enabled = 0`, deploy again → seals green without it. This proves the escape
+   hatch, and is the check that makes step 3 acceptable to ship.
+
+Record results in a gate doc beside this plan, per the Phase 1/Phase 2 precedent.
+
+## Task 7: `MaintenanceMode` — the escape hatch step 4 proved missing (ADDED 2026-07-22)
+
+**Classification:** high-risk — deploy path + schema
+**Why:** gate step 4 returned `422 ClusterEnabledNodeCountMismatch`. `Enabled = 0` on one node of a
+Warm/Hot pair fails `DraftValidator.ValidateClusterTopology` (enabled count must equal `NodeCount`),
+so the hatch Task 3 relies on cannot be used on any 2-node cluster — i.e. every cluster in the target
+topology. Task 3's behaviour change is only shippable with a hatch that works.
+
+**Decision:** split the meanings rather than weaken either rule.
+
+| Flag | Question | Enforced by |
+|---|---|---|
+| `Enabled` | part of the declared topology? | `DraftValidator` vs `ServerCluster.NodeCount` |
+| `MaintenanceMode` | expected to participate right now? | `ConfigPublishCoordinator`, `ClusterNodeAddressReconciler` |
+
+Rejected: counting configured rather than enabled nodes (removes the InvalidTopology guard);
+downgrading the rule to a warning (weakens a deploy gate for everyone); shipping without a hatch.
+
+**Done:** column + migration `AddClusterNodeMaintenanceMode`; both consumers filter on
+`Enabled && !MaintenanceMode`; deadline + reconciler messages name the right flag; docs; gate step 4
+re-run green.

docs/plans/2026-07-21-per-cluster-mesh-phase1.md.tasks.json

@@ -0,0 +1,76 @@
+{
+  "plan": "docs/plans/2026-07-21-per-cluster-mesh-phase1.md",
+  "branch": "feat/mesh-phase1",
+  "decisions": {
+    "seal-semantics": "DB-derived + Enabled hatch (confirmed 2026-07-22)",
+    "cluster-scope": "Dropped \u2014 no cluster-scoped deployment exists; expected set = all enabled ClusterNode rows",
+    "node-roles": "Every ClusterNode row is a driver node; documented + pinned by test, no Roles column",
+    "adminui-edit": "Deferred to Phase 2 (nothing reads the columns until then)",
+    "maintenance-hatch": "MaintenanceMode column, not Enabled=0 (blocked by DraftValidator on any 2-node pair)"
+  },
+  "tasks": [
+    {
+      "id": 1,
+      "subject": "Address columns on ClusterNode",
+      "status": "completed",
+      "commit": "2bbb0271",
+      "blockedBy": []
+    },
+    {
+      "id": 2,
+      "subject": "EF migration AddClusterNodeTransportPorts",
+      "status": "completed",
+      "commit": "da74ebd6",
+      "blockedBy": [
+        1
+      ]
+    },
+    {
+      "id": 3,
+      "subject": "Coordinator sources expected-ack set from ClusterNode",
+      "status": "completed",
+      "commit": "d88e2455",
+      "blockedBy": [
+        2
+      ]
+    },
+    {
+      "id": 4,
+      "subject": "Reconcile duplicated AkkaPort with the node's own Cluster:Port",
+      "status": "completed",
+      "blockedBy": [
+        2
+      ],
+      "commit": "90c1302f"
+    },
+    {
+      "id": 5,
+      "subject": "Rig seed + docs",
+      "status": "completed",
+      "blockedBy": [
+        3,
+        4
+      ],
+      "commit": "57e1d017"
+    },
+    {
+      "id": 6,
+      "subject": "Live gate on docker-dev",
+      "status": "completed",
+      "blockedBy": [
+        5
+      ],
+      "result": "live-gate PASSED (2026-07-22-mesh-phase1-live-gate.md)"
+    },
+    {
+      "id": 7,
+      "subject": "MaintenanceMode \u2014 escape hatch the live gate proved missing",
+      "status": "completed",
+      "commit": "ee69caa2",
+      "blockedBy": [
+        6
+      ],
+      "origin": "added from gate step 4 failure"
+    }
+  ]
+}

docs/plans/2026-07-22-mesh-phase1-live-gate.md

@@ -0,0 +1,134 @@
+# Per-cluster mesh Phase 1 — live gate record
+
+**Date:** 2026-07-22 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase1`
+**Plan:** [`2026-07-21-per-cluster-mesh-phase1.md`](2026-07-21-per-cluster-mesh-phase1.md)
+
+**Result: PASSED**, after the gate found a blocker that changed the design (step 4).
+
+Offline tests cannot cover the substitution's real risk — that the DB set and the live set disagree on
+a running fleet. This is that check.
+
+## Setup
+
+Migration applied by the rig's `migrator` service against the shared `OtOpcUa` database, then all six
+host nodes restarted on the rebuilt image. **All six pre-existing `ClusterNode` rows inherited
+`AkkaPort = 4053` from the migration's `DEFAULT 4053`** — the row-level evidence that the default does
+its job for rows predating the column:
+
+```
+central-1:4053|central-1|4053|NULL|1
+central-2:4053|central-2|4053|NULL|1
+site-a-1:4053 |site-a-1 |4053|NULL|1
+site-a-2:4053 |site-a-2 |4053|NULL|1
+site-b-1:4053 |site-b-1 |4053|NULL|1
+site-b-2:4053 |site-b-2 |4053|NULL|1
+   NodeId     |  Host   |Akka|Grpc|En
+```
+
+## Step 1 — deploy with all six nodes up → **PASS**
+
+`POST /api/deployments` → `202 Accepted`, deployment `38f83b37`.
+
+`Deployment.Status = 2` (Sealed). Six `NodeDeploymentState` rows, every one `Status = 1` (Applied).
+**No FK 547** — the DB-derived NodeIds satisfy the `NodeDeploymentState → ClusterNode` foreign key,
+which is the standing proof that the DB set and the membership set agree on this rig.
+
+## Step 2 — *dropped, not skipped*
+
+The plan called for a cluster-scoped deploy. **There is no such thing.** `Deployment` has no
+`ClusterId`, `ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and
+`DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. Removed
+from the plan rather than faked green.
+
+## Step 3 — deploy with one node stopped → **PASS** (the new behaviour)
+
+`docker stop otopcua-dev-site-b-2-1`, then deploy `37d72301`.
+
+`Deployment.Status = 4` (TimedOut) after the 2-minute apply deadline. Five nodes `Applied`,
+`site-b-2:4053` still `Applying`. Under the membership rule this deployment would have **sealed
+green** with five nodes and never mentioned the sixth.
+
+The operator-facing log, which the plan required to be legible:
+
+```
+[12:52:02 WRN] Deployment 37d723018e374c0eb04c7b6beccc3b3e timed out after 00:02:00 (5/6 acks
+landed). No ack from: site-b-2:4053. Each is an enabled ClusterNode row — start the node, or set
+ClusterNode.Enabled = 0 if it is down for maintenance, then redeploy
+```
+
+(The remedy in that message was corrected to `MaintenanceMode = 1` — see step 4.)
+
+## Step 4 — the escape hatch → **FAILED, then fixed and re-run → PASS**
+
+### First attempt: the hatch did not exist
+
+`UPDATE ClusterNode SET Enabled = 0 WHERE NodeId = 'site-b-2:4053'`, then deploy:
+
+```
+HTTP 422
+[ClusterEnabledNodeCountMismatch] Cluster 'SITE-B' declares NodeCount=2 but has 1 Enabled nodes.
+Toggle the missing node(s) back on or change RedundancyMode/NodeCount to match.
+```
+
+`DraftValidator.ValidateClusterTopology` rejects the deployment **before it dispatches**. So
+`Enabled = 0` cannot be used on a node of a 2-node cluster — and every cluster on this rig, and every
+cluster in the program plan's target topology, is a 2-node pair.
+
+**This was a blocker, not a wrinkle.** The plan called step 4 *"the check that makes step 3
+acceptable to ship"*. Without it, a node down for maintenance blocks every deployment to its cluster,
+with no remedy short of editing `NodeCount` / `RedundancyMode` — which changes the runtime redundancy
+posture to work around a deploy gate.
+
+The two rules were each reasonable and contradictory together: the validator reads `Enabled` as *"part
+of the declared topology"*, Phase 1 additionally read it as *"expect an ack"*.
+
+### Fix (Task 7): split the meanings
+
+New `ClusterNode.MaintenanceMode bit NOT NULL DEFAULT 0`. `Enabled` keeps its topology meaning and the
+validator is untouched; the coordinator's expected-ack set and the address reconciler both filter on
+`Enabled && !MaintenanceMode`.
+
+### Re-run: PASS
+
+`docker stop` `site-b-2` + `MaintenanceMode = 1`, then deploy `82c5e678`:
+
+- `202 Accepted` — the topology gate passes, because the node is still `Enabled` (`1|1`).
+- `Deployment.Status = 2` (Sealed).
+- **Five** `NodeDeploymentState` rows, all `Applied`. `site-b-2:4053` has no row at all — excluded
+  from the expected set rather than waited for and forgiven.
+
+## Task 4 (reconciler) — live behaviour, not a planned gate step
+
+Observed unprompted during the rolling restart, which is better evidence than a staged check:
+
+```
+[12:44:49 WRN] ClusterNode address check [EnabledRowNotInCluster] site-a-1:4053: ...
+[12:44:49 WRN] ... site-a-2:4053 ... site-b-1:4053 ... site-b-2:4053
+[12:44:56 INF] ClusterNode addresses reconcile with cluster membership (6 driver members)
+```
+
+It warned per row while the site nodes were still starting, then converged to a single clean line —
+and **logged nothing further**, confirming the change-detection guard suppresses repeats on the
+5-minute sweep.
+
+Maintenance interaction, from the step 4 re-run:
+
+```
+[13:06:02 WRN] ... site-b-2:4053 ...            ← node dropped, flag not yet set
+[13:06:05 INF] ... reconcile with cluster membership (5 driver members)   ← flag set: silent
+```
+
+The transient warning is honest — for those three seconds the row genuinely was enabled, present in
+the expected-ack set, and absent from the cluster.
+
+## What this gate did not cover
+
+- **Phase 2's premise.** `AkkaPort` / `GrpcPort` are groundwork; nothing dials them yet, so "the value
+  is correct" is only checked by the reconciler comparing it to gossip. The first real exercise is
+  Phase 2.
+- **The reconciler under split meshes.** On the current single mesh an admin node sees every driver
+  member. After Phase 6 it will not, and every site row would report `EnabledRowNotInCluster`
+  permanently. Recorded as a known limitation on the class and in the design doc.
+- **`Enabled = 0` on a single-node (`RedundancyMode.None`) cluster.** Untested — the rig has no such
+  cluster. It would fail the same topology rule (0 enabled vs `NodeCount = 1`), so `MaintenanceMode`
+  is the hatch there too.

docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md.tasks.json

@@ -0,0 +1,141 @@
+{
+  "planPath": "docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md",
+  "note": "Per-cluster mesh Phase 2. Decisions pre-made: config-flagged dark switch (MeshTransport:Mode), buffer-size=0, alarm-commands central leg in scope. Three design-doc corrections and the single-mesh duplicate-delivery trap are documented in the plan header \u2014 read them before Task 3.",
+  "tasks": [
+    {
+      "id": 0,
+      "subject": "Task 0: MeshTransportOptions + validator + registration",
+      "status": "completed",
+      "classification": "small",
+      "parallelizableWith": [
+        1
+      ],
+      "commit": "5439f148"
+    },
+    {
+      "id": 1,
+      "subject": "Task 1: receptionist extension, buffer-size=0, frame-size logging (effective-config test)",
+      "status": "completed",
+      "classification": "small",
+      "parallelizableWith": [
+        0
+      ],
+      "commit": "7b71a6a3",
+      "note": "First draft of the buffering test was VACUOUS -- Config.GetInt returns 0 for a missing key, so it passed before the feature existed. Rewritten to assert through ClusterClientSettings/ClusterReceptionistSettings."
+    },
+    {
+      "id": 2,
+      "subject": "Task 2: MeshCommand envelope + generation-suffixed ClusterClient factory",
+      "status": "completed",
+      "classification": "small",
+      "blockedBy": [
+        0
+      ],
+      "commit": "0d6669d5",
+      "note": "Dropped a third test that asserted ClusterClientSettings directly and never touched the factory."
+    },
+    {
+      "id": 3,
+      "subject": "Task 3: CentralCommunicationActor \u2014 dark-switched SendToAll fan-out, DB-sourced contacts, ApplyAck forward",
+      "status": "completed",
+      "classification": "high-risk",
+      "blockedBy": [
+        1,
+        2
+      ],
+      "parallelizableWith": [
+        4
+      ],
+      "commit": "d5b5cb6e",
+      "note": "SendToAll sabotage-verified. Two test-harness bugs found: SubscribeAck goes to the SENDER, and EventFilter matches case-INSENSITIVELY (a 'was NOT' filter also caught 'was not created')."
+    },
+    {
+      "id": 4,
+      "subject": "Task 4: NodeCommunicationActor \u2014 inbound to node-local EventStream, outbound ApplyAck",
+      "status": "completed",
+      "classification": "standard",
+      "blockedBy": [
+        1,
+        2
+      ],
+      "parallelizableWith": [
+        3
+      ],
+      "commit": "915beec1",
+      "note": "No Ask crosses the boundary in Phase 2, so this actor is far smaller than the sister project's."
+    },
+    {
+      "id": 5,
+      "subject": "Task 5: node subscribers accept EventStream too; rename _coordinatorOverride -> _ackRouter",
+      "status": "completed",
+      "classification": "standard",
+      "blockedBy": [
+        4
+      ],
+      "commit": "5cb0e721",
+      "note": "Two wiring tests added -- unit tests either side of the seam both pass with the subscription deleted. First draft raced (ActorOf returns before PreStart; EventStream publish to nobody is dropped silently); alarm version was vacuous until the warm-up moved outside the assertion window."
+    },
+    {
+      "id": 6,
+      "subject": "Task 6: wire both sides, receptionist registration, pin the wire-contract actor paths",
+      "status": "completed",
+      "classification": "standard",
+      "blockedBy": [
+        3,
+        5
+      ],
+      "commit": "16d59856",
+      "note": "Dropped a planned 'not a singleton' assertion: two candidate discriminators were both invalid (a SingletonManager also sits at /user/<name>; /singleton is null even for the known singleton config-publish). A sabotage re-registering via WithSingleton left both green, which exposed them. Property deferred to Task 8."
+    },
+    {
+      "id": 7,
+      "subject": "Task 7: route the five publish sites through the mesh router (nullable, mediator fallback)",
+      "status": "completed",
+      "classification": "standard",
+      "blockedBy": [
+        6
+      ],
+      "note": "Sabotage-verified by inverting the branch: all 6 tests red. Comm actor registration moved above the singletons so registry.Get works; an earlier ResolveOne().GetAwaiter().GetResult() blocked inside a props factory."
+    },
+    {
+      "id": 8,
+      "subject": "Task 8: real two-ActorSystem ClusterClient boundary test + falsifiability control",
+      "status": "completed",
+      "classification": "high-risk",
+      "blockedBy": [
+        7
+      ],
+      "parallelizableWith": [
+        9
+      ],
+      "commit": "5468b79d",
+      "note": "Not flaky (3/3 clean repeats; 4 tests, ~16s). Both legs sabotage-verified against production code, and the control was itself falsified by restoring the receptionist. The control necessarily removes RegisterService too -- registering would materialise the extension it is removing. Task 6's dropped per-node/singleton property does NOT land here (one node per side cannot discriminate); it is live-gate step 8, and the MeshCommActorPathTests comment was corrected to say so."
+    },
+    {
+      "id": 9,
+      "subject": "Task 9: docker-dev rig config + docs + design-doc correction",
+      "status": "completed",
+      "commit": "e4cb9a08",
+      "note": "Rig Mode is ${OTOPCUA_MESH_MODE:-Dps} rather than a hardcoded Dps, so Task 10 can flip all six nodes at `docker compose up` with no compose edit and no rebuild. Verified via `docker compose config` (6 services resolved).",
+      "classification": "small",
+      "blockedBy": [
+        7
+      ],
+      "parallelizableWith": [
+        8
+      ]
+    },
+    {
+      "id": 10,
+      "subject": "Task 10: live gate on docker-dev (11 steps; step 4 = deploy-seal semantics, step 8 = per-node/contact-rotation discriminator [replaces the dropped Task 6 assertion \u2014 do not skip], step 9 = both-centrals drop behavior)",
+      "status": "completed",
+      "note": "PASSED for the transport (steps 1-5, 8, 11 green; flip end-to-end via OTOPCUA_MESH_MODE=ClusterClient). Record: 2026-07-22-mesh-phase2-live-gate.md. Steps 6/7 NOT RUN (no UI surface for driver-control \u2014 finding D, DriverStatusPanel is orphaned; no alarms configured). Step 8's stated log-signal does not exist (HandleApplyAck is silent on success \u2014 finding E); proven instead by both centrals independently logging per-instance ClusterClient creation + rebuild, which a singleton cannot. FOUR findings: A) 10-30s post-restart delivery window is PRE-EXISTING (DPS control failed identically); B) buffer-size=0 rationale OVERSTATED \u2014 a node applies a TimedOut deploy anyway via Bootstrap() orphan-row replay at +44s, corrected in 3 docs; C) stopping BOTH centrals strands non-seed nodes at the membership layer (Phase 7 drill, transport-independent); D) driver Restart/Reconnect UI orphaned since v3 Batch 2. Rig restored to Dps default.",
+      "classification": "high-risk",
+      "blockedBy": [
+        8,
+        9
+      ]
+    }
+  ],
+  "lastUpdated": "2026-07-22T00:00:00Z"
+}

docs/plans/2026-07-22-mesh-phase2-live-gate.md

@@ -0,0 +1,228 @@
+# Per-cluster mesh Phase 2 — live gate record
+
+**Date:** 2026-07-22 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase2`
+**Plan:** [`2026-07-22-mesh-phase2-clusterclient-transport.md`](2026-07-22-mesh-phase2-clusterclient-transport.md)
+
+**Result: PASSED for the transport**, with two gate steps not run (no surface exists to drive them),
+one step whose stated pass-signal does not exist in the code, and **four findings** — one of which
+corrects a claim this phase's own documentation makes.
+
+The rig was flipped end-to-end with `OTOPCUA_MESH_MODE=ClusterClient` (no rebuild, no compose edit —
+the env indirection added in Task 9 is what made the A/B and the DPS control cheap enough to run).
+
+---
+
+## Result table
+
+| # | Step | Result |
+|---|---|---|
+| 1 | Deploy on `Mode=Dps` | **PASS** — `48d0c7c5` Sealed, 6/6 Applied |
+| 2 | Flip all six to `ClusterClient`, redeploy | **PASS** — `5db8f6ad` Sealed, 6/6 Applied |
+| 3 | Confirm the transport actually changed | **PASS** — positive log evidence, see below |
+| 4 | Stop a node, deploy | **PASS** — `59b81248` TimedOut naming `site-b-2:4053`, 5/6 |
+| 5 | `MaintenanceMode = 1`, redeploy | **PASS** — `c6cb9812` Sealed, 5 rows, contacts 6→5 |
+| 6 | AdminUI Restart + Reconnect | **NOT RUN** — no UI surface exists (finding D) |
+| 7 | AdminUI alarm ack | **NOT RUN** — no alarms configured on the rig |
+| 8 | Per-node (not singleton) discriminator | **PASS** — via a better discriminator than the plan's |
+| 9 | Both centrals down, restart a driver node | **PARTIAL** — node survives; the stated ack-drop cannot occur |
+| 10 | Restart central, redeploy | **PASS only after operator action** (finding C) |
+| 11 | Frame-size canary | **PASS** — 0 warnings across all six nodes |
+
+## Step 3 — the transport really changed
+
+Central, both nodes, at boot under the flag:
+
+```
+central-1 [22:18:16 INF] Mesh ClusterClient created with 6 contact point(s)
+central-2 [22:18:17 INF] Mesh ClusterClient created with 6 contact point(s)
+```
+
+Node side, with `Serilog__MinimumLevel__Default=Debug` overlaid on `site-a-1` (the rig pins the
+default above Debug, so this line is invisible on a stock node):
+
+```
+site-a-1  [22:10:34 DBG] Mesh: received DispatchDeployment from central; publishing node-locally
+```
+
+Under `Mode=Dps` the same node logs nothing of the sort and central logs
+`Mesh transport is Dps: commands are published on DistributedPubSub and no ClusterClient is created`.
+
+## Step 5 — the DB-sourced contact set is live
+
+Setting `MaintenanceMode = 1` on `site-b-2` shrank both centrals' contact sets on the next refresh,
+and clearing it restored them — independently, on each central's own 60 s timer:
+
+```
+central-1 [22:22:16 INF] Mesh contact set changed; replacing the ClusterClient
+central-1 [22:22:16 INF] Mesh ClusterClient created with 5 contact point(s)
+central-1 [22:23:16 INF] Mesh ClusterClient created with 6 contact point(s)
+central-2 [22:22:16 INF] … identical, same timestamps
+```
+
+No `InvalidActorNameException` on any rebuild — the generation-suffixed client name (Task 2, ported
+from the sister project's shipped bug) does its job.
+
+## Step 8 — the per-node discriminator, replaced with a stronger one
+
+**The plan's pass-signal does not exist.** It asked for the surviving central's comm actor to *log
+receipt* of a node ack inside the downing window. `CentralCommunicationActor.HandleApplyAck` logs
+**nothing** on the success path — it only warns when the coordinator is unresolvable. There is no
+line to observe. (Recorded as finding E.)
+
+The evidence above is a **cleaner** discriminator and needs no timing race. `Mesh ClusterClient
+created` is emitted from `PreStart`, so it fires **once per live actor instance**. Both centrals
+logged it, and both logged their own independent contact-set rebuild at step 5. A
+`ClusterSingletonManager` runs the actor on the oldest node only — the other node hosts a manager
+that never starts the actor and would log nothing. **Two nodes logging it proves two live instances,
+i.e. per-node registration.** The node side is proven the same way: `SendToAll` collected 6/6 acks,
+which requires six separately registered node comm actors.
+
+The failover half was run anyway, and passed: `docker stop central-1` (the oldest, and the singleton
+host) → the surviving central accepted a deployment **4 seconds later** and collected acks from all
+five live nodes (`e3161c25`; it TimedOut only because `central-1` is itself a configured node that
+was down — correct Phase 1 semantics). Graceful stop hands the singleton over immediately, so no
+downing wait was involved.
+
+## Step 9 — the stated failure mode cannot occur
+
+The plan expected a restarting driver node to emit a boot-time ack that gets dropped with a Warning.
+**Driver nodes do not ack on boot.** `DriverHostActor.SendAck` is called only from the apply paths —
+a node with no dispatch to answer has nothing to send, so there is no drop to observe.
+
+What was verified: with **both** centrals stopped, `site-a-1` restarted, came up clean, logged its
+association failures against both central addresses, retried, and stayed up — no crash loop. The
+node-side ClusterClient's forever-retry (`reconnect-timeout = off`) behaves as configured.
+
+## Step 11 — the canary stayed quiet
+
+`grep -ci "frame size|frame-size|maximum-frame"` = **0** on all six nodes, as designed: the deploy
+notify carries only `DeploymentId` + `RevisionHash` + `CorrelationId`.
+
+---
+
+## Finding A — a 10–30 s post-restart delivery window, **pre-existing, not Phase 2**
+
+A deploy issued shortly after a driver node restarts does not reach that node, and the deployment
+fails at the 2-minute deadline naming it. Measured against `site-a-1`:
+
+| Delay after the node logs readiness | Result |
+|---|---|
+| ~4 s | **missed** (`f738e1de` TimedOut, node row `Applying`) |
+| 10 s | **missed** (`0c7a6296` TimedOut) |
+| 30 s | applied, sealed |
+
+**The DPS control fails identically.** Flipping the rig back to `Mode=Dps` and repeating the 10 s
+case produced `e85f53c5` → TimedOut, `site-a-1` unacked. So the window is a property of cluster
+membership convergence, not of the transport this phase introduces. Recorded rather than fixed.
+
+> Running that control is the only reason this is a note and not a Phase 2 regression. The
+> temptation to skip it — "obviously the new transport broke it" — would have produced a false
+> attribution and a wasted investigation.
+
+## Finding B — **`buffer-size = 0` does not deliver what this phase's docs claim it does**
+
+The stated rationale, in the plan, `CLAUDE.md`, and `docs/Configuration.md`, is that buffering
+*"would replay a `DispatchDeployment` on reconnect and apply a deployment the coordinator already
+sealed as `TimedOut`, leaving the node running a configuration the database records as failed."*
+
+**That outcome happens anyway, by a different route.** Deployment `0c7a6296` timed out at 22:14:30
+with `site-a-1` unacked. The node's row:
+
+```
+NodeId          Status  StartedAtUtc              AppliedAtUtc
+site-a-1:4053   1       2026-07-22 22:12:30.145   2026-07-22 22:15:14.823
+```
+
+It applied **44 s after the deployment was sealed TimedOut**, on its next restart. The mechanism is
+`DriverHostActor.Bootstrap()`, which reads the node's most recent `NodeDeploymentState`, finds the
+orphan `Applying` row the coordinator pre-created at dispatch, and replays it:
+
+```csharp
+case NodeDeploymentStatus.Applying:
+    _log.Warning("DriverHost {Node}: found orphan Applying row for deployment {Id}; replaying", …);
+    ApplyAndAck(new DeploymentId(latest.DeploymentId), revision.Value, CorrelationId.NewId());
+```
+
+So `buffer-size = 0` removes the **transport-level, timer-driven** replay — a real difference: it is
+silent, unbounded in time, and can fire without any node event. The **boot-time DB replay remains**,
+and it is the better-behaved of the two (deliberate, Warning-logged, and bounded to a restart). The
+decision stands; the *justification* was overstated and the docs are corrected in the same change as
+this record.
+
+## Finding C — stopping **both** centrals strands the non-seed nodes permanently
+
+After `docker stop` of both centrals followed by `docker start`, the fleet did **not** recover.
+Deployments reached 2/6 then 3/6 nodes. The cause is not the transport — the Akka cluster itself had
+split:
+
+```
+Cluster hosts:  MEMBERS 3   UP 3   UNREACHABLE 0
+  central-1:4053  UP   admin,driver   leader for admin+driver
+  central-2:4053  UP   admin,driver
+  site-a-1:4053   UP   driver
+```
+
+The three site nodes that never restarted were **not members at all**, and their own gossip showed
+the stale inverse view — `central-2` as `Exiting`, `site-a-1` as `Leaving` — frozen since 22:29:
+
+```
+Gossip(members = [central-2 status = Exiting, site-a-1 status = Leaving,
+                  site-a-2 Up, site-b-1 Up, site-b-2 Up])
+```
+
+A graceful stop puts the centrals into `Leaving`/`Exiting`; the processes died before the removal
+handshake completed, leaving the survivors holding members that will never confirm. `auto-down`
+does not help — these members are not *Unreachable*, they are stuck mid-`Exiting`. The restarted
+centrals formed a fresh cluster (with `site-a-1`, which had also restarted and re-joined through its
+seed), and the three stranded nodes had no path back: they believe they are already in a cluster, so
+they never re-run the join process.
+
+**Recovery required restarting the stranded nodes.** After that, the next deploy sealed 6/6
+(`e20836ac`).
+
+This is transport-independent — membership is below both DPS and ClusterClient, and the set of nodes
+that acked matched the cluster view exactly. It is a **downing/rejoin gap of the current single-mesh
+topology**, in the same family as the registered outage gap the design doc already records, and it
+belongs to the Phase 7 drill work. Under the target per-pair topology (every node a seed of its own
+pair) the shape changes, and this drill must be re-run there rather than assumed fixed.
+
+## Finding D — the AdminUI's driver Restart/Reconnect buttons have no page
+
+Step 6 could not be run, and not for a rig reason.
+`Components/Shared/Drivers/DriverStatusPanel.razor` is the **only** component in the codebase
+carrying the `DriverOperator`-gated Reconnect/Restart buttons, and **nothing renders it** — `grep -rn
+"<DriverStatusPanel"` across `src/` returns nothing, and no `@page` route under `Components/Pages/`
+hosts it. It was orphaned when v3 Batch 2 retired the routed `/clusters/{id}/drivers` flow.
+
+So the `driver-control` channel has no operator surface **on either transport** — this is a
+pre-existing v3 regression that Phase 2 merely surfaced. Filed as a follow-up; not fixed here.
+
+**Residual risk for the channel:** central-side dispatch is sabotage-verified by
+`MeshRouterDispatchTests` (all four `AdminOperationsActor` call sites go through one `Dispatch`
+helper); node-side `EventStream` receipt by `DriverHostActor` is proven **live** by this gate, since
+`DispatchDeployment` arrives through the identical `PublishLocally` path. What is unproven live is
+only the composition for those two message types.
+
+Step 7 (alarm ack) was likewise not run: the rig has no scripted alarms configured
+(`ScriptedAlarmHost: applying (enabled=0/0, depRefs=0)`) and `/alerts` is empty, so there is nothing
+to acknowledge. Authoring a scripted alarm end-to-end was judged out of proportion to the residual
+risk above. `ScriptedAlarmHostActor`'s `AlarmCommand` EventStream subscription therefore rests on its
+(sabotage-verified) unit test.
+
+## Finding E — the central comm actor is silent on the success path
+
+`HandleApplyAck` logs only on failure. During this gate that made "did the ack reach the surviving
+central?" unanswerable from logs, which is exactly the question an operator asks during a failover.
+A Debug line on ack receipt would have made step 8 directly observable. Not added during the gate —
+adding code to make a gate step pass is the wrong order — but recorded as a follow-up.
+
+---
+
+## What this gate did not cover
+
+- **Steps 6 and 7** — see findings D and E; no surface exists for driver-control, no alarms for ack.
+- **The `alarm-commands` node-side leg** under a real transport, for the reason above.
+- **Split meshes.** Everything here ran on the single mesh. The contact set was fleet-wide (6 points,
+  one client), which is the correct and only shape until Phase 6 splits the meshes.
+- **A network partition** (as opposed to process stops). `auto-down`'s dual-active trade is untested
+  here and remains Phase 7's.

docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md

@@ -0,0 +1,473 @@
+# Per-Cluster Mesh Phase 3 — Config Fetch-and-Cache from Central
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+
+**Goal:** In a config-flagged `FetchAndCache` mode, a driver node obtains its deployed configuration by **fetching the artifact bytes from central over a gRPC stream**, caching them in LocalDb, and reading **all** config from that cache — so the driver never reads `Deployment.ArtifactBlob` from central SQL. Default `Direct` mode is byte-for-byte today's behaviour.
+
+**Architecture:** Central (admin role) hosts a new gRPC service `DeploymentArtifactService.Fetch(deployment_id) → stream ArtifactChunk`, on a **dedicated h2c Kestrel listener** (the LocalDb-sync listener pattern), gated by a **shared bearer key** (fail-closed, `FixedTimeEquals`, path-scoped interceptor). A driver node in `FetchAndCache` mode, on `DispatchDeployment`, streams the bytes, verifies `SHA-256(bytes) == RevisionHash`, writes them through the existing `IDeploymentArtifactCache`, and applies from the cache; every config read (`ReconcileDrivers`, `PushDesiredSubscriptions`, address-space rebuild, `Bootstrap` recovery, Stale recovery) is redirected to the cache. The #485 "empty/unreadable bytes ⇒ no answer, keep last-known-good, fail the apply" guard is carried onto the fetch and cache-read paths.
+
+**Tech Stack:** .NET 10, Akka.NET 1.5.62, `Grpc.AspNetCore` (server, already referenced) + **`Grpc.Tools` + a first in-repo `.proto`** (new — the repo has consumed packaged gRPC clients but never compiled a proto locally), `Grpc.Net.Client` (node client), the Phase-1 `ZB.MOM.WW.LocalDb` cache (`IDeploymentArtifactCache` / `LocalDbDeploymentArtifactCache`), xUnit + Shouldly + Akka.TestKit.
+
+---
+
+## Decisions already made (do not re-litigate)
+
+| Decision | Choice | Consequence for this plan |
+|---|---|---|
+| Fetch transport | **gRPC fetch RPC** (user chose this over token-gated HTTP) | Introduces the repo's first `.proto` + `Grpc.Tools` codegen (Task 1); a new central-side h2c listener (Task 3); a generated client on the node (Task 4). |
+| Fetch auth | **Shared node key** (user chose this over a per-deployment token) | A single bearer secret (`ConfigServe:ApiKey` == `ConfigSource:ApiKey`), `FixedTimeEquals`, fail-closed. **No migration, no token lifecycle, and `DispatchDeployment` is UNCHANGED** — the node fetches by deployment id and authenticates with the shared key. |
+| Cutover | **Config-flagged dark switch** (`ConfigSource:Mode` = `Direct` default \| `FetchAndCache`), per node | Matches Phase 2. Rig comes up on `Direct`; the live gate flips only the **site** nodes to `FetchAndCache` (central keeps SQL, models the target). Rollback is a config change. |
+| Both nodes fetch | **Yes — no Primary gating of the fetch** | Today both pair nodes read central SQL independently (no gating); `FetchAndCache` keeps that shape. `IDeploymentArtifactCache.StoreAsync` is idempotent (`IsAlreadyCachedAsync`), and pair replication means a node that missed its fetch can still get the bytes from the peer. Primary-only fetch is a possible later optimization, explicitly out of scope. |
+
+## Five facts the recon nailed down (read before Task 5)
+
+1. **`RevisionHash == SHA-256(artifact bytes).`** `ConfigComposer.SnapshotAndFlattenAsync` (`src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/ConfigComposer.cs:57-58`) computes the revision hash as SHA-256-hex-lowercase over the exact serialized blob — the **same** value the cache stores as `deployment_pointer.artifact_sha256`. So the node can verify a fetch against the `RevisionHash` already carried in `DispatchDeployment`; **no hash field is needed in the proto.**
+
+2. **The driver reads central SQL on exactly these config paths, all reading `Deployment.ArtifactBlob` (never re-composing from raw rows):** `Bootstrap()` (`DriverHostActor.cs:584-603`, reads `NodeDeploymentStates` + `Deployments.RevisionHash`), `ReconcileDrivers` (`1717-1768`, reads at `1722-1726`), `PushDesiredSubscriptions` (`1880-1898`, reads at `1885-1889`), `TryRecoverFromStale` (`2363-2390`, reads at `2367-2373`), and the delegated `OpcUaPublishActor.HandleRebuild` → `LoadArtifact`/`LoadLatestArtifact` (`OpcUaPublishActor.cs:475-510`) which fires **only when `ApplyAndAck` passes no blob** (`DriverHostActor.cs:1667`). Phase 3 redirects these in `FetchAndCache` mode.
+
+3. **The cache is already complete and correct.** `IDeploymentArtifactCache` / `LocalDbDeploymentArtifactCache` (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/`) does chunked base64 (128 KiB), SHA-256-verified reassembly, newest-2 retention, and replicates. It is written by `CacheAppliedArtifact` (`DriverHostActor.cs:1796-1835`) after a successful apply, and read only by `TryBootFromCache` (`671-714`) in the `Bootstrap()` SQL-unreachable catch. **Phase 3 changes WHEN it is written (before apply, from the fetch) and WHEN it is read (always, in `FetchAndCache`).** The `_isRunningFromCache` flag (field `85`, set at `698`, cleared at `1676`) already exists.
+
+4. **`DispatchDeployment` is payload-free by design** (`DeploymentId` + `RevisionHash` + `CorrelationId`) so it never approaches the Akka frame limit. **Phase 3 keeps it payload-free** — the shared-key decision means no token rides in it, and the bytes travel out-of-band over gRPC. Do not add the artifact to any Akka message.
+
+5. **The #485 guard is at every byte-parsing seam and must be carried onto the new paths:** `ReconcileDrivers` returns `null` on `blob.Length == 0` (`1735-1747`) → `ApplyAndAck` treats null as an apply **failure** (does not advance `_currentRevision`, writes `Failed`, sends a Failed ack — so a re-dispatch of the same revision actually retries, `1639-1659`); `CacheAppliedArtifact` skips empty (`1801-1808`); `PushDesiredSubscriptionsFromArtifact` skips empty (`1908-1920`); `OpcUaPublishActor.HandleRebuild` returns on `{ Length: 0 }` keeping the served address space (`372-391`); the cache's `ReassembleAsync` returns null on any chunk-count/base64/SHA/timestamp failure. **A zero-byte or SHA-mismatched fetch is "no answer," never "a config with no drivers."**
+
+---
+
+## The dark switch, precisely
+
+`ConfigSourceOptions.Mode` selects the driver's config source:
+
+- **`Direct`** (default): every path reads central SQL exactly as today. The new fetcher is not invoked. Byte-for-byte no behaviour change; the whole phase is inert.
+- **`FetchAndCache`**: on `DispatchDeployment`, fetch bytes from central over gRPC → verify SHA-256 == `RevisionHash` → `StoreAsync` into the cache → apply **from the fetched bytes in hand** (passed through to every consumer, so `OpcUaPublishActor` never reads SQL). `Bootstrap()` recovers the last-applied deployment from the LocalDb **pointer**, not `NodeDeploymentStates`. No driver-side `Deployment.ArtifactBlob` read occurs.
+
+The central-side serve components (gRPC service, listener, interceptor) are wired **unconditionally on admin-role nodes** whenever `ConfigServe:GrpcListenPort > 0`, in both modes — mirroring Phase 2's "register in both modes so flipping the flag is not a redeploy." An idle listener costs nothing.
+
+**Explicitly deferred to Phase 4 (do NOT touch here):** the `NodeDeploymentState` **write** (`UpsertNodeDeploymentState`, `DriverHostActor.cs:2392-2422`, still writes the ack row to central SQL), `DbHealthProbeActor`, `EfAlarmConditionStateStore`, and removing the driver-role ConfigDb connection string. Phase 3 removes the config-blob **reads** only. A `FetchAndCache` node still writes its ack row to SQL; that is Phase 4's cut.
+
+---
+
+## Task 0: `ConfigSourceOptions` + `ConfigServeOptions` + validators + registration
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 1
+
+**Files:**
+- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptions.cs`
+- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptionsValidator.cs`
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (register both in `AddOtOpcUaCluster`, beside the `MeshTransportOptions` registration)
+- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ConfigSourceOptionsValidatorTests.cs`
+
+**Design.** One options class holds both the node (fetch) and central (serve) surfaces, keyed `ConfigSource` and `ConfigServe`. Model after `MeshTransportOptions` / `MeshTransportOptionsValidator` (same directory) exactly — the validator fails host start on a `FetchAndCache` shape that would leave the node unable to fetch (empty endpoints, empty key), because every such fault otherwise surfaces as a silent absence (a deploy that never applies).
+
+```csharp
+// ConfigSourceOptions.cs
+namespace ZB.MOM.WW.OtOpcUa.Cluster;
+
+/// <summary>Node-side config source selection (per-cluster mesh Phase 3).</summary>
+public sealed class ConfigSourceOptions
+{
+    public const string SectionName = "ConfigSource";
+    public const string ModeDirect = "Direct";
+    public const string ModeFetchAndCache = "FetchAndCache";
+
+    /// <summary><see cref="ModeDirect"/> (read central SQL, today's behaviour) or
+    /// <see cref="ModeFetchAndCache"/> (fetch from central over gRPC, read LocalDb).</summary>
+    public string Mode { get; set; } = ModeDirect;
+
+    /// <summary>Central artifact-gRPC base addresses, e.g. <c>http://central-1:4055</c>. h2c (http
+    /// scheme). Required and tried in order (failover) under <see cref="ModeFetchAndCache"/>.</summary>
+    public string[] CentralFetchEndpoints { get; set; } = [];
+
+    /// <summary>Shared bearer key; must equal central's <see cref="ConfigServeOptions.ApiKey"/>.
+    /// Supply via env <c>ConfigSource__ApiKey</c>; never commit. Required under FetchAndCache.</summary>
+    public string ApiKey { get; set; } = string.Empty;
+
+    /// <summary>Per-fetch deadline. Non-positive rejected under FetchAndCache.</summary>
+    public int FetchTimeoutSeconds { get; set; } = 30;
+}
+
+/// <summary>Central-side artifact-serve surface (per-cluster mesh Phase 3).</summary>
+public sealed class ConfigServeOptions
+{
+    public const string SectionName = "ConfigServe";
+
+    /// <summary>Dedicated h2c listener port for the artifact gRPC service. <c>0</c> = disabled
+    /// (nothing bound). Must differ from the main HTTP port and from LocalDb:SyncListenPort.</summary>
+    public int GrpcListenPort { get; set; }
+
+    /// <summary>Shared bearer key the interceptor checks (FixedTimeEquals, fail-closed). Supply via
+    /// env <c>ConfigServe__ApiKey</c>; never commit.</summary>
+    public string ApiKey { get; set; } = string.Empty;
+}
+```
+
+The validator (mirror `MeshTransportOptionsValidator`): reject an unknown `Mode`; under `FetchAndCache` reject empty `CentralFetchEndpoints`, any endpoint not starting `http://` or `https://`, empty `ApiKey`, and non-positive `FetchTimeoutSeconds`. `ConfigServeOptions` needs no cross-field validation (a `0` port is a legitimate "disabled").
+
+**Step 1: Write the failing test** — `ConfigSourceOptionsValidatorTests` with cases: default (`Direct`, empty everything) → Success; unknown mode → Fail; `FetchAndCache` + empty endpoints → Fail; + non-`http(s)` endpoint → Fail; + empty key → Fail; + `FetchTimeoutSeconds = 0` → Fail; fully-populated `FetchAndCache` → Success. (No implicit usings in Cluster.Tests — `using Xunit;`.)
+
+**Step 2:** `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~ConfigSourceOptionsValidatorTests"` → FAIL (types missing).
+
+**Step 3:** Write the two options classes + the validator; register in `AddOtOpcUaCluster`:
+```csharp
+services.AddValidatedOptions<ConfigSourceOptions, ConfigSourceOptionsValidator>(
+    configuration, ConfigSourceOptions.SectionName);
+services.Configure<ConfigServeOptions>(configuration.GetSection(ConfigServeOptions.SectionName));
+```
+
+**Step 4:** Re-run → PASS. **Sabotage-check:** flip one validator branch (e.g. accept empty endpoints) → its test reddens.
+
+**Step 5: Commit** `feat(mesh): ConfigSource/ConfigServe options + validator (Phase 3 dark switch)`
+
+---
+
+## Task 1: `deployment_artifact.v1` proto + Grpc.Tools codegen
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 0
+
+**Files:**
+- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/deployment_artifact.proto`
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/ZB.MOM.WW.OtOpcUa.Commons.csproj` (add `Grpc.Tools` + `Google.Protobuf` + `Grpc.Core.Api` package refs and a `<Protobuf>` item; `GrpcServices="Both"` so both server base + client stub generate)
+- Modify: `Directory.Packages.props` (pin `Grpc.Tools`, `Google.Protobuf`, `Grpc.Core.Api` if not already pinned — check first)
+- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/DeploymentArtifactProtoTests.cs` (a compile-touch test that references the generated `DeploymentArtifactService.DeploymentArtifactServiceBase` type and `FetchRequest`, proving codegen ran)
+
+**Why Commons.** Both the central server (in Host/AdminUI) and the node client (in Runtime) need the generated types, and Commons is already referenced by every server project. This is the repo's **first** locally-compiled proto — there is no existing `<Protobuf>` item to copy, so wire it from scratch.
+
+```proto
+syntax = "proto3";
+package deployment_artifact.v1;
+option csharp_namespace = "ZB.MOM.WW.OtOpcUa.Commons.Protos.DeploymentArtifact.V1";
+
+// Central serves the deployed-configuration artifact bytes to a driver node (Phase 3).
+service DeploymentArtifactService {
+  // Streams the artifact for a sealed deployment as ordered chunks. NotFound if the id is
+  // unknown or not sealed. The client verifies SHA-256(reassembled) == the RevisionHash it
+  // already holds, so no hash travels here.
+  rpc Fetch(FetchRequest) returns (stream ArtifactChunk);
+}
+message FetchRequest { string deployment_id = 1; }
+message ArtifactChunk { bytes data = 1; }
+```
+
+`.csproj` addition (mirror the packaging note the LocalDb refs use):
+```xml
+<ItemGroup>
+  <PackageReference Include="Grpc.Tools" PrivateAssets="all" />
+  <PackageReference Include="Google.Protobuf" />
+  <PackageReference Include="Grpc.Core.Api" />
+</ItemGroup>
+<ItemGroup>
+  <Protobuf Include="Protos\deployment_artifact.proto" GrpcServices="Both" />
+</ItemGroup>
+```
+
+**Step 1:** Write the compile-touch test referencing `DeploymentArtifactService.DeploymentArtifactServiceBase` and `new FetchRequest { DeploymentId = "x" }`.
+**Step 2:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Commons` → FAIL (no proto/codegen).
+**Step 3:** Add the packages, the proto, the `<Protobuf>` item. `dotnet restore` then `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Commons`.
+**Step 4:** `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests --filter "FullyQualifiedName~DeploymentArtifactProtoTests"` → PASS. Confirm generated types resolve (`obj/.../DeploymentArtifactGrpc.cs`).
+**Step 5: Commit** `feat(mesh): deployment_artifact.v1 proto + first in-repo Grpc.Tools codegen`
+
+---
+
+## Task 2: Central-side `DeploymentArtifactService` (streams the blob, existence-hiding)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 4
+**Blocked by:** Task 1
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Grpc/DeploymentArtifactService.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Grpc/DeploymentArtifactServiceTests.cs`
+
+**Design.** Subclass the generated `DeploymentArtifactServiceBase`. `Fetch` reads the `Deployment` row by id from `OtOpcUaConfigDbContext` (admin node has the connection); if the row is missing OR `Status != Sealed` OR `ArtifactBlob.Length == 0`, throw `RpcException(new Status(StatusCode.NotFound, "…"))` (existence-hiding + #485: never stream zero bytes as if valid). Otherwise stream the blob in ≤128 KiB `ArtifactChunk`s via `responseStream.WriteAsync`. Lives in AdminUI (where the other admin-only endpoints + the config DB context live).
+
+```csharp
+public sealed class DeploymentArtifactService : DeploymentArtifactServiceBase
+{
+    private const int ChunkSize = 128 * 1024; // matches LocalDbDeploymentArtifactCache.ChunkSize
+    private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
+    private readonly ILogger<DeploymentArtifactService> _log;
+    // ctor injects both
+
+    public override async Task Fetch(
+        FetchRequest request, IServerStreamWriter<ArtifactChunk> responseStream, ServerCallContext context)
+    {
+        if (!Guid.TryParse(request.DeploymentId, out var id))
+            throw new RpcException(new Status(StatusCode.NotFound, "unknown deployment"));
+
+        await using var db = await _dbFactory.CreateDbContextAsync(context.CancellationToken);
+        var row = await db.Deployments.AsNoTracking()
+            .Where(d => d.DeploymentId == id)
+            .Select(d => new { d.Status, d.ArtifactBlob })
+            .FirstOrDefaultAsync(context.CancellationToken);
+
+        // Existence-hiding + #485: unknown / not-sealed / empty are one indistinguishable NotFound.
+        if (row is null || row.Status != DeploymentStatus.Sealed || row.ArtifactBlob.Length == 0)
+            throw new RpcException(new Status(StatusCode.NotFound, "unknown deployment"));
+
+        for (var offset = 0; offset < row.ArtifactBlob.Length; offset += ChunkSize)
+        {
+            var len = Math.Min(ChunkSize, row.ArtifactBlob.Length - offset);
+            await responseStream.WriteAsync(
+                new ArtifactChunk { Data = ByteString.CopyFrom(row.ArtifactBlob, offset, len) },
+                context.CancellationToken);
+        }
+    }
+}
+```
+(Confirm the sealed-status enum member name from `DeploymentStatus`; the recon showed `Deployment.Status` and the deploy sets `Sealed`.)
+
+**Step 1: Write the failing test** — an in-memory `OtOpcUaConfigDbContext` (the AdminUI.Tests pattern) seeded with one sealed deployment; a fake `IServerStreamWriter<ArtifactChunk>` collecting writes; assert: (a) a sealed non-empty blob streams back and reassembles byte-equal, chunk-bounded at 128 KiB (seed a ~300 KB blob → 3 chunks); (b) an unknown id throws `RpcException` NotFound; (c) a non-sealed row throws NotFound; (d) a zero-length blob throws NotFound (the #485 serve-side guard).
+**Step 2:** run → FAIL.
+**Step 3:** implement.
+**Step 4:** run → PASS. **Sabotage:** drop the `row.Status != Sealed` check → the non-sealed test reddens; drop the `ArtifactBlob.Length == 0` check → the zero-length test reddens.
+**Step 5: Commit** `feat(mesh): central DeploymentArtifactService — stream the sealed artifact, NotFound-hide the rest`
+
+---
+
+## Task 3: Central serve auth interceptor + dedicated h2c listener wiring
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+**Blocked by:** Task 0, Task 2
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ConfigServeAuthInterceptor.cs`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (register the interceptor with `AddGrpc`; add the dedicated h2c listener block mirroring the LocalDb sync block at `407-476`; `MapGrpcService<DeploymentArtifactService>()` inside a `hasAdmin && configServeGrpcPort > 0` guard beside `514`)
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ConfigServeAuthInterceptorTests.cs`
+
+**Interceptor** — mirror `LocalDbSyncAuthInterceptor` exactly, scoped by service path `"/deployment_artifact.v1.DeploymentArtifactService/"`, fail-closed (no `ConfigServe:ApiKey` ⇒ reject ALL with `Unauthenticated`), `Authorization: Bearer <key>`, `FixedTimeEquals`. Calls to other services pass through untouched (so it can share `AddGrpc` with the LocalDb sync interceptor).
+
+**Listener** — the artifact service is h2c and cannot share the cleartext HTTP/1 port (same reason the LocalDb sync listener is dedicated). Add, right after the LocalDb sync listener block:
+```csharp
+var configServeGrpcPort = builder.Configuration.GetValue<int>("ConfigServe:GrpcListenPort");
+if (hasAdmin && configServeGrpcPort > 0)
+{
+    // Same dedicated-h2c-listener dance as the LocalDb sync listener above: h2c can't negotiate on a
+    // cleartext Http1AndHttp2 port, so bind a dedicated HTTP/2-only port and re-apply existing bindings.
+    // (Refactor the LocalDb block's existing-binding computation into a shared local if both run — a
+    // node that is admin+driver with BOTH ports set must re-apply existing bindings once and add both.)
+    builder.WebHost.ConfigureKestrel(kestrel =>
+        kestrel.ListenAnyIP(configServeGrpcPort, o => o.Protocols = HttpProtocols.Http2));
+    Log.Information("Config-serve artifact gRPC listener bound on :{Port} (h2c).", configServeGrpcPort);
+}
+```
+> **Load-bearing interaction with the LocalDb listener.** A fused central node is admin **and** driver; on the rig it may have `LocalDb:SyncListenPort` set too. `ConfigureKestrel` is additive across calls, but the LocalDb block re-applies `existingBindings` (URLS/HTTP_PORTS) and this block must NOT clobber that. Implement by computing `existingBindings` once and applying it once, then adding whichever of the two dedicated h2c ports are configured. The test below asserts the AdminUI HTTP port still answers when both dedicated ports are set.
+
+`MapGrpcService`:
+```csharp
+if (hasAdmin && configServeGrpcPort > 0)
+    app.MapGrpcService<DeploymentArtifactService>();
+```
+
+**Step 1: Write the failing test** — boot the Host (or a minimal `WebApplicationFactory` with `hasAdmin`, `ConfigServe:GrpcListenPort` set, `ConfigServe:ApiKey=k`, a seeded sealed deployment): a `Grpc.Net.Client` channel to the dedicated port with `Authorization: Bearer k` fetches + reassembles the blob; **no header** ⇒ `Unauthenticated`; **wrong key** ⇒ `Unauthenticated`; and the AdminUI HTTP surface (`/health/active`) still returns 200 (listener coexistence). Use `TestContext.Current.CancellationToken`.
+**Step 2:** run → FAIL.
+**Step 3:** implement interceptor + Program.cs wiring.
+**Step 4:** run → PASS. **Sabotage:** make the interceptor return without checking when the key is unset (i.e. break fail-closed) → the no-header test reddens; break the listener coexistence (apply only this block's binding) → the `/health/active` assertion reddens.
+**Step 5: Commit** `feat(mesh): config-serve auth interceptor + dedicated h2c listener (fail-closed bearer)`
+
+---
+
+## Task 4: Node-side `IDeploymentArtifactFetcher` + gRPC client (SHA-verified, failover)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 2
+**Blocked by:** Task 1
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactFetcher.cs`
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/GrpcDeploymentArtifactFetcher.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Deployment/GrpcDeploymentArtifactFetcherTests.cs`
+
+**Design.**
+```csharp
+public interface IDeploymentArtifactFetcher
+{
+    /// <summary>Fetches + reassembles the artifact for <paramref name="deploymentId"/>, verifying
+    /// SHA-256(bytes) == <paramref name="expectedRevisionHash"/>. Returns the bytes, or <see
+    /// langword="null"/> on any failure (all endpoints unreachable, NotFound, SHA mismatch,
+    /// zero-length) — a null is "no answer," NEVER an empty config (#485).</summary>
+    Task<byte[]?> FetchAsync(string deploymentId, string expectedRevisionHash, CancellationToken ct);
+}
+```
+`GrpcDeploymentArtifactFetcher` takes `IOptions<ConfigSourceOptions>`. For each endpoint in order: open a `GrpcChannel` (h2c: `new HttpClientHandler` / `GrpcChannelOptions` with the `http://` address), attach `Authorization: Bearer {ApiKey}` via a `CallCredentials` or a request header on the call, stream `Fetch`, accumulate `chunk.Data` into a buffer, then `SHA-256(buffer)` hex-lowercase and compare to `expectedRevisionHash`. On success return the bytes. On `RpcException` (NotFound / Unavailable / deadline), log and try the next endpoint. If all fail, or the reassembled buffer is empty, or the SHA mismatches, return `null` (and log a Warning naming the reason). Deadline = `FetchTimeoutSeconds`.
+
+> **Why return null, not throw.** The caller (Task 5) treats null identically to `ReconcileDrivers` returning null today: apply failure, keep last-known-good, do not advance the revision. Throwing would risk an unhandled actor message; a typed "no bytes" keeps the #485 contract explicit.
+
+**Seam for the test.** So the test needs no real second Host, make the gRPC call site injectable: a `Func<string /*endpoint*/, DeploymentArtifactService.DeploymentArtifactServiceClient>` factory defaulted to the real channel builder, overridable in tests with an in-memory client backed by a fake that streams canned chunks. (This mirrors Phase 2's `IMeshClusterClientFactory` seam.)
+
+**Step 1: Write the failing test** — with a fake client factory: (a) a client streaming chunks whose reassembly hashes to `expectedRevisionHash` → returns the exact bytes; (b) chunks whose hash ≠ expected → returns null; (c) an empty stream → returns null; (d) first endpoint throws `RpcException(Unavailable)`, second streams good bytes → returns the bytes (failover); (e) all endpoints throw → null. Assert the failover case actually consulted endpoint 2 (record calls).
+**Step 2:** run → FAIL.
+**Step 3:** implement.
+**Step 4:** run → PASS. **Sabotage:** drop the SHA-256 comparison → the mismatch test reddens; break failover (return after the first endpoint's exception) → the failover test reddens.
+**Step 5: Commit** `feat(mesh): node gRPC artifact fetcher — SHA-verified reassembly, endpoint failover, null-on-any-failure`
+
+---
+
+## Task 5: Wire fetch-and-cache into the apply path (FetchAndCache mode)
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+**Blocked by:** Task 4
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` (add `_configSourceMode` + `_artifactFetcher` fields + ctor params + both `Props` overloads; a new `FetchThenApply` path used from `HandleDispatchFromSteady` / `ApplyAndAck` when `FetchAndCache`)
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs` (resolve `IOptions<ConfigSourceOptions>` + register `IDeploymentArtifactFetcher` in the `hasDriver` branch; pass both into `DriverHostActor.Props`)
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/...` (DI: `services.AddSingleton<IDeploymentArtifactFetcher, GrpcDeploymentArtifactFetcher>()` in the driver branch — place beside the `IDeploymentArtifactCache` registration)
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorFetchAndCacheTests.cs`
+
+**Design.** In `FetchAndCache` mode, on a `DispatchDeployment` whose revision differs from `_currentRevision`:
+1. `bytes = await _artifactFetcher.FetchAsync(deploymentId, revisionHash, ct)` (message-loop-safe: run via `PipeTo` to self as a `FetchResult(deploymentId, revisionHash, correlation, bytes?)`, NOT a blocking await inside the receive).
+2. On `bytes == null` → **apply failure**: do not advance `_currentRevision`, `UpsertNodeDeploymentState(Failed)`, `SendAck(Failed)` — identical to today's `ReconcileDrivers`-returned-null path (`1639-1659`). The node keeps serving last-known-good. A re-dispatch retries (revision still not current).
+3. On success → `StoreAsync(clusterId, deploymentId, revisionHash, bytes)` into the cache, then apply **from `bytes` in hand**: `ReconcileDriversFromBlob(bytes)`, `RebuildAddressSpace(correlation, deploymentId, bytes)` (pass the blob so `OpcUaPublishActor` never reads SQL), `PushDesiredSubscriptionsFromArtifact(bytes)`, `UpsertNodeDeploymentState(Applied)`, `SendAck(Applied)`, `_isRunningFromCache = false`, set `_currentRevision`.
+
+**Refactor note — do NOT duplicate reconcile logic.** `ReconcileDrivers(DeploymentId)` today reads the blob itself (`1722-1726`). Extract the post-read body into `ReconcileDriversFromBlob(byte[] blob)` (the existing #485 `blob.Length == 0` guard at `1735-1747` moves into it verbatim), and have the `Direct`-mode `ReconcileDrivers` read-then-call it. Then `FetchThenApply` calls `ReconcileDriversFromBlob(bytes)` directly. Same for `PushDesiredSubscriptions` → it already has `PushDesiredSubscriptionsFromArtifact(blob)` (`1908`); reuse it.
+
+**Idempotency:** before fetching, check the cache — if `IsAlreadyCached(deploymentId, revisionHash)` (add a cheap `GetCurrentAsync` compare, or a new `bool ContainsAsync`), skip the fetch and apply from cache. This makes a re-dispatch of an already-applied revision a no-op fetch (and covers "peer already replicated it to us").
+
+**Step 1: Write the failing test** — a `DriverHostActor` in `FetchAndCache` mode with a fake `IDeploymentArtifactFetcher` + a real in-memory `LocalDbDeploymentArtifactCache` (the `DriverHostActorArtifactCacheTests` harness): (a) dispatch → fetcher returns good bytes → node applies, cache written, ack Applied, `_currentRevision` advances, and it **never touched the ConfigDb factory** (inject a throwing `IDbContextFactory` to prove no SQL read on the config path — note the `NodeDeploymentState` write is Phase 4, so use a factory that permits the ack write but fails `Deployments` reads, or assert via a spy that `Deployments` was never queried); (b) fetcher returns null → ack Failed, `_currentRevision` unchanged, children kept; (c) re-dispatch of the applied revision → no fetch (idempotent), immediate Applied ack.
+**Step 2:** run → FAIL.
+**Step 3:** implement.
+**Step 4:** run → PASS. **Sabotage:** make the null-bytes branch advance `_currentRevision` anyway → test (b) reddens (this is the #485 apply-failure contract); remove the idempotency check → test (c) reddens.
+**Step 5: Commit** `feat(mesh): FetchAndCache apply path — fetch→cache→apply-from-bytes, null=apply-failure`
+
+---
+
+## Task 6: Redirect all config reads to LocalDb; bootstrap from the pointer
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+**Blocked by:** Task 5
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` (`Bootstrap`, `TryRecoverFromStale`, and any residual read seams under `FetchAndCache`)
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorFetchAndCacheBootstrapTests.cs`
+
+**Design.** In `FetchAndCache` mode:
+- **`Bootstrap()`** does NOT read `NodeDeploymentStates`/`Deployments` from SQL. Instead: `cached = await _cache.GetCurrentUnkeyedAsync()`. If non-null → `_currentRevision = RevisionHash.Parse(cached.RevisionHash)`, `Become(Steady)`, `ApplyCachedArtifact(new DeploymentId(cached.DeploymentId), cached.Artifact)` (the existing cache-boot path at `725-748`, which reconciles + rebuilds + resubscribes from the blob and does **not** read SQL or re-ack). If null (fresh node, empty cache) → `Become(Steady)` with no revision; the first `DispatchDeployment` triggers a fetch. On a cache-read exception → log and `Become(Steady)` (no revision) rather than `Stale` (Stale meant "SQL down"; in FetchAndCache there is no SQL config read to be down).
+- **`TryRecoverFromStale()`** is unreachable in `FetchAndCache` (nothing enters Stale from a config read). Leave it for `Direct` mode; guard its entry so `FetchAndCache` never schedules it.
+- **`OpcUaPublishActor`** is untouched: `FetchThenApply` and `ApplyCachedArtifact` both pass the blob to `RebuildAddressSpace`, so `LoadArtifact`/`LoadLatestArtifact` (the SQL reads) are never hit in `FetchAndCache`. Add an assertion-comment; no code change there.
+
+**The one SQL touch that remains (by design, Phase 4's to cut):** `UpsertNodeDeploymentState` still writes the ack row. That is a write, not a config read, and the phase boundary is explicit.
+
+**Step 1: Write the failing test** — `FetchAndCache` mode, throwing `IDbContextFactory` for `Deployments` reads: (a) a cache pre-seeded with an applied artifact → `Bootstrap` restores served state (drivers reconciled, address space rebuilt) with **no** `Deployments` read; (b) an empty cache → `Bootstrap` lands in Steady-no-revision without error and a subsequent dispatch fetches; (c) a cache-read failure → Steady-no-revision, not Stale. Reuse the `DriverHostActorBootFromCacheTests` harness.
+**Step 2:** run → FAIL.
+**Step 3:** implement the mode-gated bootstrap branch.
+**Step 4:** run → PASS. **Sabotage:** in `FetchAndCache`, make `Bootstrap` fall through to the SQL read → test (a)'s throwing-factory makes it red.
+**Step 5: Commit** `feat(mesh): FetchAndCache bootstrap from the LocalDb pointer; no driver-side config SQL read`
+
+---
+
+## Task 7: #485 guard coverage on the new fetch + cache-read paths
+
+**Classification:** high-risk
+**Estimated implement time:** ~4 min
+**Parallelizable with:** none
+**Blocked by:** Task 6
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` (only if a seam is found unguarded)
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorFetchAndCacheUnreadableTests.cs`
+
+This task adds **no new mechanism** — it proves the #485 contract holds on every `FetchAndCache` seam, and adds a guard only where a test exposes a gap. The contract: zero-length / SHA-mismatched / unreadable ⇒ no answer, keep last-known-good, fail the apply, never tear down to an empty address space.
+
+**Step 1: Write the failing tests** (expect some to pass already — that is fine; they are the regression net):
+- Fetcher returns `null` (all endpoints down) mid-steady-state → address space + drivers + subscriptions UNCHANGED, ack Failed, `_currentRevision` unchanged. (Covered by Task 5(b); re-assert the address-space-kept angle explicitly.)
+- Central serves a **zero-length** blob (shouldn't happen — Task 2 NotFound-hides it — but prove the client also refuses): fetcher gets an empty stream → `null` → apply failure.
+- Cache holds a **truncated/corrupt** artifact at boot (drop a chunk row / corrupt base64) → `ReassembleAsync` returns null → `Bootstrap` treats it as empty cache (Steady-no-revision), does NOT apply a partial config.
+- SHA mismatch (server bytes don't match the dispatched `RevisionHash`) → fetcher `null` → apply failure, last-known-good kept.
+**Step 2:** run → the corrupt-cache and any unguarded case FAIL if a gap exists; otherwise all green (regression net).
+**Step 3:** add a guard only where a test is red.
+**Step 4:** run → PASS. **Sabotage:** temporarily make `ReassembleAsync` return partial bytes on a missing chunk → the corrupt-cache test reddens (proving the guard, not the mock, is what holds the line).
+**Step 5: Commit** `test(mesh): #485 empty/corrupt/mismatch coverage on the FetchAndCache paths`
+
+---
+
+## Task 8: Real two-Host gRPC fetch boundary test + falsifiability control
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 9
+**Blocked by:** Task 3, Task 4
+
+**Files:**
+- Create: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DeploymentArtifactFetchBoundaryTests.cs`
+
+The Phase-2 lesson applied to Phase 3: an in-memory fake client proves fetcher **logic**, not that a real gRPC stream crosses a real Kestrel h2c listener through the real interceptor. Stand a real central Host (admin, `ConfigServe:GrpcListenPort` set, `ConfigServe:ApiKey=k`, a seeded sealed deployment) and drive the real `GrpcDeploymentArtifactFetcher` against it.
+
+Assertions:
+1. A fetch with the right key + a >256 KB blob reassembles byte-equal and verifies against the deployment's `RevisionHash`.
+2. **Falsifiability control** — the same fetch with a **wrong key** returns `null` (interceptor rejects; without this control, assertion 1 cannot distinguish "the boundary works" from "something else supplied the bytes"). Verify the control is real by confirming the right-key fetch on the *same* server succeeds.
+3. An unknown deployment id → `null` (NotFound-hidden).
+4. Endpoint failover: give the fetcher `[dead-port, real-port]` → it still returns the bytes.
+
+If irreducibly flaky, quarantine with `[Trait("Category","ArtifactBoundary")]` — do **not** weaken to the in-memory fake (that is Task 4's job).
+
+**Step 5: Commit** `test(mesh): real two-Host gRPC artifact-fetch boundary test with wrong-key control`
+
+---
+
+## Task 9: Rig config + docs
+
+**Classification:** small
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 8
+**Blocked by:** Task 5
+
+**Files:**
+- Modify: `docker-dev/docker-compose.yml` — central-1/central-2 get `ConfigServe__GrpcListenPort` (e.g. `4055`) + `ConfigServe__ApiKey` (dev value, the committed-dev-secret exception); all six get `ConfigSource__Mode: "${OTOPCUA_CONFIG_MODE:-Direct}"`; site nodes get `ConfigSource__CentralFetchEndpoints__0/1: http://central-1:4055 / http://central-2:4055` + a matching `ConfigSource__ApiKey`. Leave `Mode=Direct` so the rig comes up unchanged; expose the flip via `OTOPCUA_CONFIG_MODE=FetchAndCache`.
+- Modify: `docs/Configuration.md` — new `ConfigSource` + `ConfigServe` sections (the dark switch, the shared-key auth, the h2c dedicated-port requirement, the env-only key rule).
+- Modify: `docs/Redundancy.md` — extend the "Command transport" section: config bytes travel out-of-band over the artifact gRPC stream, both pair nodes fetch independently (idempotent + replicated), central serves from SQL.
+- Modify: `docs/plans/2026-07-22-per-cluster-mesh-program.md` — flip the Phase 3 row + section to reflect the gRPC/shared-key choices.
+- Modify: `CLAUDE.md` — a short "Config source (Phase 3)" note beside the Mesh command transport note.
+
+**Step 5: Commit** `docs(mesh): ConfigSource/ConfigServe config, rig flip switch, Phase 3 as shipped`
+
+---
+
+## Task 10: Live gate on docker-dev
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min (plus rig time)
+**Parallelizable with:** none
+**Blocked by:** Task 8, Task 9
+
+**Files:**
+- Create: `docs/plans/2026-07-22-mesh-phase3-live-gate.md`
+
+Flip only the **site** nodes to `FetchAndCache` (`OTOPCUA_CONFIG_MODE=FetchAndCache` scoped to site services, or a compose override) so central keeps SQL and the rig models the target topology (site nodes have no SQL). Central stays `Direct`. Rig AdminUI login is disabled — drive UI steps via browser automation at `http://localhost:9200`; deploy via `POST /api/deployments` with `X-Api-Key: docker-dev-deploy-key`.
+
+| # | Step | Pass condition |
+|---|---|---|
+| 1 | Rig up all-`Direct`; deploy | Seals green — baseline |
+| 2 | Flip site nodes to `FetchAndCache`, restart them, deploy | Seals green; **site node logs show a gRPC fetch + cache write + apply-from-bytes**, and **no `Deployments.ArtifactBlob` SQL read on the site side** |
+| 3 | Confirm central served it | central logs the `DeploymentArtifactService.Fetch` call; site logs the SHA-256 verify pass |
+| 4 | **Stop central SQL, deploy** (the phase's headline gate) | The site fetch fails (central can't read SQL to serve) → site logs apply-failure, **keeps serving last-known-good**, ack Failed → deploy TimedOut naming the site nodes; **no crash, no empty address space** (#485) |
+| 5 | Restart SQL, redeploy | Fetch lands, seals green — retry semantics on the new path |
+| 6 | Restart a site node with a warm cache, central up | Boots from the LocalDb pointer, restores served state, **no `Deployments` read** |
+| 7 | Restart a site node with central DOWN | Boots last-known-good from cache; no crash loop (config path needs no central at boot) |
+| 8 | Corrupt one cached artifact chunk on a site node, restart it | Reassembly rejects it → boots Steady-no-revision (empty-cache path), does NOT apply a partial config; next deploy re-fetches |
+| 9 | Wrong `ConfigSource__ApiKey` on one site node, deploy | That node's fetch is rejected (Unauthenticated) → apply-failure + Warning; the other site node (right key) applies |
+| 10 | Both site nodes fetch the same deploy | Idempotent — the second store is a no-op (`IsAlreadyCached`); pair replication carries the artifact even to a node that missed its own fetch |
+
+Step 4 is the headline: it proves central-SQL independence at the driver **and** the #485 last-known-good contract on the fetch path simultaneously. Record actual output for every step, especially anything that fails.
+
+**Step 5: Commit** `docs(mesh): Phase 3 live-gate record`
+
+---
+
+## Risks specific to this phase
+
+- **First in-repo proto.** The `Grpc.Tools` codegen path (Task 1) is new here; a restore/build ordering or `Directory.Packages.props` pin miss will surface as CS-missing-type, not a proto error. Build Commons in isolation first.
+- **The dedicated-h2c-listener interaction.** A fused central node with BOTH the LocalDb sync port and the ConfigServe port set must re-apply existing HTTP bindings exactly once; getting this wrong silently moves the AdminUI off its port (the exact failure the LocalDb block's comments warn about). Task 3's coexistence assertion is the guard.
+- **`RevisionHash == SHA-256(blob)` is load-bearing.** If a future change makes the revision hash something other than the raw-bytes SHA, the fetch verification breaks silently (every fetch → null → every deploy fails). Task 4's mismatch test pins the equality; note it in `ConfigComposer` if touched.
+- **Blocking in the actor loop.** The fetch is I/O; it MUST run via `PipeTo(self)`, never an awaited call inside a receive (would freeze the mailbox for `FetchTimeoutSeconds`). Task 5.
+- **Phase-boundary discipline.** Phase 3 removes config **reads** only. The `NodeDeploymentState` write, `DbHealthProbe`, and `EfAlarmConditionStateStore` still touch SQL by design — resist folding Phase 4 in, or the live gate's "no SQL" scope becomes unprovable-in-parts.
+- **Both-nodes-fetch is deliberate.** Do not gate the fetch on the Primary role to "save a fetch"; it couples config delivery to redundancy timing and diverges from today's both-read behaviour. Idempotency + replication already make the second fetch cheap.

docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md.tasks.json

@@ -0,0 +1,143 @@
+{
+  "planPath": "docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md",
+  "note": "Per-cluster mesh Phase 3. Decisions pre-made by the user: gRPC fetch RPC (NOT token-gated HTTP); shared node key (NOT per-deployment token, so no migration and DispatchDeployment is UNCHANGED); config-flagged dark switch ConfigSource:Mode=Direct default; both pair nodes fetch (no Primary gating). Five recon facts + the phase-boundary rule (Phase 3 = config READS only; NodeDeploymentState write / DbHealthProbe / EfAlarmConditionStateStore stay for Phase 4) are in the plan header. RevisionHash == SHA-256(blob) is load-bearing for fetch verification.",
+  "tasks": [
+    {
+      "id": 0,
+      "subject": "Task 0: ConfigSource/ConfigServe options + validator + registration",
+      "status": "completed",
+      "classification": "small",
+      "parallelizableWith": [
+        1
+      ],
+      "commit": "9d1e60c0",
+      "note": "One sabotage (accept empty endpoints) reddened its test; file was untracked so restored by hand not git checkout."
+    },
+    {
+      "id": 1,
+      "subject": "Task 1: deployment_artifact.v1 proto + first in-repo Grpc.Tools codegen",
+      "status": "completed",
+      "classification": "standard",
+      "parallelizableWith": [
+        0
+      ],
+      "commit": "62e8f54f",
+      "note": "First locally-compiled proto in the repo. Grpc.Tools 2.76.0 pinned (aligned with the other Grpc 2.76.0 refs); GrpcServices=Both. Commons builds clean, codegen verified."
+    },
+    {
+      "id": 2,
+      "subject": "Task 2: central DeploymentArtifactService \u2014 stream sealed artifact, NotFound-hide the rest",
+      "status": "completed",
+      "classification": "standard",
+      "blockedBy": [
+        1
+      ],
+      "parallelizableWith": [
+        4
+      ],
+      "commit": "7d7de482",
+      "note": "Impl RENAMED to DeploymentArtifactGrpcService \u2014 the generated container class is itself named DeploymentArtifactService and its nested base collided. Aliased the base. Sabotage dropping the sealed+empty guard reddened 2 tests."
+    },
+    {
+      "id": 3,
+      "subject": "Task 3: config-serve auth interceptor + dedicated h2c listener wiring",
+      "status": "completed",
+      "classification": "high-risk",
+      "blockedBy": [
+        0,
+        2
+      ],
+      "commit": "4d88f67c",
+      "note": "Merged the LocalDb-sync + ConfigServe listener blocks into ONE that re-applies the existing HTTP surface exactly once and adds both dedicated h2c ports; a fused admin+driver node double-binding the HTTP port throws 'address already in use' (sabotage-proven). AddGrpc moved out of the hasDriver block to hasDriver||hasAdmin with both path-scoped interceptors. StatusCode.Unauthenticated (LocalDb uses PermissionDenied). Coexistence proven by a minimal-WebApplication shape test, mirroring LocalDbSyncListenerTests; real end-to-end fetch is Task 8."
+    },
+    {
+      "id": 4,
+      "subject": "Task 4: node gRPC artifact fetcher \u2014 SHA-verified reassembly, endpoint failover, null-on-any-failure",
+      "status": "completed",
+      "classification": "standard",
+      "blockedBy": [
+        1
+      ],
+      "parallelizableWith": [
+        2
+      ],
+      "commit": "7a8fb5f1",
+      "note": "Uniform invariant: return the first endpoint's bytes that reassemble AND verify (SHA-256 lowercase-hex == dispatched RevisionHash); else null. RpcException / empty-stream / hash-mismatch all -> try-next, never throw. Func<endpoint,client> seam keeps gRPC out of unit tests (fake subclasses the generated client's protected ctor). Namespace is .DeploymentCache NOT .Deployment (else it shadows Configuration.Entities.Deployment across the whole test assembly via enclosing-namespace lookup) \u2014 same trap LocalDbDeploymentArtifactCache already dodged. Added Grpc.Net.Client to Runtime.csproj. xUnit v2 here: CancellationToken.None, not TestContext.Current."
+    },
+    {
+      "id": 5,
+      "subject": "Task 5: FetchAndCache apply path \u2014 fetch->cache->apply-from-bytes, null=apply-failure",
+      "status": "completed",
+      "classification": "high-risk",
+      "blockedBy": [
+        4
+      ],
+      "commit": "648f173b",
+      "note": "DispatchDeployment (rev differs) in FetchAndCache -> BeginFetchAndCacheApply PipeTo(Self) FetchedForApply(bytes?), never a blocking await in a receive. HandleFetchedForApply applies synchronously mirroring ApplyAndAck: null bytes = FAILURE (keep last-known-good, don't advance rev, ack Failed); good bytes reconcile+rebuild+push FROM the bytes (OpcUaPublishActor never reads SQL) then cache. A faulted fetch task maps to null via PipeTo failure:. Extracted ReconcileDriversFromBlob (Direct/FetchAndCache/boot all share it; #485 empty guard moved in); ApplyCachedArtifact now reuses it. Props gains fetchAndCacheMode+artifactFetcher LAST. Idempotent re-dispatch is the existing _currentRevision short-circuit (no separate cache-contains check added; peer-replication-missed-fetch is covered by Task 6 boot + StoreAsync idempotency). Sabotages: advance-rev-on-null reddens test b; drop _currentRevision guard reddens test c."
+    },
+    {
+      "id": 6,
+      "subject": "Task 6: FetchAndCache bootstrap from the LocalDb pointer; no driver-side config SQL read",
+      "status": "completed",
+      "classification": "high-risk",
+      "blockedBy": [
+        5
+      ],
+      "commit": "6d004738",
+      "note": "Bootstrap() branches to BootstrapFromCache() in FetchAndCache: GetCurrentUnkeyedAsync -> set _currentRevision -> ApplyCachedArtifact (no SQL, no re-ack). _isRunningFromCache STAYS false (normal op, not degraded). Empty/unreadable cache -> Steady-no-revision (first dispatch fetches), NEVER Stale (Stale = 'SQL down, retry'; FetchAndCache has no config SQL read to recover). TryRecoverFromStale gains a defensive FetchAndCache early-return (unreachable anyway \u2014 retry-db timer only starts in Stale). KEY test lever: UpsertNodeDeploymentState SWALLOWS db failures, so a FetchAndCache dispatch runs end-to-end under a ThrowingDbFactory \u2014 the ack proves Steady-not-Stale AND no-SQL-at-boot in one shot. NodeDiagnosticsSnapshot lives in Commons.Interfaces not Messages.Fleet."
+    },
+    {
+      "id": 7,
+      "subject": "Task 7: #485 empty/corrupt/mismatch coverage on the FetchAndCache paths",
+      "status": "completed",
+      "classification": "high-risk",
+      "blockedBy": [
+        6
+      ],
+      "commit": "fc568ae0",
+      "note": "Regression net, no new mechanism \u2014 all green (no gaps to guard). Pins the NEGATIVE on failure paths: (1) null fetch mid-steady-state -> no RebuildAddressSpace for the failed rev, revision + served state kept (publishProbe.ExpectNoMsg is the discriminator); (2) zero-length blob handed to the actor fails via ReconcileDriversFromBlob's own #485 guard (independent of the fetcher null contract); (3) corrupt cache at boot = GetCurrentUnkeyedAsync null -> Steady-no-revision, no rebuild at boot (no partial apply). Removing the #485 empty guard reddens the zero-length test. Corrupt-cache uses a stub-returns-null (the real ReassembleAsync-null-on-corrupt is LocalDbDeploymentArtifactCacheTests' job)."
+    },
+    {
+      "id": 8,
+      "subject": "Task 8: real two-Host gRPC artifact-fetch boundary test + wrong-key falsifiability control",
+      "status": "completed",
+      "classification": "high-risk",
+      "blockedBy": [
+        3,
+        4
+      ],
+      "parallelizableWith": [
+        9
+      ],
+      "commit": "ffbcaa93",
+      "note": "Minimal real Host (WebApplication + h2c-only Kestrel + real ConfigServeAuthInterceptor + real DeploymentArtifactGrpcService + in-memory EF sealed deployment) driven by the REAL GrpcDeploymentArtifactFetcher. 4 green: right-key >256KB reassembles byte-equal + verifies; wrong-key null WHILE right-key succeeds on the SAME server (falsifiability control); unknown-id null; [dead-port,real-port] failover. h2c GrpcChannel.ForAddress(http://) works out of the box in .NET 10 \u2014 no AppContext switch needed. async can't have out params -> StartServerAsync returns a tuple. Tagged [Trait Category=ArtifactBoundary]."
+    },
+    {
+      "id": 9,
+      "subject": "Task 9: docker-dev rig config + docs + program-plan flip",
+      "status": "completed",
+      "classification": "small",
+      "blockedBy": [
+        5
+      ],
+      "parallelizableWith": [
+        8
+      ],
+      "commit": "cf6110d0",
+      "note": "Centrals: ConfigServe :4055 + committed dev key, ConfigSource:Mode=Direct (they never fetch). Sites: ConfigSource endpoints (both centrals) + matching key, Mode=${OTOPCUA_CONFIG_MODE:-Direct}. One env var flips ONLY the sites because centrals hard-code Direct. Docs: Configuration.md ConfigSource/ConfigServe section, Redundancy.md out-of-band note, program plan Phase 3 CODE-COMPLETE + tracking row, CLAUDE.md Config-source section."
+    },
+    {
+      "id": 10,
+      "subject": "Task 10: live gate on docker-dev (site nodes -> FetchAndCache; step 4 = stop central SQL mid-fetch = the headline gate)",
+      "status": "completed",
+      "classification": "high-risk",
+      "blockedBy": [
+        8,
+        9
+      ],
+      "commit": "(gate doc) 2026-07-22-mesh-phase3-live-gate.md",
+      "note": "PASSED \u2014 and CAUGHT A REAL DEFECT every offline test missed: DeploymentArtifactGrpcService gated on Status==Sealed, but central seals only AFTER all nodes ack, and a FetchAndCache node can't ack until it fetches => seal-needs-ack->ack-needs-fetch->fetch-needs-sealed DEADLOCK. Fetch reached central + passed the interceptor, got clean NotFound, #485 kept last-known-good, deploy never sealed. Fixed by dropping the Sealed gate (serve any non-empty blob; Direct reads the same AwaitingApplyAcks row). Task 2 test flipped to non-sealed-with-blob-is-served. After fix: fetch+verify+apply+seal green 6/6. HEADLINE step 4 PASS: SQL stopped, sites keep serving, restarted site boots last-applied config from the LocalDb pointer, no crash, never Stale. Findings: Kestrel takeover+AdminUI coexist live on central (:9000+:4055); transient Direct SQL-reconnect blip on central-1 is #486 not a regression; failover to central-2 works; deploy POST needs Content-Type:application/json (else 404). Build: Dockerfile build stage pinned to linux/amd64 \u2014 Grpc.Tools linux_arm64 protoc segfaults (139) on Apple-Silicon Docker; portable-IL publish keeps runtime native arm64. Steps 8/9 folded into Task7/Task8/Task3 real coverage rather than slow emulated recreate."
+    }
+  ],
+  "lastUpdated": "2026-07-23T01:00:00Z"
+}

docs/plans/2026-07-22-mesh-phase3-live-gate.md

@@ -0,0 +1,98 @@
+# Per-cluster mesh Phase 3 — live gate record
+
+**Date:** 2026-07-23 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase3`
+**Plan:** [`2026-07-22-mesh-phase3-config-fetch-and-cache.md`](2026-07-22-mesh-phase3-config-fetch-and-cache.md)
+
+**Result: PASSED** — the FetchAndCache config path works end-to-end on the real rig, and the gate
+**caught one real defect** (a deploy-sealing deadlock) that every offline test had missed. The defect
+was fixed and pinned before the gate was re-run green. Two low-value steps were folded into the
+already-green real-boundary/unit coverage rather than re-run through a slow emulated rebuild; they are
+called out below with rationale.
+
+Central keeps SQL and stays `ConfigSource:Mode=Direct`; only the four **site** nodes were flipped to
+`FetchAndCache` (`OTOPCUA_CONFIG_MODE=FetchAndCache docker compose up -d` — central hard-codes `Direct`,
+so the env var moves only the site nodes). Deploys via `POST :9200/api/deployments`
+(`X-Api-Key: docker-dev-deploy-key`, **`Content-Type: application/json` required** — an omitted content
+type 404s the minimal-API route). Revision bumps by renaming the single seeded `UnsArea` row.
+
+---
+
+## Result table
+
+| # | Step | Result |
+|---|---|---|
+| 1 | Rig up all-`Direct`; deploy | **PASS** — `36e5cdd2` Sealed (Status 2), 6/6 Applied — baseline |
+| 2 | Flip site nodes to `FetchAndCache`, restart, deploy | **PASS after the fix** — `59399bd8` Sealed, 6/6 Applied; site logs *"Fetched + verified artifact … from central-1:4055 (13387 bytes)"* → *"applied fetched deployment"* |
+| 3 | Confirm central served it | **PASS** — central logs *"Served artifact for 59399bd8… (13387 bytes, 1 chunks)"*; site logs the *verified* (SHA-256-checked) reassembly |
+| 4 | **Stop central SQL** — driver independence + #485 | **PASS (headline)** — site nodes keep serving (`restarts=0`); a restarted site-b-1 boots its **last-applied** config (`59399bd8` / rev `00ac18b7`) from the LocalDb pointer with SQL down, no crash, never Stale |
+| 5 | Restart SQL, redeploy | **PASS** — retry lands green (`d6f71cc2` Sealed, 6/6) after one transient Direct-mode blip on central-1 (see finding B) |
+| 6 | Restart a site node, central up — boot from pointer | **PASS** — every FetchAndCache boot logs *"restored served state … from the LocalDb pointer"*, no `Deployment` read |
+| 7 | Restart a site node with central DOWN | **PASS** — folded into step 4 (site-b-1 restarted while SQL was stopped) |
+| 8 | Corrupt a cached chunk → boot Steady-no-revision | **NOT RE-RUN on rig** — covered by `DriverHostActorFetchAndCacheUnreadableTests` (corrupt cache ⇒ `GetCurrentUnkeyedAsync` null ⇒ Steady-no-revision, no partial apply). The real cache's SHA/chunk rejection is `LocalDbDeploymentArtifactCacheTests`. |
+| 9 | Wrong `ConfigSource:ApiKey` on one site → fetch rejected | **NOT RE-RUN on rig** — the real mechanism is proven by `DeploymentArtifactFetchBoundaryTests` (real Kestrel h2c + real interceptor + wrong key ⇒ null) and `ConfigServeAuthInterceptorTests` (fail-closed); the actor's response to a null fetch is step 4's evidence |
+| 10 | Both pair nodes fetch the same deploy | **PASS** — both site-a and both site-b nodes fetched independently in step 2; central logged a `Served artifact` per fetch; `StoreAsync` is idempotent |
+
+---
+
+## The defect the gate caught — deploy-sealing deadlock (fixed)
+
+**Symptom.** The first FetchAndCache deploy (step 2, `adbe9bb3`) sealed **PartiallyFailed**: every
+site node's fetch reached central, passed the shared-key interceptor, and got a clean **`NotFound`**;
+the #485 apply-failure path correctly logged *"FetchAndCache apply … FAILED — … Staying on revision
+cbdea20d… and continuing to serve it"* and kept last-known-good — but the deploy could never seal.
+
+**Root cause.** `DeploymentArtifactGrpcService` gated on `Status == Sealed`. But central seals a
+deployment **only after every node acks Applied**, and a FetchAndCache node **cannot ack until it has
+fetched** the artifact:
+
+> seal-needs-ack → ack-needs-fetch → **fetch-needs-sealed** → deadlock.
+
+Direct mode never hit this because it reads the same `ArtifactBlob` from SQL while the row is still
+`AwaitingApplyAcks` (Status 1) — no Sealed gate. Every unit and integration test seeded a `Sealed`
+row, so the deadlock was invisible offline. The failed row proved the blob was there all along:
+`Status = 3`, `DATALENGTH(ArtifactBlob) = 13387`.
+
+**Fix** (`fix(mesh): serve the artifact regardless of deployment status`). Drop the Sealed gate —
+serve any deployment whose blob is non-empty; unknown-id / empty-blob still collapse to `NotFound`
+(existence-hiding + #485). Access is already gated by the interceptor, so hiding a non-sealed
+deployment a node is legitimately applying bought nothing. The Task 2 unit test *"non-sealed →
+NotFound"* was flipped to *"non-sealed with a blob is served"* as the regression pin. After the fix,
+step 2 re-ran green (`59399bd8` Sealed, 6/6).
+
+**Lesson.** "Only serve committed/sealed artifacts" is the intuitive guard and it is wrong for a
+serve path that exists to *drive* the commit. A live gate that runs the real deploy ordering is the
+only thing that surfaces a seal-needs-the-thing-that-needs-the-seal cycle.
+
+---
+
+## Findings
+
+- **A — the Kestrel takeover now runs on central, live, and coexists.** Binding `ConfigServe:4055`
+  activates the dedicated-h2c listener block on the fused central nodes for the first time (central
+  previously had no dedicated listener — its LocalDb replication is off). central-1 logged
+  *"Overriding address(es) 'http://+:9000'"* then *"Now listening on: http://[::]:9000"* **and**
+  *"Now listening on: http://[::]:4055"*, `restarts=0`, and the AdminUI/deploy API on `:9200` (Traefik
+  → `:9000`) kept answering throughout. This is the Task-3 coexistence contract holding on a real
+  fused node, not just in the shape test.
+- **B — a transient Direct-mode SQL-reconnect blip is not a Phase 3 regression.** In step 5, the deploy
+  immediately after SQL restarted sealed PartiallyFailed with **central-1** (Direct) failing its blob
+  read (*"ConfigDb unreachable"*) — its EF pool still held broken connections. The #485 guard did
+  exactly the right thing (apply-failure, keep last-known-good, ack Failed), and a redeploy sealed
+  green. This is the pre-existing #486 behaviour firing on a reconnect edge, unrelated to FetchAndCache.
+- **C — endpoint failover works live.** site-b-1's step-5 fetch succeeded *from central-2*
+  (`Fetched + verified … from http://central-2:4055`) — the fetcher rotated past central-1 to the
+  second configured endpoint exactly as `GrpcDeploymentArtifactFetcher` intends.
+- **D — `Content-Type: application/json` is mandatory on the deploy POST.** Without it the minimal-API
+  route 404s (not 415). The baseline call carried it; a later batch of retries dropped it and 404'd
+  five times before the header was restored. Rig-operational, not a Phase 3 concern, but recorded so
+  the next operator does not chase a phantom outage.
+
+---
+
+## Build note (rig enablement)
+
+The rig image could not build until the Dockerfile's **build stage was pinned to `linux/amd64`**
+(`fix(docker-dev): pin the build stage to linux/amd64`). Phase 3 added the repo's first in-repo
+`.proto`, and `Grpc.Tools`' bundled `linux_arm64` `protoc` **segfaults (exit 139)** on Apple-Silicon
+Docker. `dotnet publish` is framework-dependent (portable IL + every RID's native assets), so pinning
+only the build stage leaves the runtime image native arm64; just the one-time build is emulated.

docs/plans/2026-07-22-per-cluster-mesh-program.md

@@ -0,0 +1,250 @@
+# Per-Cluster Mesh Program — Align OtOpcUa's Akka Topology with ScadaBridge
+
+> **For Claude:** This is a PROGRAM plan (phase roadmap + gates), not a bite-sized task plan.
+> The authoritative design is `docs/plans/2026-07-21-per-cluster-mesh-design.md` (decisions settled
+> 2026-07-21; Phases 0a/0b DONE). Per that design's own instruction, **each phase gets its own
+> detailed plan, written when the phase starts** (superpowers writing-plans → executing-plans),
+> so plans are authored against current code, not against a forecast. This document sequences the
+> phases, fixes the deployment topology (co-location with ScadaBridge — NEW constraint 2026-07-22),
+> and defines each phase's entry/exit gates.
+
+**Goal:** OtOpcUa runs the same Akka.NET mesh shape as ScadaBridge — **one Akka mesh per
+application `Cluster`, two nodes max**, central ↔ cluster joined by explicit transports
+(ClusterClient + gRPC + fetch), driver nodes holding **no ConfigDb connection** — so that every
+Primary-gated decision and every gated resource share one pair-local scope, and both products
+present one operational model on the shared site hardware.
+
+**Why now (2026-07-22):** OtOpcUa cluster pairs will run **on the same two Windows VMs as the
+ScadaBridge site nodes**. That deployment makes the current single-fleet-mesh design actively
+wrong for sites (a site's OtOpcUa nodes would gossip across the WAN to central and elect ONE
+Primary fleet-wide), and makes the ScadaBridge shape the obviously correct one: each site's two
+VMs host two independent, identically-postured 2-node clusters (one per product), surviving alone
+on the site LAN. It also means driver nodes must not depend on reaching central SQL — **sites
+have no SQL Server**, so §6.1's "driver nodes never connect to the ConfigDb" stops being an
+architectural preference and becomes a deployment requirement.
+
+---
+
+## Deployment topology (the co-location constraint, NEW)
+
+Each **site**: 2 Windows VMs. Each VM runs one ScadaBridge site node AND one OtOpcUa driver node.
+Two independent 2-node Akka clusters per site — they share hardware, never a mesh. **Central**:
+2 Windows VMs, each running a ScadaBridge central node and an OtOpcUa central node
+(`admin,driver`), again as separate pairs.
+
+**Per-VM port allocation (no collisions — verify against actual deployment configs in Phase 6):**
+
+| Port | Owner | Purpose |
+|---|---|---|
+| 8081 / 8082 | ScadaBridge | Akka remoting (central / site) |
+| 8083 | ScadaBridge site | gRPC h2c (streams + LocalDb sync) |
+| 8084 | ScadaBridge site | metrics |
+| 5000 (+Traefik) | ScadaBridge central | UI + Inbound API |
+| **4053** | OtOpcUa | Akka remoting |
+| **4840** | OtOpcUa | OPC UA endpoint |
+| OtOpcUa AdminUI port | OtOpcUa central (admin) | AdminUI (driver-only site nodes host no UI) |
+| OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) |
+
+**Aligned HA posture (both products, per VM — already true or landing via the selfform plan):**
+auto-down downing (15 s window), oldest-Up active/primary election, self-first seed ordering
+(the `SelfFormAfter` self-form fallback that briefly stood in for it was retired 2026-07-22),
+termination-watchdog → process exit → `sc.exe failure` restart recovery.
+One failover story for operators regardless of product.
+
+**Prerequisite ordering:** the fallback/manual-failover plan
+(`docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`) executes **before** this
+program — it is small, independent, and Phase 6 depends on its semantics (under per-pair meshes
+every node lists itself + partner as seeds, so the fallback covers both nodes of every pair;
+the site-node island guard then simply never triggers).
+
+---
+
+## What changes, in one table (from the design doc)
+
+| Aspect | Today (single fleet mesh) | Target (ScadaBridge shape) |
+|---|---|---|
+| Mesh | one gossip ring, all nodes, seeded by central-1 | one 2-node mesh per application `Cluster`; same ActorSystem name; separation by seed-node partitioning |
+| Roles | `admin` / `driver` fleet-wide | `driver` + cluster-specific `cluster-{ClusterId}`; singletons scoped to the cluster role |
+| Primary election | mesh-wide oldest Up driver (Phase 0b) | pair-local by construction — same rule, correct scope |
+| Command/control | 9 DPS topics + singleton over gossip | ClusterClient (one receptionist actor per side); central discovers nodes from `ClusterNode` rows |
+| Deploy | notify (DPS) + node fetches from ConfigDb | notify via ClusterClient; artifact fetched **from central**, cached in LocalDb |
+| Driver ConfigDb connection | direct EF connection to central SQL | **none** — LocalDb is the steady-state config store |
+| Live telemetry | 7 observability DPS topics | one gRPC stream contract (`oneof` event), central dials each cluster node |
+| Rig | six-node single mesh (`docker-dev`) | per-cluster meshes; rig models the real topology |
+
+---
+
+## Phases
+
+Each phase below is one row of design-doc §7, expanded with entry/exit gates. **Execution recipe
+per phase:** (1) invoke writing-plans in this repo to produce
+`docs/plans/2026-07-2X-mesh-phase-N-<name>.md` from the scope notes here + the design doc §
+references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit
+gate; (4) update this file's status column and the design doc's §7 table.
+
+### Phase 1 — `ClusterNode` address columns + DB-sourced ack set — **DONE 2026-07-22**
+Shipped per `2026-07-21-per-cluster-mesh-phase1.md`; see that plan's Task 6 gate record and the
+design doc's "Phase 1 as shipped" note for the two scope deviations (no cluster-scope filtering — no
+such deployment exists; no per-node role column) and the one addition (`ClusterNodeAddressReconciler`).
+**AdminUI node edit was deferred to Phase 2** — nothing reads the columns until then, and the
+migration default plus the rig seed cover every node today.
+
+**Scope:** `ClusterNode` gains Akka + gRPC address columns (mirroring ScadaBridge's `Site`
+entity `NodeAAddress`/`GrpcNodeAAddress` pattern, but per-node rows); EF migration; AdminUI node
+edit surfaces the fields; `ConfigPublishCoordinator` derives its expected-ack set from
+`ClusterNode` rows instead of `Akka.Cluster.State.Members` filtered by role (design §3 fact 3 —
+this removes the coordinator's one genuinely mesh-bound dependency).
+**Independent of the split:** yes — safe on the current mesh.
+**Exit gate:** deploy on the unchanged docker-dev rig completes with the coordinator's expected-ack
+set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node
+missing; a node up but row absent → its ack is not expected).
+
+### Phase 2 — Comm actors + ClusterClient transport — **DONE 2026-07-22 (live gate PASSED)**
+Shipped per [`2026-07-22-mesh-phase2-clusterclient-transport.md`](2026-07-22-mesh-phase2-clusterclient-transport.md)
+as a **dark switch**: `MeshTransport:Mode` defaults to `Dps`, so nothing changes on any deployment
+until the flag flips. The live gate ([`2026-07-22-mesh-phase2-live-gate.md`](2026-07-22-mesh-phase2-live-gate.md))
+flipped the whole rig to `ClusterClient` and passed for the transport, with four findings — most
+notably that `buffer-size = 0`'s stated rationale was overstated (a node applies a `TimedOut`
+deployment anyway via boot-time orphan-row replay) and that stopping both centrals strands non-seed
+nodes at the membership layer (a Phase 7 drill item, transport-independent). Both comm actors are spawned and receptionist-registered in **both** modes, so
+the flip is a config change rather than a redeploy. Three scope corrections landed against the
+design doc (that plan's header carries them in full):
+
+- the node path is `/user/node-communication`, not `/user/cluster-communication` (there is no
+  per-cluster scoping until Phase 6);
+- the substitution for `Publish` is **`SendToAll`**, not `Send` — `Send` reaches exactly one node and
+  would deploy to one member of the fleet while the rest silently kept their old config;
+- central creates **exactly one fleet-wide ClusterClient**. On a single mesh a receptionist serves
+  its whole cluster, so a client per application `Cluster` would fan every command out once per
+  cluster. Per-cluster clients become correct only when Phase 6 splits the meshes (`TODO(Phase 6)`
+  in `CentralCommunicationActor.RebuildClient`).
+
+**Exit-gate deviation — "an Ask timing out cleanly against a stopped node" cannot be run as written.**
+There is no cross-boundary Ask in Phase 2: every migrated command is fire-and-forget on the wire.
+`DispatchDeployment` is a `Tell` whose ack returns later as an independent `ApplyAck`;
+`RestartDriver` / `ReconnectDriver` / `AlarmCommand` reply `Ok = true` from the admin node the moment
+they are dispatched (`Ok` has always meant *dispatched*, not *applied* — the AdminUI string is
+literally "Restart dispatched"). The honest equivalents, both in the live gate: a **stopped node**
+fails the deploy at the apply deadline naming it (Phase 1 semantics), and **no reachable contact**
+drops the command with a Warning and never buffers it.
+
+**Scope:** one receptionist-registered actor per side (`/user/central-communication`,
+`/user/cluster-communication`), registered **per node, not as a singleton** (contact rotation);
+ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge
+idioms copied verbatim: sender-preserving `Tell(new ClusterClient.Send(...), Sender)` Ask relay,
+typed-failure reply for every unhandled message, no central buffering toward unreachable clusters
+(drop + warn), central discovers contacts from Phase 1's `ClusterNode` rows (60 s refresh +
+admin-change refresh), clusters know central from appsettings (static, restart to change).
+**Frame-size guard:** anything carrying payload sets both the frame limit and
+`log-frame-size-exceeding` (design §8) — the deploy path stays payload-free by design.
+**Exit gate:** on the still-single-mesh rig, deploy notify + acks and AdminUI Reconnect/Restart
+flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly
+against a stopped node.
+
+### Phase 3 — Config fetch-and-cache from central — **CODE-COMPLETE 2026-07-23 (Tasks 0–8; live gate Task 10 pending)**
+**Decisions taken:** the deploy artifact is served **by central over a gRPC fetch RPC**
+(`DeploymentArtifactService.Fetch`, streamed chunks — chosen over token-gated HTTP), authenticated by
+a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`,
+fail-closed — chosen over a per-deployment token, so **no migration** and `DispatchDeployment` stays
+payload-free). Cutover is a **config-flagged dark switch** `ConfigSource:Mode` (`Direct` default |
+`FetchAndCache`), per node. Plan + task ledger:
+[`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`](2026-07-22-mesh-phase3-config-fetch-and-cache.md).
+**Shipped (all green, sabotage-checked):** `ConfigSource`/`ConfigServe` options + validator; the repo's
+first in-repo `.proto` + `Grpc.Tools` codegen; central `DeploymentArtifactGrpcService` (streams the
+sealed blob, NotFound-hides unknown/not-sealed/empty); `ConfigServeAuthInterceptor` + a dedicated h2c
+Kestrel listener merged with the LocalDb-sync listener (re-binds the existing surface once, adds both
+dedicated ports); node `GrpcDeploymentArtifactFetcher` (SHA-verified reassembly, endpoint failover,
+null-on-any-failure); the `FetchAndCache` apply path (`PipeTo(Self)`, fetch→cache→apply-from-bytes,
+null = apply-failure keeping last-known-good) and bootstrap-from-the-LocalDb-pointer (no driver-side
+config SQL read); #485 coverage; and a **real two-Host gRPC boundary test** (right-key reassembles,
+wrong-key control returns null, failover). Chunking, SHA-256 verify, newest-2 retention, pair
+replication carry over unchanged. **Phase boundary held:** config **reads** only — the
+`NodeDeploymentState` ack-row write, `DbHealthProbe`, `EfAlarmConditionStateStore` stay for Phase 4.
+**Remaining (Tasks 9–10):** rig config + docs (done) and the live gate — deploy on a site pair flipped
+to `FetchAndCache` with central SQL stopped mid-fetch → retry lands; #485 last-known-good re-proven on
+the new path.
+**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable central
+boots into the current config; with central down it boots last-known-good.
+
+### Phase 4 — Cut the driver-side ConfigDb connection
+**Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local
+state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor`
+question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering,
+so define the replacement health input (candidate: central-reachability via the Phase 2/3
+transports) — this is a client-visible ServiceLevel semantics change and must be documented in
+`docs/Redundancy.md` + the interop playbook; audit `OpcUaPublishActor`'s ConfigDb use (TBD in the
+design) and re-source it; registration cleanup in `ServiceCollectionExtensions`; driver-role
+`Program.cs` branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only
+`AddConfigurationDatabase`).
+**Exit gate:** its own live gate — a driver pair runs a full deploy + alarm + historian cycle with
+**no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can
+resolve the ConfigDb context.
+
+### Phase 5 — gRPC stream contract for live telemetry
+**Scope:** one server-streaming contract carrying a `oneof` event, **cluster nodes host the gRPC
+server, central dials in** (the inverted direction is the load-bearing ScadaBridge finding —
+design §2); migrate the seven observability topics (`alerts`, `driver-health`,
+`driver-resilience-status`, `fleet-status`, `script-logs`, plus redundancy-state distribution and
+deployment-acks if Phase 2 left them on DPS); additive-only field evolution, contract locked by
+test; per-panel reconnect story for the AdminUI (design §8 — losing gossip loses free fleet
+observability).
+**Exit gate:** all AdminUI live panels green against a pair with DPS telemetry topics deleted;
+kill-and-reconnect of the central dialer recovers every stream.
+
+### Phase 6 — Mesh partition + co-location topology
+**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
+`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
+`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
+`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
+`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
+§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
+`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed
+of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
+central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
+including the co-location port table above (both products' compose files on shared per-site
+networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
+election is pair-local now) and the fallback's site-node island-guard docs note (moot — every
+node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair.
+**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair
+(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**);
+secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its
+topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot
+cross separate meshes).
+
+### Phase 7 — Failover drill + live gates
+**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
+directions; **close the two outstanding live gates**: (a) auto-down 1-vs-1 crash-the-oldest
+(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) self-first
+lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair;
+operator runbook for the co-located site (one page covering both products' failover on the same
+two VMs).
+**Exit gate:** drill green on every pair type (central, site); runbook merged; design doc §7
+table fully marked DONE. (Live gate (b) is now the **self-first cold-start-alone** drill, not
+`SelfFormAfter` — the watchdog it named was retired 2026-07-22.)
+
+---
+
+## Risks carried from the design (§8, unchanged — re-read before each phase plan)
+
+LocalDb becomes load-bearing for config (blast radius of the #485 class rises);
+`DbHealthProbeActor` feeds a client-visible value; the rig models the doomed topology until
+Phase 6; losing gossip loses free observability (7 panels); 128 KB ClusterClient frame drop is
+silent unless both knobs are set; never stack app-level LWW on LocalDb's HLC. **New (this
+program):** co-located VMs mean a VM loss now takes out one node of BOTH products at once — the
+drill in Phase 7 must include the shared-VM failure (both products fail over together), and
+resource sizing on the site VMs should be checked once both products run the full stack.
+
+## Tracking
+
+| Phase | Status |
+|---|---|
+| 0a downing strategy | DONE 2026-07-21 (live gate → Phase 7) |
+| 0b oldest-Up election | DONE 2026-07-21 |
+| Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. |
+| 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
+| 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
+| 3 fetch-and-cache | **CODE-COMPLETE 2026-07-23** (Tasks 0–8 merged to the phase branch, all green + sabotage-checked; live gate Task 10 pending). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
+| 4 cut driver ConfigDb | not started |
+| 5 gRPC telemetry | not started |
+| 6 mesh partition + co-location | not started |
+| 7 drill + live gates | not started |

docs/plans/2026-07-22-per-cluster-mesh-program.md.tasks.json

@@ -0,0 +1,15 @@
+{
+  "planPath": "docs/plans/2026-07-22-per-cluster-mesh-program.md",
+  "note": "PROGRAM plan: each task = write that phase's detailed plan (writing-plans), execute it (executing-plans), run its exit gate, update the tracking tables. Prereq task 0 is the separate selfform-fallback plan in this repo.",
+  "tasks": [
+    {"id": 0, "subject": "Prereq: execute 2026-07-22-selfform-fallback-and-manual-failover.md (7 tasks)", "status": "completed", "note": "Shipped as self-first seed ordering + AkkaClusterOptionsValidator, NOT the planned SelfFormAfter watchdog (live gate caught it islanding a failed-over node). Merged a78425ea."},
+    {"id": 1, "subject": "Phase 1: ClusterNode Akka+gRPC address columns; ConfigPublishCoordinator ack set from DB", "status": "completed", "note": "Merged 7654f24d. Escape hatch is ClusterNode.MaintenanceMode, not Enabled=0."},
+    {"id": 2, "subject": "Phase 2: comm actors + receptionist + ClusterClient transport (deploy notify/acks, driver-control)", "status": "in_progress", "blockedBy": [1]},
+    {"id": 3, "subject": "Phase 3: config fetch-and-cache from central; LocalDb steady-state config store (live gate)", "status": "pending", "blockedBy": [2]},
+    {"id": 4, "subject": "Phase 4: cut driver-side ConfigDb — EfAlarmConditionStateStore to LocalDb, DbHealthProbeActor ServiceLevel input, OpcUaPublishActor audit (live gate)", "status": "pending", "blockedBy": [3]},
+    {"id": 5, "subject": "Phase 5: gRPC oneof stream contract; migrate 7 observability topics; AdminUI reconnect story", "status": "pending", "blockedBy": [2]},
+    {"id": 6, "subject": "Phase 6: mesh partition — per-pair seeds, cluster-scoped roles/singletons, co-location rig rewrite, secrets replication re-scope", "status": "pending", "blockedBy": [0, 4, 5]},
+    {"id": 7, "subject": "Phase 7: failover drill per pair (incl. shared-VM failure) + close auto-down 1v1 and self-first cold-start-alone live gates + operator runbook", "status": "pending", "blockedBy": [6]}
+  ],
+  "lastUpdated": "2026-07-22T00:00:00Z"
+}

docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md

@@ -0,0 +1,405 @@
+# OtOpcUa: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan
+
+> ## ⚠ SUPERSEDED IN PART (2026-07-22, same day)
+>
+> **Half (1) of this plan — the `Cluster:SelfFormAfter` self-form watchdog — was retired within hours
+> of landing.** `ClusterBootstrapFallback`, the `SelfFormAfter` option and `SelfFormBootstrapTests`
+> are **deleted**; the cold-start-alone goal is now met by **self-first seed ordering** (each seed
+> node lists ITSELF as `seed-nodes[0]`), enforced at boot by `AkkaClusterOptionsValidator` and pinned
+> by `SelfFirstSeedBootstrapTests`. Reason: a timer outside Akka's join handshake cannot distinguish
+> "no seed answered" from "a seed answered and the join is in flight", and `Cluster.Join(SelfAddress)`
+> is not ignored mid-handshake — it wins. Task 8 below caught exactly that live and patched the one
+> observed shape with a TCP reachability guard; the race remained. Mesh-program Phase 6 had this
+> convergence queued and it was pulled forward. See `docs/Redundancy.md` §"Bootstrap: self-first seed
+> ordering" and ScadaBridge `4a6341d8`.
+>
+> **Half (2) — manual failover — is unaffected and still current.** Tasks below are kept verbatim as
+> the execution record, including the Task 8 live-gate evidence, which is the whole reason the
+> watchdog was rejected. Do not execute the Task 1/2/3 code as written.
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+>
+> Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The ScadaBridge half lives in `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution.
+>
+> **⚠ This plan is expected to GROW** — see "Reserved for additions" at the end. Append new tasks there (and to the `.tasks.json`) rather than renumbering existing ones.
+
+**Goal:** (1) Either node of a 2-node OtOpcUa pair can cold-start alone and become operational, unattended — a configurable, fast (default 10 s) InitJoin timeout falls back to self-forming a cluster, with a hard guard so site nodes seeded only by central can never island themselves. (2) An admin-only "Trigger failover" control on the Cluster Redundancy page performs a graceful, audited swap of the driver Primary.
+
+**Architecture:** New `AkkaClusterOptions.SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables, bound from the `Cluster` section) arms `ClusterBootstrapFallback` via an Akka.Hosting startup task inside `WithOtOpcUaClusterBootstrap` — so production AND the Cluster.Tests hosts get it identically. Wait for membership via `RegisterOnMemberUp`; on expiry, `Cluster.Join(SelfAddress)` — only if this node's own address is in its own `SeedNodes`. Manual failover = graceful `Cluster.Leave(<oldest Up driver member>)` via a new `IManualFailoverService` in ControlPlane, surfaced on `/clusters/{id}/redundancy`.
+
+**Tech Stack:** .NET 10, Akka.NET 1.5.62, Akka.Hosting 1.5.62 (`AddStartup`), Blazor Server (AdminUI, InteractiveServer), bUnit, xunit. No new packages.
+
+**Branch:** `feat/selfform-fallback` off `master`.
+
+---
+
+## Design essentials (from the shared design doc)
+
+**The defect:** Akka only lets the FIRST listed seed self-join to form a *new* cluster; every other node loops on `InitJoin` forever — "auto-down removes the crash outage, not this one" (`docs/Redundancy.md:244-247`). A lone cold-starting `central-2` (or a pair node under the future per-cluster mesh) never comes Up.
+
+**Behavior spec:**
+
+| Scenario | Behavior with fallback |
+|---|---|
+| Peer alive (any boot order) | Normal seed join in ms — fallback never fires |
+| Lone cold-start, self IS in own `SeedNodes` | After `SelfFormAfter`: warn log + `Cluster.Join(SelfAddress)` → Up alone (`min-nr-of-members=1`) |
+| Lone cold-start, self NOT in own `SeedNodes` | Fallback inert (info log). **This is the docker-dev site-node topology** — site-a/b nodes list only `central-1`; self-forming there would island them permanently, so they must wait. |
+| Peer boots after survivor self-formed | Its InitJoin is answered → joins as youngest. No island. |
+| Both pair nodes cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) |
+| `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior |
+| ~~Window expires mid-join-handshake~~ | ~~Benign: Akka ignores `Join` once joined~~ **FALSE — proven live (Task 8): `Join(self)` during an in-flight join abandons it and islands the node. Fixed by the TCP reachability guard: never self-form while any seed peer accepts a connection; wait a window and re-arm. ScadaBridge rejected the watchdog entirely for self-first seed ordering; OtOpcUa converges at mesh-program Phase 6.** |
+
+**Manual failover rules:** graceful `Leave` of the **oldest Up `driver` member** (identical query to `RedundancyStateActor.SelectDriverPrimary` — `Member.AgeOrdering`, `Leaving` excluded via `Status == Up`), never `Down`. The leaving node hands over via the cluster-leave phases, `CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it, and it rejoins as youngest → the survivor is Primary (ServiceLevel 250 moves; OPC UA clients re-select). Admin-only; peer guard (<2 Up driver members → disabled); confirmation dialog; audited via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam before the Leave. **Mesh-scope caveat:** until the per-cluster mesh lands, the election is mesh-wide — the button acts on THE Primary of the whole Akka cluster, not per-`ServerCluster` row; the UI must say so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md:84-99`).
+
+**Multi-node TestKit:** assessed, NOT used — everything is deterministic in-process via real hosts through `WithOtOpcUaClusterBootstrap` (the `SplitBrainResolverActivationTests` pattern), and the Akka TestKit family is xunit-v2-only while this repo's integration tree is xunit.v3. Full verdict in the shared design doc.
+
+---
+
+### Task 1 (B1): `SelfFormAfter` on `AkkaClusterOptions`
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+**Parallelizable with:** none (first task)
+
+**Files:**
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs`
+
+**Step 1: Branch**
+
+```bash
+cd ~/Desktop/OtOpcUa && git checkout master && git checkout -b feat/selfform-fallback
+```
+
+**Step 2: Add the property** (after `SplitBrainResolverStrategy`, mirroring its XML-doc depth):
+
+```csharp
+    /// <summary>
+    /// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
+    /// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
+    /// while its peer is down loops on InitJoin forever ("auto-down removes the crash outage,
+    /// not this one" — docs/Redundancy.md). When this node has waited longer than this window
+    /// without cluster membership it joins itself — but ONLY if its own address appears in
+    /// <see cref="SeedNodes"/>; a non-seed node (e.g. a site node whose only seed is central-1)
+    /// stays waiting, because self-forming there creates a permanent island. Default 10s
+    /// (same-datacenter pair; a live peer answers InitJoin in milliseconds). <c>null</c> or a
+    /// non-positive value disables the fallback. Accepted trade: both pair nodes cold-starting
+    /// inside the window while mutually unreachable form two clusters — the same dual-active
+    /// class the auto-down strategy already accepts, same recovery (restart one side).
+    /// </summary>
+    public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);
+```
+
+**Step 3:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster` — 0 warnings. Commit:
+
+```bash
+git add -A && git commit -m "feat(cluster): SelfFormAfter option (bootstrap self-form fallback window)"
+```
+
+(The default-value pin test lands with Task 3's test file — the property is inert until Task 2.)
+
+---
+
+### Task 2 (B2): `ClusterBootstrapFallback` + `AddStartup` wiring
+
+**Classification:** high-risk (cluster formation behavior)
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterBootstrapFallback.cs`
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (`WithOtOpcUaClusterBootstrap`, after the `AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend)` call at line 93)
+
+**Step 1: Implement the fallback class** (deliberately duplicated from ScadaBridge's, like the termination watchdogs — see that repo's `ClusterBootstrapFallback.cs` for the full doc comment to adapt):
+
+```csharp
+using Akka.Actor;
+using Microsoft.Extensions.Logging;
+
+namespace ZB.MOM.WW.OtOpcUa.Cluster;
+
+/// <summary>
+/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1). Akka only
+/// lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin forever —
+/// "auto-down removes the crash outage, not this one" (docs/Redundancy.md). Waits
+/// <see cref="AkkaClusterOptions.SelfFormAfter"/> for membership, then joins itself.
+/// <para><b>Island safety:</b> fires ONLY when this node's own address is in its own
+/// <see cref="AkkaClusterOptions.SeedNodes"/> — a site node seeded only by central-1 must wait,
+/// not island. Sequential recovery is island-free: Akka's join protocol prefers an existing
+/// cluster (a booting node only self-forms when NO seed answers InitJoin).</para>
+/// <para><b>Benign race:</b> Akka ignores <c>Join</c> once the node has joined a cluster this
+/// incarnation. Residual risk: both pair nodes cold-starting inside the window while mutually
+/// unreachable self-form separately — the same dual-active class auto-down accepts.</para>
+/// </summary>
+public static class ClusterBootstrapFallback
+{
+    public static void Arm(ActorSystem system, AkkaClusterOptions options, ILogger logger)
+    {
+        if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero)
+        {
+            logger.LogInformation(
+                "Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting "
+                + "while its peer is down will wait on InitJoin indefinitely.");
+            return;
+        }
+
+        var cluster = Akka.Cluster.Cluster.Get(system);
+        var self = cluster.SelfAddress;
+        var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self));
+        if (!isSeed)
+        {
+            logger.LogInformation(
+                "Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list "
+                + "[{Seeds}] — self-forming here would island it from the real seeds.",
+                self, string.Join(", ", options.SeedNodes));
+            return;
+        }
+
+        var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
+        cluster.RegisterOnMemberUp(() => joined.TrySetResult());
+
+        _ = Task.Run(async () =>
+        {
+            var winner = await Task.WhenAny(joined.Task, Task.Delay(window));
+            if (winner == joined.Task || system.WhenTerminated.IsCompleted)
+                return;
+            logger.LogWarning(
+                "No cluster membership after {Window} — no seed answered InitJoin (peer down at boot). "
+                + "Self-forming a cluster at {Self} so this node becomes operational; if the peer was "
+                + "merely partitioned (not dead), the pair is now dual-active — restart one side after "
+                + "the partition heals (accepted availability-first trade, decision 2026-07-22).",
+                window, self);
+            cluster.Join(self);
+        });
+    }
+
+    private static bool TryParseAddress(string seed, out Address address)
+    {
+        try { address = Address.Parse(seed); return true; }
+        catch { address = default!; return false; }
+    }
+}
+```
+
+(If the Cluster project lacks a `Microsoft.Extensions.Logging.Abstractions` reference, add the `PackageReference` — check `Directory.Packages.props` for the pinned version first.)
+
+**Step 2: Wire via Akka.Hosting startup hook** — in `WithOtOpcUaClusterBootstrap`, after the downing HOCON line (~93):
+
+```csharp
+        // InitJoin self-form fallback (decision 2026-07-22): registered as an Akka.Hosting
+        // startup task so every host built through this bootstrap — production AND the test
+        // hosts in Cluster.Tests — arms it identically. Guarded inside Arm: disabled when
+        // SelfFormAfter is unset; inert when this node is not in its own seed list (site nodes
+        // seeded only by central-1 must wait, not island).
+        var fallbackLogger = (serviceProvider.GetService<ILoggerFactory>()
+                              ?? Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance)
+            .CreateLogger("ZB.MOM.WW.OtOpcUa.Cluster.ClusterBootstrapFallback");
+        builder.AddStartup((system, _) =>
+        {
+            ClusterBootstrapFallback.Arm(system, options, fallbackLogger);
+            return Task.CompletedTask;
+        });
+```
+
+(Akka.Hosting 1.5.62 has `AddStartup`. If the overload signature differs, alternative: register an `IHostedService` in `AddOtOpcUaCluster` mirroring `ActorSystemTerminationWatchdog`'s lazy-accessor pattern — but prefer `AddStartup` so test hosts arm it for free.)
+
+**Step 3:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster` — 0 warnings.
+
+**Step 4: Commit**
+
+```bash
+git add -A && git commit -m "feat(cluster): InitJoin self-form fallback armed via Akka.Hosting startup task"
+```
+
+---
+
+### Task 3 (B3): `SelfFormBootstrapTests` — 4 scenarios through the production bootstrap
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 4
+
+**Files:**
+- Create: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SelfFormBootstrapTests.cs`
+
+**Step 1: Write the tests** — follow `SplitBrainResolverActivationTests`' pattern (real `Host.CreateDefaultBuilder()` + `AddAkka` + `WithOtOpcUaClusterBootstrap`; that file's remarks explain why testing through the production bootstrap is mandatory — a test that re-creates the wiring pins the test's wiring, not the shipped one). Helper + scenarios:
+
+```csharp
+public sealed class SelfFormBootstrapTests
+{
+    private static int FreePort()
+    {
+        var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
+        l.Start();
+        var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port;
+        l.Stop();
+        return p;
+    }
+
+    private static async Task<IHost> StartNodeAsync(
+        string systemName, int selfPort, int peerPort, TimeSpan? selfFormAfter, bool selfInSeeds = true)
+    {
+        var options = new AkkaClusterOptions
+        {
+            SystemName = systemName, Hostname = "127.0.0.1", PublicHostname = "127.0.0.1",
+            Port = selfPort, Roles = new[] { "admin", "driver" },
+            SelfFormAfter = selfFormAfter,
+            SeedNodes = selfInSeeds
+                ? new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}",
+                          $"akka.tcp://{systemName}@127.0.0.1:{selfPort}" }   // self SECOND — only the fallback can form it
+                : new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}" },
+        };
+        var builder = Host.CreateDefaultBuilder();
+        builder.ConfigureServices(services =>
+        {
+            services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
+            services.AddAkka(systemName, (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
+        });
+        var host = builder.Build();
+        await host.StartAsync();
+        return host;
+    }
+
+    // 1) SelfFormAfter_defaults_to_ten_seconds
+    //      new AkkaClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10));
+    // 2) Lone_non_first_seed_self_forms_after_the_window
+    //      dead peer port, window 2s → poll Cluster.Get(system).State until 1 Up member ≤ 20s.
+    // 3) Disabled_fallback_keeps_waiting  (window null → State.Members empty after 6s;
+    //      POSITIVE CONTROL: cluster.Join(cluster.SelfAddress) → 1 Up ≤ 20s — proves the node
+    //      was formable and only the fallback was absent.)
+    // 4) Site_node_seeded_only_by_central_never_self_forms  (selfInSeeds:false, window 1s →
+    //      Members empty after 5s; positive control join. THE guard test: this is the exact
+    //      docker-dev topology — site nodes list only central-1.)
+}
+```
+
+Use a unique `systemName` per test (`otopcua-selfform-1` …) so concurrent hosts cannot cross-join; dispose each host in `finally` (`await host.StopAsync(); host.Dispose();`).
+
+**Step 2: Prove the key assertion has teeth** — before relying on scenario 2, temporarily run it with `selfFormAfter: null` and confirm it times out (red), then restore. Record that in the task notes.
+
+**Step 3: Run**
+
+```bash
+dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~SelfFormBootstrap"
+```
+Expected: 4 PASS.
+
+**Step 4: Commit**
+
+```bash
+git add -A && git commit -m "test(cluster): self-form fallback — lone-seed forms, disabled waits, site-node guard"
+```
+
+---
+
+### Task 4 (B4): Docs — Redundancy.md + mesh design note
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 3
+
+**Files:**
+- Modify: `docs/Redundancy.md` (~lines 162, 244–247) — replace the bootstrap-constraint text ("Auto-down removes the crash outage, not this one") with `SelfFormAfter` semantics, the 10 s default, the site-node island guard (why driver-only nodes seeded by central-1 deliberately do NOT self-form under the CURRENT one-mesh topology), and the note that under the future per-cluster mesh each pair node is a seed so the fallback covers both.
+- Modify: `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2/§7 — one-line note: the "registered outage gap" is now closed by `SelfFormAfter` (link Redundancy.md).
+- Modify: `CLAUDE.md` — brief note in the redundancy-corrections area.
+
+Commit: `docs(redundancy): SelfFormAfter closes the seed-node bootstrap gap`.
+
+---
+
+### Task 5 (B5): Full verification + optional docker-dev live check
+
+**Classification:** standard
+**Estimated implement time:** ~5 min (suite runtime dominates)
+**Parallelizable with:** none
+
+```bash
+cd ~/Desktop/OtOpcUa
+dotnet build ZB.MOM.WW.OtOpcUa.slnx     # 0 warnings
+dotnet test  ZB.MOM.WW.OtOpcUa.slnx     # green vs the pre-existing baseline
+```
+
+Optional live check (docker-dev six-node mesh): `docker compose stop central-1 central-2 && docker compose start central-2` → central-2 (a listed seed) self-forms in ~10 s; site nodes (non-seeds) keep waiting — the island guard on the real topology. Then `docker compose start central-1` → joins central-2's cluster, mesh re-forms. If the rig isn't running, record deferred-live.
+
+---
+
+### Task 6 (D4): Manual failover service + cluster-level test
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs`
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/ManualFailoverService.cs`
+- Modify: the ControlPlane/Host DI registration point (find where `RedundancyStateActor`-adjacent services register; admin-role branch of `Program.cs`)
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ManualFailoverServiceTests.cs`
+
+**Step 1: Failing tests** — reuse the two-node fixture pattern from `RedundancyPrimaryElectionTests` (node A older on the **higher** port so age ≠ address order; that fixture even has a guard test asserting the divergence — keep it honest here too):
+
+- `FailOver_targets_the_oldest_driver_member_not_the_role_leader`: service picks node A (oldest, higher port) even though B is `RoleLeader("driver")`; A's system terminates (graceful exit); B's next `RedundancyStateActor` snapshot names B Primary.
+- `FailOver_refuses_when_no_driver_peer_exists`: single driver member → returns null, no Leave, node still Up (positive assert).
+- `Parity_with_SelectDriverPrimary`: on the same fixture, the service's target equals the member `RedundancyStateActor` names Primary — pins the two 5-line queries together.
+
+**Step 2: Implement** — same shape as ScadaBridge's `FailOverCore` but role `"driver"`:
+
+```csharp
+public static Address? FailOverCore(ActorSystem system, bool dryRun = false)
+{
+    var cluster = Akka.Cluster.Cluster.Get(system);
+    // Intentionally identical to RedundancyStateActor.SelectDriverPrimary (oldest Up driver,
+    // Member.AgeOrdering) — the node acted on must be exactly the one the election names.
+    // Pinned by ManualFailoverServiceTests.Parity_with_SelectDriverPrimary.
+    var drivers = cluster.State.Members
+        .Where(m => m.Status == MemberStatus.Up && m.Roles.Contains("driver"))
+        .OrderBy(m => m, Member.AgeOrdering)
+        .ToList();
+    if (drivers.Count < 2) return null;
+    var primary = drivers[0];
+    if (!dryRun) cluster.Leave(primary.Address);
+    return primary.Address;
+}
+```
+
+Async wrapper `FailOverDriverPrimaryAsync(string actor)`: dry-run peer guard → audit via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam (action `cluster.manual-failover`, actor, target in `DetailsJson` — copy the call shape from an existing audited AdminUI admin action) → real Leave → warn log → return address string.
+
+**Step 3:** Tests PASS → commit `feat(redundancy): manual failover service — graceful Leave of the driver Primary`.
+
+---
+
+### Task 7 (D5): ClusterRedundancy page — live panel + failover button + bUnit tests
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Clusters/ClusterRedundancy.razor` (page is already `@rendermode RenderMode.InteractiveServer` — no rendermode work; and it lives in the AdminUI host, not an RCL, so the static-router wrapper concern from lmxopcua#483 does not apply)
+- Test: the AdminUI bUnit suite (place beside the existing Clusters page tests)
+- Modify: `docs/Redundancy.md` — "Manual failover" subsection (button semantics, mesh-wide scope caveat)
+
+**Step 1: Failing bUnit tests:** (a) button hidden under the read-only policy (`AdminUiPolicies.AuthenticatedRead`), visible under the admin/write policy — check `AdminUiPolicies` for the exact policy the mutating Clusters pages use (e.g. `ClusterEdit.razor`) and use the same one; (b) disabled with tooltip when the live snapshot shows <2 Up driver members; (c) confirm flow calls a fake `IManualFailoverService` exactly once.
+
+**Step 2: Implement.** Add a "Live redundancy" panel above the static config table: current driver Primary + Up driver members from a `GetSnapshot()` read-only companion on `IManualFailoverService` (sourced from live `Cluster.State`; do NOT subscribe to the DPS topic for v1), the mesh-scope notice ("in the current single-mesh topology this acts on the whole mesh's Primary"), and the admin-gated **Trigger failover** button with confirm dialog (states: the Primary node restarts, ServiceLevel 250 moves to the survivor, OPC UA clients re-select). On confirm: call the service with the authenticated user name; surface the returned target address.
+
+**Step 3:** Tests PASS → commit `feat(adminui): manual failover control on the cluster redundancy page`.
+
+**Live check (fold into the Task 5 gate when docker-dev is up):** press the button, ServiceLevel 250 moves (old Primary drops off, survivor publishes 250), denial meters stay quiet, audit event recorded.
+
+---
+
+## Reserved for additions (this plan is expected to grow)
+
+Append new tasks below as **Task 8+** (and extend `.tasks.json`) — do not renumber Tasks 1–7. Candidate items already flagged in the family docs, awaiting a decision on which to include:
+
+- **Configurable downing window** — promote `ServiceCollectionExtensions.DowningStableAfter` (hard-coded 15 s) to `AkkaClusterOptions` (e.g. `Cluster:DowningStableAfter`), keeping the pinned invariant window > `acceptable-heartbeat-pause` (`akka_failover.md` §6.2 calls this out; ScadaBridge's equivalent `Cluster:StableAfter` is already config).
+- **Failure-detector tuning for faster failover** — expose `heartbeat-interval` / `acceptable-heartbeat-pause` (currently fixed in `Resources/akka.conf`) as options, per the availability-first tuning table in `akka_failover.md` §2.
+- ~~Auto-down 1-vs-1 live gate~~ / ~~per-cluster mesh interplay~~ — **now owned by the per-cluster mesh PROGRAM plan** (`docs/plans/2026-07-22-per-cluster-mesh-program.md`, added 2026-07-22): Phases 1–7 align OtOpcUa's mesh with ScadaBridge's shape (one 2-node mesh per application Cluster, co-located on the ScadaBridge VMs). That program lists THIS plan as its prerequisite (execute this one first); its Phase 6 removes the ClusterRedundancy mesh-scope caveat + the site-node island-guard docs note, and Phase 7 closes the auto-down 1-vs-1 and `SelfFormAfter` live gates on the real per-pair topology.
+- *(add further items here)*
+
+---
+
+## Completion
+
+- Merge decision via the finishing-a-development-branch flow (family convention: ff-merge to `master` + push to gitea `lmxopcua`, or PR — ask the user).
+- After merge: update `scadaproj/akka_failover.md` §6.1 status, `scadaproj/CLAUDE.md` index row, and memory `ha-availability-over-partition-safety` (tracked in the scadaproj index plan).
+- Verification-before-completion applies throughout; live gates may be recorded deferred-live if docker-dev is down.

docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json

@@ -0,0 +1,14 @@
+{
+  "planPath": "docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md",
+  "tasks": [
+    {"id": 0, "subject": "Task 1: SelfFormAfter on AkkaClusterOptions", "status": "completed"},
+    {"id": 1, "subject": "Task 2: ClusterBootstrapFallback + AddStartup wiring", "status": "completed", "blockedBy": [0]},
+    {"id": 2, "subject": "Task 3: SelfFormBootstrapTests — 4 scenarios incl. site-node island guard", "status": "completed", "blockedBy": [1], "notes": "Teeth proven: scenario 2 run with selfFormAfter:null failed (red) before restore. A 5th test was added by Task 8."},
+    {"id": 3, "subject": "Task 4: Docs — Redundancy.md bootstrap section + mesh design note", "status": "completed", "blockedBy": [1]},
+    {"id": 4, "subject": "Task 5: Full verification + optional docker-dev live check", "status": "completed", "blockedBy": [2, 3], "notes": "Live gate PASSED for the cold-start case (defect observed on old image, fix on new, island guard on a real site node, one six-node cluster after recovery)."},
+    {"id": 5, "subject": "Task 6: Manual failover service + cluster-level test", "status": "completed", "blockedBy": [4], "notes": "4/4. Teeth proven: sabotaging FailOverCore to address order turned the target + parity tests red."},
+    {"id": 6, "subject": "Task 7: ClusterRedundancy live panel + failover button + tests", "status": "completed", "blockedBy": [5], "notes": "PLAN DEVIATION: plan said bUnit; repo has no bUnit (PageAuthorizationGuardTests documents itself as the substitute). Adapted to a pure ManualFailoverPageModel + 9 unit tests, razor as thin shell. Policy is FleetAdmin, not the ConfigEditor the neighbouring cluster pages use (this restarts a node; ConfigEditor admits Designer). Live-verified: panel, confirm dialog, execution, singleton handover, audit row."},
+    {"id": 7, "subject": "Task 8 (NEW, from live gate): reachability guard — don't self-form while a seed peer is reachable", "status": "completed", "blockedBy": [6], "notes": "DEFECT: the Task 7 live drill caught the self-form fallback islanding the node the failover bounced. central-1 restarted, got InitJoinAck from central-2 at 10:49:05 (join in flight, healthy) but no Welcome inside the window (peer's ring still held the old incarnation until 10:49:06); at 10:49:15 the fallback fired and formed a SECOND cluster. The plan's 'Window expires mid-join-handshake -> benign, Akka ignores Join once joined' row is FALSE — the node had not joined, and Join(self) wins. Manual failover deliberately produces that restart, so the two halves of the plan collided. FIX (user chose the reachability guard): TCP-probe the other seed addresses before self-forming; a reachable peer means wait another window and re-arm. Proven three ways: (1) new regression test whose 'peer' is a bare TcpListener speaking no Akka — RED without the guard; (2) live, a throwaway node seeded at sql:1433 (TCP-reachable, not Akka) logged the guard every 10s and never self-formed; (3) positive control, the same node seeded at a dead port self-formed at +10s. Failover re-drill on the rig: no island, all six nodes one cluster."}
+  ],
+  "lastUpdated": "2026-07-22T00:00:00Z"
+}

docs/v2/Cluster.md

@@ -64,7 +64,7 @@ The Cluster.Tests project verifies these key values stay correct (`HoconLoaderTe
 - `Hostname`: interface to bind. `0.0.0.0` listens on every interface.
 - `Port`: TCP port for cluster gossip. Default 4053.
 - `PublicHostname`: address advertised in cluster gossip. Must be reachable by every other node.
-- `SeedNodes`: where new nodes go to join. List one (or two) stable nodes. First node bootstraps the cluster from its own address.
+- `SeedNodes`: where new nodes go to join. List one (or two) stable nodes. **Order is load-bearing:** Akka only lets `seed-nodes[0]` form a *new* cluster (`FirstSeedNodeProcess`); every other node retries `InitJoin` forever. A node that is one of its own seeds must therefore list ITSELF first, or it can never cold-start while its peer is down — enforced at boot by `AkkaClusterOptionsValidator`, explained in [Redundancy.md § Bootstrap: self-first seed ordering](../Redundancy.md#bootstrap-self-first-seed-ordering). Nodes seeded only by someone else (driver-only site nodes) are exempt.
 - `Roles`: free-form tags Akka gossip propagates. v2 uses `admin` + `driver`; per-role wiring in `Program.cs` reads `OTOPCUA_ROLES` env var, not this list — these two should stay in sync.
 
 Per-role overlay files (`appsettings.admin.json`, `appsettings.driver.json`, `appsettings.admin-driver.json`) layer on top of base `appsettings.json` based on the parsed `OTOPCUA_ROLES` (alphabetical, joined by `-`). See [ServiceHosting.md § Per-role configuration overlays](../ServiceHosting.md#per-role-configuration-overlays).

docs/v2/config-db-schema.md

@@ -148,10 +148,13 @@ CREATE TABLE dbo.ClusterNode (
     Host                            nvarchar(255)   NOT NULL,
     OpcUaPort                       int             NOT NULL DEFAULT 4840,
     DashboardPort                   int             NOT NULL DEFAULT 8081,
+    AkkaPort                        int             NOT NULL DEFAULT 4053,
+    GrpcPort                        int             NULL,
     ApplicationUri                  nvarchar(256)   NOT NULL,
     ServiceLevelBase                tinyint         NOT NULL DEFAULT 200,
     DriverConfigOverridesJson       nvarchar(max)   NULL CHECK (DriverConfigOverridesJson IS NULL OR ISJSON(DriverConfigOverridesJson) = 1),
     Enabled                         bit             NOT NULL DEFAULT 1,
+    MaintenanceMode                 bit             NOT NULL DEFAULT 0,
     LastSeenAt                      datetime2(3)    NULL,
     CreatedAt                       datetime2(3)    NOT NULL DEFAULT SYSUTCDATETIME(),
     CreatedBy                       nvarchar(128)   NOT NULL
@@ -167,6 +170,49 @@ CREATE UNIQUE INDEX UX_ClusterNode_Primary_Per_Cluster
     WHERE RedundancyRole = 'Primary';
 ```
 
+#### `AkkaPort` / `GrpcPort` — central's dial targets (per-cluster mesh Phase 1)
+
+These are **the addresses central dials**, not the node's own binding configuration. The node binds
+from `Cluster:Port` in its own appsettings; these columns duplicate that value for a reader that
+cannot see the node's config. Today central shares a gossip ring with every node and does not need
+them — [per-cluster mesh Phase 2](../plans/2026-07-21-per-cluster-mesh-design.md) splits the fleet
+into one mesh per `Cluster`, after which `Host` + `AkkaPort` is how central builds its ClusterClient
+contact points, and `GrpcPort` is the Phase 5 telemetry stream.
+
+`AkkaPort` is `NOT NULL DEFAULT 4053` because every node listens on a remoting port — `0` is never a
+truthful value, and rows predating the column must migrate to something real. `GrpcPort` is nullable
+with **no** default because nothing listens on it until Phase 5, and a non-null default would assert
+a port that does not exist.
+
+**The duplication against `Cluster:Port` is not enforced by the schema.** A node that binds 4054 while
+its row says 4053 is unreachable from central in Phase 2, and the symptom there is a silent absence of
+acks rather than an error. `ClusterNodeAddressReconcilerActor` (an admin-role singleton) compares the
+rows against observed cluster membership and logs an Error on mismatch. It reads membership rather
+than having each driver node assert its own row, because Phase 4 removes the driver nodes' ConfigDb
+connection entirely.
+
+#### `Enabled` vs `MaintenanceMode` — two flags, two questions
+
+Phase 1 made the rows the deploy path's **expected-ack set**: `ConfigPublishCoordinator` no longer
+derives it from cluster membership, so a row whose node is down now fails the deployment at the apply
+deadline instead of letting it seal green without that node.
+
+The escape hatch is **`MaintenanceMode = 1`**, not `Enabled = 0`:
+
+| Flag | Question it answers | Checked by |
+|---|---|---|
+| `Enabled` | Is this node part of the cluster's declared topology? | `DraftValidator.ValidateClusterTopology` — the enabled-node count must equal `ServerCluster.NodeCount` |
+| `MaintenanceMode` | Should we expect it to participate right now? | `ConfigPublishCoordinator` (expected-ack set) and `ClusterNodeAddressReconciler` (absence warnings) |
+
+`Enabled = 0` on one node of a Warm/Hot pair is **rejected at the deploy gate** with
+`ClusterEnabledNodeCountMismatch` — the enabled count drops to 1 while `NodeCount` stays 2. Since
+every cluster in the target topology is a 2-node pair, `Enabled` cannot serve as the maintenance
+hatch at all. The Phase 1 live gate found this; the two rules were independently reasonable and
+contradictory together, so the meanings were split rather than either rule being weakened.
+
+A node in maintenance is still enabled, so the topology gate still passes, and the runtime redundancy
+posture (`NodeCount` / `RedundancyMode`) is unchanged.
+
 `DriverConfigOverridesJson` shape:
 
 ```jsonc

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs

@@ -20,7 +20,31 @@ public sealed class AkkaClusterOptions
     /// </summary>
     public string PublicHostname { get; set; } = "127.0.0.1";
 
-    /// <summary>Gets or sets the seed nodes for cluster bootstrapping.</summary>
+    /// <summary>
+    /// Gets or sets the seed nodes for cluster bootstrapping.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b>ORDER IS LOAD-BEARING (decision 2026-07-22): a node that is one of its own seeds
+    ///         must list ITSELF first.</b> Akka runs <c>FirstSeedNodeProcess</c> — the only bootstrap
+    ///         path that can form a NEW cluster when no peer answers <c>InitJoin</c> — exclusively
+    ///         when <c>seed-nodes[0]</c> is this node's own address; any other node runs
+    ///         <c>JoinSeedNodeProcess</c>, which retries <c>InitJoin</c> forever and can never form a
+    ///         cluster. So listing both peers does NOT mean either can cold-start alone: a node that
+    ///         lists its partner first never comes Up while that partner is down. Self-first ordering
+    ///         closes that gap inside Akka's own handshake — unlike the retired
+    ///         <c>Cluster:SelfFormAfter</c> watchdog, which sat outside it and could not tell "no
+    ///         seed answered" from "a seed answered and the join is in flight".
+    ///     </para>
+    ///     <para>
+    ///         The rule is <b>conditional</b>: it binds only when this node's own address appears in
+    ///         this list. A node seeded exclusively by someone else — today's docker-dev site nodes,
+    ///         which list only <c>central-1</c> — is legitimately not a seed and is exempt. Enforced
+    ///         at boot by <see cref="AkkaClusterOptionsValidator"/>; identity is
+    ///         <see cref="PublicHostname"/> + <see cref="Port"/> (what Akka puts in
+    ///         <c>SelfAddress</c>), never the <see cref="Hostname"/> bind address.
+    ///     </para>
+    /// </remarks>
     public string[] SeedNodes { get; set; } = Array.Empty<string>();
 
     /// <summary>
@@ -29,4 +53,30 @@ public sealed class AkkaClusterOptions
     /// <c>admin</c>, <c>driver</c>, <c>dev</c>.
     /// </summary>
     public string[] Roles { get; set; } = Array.Empty<string>();
+
+    /// <summary>
+    /// How the cluster decides to down a node it can no longer reach. One of <c>auto-down</c>
+    /// (default) or <c>keep-oldest</c>; any other value fails the host at startup.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b><c>auto-down</c> — availability.</b> The leader among the <i>reachable</i> members
+    ///         downs the unreachable peer after
+    ///         <see cref="ServiceCollectionExtensions.DowningStableAfter"/>. A hard crash of either
+    ///         node — including the oldest — fails over to the survivor with no operator action.
+    ///         The trade is that a genuine network partition (both nodes alive, link cut) leaves
+    ///         both sides running active until an operator restarts one.
+    ///     </para>
+    ///     <para>
+    ///         <b><c>keep-oldest</c> — partition-safety.</b> The SBR resolver sacrifices the younger
+    ///         side of a split, so a partition can never run dual-active. <b>The cost is severe for a
+    ///         two-node pair:</b> it cannot survive a crash of the oldest node at all. Akka.NET's
+    ///         <c>KeepOldest.OldestDecision</c> only lets <c>down-if-alone</c> rescue a side holding
+    ///         &gt;= 2 members, so the 1-vs-1 survivor downs <i>itself</i> and shuts down — the
+    ///         redundancy pair turns a single-node crash into a total outage. Choose this only for
+    ///         clusters of three or more nodes, or where dual-active is genuinely worse than an
+    ///         outage.
+    ///     </para>
+    /// </remarks>
+    public string SplitBrainResolverStrategy { get; set; } = "auto-down";
 }

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptionsValidator.cs

@@ -0,0 +1,96 @@
+using Akka.Actor;
+using ZB.MOM.WW.Configuration;
+
+namespace ZB.MOM.WW.OtOpcUa.Cluster;
+
+/// <summary>
+///     Fail-fast startup validator for <see cref="AkkaClusterOptions"/>, built on the shared
+///     <c>ZB.MOM.WW.Configuration</c> <see cref="OptionsValidatorBase{TOptions}"/> and wired through
+///     <c>AddValidatedOptions</c> in <see cref="ServiceCollectionExtensions.AddOtOpcUaCluster"/>.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>Self-first seed ordering (decision 2026-07-22).</b> Akka runs
+///         <c>FirstSeedNodeProcess</c> — the only bootstrap path that can form a NEW cluster when no
+///         peer answers <c>InitJoin</c> — exclusively when <c>seed-nodes[0]</c> is this node's own
+///         address. Every other node runs <c>JoinSeedNodeProcess</c> and retries <c>InitJoin</c>
+///         forever. A node that lists its partner first therefore cannot cold-start while that
+///         partner is down, and the failure is <i>silent</i>: the process runs, the port listens, the
+///         node simply never becomes a member. Enforced here so it is loud, at boot, on the node
+///         that has the problem.
+///     </para>
+///     <para>
+///         <b>The rule is conditional and must stay that way.</b> It binds only when this node's own
+///         address appears in its own seed list. A driver-only site node is seeded solely by
+///         <c>central-1</c> and is legitimately not a seed of anything; an unconditional "self must
+///         be first" rule would refuse to boot every one of them.
+///     </para>
+///     <para>
+///         <b>Identity is the advertised address, never the bind address.</b> In docker-dev
+///         <c>Cluster:Hostname</c> is <c>0.0.0.0</c> and <c>Cluster:PublicHostname</c> is the
+///         container DNS name — the value Akka puts in <c>SelfAddress</c> and the only one a seed URI
+///         can match. Comparing against <c>Hostname</c> would match nothing anywhere, silently exempt
+///         every node, and leave this rule inert. <c>PublicHostname</c> falling back to
+///         <c>Hostname</c> when blank mirrors Akka's own <c>public-hostname</c> semantics.
+///     </para>
+/// </remarks>
+public sealed class AkkaClusterOptionsValidator : OptionsValidatorBase<AkkaClusterOptions>
+{
+    /// <inheritdoc />
+    protected override void Validate(ValidationBuilder builder, AkkaClusterOptions options)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+
+        var seeds = options.SeedNodes ?? Array.Empty<string>();
+        if (seeds.Length == 0)
+        {
+            // Single-node / dev default: nothing is claimed about ordering, so there is nothing to
+            // enforce. (Akka bootstraps from an empty seed list by waiting for an explicit join.)
+            return;
+        }
+
+        var selfHost = string.IsNullOrWhiteSpace(options.PublicHostname)
+            ? options.Hostname
+            : options.PublicHostname;
+
+        var selfIndex = Array.FindIndex(seeds, s => IsSelf(s, selfHost, options.Port));
+        if (selfIndex <= 0)
+        {
+            // -1 = this node is not one of its own seeds (exempt); 0 = correctly ordered.
+            return;
+        }
+
+        builder.Add(
+            $"Cluster:SeedNodes must list this node itself first: entry {selfIndex} is this node "
+            + $"('{seeds[selfIndex]}') but entry 0 is '{seeds[0]}'. Akka only lets seed-nodes[0] form "
+            + "a new cluster (FirstSeedNodeProcess); every other node retries InitJoin forever, so "
+            + "with the partner listed first this node can never start while its peer is down. Swap "
+            + "the entries — see docs/Redundancy.md → 'Bootstrap: self-first seed ordering'.");
+    }
+
+    /// <summary>
+    /// True when <paramref name="seed"/> addresses this node itself — host AND port.
+    /// </summary>
+    /// <remarks>
+    /// Host comparison is case-insensitive because DNS names are, but is otherwise exact: Akka does
+    /// no DNS canonicalisation either, so <c>central-2</c> and <c>central-2.example.com</c> are
+    /// genuinely different seed identities to the cluster. A seed entry Akka itself cannot parse is
+    /// treated as "not this node" — reporting it is Akka's job, with Akka's wording, and this rule
+    /// must not turn a parse problem into a misleading ordering complaint.
+    /// </remarks>
+    private static bool IsSelf(string seed, string selfHost, int selfPort)
+    {
+        Address address;
+        try
+        {
+            address = Address.Parse(seed);
+        }
+        catch (Exception)
+        {
+            return false;
+        }
+
+        return address.Port == selfPort
+            && string.Equals(address.Host, selfHost, StringComparison.OrdinalIgnoreCase);
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptions.cs

@@ -0,0 +1,67 @@
+namespace ZB.MOM.WW.OtOpcUa.Cluster;
+
+/// <summary>
+///     Node-side selection of where a driver reads its deployed configuration (per-cluster mesh
+///     Phase 3). The dark switch that lets a driver node stop reading <c>Deployment.ArtifactBlob</c>
+///     from central SQL and instead fetch the artifact from central over gRPC and read it from the
+///     pair-local LocalDb cache.
+/// </summary>
+public sealed class ConfigSourceOptions
+{
+    /// <summary>Configuration section name.</summary>
+    public const string SectionName = "ConfigSource";
+
+    /// <summary>Read config directly from central SQL — today's behaviour, and the default.</summary>
+    public const string ModeDirect = "Direct";
+
+    /// <summary>Fetch the artifact from central over gRPC, cache it in LocalDb, and read from there.</summary>
+    public const string ModeFetchAndCache = "FetchAndCache";
+
+    /// <summary>
+    ///     <see cref="ModeDirect"/> (default) or <see cref="ModeFetchAndCache"/>. Any other value
+    ///     fails host start.
+    /// </summary>
+    public string Mode { get; set; } = ModeDirect;
+
+    /// <summary>
+    ///     Central artifact-gRPC base addresses, e.g. <c>http://central-1:4055</c> (h2c ⇒ the
+    ///     <c>http</c> scheme). Tried in order for failover. Required under
+    ///     <see cref="ModeFetchAndCache"/>; ignored under <see cref="ModeDirect"/>.
+    /// </summary>
+    public string[] CentralFetchEndpoints { get; set; } = [];
+
+    /// <summary>
+    ///     Shared bearer key; must equal central's <see cref="ConfigServeOptions.ApiKey"/>. Supply via
+    ///     the environment (<c>ConfigSource__ApiKey</c>) — never commit it. Required under
+    ///     <see cref="ModeFetchAndCache"/>.
+    /// </summary>
+    public string ApiKey { get; set; } = string.Empty;
+
+    /// <summary>Per-fetch deadline in seconds. Must be positive under <see cref="ModeFetchAndCache"/>.</summary>
+    public int FetchTimeoutSeconds { get; set; } = 30;
+}
+
+/// <summary>
+///     Central-side artifact-serve surface (per-cluster mesh Phase 3). Configures the dedicated h2c
+///     listener + shared bearer key the <c>DeploymentArtifactService</c> is exposed behind. Only
+///     admin-role nodes (which hold the central SQL connection) serve.
+/// </summary>
+public sealed class ConfigServeOptions
+{
+    /// <summary>Configuration section name.</summary>
+    public const string SectionName = "ConfigServe";
+
+    /// <summary>
+    ///     Dedicated HTTP/2-only (h2c) listener port for the artifact gRPC service. <c>0</c> (the
+    ///     default) disables it — nothing is bound. Must differ from the main HTTP port and from
+    ///     <c>LocalDb:SyncListenPort</c> (h2c cannot share a cleartext HTTP/1 port).
+    /// </summary>
+    public int GrpcListenPort { get; set; }
+
+    /// <summary>
+    ///     Shared bearer key the serve-side interceptor checks (constant-time, fail-closed: an unset
+    ///     key rejects every call). Supply via the environment (<c>ConfigServe__ApiKey</c>) — never
+    ///     commit it.
+    /// </summary>
+    public string ApiKey { get; set; } = string.Empty;
+}

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ConfigSourceOptionsValidator.cs

@@ -0,0 +1,83 @@
+using Microsoft.Extensions.Options;
+
+namespace ZB.MOM.WW.OtOpcUa.Cluster;
+
+/// <summary>
+///     Fails the host at startup on a <see cref="ConfigSourceOptions"/> shape that would leave a
+///     driver node unable to fetch its configuration.
+/// </summary>
+/// <remarks>
+///     Like <see cref="MeshTransportOptionsValidator"/>, every fault caught here otherwise surfaces as
+///     an <i>absence</i> — a deployment that never applies because the node has no endpoint to fetch
+///     from or no key to authenticate with. An absence has no stack trace and no failing node to point
+///     at; refusing to start is cheaper to diagnose.
+/// </remarks>
+public sealed class ConfigSourceOptionsValidator : IValidateOptions<ConfigSourceOptions>
+{
+    /// <inheritdoc />
+    public ValidateOptionsResult Validate(string? name, ConfigSourceOptions options)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+
+        var errors = new List<string>();
+
+        var isDirect = string.Equals(options.Mode, ConfigSourceOptions.ModeDirect, StringComparison.OrdinalIgnoreCase);
+        var isFetch = string.Equals(
+            options.Mode, ConfigSourceOptions.ModeFetchAndCache, StringComparison.OrdinalIgnoreCase);
+
+        if (!isDirect && !isFetch)
+        {
+            errors.Add(
+                $"{ConfigSourceOptions.SectionName}:{nameof(ConfigSourceOptions.Mode)} is "
+                + $"'{options.Mode}'. Expected '{ConfigSourceOptions.ModeDirect}' or "
+                + $"'{ConfigSourceOptions.ModeFetchAndCache}'.");
+        }
+
+        // The fetch surface is only load-bearing under FetchAndCache. Requiring it under the default
+        // would make the section mandatory everywhere for a feature that is switched off — including on
+        // admin-only nodes, which never fetch.
+        if (isFetch)
+        {
+            if (options.CentralFetchEndpoints.Length == 0)
+            {
+                errors.Add(
+                    $"{ConfigSourceOptions.SectionName}:{nameof(ConfigSourceOptions.CentralFetchEndpoints)} "
+                    + "is empty. Under FetchAndCache a driver node cannot reach central to fetch its "
+                    + "configuration, and every deployment would time out naming a node that is running "
+                    + "perfectly.");
+            }
+
+            foreach (var endpoint in options.CentralFetchEndpoints)
+            {
+                if (!endpoint.StartsWith("http://", StringComparison.OrdinalIgnoreCase)
+                    && !endpoint.StartsWith("https://", StringComparison.OrdinalIgnoreCase))
+                {
+                    errors.Add(
+                        $"Central fetch endpoint '{endpoint}' must start with 'http://' (h2c) or "
+                        + "'https://' — it is a gRPC base address (scheme://host:port), not a bare "
+                        + "host:port.");
+                }
+            }
+
+            if (string.IsNullOrEmpty(options.ApiKey))
+            {
+                errors.Add(
+                    $"{ConfigSourceOptions.SectionName}:{nameof(ConfigSourceOptions.ApiKey)} is empty. "
+                    + "Under FetchAndCache the shared bearer key is the whole authentication boundary to "
+                    + "central's artifact service; without it every fetch is rejected. Supply it via the "
+                    + "environment (ConfigSource__ApiKey).");
+            }
+
+            if (options.FetchTimeoutSeconds <= 0)
+            {
+                errors.Add(
+                    $"{ConfigSourceOptions.SectionName}:{nameof(ConfigSourceOptions.FetchTimeoutSeconds)} "
+                    + $"must be greater than 0 (was {options.FetchTimeoutSeconds}).");
+            }
+        }
+
+        return errors.Count == 0
+            ? ValidateOptionsResult.Success
+            : ValidateOptionsResult.Fail(string.Join(" ", errors));
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs

@@ -0,0 +1,61 @@
+namespace ZB.MOM.WW.OtOpcUa.Cluster;
+
+/// <summary>
+///     Selects and configures the transport carrying central→node commands and node→central
+///     deployment acks (per-cluster mesh Phase 2).
+/// </summary>
+/// <remarks>
+///     <para>
+///         <see cref="Mode"/> is a deliberate <b>dark switch</b>, not a permanent knob: both
+///         transports live in the tree for one phase so the docker-dev rig can run the same
+///         deployment over each and compare, and so a bad Phase 2 rolls back with a config change
+///         rather than a revert. Phase 6 deletes the DistributedPubSub branch along with the single
+///         mesh that made it viable.
+///     </para>
+///     <para>
+///         <b>Discovery is deliberately asymmetric</b>, matching the sister project. Central
+///         discovers nodes <i>from the database</i> — enabled, non-maintenance <c>ClusterNode</c>
+///         rows, rebuilt every <see cref="ContactRefreshSeconds"/>. Nodes discover central <i>from
+///         this section</i>: static, restart to change. The asymmetry follows the rate of change —
+///         operators add and retire nodes, but central's own address is part of the deployment.
+///     </para>
+/// </remarks>
+public sealed class MeshTransportOptions
+{
+    /// <summary>Configuration section name.</summary>
+    public const string SectionName = "MeshTransport";
+
+    /// <summary>Every command stays on DistributedPubSub — the pre-Phase-2 behaviour.</summary>
+    public const string ModeDps = "Dps";
+
+    /// <summary>Commands cross the ClusterClient receptionist boundary.</summary>
+    public const string ModeClusterClient = "ClusterClient";
+
+    /// <summary>
+    ///     <see cref="ModeDps"/> (default) or <see cref="ModeClusterClient"/>. Any other value fails
+    ///     the host at startup rather than silently falling back — a transport that is neither of
+    ///     these delivers nothing, and the symptom would be commands that vanish.
+    /// </summary>
+    public string Mode { get; set; } = ModeDps;
+
+    /// <summary>
+    ///     Akka remoting addresses of the central nodes, e.g.
+    ///     <c>akka.tcp://otopcua@central-1:4053</c>. <b>Node addresses only</b> — the
+    ///     <c>/system/receptionist</c> path is appended at construction time.
+    /// </summary>
+    /// <remarks>
+    ///     Each entry must be a <i>central</i> node's remoting endpoint, never this node's own port.
+    ///     The sister project shipped a site template whose second contact was the site's own
+    ///     remoting port; it is a permanent failure in the initial-contact rotation and stays
+    ///     invisible until a command goes missing. <see cref="MeshTransportOptionsValidator"/>
+    ///     rejects the malformed shapes at boot instead.
+    /// </remarks>
+    public string[] CentralContactPoints { get; set; } = [];
+
+    /// <summary>
+    ///     How often central re-reads <c>ClusterNode</c> rows to rebuild its contact points.
+    ///     Refreshing on a timer (rather than only on admin change) means a row edited by any
+    ///     process — migration, direct SQL, a second admin node — is picked up without a restart.
+    /// </summary>
+    public int ContactRefreshSeconds { get; set; } = 60;
+}

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptionsValidator.cs

@@ -0,0 +1,83 @@
+using Microsoft.Extensions.Options;
+
+namespace ZB.MOM.WW.OtOpcUa.Cluster;
+
+/// <summary>
+///     Fails the host at startup on a <see cref="MeshTransportOptions"/> shape that would leave
+///     commands silently undelivered.
+/// </summary>
+/// <remarks>
+///     Every fault caught here otherwise surfaces as an <i>absence</i> — a deployment that never
+///     arrives, an alarm acknowledgement that does nothing, a node whose ack is dropped without a
+///     trace. An absence has no stack trace and no failing node to point at, which makes it the
+///     most expensive class of misconfiguration to diagnose in the field. Refusing to start is
+///     cheaper for everyone.
+/// </remarks>
+public sealed class MeshTransportOptionsValidator : IValidateOptions<MeshTransportOptions>
+{
+    /// <inheritdoc />
+    public ValidateOptionsResult Validate(string? name, MeshTransportOptions options)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+
+        var errors = new List<string>();
+
+        var isDps = string.Equals(options.Mode, MeshTransportOptions.ModeDps, StringComparison.OrdinalIgnoreCase);
+        var isClusterClient = string.Equals(
+            options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
+
+        if (!isDps && !isClusterClient)
+        {
+            errors.Add(
+                $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)} is "
+                + $"'{options.Mode}'. Expected '{MeshTransportOptions.ModeDps}' or "
+                + $"'{MeshTransportOptions.ModeClusterClient}'.");
+        }
+
+        if (options.ContactRefreshSeconds <= 0)
+        {
+            errors.Add(
+                $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.ContactRefreshSeconds)} "
+                + $"must be greater than 0 (was {options.ContactRefreshSeconds}). Central rebuilds its "
+                + "ClusterClient contact points on this interval; a non-positive value would leave the "
+                + "contact set frozen at whatever it held when the node booted.");
+        }
+
+        // Contact points are only load-bearing under ClusterClient mode. Requiring them under the
+        // default would make the section mandatory everywhere for a feature that is switched off —
+        // including on admin-only nodes, which have no reason to carry them at all.
+        if (isClusterClient)
+        {
+            if (options.CentralContactPoints.Length == 0)
+            {
+                errors.Add(
+                    $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.CentralContactPoints)} "
+                    + "is empty. Under ClusterClient mode a driver node cannot reach central without at "
+                    + "least one contact point, and its deployment acks would be dropped silently — the "
+                    + "deployment would then time out naming a node that is running perfectly.");
+            }
+
+            foreach (var contactPoint in options.CentralContactPoints)
+            {
+                if (!contactPoint.StartsWith("akka.tcp://", StringComparison.Ordinal))
+                {
+                    errors.Add(
+                        $"Contact point '{contactPoint}' must start with 'akka.tcp://' — it is an Akka "
+                        + "remoting address (akka.tcp://<system>@<host>:<port>), not a bare host:port.");
+                }
+                else if (contactPoint.Contains("/user/", StringComparison.Ordinal)
+                         || contactPoint.Contains("/system/", StringComparison.Ordinal))
+                {
+                    errors.Add(
+                        $"Contact point '{contactPoint}' carries an actor-path suffix. Configure the "
+                        + "node address only — '/system/receptionist' is appended at construction time, "
+                        + "and a doubled suffix produces a path that never resolves.");
+                }
+            }
+        }
+
+        return errors.Count == 0
+            ? ValidateOptionsResult.Success
+            : ValidateOptionsResult.Fail(string.Join(" ", errors));
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs

@@ -7,7 +7,7 @@ public readonly record struct NodeHealthInputs(
     bool DbReachable,
     bool OpcUaProbeOk,
     bool Stale,
-    bool IsDriverRoleLeader);
+    bool IsDriverPrimary);
 
 /// <summary>
 /// Pure ServiceLevel computation per design §6. Output range 0–255, where higher = "more
@@ -18,7 +18,9 @@ public readonly record struct NodeHealthInputs(
 /// - Member not Up/Joining: 0 (cluster cannot trust this node).
 /// - DB reachable + OPC UA probe ok + not stale: 240 (full service).
 /// - Stale config (DB reachable or not, OPC UA probe state ignored): 100 or 200 depending on DB.
-/// - +10 bonus when this node holds the role-leader lease for the "driver" role.
+/// - +10 bonus when this node is the driver Primary (the oldest Up member with the "driver" role —
+///   where the cluster singletons live). Previously this keyed off Akka's role *leader*, the
+///   lowest-addressed member, which names a different node once any node has restarted.
 /// </summary>
 public static class ServiceLevelCalculator
 {
@@ -38,6 +40,6 @@ public static class ServiceLevelCalculator
             _ => 0,
         };
 
-        return (byte)Math.Clamp(basis + (h.IsDriverRoleLeader ? 10 : 0), 0, 255);
+        return (byte)Math.Clamp(basis + (h.IsDriverPrimary ? 10 : 0), 0, 255);
     }
 }

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf

@@ -15,6 +15,12 @@ akka {
 
     extensions = [
         "Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider, Akka.Cluster.Tools"
+
+        # Per-cluster mesh Phase 2 — the central<->node command boundary. NOT auto-started: without
+        # this entry the receptionist materialises only if some code path happens to call
+        # ClusterClientReceptionist.Get(system) first, which makes "is the boundary listening?"
+        # depend on startup ordering rather than on configuration.
+        "Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"
     ]
 
     actor {
@@ -25,6 +31,20 @@ akka {
         dot-netty.tcp {
             hostname = "0.0.0.0"
             port = 4053
+
+            # FRAME SIZE. maximum-frame-size restates Akka's own default; what actually changes is
+            # log-frame-size-exceeding, which defaults to `off`. Over the limit the transport drops
+            # that ONE message without tearing down the association — heartbeats keep flowing, the
+            # node still reports healthy, and the only symptom is a command that never arrived. The
+            # sister project lost a deploy path to exactly this and could not attribute it; their
+            # postmortem also notes the default JSON serializer escapes an already-serialized
+            # payload a second time, so ~60-70 KB of real JSON is enough to blow 128 KB.
+            #
+            # OtOpcUa's deploy notify is payload-free by design (DispatchDeployment carries only
+            # DeploymentId + RevisionHash + CorrelationId), so this threshold should stay silent.
+            # It is a canary, and a quiet one is the point.
+            maximum-frame-size = 128000b
+            log-frame-size-exceeding = 32000b
         }
         transport-failure-detector {
             heartbeat-interval = 2s
@@ -37,14 +57,26 @@ akka {
         roles = []
         min-nr-of-members = 1
 
-        # Split-brain resolver (arch-review 03/S1). This block IS active: Akka.Cluster.Hosting's
-        # WithClustering enables an SBR downing provider by default (it applies
-        # SplitBrainResolverOption.Default when ClusterOptions.SplitBrainResolver is null), and that
-        # provider reads this block. ServiceCollectionExtensions.BuildClusterOptions additionally sets
-        # the typed KeepOldestOption { DownIfAlone = true } to make the strategy EXPLICIT in code —
-        # its active-strategy + keep-oldest.down-if-alone MUST match this block. (The cluster is NOT
-        # NoDowning; only an explicit downing-provider-class = "" would disable failover.)
-        # stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin.
+        # DOWNING. The provider is NOT selected here — this file is added with HoconAddMode.Append,
+        # which loses to what Akka.Hosting's WithClustering emits, so a downing-provider-class set
+        # here would be silently overridden. Selection lives in
+        # ServiceCollectionExtensions.BuildDowningHocon (Prepended, which outranks WithClustering),
+        # driven by Cluster:SplitBrainResolverStrategy.
+        #
+        # The DEFAULT strategy is "auto-down", under which this whole block is inert: Akka's
+        # AutoDowning provider is installed with auto-down-unreachable-after = 15s, and the leader
+        # among the reachable members downs the unreachable peer — so a hard crash of EITHER node,
+        # oldest included, fails over. The trade is dual-active during a real network partition.
+        #
+        # The block below applies only when an operator selects "keep-oldest". Note that keep-oldest
+        # is NOT safe for a two-node pair: Akka.NET's KeepOldest.OldestDecision only lets
+        # down-if-alone rescue a side holding >= 2 members, so a 1-vs-1 survivor downs ITSELF and
+        # (with run-coordinated-shutdown-when-down) exits — a crash of the oldest node becomes a
+        # total outage. See docs/Redundancy.md.
+        #
+        # stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin, and
+        # must equal ServiceCollectionExtensions.DowningStableAfter so both strategies fail over at
+        # the same latency. Both constraints are pinned by SplitBrainResolverActivationTests.
         split-brain-resolver {
             active-strategy = "keep-oldest"
             stable-after = 15s
@@ -62,6 +94,31 @@ akka {
         down-removal-margin = 15s
         run-coordinated-shutdown-when-down = on
 
+        # CLUSTERCLIENT — the central<->node command boundary (per-cluster mesh Phase 2).
+        client {
+            # DELIBERATE, and NOT what the sister project runs: they never wrote this section, so
+            # they inherit buffer-size = 1000. Zero means "if the receptionist is unknown, drop the
+            # message now". Buffering would replay a DispatchDeployment on reconnect and apply a
+            # deployment ConfigPublishCoordinator had already sealed as TimedOut — the node would
+            # then be running a configuration the database says failed. Fail-fast keeps the DB the
+            # single record of truth, and reproduces today's DPS behaviour exactly: a node that is
+            # down misses the publish and self-heals from the DB on restart.
+            buffer-size = 0
+
+            # Retry forever rather than self-stopping the client. Restates Akka's default; stated
+            # explicitly because the behaviour is load-bearing — a central node down for an hour
+            # must not require a driver-node restart to become reachable again.
+            reconnect-timeout = off
+
+            receptionist {
+                # Empty = every member hosts a receptionist. This is what makes the comm actors
+                # "registered per node, not as a singleton" work: contact rotation reaches whichever
+                # node answers. Restates Akka's default; do NOT scope this to a role, or the
+                # boundary becomes reachable through one node only and rotation silently fails.
+                role = ""
+            }
+        }
+
         singleton {
             singleton-name = "singleton"
         }

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs

@@ -7,6 +7,7 @@ using Akka.Remote.Hosting;
 using Microsoft.Extensions.Configuration;
 using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.Options;
+using ZB.MOM.WW.Configuration;
 using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
 
 namespace ZB.MOM.WW.OtOpcUa.Cluster;
@@ -19,13 +20,33 @@ public static class ServiceCollectionExtensions
     /// configurator via <see cref="WithOtOpcUaClusterBootstrap"/> — keeping the entire Akka graph
     /// under Akka.Hosting's management so cluster singletons land on the same ActorSystem.
     /// </summary>
+    /// <remarks>
+    /// The binding is validated at startup (<c>ValidateOnStart</c>) by
+    /// <see cref="AkkaClusterOptionsValidator"/>, so a seed list that cannot bootstrap this node —
+    /// self listed behind its partner — fails the host loudly instead of leaving it running but
+    /// permanently outside the cluster.
+    /// </remarks>
     /// <param name="services">The service collection to configure.</param>
     /// <param name="configuration">The application configuration containing cluster options.</param>
     /// <returns>The same service collection, for chaining.</returns>
     public static IServiceCollection AddOtOpcUaCluster(this IServiceCollection services, IConfiguration configuration)
     {
-        services.AddOptions<AkkaClusterOptions>()
-            .Bind(configuration.GetSection(AkkaClusterOptions.SectionName));
+        services.AddValidatedOptions<AkkaClusterOptions, AkkaClusterOptionsValidator>(
+            configuration, AkkaClusterOptions.SectionName);
+
+        // Per-cluster mesh Phase 2: which transport carries central→node commands. Validated at
+        // startup for the same reason the cluster options are — a contact point that cannot resolve
+        // produces silence, not an error, and silence is indistinguishable from "nothing to do".
+        services.AddValidatedOptions<MeshTransportOptions, MeshTransportOptionsValidator>(
+            configuration, MeshTransportOptions.SectionName);
+
+        // Per-cluster mesh Phase 3: where a driver reads its config (Direct SQL, or fetch-and-cache
+        // over gRPC). Validated at startup for the same reason — a FetchAndCache node with no endpoint
+        // or no key produces silence, not an error. ConfigServe (the central serve surface) needs no
+        // cross-field validation: a 0 port is a legitimate "disabled", so a plain Configure suffices.
+        services.AddValidatedOptions<ConfigSourceOptions, ConfigSourceOptionsValidator>(
+            configuration, ConfigSourceOptions.SectionName);
+        services.Configure<ConfigServeOptions>(configuration.GetSection(ConfigServeOptions.SectionName));
 
         services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
 
@@ -80,40 +101,133 @@ public static class ServiceCollectionExtensions
 
         builder.WithClustering(BuildClusterOptions(options));
 
+        // Must come AFTER WithClustering, which always emits a downing-provider-class of its own:
+        // Akka.Cluster.Hosting applies SplitBrainResolverOption.Default whenever the typed
+        // ClusterOptions.SplitBrainResolver is null, so there is always a competing value to beat.
+        // Prepend is the highest-precedence mode, which makes this block win irrespective of how
+        // many fragments are added or in what order.
+        //
+        // Do not take this comment's word for it. Which fragment actually wins is not obvious from
+        // the mode names — measured, Append also wins here, purely because it happens to be added
+        // last. SplitBrainResolverActivationTests reads the provider back off a *running*
+        // ActorSystem for exactly that reason: the effective value is the only one worth asserting.
+        builder.AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend);
+
+        // NOTE (2026-07-22): no bootstrap watchdog is armed here any more. The cold-start-alone gap
+        // it existed for — Akka lets only seed-nodes[0] form a NEW cluster — is now closed by
+        // self-first seed ordering (AkkaClusterOptions.SeedNodes), enforced at boot by
+        // AkkaClusterOptionsValidator. The retired ClusterBootstrapFallback timer sat OUTSIDE Akka's
+        // join handshake, so it could not distinguish "no seed answered" from "a seed answered and
+        // the join is in flight", and Cluster.Join(SelfAddress) is not ignored mid-handshake — it
+        // wins. See docs/Redundancy.md → "Bootstrap: self-first seed ordering".
+
         return builder;
     }
 
     /// <summary>
-    /// Builds the <see cref="ClusterOptions"/> for the fused-host cluster, setting the split-brain-resolver
-    /// strategy <b>explicitly in code</b>.
-    ///
-    /// <b>Activation note (arch-review 03/S1, corrected):</b> Akka.Cluster.Hosting's <c>WithClustering</c>
-    /// already enables an SBR downing provider <b>by default</b> — when <see cref="ClusterOptions.SplitBrainResolver"/>
-    /// is <c>null</c> it applies <c>SplitBrainResolverOption.Default</c>, which registers
-    /// <c>Akka.Cluster.SBR.SplitBrainResolverProvider</c> and reads the <c>split-brain-resolver</c> HOCON block in
-    /// <c>Resources/akka.conf</c>. So the cluster was <b>not</b> running <c>NoDowning</c> before this option was
-    /// set: the pre-existing akka.conf <c>keep-oldest</c> block was already active, and hard-crashed nodes already
-    /// failed over. Setting this typed option makes the strategy <b>explicit in code</b> (independent of the
-    /// framework default) rather than being the sole activator — it is reinforcing/belt-and-suspenders and
-    /// produces the same effective behavior. (Only an <i>explicit</i> <c>NoDowning</c>, e.g.
-    /// <c>downing-provider-class = ""</c>, would disable failover.)
-    ///
-    /// <see cref="KeepOldestOption"/> with <c>DownIfAlone=true</c> mirrors the HOCON intent and is the
-    /// correct strategy for a 2-node warm-redundancy pair: on an even split the oldest member (typically
-    /// the long-running primary) survives, and <c>down-if-alone</c> downs a singleton that loses its
-    /// peer. <c>keep-majority</c>/<c>static-quorum</c> are wrong for two nodes. The strategy + down-if-alone
-    /// here MUST stay consistent with the HOCON block; <c>stable-after</c> lives only in HOCON because the
-    /// typed option cannot express it (it must stay ≥ <c>failure-detector.acceptable-heartbeat-pause</c>).
+    /// How long a member must stay unreachable before it is downed. Shared by both strategies
+    /// (<c>auto-down-unreachable-after</c> and the SBR resolver's <c>stable-after</c>) so the two
+    /// have the same failover latency. Must stay above
+    /// <c>akka.cluster.failure-detector.acceptable-heartbeat-pause</c> or a merely-slow node gets
+    /// downed; both constraints are pinned by tests.
     /// </summary>
-    /// <param name="options">The bound cluster options carrying seed nodes and roles.</param>
-    /// <returns>The cluster options with seed nodes, roles, and the explicit split-brain-resolver strategy.</returns>
+    public static readonly TimeSpan DowningStableAfter = TimeSpan.FromSeconds(15);
+
+    /// <summary>
+    /// Builds the HOCON that selects the downing provider, per
+    /// <see cref="AkkaClusterOptions.SplitBrainResolverStrategy"/>.
+    /// </summary>
+    /// <param name="options">The bound cluster options carrying the strategy.</param>
+    /// <returns>
+    /// For <c>auto-down</c>, a block installing Akka's <c>AutoDowning</c> provider with a downing
+    /// window of <see cref="DowningStableAfter"/>. For <c>keep-oldest</c>, an empty string — the
+    /// typed <see cref="KeepOldestOption"/> from <see cref="BuildClusterOptions"/> already installs
+    /// the SBR provider, and the resolver's own settings live in <c>Resources/akka.conf</c>.
+    /// </returns>
+    /// <exception cref="ArgumentOutOfRangeException">The strategy is not a recognised value.</exception>
+    public static string BuildDowningHocon(AkkaClusterOptions options)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+
+        if (IsAutoDown(options))
+        {
+            // auto-down-unreachable-after is load-bearing: it defaults to `off`, under which
+            // AutoDowning is installed but never downs anything — failover silently disabled.
+            return $$"""
+                akka.cluster {
+                    downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"
+                    auto-down-unreachable-after = {{(int)DowningStableAfter.TotalMilliseconds}}ms
+                }
+                """;
+        }
+
+        if (IsKeepOldest(options))
+        {
+            return string.Empty;
+        }
+
+        throw new ArgumentOutOfRangeException(
+            nameof(options),
+            options.SplitBrainResolverStrategy,
+            $"Unknown {AkkaClusterOptions.SectionName}:{nameof(AkkaClusterOptions.SplitBrainResolverStrategy)}. "
+            + "Expected 'auto-down' (default; survives a crash of either node) or 'keep-oldest' "
+            + "(partition-safe, but a two-node pair cannot survive a crash of the oldest node).");
+    }
+
+    private static bool IsAutoDown(AkkaClusterOptions options) =>
+        string.Equals(options.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase);
+
+    private static bool IsKeepOldest(AkkaClusterOptions options) =>
+        string.Equals(options.SplitBrainResolverStrategy, "keep-oldest", StringComparison.OrdinalIgnoreCase);
+
+    /// <summary>
+    /// Builds the <see cref="ClusterOptions"/> for the fused-host cluster.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b>Activation note (arch-review 03/S1, corrected twice).</b> Akka.Cluster.Hosting's
+    ///         <c>WithClustering</c> always installs a downing provider: when
+    ///         <see cref="ClusterOptions.SplitBrainResolver"/> is <c>null</c> it applies
+    ///         <c>SplitBrainResolverOption.Default</c>, which registers
+    ///         <c>Akka.Cluster.SBR.SplitBrainResolverProvider</c> and reads the
+    ///         <c>split-brain-resolver</c> block in <c>Resources/akka.conf</c>. The cluster is
+    ///         therefore never <c>NoDowning</c>. What this method chooses is only <i>which</i>
+    ///         provider — and under the default <c>auto-down</c> strategy the choice is not made
+    ///         here at all but by the Prepended HOCON in
+    ///         <see cref="WithOtOpcUaClusterBootstrap"/>, which outranks whatever
+    ///         <c>WithClustering</c> emits.
+    ///     </para>
+    ///     <para>
+    ///         <b>Why keep-oldest is no longer the default (2026-07-21).</b> The previous
+    ///         documentation here asserted that <see cref="KeepOldestOption"/> with
+    ///         <c>DownIfAlone=true</c> was "the correct strategy for a 2-node warm-redundancy pair"
+    ///         because <c>down-if-alone</c> would down a node that lost its peer. That is the
+    ///         opposite of what Akka.NET does. In <c>KeepOldest.OldestDecision</c> the
+    ///         <c>down-if-alone</c> branch requires the surviving side to hold &gt;= 2 members, so
+    ///         in a 1-vs-1 split the survivor falls through to <c>DownReachable</c> and downs
+    ///         <i>itself</i>; with <c>run-coordinated-shutdown-when-down = on</c> it then exits. A
+    ///         two-node pair running keep-oldest converts a crash of the oldest node into a total
+    ///         outage — precisely the failure redundancy exists to absorb. Live-proven on the sister
+    ///         project's rig; see <c>docs/Redundancy.md</c> and
+    ///         <see cref="AkkaClusterOptions.SplitBrainResolverStrategy"/> for the trade.
+    ///     </para>
+    /// </remarks>
+    /// <param name="options">The bound cluster options carrying seed nodes, roles and the strategy.</param>
+    /// <returns>
+    /// The cluster options with seed nodes and roles. <see cref="ClusterOptions.SplitBrainResolver"/>
+    /// is set only under <c>keep-oldest</c>, where the SBR provider is the one wanted; under
+    /// <c>auto-down</c> it is left null because the Prepended HOCON replaces the provider anyway,
+    /// and naming a resolver that is not in force would be misleading.
+    /// </returns>
     public static ClusterOptions BuildClusterOptions(AkkaClusterOptions options)
     {
+        ArgumentNullException.ThrowIfNull(options);
+
         return new ClusterOptions
         {
             SeedNodes = options.SeedNodes,
             Roles = options.Roles,
-            SplitBrainResolver = new KeepOldestOption { DownIfAlone = true },
+            SplitBrainResolver = IsKeepOldest(options) ? new KeepOldestOption { DownIfAlone = true } : null,
         };
     }
 }

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ZB.MOM.WW.OtOpcUa.Cluster.csproj

@@ -14,6 +14,10 @@
         <PackageReference Include="Akka.Remote.Hosting"/>
         <PackageReference Include="Microsoft.Extensions.Hosting"/>
         <PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions"/>
+        <!-- Shared options-validation primitives (OptionsValidatorBase / AddValidatedOptions), the
+             same seam the Host's Ldap/OpcUa/Historian validators use. Pulled in here because the
+             self-first seed-ordering rule belongs with the options it validates. -->
+        <PackageReference Include="ZB.MOM.WW.Configuration"/>
     </ItemGroup>
 
     <ItemGroup>

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Interfaces/NodeDiagnosticsSnapshot.cs

@@ -14,8 +14,16 @@ public sealed record DriverInstanceDiagnostics(
 /// Per-node diagnostics returned by <c>IFleetDiagnosticsClient</c>. Populated by the node's
 /// local <c>DriverHostActor</c> via a request/response over Akka.
 /// </summary>
+/// <param name="RunningFromCache">
+///     True when the node booted its configuration from the node-local artifact cache because the
+///     central ConfigDb was unreachable. Such a node looks entirely healthy — full address space,
+///     live values — but its configuration is frozen and no deployment can reach it, so the state
+///     needs to be explicitly visible rather than inferred. Defaults to false, which is also what
+///     every node that booted normally reports.
+/// </param>
 public sealed record NodeDiagnosticsSnapshot(
     NodeId NodeId,
     RevisionHash? CurrentRevision,
     IReadOnlyList<DriverInstanceDiagnostics> Drivers,
-    DateTime AsOfUtc);
+    DateTime AsOfUtc,
+    bool RunningFromCache = false);

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Mesh/MeshCommand.cs

@@ -0,0 +1,28 @@
+namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
+
+/// <summary>
+///     A central→node command, plus the DistributedPubSub topic it would have been published on.
+/// </summary>
+/// <param name="Topic">
+///     The legacy DPS topic. Carried so the router can publish under
+///     <c>MeshTransport:Mode = Dps</c> without a second dispatch table; ignored entirely under
+///     <c>ClusterClient</c> mode, where the destination is an actor path rather than a topic.
+/// </param>
+/// <param name="Message">
+///     The command itself, never wrapped. Node-side handlers see exactly the type they see today,
+///     which is what lets the dark switch touch publishers only.
+/// </param>
+/// <remarks>
+///     <para>
+///         This envelope exists so the admin singletons stop knowing which transport is in force.
+///         Before per-cluster mesh Phase 2 each publisher held its own
+///         <c>DistributedPubSub.Get(Context.System).Mediator</c> call, so the switch would have had
+///         to be threaded through five call sites across two actors, and each would have been free
+///         to drift.
+///     </para>
+///     <para>
+///         Central-side only — it never crosses the wire, so it carries no serialization contract
+///         and needs no additive-evolution discipline.
+///     </para>
+/// </remarks>
+public sealed record MeshCommand(string Topic, object Message);

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Mesh/MeshPaths.cs

@@ -0,0 +1,33 @@
+namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
+
+/// <summary>
+///     Actor paths the two mesh comm actors register under. <b>These strings are the wire
+///     contract</b> of the central↔node boundary.
+/// </summary>
+/// <remarks>
+///     <para>
+///         Declared once, here, rather than as literals on each side. The <c>deployments</c> /
+///         <c>deployment-acks</c> topic names are declared twice — once in
+///         <c>ConfigPublishCoordinator</c> and again in <c>DriverHostActor</c> — and agree only by
+///         convention; renaming one would compile, deploy, and deliver nothing. A ClusterClient path
+///         has the same failure mode and no compiler to catch it, so the constant is shared.
+///     </para>
+///     <para>
+///         Nothing else pins these values, which is why <c>MeshCommActorPathTests</c> boots a real
+///         host and <c>Identify</c>-probes both paths.
+///     </para>
+/// </remarks>
+public static class MeshPaths
+{
+    /// <summary>Central-side comm actor — where a driver node sends its deployment acks.</summary>
+    public const string CentralCommunication = "/user/central-communication";
+
+    /// <summary>Node-side comm actor — where central sends commands.</summary>
+    public const string NodeCommunication = "/user/node-communication";
+
+    /// <summary>Actor name of <see cref="CentralCommunication"/> (the path's final segment).</summary>
+    public const string CentralCommunicationName = "central-communication";
+
+    /// <summary>Actor name of <see cref="NodeCommunication"/> (the path's final segment).</summary>
+    public const string NodeCommunicationName = "node-communication";
+}

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Redundancy/NodeRedundancyState.cs

@@ -8,9 +8,20 @@ public enum RedundancyRole { Primary, Secondary, Detached }
 /// Snapshot of a single node's redundancy state. Aggregated by <c>RedundancyStateActor</c>
 /// to compute fleet-wide ServiceLevel.
 /// </summary>
+/// <param name="NodeId">Canonical <c>host:port</c> id of the node this entry describes.</param>
+/// <param name="Role">The node's redundancy role in the current snapshot.</param>
+/// <param name="IsClusterLeader">Whether the node is the Akka cluster leader.</param>
+/// <param name="IsDriverPrimary">
+/// Whether the node is the driver Primary — the <b>oldest</b> Up member carrying the <c>driver</c>
+/// role, which is where <c>ClusterSingletonManager</c> places singletons. Renamed from
+/// <c>IsRoleLeaderForDriver</c>: it was derived from <c>ClusterState.RoleLeader("driver")</c>, the
+/// lowest-<i>addressed</i> member, which diverges from the oldest after any node restart. The name
+/// now describes what the value means rather than how it used to be computed.
+/// </param>
+/// <param name="AsOfUtc">When the snapshot was computed.</param>
 public sealed record NodeRedundancyState(
     NodeId NodeId,
     RedundancyRole Role,
     bool IsClusterLeader,
-    bool IsRoleLeaderForDriver,
+    bool IsDriverPrimary,
     DateTime AsOfUtc);

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/deployment_artifact.proto

@@ -0,0 +1,26 @@
+syntax = "proto3";
+
+package deployment_artifact.v1;
+
+option csharp_namespace = "ZB.MOM.WW.OtOpcUa.Commons.Protos.DeploymentArtifact.V1";
+
+// Central serves the deployed-configuration artifact bytes to a driver node (per-cluster mesh
+// Phase 3). Only admin-role nodes — which hold the central SQL connection — serve; driver nodes in
+// FetchAndCache mode are the clients. The bytes travel here, out-of-band, precisely so they never
+// ride an Akka message and hit the frame-size limit.
+service DeploymentArtifactService {
+  // Streams the artifact for a sealed deployment as ordered chunks. NotFound when the id is
+  // unknown, the deployment is not sealed, or the blob is empty (existence-hiding + the #485
+  // "empty bytes are never an empty config" rule). The client verifies
+  // SHA-256(reassembled) == the RevisionHash it already holds from DispatchDeployment, so no hash
+  // is carried here.
+  rpc Fetch(FetchRequest) returns (stream ArtifactChunk);
+}
+
+message FetchRequest {
+  string deployment_id = 1;
+}
+
+message ArtifactChunk {
+  bytes data = 1;
+}

src/Core/ZB.MOM.WW.OtOpcUa.Commons/ZB.MOM.WW.OtOpcUa.Commons.csproj

@@ -11,6 +11,24 @@
         <PackageReference Include="ZB.MOM.WW.Audit"/>
     </ItemGroup>
 
+    <!-- Per-cluster mesh Phase 3: the first locally-compiled proto in the repo (all other gRPC here
+         is packaged clients). Grpc.Tools generates both the server base and the client stub from
+         deployment_artifact.proto; Google.Protobuf + Grpc.Core.Api carry the generated code's runtime
+         types. Placed in Commons so both the central server (Host/AdminUI) and the node client
+         (Runtime) get the types from one reference. -->
+    <ItemGroup>
+        <PackageReference Include="Google.Protobuf"/>
+        <PackageReference Include="Grpc.Core.Api"/>
+        <PackageReference Include="Grpc.Tools">
+            <PrivateAssets>all</PrivateAssets>
+            <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
+        </PackageReference>
+    </ItemGroup>
+
+    <ItemGroup>
+        <Protobuf Include="Protos\deployment_artifact.proto" GrpcServices="Both"/>
+    </ItemGroup>
+
     <ItemGroup>
         <ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Core.Abstractions\ZB.MOM.WW.OtOpcUa.Core.Abstractions.csproj"/>
     </ItemGroup>

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/ClusterNode.cs

@@ -18,6 +18,33 @@ public sealed class ClusterNode
     /// <summary>The dashboard HTTP port (default 8081).</summary>
     public int DashboardPort { get; set; } = 8081;
 
+    /// <summary>
+    ///     Akka remoting port (default 4053). <b>This is central's dial target, not the node's own
+    ///     binding configuration</b> — the node binds from <c>Cluster:Port</c> in its own appsettings,
+    ///     and this column duplicates that value for a reader that cannot see the node's config.
+    ///     Per-cluster mesh Phase 2 builds its ClusterClient contact points from
+    ///     <see cref="Host"/> + this port, at which point central and the site node no longer share a
+    ///     gossip ring and central has no other way to learn it.
+    /// </summary>
+    /// <remarks>
+    ///     The duplication is real and unenforced by the schema. <c>ClusterNodeAddressReconciler</c>
+    ///     on the admin node compares this row against observed cluster membership and logs an Error
+    ///     on mismatch — a node that binds 4054 while its row says 4053 is unreachable from central in
+    ///     Phase 2, and the symptom there is a silent absence of acks rather than an error.
+    /// </remarks>
+    public int AkkaPort { get; set; } = 4053;
+
+    /// <summary>
+    ///     gRPC port central dials for the Phase 5 telemetry stream, or <see langword="null"/> when the
+    ///     node exposes none. Also central's dial target rather than the node's binding config.
+    /// </summary>
+    /// <remarks>
+    ///     Nullable by intent: nothing listens on this port yet, and a non-null default would assert a
+    ///     port that does not exist. Phase 5 populates it; until then <see langword="null"/> is the
+    ///     honest value.
+    /// </remarks>
+    public int? GrpcPort { get; set; }
+
     /// <summary>
     ///     OPC UA <c>ApplicationUri</c> — MUST be unique per node per OPC UA spec. Clients pin trust here.
     ///     Fleet-wide unique index enforces no two nodes share a value.
@@ -36,8 +63,37 @@ public sealed class ClusterNode
     public string? DriverConfigOverridesJson { get; set; }
 
     /// <summary>Gets or sets a value indicating whether this node is enabled.</summary>
+    /// <remarks>
+    ///     <b>Part of the declared topology.</b> <c>DraftValidator.ValidateClusterTopology</c> requires
+    ///     the enabled-node count to equal <see cref="ServerCluster.NodeCount"/>, so disabling one node
+    ///     of a Warm/Hot pair is a deploy-blocking validation error by design — it would boot the
+    ///     runtime into the InvalidTopology band. To take a node out of service temporarily, use
+    ///     <see cref="MaintenanceMode"/> instead.
+    /// </remarks>
     public bool Enabled { get; set; } = true;
 
+    /// <summary>
+    ///     Node is temporarily out of service: still part of the cluster's declared topology, but not
+    ///     expected to participate. A deployment does not wait for its ack, and the address reconciler
+    ///     does not warn that it is absent.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         Separate from <see cref="Enabled"/> because the two answer different questions.
+    ///         <see cref="Enabled"/> means "this node is part of the declared topology" and is checked
+    ///         against <see cref="ServerCluster.NodeCount"/>; this flag means "do not expect it right
+    ///         now". Per-cluster mesh Phase 1 made an enabled row's ack mandatory, so without this a
+    ///         node down for maintenance would block every deployment to its cluster — and clearing
+    ///         <see cref="Enabled"/> to escape that is itself rejected by the topology gate on any
+    ///         2-node pair, which is every cluster in the target deployment.
+    ///     </para>
+    ///     <para>
+    ///         Found by the Phase 1 live gate: the two rules were independently reasonable and
+    ///         contradictory together.
+    ///     </para>
+    /// </remarks>
+    public bool MaintenanceMode { get; set; }
+
     /// <summary>Gets or sets the timestamp when this node was last seen.</summary>
     public DateTime? LastSeenAt { get; set; }
 

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/GenerationSealedCache.cs

@@ -1,197 +0,0 @@
-using LiteDB;
-
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Generation-sealed LiteDB cache per <c>docs/v2/plan.md</c> and Phase 6.1
-///     Stream D.1. Each published generation writes one <b>read-only</b> LiteDB file under
-///     <c>&lt;cache-root&gt;/&lt;clusterId&gt;/&lt;generationId&gt;.db</c>. A per-cluster
-///     <c>CURRENT</c> text file holds the currently-active generation id; it is updated
-///     atomically (temp file + <see cref="File.Replace(string, string, string?)"/>) only after
-///     the sealed file is fully written.
-/// </summary>
-/// <remarks>
-///     <para>Mixed-generation reads are impossible: any read opens the single file pointed to
-///     by <c>CURRENT</c>, which is a coherent snapshot. Corruption of the CURRENT file or the
-///     sealed file surfaces as <see cref="GenerationCacheUnavailableException"/> — the reader
-///     fails closed rather than silently falling back to an older generation. Recovery path
-///     is to re-fetch from the central DB (and the Phase 6.1 Stream C <c>UsingStaleConfig</c>
-///     flag goes true until that succeeds).</para>
-///
-///     <para>This cache is the read-path fallback when the central DB is unreachable. The
-///     write path (draft edits, publish) bypasses the cache and fails hard on DB outage per
-///     Stream D.2 — inconsistent writes are worse than a temporary inability to edit.</para>
-/// </remarks>
-public sealed class GenerationSealedCache
-{
-    private const string CollectionName = "generation";
-    private const string CurrentPointerFileName = "CURRENT";
-    private readonly string _cacheRoot;
-
-    // Private per-database BsonMapper with the entity pre-registered. LiteDB's default
-    // BsonMapper.Global is a process-wide singleton whose lazy per-type member registration is
-    // not thread-safe across concurrently-constructed LiteDatabase instances; a seal racing a
-    // read (or this cache racing LiteDbConfigCache) corrupts the global mapper, surfacing as
-    // "Member … not found on BsonMapper" or a bogus duplicate-_id insert.
-    private static BsonMapper BuildMapper()
-    {
-        var mapper = new BsonMapper();
-        mapper.Entity<GenerationSnapshot>();
-        return mapper;
-    }
-
-    /// <summary>Root directory for all clusters' sealed caches.</summary>
-    public string CacheRoot => _cacheRoot;
-
-    /// <summary>Initializes a new instance of the GenerationSealedCache class.</summary>
-    /// <param name="cacheRoot">The root directory for the cache.</param>
-    public GenerationSealedCache(string cacheRoot)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(cacheRoot);
-        _cacheRoot = cacheRoot;
-        Directory.CreateDirectory(_cacheRoot);
-    }
-
-    /// <summary>
-    ///     Seal a generation: write the snapshot to <c>&lt;cluster&gt;/&lt;generationId&gt;.db</c>,
-    ///     mark the file read-only, then atomically publish the <c>CURRENT</c> pointer. Existing
-    ///     sealed files for prior generations are preserved (prune separately).
-    /// </summary>
-    /// <param name="snapshot">The generation snapshot to seal.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task representing the asynchronous operation.</returns>
-    public async Task SealAsync(GenerationSnapshot snapshot, CancellationToken ct = default)
-    {
-        ArgumentNullException.ThrowIfNull(snapshot);
-        ct.ThrowIfCancellationRequested();
-
-        var clusterDir = Path.Combine(_cacheRoot, snapshot.ClusterId);
-        Directory.CreateDirectory(clusterDir);
-        var sealedPath = Path.Combine(clusterDir, $"{snapshot.GenerationId}.db");
-
-        if (File.Exists(sealedPath))
-        {
-            // Already sealed — idempotent. Treat as no-op + update pointer in case an earlier
-            // seal succeeded but the pointer update failed (crash recovery).
-            WritePointerAtomically(clusterDir, snapshot.GenerationId);
-            return;
-        }
-
-        var tmpPath = sealedPath + ".tmp";
-        try
-        {
-            using (var db = new LiteDatabase(new ConnectionString { Filename = tmpPath, Upgrade = false }, BuildMapper()))
-            {
-                var col = db.GetCollection<GenerationSnapshot>(CollectionName);
-                col.Insert(snapshot);
-            }
-
-            File.Move(tmpPath, sealedPath);
-            File.SetAttributes(sealedPath, File.GetAttributes(sealedPath) | FileAttributes.ReadOnly);
-            WritePointerAtomically(clusterDir, snapshot.GenerationId);
-        }
-        catch
-        {
-            try { if (File.Exists(tmpPath)) File.Delete(tmpPath); } catch { /* best-effort */ }
-            throw;
-        }
-
-        await Task.CompletedTask;
-    }
-
-    /// <summary>
-    ///     Read the current sealed snapshot for <paramref name="clusterId"/>. Throws
-    ///     <see cref="GenerationCacheUnavailableException"/> when the pointer is missing
-    ///     (first-boot-no-snapshot case) or when the sealed file is corrupt. Never silently
-    ///     falls back to a prior generation.
-    /// </summary>
-    /// <param name="clusterId">The cluster ID to read the snapshot for.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task representing the asynchronous operation containing the generation snapshot.</returns>
-    public Task<GenerationSnapshot> ReadCurrentAsync(string clusterId, CancellationToken ct = default)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
-        ct.ThrowIfCancellationRequested();
-
-        var clusterDir = Path.Combine(_cacheRoot, clusterId);
-        var pointerPath = Path.Combine(clusterDir, CurrentPointerFileName);
-        if (!File.Exists(pointerPath))
-            throw new GenerationCacheUnavailableException(
-                $"No sealed generation for cluster '{clusterId}' at '{clusterDir}'. First-boot case: the central DB must be reachable at least once before cache fallback is possible.");
-
-        long generationId;
-        try
-        {
-            var text = File.ReadAllText(pointerPath).Trim();
-            generationId = long.Parse(text, System.Globalization.CultureInfo.InvariantCulture);
-        }
-        catch (Exception ex)
-        {
-            throw new GenerationCacheUnavailableException(
-                $"CURRENT pointer at '{pointerPath}' is corrupt or unreadable.", ex);
-        }
-
-        var sealedPath = Path.Combine(clusterDir, $"{generationId}.db");
-        if (!File.Exists(sealedPath))
-            throw new GenerationCacheUnavailableException(
-                $"CURRENT points at generation {generationId} but '{sealedPath}' is missing — fails closed rather than serving an older generation.");
-
-        try
-        {
-            using var db = new LiteDatabase(new ConnectionString { Filename = sealedPath, ReadOnly = true }, BuildMapper());
-            var col = db.GetCollection<GenerationSnapshot>(CollectionName);
-            var snapshot = col.FindAll().FirstOrDefault()
-                ?? throw new GenerationCacheUnavailableException(
-                    $"Sealed file '{sealedPath}' contains no snapshot row — file is corrupt.");
-            return Task.FromResult(snapshot);
-        }
-        catch (GenerationCacheUnavailableException) { throw; }
-        catch (Exception ex) when (ex is LiteException or InvalidDataException or IOException
-                                      or NotSupportedException or FormatException)
-        {
-            throw new GenerationCacheUnavailableException(
-                $"Sealed file '{sealedPath}' is corrupt or unreadable — fails closed rather than falling back to an older generation.", ex);
-        }
-    }
-
-    /// <summary>Return the generation id the <c>CURRENT</c> pointer points at, or null if no pointer exists.</summary>
-    /// <param name="clusterId">The cluster ID to get the current generation ID for.</param>
-    /// <returns>The generation ID, or null if no pointer exists.</returns>
-    public long? TryGetCurrentGenerationId(string clusterId)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
-        var pointerPath = Path.Combine(_cacheRoot, clusterId, CurrentPointerFileName);
-        if (!File.Exists(pointerPath)) return null;
-        try
-        {
-            return long.Parse(File.ReadAllText(pointerPath).Trim(), System.Globalization.CultureInfo.InvariantCulture);
-        }
-        catch
-        {
-            return null;
-        }
-    }
-
-    private static void WritePointerAtomically(string clusterDir, long generationId)
-    {
-        var pointerPath = Path.Combine(clusterDir, CurrentPointerFileName);
-        var tmpPath = pointerPath + ".tmp";
-        File.WriteAllText(tmpPath, generationId.ToString(System.Globalization.CultureInfo.InvariantCulture));
-        if (File.Exists(pointerPath))
-            File.Replace(tmpPath, pointerPath, destinationBackupFileName: null);
-        else
-            File.Move(tmpPath, pointerPath);
-    }
-}
-
-/// <summary>Sealed cache is unreachable — caller must fail closed.</summary>
-public sealed class GenerationCacheUnavailableException : Exception
-{
-    /// <summary>Initializes a new instance of the GenerationCacheUnavailableException class.</summary>
-    /// <param name="message">The error message.</param>
-    public GenerationCacheUnavailableException(string message) : base(message) { }
-    /// <summary>Initializes a new instance of the GenerationCacheUnavailableException class with an inner exception.</summary>
-    /// <param name="message">The error message.</param>
-    /// <param name="inner">The inner exception.</param>
-    public GenerationCacheUnavailableException(string message, Exception inner) : base(message, inner) { }
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/GenerationSnapshot.cs

@@ -1,20 +0,0 @@
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     A self-contained snapshot of one generation — enough to rebuild the address space on a node
-///     that has lost DB connectivity. The payload is the JSON-serialized <c>sp_GetGenerationContent</c>
-///     result; the local cache doesn't inspect the shape, it just round-trips bytes.
-/// </summary>
-public sealed class GenerationSnapshot
-{
-    /// <summary>Gets or sets the auto-generated LiteDB ID.</summary>
-    public int Id { get; set; } // LiteDB auto-ID
-    /// <summary>Gets or sets the cluster identifier.</summary>
-    public required string ClusterId { get; set; }
-    /// <summary>Gets or sets the generation identifier.</summary>
-    public required long GenerationId { get; set; }
-    /// <summary>Gets or sets the time this snapshot was cached.</summary>
-    public required DateTime CachedAt { get; set; }
-    /// <summary>Gets or sets the JSON-serialized payload content.</summary>
-    public required string PayloadJson { get; set; }
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/ILocalConfigCache.cs

@@ -1,32 +0,0 @@
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Per-node local cache of the most-recently-applied generation(s). Used to bootstrap the
-///     address space when the central DB is unreachable (degraded-but-running).
-/// </summary>
-/// <remarks>
-///     <para><b>Concurrency contract:</b> implementations must serialize writes — specifically,
-///     <see cref="PutAsync"/> for the same <c>(ClusterId, GenerationId)</c> from concurrent
-///     callers must not produce duplicate rows. Reads may run concurrently with reads and writes.
-///     The <see cref="LiteDbConfigCache"/> implementation enforces this via an instance-level
-///     <see cref="SemaphoreSlim"/> around the find-then-insert/update window.</para>
-/// </remarks>
-public interface ILocalConfigCache
-{
-    /// <summary>Retrieves the most recent generation snapshot for the specified cluster.</summary>
-    /// <param name="clusterId">The cluster identifier.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>The most recent generation snapshot, or null if none exists.</returns>
-    Task<GenerationSnapshot?> GetMostRecentAsync(string clusterId, CancellationToken ct = default);
-    /// <summary>Stores a generation snapshot in the local cache.</summary>
-    /// <param name="snapshot">The generation snapshot to store.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task that represents the asynchronous operation.</returns>
-    Task PutAsync(GenerationSnapshot snapshot, CancellationToken ct = default);
-    /// <summary>Removes old generations, keeping only the most recent N.</summary>
-    /// <param name="clusterId">The cluster identifier.</param>
-    /// <param name="keepLatest">The number of latest generations to keep.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task that represents the asynchronous operation.</returns>
-    Task PruneOldGenerationsAsync(string clusterId, int keepLatest = 10, CancellationToken ct = default);
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/LiteDbConfigCache.cs

@@ -1,129 +0,0 @@
-using LiteDB;
-
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     LiteDB-backed <see cref="ILocalConfigCache"/>. One file per node (default
-///     <c>config_cache.db</c>), one collection per snapshot. Corruption surfaces as
-///     <see cref="LocalConfigCacheCorruptException"/> on construction or read — callers should
-///     delete and re-fetch from the central DB.
-/// </summary>
-public sealed class LiteDbConfigCache : ILocalConfigCache, IDisposable
-{
-    private const string CollectionName = "generations";
-
-    // LiteDB's default BsonMapper.Global is a process-wide singleton whose per-type member
-    // registration is lazy and NOT thread-safe across concurrently-constructed LiteDatabase
-    // instances. When several caches (this one + GenerationSealedCache) initialise in parallel
-    // the global mapper races, surfacing as "Member ClusterId not found on BsonMapper" or a
-    // bogus "duplicate key _id = 0" (the int auto-id mapping was lost so Insert writes a literal
-    // 0 twice). Give each database a private, pre-registered mapper so member
-    // resolution happens once, single-threaded, at construction and never touches the global.
-    private static BsonMapper BuildMapper()
-    {
-        var mapper = new BsonMapper();
-        mapper.Entity<GenerationSnapshot>();
-        return mapper;
-    }
-
-    private readonly LiteDatabase _db;
-    private readonly ILiteCollection<GenerationSnapshot> _col;
-    // PutAsync is a find-then-insert/update; without serialization, two concurrent puts for the
-    // same (ClusterId, GenerationId) can both observe `existing is null` and both Insert,
-    // producing duplicate rows. Serialize writes through this semaphore so
-    // the read-modify-write block is atomic for a given instance. LiteDB itself only locks the
-    // page-level write, not the find-then-insert window.
-    private readonly SemaphoreSlim _writeGate = new(initialCount: 1, maxCount: 1);
-
-    /// <summary>Initializes a new instance of the <see cref="LiteDbConfigCache"/> class.</summary>
-    /// <param name="dbPath">Path to the LiteDB database file.</param>
-    public LiteDbConfigCache(string dbPath)
-    {
-        // LiteDB can be tolerant of header-only corruption at construction time (it may overwrite
-        // the header and "recover"), so we force a write + read probe to fail fast on real corruption.
-        try
-        {
-            _db = new LiteDatabase(new ConnectionString { Filename = dbPath, Upgrade = true }, BuildMapper());
-            _col = _db.GetCollection<GenerationSnapshot>(CollectionName);
-            _col.EnsureIndex(s => s.ClusterId);
-            _col.EnsureIndex(s => s.GenerationId);
-            _ = _col.Count();
-        }
-        catch (Exception ex) when (ex is LiteException or InvalidDataException or IOException
-                                      or NotSupportedException or UnauthorizedAccessException
-                                      or ArgumentOutOfRangeException or FormatException)
-        {
-            _db?.Dispose();
-            throw new LocalConfigCacheCorruptException(
-                $"LiteDB cache at '{dbPath}' is corrupt or unreadable — delete the file and refetch from the central DB.",
-                ex);
-        }
-    }
-
-    /// <inheritdoc />
-    public Task<GenerationSnapshot?> GetMostRecentAsync(string clusterId, CancellationToken ct = default)
-    {
-        ct.ThrowIfCancellationRequested();
-        var snapshot = _col
-            .Find(s => s.ClusterId == clusterId)
-            .OrderByDescending(s => s.GenerationId)
-            .FirstOrDefault();
-        return Task.FromResult<GenerationSnapshot?>(snapshot);
-    }
-
-    /// <inheritdoc />
-    public async Task PutAsync(GenerationSnapshot snapshot, CancellationToken ct = default)
-    {
-        ct.ThrowIfCancellationRequested();
-        // Serialize the find-then-insert/update so concurrent callers do not observe a stale
-        // `existing is null` and both Insert. LiteDB's per-call lock is not enough — the
-        // read and the write are independent calls.
-        await _writeGate.WaitAsync(ct).ConfigureAwait(false);
-        try
-        {
-            // upsert by (ClusterId, GenerationId) — replace in place if already cached
-            var existing = _col
-                .Find(s => s.ClusterId == snapshot.ClusterId && s.GenerationId == snapshot.GenerationId)
-                .FirstOrDefault();
-
-            if (existing is null)
-                _col.Insert(snapshot);
-            else
-            {
-                snapshot.Id = existing.Id;
-                _col.Update(snapshot);
-            }
-        }
-        finally
-        {
-            _writeGate.Release();
-        }
-    }
-
-    /// <inheritdoc />
-    public Task PruneOldGenerationsAsync(string clusterId, int keepLatest = 10, CancellationToken ct = default)
-    {
-        ct.ThrowIfCancellationRequested();
-        var doomed = _col
-            .Find(s => s.ClusterId == clusterId)
-            .OrderByDescending(s => s.GenerationId)
-            .Skip(keepLatest)
-            .Select(s => s.Id)
-            .ToList();
-
-        foreach (var id in doomed)
-            _col.Delete(id);
-
-        return Task.CompletedTask;
-    }
-
-    /// <summary>Releases all resources used by the cache.</summary>
-    public void Dispose()
-    {
-        _writeGate.Dispose();
-        _db.Dispose();
-    }
-}
-
-public sealed class LocalConfigCacheCorruptException(string message, Exception inner)
-    : Exception(message, inner);

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/ResilientConfigReader.cs

@@ -1,140 +0,0 @@
-using System.Text.RegularExpressions;
-using Microsoft.Extensions.Logging;
-using Polly;
-using Polly.Retry;
-using Polly.Timeout;
-
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Wraps a central-DB fetch function with Phase 6.1 Stream D.2 resilience:
-///     <b>timeout 2 s → retry 3× jittered → fallback to sealed cache</b>. Maintains the
-///     <see cref="StaleConfigFlag"/> — fresh on central-DB success, stale on cache fallback.
-/// </summary>
-/// <remarks>
-///     <para>Read-path only per plan. The write path (draft save, publish) bypasses this
-///     wrapper entirely and fails hard on DB outage so inconsistent writes never land.</para>
-///
-///     <para>Fallback is triggered by <b>any exception</b> the fetch raises (central-DB
-///     unreachable, SqlException, timeout). If the sealed cache also fails (no pointer,
-///     corrupt file, etc.), <see cref="GenerationCacheUnavailableException"/> surfaces — caller
-///     must fail the current request (InitializeAsync for a driver, etc.).</para>
-/// </remarks>
-public sealed class ResilientConfigReader
-{
-    private readonly GenerationSealedCache _cache;
-    private readonly StaleConfigFlag _staleFlag;
-    private readonly ResiliencePipeline _pipeline;
-    private readonly ILogger<ResilientConfigReader> _logger;
-
-    /// <summary>Initializes a resilient config reader with the given cache and options.</summary>
-    /// <param name="cache">The sealed cache for fallback.</param>
-    /// <param name="staleFlag">The stale config flag to manage.</param>
-    /// <param name="logger">The logger instance.</param>
-    /// <param name="timeout">The timeout for central fetch (default 2s).</param>
-    /// <param name="retryCount">The number of retries (default 3).</param>
-    public ResilientConfigReader(
-        GenerationSealedCache cache,
-        StaleConfigFlag staleFlag,
-        ILogger<ResilientConfigReader> logger,
-        TimeSpan? timeout = null,
-        int retryCount = 3)
-    {
-        _cache = cache;
-        _staleFlag = staleFlag;
-        _logger = logger;
-        var builder = new ResiliencePipelineBuilder()
-            .AddTimeout(new TimeoutStrategyOptions { Timeout = timeout ?? TimeSpan.FromSeconds(2) });
-
-        if (retryCount > 0)
-        {
-            builder.AddRetry(new RetryStrategyOptions
-            {
-                MaxRetryAttempts = retryCount,
-                BackoffType = DelayBackoffType.Exponential,
-                UseJitter = true,
-                Delay = TimeSpan.FromMilliseconds(100),
-                MaxDelay = TimeSpan.FromSeconds(1),
-                // Handle ALL exceptions including OperationCanceledException. A SQL command-level
-                // timeout surfaces as TaskCanceledException (derives from OperationCanceledException)
-                // when the caller's token is NOT cancelled, and must be retried just like any other
-                // transient error. Polly itself checks the cancellation token between retries and
-                // stops with OperationCanceledException on genuine caller cancellation regardless of
-                // this predicate.
-                ShouldHandle = new PredicateBuilder().Handle<Exception>(),
-            });
-        }
-
-        _pipeline = builder.Build();
-    }
-
-    /// <summary>
-    ///     Redacts connection-string fragments (Password, User Id, Pwd, etc.)
-    ///     that a caller's exception message could carry. Conservative regex pass — anything
-    ///     matching <c>Key=Value</c> with a known credential key gets its value replaced.
-    /// </summary>
-    private static readonly Regex SecretsRegex = new(
-        @"(?ix)\b(Password|Pwd|User\s*Id|Uid|AccessToken|Authorization|Api[-_]?Key)\s*=\s*[^;,)\s]*",
-        RegexOptions.Compiled);
-
-    /// <summary>Redacts sensitive credential information from a message.</summary>
-    /// <param name="message">The message to scrub.</param>
-    /// <returns>The message with redacted credentials.</returns>
-    internal static string ScrubSecrets(string? message)
-    {
-        if (string.IsNullOrEmpty(message)) return message ?? string.Empty;
-        // Replace the entire matched fragment (key + value) with a redaction marker so the
-        // key name itself doesn't leak — log scrapers grep for "Password=" too.
-        return SecretsRegex.Replace(message, "[redacted credential]");
-    }
-
-    /// <summary>
-    ///     Executes a central fetch through the resilience pipeline. On full failure
-    ///     (post-retry), reads the sealed cache and extracts the requested shape.
-    /// </summary>
-    /// <typeparam name="T">The type of configuration to read.</typeparam>
-    /// <param name="clusterId">The cluster ID to fetch for.</param>
-    /// <param name="centralFetch">Function to fetch from central DB.</param>
-    /// <param name="fromSnapshot">Function to extract the config from a snapshot.</param>
-    /// <param name="cancellationToken">Cancellation token.</param>
-    /// <returns>The configuration of type T.</returns>
-    public async ValueTask<T> ReadAsync<T>(
-        string clusterId,
-        Func<CancellationToken, ValueTask<T>> centralFetch,
-        Func<GenerationSnapshot, T> fromSnapshot,
-        CancellationToken cancellationToken)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
-        ArgumentNullException.ThrowIfNull(centralFetch);
-        ArgumentNullException.ThrowIfNull(fromSnapshot);
-
-        try
-        {
-            var result = await _pipeline.ExecuteAsync(centralFetch, cancellationToken).ConfigureAwait(false);
-            _staleFlag.MarkFresh();
-            return result;
-        }
-        // Catch all exceptions that are NOT genuine caller cancellations. A SQL command-level
-        // timeout surfaces as TaskCanceledException (derives from OperationCanceledException)
-        // but the caller's token is NOT cancelled — we must fall back to the sealed cache for
-        // that case, not propagate. Only rethrow if the caller actually requested cancellation.
-        catch (Exception ex) when (ex is not OperationCanceledException || !cancellationToken.IsCancellationRequested)
-        {
-            // Do NOT pass the raw exception object — it carries the stack
-            // and inner-exception chain, and SqlException/wrapping delegates can surface
-            // connection-string fragments (Password=…, User Id=…) embedded in messages.
-            // Log only the exception type and a scrubbed message so secrets stay out of logs.
-            _logger.LogWarning(
-                "Central-DB read failed after retries ({ExceptionType}: {SanitizedMessage}); falling back to sealed cache for cluster {ClusterId}",
-                ex.GetType().Name,
-                ScrubSecrets(ex.Message),
-                clusterId);
-            // GenerationCacheUnavailableException surfaces intentionally — fails the caller's
-            // operation. StaleConfigFlag stays unchanged; the flag only flips when we actually
-            // served a cache snapshot.
-            var snapshot = await _cache.ReadCurrentAsync(clusterId, cancellationToken).ConfigureAwait(false);
-            _staleFlag.MarkStale();
-            return fromSnapshot(snapshot);
-        }
-    }
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/StaleConfigFlag.cs

@@ -1,20 +0,0 @@
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Thread-safe <c>UsingStaleConfig</c> signal per Phase 6.1 Stream D.3. Flips true whenever
-///     a read falls back to a sealed cache snapshot; flips false on the next successful central-DB
-///     round-trip. Surfaced on <c>/healthz</c> body and on the Admin <c>/hosts</c> page.
-/// </summary>
-public sealed class StaleConfigFlag
-{
-    private int _stale;
-
-    /// <summary>True when the last config read was served from the sealed cache, not the central DB.</summary>
-    public bool IsStale => Volatile.Read(ref _stale) != 0;
-
-    /// <summary>Mark the current config as stale (a read fell back to the cache).</summary>
-    public void MarkStale() => Volatile.Write(ref _stale, 1);
-
-    /// <summary>Mark the current config as fresh (a central-DB read succeeded).</summary>
-    public void MarkFresh() => Volatile.Write(ref _stale, 0);
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Migrations/20260722122301_AddClusterNodeTransportPorts.cs

@@ -0,0 +1,39 @@
+using Microsoft.EntityFrameworkCore.Migrations;
+
+#nullable disable
+
+namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
+{
+    /// <inheritdoc />
+    public partial class AddClusterNodeTransportPorts : Migration
+    {
+        /// <inheritdoc />
+        protected override void Up(MigrationBuilder migrationBuilder)
+        {
+            migrationBuilder.AddColumn<int>(
+                name: "AkkaPort",
+                table: "ClusterNode",
+                type: "int",
+                nullable: false,
+                defaultValue: 4053);
+
+            migrationBuilder.AddColumn<int>(
+                name: "GrpcPort",
+                table: "ClusterNode",
+                type: "int",
+                nullable: true);
+        }
+
+        /// <inheritdoc />
+        protected override void Down(MigrationBuilder migrationBuilder)
+        {
+            migrationBuilder.DropColumn(
+                name: "AkkaPort",
+                table: "ClusterNode");
+
+            migrationBuilder.DropColumn(
+                name: "GrpcPort",
+                table: "ClusterNode");
+        }
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Migrations/20260722125506_AddClusterNodeMaintenanceMode.cs

@@ -0,0 +1,29 @@
+using Microsoft.EntityFrameworkCore.Migrations;
+
+#nullable disable
+
+namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
+{
+    /// <inheritdoc />
+    public partial class AddClusterNodeMaintenanceMode : Migration
+    {
+        /// <inheritdoc />
+        protected override void Up(MigrationBuilder migrationBuilder)
+        {
+            migrationBuilder.AddColumn<bool>(
+                name: "MaintenanceMode",
+                table: "ClusterNode",
+                type: "bit",
+                nullable: false,
+                defaultValue: false);
+        }
+
+        /// <inheritdoc />
+        protected override void Down(MigrationBuilder migrationBuilder)
+        {
+            migrationBuilder.DropColumn(
+                name: "MaintenanceMode",
+                table: "ClusterNode");
+        }
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Migrations/OtOpcUaConfigDbContextModelSnapshot.cs

@@ -47,6 +47,11 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
                         .HasMaxLength(64)
                         .HasColumnType("nvarchar(64)");
 
+                    b.Property<int>("AkkaPort")
+                        .ValueGeneratedOnAdd()
+                        .HasColumnType("int")
+                        .HasDefaultValue(4053);
+
                     b.Property<string>("ApplicationUri")
                         .IsRequired()
                         .HasMaxLength(256)
@@ -76,6 +81,9 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
                     b.Property<bool>("Enabled")
                         .HasColumnType("bit");
 
+                    b.Property<int?>("GrpcPort")
+                        .HasColumnType("int");
+
                     b.Property<string>("Host")
                         .IsRequired()
                         .HasMaxLength(255)
@@ -84,6 +92,9 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
                     b.Property<DateTime?>("LastSeenAt")
                         .HasColumnType("datetime2(3)");
 
+                    b.Property<bool>("MaintenanceMode")
+                        .HasColumnType("bit");
+
                     b.Property<int>("OpcUaPort")
                         .HasColumnType("int");
 

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/OtOpcUaConfigDbContext.cs

@@ -142,6 +142,11 @@ public sealed class OtOpcUaConfigDbContext(DbContextOptions<OtOpcUaConfigDbConte
             e.Property(x => x.Host).HasMaxLength(255);
             e.Property(x => x.ApplicationUri).HasMaxLength(256);
             e.Property(x => x.DriverConfigOverridesJson).HasColumnType("nvarchar(max)");
+            // Central's dial targets (per-cluster mesh Phase 1). AkkaPort carries a DB-side default so
+            // rows migrated from before the column existed come out at 4053 rather than 0 — every node
+            // in the fleet does listen on a remoting port, so 0 is never a truthful value for it.
+            // GrpcPort has no default: nothing listens on it until Phase 5.
+            e.Property(x => x.AkkaPort).HasDefaultValue(4053);
             e.Property(x => x.LastSeenAt).HasColumnType("datetime2(3)");
             e.Property(x => x.CreatedAt).HasColumnType("datetime2(3)").HasDefaultValueSql("SYSUTCDATETIME()");
             e.Property(x => x.CreatedBy).HasMaxLength(128);

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Services/ILdapGroupRoleMappingService.cs

@@ -10,10 +10,18 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Services;
 ///     Phase 6.2 compliance check on control/data-plane separation).
 /// </summary>
 /// <remarks>
-///     Per Phase 6.2 Stream A.2 this service is expected to run behind the Phase 6.1
-///     <c>ResilientConfigReader</c> pipeline (timeout → retry → fallback-to-cache) so a
-///     transient DB outage during sign-in falls back to the sealed snapshot rather than
-///     denying every login.
+///     <para>
+///         This service has no local-cache fallback: a DB outage during sign-in denies logins.
+///     </para>
+///     <para>
+///         The Phase 6.1 <c>ResilientConfigReader</c> pipeline (timeout → retry →
+///         fallback-to-sealed-snapshot) this was once expected to run behind was never wired to
+///         anything and was deleted along with the rest of the dormant LiteDB local cache, which
+///         <c>ZB.MOM.WW.LocalDb</c> supersedes. LocalDb caches the deployed-configuration artifact
+///         for driver-role nodes; it does not currently cover admin-plane reads like this one.
+///         Reviving the fallback means adding an admin-side cache on LocalDb, not restoring the
+///         old pipeline.
+///     </para>
 /// </remarks>
 public interface ILdapGroupRoleMappingService
 {

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/ZB.MOM.WW.OtOpcUa.Configuration.csproj

@@ -24,9 +24,12 @@
             <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
         </PackageReference>
         <PackageReference Include="Microsoft.AspNetCore.DataProtection.EntityFrameworkCore"/>
+        <!-- Surgical transitive pin — see the System.Security.Cryptography.Xml block in
+             Directory.Packages.props. This is the one project where the DataProtection chain
+             enters the repo; every other consumer reaches it through a ProjectReference here. -->
+        <PackageReference Include="System.Security.Cryptography.Xml"/>
         <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions"/>
         <PackageReference Include="Microsoft.Extensions.Logging.Abstractions"/>
-        <PackageReference Include="LiteDB"/>
         <PackageReference Include="Polly.Core"/>
     </ItemGroup>
 

src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmSfSchema.cs

@@ -0,0 +1,105 @@
+using System.Security.Cryptography;
+using System.Text;
+using Microsoft.Data.Sqlite;
+
+namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
+
+/// <summary>
+///     DDL for the alarm store-and-forward buffer: one row per alarm event awaiting delivery to
+///     the historian gateway.
+/// </summary>
+/// <remarks>
+///     <para>
+///         Replaces the standalone <c>alarm-historian.db</c> the sink used to own outright. Living
+///         in the consolidated LocalDb file is what lets the buffer replicate to the redundant pair
+///         peer, so a node that dies with undelivered alarm history no longer takes it to the grave.
+///     </para>
+///     <para>
+///         Deliberately depends on nothing but <see cref="SqliteConnection"/> so it can be applied
+///         to any connection — the host's <c>LocalDbSetup.OnReady</c> in production, and a bare
+///         connection in a test — without dragging the DI graph along. Mirrors
+///         <c>DeploymentCacheSchema</c>.
+///     </para>
+///     <para>
+///         <b>Why <see cref="IdColumn"/> is TEXT and app-minted.</b> The legacy queue keyed on
+///         <c>RowId INTEGER PRIMARY KEY AUTOINCREMENT</c>. Convergence is last-writer-wins over the
+///         primary key, so two nodes independently allocating rowid 7 for different alarms would
+///         silently overwrite one another. The sink mints the id from a hash of the payload, which
+///         additionally makes the same event converge to one row when both nodes of a pair enqueue
+///         it — as they legitimately do in the window before the first redundancy snapshot arrives.
+///     </para>
+/// </remarks>
+public static class AlarmSfSchema
+{
+    /// <summary>Table holding queued alarm events awaiting delivery.</summary>
+    public const string EventsTable = "alarm_sf_events";
+
+    /// <summary>The single primary-key column.</summary>
+    public const string IdColumn = "id";
+
+    /// <summary>
+    ///     Derives a row's primary key from its serialized payload.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         Deterministic rather than a fresh GUID, so that the same event arriving twice
+    ///         converges to one row under last-writer-wins instead of duplicating. That happens for
+    ///         real in two places: <c>HistorianAdapterActor</c> default-writes while its redundancy
+    ///         role is unknown, so both nodes of a pair accept the same fanned transition in the
+    ///         window before the first snapshot; and both nodes independently migrate their own
+    ///         legacy queue file, which for a warm pair holds overlapping history.
+    ///     </para>
+    ///     <para>
+    ///         Two genuinely distinct events cannot collide. <c>AlarmHistorianEvent</c> carries a
+    ///         full-precision timestamp alongside the alarm id, transition kind, message and user,
+    ///         so an equal hash means an equal event.
+    ///     </para>
+    /// </remarks>
+    /// <param name="payloadJson">The serialized <c>AlarmHistorianEvent</c>.</param>
+    /// <returns>The row's primary key.</returns>
+    public static string DeriveId(string payloadJson)
+    {
+        ArgumentNullException.ThrowIfNull(payloadJson);
+        return Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(payloadJson)));
+    }
+
+    /// <summary>
+    ///     Creates the buffer table if it does not already exist. Idempotent.
+    /// </summary>
+    /// <param name="connection">
+    ///     An already-open connection. <c>ILocalDb.CreateConnection()</c> hands out open,
+    ///     pragma-configured connections — do not call <c>Open()</c> on one.
+    /// </param>
+    public static void Apply(SqliteConnection connection)
+    {
+        ArgumentNullException.ThrowIfNull(connection);
+
+        using var cmd = connection.CreateCommand();
+
+        // Columns map 1:1 onto the legacy Queue table so the one-time migrator is a straight copy
+        // and the drain keeps its existing semantics: dead_lettered is the same 0/1 flag, and an
+        // acknowledged row is DELETEd rather than marked. Deleting keeps the table bounded without
+        // a second sweeper, and the replication engine carries the delete as a tombstone, so the
+        // peer drops its copy too.
+        //
+        // The drain index orders by enqueued_at_utc because a hashed TEXT id carries no insertion
+        // order the way the legacy AUTOINCREMENT RowId did. Timestamps are round-trip ("O") format,
+        // so lexicographic ordering is chronological; id breaks ties so the order is total and the
+        // index covers the drain's read.
+        cmd.CommandText = """
+            CREATE TABLE IF NOT EXISTS alarm_sf_events (
+                id               TEXT    NOT NULL PRIMARY KEY,
+                alarm_id         TEXT    NOT NULL,
+                enqueued_at_utc  TEXT    NOT NULL,
+                payload_json     TEXT    NOT NULL,
+                attempt_count    INTEGER NOT NULL DEFAULT 0,
+                last_attempt_utc TEXT    NULL,
+                last_error       TEXT    NULL,
+                dead_lettered    INTEGER NOT NULL DEFAULT 0
+            );
+            CREATE INDEX IF NOT EXISTS ix_alarm_sf_events_drain
+                ON alarm_sf_events (dead_lettered, enqueued_at_utc, id);
+            """;
+        cmd.ExecuteNonQuery();
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/IAlarmHistorianSink.cs

@@ -4,13 +4,13 @@ namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
 ///     The historian sink contract — where qualifying alarm events land. Ingestion routes
 ///     through the HistorianGateway alarm writer (the gateway's <c>SendEvent</c> gRPC path)
 ///     behind the durable store-and-forward queue. Tests use an in-memory fake; production uses
-///     <see cref="SqliteStoreAndForwardSink"/>.
+///     <see cref="LocalDbStoreAndForwardSink"/>.
 /// </summary>
 /// <remarks>
 ///     <para>
 ///         <see cref="EnqueueAsync"/> is fire-and-forget from the engine's perspective —
 ///         the sink MUST NOT block the emitting thread. Production implementations
-///         (<see cref="SqliteStoreAndForwardSink"/>) persist to a local SQLite queue
+///         (<see cref="LocalDbStoreAndForwardSink"/>) persist to the node's local SQLite queue
 ///         first, then drain asynchronously to the actual historian. Per the Phase 7 plan,
 ///         failed downstream writes replay with exponential backoff;
 ///         operator actions are never blocked waiting on the historian.
@@ -79,6 +79,18 @@ public enum HistorianDrainState
     Idle,
     Draining,
     BackingOff,
+
+    /// <summary>
+    ///     Ticking, but not draining: this node does not hold the Primary role, so it leaves the
+    ///     (replicated) queue for the node that does.
+    /// </summary>
+    /// <remarks>
+    ///     Distinct from <see cref="Idle"/> so a rising queue depth on a Secondary reads as the
+    ///     designed behaviour rather than a stalled drain. It is also how a misconfigured pair
+    ///     becomes diagnosable: if BOTH nodes report this, no one is draining and alarm history is
+    ///     silently accumulating toward the capacity ceiling.
+    /// </remarks>
+    NotPrimary,
 }
 
 /// <summary>Returned by the historian alarm writer per event — drain worker uses this to decide retry cadence.</summary>

src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/LocalDbStoreAndForwardSink.cs

@@ -1,53 +1,54 @@
 using System.Text.Json;
-using Microsoft.Data.Sqlite;
 using Serilog;
+using ZB.MOM.WW.LocalDb;
 
 namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
 
 /// <summary>
-///     Durable SQLite queue on the node absorbs every qualifying alarm event, a drain
-///     worker batches rows to the Wonderware historian sidecar via
-///     <see cref="IAlarmHistorianWriter"/> on an exponential-backoff cadence, and
-///     operator acks never block on the historian being reachable.
+///     Durable queue in the node's consolidated LocalDb absorbs every qualifying alarm event, a
+///     drain worker batches rows to the historian gateway via <see cref="IAlarmHistorianWriter"/>
+///     on an exponential-backoff cadence, and operator acks never block on the historian being
+///     reachable.
 /// </summary>
 /// <remarks>
 ///     <para>
-///         Queue schema:
-///         <code>
-///         CREATE TABLE Queue (
-///             RowId INTEGER PRIMARY KEY AUTOINCREMENT,
-///             AlarmId TEXT NOT NULL,
-///             EnqueuedUtc TEXT NOT NULL,
-///             PayloadJson TEXT NOT NULL,
-///             AttemptCount INTEGER NOT NULL DEFAULT 0,
-///             LastAttemptUtc TEXT NULL,
-///             LastError TEXT NULL,
-///             DeadLettered INTEGER NOT NULL DEFAULT 0
-///         );
-///         </code>
-///         Dead-lettered rows stay in place for the configured retention window (default
-///         30 days) so operators can inspect + manually
-///         retry before the sweeper purges them. Regular queue capacity is bounded —
-///         overflow evicts the oldest non-dead-lettered rows with a WARN log. The
-///         durability guarantee is therefore bounded by <see cref="DefaultCapacity"/>:
-///         under a sustained historian outage, accepted events may be evicted before
-///         delivery. The <see cref="HistorianSinkStatus.EvictedCount"/> counter makes
-///         overflow visible to operators without requiring the WARN log to be scraped.
+///         Rows live in <c>alarm_sf_events</c> (see <see cref="AlarmSfSchema"/>), which the host
+///         registers for replication. That is the point of this type living on
+///         <see cref="ILocalDb"/> rather than owning its own file: the buffer mirrors to the
+///         redundant pair peer, so a node that dies holding undelivered alarm history no longer
+///         takes it to the grave.
+///     </para>
+///     <para>
+///         <b>Only one node of a pair may drain.</b> Replication puts the Primary's queued rows in
+///         the Secondary's table too; an ungated drain there would re-deliver every event,
+///         continuously. The <c>drainGate</c> constructor argument is how the caller scopes
+///         draining to the Primary. Enqueue is gated separately and upstream, by
+///         <c>HistorianAdapterActor</c>.
+///     </para>
+///     <para>
+///         Dead-lettered rows stay in place for the configured retention window (default 30 days)
+///         so operators can inspect + manually retry before the sweeper purges them. Regular queue
+///         capacity is bounded — overflow evicts the oldest non-dead-lettered rows with a WARN log.
+///         The durability guarantee is therefore bounded by <see cref="DefaultCapacity"/>: under a
+///         sustained historian outage, accepted events may be evicted before delivery. The
+///         <see cref="HistorianSinkStatus.EvictedCount"/> counter makes overflow visible to
+///         operators without requiring the WARN log to be scraped.
 ///     </para>
 ///     <para>
 ///         Drain runs on a self-rescheduling one-shot <see cref="System.Threading.Timer"/>.
 ///         Exponential backoff on <see cref="HistorianWriteOutcome.RetryPlease"/>:
-///         1s → 2s → 5s → 15s → 60s cap — the backoff is applied to the timer's next
-///         due-time, so a historian outage genuinely slows the drain cadence.
-///         <see cref="HistorianWriteOutcome.PermanentFail"/> rows flip
-///         the <c>DeadLettered</c> flag on the individual row; neighbors in the batch
-///         still retry on their own cadence.
+///         1s → 2s → 5s → 15s → 60s cap — the backoff is applied to the timer's next due-time, so a
+///         historian outage genuinely slows the drain cadence.
+///         <see cref="HistorianWriteOutcome.PermanentFail"/> rows flip the <c>dead_lettered</c> flag
+///         on the individual row; neighbors in the batch still retry on their own cadence.
 ///     </para>
 /// </remarks>
-public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
+public sealed class LocalDbStoreAndForwardSink : IAlarmHistorianSink, IDisposable
 {
     /// <summary>Default queue capacity — oldest non-dead-lettered rows evicted past this.</summary>
     public const long DefaultCapacity = 1_000_000;
+
+    /// <summary>Default window dead-lettered rows are retained for before the sweeper purges them.</summary>
     public static readonly TimeSpan DefaultDeadLetterRetention = TimeSpan.FromDays(30);
 
     /// <summary>Default max delivery attempts before a perpetually-retrying (poison) row is dead-lettered.</summary>
@@ -62,7 +63,7 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
         TimeSpan.FromSeconds(60),
     ];
 
-    private readonly string _connectionString;
+    private readonly ILocalDb _db;
     private readonly IAlarmHistorianWriter _writer;
     private readonly ILogger _logger;
     private readonly int _batchSize;
@@ -70,8 +71,9 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     private readonly TimeSpan _deadLetterRetention;
     private readonly int _maxAttempts;
     private readonly Func<DateTime> _clock;
+    private readonly Func<bool> _drainGate;
 
-    private readonly SemaphoreSlim _drainGate = new(1, 1);
+    private readonly SemaphoreSlim _drainGateLock = new(1, 1);
     private Timer? _drainTimer;
     private TimeSpan _tickInterval;
     private volatile int _backoffIndex;
@@ -85,13 +87,16 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     private string? _lastError;
     private HistorianDrainState _drainState = HistorianDrainState.Idle;
     private long _evictedCount;
+    // Tracks the last gate decision so a denial is logged on the transition rather than on every
+    // tick. Null until the first drain runs.
+    private bool? _lastGateDecision;
 
     private long _queuedRowCount;
     // Probe counter — incremented every time we actually issue a real COUNT(*) for
     // capacity enforcement. Public for test instrumentation only.
     private long _capacityProbeCount;
     // After every Nth enqueue we resync the in-memory counter from storage to defend
-    // against silent drift (e.g. an external process editing the DB).
+    // against silent drift (e.g. the peer replicating rows in underneath us).
     private const long ResyncEnqueueInterval = 10_000;
     private long _enqueuesSinceResync;
 
@@ -99,9 +104,13 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     public long DebugCapacityProbeCount => Interlocked.Read(ref _capacityProbeCount);
 
     /// <summary>
-    ///     Initializes a new instance of the <see cref="SqliteStoreAndForwardSink"/> class with the specified configuration.
+    ///     Initializes a new instance of the <see cref="LocalDbStoreAndForwardSink"/> class.
     /// </summary>
-    /// <param name="databasePath">The filesystem path to the SQLite database file.</param>
+    /// <param name="db">
+    ///     The node's local database. Its <c>alarm_sf_events</c> table must already exist and be
+    ///     registered for replication — the host does both in <c>LocalDbSetup.OnReady</c>, before
+    ///     any consumer can resolve this sink.
+    /// </param>
     /// <param name="writer">The alarm historian writer to handle batch forwarding.</param>
     /// <param name="logger">The logger for diagnostic output.</param>
     /// <param name="batchSize">The maximum number of rows to forward in a single batch. Defaults to 100.</param>
@@ -109,18 +118,24 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     /// <param name="deadLetterRetention">The timespan to retain dead-lettered rows before purging. Defaults to 30 days.</param>
     /// <param name="maxAttempts">The maximum number of delivery attempts before a perpetually-retrying (poison) row is dead-lettered. Defaults to 10.</param>
     /// <param name="clock">Optional clock function for testing; defaults to <see cref="DateTime.UtcNow"/>.</param>
-    public SqliteStoreAndForwardSink(
-        string databasePath,
+    /// <param name="drainGate">
+    ///     Consulted at the top of every drain tick; <c>false</c> skips the tick entirely, leaving
+    ///     the queue intact. Callers running a redundant pair MUST supply a gate that is true on at
+    ///     most one node, because the queue replicates — see the class remarks. The default
+    ///     (<c>null</c> ⇒ always drain) is the correct single-node and test posture.
+    /// </param>
+    public LocalDbStoreAndForwardSink(
+        ILocalDb db,
         IAlarmHistorianWriter writer,
         ILogger logger,
         int batchSize = 100,
         long capacity = DefaultCapacity,
         TimeSpan? deadLetterRetention = null,
         int maxAttempts = DefaultMaxAttempts,
-        Func<DateTime>? clock = null)
+        Func<DateTime>? clock = null,
+        Func<bool>? drainGate = null)
     {
-        if (string.IsNullOrWhiteSpace(databasePath))
-            throw new ArgumentException("Database path required.", nameof(databasePath));
+        _db = db ?? throw new ArgumentNullException(nameof(db));
         _writer = writer ?? throw new ArgumentNullException(nameof(writer));
         _logger = logger ?? throw new ArgumentNullException(nameof(logger));
         _batchSize = batchSize > 0 ? batchSize : throw new ArgumentOutOfRangeException(nameof(batchSize));
@@ -128,50 +143,11 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
         _deadLetterRetention = deadLetterRetention ?? DefaultDeadLetterRetention;
         _maxAttempts = maxAttempts > 0 ? maxAttempts : throw new ArgumentOutOfRangeException(nameof(maxAttempts));
         _clock = clock ?? (() => DateTime.UtcNow);
-        // DefaultTimeout gives ADO.NET command-level retry; the PRAGMA busy_timeout
-        // applied in OpenConnection backs it with SQLite's own busy-handler so an
-        // enqueue/drain collision waits out the file lock instead of throwing
-        // SQLITE_BUSY immediately.
-        _connectionString = new SqliteConnectionStringBuilder
-        {
-            DataSource = databasePath,
-            DefaultTimeout = 5,
-        }.ToString();
+        _drainGate = drainGate ?? (static () => true);
 
-        InitializeSchema();
         _queuedRowCount = ProbeQueuedRowCount();
     }
 
-    /// <summary>
-    ///     Open a connection with the busy timeout + WAL journal applied. SQLite
-    ///     serializes writers with a file lock; the busy_timeout lets a writer wait
-    ///     out a competing lock (default is 0 — fail fast), and WAL lets readers and
-    ///     the single writer proceed without blocking each other.
-    /// </summary>
-    private SqliteConnection OpenConnection()
-    {
-        var conn = new SqliteConnection(_connectionString);
-        conn.Open();
-        ApplyPragmas(conn);
-        return conn;
-    }
-
-    /// <summary>Apply busy_timeout + WAL pragmas to an already-open connection (sync).</summary>
-    private static void ApplyPragmas(SqliteConnection conn)
-    {
-        using var pragma = conn.CreateCommand();
-        pragma.CommandText = "PRAGMA busy_timeout=5000; PRAGMA journal_mode=WAL;";
-        pragma.ExecuteNonQuery();
-    }
-
-    /// <summary>Apply busy_timeout + WAL pragmas to an already-open connection (async).</summary>
-    private static async Task ApplyPragmasAsync(SqliteConnection conn, CancellationToken ct)
-    {
-        using var pragma = conn.CreateCommand();
-        pragma.CommandText = "PRAGMA busy_timeout=5000; PRAGMA journal_mode=WAL;";
-        await pragma.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
-    }
-
     /// <summary>
     ///     Start the background drain worker. Not started automatically so tests can
     ///     drive <see cref="DrainOnceAsync"/> deterministically.
@@ -188,7 +164,7 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     /// <param name="tickInterval">The base interval between drain attempts.</param>
     public void StartDrainLoop(TimeSpan tickInterval)
     {
-        if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
+        if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
         _tickInterval = tickInterval;
         _drainTimer?.Dispose();
         // One-shot: dueTime = tickInterval, period = Infinite. RescheduleDrain re-arms
@@ -239,25 +215,31 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     public async Task EnqueueAsync(AlarmHistorianEvent evt, CancellationToken cancellationToken)
     {
         if (evt is null) throw new ArgumentNullException(nameof(evt));
-        if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
+        if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
 
-        using var conn = new SqliteConnection(_connectionString);
-        await conn.OpenAsync(cancellationToken).ConfigureAwait(false);
-        await ApplyPragmasAsync(conn, cancellationToken).ConfigureAwait(false);
+        await EnforceCapacityFastPathAsync(cancellationToken).ConfigureAwait(false);
 
-        await EnforceCapacityFastPathAsync(conn, cancellationToken).ConfigureAwait(false);
+        var payload = JsonSerializer.Serialize(evt);
 
-        using var cmd = conn.CreateCommand();
-        cmd.CommandText = """
-            INSERT INTO Queue (AlarmId, EnqueuedUtc, PayloadJson, AttemptCount)
-            VALUES ($alarmId, $enqueued, $payload, 0);
-            """;
-        cmd.Parameters.AddWithValue("$alarmId", evt.AlarmId);
-        cmd.Parameters.AddWithValue("$enqueued", _clock().ToString("O"));
-        cmd.Parameters.AddWithValue("$payload", JsonSerializer.Serialize(evt));
-        await cmd.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
+        // ON CONFLICT DO NOTHING: re-enqueuing an identical event (the pair's boot-window
+        // double-accept) must not resurrect a row the drain already bumped or dead-lettered.
+        var inserted = await _db.ExecuteAsync(
+            """
+            INSERT INTO alarm_sf_events
+                (id, alarm_id, enqueued_at_utc, payload_json, attempt_count)
+            VALUES (@Id, @AlarmId, @EnqueuedAtUtc, @PayloadJson, 0)
+            ON CONFLICT(id) DO NOTHING
+            """,
+            new
+            {
+                Id = AlarmSfSchema.DeriveId(payload),
+                AlarmId = evt.AlarmId,
+                EnqueuedAtUtc = _clock().ToString("O"),
+                PayloadJson = payload,
+            },
+            cancellationToken).ConfigureAwait(false);
 
-        Interlocked.Increment(ref _queuedRowCount);
+        if (inserted > 0) Interlocked.Increment(ref _queuedRowCount);
     }
 
     /// <summary>
@@ -268,15 +250,17 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     ///     through <see cref="EnforceCapacityAsync"/> which still runs a precise
     ///     COUNT to compute the exact number of rows to evict.
     /// </summary>
-    private async Task EnforceCapacityFastPathAsync(SqliteConnection conn, CancellationToken ct)
+    private async Task EnforceCapacityFastPathAsync(CancellationToken ct)
     {
         var enqueuesSinceResync = Interlocked.Increment(ref _enqueuesSinceResync);
         var cached = Interlocked.Read(ref _queuedRowCount);
 
-        // Periodic resync — bounded amount of drift even under exotic conditions.
+        // Periodic resync — bounded amount of drift even under exotic conditions. Now doubly
+        // warranted: replication applies the peer's rows straight into the table, so the counter
+        // has a second way to fall behind that no local code path can observe.
         if (enqueuesSinceResync >= ResyncEnqueueInterval)
         {
-            await ResyncQueuedRowCountAsync(conn, ct).ConfigureAwait(false);
+            await ResyncQueuedRowCountAsync(ct).ConfigureAwait(false);
             cached = Interlocked.Read(ref _queuedRowCount);
             Interlocked.Exchange(ref _enqueuesSinceResync, 0);
         }
@@ -286,55 +270,70 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
 
         // Cached counter says we're at or above the capacity wall — fall back to the
         // precise path which probes COUNT(*) and evicts whatever's needed.
-        await EnforceCapacityAsync(conn, ct).ConfigureAwait(false);
+        await EnforceCapacityAsync(ct).ConfigureAwait(false);
     }
 
     /// <summary>Synchronously query <c>COUNT(*)</c> of non-dead-lettered rows. Used at startup.</summary>
     private long ProbeQueuedRowCount()
     {
         Interlocked.Increment(ref _capacityProbeCount);
-        using var conn = OpenConnection();
+        using var conn = _db.CreateConnection();
         using var cmd = conn.CreateCommand();
-        cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
+        cmd.CommandText = "SELECT COUNT(*) FROM alarm_sf_events WHERE dead_lettered = 0";
         return (long)(cmd.ExecuteScalar() ?? 0L);
     }
 
     /// <summary>Re-sync the in-memory counter from storage (async path).</summary>
-    private async Task ResyncQueuedRowCountAsync(SqliteConnection conn, CancellationToken ct)
+    private async Task ResyncQueuedRowCountAsync(CancellationToken ct)
     {
         Interlocked.Increment(ref _capacityProbeCount);
-        using var cmd = conn.CreateCommand();
-        cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
-        var live = (long)(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false) ?? 0L);
+        var live = await CountAsync("dead_lettered = 0", ct).ConfigureAwait(false);
         Interlocked.Exchange(ref _queuedRowCount, live);
     }
 
+    private async Task<long> CountAsync(string predicate, CancellationToken ct)
+    {
+        var rows = await _db.QueryAsync(
+            $"SELECT COUNT(*) FROM alarm_sf_events WHERE {predicate}",
+            r => r.GetInt64(0),
+            parameters: null,
+            ct).ConfigureAwait(false);
+        return rows.Count > 0 ? rows[0] : 0L;
+    }
+
     /// <summary>
     ///     Read up to <see cref="_batchSize"/> queued rows, forward through the writer,
     ///     remove Ack'd rows, dead-letter PermanentFail rows, and extend the backoff
     ///     on RetryPlease. Safe to call from multiple threads; the semaphore enforces
     ///     serial execution.
     /// </summary>
+    /// <remarks>
+    ///     Returns without touching the queue when the drain gate is closed. On a redundant pair
+    ///     that is the Secondary's steady state: its table fills by replication and stays put until
+    ///     it is promoted.
+    /// </remarks>
     /// <param name="ct">Cancellation token for the operation.</param>
     /// <returns>A task that represents the asynchronous operation.</returns>
     public async Task DrainOnceAsync(CancellationToken ct)
     {
         if (_disposed) return;
-        if (!await _drainGate.WaitAsync(0, ct).ConfigureAwait(false)) return;
+        if (!await _drainGateLock.WaitAsync(0, ct).ConfigureAwait(false)) return;
         try
         {
+            if (!EvaluateDrainGate())
+            {
+                lock (_statusLock) { _drainState = HistorianDrainState.NotPrimary; }
+                return;
+            }
+
             lock (_statusLock)
             {
                 _drainState = HistorianDrainState.Draining;
                 _lastDrainUtc = _clock();
             }
 
-            // One connection per drain tick — used by purge, read, corrupt-dead-letter,
-            // and the outcome-applying transaction.
-            using var conn = OpenConnection();
-
-            PurgeAgedDeadLetters(conn);
-            var batch = ReadBatch(conn);
+            await PurgeAgedDeadLettersAsync(ct).ConfigureAwait(false);
+            var batch = await ReadBatchAsync(ct).ConfigureAwait(false);
             if (batch.Count == 0)
             {
                 lock (_statusLock) { _drainState = HistorianDrainState.Idle; }
@@ -342,22 +341,24 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
             }
 
             // A null/un-deserializable payload can never succeed — dead-letter it
-            // immediately for its own RowId so it cannot stall the queue head, and
+            // immediately for its own id so it cannot stall the queue head, and
             // exclude it from the batch handed to the writer.
-            var corruptRowIds = batch.Where(r => r.Event is null).Select(r => r.RowId).ToList();
+            var corruptIds = batch.Where(r => r.Event is null).Select(r => r.Id).ToList();
             var liveRows = batch.Where(r => r.Event is not null).ToList();
             var events = liveRows.Select(r => r.Event!).ToList();
 
-            if (corruptRowIds.Count > 0)
+            if (corruptIds.Count > 0)
             {
-                using var corruptTx = conn.BeginTransaction();
-                foreach (var rowId in corruptRowIds)
-                    DeadLetterRow(conn, corruptTx, rowId, $"corrupt payload at {_clock():O}");
-                corruptTx.Commit();
-                Interlocked.Add(ref _queuedRowCount, -corruptRowIds.Count);
+                await using (var corruptTx = await _db.BeginTransactionAsync(ct).ConfigureAwait(false))
+                {
+                    foreach (var id in corruptIds)
+                        await DeadLetterRowAsync(corruptTx, id, $"corrupt payload at {_clock():O}", ct).ConfigureAwait(false);
+                    await corruptTx.CommitAsync(ct).ConfigureAwait(false);
+                }
+                Interlocked.Add(ref _queuedRowCount, -corruptIds.Count);
                 _logger.Warning(
                     "Dead-lettered {Count} historian queue row(s) with un-deserializable payload",
-                    corruptRowIds.Count);
+                    corruptIds.Count);
             }
 
             if (events.Count == 0)
@@ -404,41 +405,44 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
             }
 
             int rowsLeavingQueue = 0;
-            using (var tx = conn.BeginTransaction())
+            await using (var tx = await _db.BeginTransactionAsync(ct).ConfigureAwait(false))
             {
                 for (var i = 0; i < outcomes.Count; i++)
                 {
                     var outcome = outcomes[i];
-                    var rowId = liveRows[i].RowId;
+                    var id = liveRows[i].Id;
                     switch (outcome)
                     {
                         case HistorianWriteOutcome.Ack:
-                            DeleteRow(conn, tx, rowId);
+                            // Deleted, not marked delivered. The replication engine carries the
+                            // delete as a tombstone, so the peer drops its copy and cannot
+                            // re-deliver the row after a failover.
+                            await DeleteRowAsync(tx, id, ct).ConfigureAwait(false);
                             rowsLeavingQueue++;
                             break;
                         case HistorianWriteOutcome.PermanentFail:
-                            DeadLetterRow(conn, tx, rowId, $"permanent fail at {_clock():O}");
+                            await DeadLetterRowAsync(tx, id, $"permanent fail at {_clock():O}", ct).ConfigureAwait(false);
                             rowsLeavingQueue++;
                             break;
                         case HistorianWriteOutcome.RetryPlease:
                             // finding 002: cap retries so a perpetually-RetryPlease (poison)
                             // row cannot retry forever at the 60s backoff floor. The incoming
-                            // AttemptCount is the count BEFORE this attempt; +1 accounts for the
+                            // attempt count is the count BEFORE this attempt; +1 accounts for the
                             // bump this drain represents. At the cap, dead-letter instead of
                             // bumping — and count it as leaving the live queue like PermanentFail.
                             if (liveRows[i].AttemptCount + 1 >= _maxAttempts)
                             {
-                                DeadLetterRow(conn, tx, rowId, $"max attempts ({_maxAttempts}) exceeded");
+                                await DeadLetterRowAsync(tx, id, $"max attempts ({_maxAttempts}) exceeded", ct).ConfigureAwait(false);
                                 rowsLeavingQueue++;
                             }
                             else
                             {
-                                BumpAttempt(conn, tx, rowId, "retry-please");
+                                await BumpAttemptAsync(tx, id, "retry-please", ct).ConfigureAwait(false);
                             }
                             break;
                     }
                 }
-                tx.Commit();
+                await tx.CommitAsync(ct).ConfigureAwait(false);
             }
             // Ack-deleted + PermanentFail-dead-lettered rows both leave the
             // non-dead-lettered queue, as do RetryPlease rows that hit the max-attempts
@@ -464,22 +468,66 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
         }
         finally
         {
-            _drainGate.Release();
+            _drainGateLock.Release();
         }
     }
 
+    /// <summary>
+    ///     Consults the injected gate, logging only when the decision changes.
+    /// </summary>
+    /// <remarks>
+    ///     Logged at all because the closed state is indistinguishable from a healthy idle queue
+    ///     from the outside, and one way to reach it is a redundancy snapshot that never names this
+    ///     node — the documented identity-mismatch shape. Without this line, that misconfiguration
+    ///     would present as alarm history quietly ceasing on both nodes of a pair. Logged on the
+    ///     transition rather than per tick because the drain runs every few seconds.
+    /// </remarks>
+    /// <returns><c>true</c> when this tick may proceed.</returns>
+    private bool EvaluateDrainGate()
+    {
+        bool open;
+        try
+        {
+            open = _drainGate();
+        }
+        catch (Exception ex)
+        {
+            // A throwing gate must not wedge the queue silently, and must not be treated as
+            // permission either: skip this tick, say why, and re-ask on the next one.
+            _logger.Error(ex, "Historian drain gate threw; skipping this tick");
+            return false;
+        }
+
+        bool? previous;
+        lock (_statusLock)
+        {
+            previous = _lastGateDecision;
+            _lastGateDecision = open;
+        }
+
+        if (previous == open) return open;
+
+        if (open)
+            _logger.Information("Historian drain resumed — this node now services the alarm queue");
+        else
+            _logger.Information(
+                "Historian drain suspended — this node is not the Primary. Queued alarm events are retained and will drain from whichever node holds the Primary role");
+
+        return open;
+    }
+
     /// <inheritdoc />
     public HistorianSinkStatus GetStatus()
     {
-        if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
+        if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
         var queued = Interlocked.Read(ref _queuedRowCount);
         if (queued < 0) queued = 0;
 
         long deadlettered;
-        using (var conn = OpenConnection())
+        using (var conn = _db.CreateConnection())
         using (var cmd = conn.CreateCommand())
         {
-            cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 1";
+            cmd.CommandText = "SELECT COUNT(*) FROM alarm_sf_events WHERE dead_lettered = 1";
             deadlettered = (long)(cmd.ExecuteScalar() ?? 0L);
         }
 
@@ -510,10 +558,13 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     /// <returns>The number of rows revived from the dead-letter state.</returns>
     public int RetryDeadLettered()
     {
-        if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
-        using var conn = OpenConnection();
+        if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
+        // Through a LocalDb connection, not a private one: the capture triggers are what make this
+        // revival replicate to the peer, and they only exist on the registered table.
+        using var conn = _db.CreateConnection();
         using var cmd = conn.CreateCommand();
-        cmd.CommandText = "UPDATE Queue SET DeadLettered = 0, AttemptCount = 0, LastError = NULL WHERE DeadLettered = 1";
+        cmd.CommandText =
+            "UPDATE alarm_sf_events SET dead_lettered = 0, attempt_count = 0, last_error = NULL WHERE dead_lettered = 1";
         var revived = cmd.ExecuteNonQuery();
         // Dead-lettered rows rejoin the non-dead-lettered queue — keep the in-memory
         // counter aligned.
@@ -523,29 +574,31 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
 
     /// <summary>
     ///     One queued row paired with its deserialized event. <see cref="Event"/> is
-    ///     <c>null</c> when the row's <c>PayloadJson</c> is corrupt or un-deserializable —
-    ///     the <see cref="RowId"/> always stays bound to its own row so outcomes can
+    ///     <c>null</c> when the row's <c>payload_json</c> is corrupt or un-deserializable —
+    ///     the <see cref="Id"/> always stays bound to its own row so outcomes can
     ///     never be mapped to the wrong row.
     /// </summary>
-    private readonly record struct QueueRow(long RowId, AlarmHistorianEvent? Event, long AttemptCount);
+    private readonly record struct QueueRow(string Id, AlarmHistorianEvent? Event, long AttemptCount);
 
-    private List<QueueRow> ReadBatch(SqliteConnection conn)
+    private async Task<List<QueueRow>> ReadBatchAsync(CancellationToken ct)
     {
-        var rows = new List<QueueRow>();
-        using var cmd = conn.CreateCommand();
-        cmd.CommandText = """
-            SELECT RowId, PayloadJson, AttemptCount FROM Queue
-            WHERE DeadLettered = 0
-            ORDER BY RowId ASC
-            LIMIT $limit
-            """;
-        cmd.Parameters.AddWithValue("$limit", _batchSize);
-        using var reader = cmd.ExecuteReader();
-        while (reader.Read())
+        // Ordered by enqueue time rather than insertion order: a hashed TEXT primary key carries
+        // no sequence the way the legacy AUTOINCREMENT rowid did. Round-trip ("O") timestamps sort
+        // lexicographically in chronological order; id makes the ordering total.
+        var raw = await _db.QueryAsync(
+            """
+            SELECT id, payload_json, attempt_count FROM alarm_sf_events
+            WHERE dead_lettered = 0
+            ORDER BY enqueued_at_utc ASC, id ASC
+            LIMIT @Limit
+            """,
+            r => (Id: r.GetString(0), Payload: r.GetString(1), AttemptCount: r.GetInt64(2)),
+            new { Limit = _batchSize },
+            ct).ConfigureAwait(false);
+
+        var rows = new List<QueueRow>(raw.Count);
+        foreach (var (id, payload, attemptCount) in raw)
         {
-            var rowId = reader.GetInt64(0);
-            var payload = reader.GetString(1);
-            var attemptCount = reader.GetInt64(2);
             AlarmHistorianEvent? evt;
             try
             {
@@ -556,77 +609,58 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
                 // Malformed JSON — carry a null event so the caller dead-letters this row.
                 evt = null;
             }
-            rows.Add(new QueueRow(rowId, evt, attemptCount));
+            rows.Add(new QueueRow(id, evt, attemptCount));
         }
         return rows;
     }
 
-    private static void DeleteRow(SqliteConnection conn, SqliteTransaction tx, long rowId)
-    {
-        using var cmd = conn.CreateCommand();
-        cmd.Transaction = tx;
-        cmd.CommandText = "DELETE FROM Queue WHERE RowId = $id";
-        cmd.Parameters.AddWithValue("$id", rowId);
-        cmd.ExecuteNonQuery();
-    }
+    private static Task DeleteRowAsync(ILocalDbTransaction tx, string id, CancellationToken ct) =>
+        tx.ExecuteAsync("DELETE FROM alarm_sf_events WHERE id = @Id", new { Id = id }, ct);
 
-    private void DeadLetterRow(SqliteConnection conn, SqliteTransaction tx, long rowId, string reason)
-    {
-        using var cmd = conn.CreateCommand();
-        cmd.Transaction = tx;
-        cmd.CommandText = """
-            UPDATE Queue SET DeadLettered = 1, LastAttemptUtc = $now, LastError = $err, AttemptCount = AttemptCount + 1
-            WHERE RowId = $id
-            """;
-        cmd.Parameters.AddWithValue("$now", _clock().ToString("O"));
-        cmd.Parameters.AddWithValue("$err", reason);
-        cmd.Parameters.AddWithValue("$id", rowId);
-        cmd.ExecuteNonQuery();
-    }
+    private Task DeadLetterRowAsync(ILocalDbTransaction tx, string id, string reason, CancellationToken ct) =>
+        tx.ExecuteAsync(
+            """
+            UPDATE alarm_sf_events
+            SET dead_lettered = 1, last_attempt_utc = @Now, last_error = @Err,
+                attempt_count = attempt_count + 1
+            WHERE id = @Id
+            """,
+            new { Now = _clock().ToString("O"), Err = reason, Id = id },
+            ct);
 
-    private void BumpAttempt(SqliteConnection conn, SqliteTransaction tx, long rowId, string reason)
-    {
-        using var cmd = conn.CreateCommand();
-        cmd.Transaction = tx;
-        cmd.CommandText = """
-            UPDATE Queue SET LastAttemptUtc = $now, LastError = $err, AttemptCount = AttemptCount + 1
-            WHERE RowId = $id
-            """;
-        cmd.Parameters.AddWithValue("$now", _clock().ToString("O"));
-        cmd.Parameters.AddWithValue("$err", reason);
-        cmd.Parameters.AddWithValue("$id", rowId);
-        cmd.ExecuteNonQuery();
-    }
+    private Task BumpAttemptAsync(ILocalDbTransaction tx, string id, string reason, CancellationToken ct) =>
+        tx.ExecuteAsync(
+            """
+            UPDATE alarm_sf_events
+            SET last_attempt_utc = @Now, last_error = @Err, attempt_count = attempt_count + 1
+            WHERE id = @Id
+            """,
+            new { Now = _clock().ToString("O"), Err = reason, Id = id },
+            ct);
 
-    // Async variant used by EnqueueAsync.
-    private async Task EnforceCapacityAsync(SqliteConnection conn, CancellationToken ct)
+    private async Task EnforceCapacityAsync(CancellationToken ct)
     {
         Interlocked.Increment(ref _capacityProbeCount);
-        long count;
-        using (var cmd = conn.CreateCommand())
-        {
-            cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
-            count = (long)(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false) ?? 0L);
-        }
+        var count = await CountAsync("dead_lettered = 0", ct).ConfigureAwait(false);
+
         // Resync the in-memory counter while we have a fresh number.
         Interlocked.Exchange(ref _queuedRowCount, count);
         if (count < _capacity) return;
 
         var toEvict = count - _capacity + 1;
-        using (var cmd = conn.CreateCommand())
-        {
-            cmd.CommandText = """
-                DELETE FROM Queue
-                WHERE RowId IN (
-                    SELECT RowId FROM Queue
-                    WHERE DeadLettered = 0
-                    ORDER BY RowId ASC
-                    LIMIT $n
+        await _db.ExecuteAsync(
+            """
+            DELETE FROM alarm_sf_events
+            WHERE id IN (
+                SELECT id FROM alarm_sf_events
+                WHERE dead_lettered = 0
+                ORDER BY enqueued_at_utc ASC, id ASC
+                LIMIT @N
             )
-                """;
-            cmd.Parameters.AddWithValue("$n", toEvict);
-            await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
-        }
+            """,
+            new { N = toEvict },
+            ct).ConfigureAwait(false);
+
         Interlocked.Add(ref _queuedRowCount, -toEvict);
         long lifetimeEvicted;
         lock (_statusLock) { _evictedCount += toEvict; lifetimeEvicted = _evictedCount; }
@@ -635,40 +669,21 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
             _capacity, toEvict, lifetimeEvicted);
     }
 
-    private void PurgeAgedDeadLetters(SqliteConnection conn)
+    private async Task PurgeAgedDeadLettersAsync(CancellationToken ct)
     {
         var cutoff = (_clock() - _deadLetterRetention).ToString("O");
-        using var cmd = conn.CreateCommand();
-        cmd.CommandText = """
-            DELETE FROM Queue
-            WHERE DeadLettered = 1 AND LastAttemptUtc IS NOT NULL AND LastAttemptUtc < $cutoff
-            """;
-        cmd.Parameters.AddWithValue("$cutoff", cutoff);
-        var purged = cmd.ExecuteNonQuery();
+        var purged = await _db.ExecuteAsync(
+            """
+            DELETE FROM alarm_sf_events
+            WHERE dead_lettered = 1 AND last_attempt_utc IS NOT NULL AND last_attempt_utc < @Cutoff
+            """,
+            new { Cutoff = cutoff },
+            ct).ConfigureAwait(false);
+
         if (purged > 0)
             _logger.Information("Purged {Count} dead-lettered row(s) past retention window", purged);
     }
 
-    private void InitializeSchema()
-    {
-        using var conn = OpenConnection();
-        using var cmd = conn.CreateCommand();
-        cmd.CommandText = """
-            CREATE TABLE IF NOT EXISTS Queue (
-                RowId INTEGER PRIMARY KEY AUTOINCREMENT,
-                AlarmId TEXT NOT NULL,
-                EnqueuedUtc TEXT NOT NULL,
-                PayloadJson TEXT NOT NULL,
-                AttemptCount INTEGER NOT NULL DEFAULT 0,
-                LastAttemptUtc TEXT NULL,
-                LastError TEXT NULL,
-                DeadLettered INTEGER NOT NULL DEFAULT 0
-            );
-            CREATE INDEX IF NOT EXISTS IX_Queue_Drain ON Queue (DeadLettered, RowId);
-            """;
-        cmd.ExecuteNonQuery();
-    }
-
     private void BumpBackoff() => _backoffIndex = Math.Min(_backoffIndex + 1, BackoffLadder.Length - 1);
     private void ResetBackoff() => _backoffIndex = 0;
 
@@ -676,12 +691,16 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     public TimeSpan CurrentBackoff => BackoffLadder[_backoffIndex];
 
     /// <summary>Disposes the sink and releases all held resources including the drain timer and the writer.</summary>
+    /// <remarks>
+    ///     The <see cref="ILocalDb"/> is NOT disposed here — it is a shared singleton owned by the
+    ///     host and used by the deployment cache too.
+    /// </remarks>
     public void Dispose()
     {
         if (_disposed) return;
         _disposed = true;
         _drainTimer?.Dispose();
-        _drainGate.Dispose();
+        _drainGateLock.Dispose();
         if (_writer is IDisposable writerDisposable) writerDisposable.Dispose();
     }
 }

src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.csproj

@@ -20,6 +20,12 @@
         -->
         <PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3"/>
         <PackageReference Include="Serilog"/>
+        <!--
+          The store-and-forward buffer lives in the node's consolidated LocalDb file so it
+          replicates to the redundant pair peer. Only the ILocalDb SQL surface is used here; the
+          Host owns the DI registration and the replication package.
+        -->
+        <PackageReference Include="ZB.MOM.WW.LocalDb"/>
     </ItemGroup>
 
     <ItemGroup>

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayAlarmHistorianWriter.cs

@@ -9,7 +9,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
 
 /// <summary>
 ///     <see cref="IAlarmHistorianWriter"/> backed by the HistorianGateway <c>SendEvent</c> path. The
-///     drain worker behind <c>SqliteStoreAndForwardSink</c> calls
+///     drain worker behind <c>LocalDbStoreAndForwardSink</c> calls
 ///     <see cref="WriteBatchAsync"/> and uses the returned per-event
 ///     <see cref="HistorianWriteOutcome"/> to decide retry vs. dead-letter, so this writer maps every
 ///     gateway result — success ack, the published client's typed exception hierarchy, raw

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianServiceCollectionExtensions.cs

@@ -47,7 +47,7 @@ public static class GatewayHistorian
     ///     <see cref="ServerHistorianOptions"/> — the <b>same single gateway</b> the read path
     ///     (<see cref="CreateDataSource"/>) targets. The Host's <c>AddAlarmHistorian</c> wiring supplies
     ///     this as the concrete <see cref="IAlarmHistorianWriter"/> the durable
-    ///     <c>SqliteStoreAndForwardSink</c> drain worker delegates to, sourcing the connection from the
+    ///     <c>LocalDbStoreAndForwardSink</c> drain worker delegates to, sourcing the connection from the
     ///     <c>ServerHistorian</c> section (endpoint/key/TLS) rather than the legacy Wonderware-shaped
     ///     <c>AlarmHistorian</c> host/port. Resolves an <see cref="ILoggerFactory"/> and the writer's
     ///     <see cref="ILogger{TCategoryName}"/> from <paramref name="services"/>, falling back to the null

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/HistorianGatewayClientAdapter.cs

@@ -57,16 +57,23 @@ public sealed class HistorianGatewayClientAdapter : IHistorianGatewayClient, IDi
                 $"ServerHistorian:Endpoint must be an absolute http(s) URI (e.g. https://host:5222); got '{options.Endpoint}'.");
         }
 
+        // TLS-only options must stay at their defaults on a plaintext h2c connection: the client rejects
+        // each of them outright when UseTls=false ("<name> is a TLS-only option and requires
+        // UseTls=true"). Forwarding them unconditionally made h2c UNREACHABLE — AllowUntrustedServer-
+        // Certificate defaults to false, so RequireCertificateValidation was always sent as true and
+        // every http:// deployment crashed at startup, even though the scheme is documented as the
+        // supported way to select h2c. There is no certificate to have a posture about here.
         var clientOptions = new HistorianGatewayClientOptions
         {
             Endpoint = endpointUri,
             ApiKey = options.ApiKey,
             UseTls = options.UseTls,
-            CaCertificatePath = options.CaCertificatePath,
-            // INVERTED mapping: ServerHistorianOptions.AllowUntrustedServerCertificate (opt-in to accept
-            // a self-signed cert) is the negation of the client's RequireCertificateValidation. Allowing
-            // an untrusted cert == not requiring validation; a pinned CaCertificatePath always verifies.
-            RequireCertificateValidation = !options.AllowUntrustedServerCertificate,
+            CaCertificatePath = options.UseTls ? options.CaCertificatePath : null,
+            // INVERTED mapping (TLS only): ServerHistorianOptions.AllowUntrustedServerCertificate
+            // (opt-in to accept a self-signed cert) is the negation of the client's
+            // RequireCertificateValidation. Allowing an untrusted cert == not requiring validation; a
+            // pinned CaCertificatePath always verifies.
+            RequireCertificateValidation = options.UseTls && !options.AllowUntrustedServerCertificate,
             DefaultCallTimeout = options.CallTimeout,
             LoggerFactory = loggerFactory,
         };

src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/AlarmsHistorian.razor

@@ -18,8 +18,9 @@
 <section class="panel notice rise" style="animation-delay:.02s">
     Snapshot from the local node's <span class="mono">HistorianAdapterActor</span>. Default sink
     is a no-op (<span class="mono">NullAlarmHistorianSink</span>); production wires
-    <span class="mono">SqliteStoreAndForwardSink</span> draining to the HistorianGateway
-    (<span class="mono">SendEvent</span>) behind it. Polling every @PollSeconds s.
+    <span class="mono">LocalDbStoreAndForwardSink</span> — buffering into this node's LocalDb, and
+    draining to the HistorianGateway (<span class="mono">SendEvent</span>) only while this node holds
+    the Primary role. Polling every @PollSeconds s.
 </section>
 
 @if (_status is null)

src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Clusters/ClusterRedundancy.razor

@@ -1,10 +1,16 @@
 @page "/clusters/{ClusterId}/redundancy"
 @attribute [Authorize(Policy = AdminUiPolicies.AuthenticatedRead)]
 @rendermode RenderMode.InteractiveServer
+@using Microsoft.AspNetCore.Authorization
 @using Microsoft.EntityFrameworkCore
+@using ZB.MOM.WW.OtOpcUa.AdminUI.Redundancy
 @using ZB.MOM.WW.OtOpcUa.Configuration
 @using ZB.MOM.WW.OtOpcUa.Configuration.Entities
+@using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy
 @inject IDbContextFactory<OtOpcUaConfigDbContext> DbFactory
+@inject IManualFailoverService FailoverService
+@inject AuthenticationStateProvider AuthState
+@inject IAuthorizationService AuthorizationService
 
 @if (!_loaded)
 {
@@ -45,6 +51,67 @@ else
         </div>
     </section>
 
+    <section class="panel rise mt-3" style="animation-delay:.11s">
+        <div class="panel-head">Live redundancy</div>
+        <div style="padding:1rem">
+            <div class="kv">
+                <span class="k">Driver Primary</span>
+                <span class="v mono">@(_failover.Snapshot?.PrimaryAddress ?? "—")</span>
+            </div>
+            <div class="kv">
+                <span class="k">Up driver members</span>
+                <span class="v mono">
+                    @(_failover.Snapshot is { DriverAddresses.Count: > 0 } s
+                        ? string.Join(", ", s.DriverAddresses)
+                        : "—")
+                </span>
+            </div>
+
+            <p class="text-muted small mt-2 mb-0">
+                Read live from cluster state on this node. <strong>Mesh-wide scope:</strong> the Primary is
+                elected once per Akka mesh, not per cluster row — in the current single-mesh topology this
+                acts on the whole mesh's Primary, which may be a node of another
+                <span class="mono">Cluster</span>. See <span class="mono">docs/Redundancy.md</span>.
+            </p>
+
+            <AuthorizeView Policy="@ManualFailoverPageModel.RequiredPolicy">
+                <Authorized>
+                    <div class="mt-3">
+                        <button class="btn btn-sm btn-outline-danger"
+                                disabled="@(!_failover.CanFailOver)"
+                                title="@(_failover.DisabledReason ?? "Gracefully move the driver Primary to its peer")"
+                                @onclick="() => _failover.RequestFailover()">
+                            Trigger failover
+                        </button>
+                        @if (_failover.DisabledReason is { } reason)
+                        {
+                            <span class="text-muted small ms-2">@reason</span>
+                        }
+                    </div>
+
+                    @if (_failover.ConfirmOpen)
+                    {
+                        <div class="panel notice mt-3">
+                            <strong>Fail over the driver Primary?</strong>
+                            <ul class="mb-2 mt-2">
+                                <li><span class="mono">@(_failover.Snapshot?.PrimaryAddress)</span> leaves the cluster and its process restarts.</li>
+                                <li>Its peer becomes Primary and advertises <span class="mono">ServiceLevel</span> 250.</li>
+                                <li>Connected OPC UA clients re-select the new Primary.</li>
+                            </ul>
+                            <button class="btn btn-sm btn-danger" @onclick="ConfirmFailoverAsync">Confirm failover</button>
+                            <button class="btn btn-sm btn-outline-secondary ms-2" @onclick="() => _failover.CancelFailover()">Cancel</button>
+                        </div>
+                    }
+                </Authorized>
+            </AuthorizeView>
+
+            @if (_failover.StatusMessage is { } msg)
+            {
+                <div class="mt-3 @(_failover.StatusIsError ? "text-danger" : "text-success")">@msg</div>
+            }
+        </div>
+    </section>
+
     <section class="panel rise mt-3" style="animation-delay:.14s">
         <div class="panel-head">Node service-level configuration</div>
         @if (_nodes is null || _nodes.Count == 0)
@@ -91,8 +158,16 @@ else
     private ServerCluster? _cluster;
     private List<ClusterNode>? _nodes;
 
+    // Everything consequential about the failover control (peer guard, confirm flow, outcome text)
+    // lives in this pure model rather than in the markup — the repo has no bUnit, so logic left in a
+    // .razor is verified only by driving the page. Covered by ManualFailoverPageModelTests.
+    private ManualFailoverPageModel _failover = default!;
+
     protected override async Task OnInitializedAsync()
     {
+        _failover = new ManualFailoverPageModel(FailoverService);
+        _failover.Refresh();
+
         await using var db = await DbFactory.CreateDbContextAsync();
         _cluster = await db.ServerClusters.AsNoTracking()
             .FirstOrDefaultAsync(c => c.ClusterId == ClusterId);
@@ -105,4 +180,22 @@ else
         }
         _loaded = true;
     }
+
+    /// <summary>
+    /// Defense-in-depth: the button is FleetAdmin-gated in markup, but this handler runs on the
+    /// server circuit — re-check the policy before bouncing a production node (the same pattern the
+    /// certificate-store actions use).
+    /// </summary>
+    private async Task ConfirmFailoverAsync()
+    {
+        var authState = await AuthState.GetAuthenticationStateAsync();
+        if (!(await AuthorizationService.AuthorizeAsync(
+                authState.User, null, ManualFailoverPageModel.RequiredPolicy)).Succeeded)
+        {
+            _failover.CancelFailover();
+            return;
+        }
+
+        await _failover.ConfirmFailoverAsync(authState.User.Identity?.Name ?? "system");
+    }
 }

src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Grpc/DeploymentArtifactService.cs

@@ -0,0 +1,100 @@
+using Google.Protobuf;
+using Grpc.Core;
+using Microsoft.EntityFrameworkCore;
+using Microsoft.Extensions.Logging;
+using ZB.MOM.WW.OtOpcUa.Commons.Protos.DeploymentArtifact.V1;
+using ZB.MOM.WW.OtOpcUa.Configuration;
+// The generated service container is itself named DeploymentArtifactService; alias its nested server
+// base so this impl can keep the natural name without colliding with the generated type.
+using GeneratedServiceBase =
+    ZB.MOM.WW.OtOpcUa.Commons.Protos.DeploymentArtifact.V1.DeploymentArtifactService.DeploymentArtifactServiceBase;
+
+namespace ZB.MOM.WW.OtOpcUa.AdminUI.Grpc;
+
+/// <summary>
+///     Central-side artifact serve (per-cluster mesh Phase 3). Streams a deployment's
+///     <c>ArtifactBlob</c> to a driver node fetching in <c>FetchAndCache</c> mode. Registered only on
+///     admin-role nodes (which hold the central SQL connection) and only when
+///     <c>ConfigServe:GrpcListenPort &gt; 0</c>.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>Serves any deployment that has a non-empty blob — status is NOT gated.</b> A node
+///         fetches the artifact <em>during apply</em>, before the deployment is sealed: central seals
+///         only once every node has acked Applied, and a <c>FetchAndCache</c> node cannot ack until it
+///         has fetched. Gating on <c>Sealed</c> would deadlock every such deploy (seal-needs-ack →
+///         ack-needs-fetch → fetch-needs-sealed), and it matches Direct mode, which reads the same
+///         <c>ArtifactBlob</c> from SQL while the row is still <c>AwaitingApplyAcks</c>. (Caught on the
+///         Phase 3 live gate.) Access is already gated by the shared-key interceptor, so there is no
+///         reason to hide a non-sealed deployment.
+///     </para>
+///     <para>
+///         <b>NotFound is indistinguishable across two cases</b> — unknown id and an empty blob. The
+///         first is existence-hiding; the second is the #485 rule (a zero-length artifact is "no
+///         answer," never "a configuration with no drivers"). Streaming empty bytes as if valid is
+///         exactly what that defect class forbids.
+///     </para>
+/// </remarks>
+public sealed class DeploymentArtifactGrpcService : GeneratedServiceBase
+{
+    /// <summary>Chunk size, matching <c>LocalDbDeploymentArtifactCache.ChunkSize</c> (128 KiB).</summary>
+    private const int ChunkSize = 128 * 1024;
+
+    private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
+    private readonly ILogger<DeploymentArtifactGrpcService> _log;
+
+    /// <summary>Initializes a new instance of the <see cref="DeploymentArtifactGrpcService"/> class.</summary>
+    /// <param name="dbFactory">Factory for the central config database holding the artifact bytes.</param>
+    /// <param name="log">Logger.</param>
+    public DeploymentArtifactGrpcService(
+        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
+        ILogger<DeploymentArtifactGrpcService> log)
+    {
+        _dbFactory = dbFactory;
+        _log = log;
+    }
+
+    /// <inheritdoc />
+    public override async Task Fetch(
+        FetchRequest request,
+        IServerStreamWriter<ArtifactChunk> responseStream,
+        ServerCallContext context)
+    {
+        ArgumentNullException.ThrowIfNull(request);
+        ArgumentNullException.ThrowIfNull(responseStream);
+        ArgumentNullException.ThrowIfNull(context);
+
+        if (!Guid.TryParse(request.DeploymentId, out var id))
+            throw new RpcException(new Status(StatusCode.NotFound, "unknown deployment"));
+
+        await using var db = await _dbFactory.CreateDbContextAsync(context.CancellationToken);
+        var row = await db.Deployments
+            .AsNoTracking()
+            .Where(d => d.DeploymentId == id)
+            .Select(d => new { d.ArtifactBlob })
+            .FirstOrDefaultAsync(context.CancellationToken);
+
+        // Existence-hiding + #485: unknown id / empty blob collapse to one NotFound. Status is NOT
+        // gated — a node fetches during apply, before the deployment is sealed (see the class remarks).
+        if (row is null || row.ArtifactBlob.Length == 0)
+        {
+            _log.LogDebug(
+                "Artifact fetch for {DeploymentId} -> NotFound (found={Found}, bytes={Bytes})",
+                request.DeploymentId, row is not null, row?.ArtifactBlob.Length ?? 0);
+            throw new RpcException(new Status(StatusCode.NotFound, "unknown deployment"));
+        }
+
+        for (var offset = 0; offset < row.ArtifactBlob.Length; offset += ChunkSize)
+        {
+            var len = Math.Min(ChunkSize, row.ArtifactBlob.Length - offset);
+            await responseStream.WriteAsync(
+                new ArtifactChunk { Data = ByteString.CopyFrom(row.ArtifactBlob, offset, len) },
+                context.CancellationToken);
+        }
+
+        _log.LogInformation(
+            "Served artifact for {DeploymentId} ({Bytes} bytes, {Chunks} chunks)",
+            request.DeploymentId, row.ArtifactBlob.Length,
+            (row.ArtifactBlob.Length + ChunkSize - 1) / ChunkSize);
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Redundancy/ManualFailoverPageModel.cs

@@ -0,0 +1,134 @@
+using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
+using ZB.MOM.WW.OtOpcUa.Security.Auth;
+
+namespace ZB.MOM.WW.OtOpcUa.AdminUI.Redundancy;
+
+/// <summary>
+/// Presentation state for the Trigger-failover control on the cluster redundancy page.
+/// </summary>
+/// <remarks>
+///     <para>
+///         A pure model behind a thin razor shell — the same split the driver-typed tag editors use,
+///         and for the same reason: this repo has no bUnit (see
+///         <c>PageAuthorizationGuardTests</c>), so anything left inside the <c>.razor</c> is verified
+///         only by driving the page live. The peer guard, the confirm flow, and the
+///         refused-vs-succeeded outcome are consequential enough to be unit-testable, so they live
+///         here.
+///     </para>
+///     <para>
+///         What is deliberately NOT here: the markup authorization gate. It is expressed in the razor
+///         as <c>&lt;AuthorizeView Policy="@ManualFailoverPageModel.RequiredPolicy"&gt;</c> and
+///         re-checked server-side before the call, so <see cref="RequiredPolicy"/> is the single
+///         source of truth for both.
+///     </para>
+/// </remarks>
+public sealed class ManualFailoverPageModel
+{
+    /// <summary>
+    /// The policy gating the control, in markup and in the server-side re-check.
+    /// </summary>
+    /// <remarks>
+    /// <see cref="AdminUiPolicies.FleetAdmin"/> (Administrator only) rather than the
+    /// <see cref="AdminUiPolicies.ConfigEditor"/> the neighbouring cluster-authoring pages use:
+    /// this does not edit configuration, it restarts a production node. ConfigEditor also admits the
+    /// Designer role, which must not be able to bounce the Primary.
+    /// </remarks>
+    public const string RequiredPolicy = AdminUiPolicies.FleetAdmin;
+
+    private readonly IManualFailoverService _service;
+
+    /// <summary>Creates the model over the failover service.</summary>
+    /// <param name="service">The manual-failover seam; faked in tests.</param>
+    public ManualFailoverPageModel(IManualFailoverService service)
+        => _service = service ?? throw new ArgumentNullException(nameof(service));
+
+    /// <summary>The last read cluster snapshot, or <see langword="null"/> if it could not be read.</summary>
+    public ManualFailoverSnapshot? Snapshot { get; private set; }
+
+    /// <summary>Whether the confirmation dialog is open.</summary>
+    public bool ConfirmOpen { get; private set; }
+
+    /// <summary>Status text from the last completed action, if any.</summary>
+    public string? StatusMessage { get; private set; }
+
+    /// <summary>Whether <see cref="StatusMessage"/> reports a failure.</summary>
+    public bool StatusIsError { get; private set; }
+
+    /// <summary>Whether the button is enabled — a peer must exist to take over.</summary>
+    public bool CanFailOver => Snapshot?.CanFailOver == true;
+
+    /// <summary>
+    /// Why the button is disabled, or <see langword="null"/> when it is enabled. Rendered as the
+    /// button's tooltip so a disabled control explains itself.
+    /// </summary>
+    public string? DisabledReason => Snapshot is null
+        ? "Cluster state is unavailable on this node."
+        : Snapshot.CanFailOver
+            ? null
+            : $"Failover needs a driver peer to take over; this mesh has {Snapshot.DriverAddresses.Count} Up driver member(s).";
+
+    /// <summary>Re-reads live cluster state. Never throws — an unreadable cluster disables the button.</summary>
+    public void Refresh()
+    {
+        try
+        {
+            Snapshot = _service.GetSnapshot();
+        }
+        catch (Exception ex)
+        {
+            // A node whose ActorSystem is not yet up (or not a cluster provider) must render the page,
+            // not 500. The button stays disabled with DisabledReason explaining why.
+            Snapshot = null;
+            StatusIsError = true;
+            StatusMessage = $"Could not read cluster state: {ex.Message}";
+        }
+    }
+
+    /// <summary>Opens the confirmation dialog. No-op when the peer guard is closed.</summary>
+    public void RequestFailover()
+    {
+        if (!CanFailOver) return;
+        StatusMessage = null;
+        StatusIsError = false;
+        ConfirmOpen = true;
+    }
+
+    /// <summary>Closes the confirmation dialog without acting.</summary>
+    public void CancelFailover() => ConfirmOpen = false;
+
+    /// <summary>
+    /// Performs the failover the dialog is confirming.
+    /// </summary>
+    /// <param name="actor">The authenticated user name, recorded in the audit event.</param>
+    public async Task ConfirmFailoverAsync(string actor)
+    {
+        if (!ConfirmOpen) return;
+        ConfirmOpen = false;
+
+        try
+        {
+            var target = await _service.FailOverDriverPrimaryAsync(actor).ConfigureAwait(false);
+            if (target is null)
+            {
+                // The service re-evaluates the peer guard against live state, which may have changed
+                // since Refresh(). A refusal is not an error — but it must not read as a success.
+                StatusIsError = true;
+                StatusMessage = "Failover refused: no driver peer is available to take over.";
+            }
+            else
+            {
+                StatusIsError = false;
+                StatusMessage =
+                    $"Failover requested. {target} is leaving the cluster; its peer takes over as Primary "
+                    + "and the node restarts as the youngest member.";
+            }
+        }
+        catch (Exception ex)
+        {
+            StatusIsError = true;
+            StatusMessage = $"Failover failed: {ex.Message}";
+        }
+
+        Refresh();
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs

@@ -4,6 +4,7 @@ using Akka.Event;
 using Microsoft.EntityFrameworkCore;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
+using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
 using ZB.MOM.WW.OtOpcUa.Commons.Types;
 using ZB.MOM.WW.OtOpcUa.Configuration;
@@ -26,6 +27,13 @@ public sealed class AdminOperationsActor : ReceiveActor
     private readonly IActorRef _coordinator;
     private readonly IReadOnlyDictionary<string, IDriverProbe> _probesByType;
     private readonly TagConfigValidationMode _tagConfigValidationMode;
+
+    /// <summary>
+    ///     Where central→node commands are handed off (per-cluster mesh Phase 2), or
+    ///     <see langword="null"/> to publish on the mediator directly.
+    /// </summary>
+    private readonly IActorRef? _meshRouter;
+
     private readonly ILoggingAdapter _log = Context.GetLogger();
 
     /// <summary>Creates actor props for the admin operations actor.</summary>
@@ -33,29 +41,38 @@ public sealed class AdminOperationsActor : ReceiveActor
     /// <param name="coordinator">Reference to the deployment coordinator actor.</param>
     /// <param name="probes">Driver probes registered in DI; keyed by DriverType (case-insensitive).</param>
     /// <param name="tagConfigValidationMode">Deploy-gate TagConfig strictness mode (Warn default | Error opt-in).</param>
+    /// <param name="meshRouter">
+    ///     Central comm actor commands are handed to, or <see langword="null"/> to publish on the
+    ///     DistributedPubSub mediator directly (the pre-Phase-2 behaviour).
+    /// </param>
     /// <returns>Props configured to create an AdminOperationsActor.</returns>
     public static Props Props(
         IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
         IActorRef coordinator,
         IEnumerable<IDriverProbe> probes,
-        TagConfigValidationMode tagConfigValidationMode = TagConfigValidationMode.Warn) =>
-        Akka.Actor.Props.Create(() => new AdminOperationsActor(dbFactory, coordinator, probes, tagConfigValidationMode));
+        TagConfigValidationMode tagConfigValidationMode = TagConfigValidationMode.Warn,
+        IActorRef? meshRouter = null) =>
+        Akka.Actor.Props.Create(() =>
+            new AdminOperationsActor(dbFactory, coordinator, probes, tagConfigValidationMode, meshRouter));
 
     /// <summary>Initializes a new instance of the AdminOperationsActor.</summary>
     /// <param name="dbFactory">Factory for creating config database contexts.</param>
     /// <param name="coordinator">Reference to the deployment coordinator actor.</param>
     /// <param name="probes">Driver probes registered in DI; keyed by DriverType (case-insensitive).</param>
     /// <param name="tagConfigValidationMode">Deploy-gate TagConfig strictness mode (Warn default | Error opt-in).</param>
+    /// <param name="meshRouter">Central comm actor, or <see langword="null"/> for the mediator.</param>
     public AdminOperationsActor(
         IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
         IActorRef coordinator,
         IEnumerable<IDriverProbe> probes,
-        TagConfigValidationMode tagConfigValidationMode = TagConfigValidationMode.Warn)
+        TagConfigValidationMode tagConfigValidationMode = TagConfigValidationMode.Warn,
+        IActorRef? meshRouter = null)
     {
         _dbFactory = dbFactory;
         _coordinator = coordinator;
         _probesByType = probes.ToDictionary(p => p.DriverType, StringComparer.OrdinalIgnoreCase);
         _tagConfigValidationMode = tagConfigValidationMode;
+        _meshRouter = meshRouter;
 
         ReceiveAsync<StartDeployment>(HandleStartDeploymentAsync);
         ReceiveAsync<TestDriverConnect>(HandleTestDriverConnectAsync);
@@ -65,6 +82,23 @@ public sealed class AdminOperationsActor : ReceiveActor
         Receive<ShelveAlarmCommand>(HandleShelveAlarm);
     }
 
+    /// <summary>
+    ///     Hands a central→node command to the mesh transport router, or publishes it directly when
+    ///     no router is wired.
+    /// </summary>
+    /// <remarks>
+    ///     The fallback keeps ~12 existing tests (and any node wired before per-cluster mesh Phase 2)
+    ///     on exactly the pre-Phase-2 behaviour. It is not a permanent seam: Phase 6 deletes it along
+    ///     with the DistributedPubSub branch and the single mesh.
+    /// </remarks>
+    /// <param name="topic">The legacy DistributedPubSub topic.</param>
+    /// <param name="message">The command.</param>
+    private void Dispatch(string topic, object message)
+    {
+        if (_meshRouter is not null) _meshRouter.Tell(new MeshCommand(topic, message));
+        else DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(topic, message));
+    }
+
     /// <summary>
     ///     AdminUI Acknowledge path. Maps the control-plane command to a
     ///     <see cref="AlarmCommand"/> (<c>Operation = "Acknowledge"</c>) and publishes it onto the
@@ -85,7 +119,7 @@ public sealed class AdminOperationsActor : ReceiveActor
                 Comment: msg.Comment,
                 UnshelveAtUtc: null);
 
-            DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(AlarmCommandsTopic.Name, alarmCmd));
+            Dispatch(AlarmCommandsTopic.Name, alarmCmd);
 
             _log.Info("AdminOps: Acknowledge published for alarm {AlarmId} by {User}", msg.AlarmId, msg.User);
             replyTo.Tell(new AcknowledgeAlarmResult(true, null, msg.CorrelationId));
@@ -132,7 +166,7 @@ public sealed class AdminOperationsActor : ReceiveActor
                 Comment: msg.Comment,
                 UnshelveAtUtc: unshelveAt);
 
-            DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(AlarmCommandsTopic.Name, alarmCmd));
+            Dispatch(AlarmCommandsTopic.Name, alarmCmd);
 
             _log.Info("AdminOps: {Operation} published for alarm {AlarmId} by {User}", operation, msg.AlarmId, msg.User);
             replyTo.Tell(new ShelveAlarmResult(true, null, msg.CorrelationId));
@@ -394,7 +428,7 @@ public sealed class AdminOperationsActor : ReceiveActor
         {
             // Broadcast to every DriverHostActor on every node via the driver-control DPS topic.
             // Only the host that owns the instance will act; others ignore it (id not found in _children).
-            DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(DriverControlTopic.Name, msg));
+            Dispatch(DriverControlTopic.Name, msg);
 
             await using var db = await _dbFactory.CreateDbContextAsync();
             db.ConfigEdits.Add(new ConfigEdit
@@ -424,7 +458,7 @@ public sealed class AdminOperationsActor : ReceiveActor
         try
         {
             // Broadcast to every DriverHostActor; only the one owning the instance reacts.
-            DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(DriverControlTopic.Name, msg));
+            Dispatch(DriverControlTopic.Name, msg);
 
             await using var db = await _dbFactory.CreateDbContextAsync();
             db.ConfigEdits.Add(new ConfigEdit

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs

@@ -0,0 +1,308 @@
+using System.Collections.Immutable;
+using Akka.Actor;
+using Akka.Cluster.Tools.Client;
+using Akka.Cluster.Tools.PublishSubscribe;
+using Akka.Event;
+using Microsoft.EntityFrameworkCore;
+using ZB.MOM.WW.OtOpcUa.Cluster;
+using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
+using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
+using ZB.MOM.WW.OtOpcUa.Configuration;
+
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
+
+/// <summary>
+///     Central-side end of the central↔node command boundary (per-cluster mesh Phase 2). Routes
+///     every central→node command out — over DistributedPubSub or ClusterClient, per
+///     <see cref="MeshTransportOptions.Mode"/> — and forwards inbound <see cref="ApplyAck"/>s to the
+///     deploy coordinator singleton.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>Registered per node, NOT as a cluster singleton.</b> A driver node's ClusterClient
+///         rotates across contact points and must find a live comm actor at whichever central node
+///         answers. A singleton would be reachable only through the one node hosting it, and the
+///         rotation would fail silently against the other.
+///     </para>
+///     <para>
+///         <b>Exactly one ClusterClient, fleet-wide.</b> See <see cref="RebuildClient"/> — this is
+///         the single most important thing to understand before changing this actor.
+///     </para>
+/// </remarks>
+public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
+{
+    private const string RefreshTimerKey = "mesh-contact-refresh";
+
+    private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
+    private readonly MeshTransportOptions _options;
+    private readonly IMeshClusterClientFactory _clientFactory;
+    private readonly Func<IActorRef?> _coordinator;
+    private readonly ILoggingAdapter _log = Context.GetLogger();
+    private readonly bool _useClusterClient;
+
+    private IActorRef? _client;
+    private ImmutableHashSet<string> _currentContacts = ImmutableHashSet<string>.Empty;
+    private CancellationTokenSource _lifecycle = new();
+
+    /// <summary>Gets the timer scheduler driving the periodic contact refresh.</summary>
+    public ITimerScheduler Timers { get; set; } = null!;
+
+    /// <summary>Creates the props for this actor.</summary>
+    /// <param name="dbFactory">Factory for the config database holding <c>ClusterNode</c> rows.</param>
+    /// <param name="options">The bound mesh-transport options.</param>
+    /// <param name="clientFactory">Creates the ClusterClient; substituted in tests.</param>
+    /// <param name="coordinator">
+    ///     Resolves the deploy-coordinator singleton proxy. A <see cref="Func{TResult}"/> rather than
+    ///     an <see cref="IActorRef"/> so this actor never depends on registration order at wiring
+    ///     time — it resolves on first use and caches.
+    /// </param>
+    /// <returns>The props.</returns>
+    public static Props Props(
+        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
+        MeshTransportOptions options,
+        IMeshClusterClientFactory clientFactory,
+        Func<IActorRef?> coordinator) =>
+        Akka.Actor.Props.Create(() =>
+            new CentralCommunicationActor(dbFactory, options, clientFactory, coordinator));
+
+    /// <summary>Initializes a new instance of the <see cref="CentralCommunicationActor"/> class.</summary>
+    /// <param name="dbFactory">Factory for the config database.</param>
+    /// <param name="options">The bound mesh-transport options.</param>
+    /// <param name="clientFactory">Creates the ClusterClient.</param>
+    /// <param name="coordinator">Lazily resolves the deploy-coordinator singleton proxy.</param>
+    public CentralCommunicationActor(
+        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
+        MeshTransportOptions options,
+        IMeshClusterClientFactory clientFactory,
+        Func<IActorRef?> coordinator)
+    {
+        _dbFactory = dbFactory;
+        _options = options;
+        _clientFactory = clientFactory;
+        _coordinator = coordinator;
+        _useClusterClient = string.Equals(
+            options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
+
+        Receive<MeshCommand>(HandleMeshCommand);
+        Receive<ApplyAck>(HandleApplyAck);
+        Receive<RefreshContacts>(_ => LoadContactsFromDb());
+        Receive<ContactsLoaded>(HandleContactsLoaded);
+
+        // A faulted LoadContactsFromDb task is piped here as a Status.Failure. Without this handler
+        // the failure is an unhandled message (debug level only) and the refresh fails silently —
+        // an operator cannot distinguish "no nodes configured" from "the database is down", and the
+        // contact set silently freezes at whatever it last held.
+        Receive<Status.Failure>(f => _log.Warning(
+            f.Cause,
+            "Failed to load ClusterNode contact points; the mesh ClusterClient contact set was NOT "
+            + "refreshed and may be stale or empty. Commands may be dropped until the next refresh"));
+
+        Receive<SubscribeAck>(_ => { /* DPS subscribe confirmation */ });
+    }
+
+    /// <inheritdoc />
+    protected override void PreStart()
+    {
+        if (!_useClusterClient)
+        {
+            _log.Info(
+                "Mesh transport is {Mode}: commands are published on DistributedPubSub and no "
+                + "ClusterClient is created", _options.Mode);
+            return;
+        }
+
+        Timers.StartPeriodicTimer(
+            RefreshTimerKey,
+            new RefreshContacts(),
+            TimeSpan.Zero,
+            TimeSpan.FromSeconds(_options.ContactRefreshSeconds));
+    }
+
+    /// <inheritdoc />
+    protected override void PostStop()
+    {
+        _lifecycle.Cancel();
+        _lifecycle.Dispose();
+    }
+
+    private void HandleMeshCommand(MeshCommand cmd)
+    {
+        if (!_useClusterClient)
+        {
+            DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(cmd.Topic, cmd.Message));
+            return;
+        }
+
+        if (_client is null)
+        {
+            // App-level drop, matching the sister project's decision and this repo's buffer-size = 0.
+            // The command is NOT queued and will NOT be retried: a deploy fails at the coordinator's
+            // apply deadline naming the silent nodes, which is a better operator signal than a
+            // command arriving minutes late against a config the DB already recorded as failed.
+            _log.Warning(
+                "No mesh ClusterClient — dropping {MessageType}. The last contact refresh produced "
+                + "no usable contact point from the enabled, non-maintenance ClusterNode rows; the "
+                + "command is not buffered and will not be retried",
+                cmd.Message.GetType().Name);
+            return;
+        }
+
+        // SendToAll, NEVER Send. Today's DPS publish reaches EVERY DriverHostActor — there is no
+        // ClusterId or node filter on the node side at all; scoping happens later, inside the
+        // artifact (DeploymentArtifact.ResolveClusterScope). ClusterClient.Send delivers to exactly
+        // ONE registered actor, which would deploy to a single node of the fleet while every other
+        // node silently kept its old configuration — and the deployment would still seal green if
+        // the ack set happened to be satisfied.
+        _client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, cmd.Message));
+    }
+
+    private void HandleApplyAck(ApplyAck ack)
+    {
+        var coordinator = _coordinator();
+        if (coordinator is null)
+        {
+            _log.Warning(
+                "Received ApplyAck for {DeploymentId} from {NodeId} but the deploy coordinator is "
+                + "not resolvable on this node; the ack is lost and the deployment will time out "
+                + "naming a node that applied successfully",
+                ack.DeploymentId, ack.NodeId);
+            return;
+        }
+
+        // Forward, not Tell: preserves the original sender across the hop so the coordinator sees
+        // the node rather than this relay.
+        coordinator.Forward(ack);
+    }
+
+    private void LoadContactsFromDb()
+    {
+        var self = Self;
+        CancellationToken ct;
+        try
+        {
+            ct = _lifecycle.Token;
+        }
+        catch (ObjectDisposedException)
+        {
+            return; // Stopping.
+        }
+
+        var systemName = Context.System.Name;
+        var dbFactory = _dbFactory;
+
+        // Off the actor thread, result piped back as a message. A synchronous DB read here would
+        // block every command dispatch behind a slow or unreachable SQL Server.
+        Task.Run(async () =>
+        {
+            await using var db = await dbFactory.CreateDbContextAsync(ct).ConfigureAwait(false);
+            var rows = await db.ClusterNodes
+                .AsNoTracking()
+                .Where(n => n.Enabled && !n.MaintenanceMode)
+                .Select(n => new { n.NodeId, n.Host, n.AkkaPort })
+                .ToListAsync(ct)
+                .ConfigureAwait(false);
+
+            var contacts = new List<string>(rows.Count);
+            var malformed = new List<string>();
+            foreach (var row in rows)
+            {
+                var address = $"akka.tcp://{systemName}@{row.Host}:{row.AkkaPort}/system/receptionist";
+
+                // Parse up front, per row, inside the loop's own guard. A single malformed row must
+                // not abort the whole refresh and leave the contact set half-built — the sister
+                // project shipped that bug and its regression test deliberately orders the bad row
+                // first.
+                if (ActorPath.TryParse(address, out _)) contacts.Add(address);
+                else malformed.Add($"{row.NodeId} -> {address}");
+            }
+
+            return new ContactsLoaded(contacts, malformed);
+        }, ct).PipeTo(self);
+    }
+
+    private void HandleContactsLoaded(ContactsLoaded msg)
+    {
+        foreach (var bad in msg.Malformed)
+        {
+            _log.Warning(
+                "ClusterNode {Row} does not yield a usable receptionist address; skipping it in this "
+                + "refresh (other nodes are unaffected). Check the row's Host and AkkaPort", bad);
+        }
+
+        var contacts = msg.Contacts.ToImmutableHashSet();
+
+        if (contacts.IsEmpty)
+        {
+            _log.Warning(
+                "No usable ClusterNode contact points; the mesh ClusterClient was not created and "
+                + "every central→node command will be dropped until a refresh finds one");
+            return;
+        }
+
+        if (_client is not null && _currentContacts.SetEquals(contacts)) return;
+
+        RebuildClient(contacts);
+    }
+
+    /// <summary>Stops any existing client and creates a replacement for the new contact set.</summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b>TODO(Phase 6): one client per application <c>Cluster</c>.</b> Today this actor
+    ///         creates exactly <b>one</b> client whose contacts are the whole fleet, and that is not
+    ///         a compromise — it is the only correct shape while the fleet is a single Akka mesh.
+    ///     </para>
+    ///     <para>
+    ///         A <c>ClusterClientReceptionist</c> serves its entire cluster, so
+    ///         <c>SendToAll</c> reaches every registered node-comm actor in the mesh <i>regardless of
+    ///         which node's address was used as the contact point</i>. One client per Cluster would
+    ///         therefore fan each command out once per cluster — N× duplicate
+    ///         <c>DispatchDeployment</c> and N× duplicate <c>ApplyAck</c>. Phase 6 splits the meshes,
+    ///         at which point per-cluster clients become both correct and necessary.
+    ///     </para>
+    ///     <para>
+    ///         Corollary worth stating because it is easy to assume otherwise: <b>the contact set
+    ///         does not scope delivery.</b> Excluding a maintenance-mode node from the contacts does
+    ///         not stop it receiving commands — it still receives them, exactly as it does under
+    ///         today's DPS broadcast. The filter is about which receptionists are worth <i>dialling</i>
+    ///         (a node in maintenance may be switched off), not about who gets the message.
+    ///     </para>
+    /// </remarks>
+    /// <param name="contacts">The receptionist addresses to dial.</param>
+    private void RebuildClient(ImmutableHashSet<string> contacts)
+    {
+        if (_client is not null)
+        {
+            _log.Info("Mesh contact set changed; replacing the ClusterClient");
+            Context.Stop(_client);
+
+            // Cleared now: if the create below throws, a stale ref would route commands into a
+            // stopping actor and they would vanish without a warning.
+            _client = null;
+            _currentContacts = ImmutableHashSet<string>.Empty;
+        }
+
+        try
+        {
+            var paths = contacts.Select(ActorPath.Parse).ToImmutableHashSet();
+            _client = _clientFactory.Create(Context.System, paths);
+            _currentContacts = contacts;
+            _log.Info("Mesh ClusterClient created with {Count} contact point(s)", contacts.Count);
+        }
+        catch (Exception ex)
+        {
+            _log.Error(ex,
+                "Failed to create the mesh ClusterClient; central→node commands are dropped until "
+                + "the next contact refresh succeeds");
+        }
+    }
+
+    /// <summary>Internal tick asking for a contact-point refresh.</summary>
+    public sealed record RefreshContacts;
+
+    /// <summary>Result of a contact-point load, piped back from the DB read.</summary>
+    /// <param name="Contacts">Receptionist addresses that parsed cleanly.</param>
+    /// <param name="Malformed">Rows that did not, described for the log.</param>
+    public sealed record ContactsLoaded(
+        IReadOnlyList<string> Contacts,
+        IReadOnlyList<string> Malformed);
+}

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/IMeshClusterClientFactory.cs

@@ -0,0 +1,47 @@
+using System.Collections.Immutable;
+using Akka.Actor;
+using Akka.Cluster.Tools.Client;
+
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
+
+/// <summary>
+///     Creates the <c>ClusterClient</c> central uses to reach driver nodes. Exists as a seam so
+///     <c>CentralCommunicationActor</c> can be tested against a <c>TestProbe</c> without standing up
+///     a second cluster.
+/// </summary>
+public interface IMeshClusterClientFactory
+{
+    /// <summary>Creates a client for the given receptionist contact points.</summary>
+    /// <param name="system">The actor system to spawn the client in.</param>
+    /// <param name="contacts">Receptionist actor paths, one per reachable node.</param>
+    /// <returns>The new client's <see cref="IActorRef"/>.</returns>
+    IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts);
+}
+
+/// <inheritdoc />
+public sealed class DefaultMeshClusterClientFactory : IMeshClusterClientFactory
+{
+    /// <summary>
+    ///     Per-incarnation generation counter, appended to the actor name.
+    /// </summary>
+    /// <remarks>
+    ///     <c>Context.Stop</c> of the previous client is <b>asynchronous</b> — its actor name stays
+    ///     reserved until termination completes — so recreating the same name while handling a single
+    ///     message throws <see cref="InvalidActorNameException"/>. That is exactly what a contact-set
+    ///     change does: stop the old client, create the replacement, both in one handler. A
+    ///     generation suffix makes every incarnation's name unique by construction. Ported from the
+    ///     sister project, where it was a shipped bug fix rather than foresight.
+    /// </remarks>
+    private long _generation;
+
+    /// <inheritdoc />
+    public IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts)
+    {
+        ArgumentNullException.ThrowIfNull(system);
+        ArgumentNullException.ThrowIfNull(contacts);
+
+        var settings = ClusterClientSettings.Create(system).WithInitialContacts(contacts);
+        var name = $"mesh-client-{Interlocked.Increment(ref _generation)}";
+        return system.ActorOf(ClusterClient.Props(settings), name);
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs

@@ -1,9 +1,9 @@
 using Akka.Actor;
-using Akka.Cluster;
 using Akka.Cluster.Tools.PublishSubscribe;
 using Akka.Event;
 using Microsoft.EntityFrameworkCore;
 using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
+using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 using ZB.MOM.WW.OtOpcUa.Commons.Types;
 using ZB.MOM.WW.OtOpcUa.Configuration;
 using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
@@ -18,8 +18,40 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
 /// has acked Applied. Per-node ACKs are persisted in <c>NodeDeploymentState</c> so a failover of
 /// this singleton can recover in-flight state from the DB.
 ///
-/// Discovery of the "expected ACK set" comes from <c>Akka.Cluster.State.Members</c> filtered by
-/// the <c>driver</c> role — the DB does not own per-node role assignment.
+/// Discovery of the "expected ACK set" comes from the <b>enabled <c>ClusterNode</c> rows</b>, not
+/// from <c>Akka.Cluster.State.Members</c> (per-cluster mesh Phase 1). Central must be able to name
+/// the nodes a deployment is for without sharing a gossip ring with them — Phase 2 splits the fleet
+/// into one mesh per <c>Cluster</c>, after which central can no longer see a site node's membership
+/// at all. This was the coordinator's last genuinely mesh-bound dependency.
+///
+/// <para>
+/// <b>Behaviour change:</b> a configured node that is switched off is now <i>expected</i>, so a
+/// deployment dispatched while it is down fails at the apply deadline instead of sealing green
+/// without it. That is deliberate — under the membership rule the operator was told the fleet was
+/// deployed when it was not. <c>ClusterNode.MaintenanceMode = 1</c> is the escape hatch for a node
+/// taken down for maintenance.
+/// </para>
+/// <para>
+/// <b>Not <c>Enabled = 0</c>.</b> That was the intended hatch until the Phase 1 live gate found it
+/// unusable: <c>DraftValidator.ValidateClusterTopology</c> requires the enabled-node count to equal
+/// <c>ServerCluster.NodeCount</c>, so clearing <c>Enabled</c> on either node of a Warm/Hot pair
+/// rejects the deployment outright — and every cluster in the target topology is a pair. The two
+/// rules were independently reasonable and contradictory together, so the meanings were split:
+/// <c>Enabled</c> = part of the declared topology, <c>MaintenanceMode</c> = do not expect it now.
+/// </para>
+/// <para>
+/// <b>Every <c>ClusterNode</c> row is assumed to be a driver node.</b> The membership rule filtered
+/// on the <c>driver</c> role; the DB has no per-node role column, and deliberately does not gain one
+/// here (a second declaration of node roles would drift from <c>Cluster:Roles</c> in the node's own
+/// appsettings). The assumption holds because the entity's whole shape — <c>Host</c>,
+/// <c>OpcUaPort</c>, <c>ApplicationUri</c>, <c>ServiceLevelBase</c> — describes an OPC UA server
+/// node. An admin-only node must not be given a <c>ClusterNode</c> row: it would be expected to ack
+/// a deployment it has no <c>DriverHostActor</c> to apply, and every deploy would time out.
+/// </para>
+/// <para>
+/// <c>ServerCluster.Enabled</c> is <b>not</b> consulted — nothing else in the codebase consults it,
+/// so honouring it here would invent semantics. Disable the nodes, not the cluster row.
+/// </para>
 /// </summary>
 public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
 {
@@ -29,6 +61,17 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
 
     private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
     private readonly TimeSpan _applyDeadline;
+
+    /// <summary>
+    ///     Where central→node commands are handed off (per-cluster mesh Phase 2), or
+    ///     <see langword="null"/> to publish on the mediator directly.
+    /// </summary>
+    /// <remarks>
+    ///     Null in tests and on any node wired before Phase 2, where the fallback reproduces the
+    ///     pre-Phase-2 behaviour exactly. The fallback is not a permanent seam — Phase 6 deletes it
+    ///     along with the DistributedPubSub branch and the single mesh.
+    /// </remarks>
+    private readonly IActorRef? _meshRouter;
     private readonly ILoggingAdapter _log = Context.GetLogger();
     // NodeId equality here is case-insensitive (by Value) to match the case-insensitive ClusterId/
     // NodeId scoping in DeploymentArtifact.ResolveClusterScope — so an ack from a node whose address
@@ -47,22 +90,31 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
     /// <param name="dbFactory">The database context factory for accessing configuration data.</param>
     /// <param name="applyDeadline">The timeout for waiting for apply acknowledgments from cluster nodes.</param>
     /// <returns>The Akka.NET <see cref="Akka.Actor.Props"/> used to spawn this actor.</returns>
+    /// <param name="meshRouter">
+    ///     Central comm actor the dispatch is handed to, or <see langword="null"/> to publish on the
+    ///     DistributedPubSub mediator directly (the pre-Phase-2 behaviour).
+    /// </param>
     public static Props Props(
         IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
-        TimeSpan? applyDeadline = null) =>
-        Akka.Actor.Props.Create(() => new ConfigPublishCoordinator(dbFactory, applyDeadline ?? DefaultApplyDeadline));
+        TimeSpan? applyDeadline = null,
+        IActorRef? meshRouter = null) =>
+        Akka.Actor.Props.Create(() =>
+            new ConfigPublishCoordinator(dbFactory, applyDeadline ?? DefaultApplyDeadline, meshRouter));
 
     /// <summary>
     ///     Initializes a new instance of the <see cref="ConfigPublishCoordinator"/> class.
     /// </summary>
     /// <param name="dbFactory">The database context factory for persisting deployment state.</param>
     /// <param name="applyDeadline">The timeout for waiting for per-node apply acknowledgments.</param>
+    /// <param name="meshRouter">Central comm actor, or <see langword="null"/> for the mediator.</param>
     public ConfigPublishCoordinator(
         IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
-        TimeSpan applyDeadline)
+        TimeSpan applyDeadline,
+        IActorRef? meshRouter = null)
     {
         _dbFactory = dbFactory;
         _applyDeadline = applyDeadline;
+        _meshRouter = meshRouter;
         Receive<DispatchDeployment>(HandleDispatch);
         Receive<ApplyAck>(HandleAck);
         Receive<DeadlineElapsed>(HandleDeadline);
@@ -119,11 +171,15 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
     {
         _current = msg.DeploymentId;
         _acks.Clear();
-        _expectedAcks = DiscoverDriverNodes();
 
         // Seed NodeDeploymentState rows so a failover knows which nodes were expected to ack.
         using (var db = _dbFactory.CreateDbContext())
         {
+            // Discovery reuses this context rather than opening a second one — it reads the same
+            // ClusterNode table the NodeDeploymentState rows below are FK-bound to, so a single
+            // context keeps the expected set and the seeded rows consistent by construction.
+            _expectedAcks = DiscoverDriverNodes(db);
+
             foreach (var node in _expectedAcks)
             {
                 db.NodeDeploymentStates.Add(new NodeDeploymentState
@@ -137,14 +193,22 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
             db.SaveChanges();
         }
 
-        DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(DeploymentsTopic, msg));
+        // Per-cluster mesh Phase 2: hand off to the transport router rather than publishing
+        // directly, so this actor no longer knows which transport is in force.
+        if (_meshRouter is not null) _meshRouter.Tell(new MeshCommand(DeploymentsTopic, msg));
+        else DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(DeploymentsTopic, msg));
         Timers.StartSingleTimer(DeadlineTimerKey, new DeadlineElapsed(msg.DeploymentId), _applyDeadline);
 
         if (_expectedAcks.Count == 0)
         {
-            // No driver-role members. Seal immediately — the alternative is hanging forever
-            // waiting for ACKs that will never come.
-            _log.Warning("DispatchDeployment {Id}: no driver-role members in cluster; sealing empty",
+            // No enabled ClusterNode rows. Seal immediately — the alternative is hanging forever
+            // waiting for ACKs that will never come. Note this is now a *configuration* error
+            // rather than a topology observation: under the old membership rule an empty set meant
+            // "no driver nodes have joined yet", which could resolve itself; an empty set here means
+            // the fleet has no enabled nodes registered, which cannot.
+            _log.Warning(
+                "DispatchDeployment {Id}: no enabled ClusterNode rows registered; sealing empty. " +
+                "Nothing will receive this deployment — check the fleet's node configuration",
                 msg.DeploymentId);
             SealDeployment();
         }
@@ -233,8 +297,15 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
         using var db = _dbFactory.CreateDbContext();
         UpdateDeploymentStatus(db, _current.Value, DeploymentStatus.TimedOut);
         db.SaveChanges();
-        _log.Warning("Deployment {Id} timed out after {Deadline} ({Acked}/{Total} acks landed)",
-            _current.Value, _applyDeadline, _acks.Count, _expectedAcks.Count);
+        // Name the silent nodes, not just the counts. Since the expected set became DB-derived this
+        // is the failure an operator meets after deploying while a configured node is switched off,
+        // so it has to say which node — "4/5 acks landed" leaves them reading logs on five machines.
+        var missing = _expectedAcks.Where(n => !_acks.ContainsKey(n)).Select(n => n.Value).Order().ToList();
+        _log.Warning(
+            "Deployment {Id} timed out after {Deadline} ({Acked}/{Total} acks landed). " +
+            "No ack from: {MissingNodes}. Each is an enabled, non-maintenance ClusterNode row — start " +
+            "the node, or set ClusterNode.MaintenanceMode = 1 if it is down for maintenance, then redeploy",
+            _current.Value, _applyDeadline, _acks.Count, _expectedAcks.Count, string.Join(", ", missing));
         ResetForNext();
     }
 
@@ -259,22 +330,29 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
         if (sealNow) d.SealedAtUtc = DateTime.UtcNow;
     }
 
-    private HashSet<NodeId> DiscoverDriverNodes()
-    {
-        var cluster = Akka.Cluster.Cluster.Get(Context.System);
-        var nodes = new HashSet<NodeId>(NodeIdComparer);
-        foreach (var member in cluster.State.Members)
-        {
-            if (member.Status is not (MemberStatus.Up or MemberStatus.Joining)) continue;
-            if (!member.Roles.Contains("driver")) continue;
-            var host = member.Address.Host;
-            if (string.IsNullOrWhiteSpace(host)) continue;
-            // Match ClusterRoleInfo's NodeId derivation (host:port) so DriverHostActor's
-            // self-identification and the coordinator's expected-ack set agree.
-            nodes.Add(NodeId.Parse($"{host}:{member.Address.Port ?? 0}"));
-        }
-        return nodes;
-    }
+    /// <summary>
+    ///     The set of nodes expected to ack this deployment: every <b>enabled</b> <c>ClusterNode</c>
+    ///     row. See the class remarks for why this is DB-derived rather than membership-derived, and
+    ///     for the two assumptions it rests on (every row is a driver node; <c>Enabled</c> is the
+    ///     maintenance hatch).
+    /// </summary>
+    /// <remarks>
+    ///     <c>ClusterNode.NodeId</c> is already in the <c>host:port</c> identifier space the
+    ///     membership rule derived — the rig seeds <c>central-1:4053</c> and so on, and
+    ///     <c>NodeDeploymentState.NodeId</c> is FK-bound to this column, so the fact that the
+    ///     membership-derived rows have always satisfied that FK is standing proof the two sets
+    ///     agree. It also has to stay that way: <c>DriverHostActor</c> self-identifies from
+    ///     <c>ClusterRoleInfo</c>'s <c>host:port</c>, so a row whose NodeId is anything else is a
+    ///     node whose acks will never match its expected-ack entry.
+    /// </remarks>
+    private static HashSet<NodeId> DiscoverDriverNodes(OtOpcUaConfigDbContext db) =>
+        db.ClusterNodes
+            .AsNoTracking()
+            .Where(n => n.Enabled && !n.MaintenanceMode)
+            .Select(n => n.NodeId)
+            .ToList()
+            .Select(NodeId.Parse)
+            .ToHashSet(NodeIdComparer);
 
     /// <summary>Case-insensitive <see cref="NodeId"/> equality (by <see cref="NodeId.Value"/>),
     /// matching the case-insensitive scoping in <c>DeploymentArtifact.ResolveClusterScope</c> so the

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconciler.cs

@@ -0,0 +1,138 @@
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
+
+/// <summary>What kind of disagreement was found between a node's real address and its row.</summary>
+public enum AddressMismatchKind
+{
+    /// <summary>
+    ///     The row was matched to a live member by <c>NodeId</c>, but its <c>Host</c> / <c>AkkaPort</c>
+    ///     columns name a different address. Phase 2 dials those columns, so central would dial
+    ///     somewhere the node is not — and the symptom there is a silent absence of acks.
+    /// </summary>
+    RowDialTargetDisagrees,
+
+    /// <summary>
+    ///     A driver-role member is in the cluster with no enabled <c>ClusterNode</c> row at its
+    ///     address. Its deployment acks are discarded (see <c>ConfigPublishCoordinator</c>), so it
+    ///     silently receives deployments it is never credited for.
+    /// </summary>
+    RunningNodeHasNoRow,
+
+    /// <summary>
+    ///     An enabled row names a node that is not currently a driver-role member. Either it is down
+    ///     — in which case the next deployment fails at the apply deadline — or it binds an address
+    ///     other than the one its row claims.
+    /// </summary>
+    EnabledRowNotInCluster,
+}
+
+/// <summary>One disagreement, with enough detail to act on without opening the DB.</summary>
+/// <param name="Kind">Which disagreement this is.</param>
+/// <param name="NodeId">The <c>host:port</c> identity the finding is about.</param>
+/// <param name="Detail">Human-readable specifics, including what to do about it.</param>
+public sealed record ClusterNodeAddressMismatch(AddressMismatchKind Kind, string NodeId, string Detail);
+
+/// <summary>A <c>ClusterNode</c> row reduced to the address facts this check cares about.</summary>
+/// <param name="NodeId">The row's primary key, in the <c>host:port</c> space.</param>
+/// <param name="Host">The host central will dial in Phase 2.</param>
+/// <param name="AkkaPort">The remoting port central will dial in Phase 2.</param>
+public sealed record ClusterNodeAddress(string NodeId, string Host, int AkkaPort);
+
+/// <summary>
+///     Compares the addresses driver nodes actually occupy against the <c>ClusterNode</c> rows that
+///     claim to describe them.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>Why this exists.</b> <c>ClusterNode.AkkaPort</c> and the node's own <c>Cluster:Port</c>
+///         are the same fact stored in two places, and nothing makes them agree. A node that binds
+///         4054 while its row says 4053 is unreachable from central once Phase 2 dials the row
+///         instead of gossiping — and an unenforced duplicated address is the exact shape of the
+///         <c>Modbus</c>/<c>ModbusTcp</c> and <c>TwinCat</c>/<c>Focas</c> drifts already recorded in
+///         this repo. Since Phase 1 also made the rows the deploy path's expected-ack set, a drifted
+///         row already costs a failed deployment today.
+///     </para>
+///     <para>
+///         <b>It runs on an admin node</b>, comparing rows against the membership an admin node can
+///         already see, rather than having each driver node assert its own row — Phase 4 removes the
+///         driver nodes' ConfigDb connection entirely, so a self-assertion written there would have
+///         to be deleted again.
+///     </para>
+///     <para>
+///         <b>Phase 2 must revisit this.</b> Once the fleet splits into one mesh per cluster, an
+///         admin node no longer sees site members at all, and every site row would report
+///         <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> forever. The check then has to
+///         move to whatever transport replaces gossip — or be scoped to the central mesh.
+///     </para>
+/// </remarks>
+public static class ClusterNodeAddressReconciler
+{
+    /// <summary>
+    ///     Reconciles observed driver-role member addresses against the configured rows.
+    /// </summary>
+    /// <param name="observedDriverMembers">
+    ///     <c>host</c>/<c>port</c> of every Up member carrying the <c>driver</c> role. Admin-only
+    ///     members must be excluded — they legitimately have no <c>ClusterNode</c> row, and including
+    ///     them would report a mismatch for correct configuration.
+    /// </param>
+    /// <param name="rows">Every <c>ClusterNode</c> row, enabled or not.</param>
+    /// <param name="expectedPresentNodeIds">
+    ///     The subset of <paramref name="rows"/> that should currently be in the cluster: enabled and
+    ///     not in maintenance. Excluded rows are deliberately still matched against membership (a
+    ///     disabled row for a running node is worth knowing) but never reported as
+    ///     <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> — being absent is the whole point
+    ///     of disabling one or putting it in maintenance.
+    /// </param>
+    /// <returns>Every disagreement found, ordered by node id for stable logging.</returns>
+    public static IReadOnlyList<ClusterNodeAddressMismatch> Reconcile(
+        IReadOnlyCollection<(string Host, int Port)> observedDriverMembers,
+        IReadOnlyCollection<ClusterNodeAddress> rows,
+        IReadOnlyCollection<string> expectedPresentNodeIds)
+    {
+        var byNodeId = new Dictionary<string, ClusterNodeAddress>(StringComparer.OrdinalIgnoreCase);
+        foreach (var row in rows) byNodeId[row.NodeId] = row;
+        var expectedPresent = new HashSet<string>(expectedPresentNodeIds, StringComparer.OrdinalIgnoreCase);
+
+        var findings = new List<ClusterNodeAddressMismatch>();
+        var seen = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
+
+        foreach (var (host, port) in observedDriverMembers)
+        {
+            var nodeId = $"{host}:{port}";
+            seen.Add(nodeId);
+
+            if (!byNodeId.TryGetValue(nodeId, out var row))
+            {
+                findings.Add(new ClusterNodeAddressMismatch(
+                    AddressMismatchKind.RunningNodeHasNoRow, nodeId,
+                    $"driver node is in the cluster at {nodeId} but no ClusterNode row has that NodeId; " +
+                    "its deployment acks are discarded. Either register the row or correct the node's " +
+                    "Cluster:PublicHostname / Cluster:Port to match an existing one"));
+                continue;
+            }
+
+            if (!string.Equals(row.Host, host, StringComparison.OrdinalIgnoreCase) || row.AkkaPort != port)
+            {
+                findings.Add(new ClusterNodeAddressMismatch(
+                    AddressMismatchKind.RowDialTargetDisagrees, nodeId,
+                    $"node is at {nodeId} but its row's dial target is {row.Host}:{row.AkkaPort}; " +
+                    "central will dial the wrong address once Phase 2 stops using gossip. " +
+                    "Correct ClusterNode.Host / ClusterNode.AkkaPort"));
+            }
+        }
+
+        foreach (var row in rows)
+        {
+            if (seen.Contains(row.NodeId) || !expectedPresent.Contains(row.NodeId)) continue;
+            findings.Add(new ClusterNodeAddressMismatch(
+                AddressMismatchKind.EnabledRowNotInCluster, row.NodeId,
+                $"enabled ClusterNode row {row.NodeId} has no matching driver member; the next " +
+                "deployment will fail at the apply deadline waiting for it. Start the node, or set " +
+                "ClusterNode.MaintenanceMode = 1 while it is down for maintenance"));
+        }
+
+        return findings
+            .OrderBy(f => f.NodeId, StringComparer.OrdinalIgnoreCase)
+            .ThenBy(f => f.Kind)
+            .ToList();
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconcilerActor.cs

@@ -0,0 +1,136 @@
+using Akka.Actor;
+using Akka.Cluster;
+using Akka.Event;
+using Microsoft.EntityFrameworkCore;
+using ZB.MOM.WW.OtOpcUa.Configuration;
+
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
+
+/// <summary>
+///     Admin-role cluster singleton that runs <see cref="ClusterNodeAddressReconciler"/> against live
+///     membership and logs what it finds. A singleton rather than a per-admin-node actor so a
+///     two-admin fleet reports each disagreement once.
+/// </summary>
+/// <remarks>
+///     Recomputes on membership change (debounced) and on a slow periodic sweep, and <b>logs only
+///     when the finding set changes</b>. A node legitimately down would otherwise reprint the same
+///     warning every sweep until someone silences it by ignoring the log, which is the failure mode
+///     this check exists to avoid.
+/// </remarks>
+public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimers
+{
+    /// <summary>Driver role name — only members carrying it are expected to have a ClusterNode row.</summary>
+    public const string DriverRole = "driver";
+
+    /// <summary>Collapses a burst of membership events (a rolling restart) into one reconcile.</summary>
+    public static readonly TimeSpan DebounceWindow = TimeSpan.FromSeconds(2);
+
+    /// <summary>Periodic sweep, so a row edited in the AdminUI is checked without a topology change.</summary>
+    public static readonly TimeSpan DefaultSweepInterval = TimeSpan.FromMinutes(5);
+
+    private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
+    private readonly Akka.Cluster.Cluster _cluster;
+    private readonly TimeSpan _sweepInterval;
+    private readonly ILoggingAdapter _log = Context.GetLogger();
+    private IReadOnlyList<ClusterNodeAddressMismatch>? _lastReported;
+
+    /// <summary>Gets the timer scheduler for this actor.</summary>
+    public ITimerScheduler Timers { get; set; } = null!;
+
+    /// <summary>Creates Props for the reconciler.</summary>
+    /// <param name="dbFactory">Factory for the config database context.</param>
+    /// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
+    /// <returns>Props for actor creation.</returns>
+    public static Props Props(
+        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, TimeSpan? sweepInterval = null) =>
+        Akka.Actor.Props.Create(() => new ClusterNodeAddressReconcilerActor(dbFactory, sweepInterval));
+
+    /// <summary>Initializes a new instance of the <see cref="ClusterNodeAddressReconcilerActor"/> class.</summary>
+    /// <param name="dbFactory">Factory for the config database context.</param>
+    /// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
+    public ClusterNodeAddressReconcilerActor(
+        IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, TimeSpan? sweepInterval = null)
+    {
+        _dbFactory = dbFactory;
+        _cluster = Akka.Cluster.Cluster.Get(Context.System);
+        _sweepInterval = sweepInterval ?? DefaultSweepInterval;
+
+        Receive<ClusterEvent.IMemberEvent>(_ => ScheduleReconcile());
+        Receive<ClusterEvent.CurrentClusterState>(_ => ScheduleReconcile());
+        Receive<ReconcileNow>(_ => Reconcile());
+    }
+
+    /// <inheritdoc />
+    protected override void PreStart()
+    {
+        _cluster.Subscribe(Self, ClusterEvent.InitialStateAsEvents, typeof(ClusterEvent.IMemberEvent));
+        Timers.StartPeriodicTimer("sweep", ReconcileNow.Instance, _sweepInterval, _sweepInterval);
+    }
+
+    /// <inheritdoc />
+    protected override void PostStop() => _cluster.Unsubscribe(Self);
+
+    private void ScheduleReconcile() =>
+        Timers.StartSingleTimer("debounce", ReconcileNow.Instance, DebounceWindow);
+
+    private void Reconcile()
+    {
+        var observed = _cluster.State.Members
+            .Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(DriverRole))
+            .Select(m => (Host: m.Address.Host ?? string.Empty, Port: m.Address.Port ?? 0))
+            .Where(a => !string.IsNullOrWhiteSpace(a.Host) && a.Port > 0)
+            .ToList();
+
+        List<ClusterNodeAddress> rows;
+        List<string> enabled;
+        try
+        {
+            using var db = _dbFactory.CreateDbContext();
+            rows = db.ClusterNodes.AsNoTracking()
+                .Select(n => new ClusterNodeAddress(n.NodeId, n.Host, n.AkkaPort))
+                .ToList();
+            // Same predicate the coordinator's expected-ack set uses — a node the deploy path will
+            // not wait for must not be reported as missing either, or the maintenance hatch trades a
+            // failed deployment for a permanent warning.
+            enabled = db.ClusterNodes.AsNoTracking()
+                .Where(n => n.Enabled && !n.MaintenanceMode).Select(n => n.NodeId).ToList();
+        }
+        catch (Exception ex)
+        {
+            // A consistency check must never take the admin node down with it. Central SQL being
+            // briefly unreachable is an operational fact, not a reason to restart this singleton.
+            _log.Warning(ex, "ClusterNode address reconcile skipped — config database unreachable");
+            return;
+        }
+
+        var findings = ClusterNodeAddressReconciler.Reconcile(observed, rows, enabled);
+
+        if (_lastReported is not null && findings.SequenceEqual(_lastReported)) return;
+        _lastReported = findings;
+
+        if (findings.Count == 0)
+        {
+            _log.Info("ClusterNode addresses reconcile with cluster membership ({Count} driver members)",
+                observed.Count);
+            return;
+        }
+
+        foreach (var f in findings)
+        {
+            // Error for the two shapes that are always a misconfiguration; Warning for a row whose
+            // node is merely absent, which is a legitimate state during maintenance.
+            if (f.Kind == AddressMismatchKind.EnabledRowNotInCluster)
+                _log.Warning("ClusterNode address check [{Kind}] {NodeId}: {Detail}", f.Kind, f.NodeId, f.Detail);
+            else
+                _log.Error("ClusterNode address check [{Kind}] {NodeId}: {Detail}", f.Kind, f.NodeId, f.Detail);
+        }
+    }
+
+    /// <summary>Self-message: run a reconcile pass now.</summary>
+    public sealed class ReconcileNow
+    {
+        /// <summary>The singleton instance.</summary>
+        public static readonly ReconcileNow Instance = new();
+        private ReconcileNow() { }
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs

@@ -0,0 +1,54 @@
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
+
+/// <summary>
+/// A read-only view of the cluster's driver membership, as the manual-failover control sees it.
+/// </summary>
+/// <param name="PrimaryAddress">
+/// The current driver Primary — the oldest Up <c>driver</c> member, the same node
+/// <see cref="RedundancyStateActor.SelectDriverPrimary"/> names — or <see langword="null"/> when
+/// there is no Up driver member at all.
+/// </param>
+/// <param name="DriverAddresses">
+/// Every Up <c>driver</c> member, oldest first. A manual failover needs at least two: with one
+/// there is nothing to fail over to.
+/// </param>
+public sealed record ManualFailoverSnapshot(string? PrimaryAddress, IReadOnlyList<string> DriverAddresses)
+{
+    /// <summary>Whether a failover can be performed — i.e. a peer exists to take over.</summary>
+    public bool CanFailOver => DriverAddresses.Count >= 2;
+}
+
+/// <summary>
+/// Operator-initiated, graceful swap of the driver Primary.
+/// </summary>
+/// <remarks>
+///     <para>
+///         The mechanism is a cluster <b>Leave</b> of the current Primary, never a <c>Down</c>. The
+///         leaving node hands its singletons over through the normal cluster-leave phases,
+///         <c>CoordinatedShutdown</c> runs, <c>ActorSystemTerminationWatchdog</c> exits the process,
+///         the service supervisor restarts it, and it rejoins as the youngest member — so the
+///         survivor becomes the oldest Up driver member and therefore Primary. ServiceLevel 250
+///         moves with it and OPC UA clients re-select.
+///     </para>
+///     <para>
+///         <b>Mesh-scope caveat.</b> Until the per-cluster mesh work lands
+///         (<c>docs/plans/2026-07-21-per-cluster-mesh-design.md</c>) the Primary is elected once per
+///         Akka mesh, not per application <c>Cluster</c> row. On a fleet running several application
+///         clusters in one mesh this acts on the whole mesh's Primary. The UI says so.
+///     </para>
+/// </remarks>
+public interface IManualFailoverService
+{
+    /// <summary>Reads the current driver membership from live cluster state.</summary>
+    ManualFailoverSnapshot GetSnapshot();
+
+    /// <summary>
+    /// Gracefully removes the current driver Primary from the cluster so its peer takes over.
+    /// </summary>
+    /// <param name="actor">The authenticated identity performing the failover; recorded in the audit event.</param>
+    /// <returns>
+    /// The address of the node asked to leave, or <see langword="null"/> when the failover was
+    /// refused because no driver peer exists (in which case nothing was changed and nothing audited).
+    /// </returns>
+    Task<string?> FailOverDriverPrimaryAsync(string actor);
+}

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/ManualFailoverService.cs

@@ -0,0 +1,128 @@
+using System.Text.Json;
+using Akka.Actor;
+using Akka.Cluster;
+using Microsoft.Extensions.Logging;
+using ZB.MOM.WW.Audit;
+
+namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
+
+/// <summary>
+/// Default <see cref="IManualFailoverService"/> — a graceful cluster <c>Leave</c> of the oldest Up
+/// <c>driver</c> member. See the interface for the full behaviour and the mesh-scope caveat.
+/// </summary>
+public sealed class ManualFailoverService : IManualFailoverService
+{
+    /// <summary>The audit <see cref="AuditEvent.Action"/> stamped onto every manual failover.</summary>
+    public const string AuditAction = "cluster.manual-failover";
+
+    /// <summary>The audit <see cref="AuditEvent.Category"/> stamped onto every manual failover.</summary>
+    public const string AuditCategory = "Redundancy";
+
+    private readonly Func<ActorSystem> _system;
+    private readonly IAuditWriter _audit;
+    private readonly ILogger<ManualFailoverService> _log;
+
+    /// <summary>
+    /// Creates the service. The ActorSystem is taken as a factory, never as a constructed instance:
+    /// this is registered before Akka's hosted service has built the system, so resolving eagerly
+    /// would race startup (the same lazy-accessor pattern <c>ActorSystemTerminationWatchdog</c> uses).
+    /// </summary>
+    /// <param name="system">Lazy accessor for the node's ActorSystem.</param>
+    /// <param name="audit">The shared audit seam; a failover is recorded before it is performed.</param>
+    /// <param name="log">Logger for the failover decision.</param>
+    public ManualFailoverService(Func<ActorSystem> system, IAuditWriter audit, ILogger<ManualFailoverService> log)
+    {
+        _system = system ?? throw new ArgumentNullException(nameof(system));
+        _audit = audit ?? throw new ArgumentNullException(nameof(audit));
+        _log = log ?? throw new ArgumentNullException(nameof(log));
+    }
+
+    /// <inheritdoc/>
+    public ManualFailoverSnapshot GetSnapshot() => SnapshotCore(_system());
+
+    /// <inheritdoc/>
+    public async Task<string?> FailOverDriverPrimaryAsync(string actor)
+    {
+        ArgumentException.ThrowIfNullOrWhiteSpace(actor);
+
+        var system = _system();
+
+        // Dry run first so the peer guard is evaluated before anything is audited: a refused
+        // failover changed nothing and must not leave an audit row implying it did.
+        var target = FailOverCore(system, dryRun: true);
+        if (target is null)
+        {
+            _log.LogWarning(
+                "Manual failover refused for {Actor}: fewer than two Up driver members — there is no peer to take over.",
+                actor);
+            return null;
+        }
+
+        await _audit.WriteAsync(new AuditEvent
+        {
+            EventId = Guid.NewGuid(),
+            OccurredAtUtc = DateTimeOffset.UtcNow,
+            Actor = actor,
+            Action = AuditAction,
+            Category = AuditCategory,
+            SourceNode = target.ToString(),
+            Outcome = AuditOutcome.Success,
+            DetailsJson = JsonSerializer.Serialize(new
+            {
+                target = target.ToString(),
+                driverMembers = SnapshotCore(system).DriverAddresses,
+            }),
+        }).ConfigureAwait(false);
+
+        _log.LogWarning(
+            "Manual failover requested by {Actor}: asking the driver Primary {Target} to leave the cluster. "
+            + "It hands over its singletons, shuts down, and is restarted by its supervisor as the youngest "
+            + "member; its peer becomes Primary and advertises the primary ServiceLevel.",
+            actor,
+            target);
+
+        FailOverCore(system, dryRun: false);
+        return target.ToString();
+    }
+
+    /// <summary>
+    /// Selects — and, unless <paramref name="dryRun"/>, gracefully removes — the driver Primary.
+    /// </summary>
+    /// <param name="system">The ActorSystem whose cluster is acted on.</param>
+    /// <param name="dryRun">When <see langword="true"/>, selects the target without leaving.</param>
+    /// <returns>The target address, or <see langword="null"/> when fewer than two Up driver members exist.</returns>
+    public static Address? FailOverCore(ActorSystem system, bool dryRun = false)
+    {
+        ArgumentNullException.ThrowIfNull(system);
+
+        var cluster = Akka.Cluster.Cluster.Get(system);
+
+        // Intentionally identical to RedundancyStateActor.SelectDriverPrimary (oldest Up driver,
+        // Member.AgeOrdering) — the node acted on must be exactly the one the election names, or a
+        // failover would remove a node that was not Primary and change nothing. Pinned by
+        // ManualFailoverServiceTests.Parity_with_SelectDriverPrimary.
+        var drivers = OrderedDrivers(cluster.State.Members);
+
+        // A failover needs somewhere to fail over TO. With a single driver member, leaving would be
+        // a shutdown dressed up as a failover.
+        if (drivers.Count < 2) return null;
+
+        var primary = drivers[0];
+        if (!dryRun) cluster.Leave(primary.Address);
+        return primary.Address;
+    }
+
+    private static ManualFailoverSnapshot SnapshotCore(ActorSystem system)
+    {
+        var drivers = OrderedDrivers(Akka.Cluster.Cluster.Get(system).State.Members);
+
+        return new ManualFailoverSnapshot(
+            drivers.FirstOrDefault()?.Address.ToString(),
+            drivers.Select(m => m.Address.ToString()).ToList());
+    }
+
+    private static List<Member> OrderedDrivers(IEnumerable<Member> members) => members
+        .Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(RedundancyStateActor.DriverRole))
+        .OrderBy(m => m, Member.AgeOrdering)
+        .ToList();
+}

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/RedundancyStateActor.cs

@@ -111,9 +111,49 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
         _log.Debug("Published RedundancyStateChanged with {Count} nodes", snapshot.Count);
     }
 
+    /// <summary>The cluster role that marks a node as carrying the driver runtime.</summary>
+    public const string DriverRole = "driver";
+
+    /// <summary>
+    /// Selects the driver Primary: the <b>oldest</b> Up member carrying the <see cref="DriverRole"/>.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         This deliberately does <b>not</b> use <c>ClusterState.RoleLeader(role)</c>, which it
+    ///         previously did. Role leader is the <i>lowest-addressed</i> Up member with the role —
+    ///         address order (host, then port), with no relationship to time. Oldest is the member
+    ///         with the lowest up-number, and it is what <c>ClusterSingletonManager</c> uses to place
+    ///         singletons.
+    ///     </para>
+    ///     <para>
+    ///         On a freshly-formed cluster the two agree, which is why the discrepancy was invisible.
+    ///         They diverge after any restart: the restarted node re-joins as the youngest but keeps
+    ///         its address, so if it holds the lower address it becomes role leader while the
+    ///         singletons — and all the work they own — stay on the other node. Electing it Primary
+    ///         would enable the Primary-gated data plane (inbound writes, alarm acks, the fleet-wide
+    ///         alerts emit, the alarm-history drain) on the node that is not hosting the work.
+    ///     </para>
+    ///     <para>
+    ///         Only <see cref="MemberStatus.Up"/> members are eligible, matching singleton placement:
+    ///         a <c>Leaving</c> node is handing its singletons over and must not be named Primary.
+    ///     </para>
+    /// </remarks>
+    /// <param name="members">The current cluster members, in any order.</param>
+    /// <returns>The oldest Up driver member's address, or <c>null</c> when there is none.</returns>
+    public static Address? SelectDriverPrimary(IEnumerable<Member> members)
+    {
+        ArgumentNullException.ThrowIfNull(members);
+
+        return members
+            .Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(DriverRole))
+            .OrderBy(m => m, Member.AgeOrdering)
+            .FirstOrDefault()
+            ?.Address;
+    }
+
     private IReadOnlyList<NodeRedundancyState> BuildSnapshot()
     {
-        var driverLeader = _cluster.State.RoleLeader("driver");
+        var driverPrimary = SelectDriverPrimary(_cluster.State.Members);
         var clusterLeader = _cluster.State.Leader;
         var now = DateTime.UtcNow;
 
@@ -123,15 +163,16 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
             var host = member.Address.Host;
             if (string.IsNullOrWhiteSpace(host)) continue;
 
-            var role = member.Roles.Contains("driver")
-                ? (driverLeader == member.Address ? CommonsRedundancyRole.Primary : CommonsRedundancyRole.Secondary)
+            var isPrimary = driverPrimary is not null && driverPrimary == member.Address;
+            var role = member.Roles.Contains(DriverRole)
+                ? (isPrimary ? CommonsRedundancyRole.Primary : CommonsRedundancyRole.Secondary)
                 : CommonsRedundancyRole.Detached;
 
             list.Add(new NodeRedundancyState(
                 ToNodeId(member.Address),
                 role,
                 IsClusterLeader: clusterLeader == member.Address,
-                IsRoleLeaderForDriver: driverLeader == member.Address,
+                IsDriverPrimary: isPrimary,
                 AsOfUtc: now));
         }
         return list;

src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs

@@ -1,12 +1,17 @@
 using Akka.Actor;
 using Akka.Cluster.Hosting;
+using Akka.Cluster.Tools.Client;
 using Akka.Hosting;
 using Microsoft.EntityFrameworkCore;
 using Microsoft.Extensions.Configuration;
 using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Options;
 using ZB.MOM.WW.OtOpcUa.Configuration;
+using ZB.MOM.WW.OtOpcUa.Cluster;
+using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.AdminOperations;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.Audit;
+using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
 using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
@@ -23,9 +28,11 @@ public static class ServiceCollectionExtensions
     public const string AuditWriterSingletonName = "audit-writer";
     public const string FleetStatusSingletonName = "fleet-status";
     public const string RedundancyStateSingletonName = "redundancy-state";
+    public const string ClusterNodeAddressReconcilerSingletonName = "cluster-node-address-reconciler";
 
     /// <summary>
-    /// Registers all five admin-role cluster singletons + their proxies on the AkkaConfigurationBuilder.
+    /// Registers the admin-role cluster singletons on the AkkaConfigurationBuilder — five with
+    /// registry proxies, plus the address reconciler, which is proxy-less because nothing addresses it.
     /// Must be called against the same builder used by <c>AkkaHostedService</c> so the singletons
     /// share the host's ActorSystem.
     /// </summary>
@@ -46,12 +53,49 @@ public static class ServiceCollectionExtensions
         var singletonOptions = new ClusterSingletonOptions { Role = AdminRole };
         var proxyOptions = new ClusterSingletonOptions { Role = AdminRole };
 
+        // Per-cluster mesh Phase 2: the central end of the central↔node command boundary.
+        //
+        // A plain WithActors, NOT a singleton — and that is the load-bearing part. A driver node's
+        // ClusterClient rotates across contact points and must find a live comm actor at whichever
+        // central node answers. As a singleton it would exist on one central node only, and the
+        // rotation would fail silently against the other: acks would land sometimes, depending on
+        // which contact the client happened to settle on.
+        builder.WithActors((system, registry, resolver) =>
+        {
+            var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
+            var options = resolver.GetService<IOptions<MeshTransportOptions>>().Value;
+
+            var actor = system.ActorOf(
+                CentralCommunicationActor.Props(
+                    dbFactory,
+                    options,
+                    new DefaultMeshClusterClientFactory(),
+                    // Resolved lazily, per message, so this actor never depends on the order in
+                    // which Akka.Hosting materialises the singleton proxy relative to this block.
+                    () => registry.TryGet<ConfigPublishCoordinatorKey>(out var coordinator)
+                        ? coordinator
+                        : null),
+                MeshPaths.CentralCommunicationName);
+
+            // Registered unconditionally, in both transport modes. Under Dps nothing dials it, and
+            // an idle registration costs nothing — whereas registering only under ClusterClient
+            // would mean flipping the flag on a running fleet requires a central restart before any
+            // node can ack, turning a config change back into a deployment.
+            ClusterClientReceptionist.Get(system).RegisterService(actor);
+            registry.Register<CentralCommunicationActorKey>(actor);
+        });
+
         builder.WithSingleton<ConfigPublishCoordinatorKey>(
             ConfigPublishSingletonName,
             (system, registry, resolver) =>
             {
                 var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
-                return ConfigPublishCoordinator.Props(dbFactory);
+                // Registered above, so it is already in the registry — the same
+                // ordering-by-registration the AdminOperations singleton relies on to resolve the
+                // coordinator. Deliberately NOT an ActorSelection.ResolveOne here: that blocks
+                // inside a props factory and races the spawn it is waiting for.
+                var meshRouter = registry.Get<CentralCommunicationActorKey>();
+                return ConfigPublishCoordinator.Props(dbFactory, applyDeadline: null, meshRouter: meshRouter);
             },
             singletonOptions);
 
@@ -67,7 +111,8 @@ public static class ServiceCollectionExtensions
                 var mode = Enum.TryParse<TagConfigValidationMode>(
                     resolver.GetService<IConfiguration>()?["Deployment:TagConfigValidationMode"],
                     ignoreCase: true, out var m) ? m : TagConfigValidationMode.Warn;
-                return AdminOperationsActor.Props(dbFactory, coordinator, probes, mode);
+                var meshRouter = registry.Get<CentralCommunicationActorKey>();
+                return AdminOperationsActor.Props(dbFactory, coordinator, probes, mode, meshRouter);
             },
             singletonOptions);
 
@@ -90,6 +135,26 @@ public static class ServiceCollectionExtensions
             (system, registry, resolver) => RedundancyStateActor.Props(),
             singletonOptions);
 
+        // Per-cluster mesh Phase 1: ClusterNode.AkkaPort duplicates the node's own Cluster:Port and
+        // nothing makes them agree. Runs here rather than on each driver node because Phase 4 removes
+        // the driver nodes' ConfigDb connection entirely.
+        //
+        // createProxyToo: false — unlike the five singletons above, nothing ever resolves
+        // ClusterNodeAddressReconcilerKey from the registry. This actor only reads cluster state and
+        // logs; no one sends it messages. A proxy would be a second actor per node whose only job is
+        // to hunt for a singleton nobody addresses — visible as the "ClusterSingletonProxy failed to
+        // find an associated singleton after 30 seconds" startup warning, and as extra cluster
+        // chatter in every two-node test harness the integration suite builds.
+        builder.WithSingleton<ClusterNodeAddressReconcilerKey>(
+            ClusterNodeAddressReconcilerSingletonName,
+            (system, registry, resolver) =>
+            {
+                var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
+                return ClusterNodeAddressReconcilerActor.Props(dbFactory);
+            },
+            singletonOptions,
+            createProxyToo: false);
+
         return builder;
     }
 }
@@ -100,3 +165,5 @@ public sealed class AdminOperationsActorKey { }
 public sealed class AuditWriterActorKey { }
 public sealed class FleetStatusBroadcasterKey { }
 public sealed class RedundancyStateActorKey { }
+public sealed class ClusterNodeAddressReconcilerKey { }
+public sealed class CentralCommunicationActorKey { }

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/AlarmSfLegacyMigrator.cs

@@ -0,0 +1,217 @@
+using Microsoft.Data.Sqlite;
+using Microsoft.Extensions.Configuration;
+using ZB.MOM.WW.LocalDb;
+using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     One-time copy of the pre-consolidation <c>alarm-historian.db</c> store-and-forward queue
+///     into the consolidated <c>ZB.MOM.WW.LocalDb</c> database.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>What is at stake.</b> Rows sit in that file precisely because the historian could not
+///         be reached. Discarding them on upgrade would throw away exactly the alarm audit trail
+///         the store-and-forward queue exists to protect — so the copy tolerates an older column
+///         set rather than bailing out, and refuses to rename a file it did not actually read.
+///     </para>
+///     <para>
+///         <b>Runs after <c>RegisterReplicated</c>, deliberately.</b> Capture is trigger-based, so
+///         rows inserted before registration never enter the oplog and never reach the peer —
+///         silently, and permanently, because nothing revisits history. Migrating after
+///         registration means the recovered backlog replicates like any other write.
+///     </para>
+///     <para>
+///         <b>Ids come from the payload, not from the legacy row id.</b> The legacy primary key was
+///         <c>RowId INTEGER PRIMARY KEY AUTOINCREMENT</c>, which cannot survive into a replicated
+///         table: each node allocated its own sequence, so node A's row 7 and node B's row 7 are
+///         different alarms that would silently overwrite one another under last-writer-wins.
+///         Hashing the payload (<see cref="AlarmSfSchema.DeriveId"/>) solves that and one more
+///         problem besides — a warm pair's two legacy files <i>overlap</i>, because
+///         <c>HistorianAdapterActor</c> default-writes while its redundancy role is unknown and
+///         both nodes therefore accepted the same transitions during every boot window. A
+///         node-prefixed id would carry that duplication into the merged buffer forever; an
+///         equal-payload id collapses it. It is also what makes a re-run harmless under
+///         <c>INSERT OR IGNORE</c>.
+///     </para>
+///     <para>
+///         <b>All-or-nothing.</b> The copy runs in one transaction and the legacy file is renamed
+///         to <c>&lt;name&gt;.migrated</c> only after the commit. A failure throws out of
+///         <c>OnReady</c>, failing host startup with the legacy file untouched — no half-migrated
+///         state to reason about, and the operator still holds the original. A later boot sees the
+///         renamed file and no-ops.
+///     </para>
+/// </remarks>
+public static class AlarmSfLegacyMigrator
+{
+    private const string MigratedSuffix = ".migrated";
+
+    /// <summary>The legacy queue table.</summary>
+    private const string LegacyTable = "Queue";
+
+    /// <summary>
+    ///     The configuration key that used to carry the standalone queue file's path. Read as a raw
+    ///     key because the corresponding <c>AlarmHistorianOptions</c> property was removed with the
+    ///     bespoke file management it configured.
+    /// </summary>
+    public const string LegacyPathKey = "AlarmHistorian:DatabasePath";
+
+    /// <summary>The pre-removal default for <see cref="LegacyPathKey"/>.</summary>
+    private const string DefaultLegacyPath = "alarm-historian.db";
+
+    /// <summary>
+    ///     The legacy column whose absence means this is not a queue file we can read. Without the
+    ///     payload there is no event and no id to derive from one.
+    /// </summary>
+    private const string RequiredColumn = "PayloadJson";
+
+    /// <summary>
+    ///     Every legacy column, mapped to its consolidated counterpart. Columns absent from an
+    ///     older file are dropped from the copy rather than failing it — see
+    ///     <see cref="PresentColumns"/>.
+    /// </summary>
+    private static readonly (string Legacy, string Current)[] ColumnMap =
+    [
+        ("AlarmId", "alarm_id"),
+        ("EnqueuedUtc", "enqueued_at_utc"),
+        ("PayloadJson", "payload_json"),
+        ("AttemptCount", "attempt_count"),
+        ("LastAttemptUtc", "last_attempt_utc"),
+        ("LastError", "last_error"),
+        ("DeadLettered", "dead_lettered"),
+    ];
+
+    /// <summary>
+    ///     Copies any legacy alarm queue into <paramref name="db"/>, then renames the legacy file.
+    /// </summary>
+    /// <param name="db">The consolidated database, with <c>alarm_sf_events</c> already registered.</param>
+    /// <param name="config">Configuration supplying the legacy queue's path.</param>
+    public static void Migrate(ILocalDb db, IConfiguration config)
+    {
+        ArgumentNullException.ThrowIfNull(db);
+        ArgumentNullException.ThrowIfNull(config);
+
+        var legacyPath = ResolveLegacyPath(config);
+        if (!ShouldMigrate(legacyPath)) return;
+
+        using (var legacy = OpenLegacyReadOnly(legacyPath))
+        {
+            var present = PresentColumns(legacy);
+
+            // An absent table probes as zero columns, so this one guard covers both "no queue
+            // table" and "a shape we do not recognise". Returning without renaming leaves the file
+            // for an operator to inspect.
+            if (!present.Contains(RequiredColumn)) return;
+
+            using var connection = db.CreateConnection();
+            using var transaction = connection.BeginTransaction();
+            CopyRows(legacy, connection, transaction, present);
+            transaction.Commit();
+        }
+
+        // Only after the commit. A crash between the two leaves the file in place and the next
+        // boot copies it again, which the payload-derived ids make harmless.
+        File.Move(legacyPath, legacyPath + MigratedSuffix, overwrite: true);
+    }
+
+    /// <summary>
+    ///     Resolves the legacy queue path from the removed configuration key, falling back to the
+    ///     code default it used to carry.
+    /// </summary>
+    /// <remarks>
+    ///     Relative paths resolve against the process working directory — not the LocalDb
+    ///     directory — because that is where the old code actually put them.
+    /// </remarks>
+    /// <param name="config">The application configuration.</param>
+    /// <returns>An absolute path, or empty when there is nothing durable to migrate from.</returns>
+    internal static string ResolveLegacyPath(IConfiguration config)
+    {
+        var path = config[LegacyPathKey] ?? DefaultLegacyPath;
+
+        // In-memory and URI-form sources (test / dev configurations) have nothing durable behind
+        // them, and Path.GetFullPath on them would produce nonsense.
+        if (string.IsNullOrWhiteSpace(path) ||
+            path.Equals(":memory:", StringComparison.OrdinalIgnoreCase) ||
+            path.StartsWith("file:", StringComparison.OrdinalIgnoreCase))
+        {
+            return string.Empty;
+        }
+
+        return Path.GetFullPath(path);
+    }
+
+    /// <summary>Whether there is a legacy file worth opening.</summary>
+    private static bool ShouldMigrate(string legacyPath) =>
+        !string.IsNullOrEmpty(legacyPath)
+        && File.Exists(legacyPath)
+        && !File.Exists(legacyPath + MigratedSuffix);
+
+    /// <summary>Opens the legacy file read-only, so a failed migration cannot damage it.</summary>
+    private static SqliteConnection OpenLegacyReadOnly(string path)
+    {
+        var connection = new SqliteConnection(new SqliteConnectionStringBuilder
+        {
+            DataSource = path,
+            Mode = SqliteOpenMode.ReadOnly,
+        }.ToString());
+        connection.Open();
+        return connection;
+    }
+
+    /// <summary>
+    ///     Which of the columns we know about the legacy table actually has.
+    /// </summary>
+    /// <remarks>
+    ///     A file written by an older build predates some columns, and naming a missing one in the
+    ///     SELECT throws "no such column" — which discards every row in the table rather than the
+    ///     one field. Intersecting first means an old file migrates its data and simply leaves the
+    ///     newer columns at their schema defaults.
+    /// </remarks>
+    private static HashSet<string> PresentColumns(SqliteConnection legacy)
+    {
+        var present = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
+
+        using var cmd = legacy.CreateCommand();
+        cmd.CommandText = $"SELECT name FROM pragma_table_info('{LegacyTable}')";
+        using var reader = cmd.ExecuteReader();
+        while (reader.Read())
+            present.Add(reader.GetString(0));
+
+        return present;
+    }
+
+    /// <summary>Copies every legacy row into the consolidated table, keyed by payload hash.</summary>
+    private static void CopyRows(
+        SqliteConnection legacy,
+        SqliteConnection target,
+        SqliteTransaction transaction,
+        HashSet<string> present)
+    {
+        var columns = ColumnMap.Where(c => present.Contains(c.Legacy)).ToArray();
+        var payloadOrdinal = Array.FindIndex(columns, c => c.Legacy == RequiredColumn);
+
+        using var read = legacy.CreateCommand();
+        read.CommandText =
+            $"SELECT {string.Join(", ", columns.Select(c => c.Legacy))} FROM {LegacyTable}";
+
+        using var reader = read.ExecuteReader();
+        while (reader.Read())
+        {
+            using var insert = target.CreateCommand();
+            insert.Transaction = transaction;
+            insert.CommandText = $"""
+                INSERT OR IGNORE INTO {AlarmSfSchema.EventsTable}
+                    ({AlarmSfSchema.IdColumn}, {string.Join(", ", columns.Select(c => c.Current))})
+                VALUES
+                    ($id, {string.Join(", ", columns.Select((_, i) => $"$c{i}"))})
+                """;
+
+            insert.Parameters.AddWithValue("$id", AlarmSfSchema.DeriveId(reader.GetString(payloadOrdinal)));
+            for (var i = 0; i < columns.Length; i++)
+                insert.Parameters.AddWithValue($"$c{i}", reader.GetValue(i) ?? DBNull.Value);
+
+            insert.ExecuteNonQuery();
+        }
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ConfigServeAuthInterceptor.cs

@@ -0,0 +1,157 @@
+using System.Security.Cryptography;
+using System.Text;
+using Grpc.Core;
+using Grpc.Core.Interceptors;
+using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.OtOpcUa.Cluster;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     Gates the central config-serve artifact endpoint (per-cluster mesh Phase 3). The
+///     <c>DeploymentArtifactService</c> streams a driver node the exact configuration it will boot
+///     on; anything able to reach central's serve port could otherwise pull the whole deployed config
+///     out-of-band. This interceptor is the ONLY inbound auth on that endpoint.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>Scoped by method path.</b> Only calls under
+///         <c>/deployment_artifact.v1.DeploymentArtifactService/</c> are gated; every other method on
+///         the shared gRPC pipeline (notably the LocalDb sync service) passes through untouched, so
+///         this can share one <c>AddGrpc</c> registration with <see cref="LocalDbSyncAuthInterceptor"/>.
+///     </para>
+///     <para>
+///         <b>Fail-closed.</b> With no <c>ConfigServe:ApiKey</c> configured, NO fetch is accepted,
+///         authenticated or not. Treating "no key" as "no auth required" would expose the deployed
+///         configuration on exactly the default shape most nodes ship with. The node dialing in must
+///         present the same <c>ConfigSource:ApiKey</c> (the shared node key), so a key typo stops
+///         config delivery outright rather than degrading to unauthenticated.
+///     </para>
+///     <para>
+///         Comparison is <see cref="CryptographicOperations.FixedTimeEquals"/> over UTF-8 bytes, so a
+///         wrong key cannot be recovered byte-by-byte from response timing. Length differences are
+///         unavoidably observable and are not sensitive.
+///     </para>
+/// </remarks>
+public sealed class ConfigServeAuthInterceptor : Interceptor
+{
+    private const string ServicePrefix = "/deployment_artifact.v1.DeploymentArtifactService/";
+    private const string AuthorizationHeader = "authorization";
+    private const string BearerPrefix = "Bearer ";
+
+    private readonly IOptions<ConfigServeOptions> _options;
+    private readonly ILogger<ConfigServeAuthInterceptor> _logger;
+
+    /// <summary>Creates the interceptor.</summary>
+    /// <param name="options">Config-serve options; <c>ApiKey</c> is the expected bearer token.</param>
+    /// <param name="logger">Logger for denial diagnostics.</param>
+    public ConfigServeAuthInterceptor(
+        IOptions<ConfigServeOptions> options,
+        ILogger<ConfigServeAuthInterceptor> logger)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+        ArgumentNullException.ThrowIfNull(logger);
+
+        _options = options;
+        _logger = logger;
+    }
+
+    /// <inheritdoc />
+    public override Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
+        TRequest request,
+        ServerCallContext context,
+        UnaryServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(request, context);
+    }
+
+    /// <inheritdoc />
+    public override Task DuplexStreamingServerHandler<TRequest, TResponse>(
+        IAsyncStreamReader<TRequest> requestStream,
+        IServerStreamWriter<TResponse> responseStream,
+        ServerCallContext context,
+        DuplexStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(requestStream, responseStream, context);
+    }
+
+    /// <inheritdoc />
+    public override Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
+        IAsyncStreamReader<TRequest> requestStream,
+        ServerCallContext context,
+        ClientStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(requestStream, context);
+    }
+
+    /// <inheritdoc />
+    public override Task ServerStreamingServerHandler<TRequest, TResponse>(
+        TRequest request,
+        IServerStreamWriter<TResponse> responseStream,
+        ServerCallContext context,
+        ServerStreamingServerMethod<TRequest, TResponse> continuation)
+    {
+        Authorize(context);
+        return continuation(request, responseStream, context);
+    }
+
+    /// <summary>
+    ///     Throws <see cref="RpcException"/> with <see cref="StatusCode.Unauthenticated"/> if this is
+    ///     a config-serve call that does not carry the configured bearer token. Non-serve calls return
+    ///     immediately.
+    /// </summary>
+    private void Authorize(ServerCallContext context)
+    {
+        if (!context.Method.StartsWith(ServicePrefix, StringComparison.Ordinal))
+            return;
+
+        var expected = _options.Value.ApiKey;
+        if (string.IsNullOrEmpty(expected))
+        {
+            _logger.LogWarning(
+                "Rejected a config-serve call to {Method}: no ConfigServe:ApiKey is configured, so the " +
+                "artifact endpoint is closed. Configure the shared node key on central and every node.",
+                context.Method);
+            throw new RpcException(new Status(
+                StatusCode.Unauthenticated,
+                "Config-serve is not accepting connections: no API key is configured on this node."));
+        }
+
+        var presented = ExtractBearerToken(context.RequestHeaders);
+        if (presented is null || !FixedTimeEquals(presented, expected))
+        {
+            _logger.LogWarning(
+                "Rejected a config-serve call to {Method}: {Reason}.",
+                context.Method,
+                presented is null ? "no bearer token presented" : "bearer token did not match");
+            throw new RpcException(new Status(
+                StatusCode.Unauthenticated,
+                "Config-serve authentication failed."));
+        }
+    }
+
+    private static string? ExtractBearerToken(Metadata headers)
+    {
+        // gRPC lowercases header keys on the wire; compare case-insensitively anyway so a hand-built
+        // Metadata in a test behaves the same as a real request.
+        foreach (var entry in headers)
+        {
+            if (!string.Equals(entry.Key, AuthorizationHeader, StringComparison.OrdinalIgnoreCase))
+                continue;
+
+            var value = entry.Value;
+            if (value is not null && value.StartsWith(BearerPrefix, StringComparison.OrdinalIgnoreCase))
+                return value[BearerPrefix.Length..];
+        }
+
+        return null;
+    }
+
+    private static bool FixedTimeEquals(string presented, string expected)
+        => CryptographicOperations.FixedTimeEquals(
+            Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/KestrelHttpBinding.cs

@@ -0,0 +1,102 @@
+using System.Net;
+using Microsoft.AspNetCore.Server.Kestrel.Core;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     One HTTP endpoint the host was asked to serve, parsed out of the configured URL list so it
+///     can be re-bound explicitly.
+/// </summary>
+/// <param name="Host">The host component as written — <c>+</c>, <c>*</c>, a hostname, or an IP.</param>
+/// <param name="Port">The port.</param>
+/// <param name="IsSecure">True when the URL used the <c>https</c> scheme.</param>
+public sealed record KestrelHttpBinding(string Host, int Port, bool IsSecure)
+{
+    /// <summary>
+    ///     Parses a semicolon-separated URL list (the <c>ASPNETCORE_URLS</c> / <c>urls</c> format)
+    ///     into bindings. Unparseable entries are skipped rather than throwing — a malformed URL
+    ///     must not take the process down at startup.
+    /// </summary>
+    /// <param name="urls">The configured URL list, or null/empty when none is configured.</param>
+    /// <returns>The parsed bindings, in the order given; empty when nothing is configured.</returns>
+    public static IReadOnlyList<KestrelHttpBinding> Parse(string? urls)
+    {
+        if (string.IsNullOrWhiteSpace(urls))
+            return [];
+
+        var result = new List<KestrelHttpBinding>();
+        foreach (var raw in urls.Split(';', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries))
+        {
+            // Uri cannot parse the "+"/"*" wildcard hosts Kestrel accepts, so substitute a
+            // placeholder host purely to get scheme/port out, then keep the original host token.
+            var isWildcard = raw.Contains("://+", StringComparison.Ordinal)
+                          || raw.Contains("://*", StringComparison.Ordinal);
+            var probe = isWildcard
+                ? raw.Replace("://+", "://placeholder", StringComparison.Ordinal)
+                     .Replace("://*", "://placeholder", StringComparison.Ordinal)
+                : raw;
+
+            if (!Uri.TryCreate(probe, UriKind.Absolute, out var uri))
+                continue;
+
+            var host = isWildcard
+                ? raw.Contains("://+", StringComparison.Ordinal) ? "+" : "*"
+                : uri.Host;
+
+            result.Add(new KestrelHttpBinding(host, uri.Port, uri.Scheme == Uri.UriSchemeHttps));
+        }
+
+        return result;
+    }
+
+    /// <summary>
+    ///     Parses the bare-port list format of <c>ASPNETCORE_HTTP_PORTS</c> / <c>HTTP_PORTS</c> (and
+    ///     the <c>HTTPS</c> variants) into wildcard (<c>+</c>, all-interfaces) bindings — the shape
+    ///     Kestrel itself gives those variables. Modern .NET base images (aspnet:8.0+) set
+    ///     <c>ASPNETCORE_HTTP_PORTS=8080</c> as the container default <b>instead of</b>
+    ///     <c>ASPNETCORE_URLS</c>, so a node that never sets <c>URLS</c> explicitly still has a real
+    ///     configured surface here that must be re-bound, not replaced with Kestrel's localhost:5000.
+    /// </summary>
+    /// <param name="ports">A <c>;</c>- or <c>,</c>-separated list of bare ports, or null/empty.</param>
+    /// <param name="isSecure">True to mark the parsed bindings <c>https</c> (the <c>HTTPS_PORTS</c> vars).</param>
+    /// <returns>One all-interfaces binding per parseable port; empty when nothing is configured.</returns>
+    public static IReadOnlyList<KestrelHttpBinding> ParsePorts(string? ports, bool isSecure)
+    {
+        if (string.IsNullOrWhiteSpace(ports))
+            return [];
+
+        var result = new List<KestrelHttpBinding>();
+        foreach (var token in ports.Split([';', ','], StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries))
+        {
+            // Skip a malformed port rather than throwing — a bad env var must not down startup.
+            if (int.TryParse(token, out var port) && port is > 0 and <= 65535)
+                result.Add(new KestrelHttpBinding("+", port, isSecure));
+        }
+
+        return result;
+    }
+
+    /// <summary>
+    ///     Applies this binding to Kestrel, choosing the narrowest listen call the host component
+    ///     allows.
+    /// </summary>
+    /// <remarks>
+    ///     A hostname that is neither a literal IP nor <c>localhost</c> cannot be reliably mapped to
+    ///     a local interface, so it falls back to <see cref="KestrelServerOptions.ListenAnyIP"/> —
+    ///     the safe superset. Narrowing it and guessing wrong would silently stop serving.
+    /// </remarks>
+    /// <param name="options">The Kestrel options being configured.</param>
+    public void Apply(KestrelServerOptions options)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+
+        if (Host is "+" or "*")
+            options.ListenAnyIP(Port);
+        else if (IPAddress.TryParse(Host, out var address))
+            options.Listen(address, Port);
+        else if (string.Equals(Host, "localhost", StringComparison.OrdinalIgnoreCase))
+            options.ListenLocalhost(Port);
+        else
+            options.ListenAnyIP(Port);
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs

@@ -0,0 +1,101 @@
+using Microsoft.Extensions.Configuration;
+using Microsoft.Extensions.DependencyInjection;
+using ZB.MOM.WW.LocalDb;
+using ZB.MOM.WW.LocalDb.Replication;
+using ZB.MOM.WW.OtOpcUa.Runtime.DeploymentCache;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+/// <summary>
+///     Single registration point for the node-local LocalDb subsystem: the embedded SQLite store
+///     that caches the deployed-configuration artifact, plus the optional gRPC replication that
+///     mirrors it to the node's redundant pair peer.
+/// </summary>
+/// <remarks>
+///     <para>
+///         Exists as a named extension rather than inline <c>AddZbLocalDb</c> calls in
+///         <c>Program.cs</c> so the wiring is covered by a DI resolution test — <c>Program.cs</c> is
+///         top-level statements and cannot be exercised directly, which is exactly how a
+///         "registered but never resolvable" defect ships unnoticed. This family has shipped that
+///         defect three times (Secrets 0.2.0, Secrets 0.2.2, ScadaBridge #22).
+///     </para>
+///     <para>
+///         <b>Driver-role nodes only.</b> Admin-only nodes have no deployed configuration to cache,
+///         and registering here would impose the <c>LocalDb:Path</c>-required-or-no-boot constraint
+///         on them for nothing.
+///     </para>
+///     <para>
+///         <b>Storage is unconditional; replication is default-OFF.</b> A node with no
+///         <c>PeerAddress</c> and no <c>SyncListenPort</c> is simply a fast local SQLite file —
+///         <c>AddZbLocalDbReplication</c> registers the engine but it stays idle. Replication is
+///         opt-in per pair because there is no production distribution story for the sync
+///         <c>ApiKey</c> yet (the same open question the Secrets KEK has).
+///     </para>
+/// </remarks>
+public static class LocalDbRegistration
+{
+    /// <summary>Configuration section holding <c>LocalDbOptions</c>.</summary>
+    public const string LocalDbSectionPath = "LocalDb";
+
+    /// <summary>Configuration section holding <c>ReplicationOptions</c>.</summary>
+    public const string ReplicationSectionPath = "LocalDb:Replication";
+
+    /// <summary>
+    ///     Configuration key carrying the dedicated h2c sync listener's port. Zero or absent means
+    ///     no listener is bound and the passive sync endpoint is not mapped.
+    /// </summary>
+    public const string SyncListenPortKey = "LocalDb:SyncListenPort";
+
+    /// <summary>
+    ///     The port the dedicated cleartext-HTTP/2 sync listener should bind, or <c>0</c> when the
+    ///     node should not listen at all. Defaults to <c>0</c> — absent configuration must mean
+    ///     "off".
+    /// </summary>
+    /// <param name="configuration">The application configuration.</param>
+    /// <returns>The configured sync port, or <c>0</c>.</returns>
+    public static int SyncListenPort(IConfiguration configuration)
+    {
+        ArgumentNullException.ThrowIfNull(configuration);
+        return configuration.GetValue(SyncListenPortKey, defaultValue: 0);
+    }
+
+    /// <summary>
+    ///     Registers the local database (with the deployment-cache schema applied and registered for
+    ///     replication) and the replication engine.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <see cref="LocalDbSetup.OnReady"/> runs inside <c>AddZbLocalDb</c>'s singleton factory,
+    ///         before the first caller receives the <c>ILocalDb</c> — so every consumer is guaranteed
+    ///         to see the tables created and their capture triggers installed.
+    ///     </para>
+    ///     <para>
+    ///         <c>AddZbLocalDbReplication</c> is called unconditionally rather than behind an
+    ///         <c>Enabled</c> flag because the library already models "off" as "no
+    ///         <c>PeerAddress</c>": the initiator background service idles and the passive endpoint
+    ///         is only reachable if <c>Program.cs</c> maps it, which it does only when
+    ///         <see cref="SyncListenPortKey"/> is set. Adding a second gate on top would give two
+    ///         ways to express the same thing and a way for them to disagree.
+    ///     </para>
+    /// </remarks>
+    /// <param name="services">The service collection to add to.</param>
+    /// <param name="configuration">The application configuration.</param>
+    /// <returns>The same <paramref name="services"/> instance, for chaining.</returns>
+    public static IServiceCollection AddOtOpcUaLocalDb(
+        this IServiceCollection services,
+        IConfiguration configuration)
+    {
+        ArgumentNullException.ThrowIfNull(services);
+        ArgumentNullException.ThrowIfNull(configuration);
+
+        services.AddZbLocalDb(configuration, db => LocalDbSetup.OnReady(db, configuration));
+        services.AddZbLocalDbReplication(configuration);
+
+        // The consumer-facing seam. WithOtOpcUaRuntimeActors resolves this optionally and threads it
+        // into DriverHostActor; registering the store without this leaves the cache tables present,
+        // replicating, and never written to.
+        services.AddSingleton<IDeploymentArtifactCache, LocalDbDeploymentArtifactCache>();
+
+        return services;
+    }
+}