test(harness): serialize Host.IntegrationTests; drop the reconciler's unused singleton proxy

Closes the last branch-only failures. Measured across five full-suite runs on
this branch versus two on master in a clean worktree: master failed only AbCip
in this assembly (2/2), while the branch additionally failed one or two E2E
deploy tests, rotating between DeployHappyPath, DriverReconnect and
EquipmentNamespaceMaterialization (4/4). Each passed in isolation, and the
assembly on its own was 195/201 throughout — the failures only appeared under
full-suite CPU contention.

Cause is contention, not correctness. Every test here builds a real two-node
Akka cluster — two Kestrel hosts, two ActorSystems with remoting, six admin
singletons, an EF context, a deploy pipeline — and xUnit ran them concurrently
with each other and with every other assembly. That was survivable while the
coordinator sealed instantly on an empty expected-ack set; now that it waits for
a real ack from every configured node, these tests measure an end-to-end
round-trip and starve.

Serialising this one assembly makes them deterministic: 195/201 with only
AbCip_Green_AgainstSim, and the full-solution failure set is now IDENTICAL to
master's — 7 shared, zero branch-only. Cost is wall-clock for this assembly,
40s -> 3m35s; it is a handful of heavyweight E2E tests, not a broad unit suite,
and a widened timeout alone did not fix it (tried first, at 45s).

Also drops createProxyToo for the address reconciler. Nothing resolves
ClusterNodeAddressReconcilerKey from the registry — the actor only reads cluster
state and logs — so the proxy was a second actor per node hunting for a
singleton nobody addresses. Kept because it removes dead machinery, not because
it fixed the flakiness; measured on its own, it did not.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

CLAUDE.md

@@ -68,9 +68,45 @@ shared-lib consumption, or per-project commands — update the **OtOpcUa entry i
    gRPC session. The gateway owns the COM apartment + STA pump
    server-side; the driver speaks `MxCommand` / `MxEvent` protos
    exclusively.
-3. **OPC UA Server** — Exposes authored equipment tags as variable nodes.
-   Galaxy tags are bound by `TagConfig.FullName` (`tag_name.AttributeName`);
-   reads/writes/subscriptions are translated to that reference for MXAccess.
+3. **OPC UA Server** — Exposes the deployed tree under **two OPC UA namespaces**
+   (see below). Galaxy tags are bound by `TagConfig.FullName`
+   (`tag_name.AttributeName`); reads/writes/subscriptions are translated to that
+   reference for MXAccess.
+
+### v3 OPC UA Address Space (Batch 4): dual namespace
+
+The server exposes **two OPC UA namespaces** (replacing the single
+`https://zb.com/otopcua/ns`), built by `AddressSpaceComposer` /
+`AddressSpaceApplier` and served by `OtOpcUaNodeManager` (which registers both
+URIs; `V3NodeIds` + `AddressSpaceRealm` are the identity authority):
+
+| Namespace URI | Realm | Subtree | NodeId `s=` scheme |
+|---|---|---|---|
+| `https://zb.com/otopcua/raw` | `AddressSpaceRealm.Raw` | the `/raw` device tree (Folder → Driver → Device → TagGroup → Tag) | the node's **RawPath** (e.g. `Plant/Modbus/dev1/Speed`) |
+| `https://zb.com/otopcua/uns` | `AddressSpaceRealm.Uns` | the UNS tree (Area → Line → Equipment → signal) | slash-joined **`Area/Line/Equipment/EffectiveName`** |
+
+- **Single source, fanned to both.** Every device value has exactly one source —
+  the raw tag's node. A `UnsTagReference` projects that raw tag into an equipment
+  as a UNS-namespace variable that carries an **`Organizes` reference to its raw
+  node** and mirrors it. One driver publish for a RawPath fans (in `DriverHostActor`)
+  to the raw NodeId AND every referencing UNS NodeId with identical
+  value/quality/timestamp — the mux key stays single (RawPath).
+- **Writes via either NodeId** route to the same driver ref under the same
+  `WriteOperate` gating (the node-manager write gate is realm-qualified and fails
+  closed); a failed device write reverts both NodeIds via the shared fan-out
+  (write-outcome self-correction).
+- **Native alarms** materialize once at the raw tag (`ConditionId = RawPath`) and
+  fan via SDK notifiers to the raw device folder AND every referencing equipment
+  folder — one `ReportEvent`, one Server-object copy; ack/confirm/shelve route on
+  `ConditionId = RawPath`.
+- **Historian dual-registration.** Both NodeIds register the **same** historian
+  tagname; the mux `HistorizedTagRef` stays single (RawPath). HistoryRead works
+  through either NodeId.
+
+The retired `EquipmentNodeIds` (`{equipmentId}/{folderPath}/{name}`) scheme and
+the single-namespace `EquipmentTags` materialization path are gone. See
+`docs/plans/2026-07-15-raw-uns-two-subtree-v3-design.md` +
+`docs/plans/2026-07-15-v3-batch4-address-space-plan.md`, and `docs/Uns.md`.
 
 ### Key Concept: Tag Name and FullName
 
@@ -183,6 +219,64 @@ The server supports configurable OPC UA transport security via the `OpcUa:Enable
 
 The server supports non-transparent warm/hot redundancy via the `Redundancy` section in `appsettings.json`. Two instances share the same Galaxy DB and the same mxaccessgw (under distinct `MxAccess.ClientName` values) but have unique `ApplicationUri` values. Each exposes `RedundancySupport`, `ServerUriArray`, and a dynamic `ServiceLevel` based on role and runtime health. The primary advertises a higher ServiceLevel than the secondary. See `docs/Redundancy.md` for the full guide.
 
+**Two corrections landed 2026-07-21 — both were total-outage or wrong-node defects, and both are worth knowing before touching this area.**
+
+- **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is still open** — the pathology only appears 1-vs-1 and docker-dev is a single six-node mesh, so the crash-the-oldest drill is deferred to the per-cluster mesh work.
+- **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**.
+
+**A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node’s own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering".
+
+Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
+
+**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2–7 not started).
+
+**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
+
+- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag.
+- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out.
+- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
+- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever.
+
+## LocalDb pair-local store (Phases 1 + 2)
+
+Every **driver-role** node keeps a consolidated `ZB.MOM.WW.LocalDb` SQLite database (the retired
+LiteDB `LocalCache` subsystem was deleted as superseded). It holds **three replicated tables**:
+`deployment_artifacts` + `deployment_pointer` (Phase 1) and `alarm_sf_events` (Phase 2).
+
+**Phase 1** caches the
+**deployed-configuration artifact** (chunked, SHA-256-verified, newest-2 retention per cluster) so a
+driver node can **boot from cache when central SQL Server is unreachable** — `DriverHostActor` writes
+the cache after each successful apply (`IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache`,
+in `Runtime/Deployment/`) and reads it as a boot fallback, flagging running-from-cache. Wired by
+`AddOtOpcUaLocalDb` in the `hasDriver` branch only (`Host/Configuration/LocalDbRegistration.cs` +
+`LocalDbSetup.OnReady`, whose DDL→`RegisterReplicated` order is load-bearing); **admin-only nodes
+never register it.** The cache can optionally replicate to the node's redundant pair peer over the
+library's gRPC sync — **default-OFF, fail-closed bearer auth** (`LocalDbSyncAuthInterceptor`), gated
+on `LocalDb:SyncListenPort` (a dedicated Kestrel h2c listener) + `LocalDb:Replication:*`. On the
+docker-dev rig the **site-a pair** has replication on; central + site-b stay default-OFF (site-b is
+the pin). ApiKey via `${secret:}`/env, never committed (the rig's dev key is the committed-dev-secret
+exception).
+
+**Phase 2** moved the **alarm store-and-forward buffer** off its standalone `alarm-historian.db`
+and into the same database as `alarm_sf_events`, so a node that dies holding undelivered alarm
+history no longer takes it to the grave. `SqliteStoreAndForwardSink` became
+`LocalDbStoreAndForwardSink` (`Core.AlarmHistorian`), and the breaking config key
+`AlarmHistorian:DatabasePath` was **removed** — it is still read, by `AlarmSfLegacyMigrator`, purely
+to find a pre-consolidation file to copy across on first boot (renamed `.migrated` after the copy
+commits). Because the buffer replicates, **only the node holding the Primary role drains it**: the
+sink takes a `drainGate` delegate, Runtime supplies it from the `IRedundancyRoleView` singleton that
+`DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, and a gated-off node reports the new
+`HistorianDrainState.NotPrimary`. Delivery is therefore **at-least-once across a failover, by
+design** (no dedup layer). Row ids are a SHA-256 of the event payload rather than a GUID, so the
+same event accepted on both nodes — which happens in every boot window, since
+`HistorianAdapterActor` default-writes while its role is unknown — converges to one row. On the
+docker-dev rig the site-a pair and site-b both run `AlarmHistorian__Enabled=true` against a
+deliberately unresolvable gateway endpoint, so the buffer is always a live, non-draining queue.
+
+See `docs/Configuration.md` (`LocalDb` section), `docs/Redundancy.md` (pair-local config
+cache), `docs/AlarmHistorian.md`, and the runbook
+`docs/operations/2026-07-20-localdb-pair-replication.md`.
+
 ## LDAP Authentication
 
 The server uses LDAP-based user authentication via the `Security:Ldap` section in `appsettings.json`. When enabled, credentials are validated by LDAP bind against a GLAuth server, and LDAP group membership maps to OPC UA permissions: `ReadOnly` (browse/read), `WriteOperate` (write FreeAccess/Operate attributes), `WriteTune` (write Tune attributes), `WriteConfigure` (write Configure attributes), `AlarmAck` (alarm acknowledgment). `LdapOpcUaUserAuthenticator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/LdapOpcUaUserAuthenticator.cs`) implements `IOpcUaUserAuthenticator`, delegating the LDAP bind + group lookup to `OtOpcUaLdapAuthService` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/OtOpcUaLdapAuthService.cs`, an `ILdapAuthService`). See `docs/security.md` for the full guide.
@@ -217,6 +311,10 @@ The AdminUI's global **UNS** page (`/uns`) is the single surface for managing th
 
 **v3 (Batch 2): device I/O is authored in the `/raw` project tree** (`GlobalRaw.razor` → `RawTree.razor`, `IRawTreeService`) — a cluster-rooted `Folder → Driver → Device → TagGroup → Tag` hierarchy with per-node context menus (the reusable `ContextMenu`), lazy loading, driver/device config modals (endpoint moved to `Device.DeviceConfig`; Test-connect probes the `DriverDeviceConfigMerger`-merged config), manual-entry + CSV import/export (RFC-4180, staged review grid, all-or-nothing), and browse-commit of discovered leaves into raw tags. The routed `/clusters/{id}/drivers` flow (`DriverTypePicker`, `DriverEditRouter`, the 8 `*DriverPage` shells, the Drivers list) is **retired** — its form bodies live on inside the `/raw` modals. The `DriverType` dispatch maps are keyed off the `DriverTypeNames` constants (fixing the `TwinCat`/`Focas` drift). A `Calculation` pseudo-driver (`DriverTypeNames.Calculation`, `IDependencyConsumer`, deploy-time `scriptId`-existence + Tarjan cycle gates) computes signal-level tags from other tags' RawPaths. See `docs/Raw.md`.
 
+**v3 (Batch 3): UNS Equipment is reference-only.** The equipment **Tags** tab no longer authors or binds tags — it holds `UnsTagReference` rows pointing at raw tags authored in `/raw`. "+ Add reference" opens the `RawTree` in picker mode, **cluster-scoped** (structurally + server-enforced `tag.cluster == equipment.cluster`); rows show effective name / RawPath / inherited DataType+AccessLevel / display-name override. **Effective-name uniqueness** (references + VirtualTags + ScriptedAlarms share the `{EquipmentId}/{EffectiveName}` NodeId space) is enforced at authoring (`IEffectiveNameGuard`) and at the deploy gate (`DraftValidator.UnsEffectiveNameCollision` — catches rename-induced collisions). Scripts use **`ctx.GetTag("{{equip}}/<RefName>")`** — resolved per-equipment through `UnsTagReference` effective names to the backing RawPath at both compose seams (`AddressSpaceComposer` + `DeploymentArtifact`, via the shared `EquipmentReferenceMap`, byte-parity); an unresolved `<RefName>` is a deploy error (`EquipReferenceUnresolved`) AND a live Monaco diagnostic (`OTSCRIPT_EQUIPREF`) — editor accepts ⇔ publish accepts. `EquipmentScriptPaths.DeriveEquipmentBase` is deleted (the dot-joint `{{equip}}.X` became the slash-joint `{{equip}}/<RefName>`). Raw rename warns when a beneath-it tag is historized-without-override / UNS-referenced / named by a script literal (`RawTreeService` substring scan). `ImportEquipmentModal` dropped the `DriverInstanceId` column. See `docs/Uns.md` + `docs/ScriptEditor.md`.
+
+**v3 (Batch 4 = v3.0): dual-namespace address space.** The OPC UA server exposes **two namespaces** — `https://zb.com/otopcua/raw` (Raw device tree, `s=<RawPath>`) + `https://zb.com/otopcua/uns` (UNS tree, `s=<Area>/<Line>/<Equipment>/<EffectiveName>`) — replacing the single `.../ns` and the retired `EquipmentNodeIds` scheme. Every value has ONE source (the raw node's publish); a driver publish for a RawPath fans in `DriverHostActor` to the raw NodeId AND every referencing UNS NodeId with identical value/quality/timestamp, and each UNS variable `Organizes`-references its raw node. Writes route via **either** NodeId to the same driver ref under the same realm-qualified `WriteOperate` gate (failed-write revert works through both); native alarms materialize ONCE at the raw tag (`ConditionId = RawPath`) and fan via SDK `AddNotifier` to the raw device folder + every referencing equipment folder (one `ReportEvent`, one Server-object copy); both NodeIds register the SAME historian tagname (mux `HistorizedTagRef` stays single, keyed by RawPath). Identity authority: `V3NodeIds` + `AddressSpaceRealm` (carried explicitly at every sink seam — never parsed out of the id string). See the "v3 OPC UA Address Space (Batch 4)" section above, `docs/Uns.md`, and `docs/plans/2026-07-15-v3-batch4-address-space-plan.md`.
+
 The `/uns` **TagModal** uses **driver-typed tag-config editors**: it dispatches by the bound driver's `DriverType` to a per-driver editor (Modbus/S7/AbCip/AbLegacy/TwinCAT/Focas/OpcUaClient) via `TagConfigEditorMap`, with client-side validation via `TagConfigValidator`; unmapped drivers (only Galaxy) fall back to the generic raw-`TagConfig`-JSON textarea. Each editor is a thin razor shell over a pure `<Driver>TagConfigModel` (`FromJson`/`ToJson`/`Validate`, preserves unknown keys). To add a driver's editor, copy the Modbus template under `Components/Shared/Uns/TagEditors/` + `Uns/TagEditors/`, reusing the driver's enums + camelCase JSON property names, and register it in `TagConfigEditorMap` + `TagConfigValidator`. See `docs/plans/2026-06-09-driver-typed-tag-editors-design.md`.
 
 ## Scripting / Script Editor
@@ -281,11 +379,13 @@ authorization uses the standard `AccessLevels.HistoryRead` bit set at materializ
 ### Alarm-history path (`AlarmHistorian` section)
 
 Alarm events are written through `GatewayAlarmHistorianWriter` (the gateway **`SendEvent`** path) behind
-the durable **`SqliteStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink`
-default for the SQLite store-and-forward queue, whose drain worker forwards batches to the gateway and uses
-per-event outcomes to decide retry vs. dead-letter (never throws). The `AlarmHistorian` section carries
-only the `Enabled` gate + the SQLite knobs (`DatabasePath`, `DrainIntervalSeconds`, `Capacity`,
-`DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection
+the durable **`LocalDbStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink`
+default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the
+replicated `alarm_sf_events` table (see the LocalDb section) and whose drain worker forwards batches to
+the gateway and uses per-event outcomes to decide retry vs. dead-letter (never throws). **Only the node
+holding the Primary role drains** (a gated node reports `HistorianDrainState.NotPrimary`). The
+`AlarmHistorian` section carries only the `Enabled` gate + the queue knobs (`DrainIntervalSeconds`,
+`Capacity`, `DeadLetterRetentionDays`, `BatchSize`, `MaxAttempts`) — the downstream gateway connection
 (endpoint/key/TLS) is sourced from the `ServerHistorian` section. **Alarm-history `ReadEvents` requires the
 target gateway deployed with `RuntimeDb:EventReadsEnabled=true`** (the C2 SQL event-read workaround).
 
@@ -357,11 +457,16 @@ The `AlarmHistorian` section's old Wonderware connection keys (`Host`/`Port`/`Us
 were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from
 `ServerHistorian`. See `docs/Historian.md` for the full guide.
 
-### KNOWN LIMITATION 1 — live-validation gate (do before merging/trusting the cutover)
+### Live-validation gate — PASSED (was "KNOWN LIMITATION 1")
 
-The cutover is code-complete but **must be live-validated against a real gateway** (VPN to
-`wonder-sql-vd03`, gateway running the prerequisites above) before it is merged or trusted. Run the
-env-gated suite:
+**No longer a limitation.** The cutover was live-validated against the real gateway on
+`wonder-sql-vd03`: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a *confirmed
+protocol limitation* (the captured `CM_EVENT` wire never carries `SourceName`, so the gateway cannot
+populate `Source_Object` — not a fixable bug). Evidence:
+`docs/plans/2026-06-27-otopcua-historian-followups.md`.
+
+Kept here because it is the recipe to **re-run** the gate after any gateway or backend change (VPN to
+`wonder-sql-vd03`, gateway running the prerequisites above):
 
 ```bash
 export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222   # absolute gateway URI; absent ⇒ all live tests skip
@@ -376,7 +481,7 @@ dotnet test --filter "Category=LiveIntegration"
 The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the
 gate the operator runs on the VPN before trusting the cutover.
 
-### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending
+### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending (issue #491)
 
 The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer +
 meters). It is spawned with an **empty *initial* ref set** (`Array.Empty<string>()` in
@@ -390,4 +495,4 @@ deployed set of historized value tags from the first deploy onward (the feed is
 path is code-complete.** What remains is the **live gate**: an end-to-end verification against a real gateway
 (deploy → dependency-mux value delta → outbox append → `WriteLiveValues`) plus a restart-convergence test
 proving the recorder re-registers the deployed set after a process restart. Until that live verification runs,
-treat continuous value-capture as unproven-in-production rather than absent.
+treat continuous value-capture as unproven-in-production rather than absent. Tracked as **issue #491**.

Directory.Build.rsp

@@ -0,0 +1,12 @@
+# MSBuild auto-response file, applied to every build run from this repo root.
+#
+# Disable the MSBuild node pool. By default MSBuild leaves worker nodes resident
+# after a build so the next one starts warm. Across a session of repeated
+# solution-wide builds and test runs this repo accumulated 55 idle nodes holding
+# ~6 GB, which is not merely untidy: the integration suites assert on timeouts,
+# and memory pressure turns those into failures indistinguishable from real
+# regressions (see the Workstation-GC note in Host.IntegrationTests.csproj for
+# the same failure mode from a different cause).
+#
+# The cost is a small startup penalty per build. That is a good trade here.
+-nodeReuse:false

Directory.Packages.props

@@ -29,10 +29,10 @@
     <PackageVersion Include="coverlet.collector" Version="6.0.4" />
     <PackageVersion Include="FluentAssertions" Version="8.3.0" />
     <PackageVersion Include="Google.Protobuf" Version="3.34.1" />
+    <PackageVersion Include="Grpc.AspNetCore" Version="2.76.0" />
     <PackageVersion Include="Grpc.Core.Api" Version="2.76.0" />
     <PackageVersion Include="Grpc.Net.Client" Version="2.76.0" />
     <PackageVersion Include="libplctag" Version="1.5.2" />
-    <PackageVersion Include="LiteDB" Version="5.0.21" />
     <PackageVersion Include="MessagePack" Version="2.5.301" />
     <PackageVersion Include="Microsoft.AspNetCore.Authorization" Version="10.0.7" />
     <PackageVersion Include="Microsoft.AspNetCore.DataProtection" Version="10.0.7" />
@@ -57,6 +57,7 @@
     <PackageVersion Include="Microsoft.EntityFrameworkCore.InMemory" Version="10.0.7" />
     <PackageVersion Include="Microsoft.EntityFrameworkCore.SqlServer" Version="10.0.7" />
     <PackageVersion Include="Microsoft.FASTER.Core" Version="2.6.5" />
+    <PackageVersion Include="Microsoft.Extensions.Configuration" Version="10.0.7" />
     <PackageVersion Include="Microsoft.Extensions.Configuration.Abstractions" Version="10.0.7" />
     <PackageVersion Include="Microsoft.Extensions.Configuration.Json" Version="10.0.7" />
     <PackageVersion Include="Microsoft.Extensions.DependencyInjection" Version="10.0.7" />
@@ -103,6 +104,23 @@
       clearing the advisory without CentralPackageTransitivePinningEnabled (which breaks the Roslyn split).
     -->
     <PackageVersion Include="SQLitePCLRaw.bundle_e_sqlite3" Version="2.1.12" />
+    <!--
+      Surgical transitive pin #2. Four NU1903 high-severity advisories (GHSA-23rf-6693-g89p,
+      GHSA-8q5v-6pqq-x66h, GHSA-cvvh-rhrc-wg4q, GHSA-g8r8-53c2-pm3f) landed in the NuGet audit
+      data against System.Security.Cryptography.Xml 10.0.7, which Microsoft.AspNetCore.DataProtection
+      10.0.7 pulls in transitively for its key storage. Under TreatWarningsAsErrors every FRESH
+      restore goes red — 204 NU1903 errors across the solution — while machines with cached audit
+      data keep building, so this surfaces first on a clean clone or a docker image build, not
+      locally. Verified 2026-07-21 by restoring a detached worktree.
+
+      10.0.10 is the patched version, pinned by direct PackageReference in Core.Configuration (the
+      one project where the DataProtection chain enters). Bumping the DataProtection parent to
+      10.0.10 instead was rejected in the sister repo for a reason that applies here too: it floors
+      Microsoft.Extensions.* and, through the EFCore adapter, Microsoft.EntityFrameworkCore at
+      10.0.10, forcing a family-wide servicing bump (NU1605 downgrade cascade) that deserves its own
+      reviewed change.
+    -->
+    <PackageVersion Include="System.Security.Cryptography.Xml" Version="10.0.10" />
     <PackageVersion Include="System.CommandLine" Version="2.0.5" />
     <PackageVersion Include="System.Data.SqlClient" Version="4.9.0" />
     <PackageVersion Include="System.IdentityModel.Tokens.Jwt" Version="8.11.0" />
@@ -122,6 +140,13 @@
     <PackageVersion Include="xunit.runner.visualstudio" Version="3.0.2" />
     <PackageVersion Include="xunit.v3" Version="1.1.0" />
     <PackageVersion Include="ZB.MOM.WW.Health" Version="0.1.0" />
+    <!--
+      LocalDb: embedded SQLite node-local cache + optional 2-node gRPC replication. The core
+      package floors the native SQLitePCLRaw.lib.e_sqlite3 at 2.1.12, so consumers inherit the
+      CVE-2025-6965-fixed native binary without needing the surgical pin below.
+    -->
+    <PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.3" />
+    <PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.3" />
     <PackageVersion Include="ZB.MOM.WW.Health.Akka" Version="0.1.0" />
     <PackageVersion Include="ZB.MOM.WW.Health.EntityFrameworkCore" Version="0.1.0" />
     <PackageVersion Include="ZB.MOM.WW.Telemetry" Version="0.1.0" />
@@ -129,9 +154,13 @@
     <PackageVersion Include="ZB.MOM.WW.MxGateway.Client" Version="0.1.1" />
     <PackageVersion Include="ZB.MOM.WW.MxGateway.Contracts" Version="0.1.1" />
     <PackageVersion Include="ZB.MOM.WW.Configuration" Version="0.1.0" />
-    <PackageVersion Include="ZB.MOM.WW.Auth.Abstractions" Version="0.1.1" />
-    <PackageVersion Include="ZB.MOM.WW.Auth.Ldap" Version="0.1.1" />
-    <PackageVersion Include="ZB.MOM.WW.Auth.AspNetCore" Version="0.1.1" />
+    <PackageVersion Include="ZB.MOM.WW.Auth.Abstractions" Version="0.1.5" />
+    <PackageVersion Include="ZB.MOM.WW.Auth.Ldap" Version="0.1.5" />
+    <PackageVersion Include="ZB.MOM.WW.Auth.AspNetCore" Version="0.1.5" />
+    <PackageVersion Include="ZB.MOM.WW.Secrets" Version="0.2.3" />
+    <PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.3" />
+    <PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.3" />
+    <PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.AkkaDotNet" Version="0.2.3" />
     <PackageVersion Include="ZB.MOM.WW.Audit" Version="0.1.0" />
     <PackageVersion Include="ZB.MOM.WW.Theme" Version="0.3.1" />
     <PackageVersion Include="ZB.MOM.WW.HistorianGateway.Client" Version="0.3.0" />

NuGet.config

@@ -25,6 +25,10 @@
       <package pattern="ZB.MOM.WW.Theme" />
       <package pattern="ZB.MOM.WW.HistorianGateway.Contracts" />
       <package pattern="ZB.MOM.WW.HistorianGateway.Client" />
+      <package pattern="ZB.MOM.WW.Secrets" />
+      <package pattern="ZB.MOM.WW.Secrets.*" />
+      <package pattern="ZB.MOM.WW.LocalDb" />
+      <package pattern="ZB.MOM.WW.LocalDb.*" />
     </packageSource>
   </packageSourceMapping>
 </configuration>

docker-dev/docker-compose.yml

@@ -13,7 +13,9 @@
 #   central-1, central-2   OTOPCUA_ROLES=admin,driver — the ONLY UI + deploy
 #                          singleton, plus the MAIN cluster's OPC UA publishers.
 #                          Reachable at http://localhost:9200 (via Traefik).
-#                          central-1 is the Akka seed node; central-2 joins it.
+#                          Both are Akka seed nodes, each listing ITSELF first
+#                          (2026-07-22) so either can cold-start while the other
+#                          is down; the site nodes seed off central-1 only.
 #   site-a-1, site-a-2     OTOPCUA_ROLES=driver — driver-only members of the same
 #   site-b-1, site-b-2     mesh, scoped to SITE-A / SITE-B by ClusterId. They
 #                          serve no UI and authenticate no users; the central
@@ -47,6 +49,29 @@
 
 name: otopcua-dev
 
+# ── Shared secret-replication env (all six host nodes) ──────────────────────────
+# Akka peer-to-peer clustered secret replication rides the existing `otopcua` Akka
+# mesh (DistributedPubSub topic `zb-mom-ww-secrets`) across all six host nodes. Two
+# hard requirements are wired here, identically, on every node:
+#   1. Secrets__Replication__Enabled=true  → SecretsRegistration.AddOtOpcUaSecrets
+#      switches from plain AddZbSecrets to AddZbSecretsAkkaReplication + the eager
+#      SecretReplicationStarter hosted service, so every node joins anti-entropy.
+#   2. ZB_SECRETS_MASTER_KEY  → the ONE shared 32-byte base64 KEK. Every node MUST
+#      carry the SAME key or peers fail closed decrypting each other's rows
+#      (kek_id mismatch). appsettings.json binds Secrets:MasterKey:Source=Environment
+#      / EnvVarName=ZB_SECRETS_MASTER_KEY, so this env var IS the KEK on every node.
+# The KEK below is a committed DEV-ONLY default (matches the rig's zero-operator-step
+# philosophy, like the committed dev SQL password). Override for a fresh key with:
+#   OTOPCUA_SECRETS_KEK=$(openssl rand -base64 32) docker compose ... up -d
+# NEVER reuse this key outside this local dev rig.
+# AnnounceInterval is shortened to 5s (default 30s) so anti-entropy convergence is
+# observable quickly during dev exercise. Each node keeps its own local SQLite store
+# (Secrets:SqlitePath=otopcua-secrets.db, per-container); replication syncs them.
+x-secrets-env: &secrets-env
+  Secrets__Replication__Enabled: "true"
+  Secrets__Replication__AnnounceInterval: "00:00:05"
+  ZB_SECRETS_MASTER_KEY: "${OTOPCUA_SECRETS_KEK:-ZYGhIX0luS/XsevpCB2W18jYHMcqO6AjM9oXy+T6Zp4=}"
+
 services:
 
   sql:
@@ -113,7 +138,8 @@ services:
 
   # ── Central cluster (2-node fused admin+driver) ─────────────────────────────
   # The only UI + deploy singleton; also the MAIN cluster's OPC UA publishers.
-  # central-1 seeds the single Akka mesh that every other node joins.
+  # central-1 seeds the single Akka mesh that every other node joins; central-2 is a seed too
+  # (listing itself first), so it can form the mesh alone if central-1 is down.
 
   central-1: &otopcua-host
     build:
@@ -145,6 +171,7 @@ services:
       sql: { condition: service_healthy }
       migrator: { condition: service_completed_successfully }
     environment:
+      <<: *secrets-env
       OTOPCUA_ROLES: "admin,driver"
       ASPNETCORE_URLS: "http://+:9000"
       ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
@@ -152,8 +179,13 @@ services:
       Cluster__Port: "4053"
       Cluster__PublicHostname: "central-1"
       # Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
-      # re-join the mesh via EITHER peer, not only central-1. With restart supervision this is
-      # the 2-node keep-oldest exit-and-rejoin recovery path.
+      # re-join the mesh via EITHER peer, not only central-1.
+      #
+      # ORDER IS LOAD-BEARING (2026-07-22): each node lists ITSELF first. Akka runs
+      # FirstSeedNodeProcess — the only path that can form a NEW cluster when no peer answers
+      # InitJoin — exclusively for seed-nodes[0]; every other node retries InitJoin forever. So a
+      # node listing its partner first cannot cold-start while that partner is down. Enforced at
+      # boot by AkkaClusterOptionsValidator; see docs/Redundancy.md.
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
       Cluster__Roles__0: "admin"
@@ -200,8 +232,15 @@ services:
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # Consolidated pair-local config cache (ZB.MOM.WW.LocalDb). Every driver-role node
+      # (central is admin,driver) gets one, on a per-node named volume so the cache survives
+      # container recreates. The central pair leaves replication OFF (default) — only the
+      # site-a pair replicates, as the enablement demo; site-b is the default-OFF pin.
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4840:4840"
+    volumes:
+      - otopcua-localdb-central-1:/app/data
 
   central-2:
     <<: *otopcua-host
@@ -210,15 +249,18 @@ services:
       central-1: { condition: service_started }
       migrator: { condition: service_completed_successfully }
     environment:
+      <<: *secrets-env
       OTOPCUA_ROLES: "admin,driver"
       ASPNETCORE_URLS: "http://+:9000"
       ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "central-2"
-      # Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1.
-      Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
-      Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
+      # Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST:
+      # central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1
+      # is down (it previously listed central-1 first and simply never came Up in that case).
+      Cluster__SeedNodes__0: "akka.tcp://otopcua@central-2:4053"
+      Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "admin"
       Cluster__Roles__1: "driver"
       Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
@@ -258,8 +300,12 @@ services:
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # Pair-local config cache; central pair leaves replication OFF (see central-1).
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4841:4840"
+    volumes:
+      - otopcua-localdb-central-2:/app/data
 
   # ── Site A cluster (2-node driver-only) ─────────────────────────────────────
   # Driver-only members of the single mesh, scoped to SITE-A by ClusterId. No UI,
@@ -278,8 +324,12 @@ services:
       Cluster__Hostname: "0.0.0.0"
       Cluster__Port: "4053"
       Cluster__PublicHostname: "site-a-1"
+      # Site nodes are deliberately NOT seeds — they join the central pair's mesh. The self-first
+      # seed rule is conditional for exactly this reason (it binds only when a node's own address
+      # is in its own seed list), so these configs are exempt rather than broken.
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
+      <<: *secrets-env
       # Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
       # mem_reservation are inherited from the *otopcua-host anchor.
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
@@ -287,8 +337,49 @@ services:
       # Resolved at runtime by GalaxyDriver.ResolveApiKey when a DriverInstance's
       # Gateway.ApiKeySecretRef = "env:GALAXY_MXGW_API_KEY".
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # Pair-local config cache + REPLICATION ON — the site-a pair is the enablement demo.
+      # site-a-1 is the initiator: it binds the h2c sync listener on 9001 AND dials the peer.
+      # Setting SyncListenPort makes Program.cs add a dedicated Http2-only Kestrel listener and
+      # re-bind the primary HTTP port (an explicit Listen* otherwise discards ASPNETCORE_URLS).
+      # Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
+      # buffer is a live, replicated table rather than an empty one — the live gate needs a real
+      # queue to watch converge, and needs to see that only the Primary drains it.
+      #
+      # There is no HistorianGateway on this rig, so the endpoint below is deliberately
+      # unresolvable: every drain attempt fails, which is exactly the historian-outage state the
+      # buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
+      # URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
+      # AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
+      #
+      # MaxAttempts is raised far above the production default of 10. With a permanently
+      # unreachable gateway the default would dead-letter the whole queue about five minutes into
+      # the outage, turning a buffering test into a dead-letter test.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
+      # Required whenever the section is consumed: the gateway client validates its own options at
+      # construction and throws on an empty key, so a keyless node crash-loops during Akka startup
+      # rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
+      # above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
+      # ServerHistorian__ApiKey from the environment.
+      ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
+      # Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
+      # direction, so an http endpoint with the default UseTls=true crash-loops the host.
+      ServerHistorian__UseTls: "false"
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
+      LocalDb__SyncListenPort: "9001"
+      LocalDb__Replication__PeerAddress: "http://site-a-2:9001"
+      # DEV-ONLY committed key (rig philosophy — like the SQL password + secrets KEK above). It
+      # MUST be byte-identical on both nodes: the interceptor is fail-closed, so any mismatch
+      # silently stops the pair converging. NEVER reuse this key outside this local dev rig.
+      LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
+      # Row-count batching against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so
+      # 16 × 171 KB ≈ 2.7 MB stays under the cap with headroom.
+      LocalDb__Replication__MaxBatchSize: "16"
     ports:
       - "4842:4840"
+    volumes:
+      - otopcua-localdb-site-a-1:/app/data
 
   site-a-2:
     <<: *otopcua-host
@@ -304,11 +395,47 @@ services:
       Cluster__PublicHostname: "site-a-2"
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
+      <<: *secrets-env
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # site-a-2 is the passive half of the replicating pair: it binds the sync listener on 9001
+      # but does NOT dial (no PeerAddress). The replication stream is bidirectional, so a-1's
+      # single dial carries both directions. Same key + MaxBatchSize as a-1 (byte-identical key
+      # is mandatory — the interceptor fail-closes on a mismatch).
+      # Alarm store-and-forward buffer, Phase 2. Enabled on BOTH halves of the pair so the
+      # buffer is a live, replicated table rather than an empty one — the live gate needs a real
+      # queue to watch converge, and needs to see that only the Primary drains it.
+      #
+      # There is no HistorianGateway on this rig, so the endpoint below is deliberately
+      # unresolvable: every drain attempt fails, which is exactly the historian-outage state the
+      # buffer exists for. Note the endpoint must still be a syntactically valid absolute http(s)
+      # URI or the host refuses to start — ServerHistorianOptionsValidator is consumer-gated, so
+      # AlarmHistorian:Enabled=true makes it required even while ServerHistorian:Enabled=false.
+      #
+      # MaxAttempts is raised far above the production default of 10. With a permanently
+      # unreachable gateway the default would dead-letter the whole queue about five minutes into
+      # the outage, turning a buffering test into a dead-letter test.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
+      # Required whenever the section is consumed: the gateway client validates its own options at
+      # construction and throws on an empty key, so a keyless node crash-loops during Akka startup
+      # rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
+      # above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
+      # ServerHistorian__ApiKey from the environment.
+      ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
+      # Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
+      # direction, so an http endpoint with the default UseTls=true crash-loops the host.
+      ServerHistorian__UseTls: "false"
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
+      LocalDb__SyncListenPort: "9001"
+      LocalDb__Replication__ApiKey: "dev-site-a-localdb-sync-key"
+      LocalDb__Replication__MaxBatchSize: "16"
     ports:
       - "4843:4840"
+    volumes:
+      - otopcua-localdb-site-a-2:/app/data
 
   # ── Site B cluster (2-node driver-only) ─────────────────────────────────────
 
@@ -326,11 +453,32 @@ services:
       Cluster__PublicHostname: "site-b-1"
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
+      <<: *secrets-env
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # site-b is the default-OFF pin: it gets the pair-local cache but NO SyncListenPort and NO
+      # replication config, so no sync listener binds and ISyncStatus stays disconnected/Healthy.
+      # This is what the live gate's check 8 verifies — the cache works locally with replication off.
+      # Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
+      # must work as a plain node-local buffer here, with no sync listener and no peer traffic.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
+      # Required whenever the section is consumed: the gateway client validates its own options at
+      # construction and throws on an empty key, so a keyless node crash-loops during Akka startup
+      # rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
+      # above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
+      # ServerHistorian__ApiKey from the environment.
+      ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
+      # Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
+      # direction, so an http endpoint with the default UseTls=true crash-loops the host.
+      ServerHistorian__UseTls: "false"
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4844:4840"
+    volumes:
+      - otopcua-localdb-site-b-1:/app/data
 
   site-b-2:
     <<: *otopcua-host
@@ -346,11 +494,30 @@ services:
       Cluster__PublicHostname: "site-b-2"
       Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
       Cluster__Roles__0: "driver"
+      <<: *secrets-env
       Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
       Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
       GALAXY_MXGW_API_KEY: "${GALAXY_MXGW_API_KEY:-mxgw_otopcua2_GI7-tNozYE6cXGUSgEzL3AHDV7bYcYIHdMwKYgyHdX4}"
+      # site-b default-OFF pin (see site-b-1): cache on, replication off.
+      # Alarm store-and-forward with replication OFF — the other half of the site-b pin. The sink
+      # must work as a plain node-local buffer here, with no sync listener and no peer traffic.
+      AlarmHistorian__Enabled: "true"
+      AlarmHistorian__MaxAttempts: "1000000"
+      ServerHistorian__Endpoint: "${OTOPCUA_HISTORIAN_ENDPOINT:-http://histgw-absent.invalid:5222}"
+      # Required whenever the section is consumed: the gateway client validates its own options at
+      # construction and throws on an empty key, so a keyless node crash-loops during Akka startup
+      # rather than degrading. DEV-ONLY placeholder — it authenticates nothing, because the endpoint
+      # above resolves nowhere. NEVER reuse this shape for a real gateway key; supply those via
+      # ServerHistorian__ApiKey from the environment.
+      ServerHistorian__ApiKey: "${OTOPCUA_HISTORIAN_APIKEY:-histgw_dev_rig-no-gateway-present}"
+      # Must agree with the endpoint scheme above — the client throws on a mismatch in EITHER
+      # direction, so an http endpoint with the default UseTls=true crash-loops the host.
+      ServerHistorian__UseTls: "false"
+      LocalDb__Path: "/app/data/otopcua-localdb.db"
     ports:
       - "4845:4840"
+    volumes:
+      - otopcua-localdb-site-b-2:/app/data
 
   traefik:
     image: traefik:v3.1
@@ -371,3 +538,13 @@ services:
 volumes:
   # SQL Server data dir — persists the OtOpcUa ConfigDb across container recreates.
   otopcua-mssql-data:
+  # Per-node pair-local config-cache (ZB.MOM.WW.LocalDb) files. One volume per node — the
+  # cache is node-local, not shared; it persists the cached deployment artifact (and, on the
+  # site-a pair, the replicated oplog state) across container recreates so a node can boot
+  # from cache after a restart even with central SQL down.
+  otopcua-localdb-central-1:
+  otopcua-localdb-central-2:
+  otopcua-localdb-site-a-1:
+  otopcua-localdb-site-a-2:
+  otopcua-localdb-site-b-1:
+  otopcua-localdb-site-b-2:

docker-dev/seed/seed-clusters.sql

@@ -64,13 +64,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ServerCluster WHERE ClusterId = 'SITE-B')
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-1:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('central-1:4053', 'MAIN', 'central-1', 4840, 8081, 'urn:OtOpcUa:central-1', 200, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('central-1:4053', 'MAIN', 'central-1', 4840, 8081, 4053, 'urn:OtOpcUa:central-1', 200, 1, 'docker-dev-seed');
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('central-2:4053', 'MAIN', 'central-2', 4840, 8081, 'urn:OtOpcUa:central-2', 150, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('central-2:4053', 'MAIN', 'central-2', 4840, 8081, 4053, 'urn:OtOpcUa:central-2', 150, 1, 'docker-dev-seed');
 
 ------------------------------------------------------------------------------
 -- ClusterNode — site A
@@ -78,13 +78,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'central-2:4053')
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-1:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('site-a-1:4053', 'SITE-A', 'site-a-1', 4840, 8081, 'urn:OtOpcUa:site-a-1', 200, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('site-a-1:4053', 'SITE-A', 'site-a-1', 4840, 8081, 4053, 'urn:OtOpcUa:site-a-1', 200, 1, 'docker-dev-seed');
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('site-a-2:4053', 'SITE-A', 'site-a-2', 4840, 8081, 'urn:OtOpcUa:site-a-2', 150, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('site-a-2:4053', 'SITE-A', 'site-a-2', 4840, 8081, 4053, 'urn:OtOpcUa:site-a-2', 150, 1, 'docker-dev-seed');
 
 ------------------------------------------------------------------------------
 -- ClusterNode — site B
@@ -92,13 +92,13 @@ IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-a-2:4053')
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-1:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('site-b-1:4053', 'SITE-B', 'site-b-1', 4840, 8081, 'urn:OtOpcUa:site-b-1', 200, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('site-b-1:4053', 'SITE-B', 'site-b-1', 4840, 8081, 4053, 'urn:OtOpcUa:site-b-1', 200, 1, 'docker-dev-seed');
 
 IF NOT EXISTS (SELECT 1 FROM dbo.ClusterNode WHERE NodeId = 'site-b-2:4053')
     INSERT INTO dbo.ClusterNode
-        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
-    VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed');
+        (NodeId, ClusterId, Host, OpcUaPort, DashboardPort, AkkaPort, ApplicationUri, ServiceLevelBase, Enabled, CreatedBy)
+    VALUES ('site-b-2:4053', 'SITE-B', 'site-b-2', 4840, 8081, 4053, 'urn:OtOpcUa:site-b-2', 150, 1, 'docker-dev-seed');
 
 ------------------------------------------------------------------------------
 -- Galaxy MxAccess gateway — INTENTIONALLY NOT SEEDED (retired 2026-06-15)

docs/AlarmHistorian.md

@@ -42,7 +42,7 @@ unless noted.
 - **`NullAlarmHistorianSink`** — the no-op default for tests and deployments
   that don't historize alarms. It is the default DI binding (registered in the
   Runtime's `AddOtOpcUaRuntime`); production overrides it with
-  `SqliteStoreAndForwardSink`.
+  `LocalDbStoreAndForwardSink`.
 - **`AlarmHistorianEvent`**
   ([`AlarmHistorianEvent.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmHistorianEvent.cs))
   — the source-agnostic event record: `AlarmId`, `EquipmentPath` (UNS path,
@@ -61,42 +61,84 @@ unless noted.
 - **`HistorianSinkStatus`** — diagnostic snapshot surfaced to the AdminUI and
   `/healthz`: `QueueDepth`, `DeadLetterDepth`, `LastDrainUtc`, `LastSuccessUtc`,
   `LastError`, `DrainState`, and `EvictedCount`.
-- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff`.
+- **`HistorianDrainState`** — `Disabled` / `Idle` / `Draining` / `BackingOff` /
+  **`NotPrimary`** (ticking, but leaving the replicated queue for the node that
+  holds the Primary role).
 
 ---
 
-## SqliteStoreAndForwardSink
+## LocalDbStoreAndForwardSink
 
-[`SqliteStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs)
-is the production `IAlarmHistorianSink`. Construction takes a SQLite database
-path, an `IAlarmHistorianWriter`, a logger, and optional `batchSize` (default
-100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30 days),
-and a test clock.
+[`LocalDbStoreAndForwardSink.cs`](../src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/LocalDbStoreAndForwardSink.cs)
+is the production `IAlarmHistorianSink`. Construction takes the node's
+`ILocalDb`, an `IAlarmHistorianWriter`, a logger, and optional `batchSize`
+(default 100), `capacity` (default 1,000,000), `deadLetterRetention` (default 30
+days), a test clock, and a **`drainGate`**.
+
+### The drain gate
+
+`drainGate` is a `Func<bool>` consulted at the top of every tick; `false` skips
+the tick entirely, leaving the queue untouched and reporting
+`DrainState = NotPrimary`.
+
+It exists because the queue **replicates**. The Secondary holds a full copy of
+the Primary's rows, and an ungated drain there would re-deliver every event,
+continuously — not just at a failover. Runtime supplies the gate from
+`IRedundancyRoleView`, a singleton `DriverHostActor` publishes its
+`PrimaryGatePolicy` verdict to on every redundancy snapshot, so the drain agrees
+with the inbound-write and native-ack gates by construction.
+
+Two postures are deliberate:
+
+- **Unset (the default) means drain.** So does an unpublished view. A deployment
+  that runs no redundancy never publishes a role, and defaulting closed would
+  silently stop its alarm history forever.
+- **A gate that throws means "not now", never permission.** Draining on a gate
+  fault would put a pair into dual delivery exactly when its redundancy signal is
+  sick.
 
 ### Queue table
 
-The sink owns one SQLite table (created on construction, WAL journal mode):
+The sink owns one table in the node's consolidated LocalDb — created and
+registered for replication by `LocalDbSetup.OnReady`, before any consumer can
+resolve the sink:
 
 ```sql
-CREATE TABLE Queue (
-    RowId          INTEGER PRIMARY KEY AUTOINCREMENT,
-    AlarmId        TEXT    NOT NULL,
-    EnqueuedUtc    TEXT    NOT NULL,
-    PayloadJson    TEXT    NOT NULL,   -- JSON-serialized AlarmHistorianEvent
-    AttemptCount   INTEGER NOT NULL DEFAULT 0,
-    LastAttemptUtc TEXT    NULL,
-    LastError      TEXT    NULL,
-    DeadLettered   INTEGER NOT NULL DEFAULT 0
+CREATE TABLE alarm_sf_events (
+    id               TEXT    NOT NULL PRIMARY KEY,   -- SHA-256 of payload_json
+    alarm_id         TEXT    NOT NULL,
+    enqueued_at_utc  TEXT    NOT NULL,
+    payload_json     TEXT    NOT NULL,   -- JSON-serialized AlarmHistorianEvent
+    attempt_count    INTEGER NOT NULL DEFAULT 0,
+    last_attempt_utc TEXT    NULL,
+    last_error       TEXT    NULL,
+    dead_lettered    INTEGER NOT NULL DEFAULT 0
 );
-CREATE INDEX IX_Queue_Drain ON Queue (DeadLettered, RowId);
+CREATE INDEX ix_alarm_sf_events_drain
+    ON alarm_sf_events (dead_lettered, enqueued_at_utc, id);
 ```
 
-`EnqueueAsync` does a single `INSERT` on the hot path. To avoid a
-`SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory non-dead-lettered
-row counter (seeded at startup, kept current by every mutation, and re-synced
-from storage every 10,000 enqueues to defend against drift). SQLite writer
-contention is handled via `PRAGMA busy_timeout=5000` + WAL so an enqueue/drain
-collision waits out the file lock instead of failing fast.
+**The primary key is a hash of the payload, not an autoincrement rowid.** Two
+reasons. Replication converges last-writer-wins *per primary key*, so two nodes
+independently allocating rowid 7 to different alarms would silently overwrite one
+another. And an equal-payload key makes the same event arriving twice collapse
+into one row — which happens for real, because `HistorianAdapterActor`
+default-writes while its redundancy role is unknown and both nodes of a pair
+therefore accept the same fanned transition in every boot window. Two genuinely
+distinct events cannot collide: `AlarmHistorianEvent` carries a full-precision
+timestamp alongside the alarm id, kind, message and user.
+
+Ordering follows from that: the drain reads in `enqueued_at_utc` order (with `id`
+as a tiebreak so the ordering is total), because a hashed key carries no
+insertion sequence.
+
+`EnqueueAsync` does a single `INSERT … ON CONFLICT DO NOTHING` on the hot path.
+To avoid a `SELECT COUNT(*)` on every enqueue, the sink keeps an in-memory
+non-dead-lettered row counter (seeded at startup, kept current by every mutation,
+and re-synced from storage every 10,000 enqueues to defend against drift — now
+doubly warranted, since replication applies the peer's rows straight into the
+table where no local code path can observe them). Pragmas and connection
+lifetime belong to LocalDb; the sink no longer manages either.
 
 ### Drain worker
 
@@ -104,13 +146,17 @@ collision waits out the file lock instead of failing fast.
 `System.Threading.Timer`** (not started automatically — tests drive
 `DrainOnceAsync` deterministically). Each tick:
 
+0. Consults the drain gate. A closed gate returns immediately, leaving the queue
+   untouched and `DrainState = NotPrimary`.
 1. Purges aged dead-lettered rows past the retention window.
-2. Reads up to `batchSize` non-dead-lettered rows in `RowId` order.
+2. Reads up to `batchSize` non-dead-lettered rows in `enqueued_at_utc, id` order.
 3. Rows with un-deserializable payloads are dead-lettered immediately (by their
-   own `RowId`) so they can't stall the queue head.
+   own `id`) so they can't stall the queue head.
 4. The remaining batch is handed to `IAlarmHistorianWriter.WriteBatchAsync`, and
-   each outcome is applied in one transaction: `Ack` deletes the row,
-   `PermanentFail` flips its `DeadLettered` flag, `RetryPlease` bumps its attempt
+   each outcome is applied in one transaction: `Ack` deletes the row (the delete
+   replicates as a tombstone, so the peer drops its copy and a promoted standby
+   cannot re-send it), `PermanentFail` flips its `dead_lettered` flag,
+   `RetryPlease` bumps its attempt
    count and leaves it queued. A row whose `AttemptCount` has reached the configured
    **`MaxAttempts`** cap (default 10) is dead-lettered automatically on the next drain
    tick rather than retried — this breaks infinite retry loops for poison events whose
@@ -131,7 +177,7 @@ an unobserved task exception.
 
 **The durability guarantee is bounded by `capacity` (default 1,000,000 rows).**
 When the non-dead-lettered queue reaches capacity, `EnqueueAsync` evicts the
-oldest non-dead-lettered rows (oldest `RowId` first) to make room, logs a WARN,
+oldest non-dead-lettered rows (oldest `enqueued_at_utc` first) to make room, logs a WARN,
 and increments `HistorianSinkStatus.EvictedCount`. Under a sustained historian
 outage, accepted alarm events can therefore be dropped before delivery. A
 non-zero `EvictedCount` is a data-loss signal that requires operator attention —
@@ -140,7 +186,7 @@ it surfaces silent loss without log scraping.
 ### Dead-letter + operator recovery
 
 `PermanentFail` and corrupt-payload rows are retained in-place with
-`DeadLettered = 1` for the retention window (default 30 days) so operators can
+`dead_lettered = 1` for the retention window (default 30 days) so operators can
 inspect them before the sweeper purges them. `RetryDeadLettered()` is the
 operator action (from the AdminUI) that clears the dead-letter flag and attempt
 count on every dead-lettered row, returning them to the regular queue with a
@@ -173,17 +219,28 @@ the `alerts` topic (future).
 The real sink is opt-in via the `AlarmHistorian` section of `appsettings.json`.
 When `Enabled` is `false` (the default), `AddAlarmHistorian` registers
 `NullAlarmHistorianSink` and the feature is dormant. When `Enabled` is `true`,
-`AddAlarmHistorian` constructs `SqliteStoreAndForwardSink` and registers
+`AddAlarmHistorian` constructs `LocalDbStoreAndForwardSink` and registers
 `GatewayAlarmHistorianWriter` as the `IAlarmHistorianWriter`. This section carries
-**only** the `Enabled` gate + the SQLite store-and-forward knobs — the downstream
+**only** the `Enabled` gate + the store-and-forward knobs — the downstream
 gateway connection (endpoint / key / TLS) is sourced from the `ServerHistorian`
-section (see [Historian.md](Historian.md)).
+section (see [Historian.md](Historian.md)), and the queue's storage location is the
+node's consolidated `LocalDb:Path` database (see
+[Configuration.md](Configuration.md)).
+
+> **Where the queue lives (changed).** The buffer is a table — `alarm_sf_events` — in the node's
+> consolidated LocalDb, not a standalone `alarm-historian.db`. That is what lets it **replicate to
+> the redundant pair peer**, so undelivered alarm history survives losing the node holding it.
+> Two consequences follow, both covered in the
+> [pair-replication runbook](operations/2026-07-20-localdb-pair-replication.md):
+> **only the Primary drains** (the Secondary holds a full replica and would otherwise re-send
+> everything), and **delivery is at-least-once across a failover** by design. On first boot after
+> the upgrade, `AlarmSfLegacyMigrator` copies any existing `alarm-historian.db` across and renames
+> it `.migrated`.
 
 ```json
 {
   "AlarmHistorian": {
     "Enabled": true,
-    "DatabasePath": "C:\\ProgramData\\OtOpcUa\\alarmhistorian.db",
     "BatchSize": 100,
     "DrainIntervalSeconds": 5,
     "Capacity": 1000000,
@@ -194,8 +251,7 @@ section (see [Historian.md](Historian.md)).
 
 | Key | Type | Default | Description |
 |---|---|---|---|
-| `Enabled` | bool | `false` | Enable the SQLite store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. |
-| `DatabasePath` | string | `alarm-historian.db` | Path to the SQLite queue file. Created on first use (WAL mode). Set an **absolute** path in production. |
+| `Enabled` | bool | `false` | Enable the durable store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. Requires LocalDb to be registered — an enabled sink with no `ILocalDb` fails at resolution rather than silently discarding events. |
 | `BatchSize` | int | `100` | Max rows per drain cycle handed to `IAlarmHistorianWriter.WriteBatchAsync`. |
 | `DrainIntervalSeconds` | int | `5` | Seconds between drain-worker ticks. |
 | `Capacity` | long | `1000000` | Max queued rows before the sink evicts the oldest (data-loss signal via `EvictedCount`). |
@@ -207,6 +263,15 @@ section (see [Historian.md](Historian.md)).
 > `RuntimeDb:EventReadsEnabled=true`. The old Wonderware connection keys (`SharedSecret` /
 > `AlarmHistorian:Host`/`Port`/`UseTls`/`ServerCertThumbprint`) were pruned.
 
+> **`DatabasePath` was removed.** The queue no longer owns a file. The key is still *read*, by
+> `AlarmSfLegacyMigrator`, purely to locate a pre-consolidation `alarm-historian.db` to migrate —
+> so leave it in place through the upgrade, then delete it once every node has an
+> `alarm-historian.db.migrated` sidecar. A leftover value configures nothing.
+
+> **`DrainState` gained `NotPrimary`.** A gated-off Secondary reports it instead of `Idle`, so a
+> rising queue depth there is distinguishable from a stalled drain. Both nodes reporting it means
+> nobody is draining — see the runbook.
+
 > Dev and docker-dev deployments leave `Enabled` unset (defaults to `false`) so alarm transitions historize to nowhere unless a HistorianGateway is configured.
 
 ---
@@ -221,3 +286,5 @@ section (see [Historian.md](Historian.md)).
 - [ScriptedAlarms.md](ScriptedAlarms.md) — the scripted-alarm engine that emits
   most events into this sink.
 - [ServiceHosting.md](ServiceHosting.md) — the external HistorianGateway backend.
+- [operations/2026-07-20-localdb-pair-replication.md](operations/2026-07-20-localdb-pair-replication.md)
+  — where the queue lives, the Primary-only drain, and the one-time legacy migration.

docs/AlarmTracking.md

@@ -24,6 +24,163 @@ condition — the dedup logic prefers the richer driver-native record
 because it carries the full operator + raise-time + category metadata
 that the value-driven path collapses.
 
+## v3 Batch 4 — multi-notifier delivery (raw + equipment folders)
+
+Under the v3 dual-namespace address space, a native driver alarm is authored on a
+**raw tag** (`/raw`) and its Part 9 `AlarmConditionState` materializes **once** at the
+raw tag — `ConditionId` = the tag's **RawPath**, parent notifier = the raw device/group
+folder (`ns=Raw`). Because equipment references raw tags (v3 reference-only UNS), the
+same condition is wired as an **event notifier of every referencing equipment folder**
+(`ns=UNS`) via the SDK `AddNotifier` pattern (`OtOpcUaNodeManager.WireAlarmNotifiers`):
+`alarm.AddNotifier(equipFolder, isInverse:true)` + `equipFolder.AddNotifier(alarm)` +
+`EnsureFolderIsEventNotifier(equipFolder)`.
+
+- A **single** `ReportEvent` fans through the SDK notifier graph to the raw device folder,
+  every referencing equipment folder, and up to the Server object. A subscriber at any one
+  root receives **exactly one** copy of the transition — the Part 9 shared-`InstanceStateSnapshot`
+  dedup (`MonitoredItem.QueueEvent` → `IsEventContainedInQueue`), **never** one copy per root.
+  Duplicating the `ReportEvent` per root is rejected by design (distinct EventIds would break
+  Server-object dedup + Part 9 ack correlation).
+- **Teardown is symmetric:** the node manager tracks the wired notifier pairs and calls
+  `RemoveNotifier(..., bidirectional:true)` on rebuild / subtree-removal / reference-removal, so
+  inverse-notifier entries never leak across redeploys.
+- **Ack/confirm/shelve route on `ConditionId = RawPath`** (never `SourceNodeId`) regardless of
+  which notifier root the operator subscribed at — an ack issued from an equipment-folder
+  subscription resolves to the same raw condition.
+- The `alerts`-topic `AlarmTransitionEvent` carries the (possibly empty) **list of referencing
+  equipment paths**, and the AdminUI `/alerts` page shows **one row per condition** (primary
+  identity RawPath + condition NodeId) with the equipment list as display metadata.
+
+## Condition event identity fields (what a client reads on the wire)
+
+Every condition event — native and scripted — carries the mandatory `BaseEventType` identity
+fields, assigned at materialize time in `OtOpcUaNodeManager.MaterialiseAlarmCondition`. The SDK
+does **not** synthesize them on this path (`Create` builds the children from the type definition
+but leaves them unset; `ReportEvent` / `InstanceStateSnapshot` copy children verbatim), so they
+are set explicitly. Leaving them unset shipped them as **null** on every event — see issues #473
+(the `BaseEventType` trio) and #475 (the `ConditionType` classification pair).
+
+| Field | Value | Notes |
+|---|---|---|
+| `EventType` | the **concrete** materialized type (`TypeDefinitionId`) | e.g. `OffNormalAlarmType`; falls back to `AlarmConditionType` for an unknown authored type. Readable as a *field*, not only via an `OfType` where-clause |
+| `SourceNode` | the condition's **own NodeId** — equal to `ConditionId` | The condition **is** the source: an alarm-bearing raw tag materializes only the condition, with no sibling value variable, so there is no other node to point at |
+| `SourceName` | the same identifying id string: **RawPath** (native) / **ScriptedAlarmId** (scripted) | Deliberately the *unique* id, **not** the leaf name |
+| `ConditionName` | the leaf / display name (e.g. `HR200`) | Where the short human-readable name lives |
+| `ConditionClassId` | always **`BaseConditionClassType`** | Part 9's "no condition class modelled" value. Unset shipped `NodeId.Null` (#475) |
+| `ConditionClassName` | always **`"BaseConditionClass"`** | Matches `ConditionClassId`. Unset shipped empty text (#475) |
+| `Quality` | the condition's **source-data quality** — native tracks the source's connectivity (`Good` / `Bad`); scripted takes the worst of its input tags' qualities (#478) | A pure annotation; never alters Active/Acked/Retain. Unset shipped the accidentally-Good default (#477) — see below |
+
+**Why `BaseConditionClassType` and not `ProcessConditionClassType`.** We hold no per-alarm
+classification at the materialize seam, and `ConditionClassId` is a wire contract clients bucket on.
+`BaseConditionClassType` is the honest, spec-conformant report of *"this server does not model
+condition classes"* — it fixes the real defect (a null that breaks conformant clients) without
+asserting a classification we cannot back. `ProcessConditionClassType` — the SDK sample's pick —
+was rejected deliberately: it would be *actively wrong* for a Galaxy alarm whose upstream category is
+Safety or Diagnostics, trading a detectable null for an undetectable lie. Real per-alarm
+classification is a separate future feature: it needs the driver's alarm category, which today lives
+only on the runtime `AlarmEventArgs` transition, carried to the deploy-time authored composition that
+`MaterialiseAlarmCondition` sees. Until then the `IAlarmSource` doc comment claiming the category
+"maps to `ConditionClassName` downstream" describes an intent, not the implementation.
+
+**Why `SourceName` is the id, not the leaf name.** The leaf is ambiguous across devices (`HR200` on
+two PLCs collides) and is already carried by `ConditionName`, so the leaf-name option would add no
+identity while costing uniqueness. Carrying the id makes the `SourceNode`/`SourceName`/`ConditionId`
+triple mutually consistent and unique. This diverges from the loose OPC UA convention that
+`SourceName` mirrors the source node's BrowseName; the divergence is intentional.
+
+**Client guidance:** key on **`ConditionId`** — the condition node's own NodeId, which equals
+`SourceNode`, and whose identifier is the RawPath for native alarms and the ScriptedAlarmId for
+scripted ones. It is the identity ack/confirm/shelve route on. `SourceName` carries the same
+identifier and is unique, so it is safe to key on *by itself* — but do **not** compose it with
+`ConditionName` (`$"{SourceName}.{ConditionName}"`), because `SourceName` already ends in the
+condition's leaf name and the result stutters (`pymodbus/plc/HR200.HR200`).
+
+Wire-level guard: `NativeAlarmEventIdentityFieldDeliveryTests` asserts the three `BaseEventType`
+fields arrive populated on a real subscription using the standard `[EventType, SourceNode,
+SourceName, Time, Message, Severity]` select clause, and — in a second test with its own clause —
+that `ConditionClassId` / `ConditionClassName` do too. The two class fields are declared on
+`ConditionType`, **not** `BaseEventType`, so a client must select them against that type.
+`NodeManagerAlarmSourceFieldsTests` guards the node itself across both realms.
+
+> **Do not correlate live events to HistoryRead on `SourceName` — the two paths disagree.**
+> The HistoryRead *events* projection (`OtOpcUaNodeManager.ProjectEventField`) returns
+> `Variant.Null` for `EventType` / `SourceNode` **by design**: it projects from the historian's
+> `HistoricalEvent` rows, which do not carry them. It **does** project `SourceName` — but the
+> alarm-history writer stamps that field with the **EquipmentPath**
+> (`AlarmEventMapper`: `SourceName = alarm.EquipmentPath`), not the RawPath a live event carries.
+> So `SourceName` is the one field populated on both paths **with different values**, and it is not
+> a live↔history join key. Correlate on `ConditionId` / the RawPath instead. (Pre-existing; the
+> live-path fix above does not change the history path.)
+
+### Condition source-data Quality (#477)
+
+`ConditionType.Quality` reports the quality of the condition's source data. It was never assigned, and
+because `StatusCodes.Good == 0x00000000` an unassigned `StatusCode` **is** `Good` — so every condition
+reported `Good` unconditionally (a wrong *value*, not a null like #473/#475). A native alarm whose device
+went offline still read `Good`, so an operator could not tell *"genuinely inactive"* from *"we have lost
+contact and do not know"*.
+
+**How it is driven now (native alarms).** An alarm-bearing raw tag materializes a condition with **no
+sibling value variable**, so the value/quality path (`WriteValue`) never touches it, and a comms-lost
+driver emits **no alarm transitions** (the feed goes silent). The quality therefore comes from the
+**driver's connectivity**, out of band from alarm transitions:
+
+- `DriverInstanceActor` Tells its host `ConnectivityChanged(driverInstanceId, connected)` on every
+  transition into `Connected` (`true`) / `Reconnecting` (`false`).
+- `DriverHostActor.OnDriverConnectivityChanged` fans that out to **every** native condition the driver
+  owns as an `OpcUaPublishActor.AlarmQualityUpdate` (`Good` on connect, `Bad` on disconnect).
+- `OtOpcUaNodeManager.WriteAlarmQuality` sets **only** the condition's `Quality` and fires a Part 9
+  event **only on a quality-bucket change** — it never touches Active/Acked/Severity/Retain (an active
+  alarm that loses comms stays active). This is a dedicated path, *not* a full-snapshot re-projection, so
+  it cannot clobber a condition's severity/message and works for a condition that never fired a transition.
+- A freshly materialized native condition starts `BadWaitingForInitialData` (the "no driver data yet"
+  convention value variables use); the first `Connected` confirms it `Good`.
+- The connectivity annotation is **ungated by redundancy role** (a Secondary keeps its condition quality
+  warm for failover) and publishes **no `/alerts` row** — driver comms health already has its own status
+  surface (`IDriverHealthPublisher`); a row per condition would be alarm-fatigue.
+
+**Scripted alarms (Layer 3, #478).** A scripted condition's state is computed from one or more input tags,
+so its `Quality` is the **worst** quality across those inputs at evaluation time ("can I trust this
+condition's state?") — mirroring the native OT semantic:
+
+- The mux now forwards each input's source quality (`DependencyValueChanged.Quality`), and the scripted host
+  pushes it into the engine's read cache (previously every mux value was treated as `Good`).
+- The `ScriptedAlarmEngine` computes the worst input quality each evaluation. A **real transition** carries
+  it on the emitted event → `ScriptedAlarmHostActor.ToSnapshot` projects it (so a transition fired while an
+  input is `Uncertain` does not clobber quality back to `Good`).
+- A **Bad input freezes the condition** (`AreInputsReady` holds its state — no transition), exactly like a
+  comms-lost native driver. So a quality-bucket change with no transition is emitted as
+  `EmissionKind.QualityChanged` and routed to the **same** dedicated `AlarmQualityUpdate → WriteAlarmQuality`
+  node path native uses (quality only, one Part 9 event on a bucket change, **no `/alerts` row**, no
+  historian write). `ScriptedAlarmSource` skips `QualityChanged` so it never fabricates a phantom
+  `IAlarmSource` event.
+- An input that has **not been published yet** (cold start) is *not* a quality signal (that is the readiness
+  guard's job) — it contributes `Good`, so scripted conditions don't flash `Bad` at every deploy. The first
+  actually-`Bad` published value flips the bucket and annotates.
+
+**Coverage boundary (#478 as shipped).** Scripted quality tracks input tags whose driver **publishes a
+data change carrying a Bad/Uncertain `StatusCode`** (e.g. an OpcUaClient input forwarding a server's
+per-item Bad). It does **not** yet cover a driver **comms loss**: a poll driver (Modbus/S7) whose device
+goes unreachable emits only `ConnectivityChanged` and goes *silent* on the value feed (see
+`DriverInstanceActor.Reconnecting`), so the scripted engine keeps the last-known Good value and the
+condition stays `Good`. Bridging driver connectivity into scripted inputs — the symmetric of the native
+`OnDriverConnectivityChanged` path above, plus resolving the null-value/cold-start asymmetry (a runtime
+`Bad` with a null value is currently indistinguishable from cold start and contributes `Good`) — is tracked
+as the Layer-4 follow-up (#481).
+
+Guards: `ScriptedAlarmEngineTests` (transition carries `Uncertain`; `Bad` input with no transition emits
+`QualityChanged(Bad)`; restore emits `QualityChanged(Good)`; unchanged bucket emits nothing),
+`ScriptedAlarmSourceTests.QualityChanged_emission_raises_no_alarm_event`,
+`DependencyMuxActorTests.Publish_quality_is_forwarded_on_DependencyValueChanged`, and
+`ScriptedAlarmHostActorTests` (`Bad_quality_dependency_publishes_AlarmQualityUpdate_and_no_alerts`,
+`Transition_snapshot_carries_worst_input_quality`).
+
+Wire-level guard: `NativeAlarmEventIdentityFieldDeliveryTests.Condition_event_Quality_tracks_source_connectivity_on_the_wire`
+subscribes with a `[Quality, Message]` clause (Quality is declared on `ConditionType`) and asserts a
+healthy source reports `Good`, a comms-lost source reports non-`Good`, and recovery returns to `Good` — on
+a real client subscription. `NodeManagerAlarmSourceFieldsTests` guards the node itself + the no-clobber /
+unknown-node-no-op invariants.
+
 ## Galaxy driver path (driver-native)
 
 Restored in PR B.2 of the epic. `GalaxyDriver` implements
@@ -208,12 +365,16 @@ AB CIP ALMD) route to AVEVA Historian via the HistorianGateway:
 - `IAlarmHistorianSink` is the DI-registered intake contract. The
   default binding is `NullAlarmHistorianSink` (registered in
   `ServiceCollectionExtensions.AddOtOpcUaRuntime`). Production
-  deployments override it with `SqliteStoreAndForwardSink` wrapping
+  deployments override it with `LocalDbStoreAndForwardSink` wrapping
   `GatewayAlarmHistorianWriter` (the HistorianGateway `SendEvent` path)
   — see [ServiceHosting.md](ServiceHosting.md) for the HistorianGateway setup.
-- `SqliteStoreAndForwardSink` queues each transition to a local
-  SQLite database and drains in the background via an
-  `IAlarmHistorianWriter`. **The durability guarantee is bounded**: the
+- `LocalDbStoreAndForwardSink` queues each transition into the node's
+  consolidated LocalDb — where it **replicates to the redundant pair peer**,
+  so undelivered alarm history survives losing the node holding it — and
+  drains in the background via an `IAlarmHistorianWriter`. **Only the node
+  holding the Primary role drains**, since the peer holds a full replica;
+  delivery is consequently at-least-once across a failover, by design. See
+  [AlarmHistorian.md](AlarmHistorian.md). **The durability guarantee is bounded**: the
   queue capacity defaults to 1,000,000 rows; under a sustained
   historian outage, older non-dead-lettered rows are evicted (oldest
   first) to make room for new events. The `HistorianSinkStatus.EvictedCount`
@@ -222,8 +383,9 @@ AB CIP ALMD) route to AVEVA Historian via the HistorianGateway:
   capacity, and dead-letter retention are tunable via the `AlarmHistorian`
   config section (`DrainIntervalSeconds`, `Capacity`,
   `DeadLetterRetentionDays`); `AlarmHistorianOptions.Validate()` logs a
-  startup warning for an empty `SharedSecret`, a relative `DatabasePath`,
-  or a non-positive knob.
+  startup warning for a non-positive knob. (It no longer warns about
+  `SharedSecret` or `DatabasePath` — both keys are gone: the connection
+  comes from `ServerHistorian`, and the queue lives in the LocalDb.)
 - `HistorianAdapterActor`
   (`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/HistorianAdapterActor.cs`)
   subscribes to the cluster `alerts` DPS topic, translates each

docs/Configuration.md

@@ -101,13 +101,15 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
 |---|---|---|---|
 | `SystemName` | string | `otopcua` | Akka actor-system name. |
 | `Hostname` | string | `0.0.0.0` | Bind hostname. |
-| `Port` | int | `4053` | Cluster transport port. |
-| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. |
-| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. |
+| `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. |
+| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. |
+| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). |
 | `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`. |
 
 > The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.
 
+> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
+
 ### `ConnectionStrings` → `ConfigDb`
 
 - **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally.
@@ -136,8 +138,9 @@ The historian backend is the external **`ZB.MOM.WW.HistorianGateway`** sidecar,
 `ZB.MOM.WW.HistorianGateway.Client` gRPC package (the retired Wonderware TCP sidecar is documented at
 [`docs/drivers/Historian.Wonderware.md`](drivers/Historian.Wonderware.md)). The OtOpcUa host reads three
 appsettings sections — `ServerHistorian` (read path + gateway connection), `ContinuousHistorization`
-(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (SQLite store-and-forward
-alarm sink draining to `SendEvent`). The gateway connection (endpoint / key / TLS) lives **only** in
+(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (the store-and-forward
+alarm sink draining to `SendEvent`; its buffer lives in the node's `LocalDb:Path` database, not a
+file of its own). The gateway connection (endpoint / key / TLS) lives **only** in
 `ServerHistorian`; the other two sections source it from there.
 
 The gateway API key is supplied via the environment variable **`ServerHistorian__ApiKey`** — never committed
@@ -146,6 +149,28 @@ key must carry the scopes `historian:read`, `historian:write`, `historian:tags:w
 [`docs/Historian.md`](Historian.md) for the full key reference, the migration note (old Wonderware keys →
 gateway keys), and the deployment prerequisites.
 
+### `LocalDb` (pair-local config cache — driver role only)
+
+Driver-role nodes keep a consolidated **`ZB.MOM.WW.LocalDb`** SQLite database that caches the deployed
+configuration artifact (chunked) so the node can **boot from cache when central SQL is unreachable**. The
+section is bound by `AddOtOpcUaLocalDb` (`Host/Configuration/LocalDbRegistration.cs`) inside the `hasDriver`
+branch — **admin-only nodes never register it, and never require `LocalDb:Path`**. Storage is unconditional;
+replication stays inert until the sync port + peer are set.
+
+| Key | Default | Meaning |
+|---|---|---|
+| `LocalDb:Path` | `./data/otopcua-localdb.db` | SQLite file path. `ValidateOnStart`-required once `AddZbLocalDb` runs (driver nodes only). In containers put it on a durable volume. |
+| `LocalDb:SyncListenPort` | `0` | `0` = replication off (no sync listener). A non-zero port binds a dedicated **h2c** sync listener; set the **same** port on both pair nodes. Setting it makes the Host add an explicit Kestrel `Listen*` and re-bind the primary HTTP port (an explicit `Listen*` otherwise discards `ASPNETCORE_URLS`). |
+| `LocalDb:Replication:PeerAddress` | `""` | The peer this node **dials** (`http://<peer>:<port>`). Set on the initiator only; leave empty on the passive node. The stream is bidirectional. |
+| `LocalDb:Replication:ApiKey` | `""` | Bearer token for the sync stream. **Fail-closed:** no key ⇒ the passive endpoint refuses everything; a mismatch ⇒ the pair silently stops converging. **Must be byte-identical on both nodes.** Supply via `${secret:...}` / env — never a cleartext literal in production. |
+| `LocalDb:Replication:MaxBatchSize` | (library default) | Rows per replication batch — **row-count-only** against gRPC's 4 MB cap. `16` for the artifact cache (chunk rows ≈171 KB, so 16 × 171 KB ≈ 2.7 MB). |
+
+Replication is **default-OFF** across the fleet; it is enabled per-pair as an opt-in. See
+[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
+for the enablement runbook (ApiKey handling, stop/start-together rule, tombstone-retention window, and the
+never-`sqlite3`-a-live-WAL-DB inspection rules) and the "pair-local config cache" section of
+[`docs/Redundancy.md`](Redundancy.md) for what boot-from-cache does and does not cover.
+
 ---
 
 ## Environment variables

docs/Historian.md

@@ -199,6 +199,15 @@ The server supports all four OPC UA HistoryRead variants:
 `Historizing=true` and `AccessLevels.HistoryRead` are set at materialization so any compliant
 OPC UA client can discover historized capability from the node's attributes.
 
+> **v3 Batch 4 — dual-namespace registration.** A historized raw tag surfaces as **two**
+> variable nodes: the Raw-namespace node (`ns=Raw, s=<RawPath>`) and, for each referencing
+> equipment, a UNS-namespace node (`ns=UNS, s=<Area>/<Line>/<Equipment>/<EffectiveName>`).
+> **Both NodeIds register the SAME historian tagname** (the raw tag's `FullName` /
+> `historianTagname`), so HistoryRead against either NodeId resolves to the same tagname and
+> returns the same series. The historization intent is single-sourced — the mux's
+> `HistorizedTagRef` set stays keyed by RawPath (no doubles), and continuous historization /
+> `EnsureTags` provisioning run once per raw tag regardless of how many equipment reference it.
+
 **Equipment-folder event-notifier nodes** serve Event history. Every equipment folder that
 owns at least one alarm condition is already an event notifier; the server registers a
 `sourceName` (the equipment id) for each such folder and maps event history reads to the
@@ -355,31 +364,39 @@ for the full alarm-historian routing.
 The `historyread` command reads historical data from any node. Supply start and end times in
 ISO 8601 UTC form. See [docs/Client.CLI.md](Client.CLI.md) for the full flag reference.
 
+> **v3 Batch 4 NodeIds.** A historized tag is readable through **either** of its two NodeIds —
+> the Raw-namespace node (`s=<RawPath>`, e.g. `Plant/Modbus/dev1/Speed`) or a referencing
+> equipment's UNS-namespace node (`s=<Area>/<Line>/<Equipment>/<EffectiveName>`). Both register
+> the same historian tagname, so the returned series is identical. The `ns=N` index is assigned
+> at connect time per the namespace URI (`https://zb.com/otopcua/raw` /
+> `https://zb.com/otopcua/uns`) — resolve it from the server's `NamespaceArray` rather than
+> hard-coding it. The examples below show a Raw-namespace read (`ns=2` illustrative).
+
 ```bash
-# Raw history for a historized Galaxy tag (last 24 hours by default)
+# Raw history for a historized tag via its Raw-namespace NodeId (last 24 hours by default)
 otopcua-cli historyread \
   -u opc.tcp://localhost:4840/OtOpcUa \
-  -n "ns=2;s=EQ-55297329838d/GalaxyTestTag" \
+  -n "ns=2;s=Plant/Modbus/dev1/Speed" \
   --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z"
 
-# Limit to 100 values
+# Same series via a referencing equipment's UNS-namespace NodeId, limited to 100 values
 otopcua-cli historyread \
   -u opc.tcp://localhost:4840/OtOpcUa \
-  -n "ns=2;s=EQ-55297329838d/GalaxyTestTag" \
+  -n "ns=3;s=filling/line1/station1/Speed" \
   --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \
   --max 100
 
 # 1-hour average aggregate
 otopcua-cli historyread \
   -u opc.tcp://localhost:4840/OtOpcUa \
-  -n "ns=2;s=EQ-55297329838d/GalaxyTestTag" \
+  -n "ns=2;s=Plant/Modbus/dev1/Speed" \
   --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \
   --aggregate Average --interval 3600000
 
 # Authenticated read (ReadOnly role or higher required)
 otopcua-cli historyread \
   -u opc.tcp://localhost:4840/OtOpcUa \
-  -n "ns=2;s=EQ-55297329838d/GalaxyTestTag" \
+  -n "ns=2;s=Plant/Modbus/dev1/Speed" \
   --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \
   -U reader -P password
 ```

docs/README.md

@@ -30,7 +30,7 @@ The project was originally called **LmxOpcUa** (a single-driver Galaxy/MXAccess
 | [Subscriptions.md](v1/Subscriptions.md) | Monitored items → `ISubscribable` + per-driver subscription refcount (v1 archive) |
 | [AlarmTracking.md](AlarmTracking.md) | `IAlarmSource` + `AlarmSurfaceInvoker` + OPC UA alarm conditions — native Galaxy alarms end-to-end (live) |
 | [AlarmTracking.md](v1/AlarmTracking.md) | Original alarm-tracking write-up (v1 archive) |
-| [AlarmHistorian.md](AlarmHistorian.md) | `Core.AlarmHistorian` store-and-forward SQLite sink — `SqliteStoreAndForwardSink`, `IAlarmHistorianWriter`, dead-letter/retry/eviction |
+| [AlarmHistorian.md](AlarmHistorian.md) | `Core.AlarmHistorian` store-and-forward sink — `LocalDbStoreAndForwardSink`, the replicated `alarm_sf_events` buffer + Primary-only drain, `IAlarmHistorianWriter`, dead-letter/retry/eviction |
 | [DataTypeMapping.md](v1/DataTypeMapping.md) | Per-driver `DriverAttributeInfo` → OPC UA variable types (v1 archive — live mapping is in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/Browse/DataTypeMap.cs`) |
 | [IncrementalSync.md](IncrementalSync.md) | Address-space rebuild on redeploy + `sp_ComputeGenerationDiff` |
 | [HistoricalDataAccess.md](v1/HistoricalDataAccess.md) | `IHistoryProvider` as a per-driver optional capability (v1 archive) |

docs/Redundancy.md

@@ -6,7 +6,7 @@ OtOpcUa supports OPC UA **non-transparent** warm/hot redundancy. Two or more `Ot
 
 > **Discovery surface.** The `ServerArray` path on the `Server` object is what each node populates with self + peer `ApplicationUri`s — see `OpcUaApplicationHost.PopulateServerArray` and the per-node `PeerApplicationUris` option below. The redundancy-object-type `ServerUriArray` proper (a child of `Server.ServerRedundancy`) remains deferred pending an SDK object-type upgrade; clients should read `Server.ServerArray` for peer discovery today.
 
-> **v2 change.** v1's operator-managed `ClusterNode.RedundancyRole` column + `RedundancyCoordinator` / `ApplyLeaseRegistry` / `PeerHttpProbeLoop` are gone. Primary/secondary is now derived from **Akka cluster role-leader** for the `driver` role. The operator no longer writes a role into the DB; cluster topology (specifically the `driver` role-leader) drives ServiceLevel automatically.
+> **v2 change.** v1's operator-managed `ClusterNode.RedundancyRole` column + `RedundancyCoordinator` / `ApplyLeaseRegistry` / `PeerHttpProbeLoop` are gone. Primary/secondary is now derived from **Akka cluster membership** — the oldest Up member carrying the `driver` role. The operator no longer writes a role into the DB; cluster topology drives ServiceLevel automatically.
 
 The runtime pieces live in:
 
@@ -19,7 +19,7 @@ The runtime pieces live in:
 | `DbHealthProbeActor` | `OtOpcUa.Runtime.Health` | Per-node; runs `SELECT 1` against ConfigDb every 5s. Read by the health endpoint AND by `OpcUaPublishActor` (the `DbReachable` ServiceLevel input). |
 | `PeerProbeSupervisor` | `OtOpcUa.Runtime.Health` | Per-node; subscribes to the `redundancy-state` topic and maintains one `PeerOpcUaProbeActor` child per OTHER driver-role peer (spawn on join, stop on departure), so every node is continuously probed by its peers. |
 | `PeerOpcUaProbeActor` | `OtOpcUa.Runtime.Health` | Spawned by `PeerProbeSupervisor`; pings a peer `opc.tcp://peer:4840` with a TCP connect (2s timeout) and publishes `OpcUaProbeResult` on the `redundancy-state` topic. A full secure-channel Hello handshake is a possible future upgrade; the TCP connect is the current real probe. |
-| `ClusterRoleInfo` | `OtOpcUa.Cluster` | Live view of cluster membership + role-leader; exposes `IClusterRoleInfo` to the rest of the host. |
+| `ClusterRoleInfo` | `OtOpcUa.Cluster` | Live view of cluster membership + role-leader; exposes `IClusterRoleInfo` to the rest of the host. (Akka's role-leader is *not* what elects the redundancy Primary — see below.) |
 
 ## ServiceLevel tiers
 
@@ -38,7 +38,7 @@ The four inputs are sourced locally per driver node:
 | `DbReachable` | Local `DbHealthProbeActor` — `OpcUaPublishActor` Asks it on each `HealthTick`; an Ask timeout is treated as `Reachable=false`. |
 | `OpcUaProbeOk` | Result of a peer probing THIS node's OPC UA endpoint: `PeerProbeSupervisor` spawns one `PeerOpcUaProbeActor` per OTHER driver-role peer; each probe publishes `OpcUaProbeResult(probed-node, ok)` on the `redundancy-state` topic; the publish actor consumes only results whose target is itself. Freshness-debounced: absent or stale (>30 s) → `true` (benefit of the doubt — single-node clusters and a departed peer never demote); only an actively-observed RECENT `false` demotes. |
 | `Stale` (derived) | `!DbReachable \|\| (now − lastDbHealth.AsOfUtc) > 30 s \|\| (now − snapshotEntry.AsOfUtc) > 30 s`. |
-| `IsDriverRoleLeader` | The local node's entry in the `RedundancyStateChanged` snapshot from `RedundancyStateActor`. |
+| `IsDriverPrimary` | The local node's entry in the `RedundancyStateChanged` snapshot from `RedundancyStateActor`. |
 
 The resulting truth table (all tiers are now reachable at runtime):
 
@@ -47,8 +47,8 @@ The resulting truth table (all tiers are now reachable at runtime):
 | Down / Detached | 0 | Member status is not `Up` or `Joining` (leaving, removed, exiting), OR node has no `driver` role (Detached). Published immediately — a starting or detached node never leaves the SDK default 255. |
 | Critically degraded | 100 | ConfigDb unreachable AND data is stale. |
 | Stale | 200 | Data stale but ConfigDb reachable. |
-| Healthy follower | 240 | DB reachable + OPC UA probe ok + not stale + not role-leader. |
-| Healthy leader | 250 | Same as healthy follower + this node is the `driver` role-leader (+10 bonus). |
+| Healthy follower | 240 | DB reachable + OPC UA probe ok + not stale + not the driver Primary. |
+| Healthy primary | 250 | Same as healthy follower + this node is the driver Primary (+10 bonus). |
 
 > **Secondary 100 → 240 (behavior change).** Previously a healthy Secondary
 > published 100 (coarse role-only mapping). It now publishes **240** — both
@@ -66,9 +66,38 @@ ServiceLevel (even 0) is always published so no node lingers at the SDK default
 255.
 
 Roles come from `RedundancyStateActor.BuildSnapshot`: a node with the `driver`
-role is `Primary` when it holds the `driver` role-leader lease, otherwise
+role is `Primary` when it is the **oldest Up member carrying the `driver` role**, otherwise
 `Secondary`; a node without the `driver` role is `Detached`.
 
+**Oldest, not role leader (changed 2026-07-21).** The election previously used
+`ClusterState.RoleLeader("driver")` — the *lowest-addressed* Up member with the role. Akka offers
+both, and they name different nodes: role leader is ordered by address (host, then port), oldest by
+up-number. They agree on a freshly-formed cluster, and diverge after any restart — a restarted node
+re-joins as the *youngest* while keeping its address, so if it holds the lower address it becomes
+role leader. Meanwhile `ClusterSingletonManager` places singletons on the **oldest**, so the old
+derivation could name a Primary that was not hosting the singletons, enabling the Primary-gated data
+plane (writes, alarm acks, the alerts emit, the alarm-history drain) on the wrong node. Pinned by
+`RedundancyPrimaryElectionTests`, which forms a real two-node cluster where the oldest node holds the
+*higher* address so the two orderings genuinely disagree. `NodeRedundancyState.IsRoleLeaderForDriver`
+was renamed `IsDriverPrimary` to stop the name asserting the old derivation.
+
+> **KNOWN LIMITATION — the election is per *Akka* cluster, not per application `Cluster`.**
+> The election yields exactly **one** Primary across the whole Akka cluster. A fleet that
+> runs several application clusters (each with its own redundant pair) inside one Akka cluster —
+> which is what `docker-dev/docker-compose.yml` models, with MAIN, SITE-A and SITE-B — therefore has
+> one Primary among *all* its driver nodes, not one per pair. Every consumer of the role inherits
+> this: `ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate below.
+>
+> The unit of redundancy everywhere else in the product is the application `Cluster` (`ClusterId`) —
+> it owns drivers, devices and `ClusterNode` rows, and the LocalDb deployment-artifact cache is
+> already keyed by it *so that a pair shares one entry*. Scoping the election the same way is the
+> fix; `RedundancyStateActor` is an admin-role singleton in the same assembly as
+> `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
+>
+> Surfaced by the LocalDb Phase 2 live gate, where it stopped the alarm-history drain on every node
+> in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`. The alarm drain no longer
+> depends on this (it defers only to a node that shares its queue), but the other three gates still do.
+
 ## Data flow
 
 ```
@@ -114,13 +143,18 @@ The admin singleton is the cluster's only `RedundancyStateActor`. If the admin l
 
 Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var:
 
-```json
+```jsonc
 {
   "Cluster": {
     "Hostname": "0.0.0.0",
     "Port": 4053,
     "PublicHostname": "node-a.lan",
-    "SeedNodes": ["akka.tcp://otopcua@node-a.lan:4053"],
+    // Self FIRST, partner second — see "Bootstrap: self-first seed ordering".
+    // node-b.lan's own config lists node-b.lan first and node-a.lan second.
+    "SeedNodes": [
+      "akka.tcp://otopcua@node-a.lan:4053",
+      "akka.tcp://otopcua@node-b.lan:4053"
+    ],
     "Roles": ["admin", "driver"]
   }
 }
@@ -130,9 +164,9 @@ Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var:
 OTOPCUA_ROLES=admin,driver
 ```
 
-Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list.
+Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. List **both** peers in `SeedNodes` on **both** nodes so either can cold-start alone — **and list each node ITSELF first**: the order decides which Akka bootstrap process runs, and a node that lists its partner first can never form the cluster while that partner is down. See [Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering).
 
-There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster role-leadership). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).
+There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).
 
 > **`RedundancyStateActor` NodeId consistency (fixed).** `RedundancyStateActor` now keys each node's `NodeRedundancyState` entry by the canonical `host:port` node id (via a `ToNodeId(Address)` helper mirroring `ClusterRoleInfo.ToNodeId`). Previously it keyed by `member.Address.Host` (host-only, e.g. `central-2`); since every subscriber matches by the canonical `host:port` form, the mismatch silently meant no node ever matched its own entry — all nodes stayed at the default ServiceLevel 255 and never learned their role. This fix makes `RedundancyStateActor` consistent with the stated contract above. Additionally, `RedundancyStateActor` now **re-publishes the current snapshot on a periodic heartbeat (default 10 s)** so any node that subscribes after the last topology-change publish converges within the interval (DistributedPubSub does not replay to late subscribers).
 
@@ -152,55 +186,201 @@ Each node advertises its partner via `OpcUaApplicationHostOptions.PeerApplicatio
 
 Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpointTests` in `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/` — boots two `OpcUaApplicationHost` instances on loopback, asserts a real OPCFoundation client `Session` reading `Server.ServerArray` from Node A sees both URIs.
 
-## Split-brain
+## Split-brain / downing
 
-The split-brain resolver is **active by default**: Akka.Cluster.Hosting's `WithClustering` enables an SBR
-downing provider whenever `ClusterOptions.SplitBrainResolver` is null (it applies
-`SplitBrainResolverOption.Default`, which registers `Akka.Cluster.SBR.SplitBrainResolverProvider`), and that
-provider reads the `split-brain-resolver` HOCON block in `akka.conf`. On top of that,
-`ServiceCollectionExtensions.BuildClusterOptions` sets the typed `ClusterOptions.SplitBrainResolver`
-(`KeepOldestOption { DownIfAlone = true }`) to make the strategy **explicit in code** rather than relying on
-the framework default — it is reinforcing, not the sole activator, and yields the same effective behavior. So
-the cluster is **not** running `NoDowning`; hard-crashed nodes are downed and the cluster recovers (how it
-recovers depends on *which* node was lost — see the table below). (Only an *explicit* `NoDowning`, e.g.
-`akka.cluster.downing-provider-class = ""`, would leave both redundancy sides at ServiceLevel 240
-indefinitely.) The HOCON block carries the tuning: `active-strategy = keep-oldest`, `stable-after = 15s`,
-`keep-oldest.down-if-alone = on`, `failure-detector.threshold = 10.0` (its `active-strategy` + `down-if-alone`
-must stay consistent with the typed option; `stable-after` lives only in HOCON because the typed option can't
-express it).
+**The default downing strategy is `auto-down` (changed 2026-07-21).** It is selected by
+`Cluster:SplitBrainResolverStrategy`, and `ServiceCollectionExtensions.BuildDowningHocon` turns that
+into the `akka.cluster.downing-provider-class` actually installed:
 
-`keep-oldest` is the correct strategy for a 2-node warm-redundancy pair (`keep-majority`/`static-quorum`
-are wrong for two nodes — no majority in a 1-1 split), but its two loss cases recover **differently**, and
-this is inherent to any 2-node cluster:
-
-| Node lost | What keep-oldest does | Recovery |
+| `Cluster:SplitBrainResolverStrategy` | Provider installed | Posture |
 |---|---|---|
-| **Younger** node (crash or partition) | The oldest is on the surviving side → it stays up and downs the younger side (`DownUnreachable`). | **In-place, fast.** The oldest keeps its singletons + `driver` role-leadership; `RedundancyStateActor` re-computes from the post-loss `Cluster.State`. |
-| **Oldest** node (crash or partition) | The lone survivor is "the side **without** the oldest" → keep-oldest downs the **survivor itself** (`DownReachable "including myself"`), and `run-coordinated-shutdown-when-down = on` terminates it. `down-if-alone` does **not** rescue a lone survivor (its rescue branch needs ≥ 2 surviving members, so it is a 3+-node feature). | **Exit-and-rejoin.** The self-downed survivor **and** the crashed oldest are both restarted by the service supervisor and re-form / rejoin. There is a brief restart-window outage; there is **no** in-place takeover. |
+| `auto-down` (**default**) | `Akka.Cluster.AutoDowning` with `auto-down-unreachable-after = 15s` | **Availability.** The leader among the *reachable* members downs the unreachable peer, so a crash of **either** node — oldest included — fails over in place. |
+| `keep-oldest` | `Akka.Cluster.SBR.SplitBrainResolverProvider` reading the `split-brain-resolver` block in `akka.conf` | **Partition-safety.** A partition can never run dual-active. **Not safe for a 2-node pair** — see below. |
 
-**Recovery therefore depends on three things being in place** (the [ScadaBridge](../../ScadaBridge/CLAUDE.md)
-sister project runs the same pattern):
+Any other value **fails the host at startup** rather than falling through to a default.
+
+### Why keep-oldest is no longer the default
+
+The previous documentation in this section — and the code comments behind it — asserted that
+`keep-oldest` with `down-if-alone = on` was "the correct strategy for a 2-node warm-redundancy pair."
+**That was wrong, and the error was not conservative: it described a total-outage configuration as the
+recommended one.**
+
+In Akka.NET 1.5.62's `KeepOldest.OldestDecision`, the `down-if-alone` rescue branch requires the
+*surviving* side to hold **≥ 2 members**. In a two-node cluster a lost peer is always a 1-vs-1 split, so
+the branch never fires and the lone survivor falls through to `DownReachable` — downing **itself** — and
+`run-coordinated-shutdown-when-down = on` then terminates it. `down-if-alone` is a 3+-node feature; it
+does not do what its name suggests here. Live-proven on the sister project's rig, whose survivor logged
+`SBR took decision …DownReachable and is downing [self] including myself, [1] unreachable of [2] members`.
+See [`../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md`](../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md)
+for the upstream decision record, which OtOpcUa follows.
+
+The alternatives do not help a pair: `static-quorum` with quorum 1 hits Akka's `IsTooManyMembers` guard
+and returns `DownAll`; with quorum 2 the survivor downs itself; `keep-majority` keeps the lowest-address
+side, which just moves the fatal crash from "the oldest node" to "the other node"; `lease-majority` needs
+a shared lease store both nodes can reach.
+
+### The accepted trade
+
+Under `auto-down`, a **genuine network partition** (both nodes alive, link cut) leaves each side downing
+the other and continuing alone: **both run active**, both hold singletons, and both advertise the primary
+`ServiceLevel`. Recovery is operator-driven — after the link heals, restart **one** side; it rejoins as a
+fresh incarnation and settles as secondary. The two sides do not merge on their own, because the mutual
+downing quarantines the association.
+
+This is a deliberate choice of availability over partition-safety, matching ScadaBridge: the pairs run one
+node per VM with no external arbiter available, so a crash is a far more likely event than a partition, and
+a crash under `keep-oldest` was unrecoverable without operator action anyway.
+
+Deployments that would rather take an outage than ever run dual-active can set
+`Cluster:SplitBrainResolverStrategy: "keep-oldest"` — but should read the section above first, because for a
+two-node pair that choice means *any* crash of the oldest node is a full outage.
+
+### Recovery still depends on supervision
 
 1. **A service supervisor that restarts the process** on exit — production `Install-Services.ps1` sets
    `sc.exe failure OtOpcUaHost … actions= restart/5000/restart/30000/restart/60000`; the docker-dev rig sets
-   `restart: unless-stopped`. Without it a downed node stays down and an oldest-crash looks like a total outage.
-2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) — it
-   watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls
-   `IHostApplicationLifetime.StopApplication()` so the process exits (and the supervisor restarts it) instead of
-   idling forever with a dead actor system.
-3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join the mesh via either peer.
+   `restart: unless-stopped`.
+2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) —
+   it watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls
+   `IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead
+   of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this
+   (it is never the one downed); the **downed** node still does.
+3. **Both peers listed in `SeedNodes`** on every node, **each node listing itself first**, so a restarted
+   node can re-join via either peer and a node cold-starting while its peer is dead still forms the
+   cluster. Without the self-first ordering it waits in `InitJoin` forever (auto-down removes the crash
+   outage, not that one) — see [Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering).
 
 > **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
-> sole `driver` role-leader, and it passes — but it simulates the crash with `provider.Transport.Shutdown()`,
-> which leaves node A's `ActorSystem` **alive** (a transport partition with the oldest still running), not a
-> real process death. It is therefore **not** representative of an oldest-process crash; a real 2-container
-> `docker kill` of the oldest downs the survivor (verified 2026-07-15, see
-> `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`). Treat the recovery guarantee for
-> the oldest as "exit-and-rejoin under supervision," not "in-place failover."
+> sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with
+> `provider.Transport.Shutdown()`, which leaves node A's `ActorSystem` **alive** (a transport partition with
+> the oldest still running), not a real process death. A real 2-container `docker kill` of the oldest downed
+> the survivor (verified 2026-07-15, `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`).
+> **It is a standing example of a green test over a fatal defect**: keep the distinction between a transport
+> partition and a process death in mind when reading it.
 
-There is no operator-driven role swap during a partition. Failover / recovery is what the cluster + supervisor
-do automatically. **Instant in-place takeover on *any* single-node loss requires 3+ members** (an odd cluster
-or a lightweight witness) — a deliberate future option, not the current 2-node posture.
+`SplitBrainResolverActivationTests` pins the strategy by starting a real host through the production
+bootstrap and reading `akka.cluster.downing-provider-class` back off the running `ActorSystem` — asserting
+the typed option or the akka.conf text is not sufficient, because several HOCON fragments compete and the
+losing one still looks correct in the file.
+
+Failover during a partition is what the cluster and supervisor do automatically. For a *planned* swap —
+patching, draining a node — see [Manual failover](#manual-failover) below.
+
+## Manual failover
+
+The cluster redundancy page (`/clusters/{id}/redundancy`) carries a **Live redundancy** panel — the current
+driver Primary and every Up `driver` member, read from live cluster state — and an admin-gated **Trigger
+failover** button.
+
+The mechanism is a graceful cluster **`Leave`** of the current Primary, never a `Down`:
+`IManualFailoverService.FailOverDriverPrimaryAsync` selects the **oldest Up `driver` member** — the identical
+query to `RedundancyStateActor.SelectDriverPrimary`, pinned together by
+`ManualFailoverServiceTests.Parity_with_SelectDriverPrimary` so the node acted on is exactly the node the
+election names. The leaving node hands its singletons over through the cluster-leave phases,
+`CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it,
+and it rejoins as the **youngest** member — so the survivor is now the oldest, becomes Primary, and
+advertises `ServiceLevel` 250. OPC UA clients re-select.
+
+| Rule | Behaviour |
+|---|---|
+| Authorization | `AdminUiPolicies.FleetAdmin` (**Administrator only**) — gated in markup *and* re-checked server-side before the call. Deliberately not `ConfigEditor`, which the neighbouring cluster pages use: this restarts a production node rather than editing config, and ConfigEditor also admits Designer. |
+| Peer guard | Disabled, with the reason as its tooltip, when fewer than two Up driver members exist — with one node a "failover" is a shutdown. Re-evaluated inside the service against live state, so a stale page cannot bypass it. |
+| Confirmation | A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select). |
+| Audit | Written through the shared `ZB.MOM.WW.Audit.IAuditWriter` seam **before** the Leave — action `cluster.manual-failover`, actor, target. A *refused* failover changes nothing and writes nothing. |
+
+> **Mesh-scope caveat.** Until the per-cluster mesh work lands
+> (`docs/plans/2026-07-21-per-cluster-mesh-design.md`), the Primary is elected once per Akka mesh rather than
+> per application `Cluster` row — so on a fleet running several clusters in one mesh this button acts on the
+> whole mesh's Primary, which may belong to a different cluster than the page you are on. The panel says so.
+> Phase 6 of the mesh program removes both the caveat and this notice.
+
+> **Live gate outstanding.** The auto-down change is verified by unit tests against the effective
+> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has
+> **not** yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where
+> the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster;
+> assert the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) should run
+> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
+> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
+
+## Bootstrap: self-first seed ordering
+
+Listing both peers in `SeedNodes` does **not** mean either node can cold-start alone — and neither does any
+downing strategy: **auto-down removes the crash outage, not this one.** Akka runs a different bootstrap
+process depending on whether `seed-nodes[0]` is the node's own address:
+
+| `seed-nodes[0]` is… | Process Akka runs | Can it form a NEW cluster? |
+|---|---|---|
+| this node's own address | `FirstSeedNodeProcess` | **Yes** — it `InitJoin`s the other seeds and self-joins once `seed-node-timeout` (5 s) passes with nobody answering. |
+| some other node | `JoinSeedNodeProcess` | **No** — it retries `InitJoin` forever, however long the peer stays down. |
+
+So the ordering *is* the mechanism. **Every node that is a seed of its own mesh lists ITSELF first and its
+partner second** (decision 2026-07-22):
+
+```jsonc
+// central-1                                     // central-2
+"SeedNodes": [                                   "SeedNodes": [
+  "akka.tcp://otopcua@central-1:4053",             "akka.tcp://otopcua@central-2:4053",
+  "akka.tcp://otopcua@central-2:4053"              "akka.tcp://otopcua@central-1:4053"
+]                                                ]
+```
+
+The rule is **conditional**: it binds only when a node's own address appears in its own seed list. A
+driver-only site node is seeded solely by `central-1` (today's docker-dev topology) — it is legitimately
+not a seed of anything and is exempt. An unconditional "self must be first" rule would refuse to boot every
+site node in the fleet.
+
+`AkkaClusterOptionsValidator` enforces it at startup (`AddValidatedOptions` → `ValidateOnStart` in
+`AddOtOpcUaCluster`), because the invariant otherwise fails **silently**: the process runs, the port
+listens, the node simply never becomes a member. Identity is compared on `Cluster:PublicHostname` (falling
+back to `Cluster:Hostname` when blank) **and** `Cluster:Port` — the address Akka puts in `SelfAddress`.
+Comparing against `Cluster:Hostname` would be worse than wrong: in docker-dev it is `0.0.0.0`, which matches
+no seed URI anywhere, so the rule would silently exempt the entire rig.
+
+Sequential recovery is island-free by construction: a `FirstSeedNodeProcess` node self-joins only when **no**
+seed answered, so a peer booting after the survivor formed simply joins it as the youngest member. Both
+nodes cold-starting *simultaneously* converge on one cluster while they are mutually reachable — the
+`InitJoin` handshake resolves it before either self-join deadline expires. The residual risk is a boot-time
+**partition** (both cold-starting, mutually unreachable): both form, which is the same dual-active class the
+`auto-down` strategy already accepts, with the same recovery (restart one side).
+
+Pinned by `SelfFirstSeedBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real
+in-process clusters through the production bootstrap at production failure-detection timings: a lone
+cold-start with a dead peer forms alone; a restarted node rejoins its live peer; two nodes cold-starting
+together converge on one cluster. Its **falsifiability control** — the old peer-first ordering, same
+scenario, never comes Up — is what keeps the other three from going vacuous, and carries a positive-control
+self-join proving the node was formable all along. `AkkaClusterOptionsValidatorTests` covers the validator,
+including the site-node exemption and the `0.0.0.0` identity trap.
+
+### Why the self-form watchdog was retired (2026-07-22)
+
+The first fix for this gap was `Cluster:SelfFormAfter` + `ClusterBootstrapFallback`: wait 10 s for
+membership, then call `Cluster.Join(SelfAddress)`. **Both are deleted.** A timer sits *outside* Akka's join
+handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight" —
+and `Cluster.Join(SelfAddress)` is **not** ignored mid-handshake: it wins.
+
+> **Live-gate finding (docker-dev, 2026-07-22 — preserved because it is the evidence).** The watchdog
+> islanded a node that manual failover had bounced. `central-1` restarted, received `InitJoinAck` from
+> `central-2` at 10:49:05 — the join was in flight and healthy — but no Welcome arrived inside the window,
+> because the peer's ring still held its previous incarnation (`Exiting` → `Down` → `Removed`, retired at
+> 10:49:06). At 10:49:15 the watchdog fired and the node formed a **second cluster**, islanded until an
+> operator restarted it. Manual failover deliberately produces exactly that restart, so the two halves of
+> that plan collided. A TCP reachability guard (`ea45ace1`) patched that one shape — never self-form while
+> a seed peer accepts a connection — and the earlier live gate for the watchdog itself had genuinely
+> passed: `central-2` cold-started alone in 10 s and a non-seed `site-a-2` correctly stayed out.
+
+The reachability guard closed the observed failure, not the class: a peer can be TCP-reachable and not
+answering, or answering and slow, and the timer cannot tell. Self-first ordering has no such race because
+the decision is made *inside* the handshake, by Akka, on the actual `InitJoin` replies. It was also inert
+for every non-seed node, so after this change it could never usefully fire. `SelfFormBootstrapTests` is
+replaced by `SelfFirstSeedBootstrapTests`, whose `Restarting_node_rejoins_its_live_peer_instead_of_islanding`
+is the drilled scenario above.
+
+> **Live gate outstanding for the new mechanism.** Self-first ordering is verified here by in-process
+> clusters at production timings, and on the sister project (ScadaBridge `4a6341d8`) on real clusters *and*
+> live docker containers. It has **not** been re-drilled on the OtOpcUa docker-dev rig since the swap — the
+> `central-2` cold-start-alone drill should be repeated on the rebuilt image. Owned by
+> `docs/plans/2026-07-22-per-cluster-mesh-program.md` Phase 7 alongside the auto-down 1-vs-1 gate.
 
 ## Primary data-plane gate (writes, acks, alerts emit)
 
@@ -221,11 +401,44 @@ Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and ev
 
 - **`alerts` topic emission** — `ScriptedAlarmHostActor` and `DriverHostActor.ForwardNativeAlarm` subscribe to the `redundancy-state` DPS topic and cache the local node's `RedundancyRole`, then gate the cluster `alerts` publish through `PrimaryGatePolicy` (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain **ungated** on both nodes so the secondary is always ready to serve clients after a failover.
 - **`HistorianAdapterActor` historization** — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the `alerts` DPS topic and translates each `AlarmTransitionEvent` → `AlarmHistorianEvent` before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
+- **The alarm sink's DRAIN** — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. `LocalDbStoreAndForwardSink` takes a `drainGate` fed from `IRedundancyRoleView` — a singleton `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, because the drain runs on a timer and cannot receive `RedundancyStateChanged` itself. A gated-off node reports `HistorianDrainState.NotPrimary`. Delivery is **at-least-once across a failover** by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See [AlarmHistorian.md](AlarmHistorian.md).
 
 Net effect: each alarm transition appears **once** on `/alerts` and would historize once, not once per node.
 
 See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals.
 
+## Pair-local store (LocalDb — Phases 1 + 2)
+
+Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated
+[`ZB.MOM.WW.LocalDb`](../CLAUDE.md) SQLite database. Phase 2 added the **alarm store-and-forward
+buffer** (`alarm_sf_events`) to it alongside the config cache, so a node that dies holding
+undelivered alarm history no longer takes it to the grave — see the Primary-only drain note above
+and [AlarmHistorian.md](AlarmHistorian.md). Phase 1 caches the **deployed-configuration
+artifact** (chunked). Its job is a single failure mode redundancy does not otherwise cover: a driver
+node restarting into a **central-SQL-Server outage** would come up with no configuration at all. With
+the cache, it **boots from the last artifact it applied** instead, and logs a running-from-cache
+signal.
+
+- **What replicates.** The cache can *optionally* replicate to the node's redundant pair peer over
+  the LocalDb library's gRPC sync (HLC-stamped, last-writer-wins). Two tables replicate:
+  `deployment_artifacts` and `deployment_pointer`. Replication is **default-OFF and fail-closed** —
+  it stays inert until `LocalDb:SyncListenPort` + a peer are configured, and a missing/mismatched
+  `LocalDb:Replication:ApiKey` refuses or silently halts sync. It is enabled per-pair as an opt-in.
+- **The replicated-cache payoff.** In a pair with replication on, *either* node can be the one that
+  applied the latest deploy; the peer holds a byte-identical copy. So a node that restarts during a
+  central outage can boot the current config **even if it never applied that deploy itself** — its
+  peer did, and the row replicated.
+- **What it does NOT cover.** The cache is a **boot fallback for central-SQL outages only**, not a
+  standby data path and not a replacement for central. A **new deployment still requires central
+  SQL**; the cache is read only when the central fetch fails at startup, and the node resumes
+  fresh-config behavior once central returns. It does not replicate live tag values, alarms, or
+  historian data — only the config artifact. It is orthogonal to the ServiceLevel/primary-gate
+  logic: a node running from cache still participates in redundancy normally.
+
+Full detail: the enablement runbook at
+[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
+and the `LocalDb` section of [`docs/Configuration.md`](Configuration.md).
+
 ## Client-side failover
 
 The OtOpcUa Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` supports `-F` / `--failover-urls` for automatic client-side failover; for long-running subscriptions the CLI monitors session KeepAlive and reconnects to the next available server, recreating the subscription on the new endpoint. See [`Client.CLI.md`](Client.CLI.md).

docs/ScriptEditor.md

@@ -212,6 +212,13 @@ The catalog is scoped per Blazor circuit; each call creates and disposes its own
 `DbContext` via the pooled factory (same pattern as `UnsTreeService`). Results
 are bounded to 200 entries to keep the completion list responsive on large fleets.
 
+> **v3 Batch 4 (dual namespace) — unchanged for scripts.** A `ctx.GetTag(...)` literal
+> still resolves by the tag's `FullName` / **RawPath** — which is precisely the identity
+> of the Raw-namespace node (`ns=Raw, s=<RawPath>`). A `{{equip}}/<RefName>` token resolves
+> through the equipment's `UnsTagReference` to that same backing RawPath (see below), so
+> scripts key off the single value source regardless of the UNS projection. The dual
+> namespace changes only the OPC UA address-space surface, not script tag-path semantics.
+
 ### Literal gate
 
 `TryGetTagPathLiteral` identifies the tag-path context by climbing from the
@@ -307,76 +314,88 @@ Register the replacement in `EndpointRouteBuilderExtensions.AddAdminUI`
 
 ### Why
 
-Today each VirtualTag bound to a script typically needs its own near-duplicate
-script because tag paths are hard-coded absolutes (e.g. `TestMachine_001.Speed`).
-The `{{equip}}` token breaks this coupling: point many VirtualTags' `ScriptId` at
-a single template script, and each resolves the token to its own equipment's tag
-base prefix at deploy time. No schema change is required — sharing a `Script`
-record across VirtualTags already works; `{{equip}}` is what makes the shared
-script resolve per-equipment.
+Each VirtualTag bound to a script would otherwise need its own near-duplicate
+script because tag paths are hard-coded absolute RawPaths (e.g.
+`Cell1/Modbus/Dev1/Speed`). The `{{equip}}/<RefName>` token breaks this coupling:
+point many VirtualTags' `ScriptId` at a single template script, and each resolves
+the token through **its own equipment's tag references** at deploy time. No schema
+change is required — sharing a `Script` record across VirtualTags already works;
+`{{equip}}/<RefName>` is what makes the shared script resolve per-equipment.
 
 ### Before / after
 
-**Before — one script per machine:**
+**Before — one script per machine (hard-coded RawPath):**
 
 ```csharp
 // Script "Calc_TestMachine_001" — hard-coded, cannot reuse
-return ctx.GetTag("TestMachine_001.Speed").Value;
+return ctx.GetTag("Cell1/Modbus/Dev1/Speed").Value;
 ```
 
 **After — one shared template:**
 
 ```csharp
 // Script "Calc_Speed" — works for any machine
-return ctx.GetTag("{{equip}}.Speed").Value;
+return ctx.GetTag("{{equip}}/Speed").Value;
 ```
 
-`TestMachine_001` and `TestMachine_002` both bind `ScriptId = "Calc_Speed"`.
-At deploy, each VirtualTag receives its own expanded copy:
-`TestMachine_001.Speed` and `TestMachine_002.Speed` respectively.
+`TestMachine_001` and `TestMachine_002` both bind `ScriptId = "Calc_Speed"`, and
+each has a **tag reference** whose effective name is `Speed`. At deploy, each
+VirtualTag receives its own expanded copy — `{{equip}}/Speed` resolves to that
+equipment's referenced raw tag's RawPath.
 
-### Token rules
+### Token rules (v3)
 
-- The token is `{{equip}}` (double braces, lowercase).
+- The token joint is a **slash**: `{{equip}}/<RefName>` (double-brace stem,
+  lowercase). `<RefName>` is a **reference effective name** — the reference's
+  `DisplayNameOverride`, else the backing raw tag's `Name`.
 - It is substituted **only inside `ctx.GetTag(…)` / `ctx.SetVirtualTag(…)` first-argument
   string literals** — comments, logger strings, and other code are untouched.
-- The operator writes the separator and tail: `ctx.GetTag("{{equip}}.Speed")`,
-  `ctx.GetTag("{{equip}}.Sub.Field")`, etc.
-- The token expands to the equipment's **tag base prefix** — the common
-  substring-before-the-first-dot of that equipment's configured driver-tag
-  `FullName` values. Example: tags `TestMachine_001.Speed` and
-  `TestMachine_001.Temp` → base `TestMachine_001`.
+- The whole post-prefix literal content is the reference name, so effective names
+  with internal spaces are supported: `ctx.GetTag("{{equip}}/Motor Speed")`.
+- `{{equip}}/<RefName>` **resolves through the owning equipment's `UnsTagReference`
+  rows** (by effective name) to the backing raw tag's `RawPath`. Example: a
+  reference named `Speed` backing `Cell1/Modbus/Dev1/Speed` → the token expands to
+  `Cell1/Modbus/Dev1/Speed`.
+- Scripted-alarm predicate scripts resolve `{{equip}}/<RefName>` the same way; alarm
+  **message-template** tokens follow the same resolution rule.
 
 ### Validation requirement
 
-Saving a VirtualTag that is bound to a `{{equip}}`-using script is rejected in
-the AdminUI if the equipment does not have at least one configured driver tag, or
-if its tags span multiple object prefixes (i.e. the common prefix is ambiguous or
-absent). The rejection message is surfaced as a clear validation error on the save
-form. This check is enforced eagerly so that an unresolved `{{equip}}` token —
-which would leave a path that resolves to nothing at runtime (Bad quality) — can
-never reach the deployed artifact.
+Saving a VirtualTag (or ScriptedAlarm) whose script uses `{{equip}}/<RefName>` is
+rejected in the AdminUI when `<RefName>` is not one of the owning equipment's
+reference effective names. The same rule runs at the deploy gate
+(`DraftValidator.ValidateEquipReferenceResolution`, error code
+`EquipReferenceUnresolved`), which also catches a **rename-induced** breakage the
+authoring check never saw. The rejection names the script/alarm, the equipment, and
+the missing ref — so an unresolved token can never reach the deployed artifact.
 
 ### Editor support
 
 In the Monaco script editor (ScriptEdit page and the `/uns` virtual-tag modal's
 inline script panel):
 
-- **Hover** — hovering a `{{equip}}` path literal shows an
-  *"Equipment-relative path — resolved at deploy"* note.
-- **Completions** — typing `{{equip}}.` inside a `GetTag`/`SetVirtualTag` literal
-  offers completion of attribute leaf names (the part after the first dot of known
-  tag references in the catalog).
+- **Hover** — hovering a `{{equip}}/<RefName>` path literal shows an
+  *"Equipment-relative path — resolved through the equipment's tag reference at
+  deploy"* note.
+- **Completions** — typing `{{equip}}/` inside a `GetTag`/`SetVirtualTag` literal
+  offers the owning equipment's **reference effective names** (needs an equipment
+  context; inert on the shared ScriptEdit page).
+- **Diagnostics** — an unresolved `{{equip}}/<RefName>` is flagged (`OTSCRIPT_EQUIPREF`)
+  identically to the deploy gate, preserving *editor accepts ⇔ publish accepts*.
 
 ### Maintainer note
 
-Substitution runs at the two compose seams —
-`Phase7Composer.Compose` and `DeploymentArtifact.BuildEquipmentVirtualTagPlans`
-— via the shared `ZB.MOM.WW.OtOpcUa.Commons.Types.EquipmentScriptPaths` helper,
-**before** dependency extraction. The runtime, the static change-trigger
-dependency graph, and the literal-only path rule are therefore all unchanged:
-by the time they see the script, `{{equip}}` has been replaced with a concrete
-tag-base prefix and the path is a normal string literal.
+Substitution runs at the two compose seams — `AddressSpaceComposer.Compose` and
+`DeploymentArtifact.ParseComposition` — via the shared
+`ZB.MOM.WW.OtOpcUa.Commons.Types.EquipmentScriptPaths.SubstituteEquipmentToken`
+helper, fed the equipment's reference map (effective name → RawPath) built by
+`EquipmentReferenceMap` over the shared `RawPathResolver`. Substitution runs
+**before** dependency extraction, so the runtime, the static change-trigger
+dependency graph, and the literal-only path rule are unchanged: by the time they
+see the script, `{{equip}}/<RefName>` has been replaced with a concrete RawPath and
+the path is a normal string literal. An unresolved ref is left un-substituted (the
+validator rejects it first — substitution never throws). The two seams build the
+reference map identically, so their plans stay byte-parity.
 
 ---
 

docs/ScriptedAlarms.md

@@ -127,9 +127,25 @@ object alongside the usual `"FullName"`:
 (unchanged behaviour). `"alarm"` **present** → the tag materialises as a Part 9
 `AlarmConditionState` under its equipment folder **instead of** a value variable.
 No EF/schema change is required; the intent rides in the schemaless `TagConfig`
-blob and is parsed byte-parity in both the compose (`Phase7Composer`) and deploy
+blob and is parsed byte-parity in both the compose (`AddressSpaceComposer`) and deploy
 (`DeploymentArtifact`) paths.
 
+> **v3 Batch 4 — multi-notifier fan-out.** The `"alarm"` object now rides on a **raw
+> tag** authored in `/raw` (referenced into equipment), and the Part 9 condition
+> materializes **once at the raw tag** — its `ConditionId` is the tag's **RawPath**, its
+> parent notifier is the raw device/group folder. For **each referencing equipment**, the
+> node manager (`WireAlarmNotifiers`) wires the SDK `AddNotifier` pattern
+> (`alarm.AddNotifier(equipFolder, isInverse:true)` + `equipFolder.AddNotifier(alarm)` +
+> `EnsureFolderIsEventNotifier(equipFolder)`) so the single condition fans to the raw device
+> folder AND every equipment folder. A **single** `ReportEvent` fans to all notifier roots —
+> a Server-object subscriber sees **exactly one** copy (the Part 9 shared-snapshot dedup),
+> never one-per-root. Teardown is symmetric (`RemoveNotifier(..., bidirectional:true)` on
+> rebuild / reference removal) so inverse-notifier entries never leak across redeploys.
+> Ack/confirm/shelve route on **`ConditionId = RawPath`** (never `SourceNodeId`), and the
+> `/alerts` row lists the referencing equipment paths. See
+> [AlarmTracking.md](AlarmTracking.md) and the WP4 section of
+> `docs/plans/2026-07-15-v3-batch4-address-space-plan.md`.
+
 ### TagConfig alarm fields
 
 | Field | Values | Default |
@@ -208,9 +224,10 @@ native alarm transitions identically. Publication to the `alerts` topic is
 the OPC UA condition-node write is ungated on all nodes so a Secondary stays warm
 for failover.
 
-The alarm is authored on the `Tags` tab of the equipment page (`/uns/equipment/{id}`)
-by editing the tag's raw `TagConfig` JSON to include the `"alarm"` object. No
-other configuration is required.
+The alarm is authored on the **raw tag** in `/raw` by including the `"alarm"` object in the
+tag's `TagConfig` JSON (v3 Batch 4 — equipment no longer authors tags; it references raw
+tags). No other configuration is required: any equipment that references the raw tag
+automatically receives the alarm at its equipment folder via the multi-notifier fan-out above.
 
 ### Native-alarm OPC UA operator operations
 

docs/ServiceHosting.md

@@ -56,6 +56,7 @@ The host joins an Akka.NET cluster bound to the address in `appsettings.json::Cl
 }
 ```
 
+- `SeedNodes` is **ordered**: a node listed in its own seed list must be entry 0, since Akka only lets `seed-nodes[0]` form a new cluster. `AkkaClusterOptionsValidator` fails startup otherwise — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering).
 - `WithOtOpcUaClusterBootstrap` (in `OtOpcUa.Cluster`) loads the embedded HOCON (split-brain resolver, pinned dispatcher, failure detector tuning) and overlays remote endpoint + cluster options.
 - All cluster singletons + per-node actors live on this single ActorSystem — there is no second Akka instance.
 

docs/Uns.md

@@ -74,26 +74,88 @@ changed by editing the area's cluster in the Area modal, which moves the
 whole branch. There is no separate "served-by" concept and no migration —
 it is simply `UnsArea.ClusterId`.
 
-### Tags
+### Tags — reference-only (v3)
 
-Tags created on the equipment page are **equipment-bound** and require a driver
-instance. The driver list on the Tags tab is scoped to the equipment's cluster
-and to drivers on an **Equipment-kind** namespace, so a driver-less equipment
-shows no eligible drivers until you bind one (edit the equipment on the Details
-tab and pick a driver).
+> **v3 (Batch 3):** UNS equipment no longer authors or binds tags. Device I/O is
+> authored once in the **Raw project tree** (`/raw` — see [`Raw.md`](Raw.md)); an
+> equipment's **Tags** tab holds **references** to those raw tags. The old
+> driver-bound Tag modal on this tab is retired.
 
-**Galaxy / AVEVA System Platform points are ordinary equipment tags** bound to
-a `GalaxyMxGateway` driver instance. Author them on the Tags tab using the
-standard Tag modal; the Galaxy address picker browses the live Galaxy hierarchy
-so you can select the attribute and set `TagConfig.FullName`. There is no
-separate alias concept or `SystemPlatform`-kind namespace.
+The **Tags** tab is a list of `UnsTagReference` rows. Each row shows the
+**effective name**, the backing tag's **RawPath**, its inherited **DataType** and
+**AccessLevel** (read-only — they come from the raw tag), and an editable
+**display-name override**. The effective name is the override else the raw tag's
+`Name`.
+
+**"+ Add reference"** opens a raw-tree picker (the `/raw` tree in picker mode)
+**scoped to the equipment's cluster** — cross-cluster tags are structurally
+unreachable, and the service also enforces `tag.cluster == equipment.cluster` on
+commit. Tick individual tag leaves, or use a device / tag-group's *"Select all
+tags below"* menu to pull many at once.
+
+**Effective-name uniqueness:** within an equipment the effective name must be
+unique across references, VirtualTags, and ScriptedAlarms (they share the
+equipment's UNS NodeId space, `{EquipmentId}/{EffectiveName}`). This is enforced
+both at authoring (a readable rejection naming the colliding source) and at the
+deploy gate (`DraftValidator`, `UnsEffectiveNameCollision`) — the deploy gate is
+what catches **rename-induced** collisions a raw rename produced after the
+reference was authored, naming both colliding sources and the equipment.
+
+Removing a raw tag in `/raw` is blocked while a `UnsTagReference` points at it
+(the error names the referencing equipment); renaming a raw tag or any ancestor
+warns when a beneath-it tag is historized (no `historianTagname` override),
+UNS-referenced, or named by a script literal — see [`Raw.md`](Raw.md).
+
+**Galaxy / AVEVA System Platform points** are now ordinary raw tags on a
+`GalaxyMxGateway` driver in `/raw`, referenced into equipment like any other raw
+tag. There is no separate alias concept or `SystemPlatform`-kind namespace.
+
+### OPC UA address-space projection (v3 Batch 4 — dual namespace)
+
+> **v3 (Batch 4):** the server now exposes the address space under **two OPC UA
+> namespaces** instead of the old single `https://zb.com/otopcua/ns`:
+>
+> | Namespace URI | Subtree | NodeId `s=` scheme |
+> |---|---|---|
+> | `https://zb.com/otopcua/raw` | the `/raw` device tree (Folder → Driver → Device → TagGroup → Tag) | the node's **RawPath** (e.g. `Plant/Modbus/dev1/Speed`) |
+> | `https://zb.com/otopcua/uns` | the UNS tree (Area → Line → Equipment → signal) | the slash-joined **`Area/Line/Equipment/EffectiveName`** |
+
+Every device value has **exactly one source** — the raw tag's node in the Raw
+namespace. A `UnsTagReference` projects that raw tag into an equipment as a
+**UNS-namespace variable** whose NodeId leaf is the reference's **effective name**
+(the display-name override else the raw tag's `Name`). The UNS variable does not
+bind a driver of its own: it carries an **`Organizes` reference to its backing raw
+node** and mirrors it. A single driver publish for a RawPath **fans out** to the
+raw NodeId AND every referencing UNS NodeId with identical value / quality /
+source-timestamp — the two NodeIds never drift.
+
+- **Reads / subscriptions** work through either NodeId and return the same data.
+- **Writes** route through either NodeId to the same backing driver ref under the
+  **same `WriteOperate` gating** (a UNS write is neither more nor less privileged
+  than the raw write it fans from); a failed device write reverts both NodeIds via
+  the shared fan-out.
+- **HistoryRead** works through either NodeId and returns the same series — both
+  NodeIds register the **same historian tagname** (see [`Historian.md`](Historian.md)).
+- **Native alarms** materialize once at the raw tag (`ConditionId = RawPath`) and
+  fan via SDK notifiers to the raw device folder AND every referencing equipment
+  folder — see [`ScriptedAlarms.md`](ScriptedAlarms.md) / [`AlarmTracking.md`](AlarmTracking.md).
+
+Note the two distinct identity strings: the wire **NodeId** is the path
+`Area/Line/Equipment/EffectiveName`, while the **effective-name uniqueness key**
+above is `{EquipmentId}/{EffectiveName}` (the logical per-equipment collision
+space the guards enforce). They are related but not the same string.
 
 ### Virtual tags
 
 A virtual tag is bound to an equipment and driven by a **script** (no driver).
 Add and edit virtual tags on the equipment page's **Virtual Tags** tab; the
 data type is chosen from the standard OPC UA type list and the Monaco script
-editor is available inline.
+editor is available inline. Scripts may read the equipment's references
+relative-to-equipment with **`ctx.GetTag("{{equip}}/<RefName>")`** — the token
+resolves through the equipment's `UnsTagReference` rows (by effective name) to
+the backing raw tag's RawPath at deploy; an unresolved `<RefName>` is a deploy
+error and a live Monaco diagnostic (editor accepts ⇔ publish accepts). See
+[`ScriptEditor.md`](ScriptEditor.md).
 
 ### Galaxy tags
 

docs/drivers/Historian.Wonderware.md

@@ -13,7 +13,7 @@ OtOpcUa now consumes the **`ZB.MOM.WW.HistorianGateway`** sidecar through the Gi
 
 - **HistoryRead** → `GatewayHistorianDataSource` over the `ServerHistorian` appsettings section.
 - **Alarm history** → `GatewayAlarmHistorianWriter` (the gateway `SendEvent` path) behind the durable
-  `SqliteStoreAndForwardSink`; alarm-history `ReadEvents` needs the gateway running
+  `LocalDbStoreAndForwardSink`; alarm-history `ReadEvents` needs the gateway running
   `RuntimeDb:EventReadsEnabled=true`.
 - **Continuous historization** → a crash-safe FasterLog outbox + `ContinuousHistorizationRecorder`
   draining to the gateway's `WriteLiveValues` (`ContinuousHistorization` section); needs the gateway

docs/operations/2026-07-16-secrets-clustered-master-key.md

@@ -0,0 +1,177 @@
+# Secrets: Clustered Master-Key Posture (All Roles)
+
+## Purpose
+
+`ZB.MOM.WW.Secrets` resolves `${secret:...}` tokens in `appsettings.*.json` via a
+pre-host expander (`SecretReferenceExpander.ExpandConfigurationAsync`, wired in
+`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`) that runs at **every** OtOpcUa node
+boot, before the host is built. It reads rows from an envelope-encrypted SQLite
+store (`Secrets:SqlitePath`) unwrapped with a key-encryption key (KEK) sourced per
+`Secrets:MasterKey:Source`. The runtime `ISecretResolver` (`AddZbSecrets`) is also
+registered unconditionally, independent of node role.
+
+OtOpcUa is Akka.NET-clustered (`builder.Services.AddAkka("otopcua", (ab, sp) => { ... })`
+in `Program.cs`), with roles parsed by `RoleParser` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs`):
+`admin`, `driver`, `dev`. A node can carry any combination of these roles (e.g. a
+fused admin+driver node, or a driver-only node). This runbook covers what a
+production deployment needs so that secret resolution behaves identically on
+**every node regardless of role** — not just admin nodes. That matters here more
+than it might elsewhere: the pre-host expander and the runtime resolver both run
+unconditionally on driver-only nodes too, and driver-only nodes are the ones that
+will resolve Layer-B `DriverConfig` secret references (coming in Slice 2) at
+runtime, not just at boot. It does **not** change any code; it is an
+operations/deployment posture, delivered out-of-band from the committed config.
+
+## The two hard requirements
+
+For the pre-host expander (and the runtime resolver) to resolve the same
+plaintext secret on every node, no matter its role:
+
+1. **Identical KEK on every node.** All nodes — admin, driver, and any fused
+   combination — must unwrap the store with the exact same master key. A
+   per-node KEK (e.g. a per-box DPAPI-protected key) would make each node
+   decrypt every *other* node's ciphertext rows to garbage.
+2. **Identical store rows on every node.** All nodes must read the same SQLite
+   database (same file, or a replicated/shared copy with the same rows) — not
+   independently-seeded stores that happen to share a KEK.
+
+`ZB.MOM.WW.Secrets` ships a SQLite-only `ISecretStore` with a `NoOpSecretReplicator`
+— there is no built-in cross-node replication today. Meeting both requirements in
+production is a deployment concern, covered below.
+
+## Recommended interim posture (G-5)
+
+Until real replication exists (G-7, below), the recommended production posture is:
+
+- **`Secrets:MasterKey:Source = File`**, with `FilePath` pointing at a **read-only
+  key file that is identical on every node of every role** — a base64-encoded
+  32-byte key, generated out-of-band (e.g. `openssl rand -base64 32`), distributed
+  to each node's filesystem/secret-mount by the deployment tooling, and **never
+  committed** to the repo. Treat it with the same discipline as any other
+  production secret (restrictive file ACLs, no logging, rotated via a future
+  KEK-rotation runbook — not yet built).
+- **`Secrets:SqlitePath`** pointing at a **single shared or replicated volume**
+  that every node mounts (admin, driver, and dev/fused alike), so every node's
+  migrator opens and reads the same rows at boot.
+
+Writes to the store are rare and human-driven — an operator using the
+`/admin/secrets` UI (admin nodes only) or the `ZB.MOM.WW.Secrets` CLI — while
+reads happen on every node's boot and on the `ResolveCacheTtl` refresh cycle,
+regardless of role. The access pattern is read-mostly / effectively
+single-writer, which is what makes a shared SQLite volume viable as an interim
+posture (see caveat below).
+
+## How it's delivered (do NOT commit these values)
+
+The File-KEK + shared-store posture is supplied per-node at deployment time —
+**never** by editing the committed `appsettings.json` or the role-overlay files
+(`appsettings.admin.json`, `appsettings.driver.json`, `appsettings.admin-driver.json`).
+Those committed overlays are also consumed by local dev and the
+`TwoNodeClusterHarness` integration tests, so hardcoding a `Source=File` path
+into them would break every dev/test/CI boot. Two acceptable delivery
+mechanisms instead:
+
+**Option A — environment variable overrides** (Windows Service / NSSM env block,
+container `env_file`, etc.), applied identically on every node regardless of role:
+
+```
+# production deployment — do not commit to the dev appsettings
+Secrets__MasterKey__Source=File
+Secrets__MasterKey__FilePath=/run/secrets/otopcua-master.key
+Secrets__SqlitePath=/shared/secrets/otopcua-secrets.db
+```
+
+**Option B — a production-only config layer** that is *not* the committed dev
+base (e.g. an untracked `appsettings.Production.json` deployed alongside the
+binaries, or an orchestrator-injected config mount):
+
+```jsonc
+// production deployment — do not commit to the dev appsettings
+{
+  "Secrets": {
+    "MasterKey": {
+      "Source": "File",
+      "FilePath": "/run/secrets/otopcua-master.key"
+    },
+    "SqlitePath": "/shared/secrets/otopcua-secrets.db"
+  }
+}
+```
+
+Either way, the file/path referenced must exist and be identical on every node
+**before** that node boots — the pre-host expander runs unconditionally on every
+role and will throw (`SecretNotFoundException` / migration failure) if the store
+or key is missing.
+
+## Caveat: SQLite over a shared volume is not real replication
+
+SQLite's file-locking model does not tolerate concurrent multi-writer access well
+over network filesystems (SMB/NFS locking is unreliable, and even on a clustered
+block volume only one writer should be active at a time). The interim posture
+above is acceptable because:
+
+- Reads dominate (every node's boot + cache-refresh cycle, across every role).
+- Writes are rare, human-initiated, and effectively single-writer in practice
+  (an operator runs the CLI/UI against one admin node at a time).
+
+It is **not** a substitute for real replication, and it is not safe if multiple
+nodes attempt concurrent writes. Do not build automation that writes secrets
+from more than one node simultaneously.
+
+## Data Protection is independent — do not touch it here
+
+OtOpcUa's cookie/session protection already has its own clustered-key story:
+`AddDataProtection().PersistKeysToDbContext<OtOpcUaConfigDbContext>()`
+(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`),
+which shares the Data Protection key ring across every node via the existing
+ConfigDb. That mechanism is unrelated to `ZB.MOM.WW.Secrets`' envelope encryption
+(KEK + SQLite store) and must **not** be reconfigured as part of secrets-adoption
+work — doing so risks invalidating active sessions for an unrelated reason.
+
+It is, however, the model for where the *next* iteration of secret storage
+should go — see the G-7 hand-off below.
+
+## The G-7 hand-off
+
+The posture above is an interim, ops-only workaround. The long-term shape,
+tracked as **G-7** in `scadaproj/components/secrets/GAPS.md`, is a
+ConfigDb-backed `ISecretStore` that mirrors the pattern OtOpcUa already uses for
+the Data Protection key ring:
+
+```csharp
+services.AddDataProtection()
+    .PersistKeysToDbContext<OtOpcUaConfigDbContext>()
+    .SetApplicationName("OtOpcUa");
+```
+
+(`src/Server/ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs:73-75`).
+`OtOpcUaConfigDbContext` already gives every node — admin, driver, and dev/fused
+alike — a single MS SQL-backed source of truth for the Data Protection key ring;
+the secret store is the natural next tenant of that same shared database instead
+of a shared SQLite file. Building this requires new `ZB.MOM.WW.Secrets` library
+code (a ConfigDb-backed `ISecretStore` implementation) that does not exist yet,
+overlaps the G-7 tracking item, and is explicitly **deferred there** — it is not
+built as part of this cut. This runbook's shared-SQLite-volume posture is the
+bridge until G-7 lands.
+
+## Dev/test/default posture (unchanged)
+
+The committed default in `appsettings.json` is:
+
+```json
+"Secrets": {
+  "SqlitePath": "otopcua-secrets.db",
+  "MasterKey": { "Source": "Environment", "EnvVarName": "ZB_SECRETS_MASTER_KEY" },
+  "RunMigrationsOnStartup": true
+}
+```
+
+This is dev-safe: `Source=Environment` needs no filesystem key, and the SQLite
+path is relative to the working directory, so local dev, the role-overlay
+appsettings (`appsettings.admin.json`, `appsettings.driver.json`,
+`appsettings.admin-driver.json`), and the `TwoNodeClusterHarness` integration
+tests all boot cleanly with no external mount. The File-KEK + shared-volume
+posture in this runbook applies only to real clustered production deployments —
+it must never be baked into the committed dev/role-overlay base, because the
+expander runs unconditionally at every node boot (any role) and would break
+dev/CI if pointed at a nonexistent `/shared` mount.

docs/operations/2026-07-20-localdb-pair-replication.md

@@ -0,0 +1,191 @@
+# LocalDb pair replication — operations runbook
+
+> **Scope.** Every driver-role OtOpcUa node keeps a consolidated
+> [`ZB.MOM.WW.LocalDb`](../../CLAUDE.md) SQLite database. It holds two things: the
+> **deployed-configuration artifact** (chunked, Phase 1) so a node can **boot from cache when
+> central SQL Server is unreachable**, and the **alarm store-and-forward buffer** (Phase 2) so a
+> node that dies holding undelivered alarm history does not take it to the grave. Both can
+> *optionally* replicate to the node's redundant pair peer over the library's gRPC sync.
+> **Replication is default-OFF and fail-closed.** This runbook covers enabling it, the operational
+> rules that keep a pair converging, and the DB-inspection safety rules.
+
+## What replicates, and what it buys you
+
+- **Replicated tables:** `deployment_artifacts` (the artifact, split into base64 chunks),
+  `deployment_pointer` (one current-deployment pointer per cluster), and `alarm_sf_events` (the
+  alarm store-and-forward buffer). Registered — in this order, after the DDL — by
+  `LocalDbSetup.OnReady`.
+- **The payoff:** in a redundant driver pair, *either* node can be the one that applied the latest
+  deploy and wrote the cache. With replication on, the peer holds a byte-identical copy. So a node
+  that restarts into a central-SQL outage can boot the current config **even if it never applied
+  that deploy itself** — its peer did, and the row replicated.
+- **Convergence model:** last-writer-wins per primary key, HLC-stamped. Both nodes also store
+  locally after every apply, so the pair converges to identical content regardless of which node
+  deployed. Retention (newest-2 deployments per cluster) prunes on one node and the deletes
+  replicate as tombstones.
+
+## Alarm store-and-forward — what replication changes
+
+Phase 2 moved the alarm buffer out of its own `alarm-historian.db` and into this database. Three
+consequences an operator needs to know:
+
+- **Only the Primary drains.** The buffer replicates, so the Secondary holds a full copy of the
+  Primary's queue. Its drain worker is gated on the same Primary decision the inbound-write and
+  native-ack gates use, and reports `DrainState = NotPrimary`. **A rising queue depth on a
+  Secondary is correct.** A rising queue depth on *both* nodes is not — see below.
+- **Delivery is at-least-once across a failover, by design.** Rows the old Primary delivered
+  moments before it died may not have had their deletes replicated yet, so the new Primary will
+  re-send them. This is accepted, not a defect: a duplicate alarm-history row is recoverable, a
+  missing one is not. There is deliberately no dedup layer.
+- **The buffer is bounded.** `AlarmHistorian:Capacity` (1,000,000 by default) still evicts the
+  oldest undelivered rows when the historian has been unreachable long enough. Watch
+  `EvictedCount` — non-zero means accepted alarm events were dropped before reaching the historian.
+
+### If BOTH nodes report `NotPrimary`
+
+Nobody is draining, and the queue is filling toward the capacity ceiling on both. The gate
+default-DENIES while the redundancy role is unknown *and* a driver peer exists, so this is what a
+redundancy snapshot that never names either node looks like — the identity-mismatch shape
+`DriverHostActor` also warns about once ("redundancy snapshot omitted this node"). Check node
+identity (`Cluster:PublicHostname` / `Cluster:Port` skew) before suspecting the sink.
+
+### One-time migration from `alarm-historian.db`
+
+On first boot after the upgrade, `AlarmSfLegacyMigrator` copies any pre-existing
+`alarm-historian.db` into `alarm_sf_events` and renames the file to `alarm-historian.db.migrated`.
+
+- **Both nodes of a pair migrate independently**, and their files overlap — row ids are derived
+  from the event payload, so the same event on both nodes converges to one row rather than
+  duplicating. The new Primary drains the union.
+- **The rename happens only after the copy commits.** A failure fails host startup with the legacy
+  file untouched; the `.migrated` sidecar is what makes a later boot a no-op.
+- **A file whose shape is unrecognised is left alone** — not renamed, not copied — so it stays
+  available for inspection.
+- The `AlarmHistorian:DatabasePath` key is **removed**. It is still *read* by the migrator, to
+  find the legacy file; it no longer configures anything. Delete it from appsettings once the
+  `.migrated` sidecar exists on every node.
+
+## Enabling replication on a pair
+
+Replication has two knobs, both under `LocalDb`:
+
+| Key | Role | Notes |
+|---|---|---|
+| `LocalDb:SyncListenPort` | binds the dedicated **h2c** sync listener | `0` (default) = no listener, replication off. Set the **same** non-zero port on **both** nodes of the pair. |
+| `LocalDb:Replication:PeerAddress` | the address this node **dials** | Set on **one** node (the initiator); leave empty on the other (passive). The stream is bidirectional — one dial carries both directions. |
+| `LocalDb:Replication:ApiKey` | bearer token for the sync stream | **Must be byte-identical on both nodes.** See fail-closed rule below. Supply via `${secret:...}` / env — never a cleartext literal in production config. |
+| `LocalDb:Replication:MaxBatchSize` | rows per replication batch | `16` for the artifact cache. Batching is **row-count-only** against gRPC's 4 MB message cap; artifact chunk rows are ≈171 KB, so 16 × 171 KB ≈ 2.7 MB stays under the cap. Raising it risks tripping the cap. |
+
+Example (site-a-1 initiator, site-a-2 passive):
+
+```
+site-a-1:  LocalDb__SyncListenPort=9001
+           LocalDb__Replication__PeerAddress=http://site-a-2:9001
+           LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key}
+           LocalDb__Replication__MaxBatchSize=16
+site-a-2:  LocalDb__SyncListenPort=9001
+           LocalDb__Replication__ApiKey=${secret:site-a-localdb-sync-key}   # no PeerAddress
+           LocalDb__Replication__MaxBatchSize=16
+```
+
+> **Kestrel note.** Setting `SyncListenPort` makes the Host add an explicit `Listen*` for the h2c
+> sync listener. An explicit `Listen*` makes Kestrel **ignore `ASPNETCORE_URLS` entirely**, so the
+> Host also re-binds the primary HTTP port in the same block. If you change the primary port,
+> verify both `:<httpPort>` and `:<syncPort>` appear in the startup "Now listening on" lines.
+
+### The ApiKey rule (fail-closed)
+
+The replication library's passive endpoint verifies **no** authentication — the host
+`LocalDbSyncAuthInterceptor` is the only gate, and it is **fail-closed**:
+
+- **No key configured ⇒ every sync call is refused** (`PermissionDenied`). "No key" is never
+  "no auth required".
+- **A key mismatch ⇒ the pair silently stops converging.** The initiator's stream is rejected at
+  the peer; nothing errors loudly. A typo in one node's key looks exactly like "replication is
+  broken". If a pair is not converging, **check the keys match first.**
+
+## Operational rules
+
+- **Stop / start the pair together where you can.** Each node keeps working (and caching locally)
+  while its peer is down; the outage is not a data-loss event — the surviving node accumulates
+  writes and the peer catches up on rejoin. But a long-lived solo node drifts further from its
+  peer, so avoid leaving a pair split for extended periods.
+- **Tombstone-retention resurrection window.** Retention prunes to the newest 2 deployments and
+  replicates the prune as tombstones. Tombstones are themselves retained only for a bounded window.
+  If a node is offline **longer than the tombstone-retention window**, a delete that happened during
+  its outage may no longer be expressible as a tombstone on rejoin — a pruned deployment could
+  briefly reappear until the next deploy re-prunes it. Keep pair outages well inside that window.
+- **A rebuilt node is back-filled by its peer.** If a node loses its LocalDb file — a wiped volume,
+  a re-imaged host, a fresh container — it rejoins with an empty database and its peer snapshots the
+  cached configuration back to it, with no new deploy and no central SQL. Two things are worth
+  knowing: this needs **`ZB.MOM.WW.LocalDb` ≥ 0.1.3**, and it heals the *cache*, so the node regains
+  boot-from-cache for future outages. (On `0.1.1` a converged pair has pruned every oplog row on ack
+  and the wiped node was never snapshotted — it stayed empty until the next deploy. On `0.1.2` it is
+  back-filled but its OWN writes are silently dropped until its restarted seq counter climbs past the
+  peer's stale watermark, so the pair looks converged right up until the moment only the rebuilt node
+  applies a deploy.) A node
+  wiped **during** a central outage is still repopulated by its peer under 0.1.2; a node with **no**
+  peer replication configured self-heals only from central on its next successful apply.
+
+- **What boot-from-cache does NOT cover.** The cache is a *fallback for central-SQL outages at
+  boot*, not a replacement for central. A **new deployment still requires central SQL** — the cache
+  is only read when the central fetch fails at startup. When central returns, the node resumes
+  fresh-config behavior. Boot-from-cache logs a running-from-cache signal; treat a node that stays
+  on it as a central-connectivity incident, not steady state.
+
+## Inspecting the LocalDb file — safety rules
+
+The database runs in **WAL mode**. These rules exist because violating them corrupted a live DB in
+the 2026-07-20 ScadaBridge incident:
+
+- **Never run `sqlite3` on the live file** (host-side, against a bind-mounted or container path).
+  Opening a live WAL DB from a second process across virtiofs poisons the WAL. **Always copy the
+  triplet first** and query the copy:
+
+  ```bash
+  docker cp <node>:/app/data/otopcua-localdb.db      /tmp/localdb.db
+  docker cp <node>:/app/data/otopcua-localdb.db-wal  /tmp/localdb.db-wal
+  docker cp <node>:/app/data/otopcua-localdb.db-shm  /tmp/localdb.db-shm
+  sqlite3 /tmp/localdb.db 'SELECT cluster_id, deployment_id FROM deployment_pointer;'
+  ```
+
+- **Metrics** come from the container, not a host curl (`aspnet:10.0` has no `curl`):
+
+  ```bash
+  docker run --rm --network container:<node> curlimages/curl:latest -s localhost:<httpPort>/metrics | grep localdb_
+  ```
+
+- **If an anomaly appears within seconds of your own measurement, suspect the measurement.**
+  Restart both nodes and re-observe untouched before blaming the library.
+
+## Useful queries (on a copied triplet)
+
+```sql
+-- What each node thinks the current deployment is
+SELECT cluster_id, deployment_id, revision_hash, applied_at_utc FROM deployment_pointer;
+
+-- Convergence check: the pointer's origin stamp should match on both nodes
+SELECT pk_json, hlc, node_id, is_tombstone
+FROM __localdb_row_version WHERE table_name = 'deployment_pointer' ORDER BY pk_json;
+
+-- Replication backlog (should drain to 0 when a pair is caught up)
+SELECT COUNT(*) FROM __localdb_oplog;
+
+-- Undelivered alarm history, and how much of it has given up
+SELECT dead_lettered, COUNT(*) FROM alarm_sf_events GROUP BY dead_lettered;
+
+-- Why the dead-lettered rows died
+SELECT alarm_id, attempt_count, last_attempt_utc, last_error
+FROM alarm_sf_events WHERE dead_lettered = 1 ORDER BY last_attempt_utc DESC LIMIT 20;
+
+-- Convergence check for the buffer: identical dumps mean the peer holds the ORIGIN-stamped rows
+SELECT pk_json, hlc, node_id, is_tombstone
+FROM __localdb_row_version WHERE table_name = 'alarm_sf_events' ORDER BY pk_json;
+```
+
+## See also
+
+- [`docs/Redundancy.md`](../Redundancy.md) — the pair-local config cache section.
+- [`docs/Configuration.md`](../Configuration.md) — the `LocalDb` appsettings section.
+- [`docs/AlarmHistorian.md`](../AlarmHistorian.md) — the alarm sink, its knobs, and the drain.
+- The two-node convergence harness: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/`.

docs/plans/2026-07-16-v3-batch3-PR.md

@@ -0,0 +1,89 @@
+# v3 Batch 3 — UNS reference-only Equipment + `{{equip}}/<RefName>` reference-relative resolution
+
+Implements `docs/plans/2026-07-15-v3-batch3-uns-rework-plan.md` (WP1–WP4), executed via the
+`/v3-batch 3` coordinator: contracts-first fan-out, worktree-isolated Opus agents per wave, a
+code-reviewer pass + green build/test after each wave, then the non-negotiable 5-item live gate on
+docker-dev.
+
+## What landed
+
+- **Contracts** — `IEffectiveNameGuard` (authoring-time effective-name uniqueness seam; WP2 implements,
+  WP1 consumes).
+- **Wave A** (3 parallel agents):
+  - **WP1** — equipment **Tags** tab → `UnsTagReference` list (effective name / RawPath / inherited
+    DataType+AccessLevel / display-name override); new `AddReferenceModal` reusing `RawTree` in opt-in
+    `PickerMode` (multi-select tag leaves + device/tag-group "Select all tags below"), **cluster-scoped**
+    (structural + server-enforced `tag.cluster == equipment.cluster`); `UnsTreeService` reference CRUD;
+    consumes `IEffectiveNameGuard` in all colliding mutations; `ImportEquipmentModal`/`EquipmentInput`
+    dropped `DriverInstanceId`; deleted the retired equipment `TagModal`.
+  - **WP2** — `EffectiveNameGuard` (injectable, ordinal) + DI registration; verified the deploy-gate
+    `UnsEffectiveNameCollision` rule (already computes ref effective name as override-else-current-raw-name,
+    so it catches rename-induced collisions; ordinal; names both sources + equipment).
+  - **WP3** — `RawTreeService.BuildRenameWarningsAsync`: refined historized warning (only tags **without**
+    a `historianTagname` override) + UNS-referenced + **new** substring scan of every `Script.SourceCode`
+    for affected tags' OLD RawPaths (recomputed pre-save via `RawPathResolver`, ordinal).
+- **Wave B** (1 integration agent):
+  - **WP4** — `{{equip}}/<RefName>` reference-relative resolution. `EquipmentScriptPaths.DeriveEquipmentBase`
+    deleted; slash joint replaces the dot joint; new shared `EquipmentReferenceMap` (`equipmentId →
+    effectiveName → RawPath`) built identically at both compose seams (`AddressSpaceComposer` +
+    `DeploymentArtifact`, byte-parity). Unresolved-ref deploy error (`EquipReferenceUnresolved`) covering VT
+    scripts, SA predicates, and SA message-template tokens; authoring guard (`ValidateEquipTokenAsync`,
+    the Wave-A M1 fix); Monaco `{{equip}}/` completion of reference effective names + unresolved diagnostic
+    (`OTSCRIPT_EQUIPREF`), `EquipmentId` threaded request→model→JS.
+
+**Address space stays dark** (values light up in Batch 4). Full solution builds 0 errors; Commons **309**,
+Configuration **121**, AdminUI **659**, OpcUaServer **335/4-skip**, Runtime **361/42-skip** (the skips are
+the Batch-1 dark-address-space tests Batch 4 un-skips).
+
+## Reviewer findings fixed in-branch
+
+- **Wave A** — no HIGH. Verified: the register-AND-consume guard seam (prod DI injects the real guard; the
+  NoOp fallback only wins under `new` in tests; guard invoked in all 6 colliding mutations); cluster scoping
+  structural + server-enforced; Razor default `RawTree` byte-unchanged; WP3 pre-save RawPath recomputation.
+  L1 (misleading concurrency comment) fixed.
+- **Wave B** — no HIGH; byte-parity + production wiring confirmed. **M1** fixed: the editor⇔authoring⇔deploy
+  invariant now holds on the **ScriptedAlarm** surface too (SA predicate + message-template `{{equip}}` refs
+  validated at authoring, matching the deploy gate). **L3** fixed: parity test hardened (unresolved-ref-intact
+  + folder/tag-group RawPath ancestry). L1 already closed at the write path (empty override normalized→null).
+- **Deferred (documented follow-ups):** absolute-path Monaco completion still projects the raw leaf name, not
+  the RawPath (out of `{{equip}}` scope; `{{equip}}/` completion works) — rework `ScriptTagCatalog` to RawPaths
+  in Batch 4; a broken raw-topology-chain reference is deploy-accepted but compose-dropped (low reachability);
+  message-template `{{equip}}/X` is gate-validated but rendered/substituted only in Batch 4.
+
+## Live `/run` gate (docker-dev `:9200`, both central nodes rebuilt on Batch-3 code)
+
+Substrate: an Area→Line→Equipment seeded in MAIN, plus a SITE-A raw tag so cross-cluster exclusion is
+demonstrable. (Hand-seeded equipment surfaced the Batch-1 `EquipmentIdNotDerived` invariant — the canonical
+`EQ-<hash>` id was applied before the gate.)
+
+1. **Reference picker + list** — "+ Add reference" header *"The tree is scoped to the equipment's cluster"*;
+   only the **Main cluster** root shows (SITE-A/B absent — the seeded `SiteAOnly` tag unreachable); lazy
+   expansion; a device **"Select all tags below"** pulled tags across a collapsed group → **4 references in
+   one commit**. Rows show effective name / RawPath (incl. deep `opcua1/plc/OpcPlc/Telemetry/Basic/…`
+   ancestry) / DataType / Access / override. Set override "MainPressure" → effective name updated
+   `HR200 → MainPressure`, RawPath unchanged.
+2. **Collision** — *authoring:* setting a reference override to an existing VirtualTag's name → red banner
+   **"effective name 'GateVt' already used by VirtualTag 'VT-gatevt' in equipment '…'"** (not persisted).
+   *rename-induced:* renamed a raw tag so two references collide (allowed at rename) → deploy **422**
+   `[UnsEffectiveNameCollision] 2 UNS signals collide on effective name 'ImportedTag' … reference '…',
+   reference '…'`.
+3. **`{{equip}}` (editor ⇔ deploy)** — resolving `{{equip}}/MainPressure` deploys **202**; misspelled
+   `{{equip}}/MainPresure` deploys **422** `[EquipReferenceUnresolved] … has no reference named 'MainPresure'`.
+   Monaco: red squiggle + hover *"'{{equip}}/MainPresure' does not resolve — … no reference named
+   'MainPresure' … (OTSCRIPT_EQUIPREF)"*; typing `{{equip}}/` completes the four reference **effective**
+   names (AlternatingBoolean, ImportedTag, **MainPressure**, RandomSignedInt32).
+4. **Rename warning** — renaming a driver whose beneath-it tag is historized-without-override + UNS-referenced
+   + named in a script literal → **"Renamed — with warnings"** listing all three (historized fork,
+   UNS-referenced by 'gateequip', "2 scripts ('cval','gatevt') reference tags … by their raw paths"); renaming
+   an unrelated driver → **no warning dialog**. (Note: direct *tag* rename flows through `UpdateTagAsync`
+   which has no warnings surface — warnings fire on container renames affecting descendants; documented
+   follow-up if per-tag-rename warnings are wanted.)
+5. **Import without driver column** — the Import equipment CSV modal columns are `Name, MachineCode, UnsLineId`
+   (+ optional `ZTag, SAPID, Manufacturer, Model`); **no `DriverInstanceId`**.
+
+## Docs
+
+`docs/Uns.md` Tags section rewritten to the reference-only model; `docs/ScriptEditor.md` updated (WP4) to the
+slash/reference `{{equip}}` semantics; `CLAUDE.md` gains a v3 Batch 3 paragraph.
+
+https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox

docs/plans/2026-07-16-v3-batch4-PR.md

@@ -0,0 +1,116 @@
+# v3 Batch 4 — dual-namespace address space + raw-only runtime binding (= v3.0)
+
+Implements `docs/plans/2026-07-15-v3-batch4-address-space-plan.md` (B4-WP1–WP5), executed via the
+`/v3-batch 4` coordinator: contracts-first fan-out, worktree-isolated Opus agents per wave, a
+code-reviewer pass + green build/test after each wave, then the non-negotiable **7-leg v3.0 live gate**
+on docker-dev + Client.CLI. **This batch lights up the address space** — v3.0.
+
+## What landed
+
+- **Contracts** (`a1a56e22`) — two namespace URIs (`…/otopcua/raw`, `…/otopcua/uns`, replacing the single
+  `…/ns`), `V3NodeIds` (raw = `s=<RawPath>`, UNS = `s=<Area>/<Line>/<Equipment>/<EffectiveName>`), and the
+  `AddressSpaceRealm` (`Raw | Uns`) discriminator carried at every sink seam.
+- **Wave A** (2 agents ∥):
+  - **WP1** — composer/planner un-darkened: emit the **Raw** subtree (Folder→Driver→Device→TagGroup→Tag,
+    tags as `(realm=Raw, s=RawPath)`) + the **UNS** subtree (each `UnsTagReference` a Variable
+    `(realm=Uns, s=Area/Line/Equipment/EffectiveName)` carrying its backing RawPath + an `Organizes`
+    UNS→Raw ref); native-alarm plans at the **raw** tag (ConditionId=RawPath) carrying referencing-equipment
+    paths; per-realm planner diff (raw rename = remove+add Raw + re-point UNS); reactivated the Batch-1-skipped
+    `EquipmentNamespaceMaterializationTests`.
+  - **WP2** — node manager registers both namespaces; `AddressSpaceRealm` on every node-naming sink method
+    (maps keyed by ns-qualified NodeId so a Raw + UNS node sharing a bare `s=` id stay distinct; historized
+    UNS node registers the **same** tagname, mux single by RawPath); everything **forwarded through
+    `DeferredAddressSpaceSink`** + reflection guard extended (the forwarding trap). **M1 fix**: new
+    `AddReference` sink seam wiring the `Organizes` UNS→Raw edge bidirectionally (idempotent, missing-endpoint
+    no-op, forwarded + reflection-covered).
+- **Wave B** (1 agent — the delicate one):
+  - **WP3** — applier materializes both realms + wires the Organizes edge; `DriverHostActor` dual-NodeId
+    registration + single-source fan-out (drift-free — one publish → raw NodeId AND every referencing UNS
+    NodeId, identical value/quality/timestamp); write routing + failed-write revert through both NodeIds;
+    `EquipmentNodeIds` retired; transitional realm defaults removed from the sink surface. **Review fixes**:
+    **H1** realm-qualified `(realm, bareId)` write-routing key (a raw RawPath that coincidentally equals a UNS
+    path can no longer misroute a write); **M1** coherent-dormant discovery-injection guard; **M2** dual-node
+    self-correction tests; **L1** node-manager routing defaults removed (~180 call sites); byte-parity
+    round-trip test.
+- **Wave C** (2 agents ∥):
+  - **WP4** — multi-notifier native alarms: one condition at the raw tag, `AddNotifier`-fanned to the raw
+    device folder + every referencing equipment folder, **one `ReportEvent`** (exactly one Server-object copy —
+    proven by an over-the-wire event-delivery test), teardown symmetry (`RemoveNotifier(bidirectional:true)` on
+    rebuild/condition-removal/subtree-removal + reconcile), ack/confirm/shelve still on ConditionId=RawPath;
+    `AlarmTransitionEvent` + `/alerts` gain the referencing-equipment list. **Review fixes**: M1 event-delivery
+    test, M2 resolution-failure meter + composer↔applier path-agreement test, M3 reconcile, L1/L3.
+  - **WP5** — over-the-wire dual-namespace integration tests (`DualNamespaceAddressSpaceTests`) + 2-node
+    harness materialization round-trip; docs (`CLAUDE.md`, `docs/Uns.md`, `docs/ScriptEditor.md`,
+    `docs/Historian.md`, `docs/ScriptedAlarms.md`, `docs/AlarmTracking.md`).
+- **Coordinator seam-fix** — threaded `ReferencingEquipmentPaths` into `DriverHostActor.ForwardNativeAlarm`'s
+  `AlarmTransitionEvent` so `/alerts` chips populate in production (WP3-owned file WP4 couldn't edit).
+- **Live-gate bug fix** — the gate caught a genuine prod-inert defect: **Equipment VirtualTags never resolved
+  live** (a redeploy resets the VT's UNS node to `BadWaitingForInitialData`, but a surviving unchanged-plan
+  `VirtualTagActor` keeps its value-dedup state and suppresses the re-publish → the reset node stays Bad
+  forever with a static dependency). Fix: `ReassertValue` on apply re-emits the last value after the node
+  reset. Regression test fail-before/pass-after; live-verified Good. (Reviewer follow-ups: reassert skips
+  historian re-record (M1); scripted-alarm redeploy recovery (M2); comment (LOW-3).)
+
+## Reviewer verdict per wave
+
+No HIGH survived any wave. Wave A: forwarding trap verified solid (DispatchProxy guard catches unforwarded
+methods), collision-free ns-qualified keying, planner raw-rename-repoints-UNS holds; M1 (Organizes seam) fixed.
+Wave B: **H1** write-route realm collision fixed with a regression test; M1/M2/L1 fixed. Wave C: single-ReportEvent
++ teardown symmetry + forwarding trap all confirmed; M1/M2/M3/L1/L3 fixed. VT-fix review: ordering confirmed real,
+dedup/02-S13 semantics intact; M1/M2/LOW-3 closed.
+
+## v3.0 live gate (docker-dev `:9200` / `opc.tcp://:4840`+`:4841`, both centrals rebuilt on Batch-4 code)
+
+1. **Browse both namespaces** — `ns=2` Raw tree (`pymodbus/plc/HR200`, `s=<RawPath>`) + `ns=3` UNS tree
+   (`gatearea/gateline/gateequip/MainPressure`, effective names); both materialize `failed=0`.
+2. **Single-source fan-out** — `MainPressure` (UNS) = `HR200` (Raw) = **1234**, identical value AND timestamp;
+   Calculation tag `Cval` computes (=1235); the `{{equip}}/MainPressure` VirtualTag reads Good (post-fix).
+3. **Writes** — write **4242** via the UNS NodeId → read-back **4242** via the raw NodeId; write **777** via the
+   raw NodeId; `opc-readonly` rejected **0x801F0000** on the UNS NodeId (role gating symmetric). Failed-write
+   dual-node revert is unit-covered (`exception_injector` fixture not provisioned on this rig).
+4. **History** — HistoryRead via the raw NodeId and the UNS NodeId both return **GoodNoData** under the same
+   historian tagname (Null source — no gateway on docker-dev); provisioning tally `dispatched=1 … failed=0`.
+5. **Alarms (multi-notifier)** — the native alarm materialized as a Part 9 **AlarmCondition** at the raw tag
+   (ConditionId = RawPath); the **equipment folder** accepts an alarm subscription + ConditionRefresh, proving
+   it is a live event-notifier wired to the raw condition; the Server object likewise. The transition delivery
+   (exactly-one Server copy + delivery at both notifiers + ack) is covered by the over-the-wire
+   `NativeAlarmMultiNotifierEventDeliveryTests` — docker-dev has no `IAlarmSource` driver (only Galaxy is one)
+   so a live transition can't be tripped on the rig.
+6. **Rename cascade** — renamed `HR200`→`HR200X` + deployed: old raw NodeId gone, new present; the UNS
+   reference `MainPressure` follows automatically; the `{{equip}}` VirtualTag keeps working and tracks the
+   renamed tag live (3610→3611 Good); the **absolute** RawPath script literal (`Cval`) now fails **Bad
+   0x80000000**. (Caveat: a raw-tag rename needs a node recreate/restart to hot-rebind the renamed tag's live
+   driver value — the pre-existing deploy-THEN-recreate pattern for structural edits.)
+7. **Redundancy** — ServiceLevel split **central-1 = 250 / central-2 = 240** (both Good) — the Primary/Secondary
+   differentiation is intact with the second namespace present; single-emit-per-condition is unit-covered (the
+   alarm-emit gate is Primary-only).
+
+## Tests
+
+Full solution builds **0 errors**. Commons 306, Configuration 121, OpcUaServer 375/4-skip (+ OpcUaServer.IntegrationTests
+dual-namespace + event-delivery), Runtime 377 (+ VT reassert regression), AdminUI 659, Host.IntegrationTests green
+(the pre-existing Batch-2 `Calculation`-probe stale test fixed here). The remaining Runtime skips are legacy
+`EquipmentTags`-model dark tests + the dormant discovery-injection scenarios (accurate skip reasons).
+
+## Documented follow-ups (non-blocking)
+
+- **Scripted-alarm redeploy recovery (CONFIRMED pre-existing bug, deferred — deserves its own careful fix). Filed as issue #487.**
+  The scripted-alarm Part 9 condition node has the same redeploy-reset race the VT `ReassertValue` fix closed:
+  `RebuildAddressSpace` clears `_alarmConditions` and `MaterialiseScriptedAlarms` recreates each condition
+  fresh/normal, but `ScriptedAlarmEngine.LoadAsync` reloads from the persisted state and yields
+  `EmissionKind.None` for a still-active condition (dropped by `OnEngineEmission`), so an **active scripted
+  alarm with static dependencies under-reports as normal after a full-rebuild deploy until its next real
+  transition** (the known "snapshot-is-one-shot / restart-to-re-deliver" gotcha — NOT a Batch-4 regression).
+  Deferred because the fix is a Core-engine emission-contract change carrying **double-alarm-history risk**
+  (a naive re-emit would append a duplicate AVEVA/historian row per deploy — the alarm analogue of the VT M1
+  historian issue). Proposed shape: a host-driven, node-only `AlarmStateUpdate` re-assert in `OnAlarmsLoaded`
+  (same safe post-materialise ordering as the VT fix), reusing `_engine.GetState(alarmId)` + `LoadedAlarmIds`,
+  **never touching the `alerts` topic**.
+- Raw-tag rename hot-rebind of the renamed tag's live value without a recreate (today: deploy-THEN-recreate). Filed as issue #489.
+- Absolute-path Monaco completion → RawPath (from Batch 3; the `{{equip}}/` completion works). Filed as issue #490.
+- Cross-repo (post-merge): ScadaBridge Data-Connection-Layer NodeId/namespace cutover (raw `s=<RawPath>` /
+  UNS `s=<Area>/<Line>/<Equipment>/<EffectiveName>`, retired `EquipmentNodeIds`) — tracked in the
+  ScadaBridge repo as its issue #14 (its #20 landed the `nsu=` binding + native-alarm routing). The umbrella
+  index `../scadaproj/CLAUDE.md` OtOpcUa entry is **DONE** (scadaproj `ede5275`).
+
+https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox

docs/plans/2026-07-17-alarm-condition-quality-477-design.md

@@ -0,0 +1,187 @@
+# Alarm condition Quality (issue #477) — design
+
+**Status:** implemented (L1+L2) · **Date:** 2026-07-17 · **Issue:** #477 (follow-up chain #473 → #475 → #477)
+**Scope decision:** Layer 1 + Layer 2, Bad-direct, annotate-only. Layer 3 (scripted worst-of-input) deferred → **#478**.
+
+## Problem
+
+`AlarmConditionState.Quality` is never assigned anywhere in `src/` — neither by
+`OtOpcUaNodeManager.MaterialiseAlarmCondition` nor by the `WriteAlarmCondition` transition path.
+Because `StatusCodes.Good == 0x00000000`, `default(StatusCode)` **is** `Good`, so the field is
+*accidentally valid* — clients parse it, but it reports **`Good` unconditionally regardless of the
+backing tag's real quality**.
+
+This is a wrong-*value* bug, not the null-value bug class of #473/#475. Part 9 defines
+`ConditionType.Quality` as "the quality of the Condition's source data". OT impact: when a native
+alarm's device goes offline (comms lost) the condition still reports `Quality = Good`, so an operator
+(or an HMI bucketing on `IsGood`) cannot distinguish *"genuinely not active"* from *"we have lost
+contact and do not know"*.
+
+## Why it isn't a 2-line default (confirmed by code)
+
+1. **Alarm-bearing raw tags have no value variable.** `AddressSpaceApplier` materialises a raw tag as
+   *either* a condition node (`tag.Alarm is not null`) *or* a value variable (`else`) — never both,
+   since they'd share the same `s=<RawPath>` NodeId. So `WriteValue` (the only path carrying
+   `OpcUaQuality`) is never invoked for an alarm node. Quality has nowhere to land today.
+2. **The alarm channel is quality-blind.** `AlarmEventArgs` (driver → host) and `AlarmConditionSnapshot`
+   (host → SDK sink) both carry no quality field.
+3. **On comms-loss the alarm feed goes silent.** `DriverInstanceActor` on `DisconnectObserved` detaches
+   the alarm subscription and re-enters `Reconnecting` — no transition event ever arrives to carry Bad.
+   So the "device offline" signal must come from **driver connectivity**, independently of alarm
+   transitions.
+
+## Decisions (the issue's open questions)
+
+| # | Question | Decision | Rationale |
+|---|----------|----------|-----------|
+| 1 | Does an alarm tag get quality today? | No | Confirmed above — new plumbing required. |
+| 2 | Direct status code vs. policy map | **Direct Bad** on comms-loss; Good on reconnect | Matches how a value variable would read; unambiguous for `IsGood` bucketing. |
+| 3 | Does Bad also suppress transitions / touch Retain? | **No — annotate only** | A comms-lost *active* condition must stay active + retained. Silently clearing an active alarm on comms-loss is the unsafe direction. Quality is a pure annotation; the Active/Ack/Retain state machine is untouched. |
+| 4 | Scripted alarms: worst-of-inputs quality? | **Deferred (Layer 3)** | Scripted conditions stay `Good`. Filed as a follow-up issue. |
+
+## Architecture — reuse the existing publish path, add no sink method
+
+The key move: **do not add a new `IOpcUaAddressSpaceSink` method.** A new sink-interface surface would
+have to be forwarded through `DeferredAddressSpaceSink` or it is inert on driver hosts (the F10b
+prod-inertness trap). Instead the `NativeAlarmProjector` becomes the single owner of per-condition
+state *and* quality, and a connectivity change re-projects the *last* snapshot with a swapped quality
+through the **existing** `AlarmStateUpdate → OpcUaPublishActor → WriteAlarmCondition` path.
+
+### Layer 1 — make Quality a real, plumbed field
+
+- `AlarmConditionSnapshot` (Commons) gains `OpcUaQuality Quality` (last positional param, default
+  `OpcUaQuality.Good` so scripted callers and existing tests keep compiling; Commons already knows
+  `OpcUaQuality` via `IOpcUaAddressSpaceSink`).
+- `MaterialiseAlarmCondition` sets `alarm.Quality.Value` at build time:
+  **native → `BadWaitingForInitialData`** (honest until connectivity confirms Good, matching the
+  value-variable "waiting for initial data" convention), **scripted → `Good`** (script-computed, always
+  live in this scope).
+- `WriteAlarmCondition` projects `StatusFromQuality(state.Quality)` onto `condition.Quality.Value`
+  (+ `SourceTimestamp`).
+- The delta-gate (`AlarmConditionDelta` / `ReadConditionDelta` / `ToConditionDelta`) gains a `Quality`
+  member, so a Good→Bad bucket change is a genuine delta and **fires a Part 9 condition event**.
+
+### Layer 2 — drive native quality from driver connectivity
+
+- `DriverInstanceActor`: new `public sealed record ConnectivityChanged(string DriverInstanceId, bool Connected)`.
+  `Context.Parent.Tell` it on `Become(Connected)` entry (`true`) and on the transitions into
+  `Reconnecting` (`DisconnectObserved` / `ForceReconnect`) (`false`). Fire-and-forget, mirrors
+  `DeltaApplied`.
+- `NativeAlarmProjector`: per-node state becomes `(bool Active, bool Acked, OpcUaQuality Quality)`.
+  `Project(transition)` preserves the current quality; new `ProjectQuality(nodeId, quality)` preserves
+  Active/Acked and swaps only the quality, returning a full snapshot.
+- `DriverHostActor`: `Receive<ConnectivityChanged>` iterates `_alarmNodeIdByDriverRef` for that driver
+  instance and Tells one `AlarmStateUpdate` per condition with the re-projected snapshot
+  (`connected ? Good : Bad`). **Ungated** — both redundancy nodes track their own driver's comms, matching
+  the existing "condition write stays ungated (Secondary keeps its address space warm)" rule.
+  **No `/alerts` row** for a quality-only change — driver health already has its own status/alerts surface
+  via `IDriverHealthPublisher`; a row here would be alarm-fatigue.
+
+Scripted alarms are unaffected: they are not driver instances, receive no `ConnectivityChanged`, and
+their snapshot quality stays `Good`.
+
+## Files
+
+**Layer 1**
+- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/AlarmConditionSnapshot.cs`
+- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs` (`MaterialiseAlarmCondition`,
+  `WriteAlarmCondition`, `AlarmConditionDelta`/`ReadConditionDelta`/`ToConditionDelta`)
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs` (`ToSnapshot` — Quality=Good, or rely on default)
+
+**Layer 2**
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverInstanceActor.cs`
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/NativeAlarmProjector.cs`
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
+
+## Tests (TDD, RED-first)
+
+1. **Wire-level (the issue's suggested guard)** — extend `NativeAlarmEventIdentityFieldDeliveryTests`
+   (OpcUaServer.IntegrationTests): active alarm → event `Quality.IsGood`; driver disconnect → condition
+   event `Quality.IsGood == false`; reconnect → Good. Verify RED against pre-fix.
+2. **Node-level** — `NodeManagerAlarmSourceFieldsTests`: materialise sets Quality (native
+   `BadWaitingForInitialData`, scripted `Good`); `WriteAlarmCondition` projects snapshot quality and
+   fires on a quality-bucket change only.
+3. **`NativeAlarmProjector`** unit: `ProjectQuality` keeps Active/Acked + swaps quality; `Project`
+   preserves quality.
+4. **`DriverInstanceActor`**: `Connected` entry Tells `ConnectivityChanged(true)`; `DisconnectObserved`
+   Tells `ConnectivityChanged(false)`.
+5. **`DriverHostActor`**: `ConnectivityChanged(false)` pushes a Bad-quality `AlarmStateUpdate` to every
+   condition of that driver instance.
+
+## Deferred / notes
+
+- **Layer 3** (scripted worst-of-input quality) → **Gitea #478**.
+- **Implementation note:** L2 uses a **dedicated `IOpcUaAddressSpaceSink.WriteAlarmQuality`** path (not a
+  full-snapshot re-projection). Rationale: a connectivity change must set *only* Quality; re-projecting a full
+  snapshot would clobber a cold condition's severity/message and can't annotate a condition that never fired a
+  transition. The new sink method is forwarded through `DeferredAddressSpaceSink` (the F10b inertness trap) —
+  auto-verified by `DeferredSinkForwardingReflectionTests` (reflection guard) + its realm-discriminator guard.
+- **Test-harness note:** the new `DriverInstanceActor → parent` `ConnectivityChanged` Tell polluted existing
+  parent-`TestProbe` assertions in 3 `DriverInstanceActor*Tests` files; those tests now
+  `parent.IgnoreMessages(m => m is ConnectivityChanged)` since they assert on data/alarm/discovery forwards,
+  not connectivity.
+- `Bad_NoCommunication` vs generic `Bad`: v1 maps `OpcUaQuality.Bad → StatusCodes.Bad`; refining
+  `StatusFromQuality` to emit `BadNoCommunication` for the comms-loss case is a one-line nicety, noted in
+  the issue.
+- `docs/AlarmTracking.md` §"Condition event identity fields" gains a Quality subsection (Good/Bad
+  semantics, annotation-not-state-change, quality-bucket change fires an event).
+
+## Layer 3 — scripted worst-of-input quality (Gitea #478, implemented 2026-07-17)
+
+**Problem.** A scripted alarm is computed from one or more input tags. Its condition should report the
+**worst** quality of those inputs ("can I trust this condition's state?"), not the hardcoded `Good` Layer 1
+left at `ScriptedAlarmHostActor.ToSnapshot`.
+
+**Two blockers discovered in the live path (both silently discard quality):**
+1. `DependencyMuxActor.OnAttributeValuePublished` builds `VirtualTagActor.DependencyValueChanged` **without**
+   the `AttributeValuePublished.Quality` it already carries.
+2. `ScriptedAlarmHostActor.OnDependencyChanged` pushes each mux value into the engine's upstream with a
+   **hardcoded `0u` (Good)** StatusCode.
+So even a `Bad` driver value reached the scripted engine as Good — Layer 3 has to plumb quality first.
+
+**Design (mirrors Layer 2's native OT semantic through the scripted channel):**
+- **Plumb quality end-to-end.** `DependencyValueChanged` gains `OpcUaQuality Quality` (defaulted `Good`, so the
+  virtual-tag engine's calls are unchanged); the mux forwards `msg.Quality`; `OnDependencyChanged` maps it to a
+  StatusCode (`Good→0`, `Uncertain→0x40000000`, `Bad→0x80000000`) on the pushed `DataValueSnapshot`.
+- **Engine computes the worst input quality** each evaluation (over the refilled read cache, **before** the
+  `AreInputsReady` short-circuit so a `Bad` input is still observed) and carries it as
+  `ScriptedAlarmEvent.WorstInputStatusCode` (a raw `uint` StatusCode — `Core.ScriptedAlarms` doesn't reference
+  Commons, so it stays in the engine's existing StatusCode vocabulary; the host maps it to `OpcUaQuality`).
+- **Transitions carry the current worst quality** → `ToSnapshot` projects it (no clobber-back-to-Good when a
+  transition fires while an input is `Uncertain`).
+- **Quality-only changes emit out of band.** A `Bad` input freezes the condition (`AreInputsReady` returns
+  false → no transition), exactly like a comms-lost native driver — so quality can't ride a transition. The
+  engine tracks the last worst-quality **bucket** per alarm and, when the bucket changes with **no** transition
+  emission, emits a new `EmissionKind.QualityChanged` event. The host routes that to the **existing** Layer 2
+  `OpcUaPublishActor.AlarmQualityUpdate → IOpcUaAddressSpaceSink.WriteAlarmQuality` path (sets ONLY Quality,
+  one Part 9 event on a bucket change, **no `/alerts` row**, no historian write). No new sink surface.
+- **`ScriptedAlarmSource` (the `IAlarmSource` fan-out adapter) skips `QualityChanged`** — quality is delivered
+  through the dedicated node path, never as a phantom `AlarmEventArgs` (which would materialize/historize a
+  native condition).
+
+**Files (Layer 3):**
+- `src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/Part9StateMachine.cs` — `EmissionKind.QualityChanged`.
+- `src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmEngine.cs` — worst-of-input, bucket tracking,
+  `WorstInputStatusCode` on the event, quality-only emission.
+- `src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmSource.cs` — skip `QualityChanged`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs` — `DependencyValueChanged.Quality`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/DependencyMuxActor.cs` — forward `msg.Quality`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs` — push real quality;
+  `ToSnapshot` maps `WorstInputStatusCode`; `OnEngineEmission` routes `QualityChanged → AlarmQualityUpdate`.
+
+**Tests (RED-first):** engine — transition carries `Uncertain` worst; `Bad` input with no transition emits
+`QualityChanged(Bad)`; restore emits `QualityChanged(Good)`; no spurious emit when the bucket is unchanged.
+`ScriptedAlarmSource` — `QualityChanged` raises no `OnAlarmEvent`. Mux — `DependencyValueChanged` carries the
+published quality. Host — `Bad` dependency → `AlarmQualityUpdate(Bad)`, no `/alerts` publish; `ToSnapshot`
+maps the event's worst quality.
+
+**Coverage boundary → Layer 4 (#481).** L3 covers inputs whose driver **publishes a Bad/Uncertain-status
+data change** (the mux quality path). It does **not** cover a driver **comms loss**: a poll driver
+(Modbus/S7) whose device goes unreachable emits only `ConnectivityChanged` and goes silent on the value feed
+(`DriverInstanceActor.Reconnecting`), so the scripted engine keeps the last-known Good value and the
+condition stays Good — the same silent-feed problem native solved in L2, but native's `OnDriverConnectivityChanged`
+bridge fans only to **native** condition nodes (`_alarmNodeIdByDriverRef`), not into the mux the scripted
+engine reads. Bridging connectivity into scripted inputs — plus resolving the null-value/cold-start
+asymmetry (a runtime `Bad` with a null value is currently indistinguishable from cold start and contributes
+`Good`) and its ripple into virtual-tag quality — is **Gitea #481 (Layer 4)**. Found by the post-implementation
+code review; the code as shipped faithfully implements #478's written scope (mux-delivered input quality).

docs/plans/2026-07-20-localdb-adoption-phase1.md

@@ -0,0 +1,706 @@
+# OtOpcUa LocalDb Adoption — Phase 1 Implementation Plan
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+>
+> **Execution model:** This plan is optimized for **Claude Opus** agents (`claude --model opus`).
+> Dispatch implementer subagents on Opus for every task classified `high-risk`; `small`/`trivial`
+> tasks may use Sonnet. Work on branch **`feat/localdb-phase1`** in this repo (`~/Desktop/OtOpcUa`,
+> remote `lmxopcua`). Do NOT merge to `master` as part of this plan — stop at the DoD task and
+> report.
+>
+> **Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md`.
+> Read it before Task 0. The reference adoption is ScadaBridge (merged PR #23) — when in doubt,
+> mirror `~/Desktop/ScadaBridge` (files named per task below).
+
+**Goal:** Give every driver-role OtOpcUa node a consolidated `ZB.MOM.WW.LocalDb` database that caches
+the deployed-configuration artifact (chunked), boots from that cache when central SQL Server is
+unreachable, and optionally replicates it to the node's redundant pair peer over the library's gRPC
+sync (default-OFF, fail-closed bearer auth).
+
+**Architecture:** `AddZbLocalDb`/`AddZbLocalDbReplication` wired in a new `LocalDbRegistration`
+(mirroring `SecretsRegistration`), DDL + `RegisterReplicated` in `LocalDbSetup.OnReady`, a dedicated
+Kestrel h2c listener for `MapZbLocalDbSync` gated on `LocalDb:SyncListenPort`, a consumer-supplied
+fail-closed bearer interceptor, and an `IDeploymentArtifactCache` seam written by `DriverHostActor`
+after each successful apply and read as a boot fallback. The dormant LiteDB LocalCache subsystem is
+deleted as superseded.
+
+**Tech stack:** .NET 10, `ZB.MOM.WW.LocalDb`/`.Replication`/`.Contracts` **0.1.1** (Gitea feed),
+`Grpc.AspNetCore` 2.76.0, Akka.NET (existing), xunit.v3 (Host.IntegrationTests) + xunit2 TestKit
+(actor tests).
+
+**Hard rules (from the ScadaBridge adoption — violating any of these is a defect):**
+1. Order inside `OnReady` is load-bearing: **DDL → `RegisterReplicated` → writes**. Rows written
+   before registration are never captured or snapshotted — silently, forever.
+2. No autoincrement PKs, no BLOB columns in replicated tables.
+3. The library's passive sync endpoint verifies **no** ApiKey — the host interceptor is the only
+   auth, and it must be fail-closed (no key configured ⇒ deny everything).
+4. `ILocalDb.CreateConnection()` returns an **already-open**, pragma-configured connection with the
+   `zb_hlc_next()` UDF. Never call `Open()` on it. There is **no in-memory mode** — tests use temp
+   files + `SqliteConnection.ClearAllPools()` before delete.
+5. `Grpc.Core.Testing` does not exist on grpc-dotnet — hand-roll fake `ServerCallContext`s.
+6. Explicit Kestrel `Listen*` calls make Kestrel **ignore `ASPNETCORE_URLS`** — when adding the sync
+   listener you must re-bind the existing HTTP port in the same block, and verify it on the rig.
+7. Never run host `sqlite3` against a live bind-mounted WAL DB (copy the `db`/`-wal`/`-shm` triplet
+   with `cp` instead). `aspnet:10.0` containers have no `curl` — use a
+   `curlimages/curl` sidecar with `--network container:<name>`.
+
+---
+
+### Task 0: Preflight + recon (produces `docs/plans/2026-07-20-localdb-phase1-recon.md`)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (everything downstream consumes its findings)
+
+**Files:**
+- Create: `docs/plans/2026-07-20-localdb-phase1-recon.md`
+- Read-only: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DeploymentArtifact.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/SecretsRegistration.cs`,
+  `Directory.Packages.props`, `docker-dev/docker-compose.yml`,
+  the observability registration (`AddOtOpcUaObservability` implementation),
+  `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/*`
+
+**Step 1: Verify the feed packages restore.**
+
+Run:
+```bash
+cd ~/Desktop/OtOpcUa && dotnet package search ZB.MOM.WW.LocalDb --source dohertj2-gitea --format json | head -50
+```
+Expected: `ZB.MOM.WW.LocalDb`, `.Replication`, `.Contracts` at `0.1.1`. If absent → STOP, report.
+
+**Step 2: Map and record, with `file:line` citations, each of:**
+1. **Deploy fetch path:** where `DriverHostActor` loads `Deployment.ArtifactBlob` (the
+   `_dbFactory.CreateDbContext()` sites, ~L1472/1543), the artifact's runtime type
+   (`DeploymentArtifact` — how it's deserialized from the blob), the type/format of
+   `DeploymentId` and `RevisionHash`, and where a successful apply completes (the point after
+   which the cache write belongs).
+2. **Cold-boot path:** what the actor does at startup before any `DispatchDeployment` arrives —
+   does it query central SQL for the current deployment? Where does the failure path land
+   (the `Stale` state, `DbHealthProbeActor` interaction, L1345 dispatch-ignore)? Identify the
+   exact seam where "central fetch failed at boot" is known — that is where the cache fallback goes.
+3. **Cluster identity:** how a driver node knows its `ClusterId` (config key / `ClusterNode` row /
+   `IClusterRoleInfo`) — the cache is keyed by it.
+4. **DriverHostActor construction:** how its dependencies are injected
+   (`WithOtOpcUaRuntimeActors`, `Props` wiring) so `IDeploymentArtifactCache` can be threaded in.
+   ⚠ `Props.Create` builds an expression tree — adding a parameter silently rebinds
+   out-of-position named args at call sites; list every construction site.
+5. **Telemetry allowlist:** does `AddOtOpcUaObservability` restrict meters
+   (`ZbTelemetryOptions.Meters` or equivalent)? If yes, record where the list lives.
+6. **Kestrel/URLs:** confirm the Host binds via `ASPNETCORE_URLS=http://+:9000` with no
+   `ConfigureKestrel` calls; record any existing `builder.WebHost` usage.
+7. **LiteDB LocalCache:** confirm (grep) that nothing outside
+   `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/` and its tests references
+   `ILocalConfigCache`/`GenerationSealedCache`/`ResilientConfigReader`/`LiteDB`.
+8. **docker-dev volumes:** whether nodes have a writable data volume; record what `LocalDb:Path`
+   should be per node and how env vars are passed (`Cluster__*` style double-underscore).
+
+**Step 3: STOP conditions — halt the plan and report instead of improvising if:**
+- the artifact apply path cannot be given a post-apply hook without restructuring the actor;
+- `DeploymentId`/cluster identity are not stable strings/GUIDs;
+- LiteDB LocalCache turns out to be referenced by live code.
+
+**Step 4: Commit.**
+```bash
+git add docs/plans/2026-07-20-localdb-phase1-recon.md
+git commit -m "docs(localdb): phase-1 recon findings"
+```
+
+---
+
+### Task 1: Package references and pins
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+**Parallelizable with:** none (all code tasks build on it)
+
+**Files:**
+- Modify: `Directory.Packages.props`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ZB.MOM.WW.OtOpcUa.Runtime.csproj`
+
+**Step 1:** Add to `Directory.Packages.props` (versions exact):
+```xml
+<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.1" />
+<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.1" />
+<PackageVersion Include="Grpc.AspNetCore" Version="2.76.0" />
+```
+Check the existing `Grpc.Net.Client`/`Grpc.Core.Api` are already ≥ 2.76.0 and
+`Google.Protobuf` ≥ 3.34.1 (the LocalDb.Replication nuspec floors); raise if not. Confirm a
+`SQLitePCLRaw` pin ≥ 2.1.12 exists for the transitive `Microsoft.Data.Sqlite` chain (the family
+advisory GHSA-2m69-gcr7-jv3q); Core.AlarmHistorian already pins `bundle_e_sqlite3` 2.1.12 — add
+`SQLitePCLRaw.lib.e_sqlite3` 2.1.12 as an explicit reference in the Host if the audit flags it.
+
+**Step 2:** Host csproj: `ZB.MOM.WW.LocalDb`, `ZB.MOM.WW.LocalDb.Replication`, `Grpc.AspNetCore`.
+Runtime csproj: `ZB.MOM.WW.LocalDb` only (it needs `ILocalDb`, not the sync engine).
+
+**Step 3:** `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → 0 warnings/errors (repo builds warnings-as-errors).
+
+**Step 4: Commit.** `git commit -m "build(localdb): reference ZB.MOM.WW.LocalDb 0.1.1 + Grpc.AspNetCore"`
+
+---
+
+### Task 2: Schema + `LocalDbSetup.OnReady` + registration tests
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 3
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/DeploymentCacheSchema.cs`
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/LocalDbSetupTests.cs` (create the test project
+  reference layout to match neighboring Host unit tests; if Host has no unit-test project, place in
+  the closest existing one and note the deviation)
+
+**Step 1: Write the failing tests first** — using a real temp-file `ILocalDb` (build it via
+`new ServiceCollection().AddZbLocalDb(config, LocalDbSetup.OnReady)` with `LocalDb:Path` pointed at
+a temp file; dispose + `SqliteConnection.ClearAllPools()` in cleanup):
+- `OnReady_RegistersExactlyTheTwoDeploymentTables` — `db.ReplicatedTables.Keys` ordinal-sorted
+  equals `["deployment_artifacts", "deployment_pointer"]` (assert BOTH directions: no fewer, no more).
+- `DeploymentArtifacts_PkIsDeploymentIdPlusChunkIndex` and `DeploymentPointer_PkIsClusterId`
+  (pin `ReplicatedTable.PkColumns`).
+- `RowsWrittenAfterOnReady_EnterTheOplog` — insert a row, assert `SELECT COUNT(*) FROM __localdb_oplog` ≥ 1
+  (pins the DDL→register ordering; this is the assertion that catches a silently-broken CDC).
+
+**Step 2: Run tests, watch them fail** (types don't exist yet).
+
+**Step 3: Implement.** `DeploymentCacheSchema` depends only on `Microsoft.Data.Sqlite` (the
+ScadaBridge `*Schema.Apply` pattern — self-sufficient for direct construction in tests):
+
+```csharp
+namespace ZB.MOM.WW.OtOpcUa.Runtime.Deployment;
+
+public static class DeploymentCacheSchema
+{
+    public static void Apply(SqliteConnection connection)
+    {
+        using var cmd = connection.CreateCommand();
+        cmd.CommandText = """
+            CREATE TABLE IF NOT EXISTS deployment_artifacts (
+                deployment_id   TEXT    NOT NULL,
+                chunk_index     INTEGER NOT NULL,
+                cluster_id      TEXT    NOT NULL,
+                revision_hash   TEXT    NOT NULL,
+                chunk_count     INTEGER NOT NULL,
+                chunk_base64    TEXT    NOT NULL,
+                cached_at_utc   TEXT    NOT NULL,
+                PRIMARY KEY (deployment_id, chunk_index)
+            );
+            CREATE TABLE IF NOT EXISTS deployment_pointer (
+                cluster_id        TEXT NOT NULL PRIMARY KEY,
+                deployment_id     TEXT NOT NULL,
+                revision_hash     TEXT NOT NULL,
+                artifact_sha256   TEXT NOT NULL,
+                applied_at_utc    TEXT NOT NULL
+            );
+            """;
+        cmd.ExecuteNonQuery();
+    }
+}
+```
+
+`LocalDbSetup` (Host):
+```csharp
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+internal static class LocalDbSetup
+{
+    // ORDER IS LOAD-BEARING: DDL, then RegisterReplicated, then (nothing else in Phase 1).
+    // Rows written before RegisterReplicated are invisible to the peer forever, silently.
+    public static void OnReady(ILocalDb db)
+    {
+        using var connection = db.CreateConnection(); // already open — do NOT call Open()
+        DeploymentCacheSchema.Apply(connection);
+        db.RegisterReplicated("deployment_artifacts");
+        db.RegisterReplicated("deployment_pointer");
+    }
+}
+```
+
+**Step 4:** `dotnet test --filter "FullyQualifiedName~LocalDbSetupTests"` → PASS.
+
+**Step 5: Commit.** `git commit -m "feat(localdb): deployment-cache schema + OnReady registration"`
+
+---
+
+### Task 3: Fail-closed sync auth interceptor + tests
+
+**Classification:** standard
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 2
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/LocalDbSyncAuthInterceptorTests.cs`
+
+Port ScadaBridge's `src/ZB.MOM.WW.ScadaBridge.Host/LocalDbSyncAuthInterceptor.cs` (read it first;
+keep behavior identical, adjust namespace only). Semantics to preserve exactly:
+- `ServicePrefix = "/localdb_sync.v1.LocalDbSync/"`; any other method path passes through untouched.
+- Expected token from `IOptions<ReplicationOptions>.Value.ApiKey`.
+- **Fail-closed:** null/empty configured key ⇒ `RpcException(new Status(StatusCode.PermissionDenied, ...))`
+  for every sync call. Missing/mismatched bearer ⇒ same.
+- `CryptographicOperations.FixedTimeEquals` over UTF-8 bytes; header `authorization`,
+  case-insensitive, `"Bearer "` prefix. Override all four server handler kinds.
+
+**Tests** (hand-rolled fake `ServerCallContext` — `Grpc.Core.Testing` does not exist on grpc-dotnet;
+copy ScadaBridge's `LocalDbSyncAuthInterceptorTests.cs` fake): non-sync method passes with no key;
+sync + no configured key → PermissionDenied; wrong bearer → PermissionDenied; correct bearer → passes.
+
+Write tests first (fail), implement, `dotnet test --filter "FullyQualifiedName~LocalDbSyncAuthInterceptor"`,
+commit `feat(localdb): fail-closed bearer interceptor for the sync endpoint`.
+
+---
+
+### Task 4: `LocalDbRegistration` + Program.cs DI wiring + config defaults
+
+**Classification:** high-risk (touches Program.cs / role gating)
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (Task 5 edits the same file)
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (inside the `hasDriver` branch)
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json`
+
+**Step 1:** `LocalDbRegistration`, mirroring `SecretsRegistration`'s shape:
+```csharp
+namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+internal static class LocalDbRegistration
+{
+    /// Driver-role nodes only. Storage is unconditional; replication stays inert until
+    /// LocalDb:Replication:PeerAddress (initiator) / LocalDb:SyncListenPort (listener) are set.
+    public static IServiceCollection AddOtOpcUaLocalDb(
+        this IServiceCollection services, IConfiguration configuration)
+    {
+        services.AddZbLocalDb(configuration, LocalDbSetup.OnReady);
+        services.AddZbLocalDbReplication(configuration);
+        return services;
+    }
+
+    public static int SyncListenPort(IConfiguration configuration) =>
+        configuration.GetValue<int>("LocalDb:SyncListenPort");
+}
+```
+
+**Step 2:** Program.cs, in the `hasDriver` block (near `AddAlarmHistorian`):
+`builder.Services.AddOtOpcUaLocalDb(builder.Configuration);` and
+`builder.Services.AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>());`
+(AddGrpc only under `hasDriver` — admin-only nodes expose no sync surface).
+
+**Step 3:** appsettings.json:
+```json
+"LocalDb": {
+  "Path": "./data/otopcua-localdb.db",
+  "SyncListenPort": 0,
+  "Replication": {}
+}
+```
+`LocalDb:Path` is `ValidateOnStart`-required once `AddZbLocalDb` runs — since registration is
+driver-gated, admin-only configs need nothing. Check the role-overlay files
+(`appsettings.driver.json` / `appsettings.admin-driver.json`) and any deploy templates for
+conflicting `LocalDb` keys.
+
+**Step 4:** Build + full Host unit tests. Verify an admin-only graph doesn't require the key: this
+is pinned properly in Task 10's integration tests, but do a quick
+`OTOPCUA_ROLES=admin dotnet run` smoke only if cheap; otherwise rely on Task 10.
+
+**Step 5: Commit.** `feat(localdb): wire AddOtOpcUaLocalDb into the driver role`
+
+---
+
+### Task 5: Dedicated h2c sync listener + endpoint mapping (THE Kestrel task)
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (Program.cs)
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`
+
+**Why high-risk:** the Host today binds exclusively via `ASPNETCORE_URLS=http://+:9000`. Any
+explicit `ConfigureKestrel(... Listen*)` makes Kestrel **ignore URLs entirely** (it logs
+"Overriding address(es)"), which would silently kill the AdminUI/deploy API behind Traefik.
+
+**Step 1:** Add, before `builder.Build()`:
+```csharp
+var syncPort = LocalDbRegistration.SyncListenPort(builder.Configuration);
+if (hasDriver && syncPort > 0)
+{
+    // Explicit Listen* replaces ASPNETCORE_URLS wholesale, so re-bind the primary HTTP
+    // endpoint here too. Parse it from the configured URLs rather than hard-coding 9000.
+    var urls = builder.Configuration["ASPNETCORE_URLS"] ?? builder.Configuration["urls"] ?? "http://+:9000";
+    var httpPort = new Uri(urls.Split(';')[0].Replace("+", "localhost").Replace("*", "localhost")).Port;
+    builder.WebHost.ConfigureKestrel(k =>
+    {
+        k.ListenAnyIP(httpPort);                                            // HTTP/1.1 (existing surface)
+        k.ListenAnyIP(syncPort, o => o.Protocols = HttpProtocols.Http2);    // h2c, prior-knowledge gRPC
+    });
+}
+```
+When `syncPort == 0` (the default) nothing changes — URLs binding stays untouched. Cleartext
+`Http1AndHttp2` cannot serve prior-knowledge h2c, hence the dedicated `Http2`-only listener.
+
+**Step 2:** After the app pipeline's other `Map*` calls, driver-gated:
+```csharp
+if (hasDriver && syncPort > 0)
+{
+    app.MapZbLocalDbSync();
+}
+```
+(Mapping is harmless when unauthenticated — the interceptor fail-closes — but gating on the port
+keeps admin-only and default-OFF graphs entirely free of the endpoint.)
+
+**Step 3: Verify both surfaces.**
+```bash
+dotnet build ZB.MOM.WW.OtOpcUa.slnx
+```
+Then a local smoke with the port on:
+`ASPNETCORE_URLS=http://+:9000 OTOPCUA_ROLES=driver LocalDb__SyncListenPort=9001 dotnet run --project src/Server/ZB.MOM.WW.OtOpcUa.Host ...`
+— expect startup logs to show BOTH `:9000` and `:9001` bound, and `curl -s localhost:9000/healthz`
+(or the mapped health route) still answering. If the app needs SQL to boot, defer the smoke to the
+Task 12 rig check but say so explicitly in the task notes.
+
+**Step 4: Commit.** `feat(localdb): dedicated h2c listener + MapZbLocalDbSync gated on LocalDb:SyncListenPort`
+
+---
+
+### Task 6: `IDeploymentArtifactCache` + chunked LocalDb implementation + tests
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 3, Task 5
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactCache.cs`
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/LocalDbDeploymentArtifactCache.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Deployment/LocalDbDeploymentArtifactCacheTests.cs`
+
+**Step 1: Failing tests** (real temp-file `ILocalDb` with `LocalDbSetup.OnReady` applied):
+- round-trip: store a 300 KiB random artifact → `GetCurrentAsync` returns byte-identical payload
+  (forces ≥ 3 chunks at 128 KiB);
+- retention: store 3 deployments for one cluster → only the newest 2 remain in
+  `deployment_artifacts`; pointer names the newest;
+- integrity: corrupt one chunk row via SQL → `GetCurrentAsync` returns null (miss), never a
+  truncated artifact;
+- missing pointer → null.
+
+**Step 2: Implement:**
+```csharp
+public interface IDeploymentArtifactCache
+{
+    Task StoreAsync(string clusterId, string deploymentId, string revisionHash,
+                    byte[] artifact, CancellationToken ct = default);
+    Task<CachedDeploymentArtifact?> GetCurrentAsync(string clusterId, CancellationToken ct = default);
+}
+
+public sealed record CachedDeploymentArtifact(
+    string DeploymentId, string RevisionHash, byte[] Artifact, DateTimeOffset AppliedAtUtc);
+```
+`LocalDbDeploymentArtifactCache(ILocalDb db)`:
+- `StoreAsync`: one `ILocalDbTransaction` — delete any existing chunks for this `deployment_id`
+  (idempotent re-store), insert chunks (raw 128 * 1024 bytes per chunk, `Convert.ToBase64String`),
+  upsert the pointer (`INSERT ... ON CONFLICT(cluster_id) DO UPDATE`), then prune: delete
+  `deployment_artifacts` rows whose `deployment_id` is not among the newest 2 `cached_at_utc` for
+  this `cluster_id`. SHA-256 over the raw artifact into the pointer. ISO-8601 UTC timestamps.
+- `GetCurrentAsync`: read pointer; read chunks `ORDER BY chunk_index`; verify count ==
+  `chunk_count` and SHA-256 matches; return null on any mismatch (log a warning naming which check
+  failed).
+- Use `db.ExecuteAsync`/`QueryAsync` with anonymous-object parameters (`@Name` markers). Remember
+  a `Dictionary<string,string>` parameter **throws** — use anonymous objects.
+
+**Step 3:** run tests → PASS. **Step 4: Commit.**
+`feat(localdb): chunked deployment-artifact cache over ILocalDb`
+
+---
+
+### Task 7: Cache write path in `DriverHostActor`
+
+**Classification:** high-risk (actor model, `Props` expression-tree trap)
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:** (exact edit sites come from the Task 0 recon doc — cite it in the commit)
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
+- Modify: the actor-registration site (`WithOtOpcUaRuntimeActors` / DI extension) to provide
+  `IDeploymentArtifactCache`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` or `LocalDbRegistration` — register
+  `IDeploymentArtifactCache` → `LocalDbDeploymentArtifactCache` singleton (driver branch)
+- Test: the **xunit2** TestKit project used for existing DriverHostActor tests (recon names it)
+
+**Steps:**
+1. **Failing test:** after a successful artifact apply, the cache received `StoreAsync` with the
+   applied deployment's id/hash/bytes (inject a recording fake `IDeploymentArtifactCache`).
+   Also: a cache that throws on `StoreAsync` does NOT fail the apply (apply result unchanged, error
+   logged).
+2. Thread the dependency through the actor's constructor. ⚠ `Props.Create` is an expression tree:
+   after adding the parameter, re-check **every** construction site for positional/named-arg
+   rebinding (the recon lists them) — do not rely on the compiler.
+3. Invoke `StoreAsync` fire-and-forget (`PipeTo`-style or a guarded `Task.Run` per the actor's
+   existing async conventions — match whatever pattern the actor already uses for side-effect IO)
+   at the post-apply point identified in recon. Cluster id from the recon-identified source.
+4. Run the actor test suite for this project. Commit:
+   `feat(localdb): DriverHostActor stores applied artifacts in the pair-local cache`
+
+---
+
+### Task 8: Boot-from-cache read path + running-from-cache signal
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs`
+- Test: same xunit2 TestKit project as Task 7
+
+**Steps:**
+1. **Failing tests:**
+   - central fetch fails at startup AND cache has an artifact → the actor applies the cached
+     artifact (assert via whatever observable the apply already exposes) and logs/flags
+     running-from-cache;
+   - central fetch fails AND cache empty → today's behavior exactly (Stale, no apply);
+   - central fetch **succeeds** → cache is NOT consulted (fresh config wins; assert the fake
+     cache's `GetCurrentAsync` was never called on the happy path).
+2. Implement at the recon-identified boot-failure seam. The signal: a log warning at minimum plus,
+   if the actor already publishes health/status (recon says how), a `RunningFromCache` marker on it.
+   Do NOT touch `DispatchDeployment` handling — a new deployment still requires central.
+3. Run tests; commit `feat(localdb): boot from the pair-local artifact cache when central SQL is unreachable`.
+
+---
+
+### Task 9: Delete the dormant LiteDB LocalCache subsystem
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+**Parallelizable with:** Task 6, Task 10, Task 11
+
+**Files:**
+- Delete: `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/` (entire directory:
+  `ILocalConfigCache`, `LiteDbConfigCache`, `GenerationSealedCache`, `ResilientConfigReader`,
+  `GenerationSnapshot`, `StaleConfigFlag`, exceptions)
+- Delete: `tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/GenerationSealedCacheTests.cs`,
+  `ResilientConfigReaderTests.cs`
+- Modify: `Directory.Packages.props` + the Configuration csproj — remove the `LiteDB` package if
+  nothing else references it (grep first)
+
+**Steps:** Confirm the Task 0 recon's "nothing references it" finding still holds
+(`grep -rn "ILocalConfigCache\|GenerationSealedCache\|ResilientConfigReader\|LiteDB" src/ tests/`),
+delete, build the full solution, run the Configuration test project. DoD phrasing: **no references
+from code** — leave any explanatory prose/comments that point readers at LocalDb instead. Add one
+line to the recon doc noting LiteDB's removal. Commit:
+`refactor(localdb): delete the dormant LiteDB LocalCache (superseded by ZB.MOM.WW.LocalDb)`
+
+---
+
+### Task 10: Health check + telemetry meter allowlist
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 9, Task 11
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/LocalDbReplicationHealthCheck.cs` (or the
+  directory `AddOtOpcUaHealth` uses — recon names it)
+- Modify: the `AddOtOpcUaHealth` registration (driver branch)
+- Modify: the observability meter list IF Task 0 found an allowlist
+
+**Steps:**
+1. Failing unit tests: replication unconfigured (no peer, no listener) → `Healthy` (default-OFF
+   must not degrade a plain node); peer configured + `ISyncStatus.Connected == false` → `Degraded`;
+   connected + backlog `null` → `Degraded` (unknown backlog is not healthy); connected + backlog
+   small → `Healthy`.
+2. Implement against `ISyncStatus` + `IOptions<ReplicationOptions>` (peer-configured = non-empty
+   `PeerAddress` OR `SyncListenPort > 0` — pass the latter in via options/config).
+3. **Meter allowlist:** if `AddOtOpcUaObservability` restricts meters, add
+   `"ZB.MOM.WW.LocalDb.Replication"` (use `LocalDbMetrics.MeterName`) — this is a silent allowlist;
+   the ScadaBridge live gate is the proof it bites. If there is no allowlist, record that in the
+   recon doc and skip.
+4. Commit `feat(localdb): replication health check + meter export`.
+
+---
+
+### Task 11: DI-pin integration tests over the real Program.cs
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 9, Task 10
+
+**Files:**
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDbWiringTests.cs` (xunit.v3,
+  `WebApplicationFactory<Program>` — the project already builds the real Program.cs; set
+  `OTOPCUA_ROLES` per test case the way neighboring tests do)
+
+**Pins (each its own test):**
+1. Driver graph: `ILocalDb` resolves, is a singleton, `ReplicatedTables` ordinal-sorted ==
+   `["deployment_artifacts", "deployment_pointer"]` — exact set, both directions load-bearing.
+2. Driver graph: `IDeploymentArtifactCache` resolves to `LocalDbDeploymentArtifactCache`.
+3. Default-OFF pin: `ISyncStatus` resolves with `Connected == false`, `PeerNodeId == null`.
+4. Admin-only graph: `ILocalDb` is NOT registered (`GetService<ILocalDb>()` is null) and boot does
+   not demand `LocalDb:Path`.
+5. Health: the LocalDb health check is registered in the driver graph.
+
+Point `LocalDb:Path` at a per-test temp file via the factory's config overrides;
+`ClearAllPools` + delete in cleanup. These tests exist because DI extensions without a
+container-built test have shipped inert three times in this family (Secrets 0.2.0/0.2.2,
+ScadaBridge#22). Commit `test(localdb): DI pins over the real host graph`.
+
+---
+
+### Task 12: Two-node convergence harness + tests
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min (harness) + ~4 min (scenarios) — split the commit if needed
+**Parallelizable with:** none (depends on Tasks 2, 3, 5, 6)
+
+**Files:**
+- Create: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairHarness.cs`
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/LocalDbPairConvergenceTests.cs`
+
+**Harness:** copy the pattern from
+`~/Desktop/ScadaBridge/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs` and
+the library's `ConvergenceFixture` (`~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/tests/.../Convergence/ConvergenceFixture.cs`):
+- Two full stacks over a real loopback Kestrel h2c socket (port 0), **through the real
+  `LocalDbSyncAuthInterceptor`** with a shared test key; node A initiator, node B passive.
+- Both initialized via the **production `LocalDbSetup.OnReady`** — a hand-written schema proves
+  only that the test agrees with itself.
+- DBs owned by the harness, registered as pre-constructed singletons (so MS.DI won't dispose them
+  on host teardown); `KillTransportAsync`/`RestartTransportAsync`; tight `FlushInterval` (50 ms),
+  bounded backoff (2 s); temp files, `ClearAllPools` cleanup; a collection definition to serialize
+  these tests.
+
+**Scenarios:**
+1. `ArtifactStoredOnA_ConvergesToB_ByteIdentical` — store a multi-chunk artifact via
+   `LocalDbDeploymentArtifactCache` on A; B's cache returns byte-identical bytes; both nodes'
+   `__localdb_row_version` rows for the pointer carry the **same HLC and origin node id** (B holds
+   A's row, not a re-derived one).
+2. `RetentionPruneOnA_TombstonesReachB` — third deployment on A prunes the first; B's chunk rows
+   for the pruned deployment disappear and `__localdb_row_version WHERE is_tombstone=1` rows exist
+   on B.
+3. `WritesWhileTransportDown_SurviveRejoin` — kill transport, store on A, restart, converge.
+4. `WrongApiKey_NeverConverges` — harness with mismatched keys: assert **no** convergence after a
+   bounded wait AND (positive control) that the same scenario with matching keys converges — an
+   absence assertion without a positive control passed vacuously in ScadaBridge.
+
+**DoD within this task:** temporarily comment out the two `RegisterReplicated` calls and confirm
+scenarios 1–3 go red (run locally, do not commit the red state); restore. Record the red/green
+evidence in the task notes. Commit `test(localdb): 2-node convergence harness + scenarios`.
+
+---
+
+### Task 13: docker-dev rig configuration
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 14
+
+**Files:**
+- Modify: `docker-dev/docker-compose.yml`
+
+**Steps** (adjust to the recon's findings on volumes/env style):
+1. Every driver node: a named volume (or existing data mount) backing `/app/data`, and
+   `LocalDb__Path=/app/data/otopcua-localdb.db`.
+2. **site-a pair only** (mirror the ScadaBridge default-OFF posture — site-b stays unreplicated as
+   the pin):
+   - site-a-1: `LocalDb__SyncListenPort=9001`, `LocalDb__Replication__PeerAddress=http://site-a-2:9001`,
+     `LocalDb__Replication__ApiKey=dev-site-a-localdb-sync-key`, `LocalDb__Replication__MaxBatchSize=16`
+   - site-a-2: `LocalDb__SyncListenPort=9001`, same `ApiKey`, same `MaxBatchSize`, **no PeerAddress**
+     (passive; the stream is bidirectional).
+   The key must be byte-identical on both — the interceptor fail-closes on any mismatch and the
+   pair silently stops converging.
+3. `MaxBatchSize=16` because batching is row-count-only against gRPC's 4 MB cap and artifact chunk
+   rows are ≈ 171 KB (16 × 171 KB ≈ 2.7 MB worst case).
+4. `docker compose config` to validate; do NOT bring the rig up in this task (the live gate task
+   owns rig runs).
+5. Commit `chore(localdb): rig config — consolidated path everywhere, replication on the site-a pair`.
+
+---
+
+### Task 14: Documentation
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 13
+
+**Files:**
+- Create: `docs/operations/2026-07-20-localdb-pair-replication.md` (runbook: enable/disable,
+  ApiKey handling via `${secret:}`, stop/start-together rule, tombstone-retention resurrection
+  window, MaxBatchSize row-count-vs-4MB, never host-`sqlite3` a live WAL DB / cp-triplet recipe)
+- Modify: `docs/Redundancy.md` (a short "pair-local config cache" section: what replicates, what
+  boot-from-cache does and does not cover)
+- Modify: `CLAUDE.md` (this repo): LocalDb adoption row/paragraph — state Phase 1 scope, default-OFF,
+  rig-only enablement
+- Modify: `docs/Configuration.md` if it documents config keys (add the `LocalDb` section)
+
+Commit `docs(localdb): phase-1 runbook + redundancy/config docs`.
+
+---
+
+### Task 15: DoD sweep (offline)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none (final offline task)
+
+**Steps:**
+1. `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → **0 warnings** (warnings-as-errors).
+2. `dotnet test ZB.MOM.WW.OtOpcUa.slnx` → full suite green; record counts. Baseline any
+   pre-existing failures BEFORE this branch (`git stash` → run → unstash) rather than assuming;
+   report deltas only.
+3. Greps (phrase as "no references from code"; explanatory comments may remain):
+   `grep -rn "LiteDB\|ILocalConfigCache\|GenerationSealedCache" src/ tests/` → no code references.
+4. Confirm the positive-control evidence from Task 12 is recorded.
+5. Update `docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json` statuses.
+6. Commit `chore(localdb): phase-1 DoD sweep` and STOP. Report: branch name, commit list, test
+   counts, and that the live gate (Task 16) needs the rig + operator go-ahead.
+
+---
+
+### Task 16: Live gate on the docker-dev rig (run only with explicit user go-ahead)
+
+**Classification:** high-risk (rig)
+**Estimated implement time:** ~30 min wall-clock (mostly waiting)
+**Parallelizable with:** none
+
+**Preamble:** All DB inspection via `cp` of the `db`/`-wal`/`-shm` triplet out of the container
+(`docker cp`) and querying the copy — never host `sqlite3` on the live file (it poisons the WAL
+across virtiofs; root cause of the 2026-07-20 ScadaBridge incident). Metrics via
+`docker run --rm --network container:<node> curlimages/curl:latest -s localhost:<port>/metrics`.
+If an anomaly appears within seconds of one of your own measurements, suspect the measurement
+first — restart both nodes and re-run untouched before blaming the library.
+
+**Checks (all must PASS; record evidence in `docs/plans/2026-07-20-localdb-phase1-live-gate.md`):**
+1. Rig up, site-a pair healthy, `/metrics` on both shows `localdb_` series (proves the meter
+   allowlist work).
+2. Deploy a config through the normal deploy API → both site-a nodes' `deployment_pointer` +
+   `deployment_artifacts` rows are byte-identical with **identical HLC + origin node id** (one
+   node's row replicated, not two independent derivations — both nodes also locally store after
+   apply, so expect LWW-converged identical content either way; assert convergence, and note which
+   origin won).
+3. **Boot-from-cache:** stop the SQL container; restart site-a-2; it comes up serving the cached
+   config with the running-from-cache signal in its log; start SQL again; node recovers fresh
+   behavior.
+4. **Replicated-cache payoff:** wipe site-a-2's local DB file (container stopped), restart it with
+   SQL **up**, let replication repopulate the cache from site-a-1, then repeat check 3's SQL-down
+   restart — it boots from a cache it never wrote itself.
+5. Transport kill (pause site-a-2 container): store/deploy on a-1, oplog depth rises; unpause;
+   drains to 0; converged.
+6. Retention: deploy a 3rd config; pruned deployment's chunks gone on BOTH nodes with tombstone
+   rows present on both.
+7. Both-nodes-together restart: clean rejoin, identical counts, zero
+   `disk I/O error`/`SQLITE_IOERR`/`corrupt` in either log.
+8. site-b pair (default-OFF pin): LocalDb file exists and works locally, `ISyncStatus` health
+   Healthy, no sync connections, no listener on 9001.
+
+Record PASS/FAIL per check with the actual evidence (row counts, md5s, log lines). Commit the gate
+doc. Do not merge — report.
+
+---
+
+## Task persistence
+
+Tasks file: `docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json` (same directory).
+Update statuses as tasks complete; any deviation from this plan gets a `deviation` note on the task
+(the ScadaBridge tasks.json convention — the record of *why* is what makes the plan auditable).

docs/plans/2026-07-20-localdb-adoption-phase1.md.tasks.json

@@ -0,0 +1,153 @@
+{
+  "planPath": "docs/plans/2026-07-20-localdb-adoption-phase1.md",
+  "tasks": [
+    {
+      "id": 0,
+      "subject": "Task 0: Preflight + recon (findings doc, STOP conditions)",
+      "status": "completed",
+      "notes": "Feed verified: LocalDb/.Contracts/.Replication all 0.1.1. All 3 STOP conditions clear. Findings: docs/plans/2026-07-20-localdb-phase1-recon.md. 8 deviations recorded (D-1..D-8); D-1/D-3/D-5/D-6 change downstream work materially.",
+      "deviation": "D-1 cache read cannot be keyed by ClusterId at the boot seam (unkeyed newest-pointer read instead); D-3 a third LiteDB test file must be deleted; D-5 WebApplicationFactory<Program> is deliberately unused in this repo (use TwoNodeClusterHarness); D-6 driver-only nodes have no ASPNETCORE_URLS so the Kestrel re-bind fallback is wrong. See recon doc \u00a79."
+    },
+    {
+      "id": 1,
+      "subject": "Task 1: Package references and pins (LocalDb 0.1.1, Grpc.AspNetCore, SQLitePCLRaw)",
+      "status": "completed",
+      "blockedBy": [
+        0
+      ]
+    },
+    {
+      "id": 2,
+      "subject": "Task 2: Schema + LocalDbSetup.OnReady + registration tests",
+      "status": "completed",
+      "blockedBy": [
+        1
+      ]
+    },
+    {
+      "id": 3,
+      "subject": "Task 3: Fail-closed sync auth interceptor + tests",
+      "status": "completed",
+      "blockedBy": [
+        1
+      ]
+    },
+    {
+      "id": 4,
+      "subject": "Task 4: LocalDbRegistration + Program.cs DI wiring + config defaults",
+      "status": "completed",
+      "blockedBy": [
+        2,
+        3
+      ]
+    },
+    {
+      "id": 5,
+      "subject": "Task 5: Dedicated h2c sync listener + MapZbLocalDbSync (Kestrel URLs-override risk)",
+      "status": "completed",
+      "blockedBy": [
+        4
+      ]
+    },
+    {
+      "id": 6,
+      "subject": "Task 6: IDeploymentArtifactCache + chunked LocalDb implementation + tests",
+      "status": "completed",
+      "blockedBy": [
+        2
+      ]
+    },
+    {
+      "id": 7,
+      "subject": "Task 7: Cache write path in DriverHostActor (Props expression-tree trap)",
+      "status": "completed",
+      "blockedBy": [
+        0,
+        6
+      ]
+    },
+    {
+      "id": 8,
+      "subject": "Task 8: Boot-from-cache read path + running-from-cache signal",
+      "status": "completed",
+      "blockedBy": [
+        7
+      ]
+    },
+    {
+      "id": 9,
+      "subject": "Task 9: Delete the dormant LiteDB LocalCache subsystem",
+      "status": "completed",
+      "blockedBy": [
+        0
+      ]
+    },
+    {
+      "id": 10,
+      "subject": "Task 10: Health check + telemetry meter allowlist",
+      "status": "completed",
+      "blockedBy": [
+        4
+      ]
+    },
+    {
+      "id": 11,
+      "subject": "Task 11: DI-pin integration tests over the real Program.cs",
+      "status": "completed",
+      "blockedBy": [
+        5,
+        10
+      ]
+    },
+    {
+      "id": 12,
+      "subject": "Task 12: Two-node convergence harness + scenarios (+ positive control)",
+      "status": "completed",
+      "blockedBy": [
+        5,
+        6
+      ]
+    },
+    {
+      "id": 13,
+      "subject": "Task 13: docker-dev rig configuration (site-a pair on, site-b pin off)",
+      "status": "completed",
+      "blockedBy": [
+        5
+      ]
+    },
+    {
+      "id": 14,
+      "subject": "Task 14: Documentation (runbook, Redundancy.md, CLAUDE.md)",
+      "status": "completed",
+      "blockedBy": [
+        8
+      ]
+    },
+    {
+      "id": 15,
+      "subject": "Task 15: DoD sweep (offline) \u2014 STOP and report after this",
+      "status": "completed",
+      "blockedBy": [
+        7,
+        8,
+        9,
+        11,
+        12,
+        13,
+        14
+      ],
+      "notes": "Offline DoD sweep: full-solution build 0 errors (824 pre-existing OTOPCUA0001 analyzer warnings in driver *test* projects; none reference any LocalDb/DeploymentCache file). Grep: no LiteDB/ILocalConfigCache/GenerationSealedCache code refs (2 explanatory doc-comment lines in ILdapGroupRoleMappingService remain, allowed). Runtime.Tests 407/0/31 twice (the previously-flagged intermittent did NOT reproduce). Host.IntegrationTests LocalDb subset 40/0/0. Full-solution `dotnet test` with stash-baseline NOT run: many suites are infra-gated (driver fixtures, full Akka mesh, LDAP real-bind, shared-SQL DB tests) and cannot run offline on macOS \u2014 deferred to Task 16 / CI. Task 12 positive-control evidence recorded in commit afa5be71. Did NOT merge to master \u2014 stopped here per plan."
+    },
+    {
+      "id": 16,
+      "subject": "Task 16: Live gate on the docker-dev rig (needs explicit user go-ahead)",
+      "status": "completed",
+      "blockedBy": [
+        15
+      ],
+      "notes": "Live gate RAN on the docker-dev rig with explicit user go-ahead. 8/8 checks pass (check 4 with a documented limitation). Caught + fixed FOUR real defects offline tests missed (4b2f0e6e NU1101 packageSourceMapping, ce9fa07f ASPNETCORE_HTTP_PORTS re-bind, 9137cb41 empty-address-space-on-cache-boot, c6a9f93a cache-not-repopulated-on-RestoreApplied), each with a regression test. Two documented limitations (follow-ups): no replication back-fill of a fully-wiped node; oplog growth on default-OFF nodes. Post-fix: full build 0 errors, Runtime.Tests 409/0, all LocalDb Host.IntegrationTests green; only consistent failure is the infra-gated AbCip_Green_AgainstSim (sim not up). Evidence: 2026-07-20-localdb-phase1-live-gate.md. NOT merged."
+    }
+  ],
+  "lastUpdated": "2026-07-20T00:00:00Z"
+}

docs/plans/2026-07-20-localdb-adoption-phase2.md

@@ -0,0 +1,316 @@
+# OtOpcUa LocalDb Adoption — Phase 2 Implementation Plan (alarm-historian store-and-forward)
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+>
+> **Execution model:** Optimized for **Claude Opus** agents (`claude --model opus`); dispatch
+> `high-risk` tasks on Opus. Branch **`feat/localdb-phase2`** in `~/Desktop/OtOpcUa` (remote
+> `lmxopcua`). **Prerequisite: Phase 1 (`docs/plans/2026-07-20-localdb-adoption-phase1.md`) is
+> merged (or this branch is stacked on it) and its live gate passed.** Do not merge as part of this
+> plan; stop at the DoD task.
+>
+> **Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md` §3.4,
+> D8, D9. Reference implementation for "replace a bespoke store" phasing: ScadaBridge Phase 2
+> (`~/Desktop/ScadaBridge/docs/plans/2026-07-19-localdb-adoption-phase2.md` + its live gate doc).
+
+**Goal:** Move the alarm-historian store-and-forward buffer (today the standalone
+`alarm-historian.db` owned by `SqliteStoreAndForwardSink`) into the consolidated LocalDb file as a
+replicated table with a primary-gated drain, so a redundant pair no longer loses buffered alarm
+history when a node dies — and delete the sink's bespoke file/connection management outright.
+
+**Architecture:** `alarm_sf_events` (TEXT GUID PK) registered in `LocalDbSetup.OnReady`; the sink
+rewired onto `ILocalDb` behind its unchanged public seam; drain gated on the delivered-snapshot
+Primary role via `PrimaryGatePolicy` (at-least-once across failover, accepted and documented); a
+one-time idempotent migrator from the legacy file, running **after** registration.
+
+**Risk framing (from ScadaBridge):** Phase 2 replaces a *working* mechanism — a harder risk class
+than Phase 1's "add where none existed." Cutover (delete + rewire) lands in **one commit**; there is
+no dual-mechanism period, and the cutover is the test.
+
+**Hard rules:** identical to Phase 1's list (OnReady ordering, no autoincrement/BLOB, fail-closed
+auth already in place, no in-memory SQLite in tests, cp-triplet-only rig inspection), plus:
+- **Legacy-copy column lists must INTERSECT** with what the legacy file actually has
+  (`pragma_table_info` probe) — a missing column throws and readers silently discard every row.
+- Absence assertions need a positive control.
+- DoD greps phrased as "no references from code" (explanatory comments may survive).
+
+---
+
+### Task 0: Recon (produces `docs/plans/2026-07-20-localdb-phase2-recon.md`)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Create: `docs/plans/2026-07-20-localdb-phase2-recon.md`
+- Read-only: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs` (and the
+  whole `Core.AlarmHistorian` project), `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/AlarmHistorianOptions.cs`,
+  the drain worker (whatever forwards batches to `GatewayHistorian`/`SendEvent`),
+  `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/PrimaryGatePolicy.cs`,
+  `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` (how `_localRole` +
+  driver-member-count reach the gate today)
+
+**Record, with `file:line` citations:**
+1. The sink's exact table schema: name, columns, PK type (**autoincrement?** — decides whether the
+   migrator needs deterministic `mig-{node}-{legacyId}` ids), indices, and any BLOB columns
+   (**STOP condition:** a BLOB payload column cannot be registered — it must become base64 TEXT in
+   the new schema, and the recon must size the largest realistic payload against the 171 KB-ish
+   chunk guidance; alarm events are small JSON, so expect this to be fine, but verify).
+
+   > **PRE-ANSWERED 2026-07-21 (verified against `master` `d218282c`) — the STOP condition does NOT
+   > fire.** Re-verify cheaply during recon, but do not expect a surprise:
+   >
+   > - **Executed DDL:** `SqliteStoreAndForwardSink.cs:657-667`. It matches the class doc-comment at
+   >   `:17-26` exactly — no drift between the documented and executed schema.
+   > - **No BLOB.** All 8 columns are TEXT/INTEGER; the payload column is **`PayloadJson TEXT NOT
+   >   NULL`**. So no base64 conversion is needed and the chunk-size sizing is moot.
+   > - **Payload is small and bounded by shape.** `PayloadJson` is a serialized
+   >   `AlarmHistorianEvent` — 10 scalar fields (`AlarmId`, `EquipmentPath`, `AlarmName`,
+   >   `AlarmTypeName`, `Severity`, `EventKind`, `Message`, `User`, `Comment?`, `TimestampUtc`).
+   >   No collections, no nesting. Realistic worst case is low single-digit KB (operator `Comment`
+   >   is the only unbounded-ish field), far under the 171 KB chunk guidance.
+   > - **PK IS autoincrement** — `RowId INTEGER PRIMARY KEY AUTOINCREMENT`. This confirms the
+   >   plan's own prediction: LocalDb cannot replicate an autoincrement key, so the migrator
+   >   **must** mint deterministic `mig-{node}-{legacyId}` ids, and the new table needs a TEXT
+   >   GUID PK as already specified in the Architecture note.
+   > - **One index to carry across:** `IX_Queue_Drain ON Queue (DeadLettered, RowId)` (`:667`) —
+   >   the drain's covering index. The `alarm_sf_events` equivalent wants the same shape over
+   >   (dead-lettered flag, insertion order) so the drain query stays index-covered.
+   > - **Legacy column list for the migrator's `pragma_table_info` intersection check:**
+   >   `RowId, AlarmId, EnqueuedUtc, PayloadJson, AttemptCount, LastAttemptUtc, LastError,
+   >   DeadLettered`.
+2. The public seam: the interface the drain worker and producers use (e.g. `IAlarmHistorianSink` /
+   enqueue+dequeue+markDelivered+deadLetter methods), so the rewire can keep it byte-compatible.
+3. Semantics to preserve: `Capacity` (1,000,000) enforcement, `MaxAttempts` (10), dead-letter
+   retention (30 d), `BatchSize` (100), `DrainIntervalSeconds` (5) — where each lives.
+4. The drain worker's lifecycle: hosted service or actor? Where a Primary-role check can be
+   injected, and how the delivered-snapshot role (`RedundancyStateChanged` cache) is accessible
+   from it (via `DriverHostActor`, a shared status service, or a message). If the role is only
+   available inside `DriverHostActor`, note the cleanest bridge (e.g. an `IRedundancyRoleView`
+   singleton the actor updates) — that becomes Task 4's shape.
+5. Whether the sink is constructed per-node config path (`AlarmHistorian:DatabasePath`) anywhere
+   else (tests, tooling).
+6. How `AlarmHistorian:Enabled=false` short-circuits (NullAlarmHistorianSink) — the rewire must
+   keep the disabled path allocating no LocalDb tables? No: tables are created unconditionally in
+   `OnReady` (cheap, empty); only the sink/drain stay Null. Note this in the doc.
+
+Commit: `docs(localdb): phase-2 recon findings`.
+
+---
+
+### Task 1: `alarm_sf_events` schema + registration (+ tests)
+
+**Classification:** standard
+**Estimated implement time:** ~4 min
+**Parallelizable with:** none (Task 2 depends on it)
+
+**Files:**
+- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmSfSchema.cs` (depends only on
+  `Microsoft.Data.Sqlite`)
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs`
+- Test: extend `LocalDbSetupTests`
+
+Schema shape (adjust column names to the recon's findings — preserve today's semantics):
+```sql
+CREATE TABLE IF NOT EXISTS alarm_sf_events (
+    id               TEXT    NOT NULL PRIMARY KEY,   -- app-minted GUID (never autoincrement)
+    payload_json     TEXT    NOT NULL,
+    enqueued_at_utc  TEXT    NOT NULL,
+    attempts         INTEGER NOT NULL DEFAULT 0,
+    status           TEXT    NOT NULL DEFAULT 'pending',  -- pending | delivered | dead
+    last_attempt_utc TEXT    NULL,
+    dead_at_utc      TEXT    NULL
+);
+CREATE INDEX IF NOT EXISTS ix_alarm_sf_events_status ON alarm_sf_events(status, enqueued_at_utc);
+```
+`OnReady` order becomes: Phase-1 DDL → `AlarmSfSchema.Apply` → the two Phase-1
+`RegisterReplicated` calls → `RegisterReplicated("alarm_sf_events")` → **migrator (Task 5) last**.
+(All DDL may run before all registrations; the invariant is registration-before-writes.)
+
+TDD: failing test first — exact replicated set becomes
+`["alarm_sf_events", "deployment_artifacts", "deployment_pointer"]` (ordinal-sorted; update the
+Phase-1 exact-set pins in the same commit — they are *supposed* to go red here, that's them
+working). Oplog-capture test for an `alarm_sf_events` insert. Commit
+`feat(localdb): alarm_sf_events replicated table`.
+
+---
+
+### Task 2: Rewire the sink onto `ILocalDb` + delete bespoke file management (the cutover commit, part 1 of 2 — see Task 3)
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/SqliteStoreAndForwardSink.cs` (or replace
+  with `LocalDbStoreAndForwardSink.cs` — keep the public seam identical either way)
+- Modify: its registration (`AddAlarmHistorian`) to inject `ILocalDb`
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/AlarmHistorianOptions.cs` — remove
+  `DatabasePath` (a breaking config key removal: note it in the runbook/CHANGELOG task)
+- Test: the sink's existing unit tests, rewired to a temp-file `ILocalDb` via `TestLocalDb`-style
+  helper (create `tests/.../TestSupport` helper if none exists — real DB, never a stub: a stubbed
+  bare `SqliteConnection` lacks `zb_hlc_next()` and fails closed on registered tables)
+
+**Steps:**
+1. Write/port failing tests for the seam's semantics: enqueue, drain batch of `BatchSize`,
+   `MaxAttempts` → dead-letter, capacity enforcement, dead-letter retention purge.
+2. Implement over `ILocalDb.ExecuteAsync/QueryAsync` (anonymous-object params). Delete the private
+   connection/pragma/file-open code and any `PRAGMA journal_mode` calls (LocalDb owns pragmas).
+   GUIDs minted at enqueue (`Guid.NewGuid().ToString("N")`).
+3. `Capacity` enforcement: count-based insert guard (preserve today's overflow behavior per recon).
+4. Core.AlarmHistorian gains a package ref on core `ZB.MOM.WW.LocalDb` (interface only).
+5. Build + project tests green. **Commit together with Task 3** if the drain gate can't compile
+   separately (the ScadaBridge tasks-14/15/16 circular-dependency landmine — check before assuming
+   they're independent commits).
+
+---
+
+### Task 3: Primary-gated drain
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:** (exact shape from recon item 4)
+- Modify: the drain worker
+- Create (if recon says so): `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Redundancy/IRedundancyRoleView.cs`
+  — a singleton snapshot (`RedundancyRole? LocalRole`, `int DriverMemberCount`) updated by
+  `DriverHostActor` where it already caches `_localRole`
+- Test: drain-worker tests + an actor test pinning that `DriverHostActor` publishes role changes to
+  the view
+
+**Semantics:**
+- Drain runs only when `PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount)` is
+  true — same policy, same boot-window posture (unknown role drains only when the node is alone).
+- Delivered/dead-letter marks are row UPDATEs → they replicate, so the standby's copy tracks drain
+  progress and does not re-deliver already-marked rows after failover.
+- **At-least-once across failover is accepted:** rows delivered on the old primary whose
+  `delivered` mark hadn't replicated yet will be re-sent by the new primary. Document in the
+  runbook (Task 7); do NOT build dedup.
+- When replication is OFF (default), the gate still applies but `driverMemberCount` for a solo
+  node keeps today's behavior — verify with a test: single-node, role unknown → drains (no
+  regression for unpaired deployments).
+
+TDD: failing tests — secondary role does not drain; primary drains; unknown+alone drains;
+unknown+paired does not. Commit (with Task 2 if coupled):
+`feat(localdb): alarm S&F on LocalDb with primary-gated drain (cutover)`.
+
+---
+
+### Task 4: One-time legacy migrator (`alarm-historian.db` → consolidated)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 6
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/AlarmSfLegacyMigrator.cs`
+- Modify: `LocalDbSetup.OnReady` — call it **last**, after all `RegisterReplicated` calls
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/AlarmSfLegacyMigratorTests.cs`
+
+Mirror `~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs`:
+- Source path from the pre-removal `AlarmHistorian:DatabasePath` default (`alarm-historian.db`) —
+  read the raw config key even though the option property is gone.
+- `ShouldMigrate`: skip if file missing or `<file>.migrated` exists. `:memory:`/`file:` sources → no-op.
+- **If the legacy PK is autoincrement (recon):** deterministic ids `mig-{NodeName}-{legacyId}`
+  (node name from the recon-identified config key) — rerunnable without duplicates under
+  `INSERT OR IGNORE`, and no cross-node collision. If already GUIDs, copy as-is.
+- One transaction for the whole copy; `File.Move(path, path + ".migrated")` only after commit;
+  failure throws out of `OnReady` → boot fails, legacy untouched.
+- **Column intersection** via `pragma_table_info` on the legacy table; required-PK guard.
+- Both pair nodes migrate independently; their rows have distinct ids (node-prefixed), so the
+  merged buffer is the union — expected; note that the new primary will drain the standby's
+  migrated rows too.
+
+TDD: failing tests — happy path row counts; idempotent re-run; failure leaves legacy file
+untouched; migrated rows **enter the oplog** (assert `__localdb_oplog` — the registration-order
+pin); older legacy file missing a column still migrates (intersection). Commit
+`feat(localdb): one-time alarm-historian.db migrator`.
+
+---
+
+### Task 5: Convergence + failover scenarios in the pair harness
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 4
+
+**Files:**
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/LocalDb/AlarmSfConvergenceTests.cs`
+  (reuses the Phase-1 `LocalDbPairHarness`)
+
+**Scenarios:**
+1. `AlarmBurstOnA_ConvergesToB_AndOplogDrains` — enqueue N events on A; identical rowset on B;
+   oplog → 0.
+2. `DeliveredMarksReplicate` — mark rows delivered on A; B's copies show delivered (the
+   no-redeliver-after-failover property, asserted at the data layer).
+3. `WritesWhileTransportDown_SurviveRejoin`.
+4. Positive control: with `RegisterReplicated("alarm_sf_events")` commented out, scenarios 1–3 go
+   red (run locally, record, restore — do not commit red).
+
+Commit `test(localdb): alarm S&F convergence scenarios`.
+
+---
+
+### Task 6: Rig config + docs
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 4
+
+**Files:**
+- Modify: `docker-dev/docker-compose.yml` — enable `AlarmHistorian__Enabled=true` on the site-a
+  pair if not already (the gate needs a live buffer); remove any `AlarmHistorian__DatabasePath`
+  env vars (key deleted).
+- Modify: `docs/operations/2026-07-20-localdb-pair-replication.md` — add: alarm S&F replication
+  semantics, at-least-once-across-failover statement, migrator behavior (`.migrated` sidecar),
+  `DatabasePath` key removal.
+- Modify: `docs/AlarmHistorian.md` + `CLAUDE.md` — sink now lives in the consolidated LocalDb;
+  drain is primary-gated.
+
+Commit `docs+chore(localdb): phase-2 rig config + docs`.
+
+---
+
+### Task 7: DoD sweep (offline)
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+1. Full solution build → 0 warnings; full test suite green (deltas vs pre-branch baseline only).
+2. Greps, phrased as "no references from **code**": the old bespoke connection management
+   (`AlarmHistorian:DatabasePath`, direct `new SqliteConnection` inside Core.AlarmHistorian except
+   via schema helpers/tests) — explanatory comments may remain.
+3. Positive-control evidence from Task 5 recorded.
+4. Exact-set replicated-tables pin = 3 tables, both directions.
+5. Update `…phase2.md.tasks.json`; commit `chore(localdb): phase-2 DoD sweep`; STOP and report.
+
+---
+
+### Task 8: Live gate on the docker-dev rig (run only with explicit user go-ahead)
+
+**Classification:** high-risk (rig)
+**Estimated implement time:** ~30 min wall-clock
+**Parallelizable with:** none
+
+Same inspection rules as Phase 1's gate (cp-triplet, curl sidecar, observer-suspicion rule).
+Record in `docs/plans/2026-07-20-localdb-phase2-live-gate.md`:
+1. Migration ran: `.migrated` sidecar present on both site-a nodes; row counts match legacy.
+2. Alarm burst (drive a real driver alarm or the historian-gateway-unreachable path) converges:
+   identical rowsets, oplog drains to 0, dead letters 0.
+3. Only the primary drains: stop the historian gateway egress, buffer builds on BOTH nodes'
+   tables (replicated), but only the primary's drain worker logs attempts.
+4. Failover: stop the primary; the standby (new primary) resumes draining the shared buffer;
+   count of double-delivered events observed and recorded (at-least-once evidence, not a failure).
+5. Both-nodes-together restart clean; zero `disk I/O error`/`SQLITE_IOERR` in logs.
+6. site-b (default-OFF pin): sink works locally, no sync traffic.
+
+Commit the gate doc; report; do not merge.
+
+---
+
+## Task persistence
+
+Tasks file: `docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json`. Record deviations per
+task — especially if Tasks 2/3 had to land as one commit (the expected outcome).

docs/plans/2026-07-20-localdb-adoption-phase2.md.tasks.json

@@ -0,0 +1,86 @@
+{
+  "planPath": "docs/plans/2026-07-20-localdb-adoption-phase2.md",
+  "tasks": [
+    {
+      "id": 0,
+      "subject": "Task 0: Recon \u2014 sink schema, seam, drain lifecycle, role-view bridge (STOP conditions)",
+      "status": "completed",
+      "note": "STOP condition does NOT fire (no BLOB; PayloadJson TEXT). Recon doc: docs/plans/2026-07-20-localdb-phase2-recon.md. Key finding: the drain worker is an internal Timer inside the sink (not a hosted service/actor), so the gate is a Func<bool> ctor param; and HistorianAdapterActor already primary-gates ENQUEUE with a different policy (ShouldHistorize) - which is why today's ungated drain is safe and why replication breaks it. Deviations D-1..D-4 recorded."
+    },
+    {
+      "id": 1,
+      "subject": "Task 1: alarm_sf_events schema + registration (+ exact-set pin update)",
+      "status": "completed",
+      "blockedBy": [
+        0
+      ],
+      "note": "alarm_sf_events created in AlarmSfSchema (Core.AlarmHistorian) + registered third in LocalDbSetup.OnReady. Exact-set pin updated to 3 tables. DEVIATION D-5: kept the legacy delete-on-ack + dead_lettered flag + last_error rather than the plan's status column (no sweeper for 'delivered' rows; last_error is the only record of why a row died). Drain ORDER BY moves to (enqueued_at_utc, id)."
+    },
+    {
+      "id": 2,
+      "subject": "Task 2: Rewire sink onto ILocalDb + delete bespoke file management (cutover 1/2)",
+      "status": "completed",
+      "blockedBy": [
+        1
+      ],
+      "note": "Sink rewritten as LocalDbStoreAndForwardSink over ILocalDb; bespoke file/pragma/schema management deleted with the old class. AlarmHistorian:DatabasePath removed (breaking config key). Ids are a deterministic payload hash (D-1), not GUIDs."
+    },
+    {
+      "id": 3,
+      "subject": "Task 3: Primary-gated drain via PrimaryGatePolicy (cutover 2/2 \u2014 may co-commit with Task 2)",
+      "status": "completed",
+      "blockedBy": [
+        2
+      ],
+      "note": "Drain gated on IRedundancyRoleView, a singleton DriverHostActor publishes PrimaryGatePolicy's verdict to on every snapshot. Fails closed on a throwing gate; seeded OPEN so a non-redundant deployment is never silently stopped. New HistorianDrainState.NotPrimary + transition-logged (D-2). Landed with Task 2 as one commit (D-3)."
+    },
+    {
+      "id": 4,
+      "subject": "Task 4: One-time alarm-historian.db legacy migrator",
+      "status": "completed",
+      "blockedBy": [
+        3
+      ],
+      "note": "AlarmSfLegacyMigrator runs LAST in OnReady (which now takes IConfiguration - no skip-migration overload exists). DEVIATION D-6: ids are the payload hash (AlarmSfSchema.DeriveId, lifted out of the sink) rather than mig-{node}-{legacyId} - a warm pair's two legacy files OVERLAP, and node-prefixing would carry that duplication forward forever. DEVIATION: tests live in Host.IntegrationTests; the plan's Host.Tests project does not exist."
+    },
+    {
+      "id": 5,
+      "subject": "Task 5: Convergence + failover scenarios in the pair harness (+ positive control)",
+      "status": "completed",
+      "blockedBy": [
+        3
+      ],
+      "note": "4 scenarios green. POSITIVE CONTROL RUN (RegisterReplicated(alarm_sf_events) commented out): 3/4 went red immediately; the 4th (same-event-on-both-nodes) passed VACUOUSLY - with replication off each node trivially held its own single row. Strengthened by enqueuing a second distinct event on B and asserting BOTH nodes hold 2; it then went red under the control too. Control restored, all 4 green. Also fixed a SECOND exact-set pin the plan predicted: LocalDbWiringTests."
+    },
+    {
+      "id": 6,
+      "subject": "Task 6: Rig config + docs",
+      "status": "completed",
+      "blockedBy": [
+        3
+      ],
+      "note": "Rig: AlarmHistorian__Enabled=true on site-a-1/2 AND site-b-1/2, with MaxAttempts raised to 1e6 (D-4: the default 10 would dead-letter the queue ~5min into the rig's permanent gateway outage) and a deliberately unresolvable ServerHistorian__Endpoint. NOTE: the endpoint is REQUIRED - ServerHistorianOptionsValidator is consumer-gated on AlarmHistorian:Enabled, so an empty one fails host start. Docs: runbook, AlarmHistorian.md, AlarmTracking.md, Configuration.md, Redundancy.md, docs/README.md, CLAUDE.md."
+    },
+    {
+      "id": 7,
+      "subject": "Task 7: DoD sweep (offline) \u2014 STOP and report after this",
+      "status": "completed",
+      "blockedBy": [
+        4,
+        5,
+        6
+      ],
+      "note": "Build: 0 errors solution-wide; 0 warnings from every project this branch touches (the ~816 solution-wide warnings are pre-existing xUnit1051/OTOPCUA0001/CS86xx in untouched driver + client test projects). GREPS FOUND REAL DRIFT Task 6 missed: 8 live sites still named the deleted SqliteStoreAndForwardSink - CLAUDE.md, the AdminUI /alarms/historian panel text (user-visible), HistorianAdapterActor (a <see cref> that did NOT warn), Runtime + Host + 2 Driver.Historian.Gateway doc comments, 1 gateway test comment, docs/drivers/Historian.Wonderware.md; plus docs/AlarmTracking.md still claimed a Validate() startup warning for a relative DatabasePath (that branch is gone). All fixed. Code refs to AlarmHistorian:DatabasePath now reduce to exactly two intentional ones: AlarmSfLegacyMigrator.LegacyPathKey and its test. No 'new SqliteConnection' in Core.AlarmHistorian. Both exact-set pins assert the 3-table set, both directions. Guard-deletion evidence (both vacuous passes + the pin inventory) consolidated into the recon doc."
+    },
+    {
+      "id": 8,
+      "subject": "Task 8: Live gate on the docker-dev rig (needs explicit user go-ahead)",
+      "status": "completed",
+      "blockedBy": [
+        7
+      ],
+      "note": "Gate doc: docs/plans/2026-07-20-localdb-phase2-live-gate.md. Checks 1/2/5/6 PASS; checks 3+4 NOT SATISFIED (see below). FOUR production defects found + fixed, three of which crash-looped every driver node before check 1 could run: (1) empty ServerHistorian:ApiKey crashes the host - the validator had explicitly classified it as 'degrades', which is false because the gateway client validates its own options at construction; (2) UseTls vs endpoint scheme mismatch crashes likewise, both directions; (3) plaintext h2c was UNREACHABLE - the adapter forwarded TLS-only options unconditionally so every documented http:// deployment crashed; (4) THE BLOCKER - the drain gate deferred to a CLUSTER-WIDE elected Primary while the queue is PAIR-LOCAL. On the rig that Primary is central-1 (carries the driver Akka role, replicates nobody, runs no historian), so ALL FOUR driver nodes suspended their drains including unpaired site-b: silent permanent loss where pre-Phase-2 drained fine. Fixed in 3 layers (separate ShouldDrainAlarmHistory policy; peer-host matching in DriverHostActor; gate short-circuited entirely when replication is unconfigured, testing BOTH PeerAddress and SyncListenPort since only the dialer sets the former). Also fixed a THIRD vacuous test: AwaitAssert on the seeded-open value passed with the guard deleted; now asserts the sequence of PUBLISHED values via a recording view. Migration evidence: 11 legacy rows across two overlapping files -> exactly 9 identical rows on both nodes, proving D-6's payload-hash identity live. OPEN DESIGN FORK (in the gate doc): a pair cannot identify its own Primary, so both halves drain - safe (no loss, duplicates only) but the gate's de-dup benefit is unrealised."
+    }
+  ],
+  "lastUpdated": "2026-07-21T00:00:00Z"
+}

docs/plans/2026-07-20-localdb-phase1-live-gate.md

@@ -0,0 +1,93 @@
+# LocalDb Phase 1 — live gate (docker-dev rig)
+
+> Task 16 of `2026-07-20-localdb-adoption-phase1.md`. Evidence log for the 8 live-rig checks.
+> Rig: local `docker-dev/docker-compose.yml` (6 host nodes, one shared SQL). site-a pair replicates;
+> central + site-b are default-OFF.
+>
+> **Safety rules followed:** all DB inspection via `docker cp` of the `db`/`-wal`/`-shm` triplet out
+> of the container, querying the copy — never host `sqlite3` on the live WAL file. Metrics via a
+> `curlimages/curl` sidecar with `--network container:<node>` (`aspnet:10.0` has no curl).
+
+## Rig build/up
+
+Two real defects surfaced by the rebuild (both fixed on-branch, re-verified):
+
+1. **`NU1101` — packageSourceMapping gap** (`4b2f0e6e`). `ZB.MOM.WW.LocalDb*` was not mapped to the
+   `dohertj2-gitea` feed, so any *clean* restore (Docker image build, CI, fresh clone) failed. Local
+   dev builds masked it via the warm global NuGet cache. Fixed by adding the two patterns; verified
+   with a temp-packages-dir restore.
+2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`). The Kestrel re-bind read only
+   `urls`/`ASPNETCORE_URLS`; the aspnet:8.0+ base image sets `ASPNETCORE_HTTP_PORTS=8080` (all
+   interfaces) as the container default instead. A driver node that never sets `URLS` fell through to
+   `localhost:5000`, silently moving its health/metrics surface to loopback. Central nodes were fine
+   (they set `ASPNETCORE_URLS=9000` explicitly). Fixed to fall through URLS → HTTP_PORTS/HTTPS_PORTS →
+   5000; 4 new unit tests. After the fix, driver nodes bind `[::]:8080` + `[::]:9001` (site-a) /
+   `[::]:8080` only (site-b).
+
+Rig: 6 nodes rebuilt from branch `feat/localdb-phase1` @ `ce9fa07f`, all up.
+
+## Checks
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 1 | Rig up, site-a pair healthy, `/metrics` shows `localdb_` series | ✅ PASS | Both site-a nodes `/healthz` 200 Healthy; `/metrics` shows `localdb_oplog_depth` + `localdb_sync_reconnects_total` on meter `ZB.MOM.WW.LocalDb.Replication` (allowlist works); site-a-2 served `/localdb_sync.v1.LocalDbSync/Sync` with routing `match_status=success` (interceptor passed, session established); both oplogs depth 0 |
+| 2 | Deploy → both site-a nodes' pointer + artifacts byte-identical, same HLC + origin | ✅ PASS | Deploy `6e687451…` (rev `efb04c79…`) via `POST :9200/api/deployments` → both site-a-1 and site-a-2 `deployment_pointer` = `SITE-A / 6e687451… / efb04c79… / sha EFB04C79…` (identical); artifact 1 chunk each; **pointer `__localdb_row_version` identical on both: hlc `116955620225646592`, origin node_id `a0576df6-…`, tombstone 0** — one row replicated, not two derivations; both oplogs depth 0 |
+| 3 | Boot-from-cache: SQL down, restart site-a-2, serves cached config w/ signal | ✅ PASS (found + fixed a defect) | With SQL stopped, site-a-2 restart logged `RUNNING FROM CACHE — … booted deployment 6e687451…`; **address space materialised from the cached blob**: `AddressSpaceApplier: applied plan (added=18)` / `OpcUaPublish: applied rebuild (added=18)`. Browse confirms **16 nodes on site-a-2 = 16 on site-a-1**. **Defect the gate caught (`9137cb41`):** pre-fix the rebuild re-read the artifact from the down ConfigDb and no-op'd, so the cache-booted node served **1** node vs the healthy peer's 16 — it logged success but browsed empty. Fixed by passing the in-hand cached blob to `RebuildAddressSpace`; regression test added |
+| 4 | Replicated-cache payoff: wipe a-2 DB, repopulate, then SQL-down boot | ⚠️ PARTIAL — 1 defect fixed; the limitation is since **CLOSED** (LocalDb 0.1.2 — see Summary) | Wiped a-2's LocalDb volume, restarted with **SQL down**: sync session re-established (a-2 served a `Sync` call) but a-2's cache stayed **0/0/0** and a-1's oplog was **0** — the deploy's rows had been acked by the old a-2 and pruned, so there was no delta to send and **no snapshot-resync fired**. ⇒ **Limitation: pure replication does not back-fill a fully-wiped, already-converged node** (library delta-replication; snapshot-resync is gated on the oplog cap, which the running engine never hits). Separately found a **defect (`c6a9f93a`)**: recovery via `RestoreApplied` never wrote the cache (only fresh `ApplyAndAck` did), so a wiped node stayed cache-less. Fixed: `RestoreApplied` now re-caches. Verified — after the fix, wiped a-2 restarted with SQL up: `restored served state … on bootstrap` → cache repopulated `SITE-A / 6e687451`, chunks=1. **Net: a wiped node self-heals its cache from central on next boot (regaining boot-from-cache for future outages); it is not back-filled by its peer during the same outage.** |
+| 5 | Transport kill: pause a-2, oplog rises; unpause, drains to 0 | ✅ PASS | `docker pause` on a-2 severed the sync transport; during the window a-2 accumulated **oplog=2** unacked local writes (its re-cache) while a-1 held 0. `docker unpause` → a-2's oplog **drained to 0 within 5s**, both converged on the same pointer. Note: the rig's redeploys carry an identical revision, so a fresh cache write on a-1 short-circuits (no-op) — the store-during-partition variant is covered by the automated harness scenario `WritesWhileTransportDown_SurviveRejoin` (proven red without `RegisterReplicated`) |
+| 6 | Retention: 3rd deploy prunes oldest; chunks gone + tombstones on BOTH nodes | ✅ PASS | Cached 3 distinct deployment-ids (via deploy + restart re-cache). Both nodes retain exactly the newest 2 (`475df87a`, `962962dc`); the oldest `6e687451`'s chunks are **0 on both** (pruned); **1 artifact tombstone on both** — the prune replicated as a tombstone, not left as a live chunk |
+| 7 | Both-nodes-together restart: clean rejoin, identical counts, no I/O errors | ✅ PASS | Restarted site-a-1 + site-a-2 together: **0** `disk I/O error`/`SQLITE_IOERR`/`corrupt` lines in either log; both re-cached and converged to **identical** counts (ptr `962962dc…`, chunks=2, rowver=3). Byte-identical convergence preserved across a simultaneous restart |
+| 8 | site-b default-OFF pin: cache works locally, Healthy, no sync listener on 9001 | ✅ PASS (+1 finding, since **CLOSED** by the skip-if-unchanged cache write — see Summary) | site-b-1 cache works locally (`SITE-B / 6e687451`, chunks=1, from its own apply); listeners **`[::]:8080` only — no 9001**; port 9001 **not bound** (`/proc/net/tcp`); `/healthz` **200**; no sync/reconnect metric series. **Finding (follow-up, not a blocker):** `localdb_oplog_depth=5` — because `OnReady` registers replication unconditionally, a default-OFF node captures cache writes into its oplog but has no peer to ack/prune them, so the oplog grows slowly (≈2 rows/deploy, amplified by the re-cache-on-restore fix). Bounded by the library `MaxOplogRows` cap (1M); worth a periodic pruner or skip-if-unchanged in a follow-up |
+
+## Summary
+
+**8/8 checks pass** (check 4 with a documented limitation). The gate exercised the real image built
+from the branch, the real interceptor over a real loopback+cross-container h2c transport, and real
+`docker cp`-triplet DB inspection — and caught **four real defects that every offline test had
+missed**, each now fixed on-branch with a regression test:
+
+1. **`NU1101` packageSourceMapping gap** (`4b2f0e6e`) — `ZB.MOM.WW.LocalDb*` unmapped to the gitea
+   feed; every clean restore (Docker, CI, fresh clone) failed. Masked locally by the warm NuGet cache.
+2. **`ASPNETCORE_HTTP_PORTS` re-bind gap** (`ce9fa07f`) — the sync-listener re-bind ignored the
+   aspnet:8.0+ container-default port var, silently moving driver health/metrics to loopback:5000.
+3. **Empty address space on boot-from-cache** (`9137cb41`) — the rebuild re-read the artifact from the
+   down ConfigDb and no-op'd, so a cache-booted node logged success but served clients **1 node vs the
+   healthy peer's 16**. The single most important find: boot-from-cache was half-working.
+4. **Cache not repopulated on `RestoreApplied`** (`c6a9f93a`) — a wiped/fresh-volume node recovered its
+   served state from central but stayed cache-less, unable to boot-from-cache on the next outage.
+
+**Two documented limitations (follow-ups, not blockers) — both since CLOSED:**
+
+- **No replication back-fill of a fully-wiped node** (check 4): a peer's already-acked cache rows are
+  pruned from its oplog and snapshot-resync is gated on the (never-hit) oplog cap, so a wiped node is
+  not healed by its peer — it self-heals from central on next boot instead.
+  → **FIXED in `ZB.MOM.WW.LocalDb` 0.1.2 + 0.1.3** (scadaproj `cad3bcb` + `3b7489a`), live-gated in
+  `2026-07-21-localdb-followups-live-gate.md`. The diagnosis in this doc was
+  slightly off: the cap was not the gate. `ComputeSnapshotRequiredAsync` measured the peer's gap
+  against the *oldest surviving oplog row* and read an empty oplog as "no gap possible" — and an
+  empty oplog is the **steady state of a converged pair**, since ack-pruning deletes everything the
+  peer has confirmed. So the healthy state was the one state that could not heal a wiped peer. With
+  an empty oplog the gap is now measured against `last_acked_seq`. Covered at both levels:
+  `SnapshotResyncTests.WipedPeer_AfterEverythingWasAckedAndPruned_StillGetsSnapshot` in the library,
+  and `LocalDbPairConvergenceTests.WipedNode_IsBackFilledByItsPeer_WithoutAnyNewDeploy` here —
+  the latter verified RED against the pinned 0.1.1 and green on 0.1.2. **0.1.3 closes a second,
+  deeper half that only became reachable once back-fill worked:** the rebuilt node's OWN writes were
+  silently dropped, because `last_applied_remote_seq` is a watermark in the *peer's* seq space and a
+  rebuilt peer numbers from 1 again. The follow-up live gate caught that one — offline tests could
+  not have, since the scenario had no test while it was still a documented limitation.
+- **Oplog growth on default-OFF nodes** (check 8): `OnReady` registers replication unconditionally, so
+  a node with no peer accumulates unacked/unpruned oplog rows (~2/deploy). Slow, bounded by the 1M cap;
+  a periodic pruner or a skip-if-unchanged cache write would close it.
+  → **FIXED at the source**: `LocalDbDeploymentArtifactCache.StoreAsync` now **skips entirely** when
+  the pointer already names this deployment/revision, the SHA matches the bytes, and the expected
+  chunk count is present — so re-caching an artifact the node already holds (every restart's
+  boot-from-cache, every `RestoreApplied`) writes nothing and mints no oplog rows. Two guard tests
+  pin what the skip must NOT do (stale bytes under an unchanged revision; a matching pointer over
+  missing chunks), both verified RED against a naive pointer-only skip.
+  Also correcting this row's bound: the growth was never headed for the 1M cap. `AddZbLocalDbReplication`
+  is registered unconditionally, so `MaintenanceBackgroundService` runs on default-OFF nodes too and
+  the **7-day `MaxOplogAge`** cap prunes — the true bound was ~7 days of writes, not 1M rows.
+
+Merged to `master` as `c957db52` (2026-07-21) and pushed. Both follow-ups closed on
+`fix/localdb-phase1-followups` (2026-07-21) and live-gated — see
+`2026-07-21-localdb-followups-live-gate.md`, which found a third defect in the process.

docs/plans/2026-07-20-localdb-phase1-progress.md

@@ -0,0 +1,72 @@
+# LocalDb Phase 1 — Execution Progress (resume state)
+
+**Branch:** `feat/localdb-phase1` (off `master`). Working tree CLEAN, all work committed, nothing pushed.
+**Plan:** `docs/plans/2026-07-20-localdb-adoption-phase1.md`
+**Recon:** `docs/plans/2026-07-20-localdb-phase1-recon.md` (READ THIS — it has the 8 deviations D-1..D-8 that override the plan text).
+**Skill in use:** `superpowers-extended-cc:executing-plans` — batch, verify, commit per task, report between batches.
+**Do NOT merge to master** — plan stops at Task 15 (DoD) and reports; Task 16 is a rig live-gate needing explicit user go-ahead.
+
+## Status: Tasks 0–9 DONE (10 commits). Tasks 10–16 REMAIN.
+
+| # | Task | State | Commit |
+|---|---|---|---|
+| 0 | Preflight + recon | ✅ | `42f85507` |
+| 1 | Package refs (LocalDb 0.1.1, Grpc.AspNetCore 2.76.0) | ✅ | `44255fcb` |
+| 2 | Schema + LocalDbSetup.OnReady + tests | ✅ | `3b65c24b` |
+| 3 | Fail-closed sync auth interceptor + tests | ✅ | `3d29be83` |
+| 4 | LocalDbRegistration + Program.cs DI + appsettings | ✅ | `6aff9a83` |
+| 5 | h2c sync listener + MapZbLocalDbSync (Kestrel) | ✅ | `b9ddf20e` |
+| (fix) | rename Runtime.Deployment → Runtime.DeploymentCache | ✅ | `e771a11a` |
+| 6 | IDeploymentArtifactCache + chunked impl + tests | ✅ | `a38a52b8` |
+| 9 | Delete LiteDB LocalCache | ✅ | `2bae1b4c` |
+| 7 | Cache write path in DriverHostActor | ✅ | `1becf591` |
+| 8 | Boot-from-cache + RunningFromCache signal | ✅ | `a27eff32` |
+| 10 | Health check + telemetry meter allowlist | ⬜ TODO | |
+| 11 | DI-pin integration tests over real host graph | ⬜ TODO | |
+| 12 | 2-node convergence harness + scenarios | ⬜ TODO | |
+| 13 | docker-dev rig config | ⬜ TODO | |
+| 14 | Documentation | ⬜ TODO | |
+| 15 | DoD sweep (offline) — STOP + report | ⬜ TODO | |
+| 16 | Live gate on docker-dev rig — NEEDS USER GO-AHEAD | ⬜ TODO | |
+
+Native task list (TaskCreate ids 1–17 map to plan tasks 0–16) tracks the same; tasks.json in plan dir mirrors it.
+
+## What exists now (key files)
+
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/DeploymentCacheSchema.cs` — DDL, namespace `Runtime.DeploymentCache`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/IDeploymentArtifactCache.cs` — `StoreAsync` / `GetCurrentAsync` / **`GetCurrentUnkeyedAsync`** (D-1 addition) + `CachedDeploymentArtifact` record.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Deployment/LocalDbDeploymentArtifactCache.cs` — chunked impl. NOTE the prune guard `AND deployment_id <> @DeploymentId` (bug I found: pointer-target could be pruned under same-tick GUIDs → permanent silent miss).
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSetup.cs` — `OnReady` (DDL→RegisterReplicated). PUBLIC (Host has no InternalsVisibleTo).
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbRegistration.cs` — `AddOtOpcUaLocalDb`, `SyncListenPort(config)`. Registers `IDeploymentArtifactCache → LocalDbDeploymentArtifactCache` singleton.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/LocalDbSyncAuthInterceptor.cs` — fail-closed bearer.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/KestrelHttpBinding.cs` — URL parse/re-bind helper (D-6).
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` — LocalDb wired in `hasDriver` branch (~L176 after AddAlarmHistorian: `AddOtOpcUaLocalDb` + `AddGrpc(interceptor)`); Kestrel re-bind block before `builder.Build()`; `MapZbLocalDbSync()` gated `hasDriver && syncListenPort>0` before `MapOtOpcUaHealth()`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json` — `LocalDb` section (Path, SyncListenPort:0, Replication{}), between AlarmHistorian and Deployment.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs` — cache dep threaded LAST in Props/ctor/forwarding; `CacheAppliedArtifact` (write, after PushDesiredSubscriptions, empty-blob skip, own try/catch); `TryBootFromCache` + `ApplyCachedArtifact` (in Bootstrap catch); `PushDesiredSubscriptionsFromArtifact` split out; `_isRunningFromCache` field; `SingleClusterCacheKey="__single"`; `ReconcileDrivers` now returns `byte[]?`.
+- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs` — resolves `IDeploymentArtifactCache` (optional, nullable) and passes into DriverHostActor.Props.
+- `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Interfaces/NodeDiagnosticsSnapshot.cs` — added `bool RunningFromCache = false`.
+- Tests (all green): `Host.IntegrationTests/LocalDbSetupTests.cs`, `LocalDbSyncAuthInterceptorTests.cs`, `LocalDbSyncListenerTests.cs`; `Runtime.Tests/Deployment/LocalDbDeploymentArtifactCacheTests.cs`, `Runtime.Tests/Drivers/DriverHostActorArtifactCacheTests.cs`, `DriverHostActorBootFromCacheTests.cs`.
+
+## Load-bearing facts for the remaining tasks
+
+- **Namespace trap (recurs!):** any `Deployment` child namespace under `Runtime` OR `Runtime.Tests` shadows the `Deployment` EF entity → CS0118 across the assembly. Use `...DeploymentCache`. Only a FULL-SOLUTION build catches it (per-project builds pass).
+- **xUnit1051 is an error** in these test projects: every `ILocalDb`/DB call in a test must pass `TestContext.Current.CancellationToken` (xunit.v3 Host.IntegrationTests) — but `Runtime.Tests` is **xunit v2** (no `TestContext.Current`).
+- **Test project reality (D-5):** NO `Host.Tests` project; NO `WebApplicationFactory<Program>` (deliberately — `TwoNodeClusterHarness.cs:48` explains why). Host unit tests live in `Host.IntegrationTests` (xunit.v3). Actor tests in `Runtime.Tests` (xunit v2 + Akka.TestKit.Xunit2).
+- **Task 10 (D-7):** `AddOtOpcUaHealth()` takes NO args, called unconditionally at `Program.cs:365`. DON'T change its signature — register the LocalDb check unconditionally and return `Healthy` when LocalDb absent (ActiveNodeHealthCheck precedent). Health check classes: there are ZERO in-repo today; `Host/Health/` has only `HealthEndpoints.cs`. Meter allowlist is REAL and REQUIRED: `Host/Observability/ObservabilityExtensions.cs:30` `o.Meters = [OtOpcUaTelemetry.MeterName]` — add `LocalDbMetrics.MeterName` (from `ZB.MOM.WW.LocalDb.Replication`). Health check reads `ISyncStatus` + `IOptions<ReplicationOptions>`.
+- **Task 11:** copy the `TwoNodeClusterHarness` direct-host-build pattern, NOT WebApplicationFactory. Pins: `ILocalDb` singleton + `ReplicatedTables` == `["deployment_artifacts","deployment_pointer"]`; `IDeploymentArtifactCache`→concrete; `ISyncStatus` default-OFF (`Connected==false`); admin-only graph has NO `ILocalDb` + doesn't demand `LocalDb:Path`; health check registered. Set `OTOPCUA_ROLES` per case; `LocalDb:Path`→temp file, `ClearAllPools`+delete triplet in cleanup.
+- **Task 12:** harness copied from `~/Desktop/ScadaBridge/tests/.../LocalDbSitePairHarness.cs` + `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/tests/.../Convergence/ConvergenceFixture.cs`. Both nodes init via PRODUCTION `LocalDbSetup.OnReady`. Real loopback Kestrel h2c through the REAL interceptor. DBs as pre-constructed singletons. Positive control: comment out both `RegisterReplicated`, confirm scenarios 1–3 red, restore (don't commit red).
+- **Task 13 (D-8):** NO host node has a writable volume today — must ADD named volumes. `docker-dev/docker-compose.yml` services: `central-1/2` (admin,driver, MAIN), `site-a-1/2` (driver, SITE-A), `site-b-1/2` (driver, SITE-B). Env style `Section__Key`. `9001` free container-internal. Enable replication on **site-a pair only** (a-1 initiator w/ PeerAddress, a-2 passive, byte-identical ApiKey, MaxBatchSize=16). site-b stays OFF = default-OFF pin. Validate `docker compose config`; DON'T bring rig up.
+- **Task 15:** baseline pre-existing test failures FIRST (`git stash` → run full `dotnet test` → unstash), report deltas only. **KNOWN intermittent:** one `Runtime.Tests` failure seen ~3× across runs (mine + subagent's), never reproduced under trx logger, NOT one of the new tests — investigate/baseline here.
+- **Warnings-as-errors** across src + most test projects. Pre-existing CS0618/xUnit1051 warnings exist ONLY in `Client.Shared*` (which don't set TreatWarningsAsErrors) — not mine, ignore.
+- **SQLitePCLRaw:** no pin needed — LocalDb 0.1.1 nuspec floors `lib.e_sqlite3` at 2.1.12 (CVE-fixed). Confirmed.
+
+## Follow-ups logged (NOT in scope, don't action without asking)
+
+- `Polly.Core` in `Configuration.csproj` now orphaned (ResilientConfigReader was its only real consumer). Left in place — recorded in recon §7.3b.
+- Pre-existing: `TryRecoverFromStale` marks Applied w/o reconciling drivers (recon §3.5); `Install-Services.ps1` gives driver-only nodes no `ASPNETCORE_URLS` (recon §6.4, worked around by D-6).
+
+## Verify-anything-quick
+
+Full build: `dotnet build ZB.MOM.WW.OtOpcUa.slnx` (expect 0 errors). LocalDb tests:
+`dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LocalDb"` and
+`dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~ArtifactCache|FullyQualifiedName~BootFromCache"`.

docs/plans/2026-07-20-localdb-phase1-recon.md

@@ -0,0 +1,617 @@
+# LocalDb Phase 1 — Task 0 Recon Findings
+
+**Date:** 2026-07-20
+**Branch:** `feat/localdb-phase1` (based on `master`)
+**Plan:** `docs/plans/2026-07-20-localdb-adoption-phase1.md`
+**Design authority:** `~/Desktop/scadaproj/docs/plans/2026-07-20-otopcua-localdb-design.md`
+
+All `file:line` citations are against the branch base commit.
+
+---
+
+## Step 1 — Feed verification: PASS
+
+```
+$ dotnet package search ZB.MOM.WW.LocalDb --source dohertj2-gitea --format json
+ZB.MOM.WW.LocalDb              0.1.1
+ZB.MOM.WW.LocalDb.Contracts    0.1.1
+ZB.MOM.WW.LocalDb.Replication  0.1.1
+```
+
+All three present at **0.1.1**. No STOP. (Note: the local NuGet cache holds both `0.1.0` and `0.1.1`
+— pin exactly `0.1.1` so a stale cache cannot win.)
+
+---
+
+## Step 3 — STOP conditions: NONE TRIGGERED
+
+| STOP condition | Verdict |
+|---|---|
+| Artifact apply path cannot take a post-apply hook without restructuring the actor | **Clear** — `ApplyAndAck` has a clean insertion point (§2.2) |
+| `DeploymentId` / cluster identity are not stable strings/GUIDs | **Clear** — both are stable strings, but cluster identity is **not resolvable at boot**; see D-1 |
+| LiteDB LocalCache is referenced by live code | **Clear** — zero live-code references (§7) |
+
+Three STOP conditions clear. **Eight deviations** from the plan's stated assumptions are recorded in
+§9 — four of them (D-1, D-3, D-5, D-6) change the work materially.
+
+---
+
+## 1. Deploy fetch path
+
+### 1.1 Blob-load sites — there are **five** `CreateDbContext()` calls, not two
+
+| Line | Method | Purpose |
+|---|---|---|
+| `DriverHostActor.cs:513` | `Bootstrap()` | `NodeDeploymentStates` + `Deployments` at PreStart |
+| `DriverHostActor.cs:1472` | `ReconcileDrivers(DeploymentId)` | **loads `ArtifactBlob`** |
+| `DriverHostActor.cs:1543` | `PushDesiredSubscriptions(DeploymentId)` | **loads `ArtifactBlob` again** |
+| `DriverHostActor.cs:2000` | `TryRecoverFromStale()` | latest `Sealed` deployment (id + hash only, no blob) |
+| `DriverHostActor.cs:2029` | `UpsertNodeDeploymentState(...)` | writes `NodeDeploymentState` |
+
+`DriverHostActor.cs:1472-1476`:
+```csharp
+using var db = _dbFactory.CreateDbContext();
+blob = db.Deployments.AsNoTracking()
+    .Where(d => d.DeploymentId == deploymentId.Value)
+    .Select(d => d.ArtifactBlob)
+    .FirstOrDefault() ?? Array.Empty<byte>();
+```
+`:1543-1547` is byte-identical apart from the local. **The same blob is read twice per apply.**
+
+### 1.2 Artifact runtime type — there is no instance type
+
+`DeploymentArtifact` (`DeploymentArtifact.cs:48`) is a **static parser class** over
+`ReadOnlySpan<byte>`. The blob is never materialised into an object graph.
+
+- Storage: `byte[]` — `Deployment.cs:29` `public byte[] ArtifactBlob { get; init; } = Array.Empty<byte>();`
+- Format: **UTF-8 JSON**, `JsonDocument.Parse` (`DeploymentArtifact.cs:66`, `:490`). Not MessagePack.
+- Leniency contract: empty/malformed blob → empty result, **never throws** (`:62`, `:537-541`).
+
+**Consequence for the cache:** we cache raw `byte[]`. No serializer coupling. Good.
+
+### 1.3 Identity types
+
+| Type | Definition | Shape |
+|---|---|---|
+| `DeploymentId` | `Commons/Types/DeploymentId.cs:3` — `readonly record struct DeploymentId(Guid Value)` | `ToString()` → `Value.ToString("N")`, 32 hex chars, **no hyphens** (`:10`) |
+| `RevisionHash` | `Commons/Types/RevisionHash.cs:8` — `readonly record struct RevisionHash(string Value)` | SHA-256 hex, **lowercase 64 chars**, no `0x`; empty invalid (`:3-6`) |
+
+Both are stable, filesystem/SQL-safe strings. Use `DeploymentId.ToString()` (the "N" form) as the
+`deployment_id` TEXT column value.
+
+---
+
+## 2. Post-apply point
+
+### 2.1 `ApplyAndAck` control flow — `DriverHostActor.cs:1413-1458`
+
+```
+1415  _applyingDeploymentId = deploymentId;
+1416  Become(Applying);
+1425  UpsertNodeDeploymentState(..., Applying, null)
+1427  try {
+1429      ReconcileDrivers(deploymentId);            // blob fetch #1 — SWALLOWS DB errors
+1430      _currentRevision = revision;
+1431      UpsertNodeDeploymentState(..., Applied, null);
+1432      SendAck(..., ApplyAckOutcome.Applied, null, correlation);
+1436      _opcUaPublishActor?.Tell(RebuildAddressSpace(...));
+1439      PushDesiredSubscriptions(deploymentId);    // blob fetch #2 — SWALLOWS DB errors
+1440      OtOpcUaTelemetry.DeploymentApplied.Add(1, "outcome"="ack");
+1441      _log.Info("applied deployment ...");
+1443  }
+1444  catch (Exception ex) { ...Failed ACK... }
+1452  finally { _applyingDeploymentId = null; Become(Steady); }
+```
+
+### 2.2 Where the cache write belongs
+
+**Immediately after `DriverHostActor.cs:1439`**, before `:1440` — but **wrapped in its own
+try/catch**. Reason: `:1432` already sent an `Applied` ACK to the coordinator. An unguarded cache
+write that throws would fall into the `catch` at `:1444` and send a *second*, contradictory
+`Failed` ACK for a deployment the fleet already believes is applied.
+
+### 2.3 Paths that must NOT write to the cache
+
+1. **`catch` at `:1444`** — any apply failure.
+2. **Idempotent short-circuit at `:1402-1409`** — `HandleDispatchFromSteady` returns early with an
+   `Applied` ACK when `_currentRevision == msg.RevisionHash`. `ApplyAndAck` is never entered and no
+   blob is read.
+3. **⚠ THE SILENT-DEGRADATION TRAP.** `ReconcileDrivers:1478-1483` and
+   `PushDesiredSubscriptions:1549-1553` catch DB failures, log a **warning**, and `return` **without
+   rethrowing**. `ApplyAndAck` therefore reaches `:1440` and declares success having applied **zero
+   drivers** when the blob load failed. Caching "whatever we read" would **persist an empty artifact
+   as a successful apply** — poisoning the boot-from-cache path with a config that starts no drivers.
+
+   **Mitigation (load-bearing):** the cache write must be gated on the blob having actually loaded
+   non-empty. Have `ReconcileDrivers` surface the loaded blob upward (a field or an out-param)
+   rather than re-reading it a third time, and skip the cache write when it is
+   `Array.Empty<byte>()`. This also removes the duplicate read noted in §1.1.
+
+4. **`RestoreApplied` (`:1514-1528`)** — the bootstrap replay path. Restoring from central; caching
+   here is harmless and arguably desirable, but it is a distinct seam. **Phase 1 scope: skip it.**
+5. **`TryRecoverFromStale` (`:1996-2022`)** — never reads the artifact at all (see §3.4).
+
+---
+
+## 3. Cold-boot path
+
+### 3.1 PreStart — `DriverHostActor.cs:415-436`
+
+Ctor already did `Become(Steady)` at `:411`. PreStart subscribes to topics, spawns the virtual-tag
+and scripted-alarm hosts, then calls `Bootstrap()` at `:435`.
+
+### 3.2 `Bootstrap()` — `DriverHostActor.cs:506-565` — **yes, it queries central SQL at boot**
+
+Two queries inside one `try`: latest `NodeDeploymentState` for self (`:514-518`), then the matching
+`Deployment` row (`:527-532`). Four outcomes: no prior deployments → Steady; `Applied` →
+`RestoreApplied` (`:544`); `Applying` orphan → `ApplyAndAck` replay (`:549`); `Failed` → Steady.
+
+### 3.3 **THE SEAM** — `DriverHostActor.cs:560-564`
+
+```csharp
+catch (Exception ex)
+{
+    _log.Warning(ex, "DriverHost {Node}: ConfigDb unreachable on bootstrap; entering Stale", _localNode);
+    Become(Stale);
+}
+```
+
+**This is the exact point where "central fetch failed at boot" is known.** The cache fallback goes
+between `:562` and `:563`: attempt a local-cache load, and only `Become(Stale)` if the cache is also
+a miss.
+
+⚠ The catch also fires if the *second* query throws after the first succeeded, so `latest` is not in
+scope-valid state from the catch. **The cache must be self-describing** — it must carry its own
+`DeploymentId` + `RevisionHash`, because at this seam the actor has no idea what it should be
+restoring. The planned `deployment_pointer` shape already satisfies this.
+
+### 3.4 The `Stale` state — `DriverHostActor.cs:1343-1379`
+
+`ReconnectInterval = 30s` (`:55`), retry timer started at `:1378`.
+
+The dispatch-ignore at **`:1345-1348`** discards `DispatchDeployment` entirely — no ACK, no
+buffering (contrast `Applying:599` which does `Self.Forward(msg)`). A Stale node looks like a
+timeout to the coordinator. Phase 1 does **not** change this (per design §3.3: a *new* deployment
+still requires central).
+
+`RouteNodeWrite` fast-fails at `:1359-1360` with `"driver host stale (config DB unreachable)"`.
+
+### 3.5 Pre-existing gap worth recording (not Phase 1 scope)
+
+`TryRecoverFromStale` (`:1996-2022`) sets `_currentRevision` and marks `Applied` **without calling
+`ReconcileDrivers` or `PushDesiredSubscriptions`** — documented at `:1692-1695` and `:1706`. A
+Stale-recovered node reports Applied while running **zero drivers**, and
+`HandleDispatchFromSteady:1402` then short-circuits on revision match. Boot-from-cache largely
+neutralises this in practice, but the bookkeeping remains optimistic. **Logged as a follow-up, not
+fixed here.**
+
+### 3.6 `DbHealthProbeActor`
+
+**No interaction with `DriverHostActor` whatsoever.** Spawned as a sibling
+(`ServiceCollectionExtensions.cs:243-246`), consumed only by `OpcUaPublishActor`
+(`OpcUaPublishActor.cs:105`, `:256`, `:528`, `:543-544`). `DriverHostActor` runs its own independent
+30s `retry-db` timer. If running-from-cache should later influence OPC UA ServiceLevel,
+`OpcUaPublishActor`'s DB-health seam is the precedent to mirror.
+
+---
+
+## 4. Cluster identity — **the one real design problem (see D-1)**
+
+**There is no `ClusterId` field on `DriverHostActor` and no accessor for it.** The actor knows only
+`_localNode` (`DriverHostActor.cs:61`, type `Commons.Types.NodeId`, aliased at `:28`).
+
+`ClusterId` is resolved **per-artifact**, by `DeploymentArtifact.ResolveClusterScope(blob, nodeId)`
+(`DeploymentArtifact.cs:485-516`) — it scans the artifact's `Nodes[]` array for a matching `NodeId`
+and reads that element's `ClusterId`. Types: `ClusterScope` (`:46`), `ClusterFilterMode` (`:28`).
+
+`IClusterRoleInfo` (`Commons/Interfaces/IClusterRoleInfo.cs:12`) exposes `LocalNode`, `LocalRoles`,
+`HasRole`, `MembersWithRole`, `RoleLeader`, `RoleLeaderChanged` — **no `ClusterId`**.
+`docker-dev/docker-compose.yml` has `Cluster__Hostname`/`Port`/`Roles`/`SeedNodes` — **no
+`Cluster__ClusterId`**. The `ClusterNode` entity
+(`Configuration/Entities/ClusterNode.cs:7,10`) carries both, but lives in central SQL — exactly
+what is unreachable at the seam that needs it.
+
+> **The bind:** the design keys the cache by `ClusterId` so a replicated pair shares one cache
+> entry. But `ClusterId` is only discoverable *from a blob you already have* or *from central SQL*.
+> At the §3.3 boot-failure seam we have neither.
+
+**Resolution adopted (D-1):** keep `cluster_id` as the `deployment_pointer` PK — pair sharing is the
+whole point of replication and must not be given up. Resolve it at each end as follows:
+
+- **Write path** (`ApplyAndAck`, §2.2): call `DeploymentArtifact.ResolveClusterScope(blob, nodeId)`
+  on the blob just applied. `ClusterFilterMode.ScopeTo` → use its `ClusterId`. `None` (single-cluster
+  artifact) or `Suppress` → fall back to the literal `"__single"` sentinel, matching the actor's own
+  existing degenerate-case convention at `:1827-1830`.
+- **Read path** (boot seam, §3.3): we cannot compute the key, so **do not key the read**. Select the
+  single `deployment_pointer` row, ordered by `applied_at_utc DESC`, `LIMIT 1`. A node belongs to
+  exactly one cluster and its pair peer replicates the same cluster's row, so the table holds one
+  row in every real topology. If more than one row is present (a node was re-homed between
+  clusters), take the newest and **log a warning naming both cluster ids** — an operator-visible
+  signal rather than a silent wrong-config boot.
+- Optional escape hatch: honour a `Cluster:ClusterId` config key when present. **Deferred** — not
+  needed for the rig, and adding a required key contradicts the design's "storage ships
+  unconditionally" posture.
+
+This preserves the replication payoff (design §3.3) and the schema exactly as specified, and
+confines the change to the read query's `WHERE` clause.
+
+---
+
+## 5. `DriverHostActor` construction — the `Props` expression tree
+
+### 5.1 The factory — `DriverHostActor.cs:326-346`
+
+`static Props(...)` takes **16 parameters**, 14 optional, and forwards them **fully positionally**
+into `Akka.Actor.Props.Create(() => new DriverHostActor(...))` at **`:343-346`**. Ctor at
+`:375-412`, assignments `:393-408`.
+
+> ⚠ Six `IActorRef?` parameters and three interface-typed parameters mean many mis-bindings are
+> **type-compatible and therefore compile clean**. Callers use *named* arguments (safe); the
+> internal forwarding at `:343-346` is *positional* (unsafe).
+>
+> **Rule for Task 7: append the new parameter LAST in the `Props` signature, LAST in the ctor
+> signature, and LAST in the positional forwarding list at `:346`. Change nothing else.**
+
+### 5.2 Construction sites — **27 total** (1 production + 26 test)
+
+**Production (1):** `Runtime/ServiceCollectionExtensions.cs:332`, inside `WithOtOpcUaRuntimeActors`
+(`:193`); registered as `DriverHostActorKey` (`:344`). All args named except the first three. Does
+not pass `virtualTagHostOverride`, `scriptedAlarmHostOverride`, or `driverMemberCountProvider`.
+
+`WithOtOpcUaRuntimeActors` callers: `Host/Program.cs:331`,
+`Host.IntegrationTests/TwoNodeClusterHarness.cs:398`, `ServiceCollectionExtensionsTests.cs:42,144`.
+
+**Tests (26)**, all in `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/`:
+
+| File | Lines |
+|---|---|
+| `Drivers/DriverHostActorTests.cs` | 28, 60, 81, 118, 143, 189 |
+| `Drivers/DriverHostActorReconcileTests.cs` | 37, 59, 90, 119, 156, 193 |
+| `Drivers/DriverHostActorDiscoveryTests.cs` | 385, 671, 748, 1023 |
+| `Drivers/DriverHostActorWriteRoutingTests.cs` | 148, 208 |
+| `Drivers/DriverHostActorPrimaryGateTests.cs` | 228, 247 |
+| `Drivers/DriverHostActorVirtualTagTests.cs` | 44, 76 |
+| `Drivers/DriverHostActorNativeAlarmTests.cs` | 350 |
+| `Drivers/DriverHostActorNativeAlarmAckRoutingTests.cs` | 126 |
+| `Drivers/DriverHostActorProbeResultDropTests.cs` | 66 |
+| `Drivers/DriverHostActorHistoryWriterTests.cs` | 59 |
+| `Drivers/DiscoveryInjectionEndToEndTests.cs` | 219 |
+| `VirtualTags/DependencyMuxActorTests.cs` | 139 |
+
+No site calls `Props.Create<DriverHostActor>()` or `new DriverHostActor(...)` directly — the only
+`new DriverHostActor` in the tree is inside the factory at `:343`. Because the new parameter is
+optional and appended last, **no test site should need editing.**
+
+### 5.3 Status object for the running-from-cache signal
+
+`NodeDiagnosticsSnapshot` (`Commons/Interfaces/NodeDiagnosticsSnapshot.cs:17-22`) — built in
+`HandleGetDiagnostics` (`DriverHostActor.cs:1381-1398`), a positional record of
+`(NodeId, RevisionHash? CurrentRevision, IReadOnlyList<DriverInstanceDiagnostics> Drivers, DateTime AsOfUtc)`.
+`GetDiagnostics` is handled in **all three states** (`Steady:570`, `Applying:601`, `Stale:1349`), so
+a flag here is observable regardless of state. **This is the natural home for `RunningFromCache`**
+(append last to the record). Consumer: `IFleetDiagnosticsClient` → AdminUI.
+
+`IDriverHealthPublisher` (`:73`, defaulted `:402`) is **not** a fit — the host never calls it; it
+only passes it down to `DriverInstanceActor` children (`:1845`, `:1859`). Per-driver, not per-node.
+
+### 5.4 Async / side-effect IO convention
+
+The actor is **overwhelmingly synchronous-blocking on DB IO** — all five `CreateDbContext()` sites
+use `using var db = ...` with synchronous LINQ on the actor thread. No `async`/`await` in the file,
+no `Task.Run`.
+
+The single async pattern is `ContinueWith(..., TaskScheduler.Default).PipeTo(replyTo)` in
+`HandleRouteNodeWrite` (`:1188-1200`), with an explicit `Sender` capture before the continuation.
+
+**Recommendation for Task 7:** a **synchronous** cache write at the §2.2 point is *consistent with
+the file's existing style* and is correct there — the actor is mid-apply, holds `Applying`, and
+every other DB call on that path already blocks. A fire-and-forget `PipeTo(Self)` would need a
+handler registered in `Steady`, `Applying` **and** `Stale`, or it dead-letters after the
+`Become(Steady)` at `:1456`. **Prefer synchronous + try/catch.** (This deviates from the plan's
+"fire-and-forget"; see D-4.)
+
+`IWithTimers` is already implemented (`:50`, `:282`); `"retry-db"` key in use at `:1378`/`:2009`.
+
+---
+
+## 6. Host composition root
+
+### 6.1 Role flag — `Program.cs:47-50`
+
+```csharp
+var roles = RoleParser.Parse(Environment.GetEnvironmentVariable("OTOPCUA_ROLES"));
+var hasAdmin = roles.Contains("admin");
+var hasDriver = roles.Contains("driver");
+```
+Allowed roles: `admin`, `driver`, `dev` (`Cluster/RoleParser.cs:5-8`); unknown throws (`:25-27`).
+
+### 6.2 `hasDriver` branch — `Program.cs:131-310`
+
+Anchor for the new call: **`Program.cs:173-175`** `builder.Services.AddAlarmHistorian(...)`, inside
+the config-gated cluster at `:154-195`. Must land **before** `AddAkka` at `:314`.
+
+### 6.3 Pipeline — `Program.cs:368-404`
+
+`:383` `MapStaticAssets` · `:385-396` `if (hasAdmin) { ... }` · **`:398` `MapOtOpcUaHealth()`** ·
+`:399` `MapOtOpcUaMetrics()` · `:404` `RunAsync()`.
+
+⚠ **There is no `hasDriver` block in the pipeline half** — only `hasAdmin` is branched on. A
+driver-gated `app.Map*` is a new construct; it goes between `:396` and `:398`.
+
+`Program.cs:406-410` re-exports `public partial class Program`.
+
+### 6.4 Kestrel / URLs — **CONFIRMED: no Kestrel configuration exists**
+
+- `Program.cs:57` `builder.WebHost.UseStaticWebAssets();` — the **only** `builder.WebHost` call.
+- **Zero** `ConfigureKestrel` / `UseUrls` / `ListenAnyIP` / `UseKestrel` anywhere in `src/`.
+- Binding is exclusively via `ASPNETCORE_URLS`: `docker-compose.yml:173` (central-1) and `:240`
+  (central-2) set `http://+:9000`; `launchSettings.json:7` sets `http://localhost:9000`.
+
+Two traps for Task 5:
+
+1. ⚠ **`scripts/install/Install-Services.ps1:186`** (`$hostEnv += "ASPNETCORE_URLS=..."`) is inside
+   `if ($hasAdmin)` at `:185`. **A driver-only Windows-service node gets no `ASPNETCORE_URLS` at
+   all** and binds the ASP.NET default. The plan's parse-URLs-then-rebind snippet would re-bind
+   `http://+:9000` there — a port that node never listened on. The `?? "http://+:9000"` fallback in
+   the plan's snippet is therefore **wrong for driver-only nodes**; see D-6.
+2. ⚠ **`Host.IntegrationTests/TwoNodeClusterHarness.cs:331`** already calls
+   `builder.WebHost.UseKestrel(o => o.Listen(IPAddress.Parse(LoopbackHost), 0));`. Adding a
+   production `ConfigureKestrel` gives two competing explicit configurations in integration tests —
+   last-one-wins. The `syncPort > 0` gate keeps this dormant by default (harness sets no sync port),
+   but any harness that *does* set one will fight the port-0 binding.
+
+**Health route for smoke tests:** `/healthz` — liveness, runs no checks, always 200 while the
+process is up, `AllowAnonymous`. (`Health/HealthEndpoints.cs:48` → `MapZbHealth()`; routes from the
+`ZB.MOM.WW.Health` package.) Also `/health/ready`, `/health/active`, and `/metrics` (`Program.cs:399`).
+
+### 6.5 `SecretsRegistration` template — `Host/Configuration/SecretsRegistration.cs`
+
+Shape for `LocalDbRegistration` to mirror: `public static class` in
+`ZB.MOM.WW.OtOpcUa.Host.Configuration`; `public const string` section/key path constants; a
+`public static bool IsXEnabled(IConfiguration)` predicate using
+`GetValue(key, defaultValue: false)` (**default-deny**) that `Program.cs` can also call;
+`AddOtOpcUaX(this IServiceCollection, IConfiguration)` returning `IServiceCollection`, both args
+`ArgumentNullException.ThrowIfNull`-guarded; XML docs with `<remarks><para>` explaining *why* the
+gate is default-deny.
+
+Call-site precedent: `Program.cs:363` `AddOtOpcUaSecrets(...)`; predicate reused inside the `AddAkka`
+lambda at `Program.cs:323`.
+
+### 6.6 Telemetry meter allowlist — **EXISTS. Task 10 step 3 is REQUIRED, not conditional.**
+
+`Host/Observability/ObservabilityExtensions.cs:25-38`:
+```csharp
+return services.AddZbTelemetry(o =>
+{
+    o.ServiceName = "otopcua";
+    o.Meters = [OtOpcUaTelemetry.MeterName];          // <-- ObservabilityExtensions.cs:30
+    o.ActivitySources = [OtOpcUaTelemetry.ActivitySourceName];
+```
+
+`o.Meters` is a **strict allowlist** — `ZbTelemetryExtensions.cs:51-54` iterates it calling
+`metrics.AddMeter(name)`; **no wildcard support**. Anything unlisted is silently not exported.
+Single element today: `OtOpcUaTelemetry.MeterName` = `"ZB.MOM.WW.OtOpcUa"`
+(`Commons/Observability/OtOpcUaTelemetry.cs:19`).
+
+**Action:** add `LocalDbMetrics.MeterName` at `ObservabilityExtensions.cs:30`. This is exactly the
+omission the ScadaBridge live gate caught.
+
+### 6.7 Health registration — `Host/Health/HealthEndpoints.cs:19-41`
+
+`AddOtOpcUaHealth()` **takes no parameters** and is called **unconditionally** at `Program.cs:365`,
+outside both role blocks. Three `.AddTypeActivatedCheck<>` calls (`configdb`, `akka`,
+`admin-leader`); **no registration-time role gating exists** — `ActiveNodeHealthCheck` does runtime
+role scoping instead (returns `Healthy` when the node lacks the role).
+
+⚠ **There are zero `IHealthCheck` implementations in this repo** — all three come from the
+`ZB.MOM.WW.Health.*` packages (`Directory.Packages.props:124-126`, v0.1.0). A LocalDb check would be
+the **first health-check class in the tree**; `Host/Health/` holds only `HealthEndpoints.cs` today.
+
+**Consequence for Task 10:** driver-gating the new check requires changing
+`AddOtOpcUaHealth()`'s signature (add `bool hasDriver` or `IConfiguration`) and its `Program.cs:365`
+call site. **Preferred alternative:** follow the `ActiveNodeHealthCheck` precedent — register
+unconditionally and have the check return `Healthy` when LocalDb is not registered. That keeps the
+signature and matches the "default-OFF must not degrade a plain node" requirement for free. See D-7.
+
+### 6.8 Config files
+
+| File | Top-level sections |
+|---|---|
+| `appsettings.json` | `Serilog`, `Security`, `Secrets`, `ServerHistorian`, `ContinuousHistorization`, `AlarmHistorian`, `Deployment` |
+| `appsettings.driver.json` / `.admin.json` / `.admin-driver.json` / `.Development.json` | `Serilog`, `Security` only |
+
+Overlay selection: `Program.cs:67` sorts roles ordinally and joins with `-` → `admin,driver`
+becomes `appsettings.admin-driver.json`, added `optional: true` at `:70`; env vars (`:77`) and CLI
+args (`:78`) are re-appended after, so deployment overrides outrank the overlay.
+
+**`LocalDb` key search: CONFIRMED ABSENT.** Case-insensitive `localdb` across all `*.json`,
+`*.yml`, `*.ps1`, `*.csproj`, `*.props` matches only the two Phase-1/Phase-2 planning artifacts. No
+conflicting keys in any overlay or in `Install-Services.ps1`.
+
+**Placement convention:** each feature section opens with a `"_comment"` string then `"Enabled":
+false`; per-field caveats use sibling `"_<Field>Comment"` keys (`appsettings.json:34`, `:45`). Place
+`LocalDb` between `AlarmHistorian` (ends `:61`) and `Deployment` (`:62`).
+
+---
+
+## 7. LiteDB LocalCache — dormant, safe to delete
+
+### 7.1 Files (6) under `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/`
+
+`ILocalConfigCache.cs`, `GenerationSnapshot.cs`, `StaleConfigFlag.cs`, `LiteDbConfigCache.cs`
+(also declares `LocalConfigCacheCorruptException` at `:128`), `GenerationSealedCache.cs`,
+`ResilientConfigReader.cs`. Single namespace `...Configuration.LocalCache`. The only two
+`using LiteDB;` statements in `src/` are `LiteDbConfigCache.cs:1` and `GenerationSealedCache.cs:1`.
+
+### 7.2 Reference classification — **no live-code references. NO STOP.**
+
+- **(c) live code elsewhere: ZERO.** Also confirms full dormancy: `SealedBootstrap` (referenced by
+  `docs/v2/v2-release-readiness.md:46`) has **zero hits** in `src/` and `tests/` — it no longer exists.
+- Two benign near-misses:
+  - `Configuration/Services/ILdapGroupRoleMappingService.cs:14` — the string `ResilientConfigReader`
+    inside an **XML doc comment**. Prose only. Needs a comment edit or the doc goes stale.
+  - `Configuration/ZB.MOM.WW.OtOpcUa.Configuration.csproj:29` — `<PackageReference Include="LiteDB"/>`,
+    the only one in the repo.
+- Dozens of `LocalConfigCacheCorruptException` hits under `src/**/bin/**/*.xml` are **generated doc
+  build artifacts** (`GenerateDocumentationFile=true`), not source.
+
+### 7.3 ⚠ Test-file drift — the plan's delete list is incomplete
+
+Plan `…phase1.md:484-485` names only `GenerationSealedCacheTests.cs` and
+`ResilientConfigReaderTests.cs`. Actual test files in
+`tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/`:
+
+| File | Status |
+|---|---|
+| `GenerationSealedCacheTests.cs` | in plan ✅ |
+| `ResilientConfigReaderTests.cs` | in plan ✅ — note `StaleConfigFlagTests` is co-located here at `:323` |
+| **`LiteDbConfigCacheTests.cs`** | **MISSING from the plan** — will fail to compile if the source is deleted (`:125` calls `new LiteDB.LiteDatabase(...)` directly) |
+
+See D-3.
+
+### 7.3b Outcome (Task 9, executed 2026-07-20)
+
+Deleted: the 6 `LocalCache/` sources and **all three** test files (including
+`LiteDbConfigCacheTests.cs`, which the plan omitted — D-3). `LiteDB` removed from both
+`Configuration.csproj` and `Directory.Packages.props`. The stale XML-doc reference at
+`ILdapGroupRoleMappingService.cs:14` was rewritten to state the present truth (no fallback exists;
+reviving it means an admin-side cache on LocalDb, not restoring the old pipeline) rather than
+deleted, per the DoD's "explanatory prose may remain". Configuration tests: 92/92 green.
+
+**Follow-up found while executing:** `Polly.Core` in `Configuration.csproj` is now orphaned too —
+`ResilientConfigReader` was its only real consumer in that project (remaining `Polly` mentions are
+comments). Left in place deliberately: removing it is outside Task 9's scope and could break a
+consumer relying on the transitive flow. Worth a separate cleanup.
+
+### 7.4 Package removal
+
+- `PackageReference`: `Configuration.csproj:29` — only consumer.
+- `PackageVersion`: `Directory.Packages.props:35` — `LiteDB` `5.0.21`.
+
+Both safe to delete; no transitive consumer.
+
+---
+
+## 8. Test projects, rig, packages
+
+### 8.1 Test project layout
+
+| Question | Answer |
+|---|---|
+| `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/` | **DOES NOT EXIST** — there is no Host *unit*-test project at all |
+| Closest Host project | `Host.IntegrationTests` — despite the name it carries plenty of pure unit tests (`LdapOptionsValidatorTests`, `ServerHistorianOptionsValidatorTests`, `ResilienceInvokerFactoryRegistrationTests`, …) |
+| `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/` | **EXISTS** |
+| DriverHostActor tests | `Runtime.Tests/Drivers/` — 11 `DriverHostActor*Tests.cs` files. **xunit v2** (`xunit` 2.9.3), references **`Akka.TestKit.Xunit2`** ✅ |
+| `Host.IntegrationTests` xunit version | **xunit.v3** 1.1.0; SDK `Microsoft.NET.Sdk.Web` |
+
+`Directory.Packages.props` documents that AdminUI.Tests / ControlPlane.Tests / **Runtime.Tests** are
+the three xunit-v2 holdouts, held there by `Akka.TestKit.Xunit2` (verified 2026-07-13: no xunit.v3
+Akka TestKit exists). This matches the design's "actor tests live in an xunit2 TestKit project".
+
+### 8.2 ⚠ `WebApplicationFactory<Program>` is **deliberately not used** — Task 11 must be rewritten
+
+The only mention of the type in the entire `tests/` tree is a comment explaining why it is avoided:
+
+`Host.IntegrationTests/TwoNodeClusterHarness.cs:48`
+```
+/// <para>Why not <c>WebApplicationFactory&lt;Program&gt;</c>? Program.cs reads <c>OTOPCUA_ROLES</c> ...
+```
+
+The actual pattern builds the host directly — `TwoNodeClusterHarness.cs:329`:
+```csharp
+var builder = WebApplication.CreateBuilder(new WebApplicationOptions { Args = [] });
+```
+`Microsoft.AspNetCore.Mvc.Testing` is still referenced (csproj `:15`) but only for transitive
+test-host plumbing. **Copy `TwoNodeClusterHarness`, not `WebApplicationFactory`.** See D-5.
+
+`Host.IntegrationTests` also has its own `docker-compose.yml` (sql `14331`, ldap `3894`) gated by
+`DockerFixtureAvailability.cs`.
+
+### 8.3 docker-dev rig topology — plan's site-a/site-b assumption is **correct**
+
+Single Akka mesh (`otopcua`), seeded by `central-1`. Tenant separation is by `ServerCluster.ClusterId`
+rows (MAIN / SITE-A / SITE-B) inside **one shared ConfigDb** — not separate meshes.
+
+| Service | Roles | Cluster |
+|---|---|---|
+| `central-1`, `central-2` | `admin,driver` | MAIN |
+| `site-a-1`, `site-a-2` | `driver` | SITE-A |
+| `site-b-1`, `site-b-2` | `driver` | SITE-B |
+| `sql`, `migrator`, `cluster-seed`, `traefik` | — | infra |
+
+**There is no admin-only service** — the "admin" nodes are fused `admin,driver`. All six host nodes
+share the `&otopcua-host` YAML anchor.
+
+⚠ **NO HOST NODE HAS A WRITABLE VOLUME.** The only `volumes:` in the file are `sql` →
+`otopcua-mssql-data`, plus two read-only binds (`./seed:ro`, `./traefik-dynamic.yml:ro`). All six
+nodes write to the container's ephemeral layer, destroyed on recreate. This is already an accepted
+pattern (per-container `Secrets:SqlitePath=otopcua-secrets.db`, replicated over Akka rather than
+persisted). **Task 13 must add six named volumes** — there is nothing to piggyback on. See D-8.
+
+**Env style: confirmed double-underscore `Section__Key`**, with array indices and dictionary keys as
+a third segment (`Cluster__SeedNodes__0`, `Security__Ldap__GroupToRole__ReadOnly`). Exceptions that
+are plain SCREAMING_SNAKE (read directly, not config-bound): `OTOPCUA_ROLES`,
+`ZB_SECRETS_MASTER_KEY`, `GALAXY_MXGW_API_KEY`, `OTOPCUA_CONFIG_CONNECTION`, `ASPNETCORE_URLS`.
+
+**Ports.** Host-published: `14330`→sql, `4840`-`4845`→the six nodes' OPC UA, `9200`→traefik web,
+`8089`→traefik dashboard. Container-internal on every node: `4053` (Akka remoting), `9000`
+(Kestrel), `4840` (OPC UA). Also reserved on this machine: `14331`/`3894` (Host.IntegrationTests
+compose), `80`/`8080` (sibling scadabridge/scadalink stack), `10.100.0.35:3893` (shared GLAuth).
+
+**`9001` (the plan's choice) is free** container-internal and need not be published. ✅
+
+### 8.4 Package versions — `Directory.Packages.props`
+
+| Package | Version | Line |
+|---|---|---|
+| `Grpc.Core.Api` | 2.76.0 | 32 |
+| `Grpc.Net.Client` | 2.76.0 | 33 |
+| **`Grpc.AspNetCore`** | **NOT PRESENT** | — |
+| `Google.Protobuf` | 3.34.1 | 31 |
+| `Microsoft.Data.Sqlite` | 10.0.7 | ~55 |
+| `SQLitePCLRaw.bundle_e_sqlite3` | 2.1.12 | ~108 |
+| `LiteDB` | 5.0.21 | 35 |
+
+Both Grpc floors and the Protobuf floor are already satisfied — **no bumps needed**, only the new
+`Grpc.AspNetCore` entry (use 2.76.0 to match the train; watch for NU1605/NU1608 if it wants a
+Protobuf newer than the pinned 3.34.1).
+
+⚠ **`SQLitePCLRaw` is a surgical direct pin, not a transitive one.**
+`CentralPackageTransitivePinningEnabled` is deliberately **off** (it breaks the Roslyn version
+split). The 2.1.12 pin is promoted by a **direct `PackageReference` in `Core.AlarmHistorian`**
+(`Core.AlarmHistorian.csproj:15` + `:21`) — today the only Sqlite consumer in `src/`.
+
+> **Any new project that pulls `Microsoft.Data.Sqlite` MUST also take a direct
+> `<PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3"/>`,** or it silently resolves the
+> vulnerable 2.1.11 native bundle. This applies to **Host** and **Runtime** (both gain `ZB.MOM.WW.LocalDb`,
+> which depends on `Microsoft.Data.Sqlite`) and to every test project that builds a real `ILocalDb`.
+> Copy the `Core.AlarmHistorian.csproj:15,21` pair exactly. The plan's Task 1 mentions
+> `SQLitePCLRaw.lib.e_sqlite3` "if the audit flags it" — the correct package is
+> **`bundle_e_sqlite3`**, and it is **required, not conditional**.
+
+---
+
+## 9. Deviations from the plan
+
+Recorded per the tasks.json `deviation` convention. **D-1, D-3, D-5, D-6 change the work
+materially.**
+
+| # | Plan says | Reality | Resolution |
+|---|---|---|---|
+| **D-1** | Cache keyed by `ClusterId`; "cluster identity" is a recon item with a STOP condition | `ClusterId` is stable, but **not resolvable at the boot seam** — it lives inside the artifact or in the unreachable central DB (§4) | Keep `cluster_id` as PK (pair sharing is the point). **Write:** resolve via `DeploymentArtifact.ResolveClusterScope`, `"__single"` sentinel for the degenerate case. **Read:** unkeyed `ORDER BY applied_at_utc DESC LIMIT 1`, warn if >1 row. No STOP. |
+| **D-2** | Cache write "fire-and-forget (`PipeTo`-style or guarded `Task.Run`)" | The actor has **no async DB IO at all**; a `PipeTo(Self)` needs handlers in all three states or it dead-letters after `Become(Steady)` (§5.4) | Synchronous write in its own try/catch at `:1439`. Matches the file's style; failure still cannot fail the apply. |
+| **D-3** | Task 9 deletes 2 test files | **3** test files reference the subsystem — `LiteDbConfigCacheTests.cs` is missing from the list (§7.3) | Delete all three. Also fix the stale XML-doc mention at `ILdapGroupRoleMappingService.cs:14`. |
+| **D-4** | Cache the artifact after a successful apply | `ReconcileDrivers`/`PushDesiredSubscriptions` **swallow** DB errors, so a "successful" apply can have loaded an **empty blob** and started zero drivers (§2.3) | Gate the write on a non-empty blob; surface the blob from `ReconcileDrivers` instead of a third read. **Load-bearing — without it the cache can be poisoned with an empty config.** |
+| **D-5** | Task 11 uses `WebApplicationFactory<Program>` | Deliberately **not used** in this repo; `TwoNodeClusterHarness.cs:48` documents why (Program.cs reads `OTOPCUA_ROLES` from the environment) (§8.2) | Rewrite Task 11 against the `TwoNodeClusterHarness` direct-host-build pattern. |
+| **D-6** | Task 5 re-binds `ASPNETCORE_URLS ?? "http://+:9000"` | `Install-Services.ps1:185-186` sets `ASPNETCORE_URLS` **only when `hasAdmin`** — the fallback would bind a driver-only node to a port it never served (§6.4) | Re-bind only when a URL is actually configured; when absent, add **only** the sync listener. Also note `TwoNodeClusterHarness.cs:331` already calls `UseKestrel(Listen(…, 0))`. |
+| **D-7** | Task 10 driver-gates the health check | `AddOtOpcUaHealth()` takes no args and is called unconditionally at `Program.cs:365`; no registration-time gating exists anywhere (§6.7) | Register unconditionally; return `Healthy` when LocalDb is absent — the `ActiveNodeHealthCheck` precedent. Avoids a signature change and satisfies "default-OFF must not degrade a plain node". |
+| **D-8** | Task 13 "existing data mount" | **No host node has any writable volume** (§8.3) | Add six named volumes. Note: `9001` is free container-internal. Meter allowlist (§6.6) is **required**, not conditional. |
+
+### Follow-ups logged, not fixed here
+
+1. **`TryRecoverFromStale` marks `Applied` without reconciling drivers** (§3.5) — pre-existing,
+   documented in-code at `:1692-1695`. Out of Phase 1 scope.
+2. **Duplicate blob read per apply** (`:1472` and `:1543`, §1.1) — D-4's fix removes one of them as a
+   side effect.
+3. **`Install-Services.ps1` gives driver-only nodes no `ASPNETCORE_URLS`** (§6.4) — pre-existing;
+   D-6 works around it rather than fixing it.

docs/plans/2026-07-20-localdb-phase2-live-gate.md

@@ -0,0 +1,180 @@
+# LocalDb Phase 2 — live gate (docker-dev rig)
+
+> Task 8 of `2026-07-20-localdb-adoption-phase2.md`. Evidence log.
+> Rig: local `docker-dev/docker-compose.yml` — 2 central + site-a pair (replication ON) + site-b pair
+> (default-OFF), one Akka cluster, one shared SQL.
+>
+> **Safety rules followed** (same as the Phase 1 gate): all DB inspection via `docker cp` of the
+> `db`/`-wal`/`-shm` triplet out of the container, querying the copy — never host `sqlite3` on the
+> live WAL file.
+
+## Headline
+
+**The gate did not get past its own first boot before finding real defects.** Four production bugs,
+all fixed on-branch with regression tests, plus a fifth finding in the test suite itself. Three of
+the four crash-looped every driver node; the fourth silently stopped alarm history everywhere.
+
+Checks 1, 2, 5 and 6 pass. **Checks 3 and 4 are NOT satisfied** — see "The blocker" below; they
+cannot be run as written on a rig that has no per-pair redundancy roles, and the defect they were
+meant to confirm turned out to be the opposite of what the plan assumed.
+
+## Seeding the migration
+
+The rig had no pre-consolidation `alarm-historian.db`, so one was synthesised per node and `docker
+cp`-ed to `/app/alarm-historian.db` (the WORKDIR-relative default) into created-but-not-started
+containers — the exact upgrade shape.
+
+Deliberately overlapping content, to test D-6 in production: **6 rows on site-a-1, 5 on site-a-2,
+2 of them byte-identical payloads** (the warm-pair duplication `HistorianAdapterActor` produces in
+every boot window). 11 legacy rows, **9 distinct**.
+
+## Defects found and fixed
+
+### 1. Empty `ServerHistorian:ApiKey` crash-loops the host (was classified "degrades")
+
+`AlarmHistorian:Enabled=true` requires the gateway alarm writer, which requires a key.
+`HistorianGatewayClientOptions.Validate()` throws `The gateway API key must not be empty` inside the
+client factory during Akka startup, so every driver node crash-looped.
+
+`ServerHistorianOptionsValidator` exists precisely to convert that class of failure into a named
+`OptionsValidationException` — but its documented fail tier explicitly excluded `ApiKey`, reasoning
+that a keyless client "degrades rather than crashes (the gateway rejects calls)". **That premise is
+false**: the client validates its own options at construction, so the process dies before any call.
+Fixed; the key is never echoed in the message.
+
+### 2. `UseTls` / endpoint-scheme mismatch crash-loops the host
+
+Immediately after fixing #1, the rig crash-looped again: `UseTls` defaults to `true`, the rig
+endpoint is `http://`, and the client throws `UseTls requires an https gateway endpoint`. The
+inverse (`https://` + `UseTls=false`) throws too — both strings confirmed in the shipped client
+assembly. Setting an `http` endpoint without clearing `UseTls` is an ordinary migration slip that
+takes the host down. Now validated in both directions.
+
+### 3. Plaintext h2c was *unreachable* — every `http://` deployment crashed
+
+Third crash-loop, and a genuine product bug rather than operator config.
+`HistorianGatewayClientAdapter.Create` forwarded the TLS-only options unconditionally, and
+`AllowUntrustedServerCertificate` defaults to `false`, so it always sent
+`RequireCertificateValidation=true` — which the client rejects outright when `UseTls=false`
+(`RequireCertificateValidation is a TLS-only option and requires UseTls=true`).
+
+CLAUDE.md and `docs/Historian.md` both document `http://` as the supported way to select h2c, but
+**no h2c configuration could start**, and the only workaround was to claim
+`AllowUntrustedServerCertificate=true` — a certificate posture for a connection with no certificate.
+Fixed: TLS-only options stay at their defaults when `UseTls=false`.
+
+### 4. THE BLOCKER — the drain gate deferred to a Primary that cannot deliver
+
+With the rig finally up, **every driver node logged `Historian drain suspended`. Nothing drained
+anywhere** — including the two site-b nodes, which have no redundancy peer and no replication at all.
+Before Phase 2 this deployment drained fine, because the drain was ungated.
+
+Root cause, established from the rig rather than guessed:
+
+- `RedundancyStateActor` derives roles from Akka's **cluster-wide** `RoleLeader("driver")`.
+- `central-1`/`central-2` carry **both** `admin` and `driver` Akka roles (`OTOPCUA_ROLES=admin,driver`).
+- So the elected driver Primary is **central-1** — a node that replicates nobody's LocalDb and does
+  not even run the alarm historian.
+- Every site node was `Secondary`, saw that a Primary existed, and stood down in its favour.
+
+**The redundancy role is a cluster-wide election; the alarm queue is pair-local.** Deferring on
+"somebody is Primary" is therefore wrong in principle: in a multi-pair fleet the Primary is usually
+in another pair and holds none of this node's rows. The consequence is not a duplicate — it is the
+buffer growing on every node until the capacity wall evicts the oldest events, i.e. silent permanent
+loss of the audit trail the queue exists to protect.
+
+Fixed in three layers, each with a positive control:
+
+1. `PrimaryGatePolicy.ShouldDrainAlarmHistory(role, queueSharingPeerIsPrimary)` — a **separate**
+   policy from `ShouldServiceAsPrimary`. Unknown role ⇒ drain. A node stands down only for a Primary
+   that holds its rows. The two gates now deliberately disagree, and a test pins that disagreement so
+   nobody "harmonises" them back.
+2. `DriverHostActor` publishes that verdict, matching the snapshot's Primary against this node's
+   configured LocalDb replication peer host.
+3. `AddAlarmHistorian` short-circuits the gate entirely when replication is not configured — a node
+   whose rows exist nowhere else must never defer. The predicate tests **both** `Replication:PeerAddress`
+   and `SyncListenPort`, because only the dialing half sets the former and both halves share the queue.
+
+The asymmetry of the error costs is what drives every one of these: a false allow costs a duplicate
+history row, which the design already accepts (*at-least-once across a failover, by design*, and
+payload-hash ids collapse identical events); a false deny loses data silently.
+
+### 5. A third vacuous test — `AwaitAssert` on the seeded value
+
+Deleting the guard (publishing the old write-gate verdict) left `DriverHostActorRoleViewTests`
+**green**. `AwaitAssert` polls until an assertion passes, and the assertion was "the view reads
+open" — which is also the value the view is *seeded* with, so it succeeded at the first poll, before
+the actor had processed anything. It could not distinguish a correct publish from no publish at all.
+
+Fixed by asserting on the **sequence of published values** via a recording view, never on current
+state. Re-run under the control: red, for exactly the cases that matter. This is the same
+"absence-assertion passes by race" trap recorded for #485/#486, in a new disguise.
+
+## Checks
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 1 | Migration ran; `.migrated` sidecar on both site-a nodes; row counts match | ✅ PASS | `alarm-historian.db.migrated` present on both. 11 legacy rows across the two files → **exactly 9 rows on each node**, id sets byte-identical — the 2 shared payloads collapsed, proving D-6's payload-hash identity in production. Each node holds the *other's* rows (`legacy/a1-only-*` present on a-2 and vice versa). Legacy `attempt_count` distinction survived the copy (the 3-attempt row still leads the 0-attempt rows), `last_error` carried, 0 dead-lettered |
+| 2 | Alarm burst converges: identical rowsets, oplog drains to 0, dead letters 0 | ✅ PASS | Identical id sets on both nodes; `__localdb_oplog` depth **0** on both; **0** dead-lettered. Drain visibly live against the deliberately unresolvable gateway — `attempt_count` climbing 33 → 42 → 168 with `last_error='retry-please'`, which is exactly the buffering behaviour the queue exists for |
+| 3 | Only the primary drains | ❌ **NOT SATISFIED** — see defect 4 | As written the check assumes a per-pair Primary exists. On this rig the elected Primary is a central node outside both pairs, so the honest post-fix behaviour is that **all four nodes drain**. No node can currently identify a queue-sharing Primary: site-b is unpaired, and of the site-a pair only the dialing half knows its peer. Safe (no loss, duplicates only), but the check's intent is unproven |
+| 4 | Failover: standby resumes draining; count double-delivered events | ⛔ **NOT RUN** | Superseded by defect 4. A meaningful failover test needs a rig where the pair itself elects a Primary; measuring "double delivery" is meaningless while both halves drain unconditionally |
+| 5 | Both nodes restarted together: clean rejoin, identical counts, no I/O errors | ✅ PASS | `docker compose restart site-a-1 site-a-2` → **0** `disk I/O error` / `SQLITE_IOERR` / `corrupt` lines and 0 fatals in either log; both back to 9 rows / 0 dead / oplog 0, still identical |
+| 6 | site-b default-OFF pin: sink works locally, no sync traffic | ✅ PASS | `alarm_sf_events` created and registered on both site-b nodes; port **9001 not bound** (`/proc/net/tcp`), no sync listener, no replication config; both drain their own queue (defect 4's third fix) with 0 fatals |
+
+## The open design question
+
+The gate leaves one question that is a design fork, not a bug to patch:
+
+**A pair cannot currently identify its own Primary.** Redundancy roles are elected cluster-wide, but
+the queue is pair-local. Only the dialing half of a pair knows its peer (`Replication:PeerAddress`);
+the listening half sets `SyncListenPort` only. So the drain gate can never fully engage, and both
+halves of a replicated pair drain — safe, but duplicated.
+
+### Follow-up analysis — the root cause is wider than the drain gate
+
+Two things assumed while writing the section above turned out to be wrong, and correcting them
+changes the recommendation.
+
+**The rig is not misconfigured.** `central-1`/`central-2` carry `admin,driver` deliberately — the
+compose says so in as many words ("central is admin,driver"), because they are themselves a
+redundant pair that also runs the MAIN cluster's drivers. Stripping the driver role would change what
+the rig models, not fix anything.
+
+**A "pair" is not a missing concept — it already exists, as the application Cluster.** OtOpcUa's
+`Cluster`/`ClusterId` (the config-DB entity that owns drivers, devices and `ClusterNode` rows) is
+exactly the unit a redundant pair belongs to. Phase 1 already relies on this: the deployment-artifact
+cache is *keyed by ClusterId so a pair shares one entry*, and the rig's three application clusters
+(MAIN, SITE-A, SITE-B) are precisely its three pairs.
+
+So the real defect is that **`RedundancyStateActor` elects one Primary per *Akka* cluster, while
+every meaningful unit of redundancy is an *application* cluster.** The rig runs six driver nodes
+across three application clusters in one Akka cluster — a topology the product otherwise supports
+throughout — and exactly one of those six can ever be Primary.
+
+That mis-scoping is not confined to the alarm drain. Every consumer of the role inherits it:
+`ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate. On a multi-cluster fleet
+today, only one driver node cluster-wide will service Primary-gated work at all.
+
+### Revised recommendation
+
+**(C), as a separate change with its own live gate — and explicitly *not* (A).**
+
+- (C) is no longer "invent a pair concept"; it is "elect per `ClusterId` instead of per Akka cluster",
+  using a first-class entity that already exists. It is feasible where it needs to be:
+  `RedundancyStateActor` has no DB access today, but it is an admin-role singleton in the same
+  assembly as `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
+- **(A) is now actively undesirable.** Deriving pair identity from `LocalDb:Replication:PeerAddress`
+  would introduce a *second*, parallel notion of "who is my partner" that (C) immediately obsoletes —
+  and it cannot be done cleanly anyway: the library exposes no initiator/passive flag, so making both
+  halves declare a peer means both halves *dial*, opening two duplex sessions purely to obtain a
+  configuration value.
+- (C) touches the inbound device-write gate, which is safety-critical (the archreview 03/S4
+  dual-primary fix). It therefore belongs in its own change, with its own live gate, rather than
+  bolted onto Phase 2.
+
+Until then this branch is safe in every topology — no configuration loses data — so what remains is
+reclaiming the de-duplication benefit, not an outstanding hazard.
+
+## Not merged
+
+Per the plan, nothing on this branch is merged.

docs/plans/2026-07-20-localdb-phase2-recon.md

@@ -0,0 +1,371 @@
+# LocalDb Phase 2 — Recon findings
+
+**Date:** 2026-07-21 · **Branch:** `feat/localdb-phase2` · **Base:** `master` `2e46d054`
+
+Answers Task 0 of `2026-07-20-localdb-adoption-phase2.md`. Every claim carries a `file:line`
+citation against the base commit. Read the **Deviations** section last — two of them change the
+shape of Tasks 2/3 from what the plan assumed.
+
+---
+
+## 1. The sink's table schema
+
+**STOP condition does NOT fire** — confirmed by re-reading the executed DDL, as the plan's
+pre-answer instructed.
+
+Executed DDL: `SqliteStoreAndForwardSink.cs:652-670` (`InitializeSchema`). It matches the class
+doc-comment at `:17-26` exactly — no drift between documented and executed schema.
+
+```sql
+CREATE TABLE IF NOT EXISTS Queue (
+    RowId          INTEGER PRIMARY KEY AUTOINCREMENT,
+    AlarmId        TEXT    NOT NULL,
+    EnqueuedUtc    TEXT    NOT NULL,
+    PayloadJson    TEXT    NOT NULL,
+    AttemptCount   INTEGER NOT NULL DEFAULT 0,
+    LastAttemptUtc TEXT    NULL,
+    LastError      TEXT    NULL,
+    DeadLettered   INTEGER NOT NULL DEFAULT 0
+);
+CREATE INDEX IF NOT EXISTS IX_Queue_Drain ON Queue (DeadLettered, RowId);
+```
+
+| Question | Answer |
+|---|---|
+| BLOB columns? | **None.** All 8 columns are TEXT/INTEGER. Payload is `PayloadJson TEXT NOT NULL`. No base64 conversion needed; the 171 KB chunk sizing is moot. |
+| PK autoincrement? | **Yes** — `RowId INTEGER PRIMARY KEY AUTOINCREMENT`. The migrator must mint deterministic `mig-{node}-{legacyId}` ids and the new table needs a TEXT PK, as the plan specified. |
+| Indices to carry | One: `IX_Queue_Drain (DeadLettered, RowId)` — the drain's covering index. `alarm_sf_events` wants the same shape over (status, insertion order). |
+| Largest realistic payload | `PayloadJson` is a serialized `AlarmHistorianEvent` — 10 scalar fields (`AlarmHistorianEvent.cs`), no collections, no nesting. Low single-digit KB worst case; `Comment` is the only loosely-bounded field. |
+
+**Legacy column list** for the migrator's `pragma_table_info` intersection check:
+`RowId, AlarmId, EnqueuedUtc, PayloadJson, AttemptCount, LastAttemptUtc, LastError, DeadLettered`.
+
+`RegisterReplicated` will accept this table: no BLOB, and the PK will be app-minted TEXT.
+
+## 2. The public seam
+
+`IAlarmHistorianSink` (`IAlarmHistorianSink.cs:23-34`) is **two members**:
+
+```csharp
+Task EnqueueAsync(AlarmHistorianEvent evt, CancellationToken cancellationToken);
+HistorianSinkStatus GetStatus();
+```
+
+The concrete `SqliteStoreAndForwardSink` additionally exposes, and these are all consumed
+somewhere, so the rewire must keep them:
+
+| Member | Cited | Consumer |
+|---|---|---|
+| `StartDrainLoop(TimeSpan)` | `:189` | `ServiceCollectionExtensions.cs:105` (DI factory) |
+| `DrainOnceAsync(CancellationToken)` | `:320` | tests only — the deterministic drive |
+| `RetryDeadLettered()` | `:511` | operator action (AdminUI) |
+| `CurrentBackoff` | `:676` | tests |
+| `DebugCapacityProbeCount` | `:99` | tests |
+| `DefaultCapacity` / `DefaultMaxAttempts` consts | `:49`,`:53` | `AlarmHistorianOptions.cs:38,45` |
+| `IDisposable` | `:679` | DI + tests; **disposes the injected writer too** |
+
+There is **no separate enqueue/dequeue/markDelivered/deadLetter interface** — the plan's item 2
+guessed at one. Dequeue, delivered-marking and dead-lettering are all *private* to the sink
+(`ReadBatch :532`, `DeleteRow :564`, `DeadLetterRow :573`, `BumpAttempt :587`). That is good news:
+the rewire is entirely internal to one class and the public seam is untouched.
+
+## 3. Semantics to preserve
+
+All defaults live in **two** places that must stay in agreement:
+
+| Semantic | Sink default | Options default | Enforced at |
+|---|---|---|---|
+| `Capacity` 1,000,000 | `:49` | `AlarmHistorianOptions.cs:38` | `EnforceCapacityFastPathAsync :271` → `EnforceCapacityAsync :602` |
+| `MaxAttempts` 10 | `:53` | `:45` | drain loop `:429` |
+| Dead-letter retention 30 d | `:50` | `:41` | `PurgeAgedDeadLetters :638`, run once per drain tick |
+| `BatchSize` 100 | ctor arg `:116` | `:31` | `ReadBatch :542` |
+| `DrainIntervalSeconds` 5 | — | `:34` | `StartDrainLoop` via `ServiceCollectionExtensions.cs:105` |
+
+Behaviours that are **not** in the plan's list but are load-bearing, and which the cutover must
+not silently drop:
+
+- **Backoff ladder** 1s→2s→5s→15s→60s (`:57-64`), applied to the timer's *next due time*
+  (`RescheduleDrain :224`) so an outage genuinely slows the cadence.
+- **Overflow evicts oldest non-dead-lettered rows** with a WARN and a lifetime
+  `EvictedCount` counter (`:630-635`) — the durability guarantee is explicitly bounded.
+- **Corrupt-payload rows dead-letter immediately** for their own RowId so they cannot stall the
+  queue head (`:344-361`).
+- **Cardinality guard**: a writer returning ≠1 outcome per event is treated as a batch retry
+  (`:393-404`).
+- **In-memory row counter with periodic resync** every 10,000 enqueues (`:89-96`, `:271-290`) —
+  a perf optimisation that avoids `COUNT(*)` on the hot enqueue path.
+- **Post-dispose throwing contract**: `GetStatus`/`RetryDeadLettered`/`EnqueueAsync`/`StartDrainLoop`
+  throw `ObjectDisposedException`; `DrainOnceAsync` returns quietly (`:322`). Pinned by a
+  regression test (`SqliteStoreAndForwardSinkTests.cs:854-865`).
+
+## 4. Drain-worker lifecycle — **the finding that reshapes Task 3**
+
+> The plan asked "hosted service or actor?". **Neither.**
+
+The drain worker is a **self-rescheduling one-shot `System.Threading.Timer` owned by the sink
+itself** (`_drainTimer`, `:76`; started by `StartDrainLoop :189`; callback `DrainTimerCallback :199`;
+re-armed by `RescheduleDrain :224`). It is started from inside the DI singleton factory at
+`ServiceCollectionExtensions.cs:105`, so it begins ticking the moment the first consumer resolves
+`IAlarmHistorianSink`.
+
+Consequences for the Primary gate:
+
+1. **The gate has to live inside `DrainOnceAsync`**, not in a wrapper. There is no external
+   scheduler to gate.
+2. **`Core.AlarmHistorian` cannot see `PrimaryGatePolicy`.** The policy is in
+   `Runtime/Drivers/PrimaryGatePolicy.cs`, and Runtime → Core is the reference direction.
+   `Core.AlarmHistorian` references only `Core.Abstractions` (`.csproj`). So the sink takes a
+   **`Func<bool>` gate delegate** (default: always-drain) and Runtime supplies the lambda that
+   calls `PrimaryGatePolicy`. No new project reference, no new interface in Core.
+3. **A role view is still needed** because the timer fires outside any actor. See the next point
+   for who should own it.
+
+### There is already a Primary gate — on the *enqueue* side, with a different policy
+
+`HistorianAdapterActor` (`Historian/HistorianAdapterActor.cs`) already subscribes to
+`RedundancyStateChanged` (`:86`, handler `:145`), caches `_localRole` (`:42`), and gates the
+enqueue on `ShouldHistorize()` (`:97`):
+
+```csharp
+private bool ShouldHistorize() => _localRole is not (RedundancyRole.Secondary or RedundancyRole.Detached);
+```
+
+This is **not** `PrimaryGatePolicy` — it ignores driver-member count and defaults to *write* while
+the role is unknown. Its comment (`:68-71`) explains why it exists: DPS fans the Primary's single
+`alerts` publish to **every** node's adapter, so without the gate both nodes of a pair would
+double-write.
+
+**This is why today's ungated drain is safe, and why Phase 2 breaks that.** Today the Secondary
+never enqueues, so its queue is empty and an ungated drain has nothing to send. Once
+`alarm_sf_events` replicates, the Primary's rows land in the Secondary's table — and an ungated
+drain there would double-deliver *every* event, continuously, not just at failover. **The drain
+gate is not a nicety; it is required by the same commit that registers the table.**
+
+### Who owns the role view
+
+`HistorianAdapterActor`, not `DriverHostActor`. Both are spawned under
+`WithOtOpcUaRuntimeActors` (`ServiceCollectionExtensions.cs:353`) which `Program.cs:133` calls only
+under `hasDriver` — as are `AddAlarmHistorian` (`:175`) and `AddOtOpcUaLocalDb` (`:185`). So sink,
+LocalDb and adapter are co-located by construction. The adapter wins over `DriverHostActor`
+because it **already** holds the role for exactly this purpose; using it adds no new subscription
+and no new way for the view to go stale while the sink keeps draining.
+
+`DriverMemberCount` is read off Akka cluster state in `DriverHostActor.DriverMemberCount() :1446`
+(try/catch → 0 on a non-cluster provider ⇒ single-node posture). Task 3 will lift that into a
+small shared helper rather than copy it.
+
+## 5. Other `DatabasePath` construction sites
+
+- `ServiceCollectionExtensions.cs:98` — the only production site.
+- `appsettings.json:57` — `"DatabasePath": "alarm-historian.db"`.
+- `AlarmHistorianOptions.Validate() :54-55` — warns when the path is relative. **This warning
+  disappears with the key**; the validator loses one of its five checks.
+- `SqliteStoreAndForwardSinkTests.cs` — ~30 construction sites, all passing the per-test temp
+  `_dbPath` seeded at `:25` and deleted in `Dispose :32`. These are the tests Task 2 rewires onto
+  a temp-file `ILocalDb`.
+- **`docker-dev/docker-compose.yml` has no `AlarmHistorian` keys at all** — the rig runs with
+  `Enabled=false` today. Task 6 is therefore an *addition*, not an edit.
+
+No tooling or CLI constructs the sink.
+
+## 6. How `Enabled=false` short-circuits
+
+`AddAlarmHistorian` returns before registering anything when the section is absent or
+`Enabled != true` (`ServiceCollectionExtensions.cs:87`), leaving the `NullAlarmHistorianSink`
+seeded by `AddOtOpcUaRuntime :49`. The Null sink is a pure no-op (`IAlarmHistorianSink.cs:37-52`).
+
+Confirming the plan's own answer: **the tables are created unconditionally in `OnReady`** (cheap,
+empty) because `AddOtOpcUaLocalDb` is independent of the `AlarmHistorian` gate. Only the
+sink/drain stay Null. This is also required for correctness — `RegisterReplicated` must run on
+both pair members regardless of their historian config, or a node that later enables the sink
+would write rows that are never captured (see the CDC note below).
+
+## 7. Library constraints re-confirmed (README, `ZB.MOM.WW.LocalDb` 0.1.3)
+
+- **BLOB columns are rejected at registration.** Not a problem here (§1).
+- **CDC semantics** — rows that existed *before* `RegisterReplicated` are neither captured nor
+  snapshotted. This is the mechanical reason the migrator must run **after** registration, and
+  the reason `LocalDbSetup.OnReady`'s ordering comment is load-bearing (`LocalDbSetup.cs:23-31`).
+- **LWW keyed on the PK** — two nodes minting the same PK for *different* rows silently overwrite
+  each other; two nodes minting the same PK for the *same* row converge. Decision D-1 below turns
+  that second property into a feature.
+- `ILocalDb` surface in use: `ExecuteAsync` / `QueryAsync(sql, rowMapper, params, ct)` /
+  `BeginTransactionAsync` / `CreateConnection` / `RegisterReplicated`
+  (`Deployment/LocalDbDeploymentArtifactCache.cs:77,82,205`).
+
+---
+
+## Deviations and decisions
+
+Recorded here rather than raised as questions, per the plan's follow-up convention.
+
+### D-1 — Enqueue ids are a deterministic content hash, not `Guid.NewGuid()`
+
+**The plan says** GUIDs minted at enqueue (Task 2 step 2).
+
+**The problem.** The enqueue gate (`ShouldHistorize`, default-**write** on unknown role) and the
+drain gate the plan specifies (`PrimaryGatePolicy`, default-**deny** on unknown role when paired)
+disagree during the boot window. On a two-node pair before the first redundancy snapshot arrives,
+**both** adapters enqueue the same DPS-fanned transition. With random GUIDs those are two distinct
+rows that replicate into each other's tables — the buffer *duplicates* instead of converging, and
+whichever node becomes Primary later delivers both copies.
+
+**The decision.** Mint `id` as a hash over the serialized payload. Two nodes independently
+enqueuing the same fanned event produce the same PK, so LWW converges them to one row for free.
+Two genuinely distinct events cannot collide: `AlarmHistorianEvent` carries a full-precision
+`TimestampUtc` plus alarm id, kind, message and user, so an identical hash means an identical
+event.
+
+This costs one `SHA256.HashData` per enqueue and removes a whole duplicate-row class. It also
+slightly *improves* today's behaviour, where the same boot-window fan-out is already
+double-delivered because the drain is ungated on both nodes.
+
+`ShouldHistorize` itself is left unchanged — out of scope, and with D-1 in place the two gates
+disagreeing is no longer harmful.
+
+### D-2 — Gate denial must be observable
+
+`PrimaryGatePolicy` defaults to **deny** when the role is unknown on a multi-driver cluster. That
+is the correct safety posture, but it means a redundancy-snapshot identity mismatch — the
+documented 03/S5 shape that `DriverHostActor.cs:1485-1491` already warns about once — would
+silently stop alarm history on *both* nodes, degrading only as retry/eviction growth minutes later.
+
+Task 3 will therefore: log the skip once per gate-state transition (not per tick — the drain runs
+every 5 s), and surface it in `HistorianSinkStatus` so `/healthz` and the AdminUI can see
+"buffering, not draining, because not Primary" rather than an unexplained rising queue depth.
+Adding a `HistorianDrainState` member is additive; Task 3 will check every `switch` over it.
+
+### D-3 — Tasks 2 and 3 will land as one commit
+
+The plan anticipated this. Confirmed necessary: the gate delegate is a constructor parameter of
+the rewritten sink, so the sink cannot compile in its final shape without it, and an intermediate
+commit with a rewired-but-ungated sink would be a commit in which a replicated table is drained by
+both nodes. Not a state worth having in history.
+
+### D-5 — The table keeps the legacy delete-on-ack shape, not a `status` column
+
+**The plan proposes** `status TEXT NOT NULL DEFAULT 'pending'` with values `pending | delivered |
+dead`, and drops the legacy `LastError` column.
+
+**Two problems.** A `delivered` status means acknowledged rows accumulate in the table forever —
+the plan specifies no sweeper for them, and they would also have to be excluded from the capacity
+count, changing a semantic §3 says to preserve. And `LastError` is not dead weight: `DeadLetterRow`
+writes the failure reason into it and `RetryDeadLettered` clears it, so it is the only operator-
+facing record of *why* a row was dead-lettered.
+
+**The decision.** Keep the legacy shape: DELETE on acknowledgement, `dead_lettered` as the same 0/1
+flag, and retain `last_error`. Deleting keeps the table bounded with no second sweeper, and the
+replication engine carries the delete as a tombstone (pruned after `TombstoneRetention`, 7 d), so
+the peer drops its copy too — which is the property the plan wanted the `delivered` status for.
+Columns then map 1:1 onto the legacy table, making the Task-4 migrator a straight copy.
+
+One genuine change: the drain's `ORDER BY` moves from `RowId ASC` to `enqueued_at_utc, id`, because
+a hashed TEXT id carries no insertion order. Round-trip ("O") timestamps sort lexicographically in
+chronological order, and `id` makes the ordering total; the index covers both.
+
+### D-6 — The migrator keys on the payload hash, not `mig-{node}-{legacyId}`
+
+**The plan says** deterministic ids of the form `mig-{NodeName}-{legacyId}`, mirroring ScadaBridge.
+
+**Why that is not enough here.** Node-prefixing does solve the collision the legacy AUTOINCREMENT
+key would otherwise cause — node A's `RowId 7` and node B's `RowId 7` are different alarms — but it
+*preserves* a duplication that ought to be collapsed. A warm pair's two legacy files **overlap**:
+`HistorianAdapterActor` default-writes while its redundancy role is unknown, so both nodes accepted
+the same fanned transitions during every boot window they ever had. Prefixed ids would carry every
+one of those duplicates into the merged buffer permanently, for the eventual Primary to deliver
+twice.
+
+**The decision.** Reuse D-1's `AlarmSfSchema.DeriveId` (lifted out of the sink so both call sites
+share one identity rule). Distinct events cannot collide; identical ones converge. It also removes
+the migrator's only need for node identity from configuration, and makes a crash between commit and
+rename harmless under `INSERT OR IGNORE` — which the plan's own idempotency requirement wanted
+anyway.
+
+### D-7 — The migrator's tests live in `Host.IntegrationTests`
+
+The plan names `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests`. **That project does not exist.** The
+tests sit beside `LocalDbSetupTests` in `Host.IntegrationTests`, which is where every other
+LocalDb-facing test already lives. They are offline — a temp file and no external services.
+
+### D-4 — The live gate needs `MaxAttempts` raised on the rig
+
+Not actionable until Task 8, recorded now so it is not rediscovered there. The rig has no
+HistorianGateway, so every drain attempt fails. With `MaxAttempts: 10` and the backoff ladder
+capping at 60 s, buffered rows dead-letter roughly five minutes into the outage — which would make
+the gate's "dead letters 0" check (step 2) fail for reasons that have nothing to do with
+replication. Task 6 should set a high `MaxAttempts` on the rig, or Task 8 should assert on
+dead-letter *timing* instead.
+
+Related: `Program.cs:170-173` shows an `AlarmHistorian:Enabled=true` /
+`ServerHistorian:Enabled=false` deployment is explicitly supported and warned about — which is
+exactly the rig's shape. Task 6 must confirm `ServerHistorianOptionsValidator` does not fail host
+start for a disabled section with an empty `Endpoint`.
+
+---
+
+## Guard-deletion evidence (recorded at the Task 7 DoD sweep)
+
+Every guard this phase adds was proved by deleting it and watching the tests go red. Two of those
+runs found tests that would have passed over a broken implementation. Both are recorded here because
+the *shape* of the vacuity is reusable, not just the fix.
+
+### Control 1 — remove `IRedundancyRoleView.Publish` from `DriverHostActor`
+
+**3 of 6** `DriverHostActorRoleViewTests` went red. The other three assert
+`ShouldServiceAsPrimary == true`, which is exactly the value the view is **seeded** with — so they
+cannot distinguish "published true" from "never published at all". That split is expected and
+acceptable *given* the three that do go red pin the false cases, which only a real publish can
+produce. Worth knowing before someone reads a future 3/6 result as a regression.
+
+### Control 2 — remove `RegisterReplicated("alarm_sf_events")` from `LocalDbSetup.OnReady`
+
+**3 of 4** `AlarmSfConvergenceTests` went red immediately. The fourth,
+`TheSameEventOnBothNodes_ConvergesToOneRow`, **passed vacuously**: with replication switched off
+each node trivially held its own single row, and "count == 1 on both nodes" was satisfied by two
+completely unconverged databases. The assertion could not tell convergence from isolation.
+
+**Fix:** enqueue a second, *distinct* event on node B and require **both** nodes to hold two rows.
+Isolation now yields 1 and 2; only convergence yields 2 and 2. Re-run under the control: red.
+Control restored: all 4 green.
+
+The general trap — an assertion whose expected value is also what a *disabled* system produces —
+is the same one recorded in the `#485`/`#486` unreadable-artifact defect class.
+
+### Exact-set replicated-table pins
+
+Two, not one, and both assert set equality (ordered `ShouldBe` against the literal three-table list,
+so an added *or* dropped registration fails):
+
+- `LocalDbSetupTests` — drives the production `OnReady` callback directly.
+- `LocalDbWiringTests` — resolves `ILocalDb` from the full driver DI graph, which is what catches a
+  registration that never reaches a running host.
+
+## Test-suite evidence (Task 7 DoD sweep)
+
+Full solution run on the branch, compared against a **full solution run on a detached worktree at
+the pre-branch baseline `2e46d054`**. Comparing failure *sets*, not counts, because the suite has a
+standing set of environment- and load-dependent failures that a raw count would hide.
+
+**The two failure sets are identical — all 13 tests, same names, both runs. Zero new failures.**
+
+| Assembly | Baseline | Branch | Delta |
+|---|---|---|---|
+| `Core.AlarmHistorian.Tests` | 0 failed / 29 passed | 0 failed / **32** passed | **+3** (the drain-gate tests; 29 ported from the old sink's suite) |
+| `Runtime.Tests` | 1 failed / 418 passed | 1 failed / **424** passed | **+6** (`DriverHostActorRoleViewTests`) |
+| `Host.IntegrationTests` | 2 failed / 169 passed | 2 failed / **186** passed | **+17** (migrator + convergence + pin updates) |
+
+The 13 standing failures, and why each is not this branch's:
+
+- **3 × `DriverTypeNamesGuardTests`** — `ArgumentNullException (secretResolver)` out of
+  `GalaxyDriverFactoryExtensions.Register`, reached by reflection. A real pre-existing defect in a
+  guard test, reproduced exactly on the baseline worktree. Unrelated to this phase.
+- **4 × `AbLegacy.IntegrationTests`**, **3 × `OpcUaClient.IntegrationTests`**,
+  **1 × `DriverProbeHandshakeE2eTests.AbCip_Green_AgainstSim`** — driver fixtures on the
+  `10.100.0.35` docker host are not running.
+- **1 × `RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed`**
+  — times out under full-suite load; passes in isolation.
+- **1 × `ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks`** — fails
+  under full-suite load ("writer must have been called at least twice"), and `Runtime.Tests` passes
+  **425/425** when run alone. Also fails on the baseline. Worth noting for anyone reading a future
+  run: this one is *newly observed* here only because the previous sweep's log was truncated, not
+  because the branch introduced it.

docs/plans/2026-07-21-issue-485-live-gate.md

@@ -0,0 +1,76 @@
+# Issue #485 — live gate (docker-dev rig)
+
+> Verifies the two fixes for #485 — `OpcUaPublishActor` (master `c7f5b9cf`) and `DriverHostActor`
+> (master `5aad1e33`) — on the local `docker-dev` rig: 6 nodes, shared SQL, real Modbus fixture on
+> `10.100.0.35:5020`.
+>
+> **Safety rules followed**: all DB inspection + mutation via `sqlcmd` inside the SQL container; the
+> rig's Modbus fixture is read-only for this gate (no writes to the PLC sim).
+
+## Result
+
+**Both fixes verified, all three guards observed firing in production, with a live RED control that
+was not staged.** The gate also confirmed a **pre-existing** defect the fixes do not address — the node
+records `Applied` for a deployment it skipped (see "Finding" below).
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 1 | RED control — the bug in its natural habitat, unforced | ✅ | site-a-1 hit #485 during the *previous* gate's SQL flapping and had been serving an **empty address space for 34 minutes**: `failed to load artifact … rebuild becomes no-op` followed immediately by `applied plan (kind=PureRemove, added=0, removed=17, rebuild=True)`. Its 17 `ns=2` nodes were gone; a browse returned only the root `OtOpcUa` folder. Nothing was staged to produce this — it was already true on the pre-fix image. |
+| 2 | Rebuilt on the fixed image, the node recovers | ✅ | After `docker compose build` + `--force-recreate`, site-a-1 restored to `raw subtree materialised (containers=16, tags=1)` — 18 `ns=2` nodes. |
+| 3 | Address-space guard fires; nodes survive | ✅ | site-a-1, empty artifact: `OpcUaPublish: no usable artifact for deployment f4b9dd63… — KEEPING the currently served address space rather than tearing it down`. **No `PureRemove`.** Browse before vs after: 18 vs 18, `diff` **identical**. |
+| 4 | Reconcile guard fires; drivers survive | ✅ | `artifact for f4b9dd63… read back empty; skipping reconcile and keeping the 1 running driver(s)`, and the apply still reported `children=1`. Repeated on central-2 with a bigger blast radius: `keeping the 5 running driver(s)`, `children=5`. |
+| 5 | SubscribeBulk guard fires; **live values keep flowing** | ✅ | central-2, `artifact for 8c36a673… read back empty; skipping SubscribeBulk and keeping the live subscriptions`. The decisive proof is the value, not the log: `ns=2;s=pymodbus/plc/HR200X` read **56278 @ 06:59:51** before the induction, then **56336 @ 07:00:21** and **56354 @ 07:00:30** after it (induction at 07:00:07). The subscription was still delivering fresh Modbus data across the event. |
+| 6 | Positive control — a readable artifact still applies | ✅ | Immediately after the guards fired, a genuine deploy applied normally on both nodes: site-a-1 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 1 references`; central-2 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 5 references across 4 driver(s)`. The guards suppress the teardown, **not** deployment. |
+| 7 | Control — only the node that saw empty bytes took the guard path | ✅ | Of the six nodes, exactly **one** logged `read back empty` (the paused one); the other five read the real artifact before it was blanked and applied it normally. The induction window is what produced the condition, not a global change. |
+
+## How the failure was induced deterministically
+
+The original incident was a race (a transient ConfigDb blip landing between the apply and the artifact
+read). Racing it again would be unreliable, so the gate drives the **empty-bytes** branch instead — which
+is byte-for-byte what the code sees when the row cannot be read, and is the branch the driver-side guards
+key on (their *throw* paths already returned early before this work):
+
+1. `docker pause` the target node, freezing it mid-cluster.
+2. `POST /api/deployments` — the deployment seals and dispatches while the node cannot process it.
+3. `UPDATE Deployment SET ArtifactBlob = 0x WHERE DeploymentId = <new>`.
+4. `docker unpause` — the node now processes the dispatch and reads an empty artifact.
+
+The pause measured **<1s** on both runs, well inside `acceptable-heartbeat-pause = 10s` and SBR's
+`stable-after = 15s`, so the node was never flagged unreachable and no failover was triggered.
+
+## Finding — the node records `Applied` for a deployment it skipped (pre-existing, NOT fixed here)
+
+Both guards leave the node running its previous configuration, which is the intent. But `ApplyAndAck`
+advances `_currentRevision = revision` and writes `NodeDeploymentState = Applied` **before** it knows
+whether anything was applied — that assignment sits immediately after `ReconcileDrivers`, which returns
+`null` on both the (pre-existing) throw path and the new empty path:
+
+```
+[07:00:07 WRN] DriverHost central-2:4053: artifact … read back empty; skipping reconcile …
+[07:00:07 INF] DriverHost central-2:4053: applied deployment 8c36a673… (rev 03b33904…, children=5)
+```
+
+`NodeDeploymentState` for that deployment reads `Status = Applied` for central-2. The node now claims a
+revision whose configuration it never applied — and because `HandleDispatchFromSteady` short-circuits on
+a revision match (`dispatch matches current rev; immediate ACK`), **re-dispatching that same revision is
+a no-op**, so the node cannot self-heal. Only a later deployment with a *different* revision recovers it
+(check 6 confirms that path works).
+
+This is pre-existing — the throw path has always done it — and is strictly a reporting/convergence bug,
+not a data-plane one: the node keeps serving its last good configuration throughout. The honest fix is to
+ACK `Failed` and leave `_currentRevision` alone when the artifact could not be read, so the normal deploy
+retry heals it. Tracked separately rather than folded in here, because it changes deploy ACK semantics
+fleet-wide.
+
+## Addendum — issue #486 fix, live-gated (2026-07-21)
+
+The finding above is now fixed (`DriverHostActor.ApplyAndAck` fails the apply when `ReconcileDrivers`
+returns null) and gated on the same rig with the same induction.
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 8 | Regression — a normal deploy still ACKs Applied fleet-wide | ✅ | The change alters ACK semantics, so this is the risk that matters. A genuine deploy recorded `Status = Applied` on **all six** nodes. |
+| 9 | An unreadable artifact now ACKs Failed, with a reason | ✅ | central-2 recorded `Status = Failed` + `the deployment artifact could not be read (ConfigDb unreachable, or the row is missing/empty); nothing was applied`. The five nodes that read the real artifact recorded Applied — so the fleet view now distinguishes them. Pre-fix, all six claimed Applied. |
+| 10 | The revision is NOT advanced | ✅ | `apply of a855dd9e… FAILED — … Staying on revision 5bc0d9f3…` — the **previous** revision, not the dispatched `583122b7…`. This is what stops `HandleDispatchFromSteady` from short-circuiting a retry of that revision. |
+| 11 | The node keeps serving throughout (#485 still holds) | ✅ | The #485 guard fired first (`keeping the 5 running driver(s)`), and `ns=2;s=pymodbus/plc/HR200X` read **59603 @ 07:27:28**, live and current, after the Failed apply. Failing the apply reports the truth; it does not tear anything down. |
+| 12 | The node recovers on the next real deploy | ✅ | The following deployment applied on central-2 (`children=5`) and recorded Applied on all six nodes. |

docs/plans/2026-07-21-localdb-followups-live-gate.md

@@ -0,0 +1,86 @@
+# LocalDb Phase 1 follow-ups — live gate (docker-dev rig)
+
+> Closes the two limitations left open by `2026-07-20-localdb-phase1-live-gate.md` (checks 4 and 8).
+> Rig: local `docker-dev/docker-compose.yml`, 6 nodes rebuilt from `fix/localdb-phase1-followups`.
+> site-a pair replicates; central + site-b stay default-OFF.
+>
+> **Safety rules followed** (same as the Phase 1 gate): all DB inspection via `docker cp` of the
+> `db`/`-wal`/`-shm` triplet, querying the copy — never host `sqlite3` on the live WAL file.
+
+## Result
+
+**Both follow-ups closed, and the gate found a third defect** — one that was *unreachable* until the
+first fix landed. Library `ZB.MOM.WW.LocalDb` went `0.1.1` → `0.1.2` → `0.1.3` in the course of this
+gate; both consumers (OtOpcUa, ScadaBridge) are pinned to `0.1.3`.
+
+| # | Check | Result | Evidence |
+|---|---|---|---|
+| 8 | Oplog stops growing on a default-OFF node | ✅ PASS | site-b-1 restarted twice with an unchanged cached artifact: `oplog_depth` **7 → 7 → 7**, `deployment_pointer.applied_at_utc` frozen at `05:34:58` across both — the re-cache was skipped outright. Repeated on the `0.1.3` build after a real deploy: **13 → 13**. Pre-fix this grew ~2 rows per restart forever, because a default-OFF node has no peer to ack and prune them. |
+| 8b | …but a genuinely new deployment still writes (positive control) | ✅ PASS | A real config change (renamed tag `SITEA-tag`) produced rev `822bdf34`, then `002c16b3`. site-b-1's oplog **7 → 10 → 13** and its pointer advanced each time. The skip suppresses *identical* re-writes only; it is not a dead cache-write path. |
+| 4 | A wiped node is back-filled by its peer, no new deploy, central down | ✅ PASS (was ⚠️ PARTIAL) | Precondition asserted first — site-a-1's oplog **0**, `needs_snapshot 0`: the exact state that used to make back-fill impossible. Then SQL stopped, site-a-2's LocalDb **volume destroyed**, container recreated. a-1 logged `Snapshot sent (as_of_seq 0, 4 rows)` — a line that could not appear on `0.1.1`. a-2 came back with pointer `SITE-A / 475df87a @ 03:09:12` (a-1's **original** timestamp, not a re-derived one), chunks 2, and a `__localdb_row_version` dump **byte-identical to a-1's**, carrying origin node ids `6812ca9c` and `935659a5` — the rebuilt node's own id appears nowhere, so the rows were replicated, not recreated. |
+| 4b | …and the back-filled node then boots from cache | ✅ PASS | With SQL still down, site-a-2 logged `RUNNING FROM CACHE — … booted deployment 475df87a… cached at 2026-07-21T03:09:12`, `raw subtree materialised (containers=16, tags=1)`. OPC UA browse of both nodes: **17 ns=2 nodes each, diff-identical** — the rebuilt node serves exactly what its healthy peer does, from a configuration it never applied itself and could not have fetched. |
+| 4c | The rebuilt node's OWN writes reach its peer | ✅ PASS **after a third fix** (see below) | On `0.1.2` this failed: a-1 held `last_applied_remote = 10` while the rebuilt a-2's oplog was seqs **1,2,3** and its `last_acked` stayed **0** — never accepted. On `0.1.3`, a-1 logs `Replication peer identity changed (ebb10854 -> c85d8b1d): the peer was rebuilt … Resetting the inbound watermark`, and after the next deploy a-1 shows `last_applied_remote = 3` / a-2 shows `last_acked = 3` with a drained oplog and identical row_versions. |
+
+## The third defect (the reason this gate was worth running)
+
+`0.1.2` made back-fill work, which for the first time let a rebuilt node be observed *writing*. It
+turned out its writes went nowhere.
+
+`last_applied_remote_seq` is a watermark in the **peer's** seq space, and a rebuilt peer is a new
+database whose oplog numbers from 1. Two independent bugs stacked, and fixing either alone leaves the
+data stuck:
+
+1. The healthy node kept advertising the **old** peer's watermark. The *sending* side seeds its pump
+   directly from that number (`_sentThruSeq`), so the rebuilt node skipped its entire oplog.
+   → the watermark (and the observed peer clock) is now reset when the peer's node id changes.
+   `last_acked_seq` is deliberately **not** reset: it describes our own oplog's pruned horizon and is
+   exactly what `0.1.2` keys the snapshot decision off — clearing it would switch back-fill back off.
+2. A peer claiming to have applied more of our stream than we have ever produced can only be
+   remembering a previous incarnation of *us*. → that claim is now rejected in favour of pumping from
+   the start; LWW makes the re-send harmless. This mirrors the clamp `RecordPeerAckAsync` already
+   applies to acks.
+
+Both were reproduced offline as a RED test before fixing
+(`SnapshotResyncTests.RebuiltPeer_OwnWritesReachTheHealthyNode_DespiteTheOldPeersWatermark`).
+
+**Why offline tests could not have found it first:** the wiped-peer scenario had no test because the
+scenario was believed impossible to recover from — it was a *documented limitation*, not a bug. The
+gate is what turned the limitation into a reproducible case, and the second defect only existed
+behind the first.
+
+## Unrelated finding (NOT a follow-up of this work — pre-existing; `lmxopcua` issue #485, since FIXED)
+
+While SQL was being stopped and started to drive check 4, site-b-1 took a deploy whose artifact load
+hit a transient ConfigDb error, and **emptied its served address space**:
+
+```
+[06:11:20 ERR] An error occurred using the connection to database 'OtOpcUa' on server 'sql,1433'.
+[06:11:20 WRN] OpcUaPublish: failed to load artifact for deployment 43d54397…; rebuild becomes no-op
+[06:11:20 INF] AddressSpaceApplier: applied plan (kind=PureRemove, added=0, removed=16, rebuild=True)
+```
+
+The log says "rebuild becomes no-op" and the applier then removed all 16 nodes — those two disagree.
+The cache layer behaved correctly and refused to store the bad artifact ("not caching deployment … its
+artifact was empty or could not be loaded, so it is not a usable last-known-good configuration"), and
+the node recovered fully on restart (`containers=16`). But a transient ConfigDb blip emptying the
+served address space instead of holding last-known-good is the same failure *class* as the Phase 1
+gate's check-3 defect. Untouched by this work — site-b-1 has replication off entirely — and induced by
+this gate's own SQL flapping.
+
+**Fixed** (issue #485): `OpcUaPublishActor.HandleRebuild` now treats an artifact it could not obtain as
+*no answer* rather than *an empty configuration*, and abandons the rebuild with the served address space
+(and `_lastApplied`) intact. Nothing legitimate produces a zero-length blob — an operator who really
+deletes everything still deploys a JSON document with empty arrays — so "no bytes" can only mean the read
+failed. The log excerpt above is now impossible: the `PureRemove` never runs, and the loader's own
+"the rebuild is abandoned" line is finally true. Covered by
+`OpcUaPublishActorRebuildTests.Rebuild_keeps_the_last_known_good_address_space_when_the_artifact_load_fails`,
+with a positive control proving a *readable* artifact that genuinely drops an equipment still tears its
+subtree down.
+
+The **driver-side half** of the same mistake was fixed alongside it: `DriverHostActor.ReconcileDrivers`
+and `PushDesiredSubscriptionsFromArtifact` read the artifact the same way, so empty bytes meant zero
+driver specs — `DriverSpawnPlanner` planned every running child for `StopChild` — and then an empty
+desired set dropped each surviving driver's live subscription handle. A node lost its entire field I/O
+and still ACKed Applied. Both now skip on empty. Covered by `DriverHostActorUnreadableArtifactTests`,
+whose absence assertion is calibrated by a control that observes the very same unsubscribe *arriving*
+inside the settle window (without it the assertion passed on a race).

docs/plans/2026-07-21-per-cluster-mesh-design.md

@@ -0,0 +1,337 @@
+# Per-Cluster Akka Mesh — design
+
+> **Status:** design for review. **Phases 0a and 0b are DONE and merged** (2026-07-21); Phases 1–7
+> are not started and each needs its own plan.
+> **Decisions settled 2026-07-21** — see §6: one central DB with driver nodes disconnected from it,
+> the two-node keep-oldest gap closed upstream and now closed here too (§6.2), and their auth posture
+> matched for now.
+
+## 1. What this changes
+
+Today OtOpcUa runs **one Akka mesh containing every node of every application Cluster**. That was a
+deliberate choice — `docs/plans/2026-06-07-per-cluster-scoping.md`, "Per-ClusterId Scoping
+(hub-and-spoke **single mesh**)" — so the deploy channel could stay in-mesh: AdminUI →
+`admin-operations` singleton → `deployments` topic → every node filters to its own `ClusterId`.
+
+The proposal is to move to the sister project's shape: **one Akka mesh per application Cluster,
+two nodes maximum**, with central and clusters joined by explicit transports instead of cluster
+gossip.
+
+The motivation is that redundancy becomes correct *by construction*. Every Primary-gated decision
+(inbound device writes, native alarm acks, fleet alerts, ServiceLevel, and — since Phase 2 — the
+alarm-history drain) currently asks "am I the driver Primary?" of a **cluster-wide** election, while
+every resource being gated is **pair-local**. Splitting the mesh collapses the two scopes into one
+and the question stops being ambiguous.
+
+## 2. What ScadaBridge actually does
+
+Researched directly from `/Users/dohertj2/Desktop/ScadaBridge`. Five things differ from the
+summary in its own CLAUDE.md, and they matter:
+
+**Three transports cross the boundary, not two.**
+
+| Transport | Direction | Carries |
+|---|---|---|
+| Akka **ClusterClient** | central → site (+ replies) | command/control: deploy notify, lifecycle, queries, subscribe handshakes |
+| **gRPC** server-streaming | **central dials into the site** | real-time attribute + alarm events |
+| Plain **HTTP**, token-gated | site → central | the deployment config itself (notify-and-fetch) |
+
+**The gRPC direction is inverted from the obvious guess.** Data flows site→central, but *each site
+node hosts the gRPC server* (Kestrel h2c :8083) and *central is the client*. There is no gRPC server
+on central at all.
+
+**Active node = OLDEST Up member, explicitly not the cluster leader.**
+`ActiveNodeEvaluator.SelfIsOldestUp` is documented as *"THE single definition of 'active node'"*:
+
+> *"Cluster LEADERSHIP (lowest address) is an Akka-internal concept that diverges from singleton
+> placement permanently once the original first node restarts and rejoins; every product-level
+> active/standby decision must use this evaluator, never `cluster.State.Leader`."*
+
+The equivalence **oldest-Up == where `ClusterSingletonManager` places singletons** *is* the design.
+
+**All clusters share one ActorSystem name** (`"scadabridge"`, hardcoded). They are separate clusters
+only by seed-node partitioning — required, because Akka.Remote address matching means ClusterClient
+could not reach a differently-named system.
+
+**Site nodes carry two roles:** `"Site"` and `"site-{SiteId}"`. Singletons scope to the
+**site-specific** role.
+
+Other load-bearing details:
+
+- **Central discovers sites from the database** (a `Site` entity with `NodeAAddress`/`NodeBAddress`
+  + `GrpcNodeAAddress`/`GrpcNodeBAddress`), refreshed every 60 s and on admin change. Sites discover
+  central from **appsettings** — static, restart required. The asymmetry is deliberate.
+- **Exactly one actor is exposed per side** via `ClusterClientReceptionist.RegisterService` —
+  `/user/central-communication` and `/user/site-communication` — registered **per node, not as a
+  singleton**, so contact rotation reaches whichever node answers.
+- **No central buffering when a site is unreachable.** The send is dropped with a warning and the
+  caller's Ask times out. *"It keeps the central coordinator stateless with respect to site
+  availability."* A `ConnectionStateChanged` mechanism was built for this and **deleted as dead
+  code**.
+- **Every unhandled message replies with a typed failure rather than dropping**, so a central Ask
+  can never stall on a missing handler.
+- **Sender preservation is the whole Ask idiom**: `_client.Tell(new ClusterClient.Send(path, msg),
+  Sender)` routes the reply straight back to the waiting Ask, bypassing the comm actor.
+
+### The registered outage gap — read before copying
+
+> `Component-ClusterInfrastructure.md:113`: *"Only the FIRST seed listed in `Cluster:SeedNodes` may
+> self-join to form a *new* cluster… a lone restarted non-first-seed node (with the first seed still
+> down) loops on `InitJoin` forever — never `Up`, never routable. This is why the two-node
+> keep-oldest **oldest/active-node crash is a total-outage gap**: after the oldest dies the younger
+> survivor self-downs, and it cannot re-bootstrap alone. Recovery is operator-driven."*
+
+Their failover drill has a mode that exists *"to make the registered gap observable — not to pretend
+it is covered."* **A two-node keep-oldest mesh has an acknowledged total-outage hole.** If OtOpcUa
+adopts 2-node meshes, this must be decided deliberately, not inherited.
+
+> **CLOSED 2026-07-22.** The `InitJoin` half of this gap is fixed in both repos by **self-first seed
+> ordering**: every node that is one of its own seeds lists ITSELF as `seed-nodes[0]`, which is the
+> only ordering under which Akka runs `FirstSeedNodeProcess` and can form a cluster with no peer
+> answering. (An earlier `Cluster:SelfFormAfter` watchdog was tried here and retired — it raced the
+> join handshake.) See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Today's
+> driver-only site nodes are seeded by `central-1` only and are exempt; the per-cluster mesh removes
+> even that exception by making every pair node a seed of its own mesh.
+
+### Both inter-cluster transports are unauthenticated
+
+Akka remoting has no TLS, no secure cookie, no `trusted-selection-paths`. The gRPC listener is h2c
+with no auth covering `SiteStreamService` — including two Pull RPCs that return audit rows. The only
+authenticated pieces are the HTTP fetch token and `LocalDbSyncAuthInterceptor`, which gates *only*
+`/localdb_sync.v1.LocalDbSync/`. Their boundary assumes a trusted network.
+
+**OtOpcUa already ships that same fail-closed interceptor** (LocalDb Phase 1) — it is a ready-made
+template if we choose to authenticate more than they did.
+
+## 3. What OtOpcUa has today
+
+**Nine cross-node DPS topics plus a singleton**, all currently relying on one mesh:
+
+| Channel | Direction | Purpose |
+|---|---|---|
+| `deployments` / `deployment-acks` | central ↔ nodes | deploy dispatch + acks |
+| `driver-control` | central → nodes | AdminUI Reconnect/Restart |
+| `redundancy-state` | central → nodes | roles + peer probe results |
+| `alerts` | nodes → central | fleet alarms → `/alerts` |
+| `driver-health`, `driver-resilience-status`, `fleet-status`, `script-logs` | nodes → central | AdminUI live panels |
+| `admin-operations` singleton | central | operator ack/shelve into engines |
+
+Three facts make the split cheaper than that table suggests:
+
+1. **The deploy path is already notify-and-fetch.** `DispatchDeployment` carries only
+   `DeploymentId` + `RevisionHash` + `CorrelationId`; the node fetches the artifact itself. We
+   independently arrived at the pattern ScadaBridge had to retrofit, and we do **not** carry its
+   128 KB Akka frame exposure on this path.
+2. **Deploy state already has a DB substrate.** `ConfigPublishCoordinator` persists per-node ACKs to
+   `NodeDeploymentState` so a singleton failover recovers in-flight state from the DB.
+3. **`ClusterNode` rows already enumerate every node per application Cluster** — the same table
+   `AdminOperationsActor` groups by cluster. This replaces the coordinator's one genuinely
+   mesh-bound dependency: it derives its expected-ack set from `Akka.Cluster.State.Members` filtered
+   by role.
+
+So the deploy channel needs only a small notify + ack transport. **The nine live-telemetry channels
+are the real work**, and they are exactly what ScadaBridge built ClusterClient + gRPC for.
+
+## 4. A defect to fix regardless of this design
+
+OtOpcUa currently uses **two different node-selection rules at once**:
+
+- Split-brain resolution is **keep-oldest** (`Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49`).
+- `ClusterSingletonManager` places singletons on the **oldest** member.
+- But `RedundancyStateActor.BuildSnapshot` elects Primary from **`RoleLeader("driver")`** — lowest
+  address among driver members.
+
+Those select the same node only by coincidence. After a restart-and-rejoin the role leader and the
+oldest member diverge permanently, which means the node the SBR protects and hosts every singleton
+on need not be the node the data-plane gates consider Primary. This is precisely the rule ScadaBridge
+forbids by name, and for the reason it gives: *"both sides claim leadership during a partition"* —
+the dual-primary shape archreview 03/S4 exists to prevent.
+
+**Switching the role derivation to oldest-Up-with-role is small, is correct under either topology,
+and should not wait for this design.** It also happens to be exactly what the separate-mesh model
+needs, so it is not throwaway work.
+
+**FIXED 2026-07-21 (Phase 0b).** `RedundancyStateActor.BuildSnapshot` selects the oldest Up member
+carrying the `driver` role, matching singleton placement; `NodeRedundancyState.IsRoleLeaderForDriver`
+was renamed `IsDriverPrimary` so the contract stops asserting the old derivation. Pinned by
+`RedundancyPrimaryElectionTests`, which builds a real two-node cluster in which the oldest node holds
+the *higher* address, so the two orderings genuinely disagree.
+
+## 5. Target architecture
+
+Mirror ScadaBridge's shape, adapted to OtOpcUa's existing substrate:
+
+- **One Akka mesh per application `Cluster`, two nodes max.** Same ActorSystem name (`otopcua`)
+  everywhere; separation by seed-node partitioning only.
+- **Roles per node:** `driver` plus a cluster-specific `cluster-{ClusterId}`, with singletons scoped
+  to the cluster-specific role.
+- **Active node = oldest Up member with role** (§4), one definition, used by the data-plane gates,
+  ServiceLevel, and the alarm drain alike.
+- **Central ↔ cluster command/control over ClusterClient**, with exactly one receptionist-registered
+  actor per side and central discovering cluster node addresses from `ClusterNode` rows (extended
+  with Akka + gRPC addresses, mirroring their `Site` entity).
+- **Deploy stays notify-and-fetch, but the fetch changes source.** The notify crosses via
+  ClusterClient; the artifact then comes **from central**, not from the ConfigDb the driver node no
+  longer connects to (§6.1), and is cached in LocalDb. The coordinator's expected-ack set comes from
+  `ClusterNode` rows instead of cluster membership.
+- **Driver nodes hold no ConfigDb connection.** LocalDb becomes their steady-state configuration
+  store; five current DB consumers are re-homed or re-sourced per §6.1.
+- **Live telemetry over gRPC**, central dialling into each cluster node, replacing the seven
+  observability topics with one stream contract carrying a `oneof` event — additive-only field
+  evolution, contract-locked by test.
+
+**Copied:** their node-local configuration store fed by fetch-and-cache (§6.1), their oldest-Up
+active-node rule (§4), their two-node keep-oldest posture including its acknowledged outage gap
+(§6.2), and their unauthenticated inter-cluster transports (§6.3).
+
+**Deliberately not copied:** their *transient-only* central alarm cache. OtOpcUa persists alarm
+history centrally through the historian, and Phase 2's replicated store-and-forward buffer already
+guarantees delivery across a failover — so there is no reason to adopt a design whose stated
+trade-off is that a new active node re-seeds alarm state from scratch.
+
+## 6. Decisions (settled 2026-07-21)
+
+### 6.1 DECIDED — one central DB, and driver nodes never connect to it
+
+**Verified in ScadaBridge before adopting.** `Program.cs` calls
+`AddConfigurationDatabase(configDbConnectionString)` at line 264 — inside the Central branch
+(89–478). The Site branch begins at line 479. **Site nodes never register the central configuration
+database at all**; they receive config from central (notify → token-gated HTTP fetch) and persist it
+locally. There is still exactly one authoritative configuration database — it is simply
+central-only.
+
+OtOpcUa adopts the same shape: **the ConfigDb stays the single source of truth, and driver nodes stop
+connecting to it.** They obtain their configuration from the central nodes and cache it locally in
+LocalDb.
+
+**This promotes LocalDb from an outage cache to the driver node's steady-state configuration store.**
+That is a reframing of the Phase 1 work rather than a contradiction of it: boot-from-cache stops
+being the exceptional path and becomes the normal one, and "central SQL unreachable" stops being a
+degraded mode for driver nodes because they never held a connection to lose. The chunking,
+SHA-256 verification, newest-2 retention and pair replication all carry over unchanged.
+
+**Scope — five driver-side ConfigDb consumers must be re-homed or re-sourced.** Audited:
+
+| Consumer | Current use | Disposition |
+|---|---|---|
+| `DriverHostActor` | artifact fetch + ack writes (5 `CreateDbContext` sites) | artifact via fetch-and-cache; acks via the ack transport |
+| `EfAlarmConditionStateStore` | Part 9 alarm condition state | **move to LocalDb** — it is pair-local state, the same journey Phase 2 made for the alarm S&F buffer |
+| `DbHealthProbeActor` | DB reachability feeding ServiceLevel tiering | **semantics change** — a driver node has no DB to probe; decide what health input replaces it |
+| `OpcUaPublishActor` | (audit required) | TBD |
+| `ServiceCollectionExtensions` | registration | follows the above |
+
+`DbHealthProbeActor` deserves particular care: DB health is currently a ServiceLevel input, and
+under this model it stops existing for driver nodes. That is a behaviour change on a client-visible
+value, not just an internal refactor.
+
+### 6.2 CORRECTED — the gap was closed in ScadaBridge, and OtOpcUa still has it
+
+**An earlier revision of this section was wrong.** It reported the two-node keep-oldest outage gap as
+still open, citing `Component-ClusterInfrastructure.md:113` and Gitea PR
+[#12](https://gitea.dohertylan.com/dohertj2/ScadaBridge/pulls/12) (which made the failover drill
+*honest* about the gap rather than closing it). That doc line was stale: ScadaBridge closed the gap
+in `cf3bd52f`, *"feat(cluster): auto-down downing strategy — either-node crash now fails over (owner
+decision 2026-07-21: availability over partition-safety)"*.
+
+Its commit message carries the analysis, live-proven on their rig:
+
+> *"Two-node keep-oldest could NEVER survive a crash of the oldest/active node: Akka.NET 1.5.62
+> `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side with >= 2 members, so the 1-vs-1
+> survivor takes `DownReachable` and downs ITSELF — proven live on the rig ('SBR took decision …
+> including myself') before this change. `static-quorum(1)` is worse (`IsTooManyMembers` -> `DownAll`);
+> `keep-majority` just re-keys the fatal crash to the lowest address."*
+
+**OtOpcUa is running that exact configuration today** —
+`Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49-53`: `active-strategy = "keep-oldest"`,
+`keep-oldest { down-if-alone = on }`, `stable-after = 15s`, with
+`BuildClusterOptions` setting the matching typed `KeepOldestOption { DownIfAlone = true }`.
+
+So a two-node OtOpcUa pair **could not fail over when the oldest node crashed**: the survivor
+downed itself. That was a total-outage defect in the redundancy feature itself, independent of this
+design.
+
+**FIXED 2026-07-21 (Phase 0a).** `Cluster:SplitBrainResolverStrategy` now selects the downing
+provider and defaults to `auto-down` — Akka's `AutoDowning` with a 15s window — so the leader among
+the reachable members downs the unreachable peer and a crash of either node fails over. `keep-oldest`
+remains selectable for deployments that prefer partition-safety. Pinned by
+`SplitBrainResolverActivationTests`, which asserts the provider read back off a running ActorSystem
+rather than the configured one. See `docs/Redundancy.md`.
+
+**Its live gate is still outstanding**, and cannot run on today's rig: the failure only appears in a
+1-vs-1 split, and `docker-dev` is a single six-node mesh. The crash-the-oldest drill therefore lands
+with Phase 6/7, which makes every mesh exactly two nodes.
+
+**The bootstrap half of the gap is also closed (2026-07-22).** Downing was only ever one of the two
+outage modes: even with `auto-down`, a node cold-starting while its peer was dead looped on `InitJoin`
+forever, because Akka lets only `seed-nodes[0]` form a new cluster. The fix is **self-first seed
+ordering** — each seed node lists itself first, enforced at boot by `AkkaClusterOptionsValidator`.
+The `Cluster:SelfFormAfter` watchdog first shipped for this is deleted: a timer outside Akka's join
+handshake cannot tell "no seed answered" from "a seed answered and the join is in flight", and it
+islanded a manually-failed-over node live. Pinned by `SelfFirstSeedBootstrapTests`; see
+`docs/Redundancy.md` §"Bootstrap: self-first seed ordering". Under this design each pair node is a
+seed of its own two-node mesh, so the ordering covers both sides and the current site-node exemption
+disappears.
+
+### 6.3 DECIDED — match ScadaBridge's auth posture for now
+
+Inter-cluster transports stay unauthenticated and unencrypted, as theirs are: the boundary assumes a
+trusted network. Recorded as an accepted risk rather than an oversight, with two notes for whenever
+it is revisited: `LocalDbSyncAuthInterceptor` is already in our tree as a fail-closed template, and
+the surface most worth gating first is any Pull-style RPC that returns historical rows.
+
+## 7. Sequencing sketch
+
+Deliberately not a task plan — per-phase plans follow, one at a time.
+
+| Phase | Content | Independent of the split? |
+|---|---|---|
+| 0a | Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2) | **Yes** — **DONE 2026-07-21**, live gate deferred to Phase 7 |
+| 0b | Oldest-Up role derivation (§4) | **Yes** — **DONE 2026-07-21** |
+| 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | **Yes** — **DONE 2026-07-22** (see below) |
+| 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No |
+| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |
+| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No |
+| 5 | gRPC stream contract; migrate the seven observability topics | No |
+| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No |
+| 7 | Failover drill (both directions, per §6.2); live gate | No |
+
+Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each
+deserves its own live gate.
+
+### Phase 1 as shipped (2026-07-22)
+
+Plan: `2026-07-21-per-cluster-mesh-phase1.md`. Two deviations from the sketch above, both decided
+before implementing:
+
+- **No cluster-scope filtering of the expected-ack set.** The plan called for it, but there is no
+  cluster-scoped deployment to filter on: `Deployment` has no `ClusterId`,
+  `ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and
+  `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The
+  expected set is every enabled `ClusterNode` row, which is what the membership rule produced too.
+- **No per-node role column.** The membership rule filtered on the `driver` role and the DB has none.
+  Rather than add one — a second declaration of node roles, free to drift from `Cluster:Roles` — every
+  `ClusterNode` row is now *defined* to be a driver node. An admin-only node must not be given one.
+
+Also shipped beyond the sketch: `ClusterNodeAddressReconciler` (Task 4), an admin singleton that
+catches `AkkaPort` drifting from the node's own `Cluster:Port`. **Phase 2 must revisit it** — once the
+meshes split, an admin node cannot see site members and every site row would report
+`EnabledRowNotInCluster` forever.
+
+## 8. Risks
+
+- **LocalDb becomes load-bearing for configuration, not just resilient.** Phase 1 built it as a
+  fallback; §6.1 makes it the driver node's only config store. A cache defect that was previously a
+  degraded-mode bug becomes a total-availability bug — the #485 unreadable-artifact class, with the
+  blast radius raised.
+- **`DbHealthProbeActor` feeds a client-visible value.** Removing the driver-side DB connection
+  changes what ServiceLevel means on those nodes; that is an interop-visible change, not internal.
+- **The rig currently models the topology that would be abolished.** Phase 6 rewrites
+  `docker-dev/docker-compose.yml` substantially, and every live gate that depends on it.
+- **Losing gossip loses free fleet-wide observability.** Seven AdminUI panels are fed by DPS today;
+  each needs an explicit stream and a reconnect story.
+- **Akka frame size.** We are clean on the deploy path, but anything new that carries payload over
+  ClusterClient inherits the 128 KB default with `log-frame-size-exceeding` off — a silent
+  single-message drop that leaves the association healthy. If we send payload at all, set both.
+- **Do not stack application-level last-write-wins on LocalDb's HLC.** Their `SiteStorageService`
+  carries this warning in code. Verified: our `deployment_pointer` upsert is unconditional and the
+  alarm sink is idempotent on a payload hash, so we are currently clean — keep it that way.

docs/plans/2026-07-21-per-cluster-mesh-phase1.md

@@ -0,0 +1,222 @@
+# Per-Cluster Mesh — Phase 1 plan
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+
+**Parent design:** [`2026-07-21-per-cluster-mesh-design.md`](2026-07-21-per-cluster-mesh-design.md).
+Phases 0a and 0b are done and merged; this is the next phase.
+
+**Goal:** Give `ClusterNode` the transport addresses central will need to reach each cluster
+directly, and cut `ConfigPublishCoordinator`'s last dependency on shared cluster membership by
+sourcing its expected-ack set from those rows.
+
+**Architecture:** Additive schema change plus one substitution inside the coordinator. Nothing else
+moves; the deploy channel stays on DistributedPubSub until Phase 2.
+
+**Status: DONE 2026-07-22** (branch `feat/mesh-phase1`). Live gate:
+[`2026-07-22-mesh-phase1-live-gate.md`](2026-07-22-mesh-phase1-live-gate.md) — **PASSED**, after gate
+step 4 found that the escape hatch this phase depends on did not exist. Three deviations from the
+plan below, all decided before implementing or forced by the gate:
+
+1. **Task 3's cluster-scope filtering was dropped** — there is no cluster-scoped deployment to filter
+   on. See the gate doc, step 2.
+2. **No per-node role column.** Every `ClusterNode` row is now *defined* to be a driver node.
+3. **Task 7 added:** `ClusterNode.MaintenanceMode`. `Enabled = 0` is rejected by
+   `DraftValidator.ValidateClusterTopology` on any 2-node pair, so it could never be the maintenance
+   hatch — which the plan called "the check that makes step 3 acceptable to ship".
+
+---
+
+## Why this is smaller than the design implies
+
+The design says "`ClusterNode` gains Akka + gRPC address columns; coordinator sources its
+expected-ack set from the DB", which reads as though the two are one change. They are not, and the
+second does not depend on the first.
+
+`ConfigPublishCoordinator.DiscoverDriverNodes` builds `NodeId.Parse($"{host}:{port}")` from
+`Cluster.State.Members`. `ClusterNode.NodeId` is **already in that identifier space** — the rig seeds
+`central-1:4053`, `site-a-1:4053` and so on (`docker-dev/seed/seed-clusters.sql:65`), and its comment
+says so explicitly, because `NodeDeploymentState.NodeId` is FK-bound to `ClusterNode.NodeId` and a
+mismatch throws FK 547 on deploy. The coordinator already writes `NodeDeploymentState` rows keyed by
+the membership-derived id and they already satisfy that FK — which is standing proof the two sets
+agree today.
+
+So the coordinator switch is a substitution inside one private method, and the address columns are
+purely forward-looking groundwork for Phase 2's ClusterClient and Phase 5's gRPC dial.
+
+## The decision this phase forces
+
+**Membership-derived and DB-derived expected-ack sets are not the same set, and the difference is a
+behaviour change on the deploy path.**
+
+| Situation | Today (membership) | After (DB rows) |
+|---|---|---|
+| Configured node is running | expected, acks, seals | same |
+| Configured node is **stopped** | not expected — deploy seals **successfully** without it | expected — deploy waits, then **fails at the apply deadline** |
+| Node running but not in `ClusterNode` | expected (and then FK 547 on the state row) | not expected; its ack is discarded |
+
+The middle row is the one that matters. Today a deployment can seal green while a node that should
+have received it was down and did not — the operator is told the fleet is deployed when it is not.
+Failing instead is arguably the correct behaviour and is the reason the design moves this to the DB
+(central cannot see cluster membership once the meshes split).
+
+But it is a behaviour change, and it needs an operator escape hatch or a node taken down for
+maintenance blocks every deployment. `ClusterNode.Enabled` already exists and is exactly that hatch,
+so the query filters on it: **`Enabled = 1` rows are expected to ack; disabled rows are not.**
+
+**Confirmed 2026-07-22 before executing Task 3** — DB-derived with the hatch. (The hatch turned out to be `MaintenanceMode`, not `Enabled`; see Task 7.) If the preference is to keep sealing green past a down
+node, the alternative is to intersect the DB set with current membership, which preserves today's
+behaviour but does not survive the mesh split — it would have to be revisited in Phase 2 anyway.
+
+---
+
+## Task 1: Address columns on `ClusterNode`
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** none (Task 2 migrates what this declares)
+
+**Files:**
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/ClusterNode.cs`
+- Modify: the `ClusterNode` entity configuration in `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/` (find the `IEntityTypeConfiguration<ClusterNode>` or the `OnModelCreating` block)
+- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Configuration.Tests/` (mirror an existing entity-mapping test)
+
+Add two nullable columns beside the existing `OpcUaPort` / `DashboardPort`:
+
+- `int AkkaPort` — default `4053`. The remoting port central will use to build the ClusterClient
+  contact point. Non-nullable with a default, because every node has one.
+- `int? GrpcPort` — nullable, no default. The Phase 5 telemetry-stream port. Nullable is deliberate:
+  it does not exist yet, and a non-null default would assert a port nothing listens on.
+
+Document on each property that these are **central's dial targets**, not the node's own binding
+configuration — the node binds from `Cluster:Port` in its own appsettings, and these columns
+duplicate that value for a reader that cannot see the node's config. That duplication is the reason
+Task 4 exists.
+
+**Step 1:** Write a mapping test asserting a round-tripped `ClusterNode` preserves both values and
+that `AkkaPort` defaults to 4053.
+**Step 2:** Run it; expect failure to compile.
+**Step 3:** Add the properties + mapping.
+**Step 4:** Run; expect pass.
+**Step 5:** Commit.
+
+## Task 2: EF migration
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+**Parallelizable with:** none
+
+**Files:**
+- Create: a new migration under the Configuration project's `Migrations/` folder
+
+```bash
+dotnet ef migrations add AddClusterNodeTransportPorts \
+  --project src/Core/ZB.MOM.WW.OtOpcUa.Configuration
+```
+
+Verify the generated `Up` adds `AkkaPort` with `defaultValue: 4053` and `GrpcPort` as nullable, and
+that `Down` drops both. **Check the migration is additive only** — an `AlterColumn` on anything else
+means the model had drifted from the last migration and that drift must be understood before this
+ships, not carried along by it.
+
+## Task 3: Coordinator sources the expected-ack set from `ClusterNode`
+
+**Classification:** high-risk — this is the deploy path
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs` (`DiscoverDriverNodes`, ~line 262, and the class doc comment at line 21 which states the membership rule)
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ConfigPublishCoordinatorTests.cs`
+
+Replace the `Cluster.State.Members` scan with a query over enabled `ClusterNode` rows. Keep
+`NodeIdComparer` — the case-insensitivity rationale in its comment applies unchanged, and SQL
+collation makes it more relevant, not less.
+
+Points to get right:
+
+- **Filter on `Enabled`.** See "The decision this phase forces".
+- **Scope to the deployment's cluster if the deployment is cluster-scoped.** `DeploymentArtifact.ResolveClusterScope`
+  already defines that scoping; a fleet-wide deployment expects every enabled node, a cluster-scoped
+  one expects only that cluster's. Getting this wrong makes every cluster-scoped deploy hang on nodes
+  that were never sent it. **Read `ResolveClusterScope` before writing the query.**
+- **The empty-set branch still matters.** `HandleDispatch` seals immediately when the set is empty,
+  and its log message says "no driver-role members in cluster" — reword it, because the reason is now
+  "no enabled ClusterNode rows", which is a configuration error rather than a topology observation.
+- The DB is read inside `HandleDispatch`, which already opens a context; reuse it rather than opening
+  a second.
+
+**Tests to add:** enabled-only filtering; cluster-scope filtering; empty set seals with the new
+reason; an ack from a node absent from `ClusterNode` is discarded. **Positive control on each:**
+delete the filter and confirm the test goes red — the existing suite passed throughout the
+membership era, so a test that also passes against the old derivation is testing nothing.
+
+## Task 4: Reconcile the duplicated port with the node's own config
+
+**Classification:** standard
+**Estimated implement time:** ~4 min
+**Parallelizable with:** none
+
+`ClusterNode.AkkaPort` and the node's own `Cluster:Port` are the same fact in two places, and nothing
+makes them agree. A node that binds 4054 while its row says 4053 is unreachable from central in Phase
+2, and the symptom will be a silent absence of acks rather than an error.
+
+Options, in preference order:
+
+1. **Have the node assert its row on startup** — on join, compare `Cluster:Port` /
+   `Cluster:PublicHostname` to its own `ClusterNode` row and log an Error (not a Warning) on
+   mismatch. Cheap, no schema change, no write path from driver nodes to the ConfigDb (which Phase 4
+   removes anyway — so put this check on an **admin** node, reading the membership it can already see).
+2. Deploy-time validation in `DraftValidator`, alongside the existing gates.
+
+Pick one and implement it. Do not skip this task: an unenforced duplicated address is the exact shape
+of the `Modbus`/`ModbusTcp` drift and the `TwinCat`/`Focas` drift already recorded in this repo.
+
+## Task 5: Rig seed + docs
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+
+**Files:**
+- Modify: `docker-dev/seed/seed-clusters.sql` — add `AkkaPort` to the six INSERTs (all 4053) so a
+  freshly-seeded rig matches a migrated one. Leave `GrpcPort` null.
+- Modify: `docs/Configuration.md` — document both columns.
+- Modify: `docs/plans/2026-07-21-per-cluster-mesh-design.md` — mark Phase 1 done in §7.
+
+## Task 6: Live gate
+
+**Classification:** high-risk
+**Estimated implement time:** n/a — operator-driven
+
+Offline tests cannot cover the substitution's real risk, which is that the DB set and the live set
+disagree on the rig. Run on `docker-dev`:
+
+1. Deploy with all six nodes up → seals green, six `NodeDeploymentState` rows, no FK 547.
+2. Deploy a **cluster-scoped** config → expected set is that cluster's nodes only.
+3. `docker stop` one node, deploy → **fails at the apply deadline** naming the missing node. This is
+   the new behaviour; confirm the failure is legible, because an operator will meet it.
+4. Set that node's `Enabled = 0`, deploy again → seals green without it. This proves the escape
+   hatch, and is the check that makes step 3 acceptable to ship.
+
+Record results in a gate doc beside this plan, per the Phase 1/Phase 2 precedent.
+
+## Task 7: `MaintenanceMode` — the escape hatch step 4 proved missing (ADDED 2026-07-22)
+
+**Classification:** high-risk — deploy path + schema
+**Why:** gate step 4 returned `422 ClusterEnabledNodeCountMismatch`. `Enabled = 0` on one node of a
+Warm/Hot pair fails `DraftValidator.ValidateClusterTopology` (enabled count must equal `NodeCount`),
+so the hatch Task 3 relies on cannot be used on any 2-node cluster — i.e. every cluster in the target
+topology. Task 3's behaviour change is only shippable with a hatch that works.
+
+**Decision:** split the meanings rather than weaken either rule.
+
+| Flag | Question | Enforced by |
+|---|---|---|
+| `Enabled` | part of the declared topology? | `DraftValidator` vs `ServerCluster.NodeCount` |
+| `MaintenanceMode` | expected to participate right now? | `ConfigPublishCoordinator`, `ClusterNodeAddressReconciler` |
+
+Rejected: counting configured rather than enabled nodes (removes the InvalidTopology guard);
+downgrading the rule to a warning (weakens a deploy gate for everyone); shipping without a hatch.
+
+**Done:** column + migration `AddClusterNodeMaintenanceMode`; both consumers filter on
+`Enabled && !MaintenanceMode`; deadline + reconciler messages name the right flag; docs; gate step 4
+re-run green.

docs/plans/2026-07-21-per-cluster-mesh-phase1.md.tasks.json

@@ -0,0 +1,76 @@
+{
+  "plan": "docs/plans/2026-07-21-per-cluster-mesh-phase1.md",
+  "branch": "feat/mesh-phase1",
+  "decisions": {
+    "seal-semantics": "DB-derived + Enabled hatch (confirmed 2026-07-22)",
+    "cluster-scope": "Dropped \u2014 no cluster-scoped deployment exists; expected set = all enabled ClusterNode rows",
+    "node-roles": "Every ClusterNode row is a driver node; documented + pinned by test, no Roles column",
+    "adminui-edit": "Deferred to Phase 2 (nothing reads the columns until then)",
+    "maintenance-hatch": "MaintenanceMode column, not Enabled=0 (blocked by DraftValidator on any 2-node pair)"
+  },
+  "tasks": [
+    {
+      "id": 1,
+      "subject": "Address columns on ClusterNode",
+      "status": "completed",
+      "commit": "2bbb0271",
+      "blockedBy": []
+    },
+    {
+      "id": 2,
+      "subject": "EF migration AddClusterNodeTransportPorts",
+      "status": "completed",
+      "commit": "da74ebd6",
+      "blockedBy": [
+        1
+      ]
+    },
+    {
+      "id": 3,
+      "subject": "Coordinator sources expected-ack set from ClusterNode",
+      "status": "completed",
+      "commit": "d88e2455",
+      "blockedBy": [
+        2
+      ]
+    },
+    {
+      "id": 4,
+      "subject": "Reconcile duplicated AkkaPort with the node's own Cluster:Port",
+      "status": "completed",
+      "blockedBy": [
+        2
+      ],
+      "commit": "90c1302f"
+    },
+    {
+      "id": 5,
+      "subject": "Rig seed + docs",
+      "status": "completed",
+      "blockedBy": [
+        3,
+        4
+      ],
+      "commit": "57e1d017"
+    },
+    {
+      "id": 6,
+      "subject": "Live gate on docker-dev",
+      "status": "completed",
+      "blockedBy": [
+        5
+      ],
+      "result": "live-gate PASSED (2026-07-22-mesh-phase1-live-gate.md)"
+    },
+    {
+      "id": 7,
+      "subject": "MaintenanceMode \u2014 escape hatch the live gate proved missing",
+      "status": "completed",
+      "commit": "ee69caa2",
+      "blockedBy": [
+        6
+      ],
+      "origin": "added from gate step 4 failure"
+    }
+  ]
+}

docs/plans/2026-07-22-mesh-phase1-live-gate.md

@@ -0,0 +1,134 @@
+# Per-cluster mesh Phase 1 — live gate record
+
+**Date:** 2026-07-22 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase1`
+**Plan:** [`2026-07-21-per-cluster-mesh-phase1.md`](2026-07-21-per-cluster-mesh-phase1.md)
+
+**Result: PASSED**, after the gate found a blocker that changed the design (step 4).
+
+Offline tests cannot cover the substitution's real risk — that the DB set and the live set disagree on
+a running fleet. This is that check.
+
+## Setup
+
+Migration applied by the rig's `migrator` service against the shared `OtOpcUa` database, then all six
+host nodes restarted on the rebuilt image. **All six pre-existing `ClusterNode` rows inherited
+`AkkaPort = 4053` from the migration's `DEFAULT 4053`** — the row-level evidence that the default does
+its job for rows predating the column:
+
+```
+central-1:4053|central-1|4053|NULL|1
+central-2:4053|central-2|4053|NULL|1
+site-a-1:4053 |site-a-1 |4053|NULL|1
+site-a-2:4053 |site-a-2 |4053|NULL|1
+site-b-1:4053 |site-b-1 |4053|NULL|1
+site-b-2:4053 |site-b-2 |4053|NULL|1
+   NodeId     |  Host   |Akka|Grpc|En
+```
+
+## Step 1 — deploy with all six nodes up → **PASS**
+
+`POST /api/deployments` → `202 Accepted`, deployment `38f83b37`.
+
+`Deployment.Status = 2` (Sealed). Six `NodeDeploymentState` rows, every one `Status = 1` (Applied).
+**No FK 547** — the DB-derived NodeIds satisfy the `NodeDeploymentState → ClusterNode` foreign key,
+which is the standing proof that the DB set and the membership set agree on this rig.
+
+## Step 2 — *dropped, not skipped*
+
+The plan called for a cluster-scoped deploy. **There is no such thing.** `Deployment` has no
+`ClusterId`, `ConfigComposer.SnapshotAndFlattenAsync` always snapshots the whole DB, and
+`DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. Removed
+from the plan rather than faked green.
+
+## Step 3 — deploy with one node stopped → **PASS** (the new behaviour)
+
+`docker stop otopcua-dev-site-b-2-1`, then deploy `37d72301`.
+
+`Deployment.Status = 4` (TimedOut) after the 2-minute apply deadline. Five nodes `Applied`,
+`site-b-2:4053` still `Applying`. Under the membership rule this deployment would have **sealed
+green** with five nodes and never mentioned the sixth.
+
+The operator-facing log, which the plan required to be legible:
+
+```
+[12:52:02 WRN] Deployment 37d723018e374c0eb04c7b6beccc3b3e timed out after 00:02:00 (5/6 acks
+landed). No ack from: site-b-2:4053. Each is an enabled ClusterNode row — start the node, or set
+ClusterNode.Enabled = 0 if it is down for maintenance, then redeploy
+```
+
+(The remedy in that message was corrected to `MaintenanceMode = 1` — see step 4.)
+
+## Step 4 — the escape hatch → **FAILED, then fixed and re-run → PASS**
+
+### First attempt: the hatch did not exist
+
+`UPDATE ClusterNode SET Enabled = 0 WHERE NodeId = 'site-b-2:4053'`, then deploy:
+
+```
+HTTP 422
+[ClusterEnabledNodeCountMismatch] Cluster 'SITE-B' declares NodeCount=2 but has 1 Enabled nodes.
+Toggle the missing node(s) back on or change RedundancyMode/NodeCount to match.
+```
+
+`DraftValidator.ValidateClusterTopology` rejects the deployment **before it dispatches**. So
+`Enabled = 0` cannot be used on a node of a 2-node cluster — and every cluster on this rig, and every
+cluster in the program plan's target topology, is a 2-node pair.
+
+**This was a blocker, not a wrinkle.** The plan called step 4 *"the check that makes step 3
+acceptable to ship"*. Without it, a node down for maintenance blocks every deployment to its cluster,
+with no remedy short of editing `NodeCount` / `RedundancyMode` — which changes the runtime redundancy
+posture to work around a deploy gate.
+
+The two rules were each reasonable and contradictory together: the validator reads `Enabled` as *"part
+of the declared topology"*, Phase 1 additionally read it as *"expect an ack"*.
+
+### Fix (Task 7): split the meanings
+
+New `ClusterNode.MaintenanceMode bit NOT NULL DEFAULT 0`. `Enabled` keeps its topology meaning and the
+validator is untouched; the coordinator's expected-ack set and the address reconciler both filter on
+`Enabled && !MaintenanceMode`.
+
+### Re-run: PASS
+
+`docker stop` `site-b-2` + `MaintenanceMode = 1`, then deploy `82c5e678`:
+
+- `202 Accepted` — the topology gate passes, because the node is still `Enabled` (`1|1`).
+- `Deployment.Status = 2` (Sealed).
+- **Five** `NodeDeploymentState` rows, all `Applied`. `site-b-2:4053` has no row at all — excluded
+  from the expected set rather than waited for and forgiven.
+
+## Task 4 (reconciler) — live behaviour, not a planned gate step
+
+Observed unprompted during the rolling restart, which is better evidence than a staged check:
+
+```
+[12:44:49 WRN] ClusterNode address check [EnabledRowNotInCluster] site-a-1:4053: ...
+[12:44:49 WRN] ... site-a-2:4053 ... site-b-1:4053 ... site-b-2:4053
+[12:44:56 INF] ClusterNode addresses reconcile with cluster membership (6 driver members)
+```
+
+It warned per row while the site nodes were still starting, then converged to a single clean line —
+and **logged nothing further**, confirming the change-detection guard suppresses repeats on the
+5-minute sweep.
+
+Maintenance interaction, from the step 4 re-run:
+
+```
+[13:06:02 WRN] ... site-b-2:4053 ...            ← node dropped, flag not yet set
+[13:06:05 INF] ... reconcile with cluster membership (5 driver members)   ← flag set: silent
+```
+
+The transient warning is honest — for those three seconds the row genuinely was enabled, present in
+the expected-ack set, and absent from the cluster.
+
+## What this gate did not cover
+
+- **Phase 2's premise.** `AkkaPort` / `GrpcPort` are groundwork; nothing dials them yet, so "the value
+  is correct" is only checked by the reconciler comparing it to gossip. The first real exercise is
+  Phase 2.
+- **The reconciler under split meshes.** On the current single mesh an admin node sees every driver
+  member. After Phase 6 it will not, and every site row would report `EnabledRowNotInCluster`
+  permanently. Recorded as a known limitation on the class and in the design doc.
+- **`Enabled = 0` on a single-node (`RedundancyMode.None`) cluster.** Untested — the rig has no such
+  cluster. It would fail the same topology rule (0 enabled vs `NodeCount = 1`), so `MaintenanceMode`
+  is the hatch there too.

docs/plans/2026-07-22-per-cluster-mesh-program.md

@@ -0,0 +1,208 @@
+# Per-Cluster Mesh Program — Align OtOpcUa's Akka Topology with ScadaBridge
+
+> **For Claude:** This is a PROGRAM plan (phase roadmap + gates), not a bite-sized task plan.
+> The authoritative design is `docs/plans/2026-07-21-per-cluster-mesh-design.md` (decisions settled
+> 2026-07-21; Phases 0a/0b DONE). Per that design's own instruction, **each phase gets its own
+> detailed plan, written when the phase starts** (superpowers writing-plans → executing-plans),
+> so plans are authored against current code, not against a forecast. This document sequences the
+> phases, fixes the deployment topology (co-location with ScadaBridge — NEW constraint 2026-07-22),
+> and defines each phase's entry/exit gates.
+
+**Goal:** OtOpcUa runs the same Akka.NET mesh shape as ScadaBridge — **one Akka mesh per
+application `Cluster`, two nodes max**, central ↔ cluster joined by explicit transports
+(ClusterClient + gRPC + fetch), driver nodes holding **no ConfigDb connection** — so that every
+Primary-gated decision and every gated resource share one pair-local scope, and both products
+present one operational model on the shared site hardware.
+
+**Why now (2026-07-22):** OtOpcUa cluster pairs will run **on the same two Windows VMs as the
+ScadaBridge site nodes**. That deployment makes the current single-fleet-mesh design actively
+wrong for sites (a site's OtOpcUa nodes would gossip across the WAN to central and elect ONE
+Primary fleet-wide), and makes the ScadaBridge shape the obviously correct one: each site's two
+VMs host two independent, identically-postured 2-node clusters (one per product), surviving alone
+on the site LAN. It also means driver nodes must not depend on reaching central SQL — **sites
+have no SQL Server**, so §6.1's "driver nodes never connect to the ConfigDb" stops being an
+architectural preference and becomes a deployment requirement.
+
+---
+
+## Deployment topology (the co-location constraint, NEW)
+
+Each **site**: 2 Windows VMs. Each VM runs one ScadaBridge site node AND one OtOpcUa driver node.
+Two independent 2-node Akka clusters per site — they share hardware, never a mesh. **Central**:
+2 Windows VMs, each running a ScadaBridge central node and an OtOpcUa central node
+(`admin,driver`), again as separate pairs.
+
+**Per-VM port allocation (no collisions — verify against actual deployment configs in Phase 6):**
+
+| Port | Owner | Purpose |
+|---|---|---|
+| 8081 / 8082 | ScadaBridge | Akka remoting (central / site) |
+| 8083 | ScadaBridge site | gRPC h2c (streams + LocalDb sync) |
+| 8084 | ScadaBridge site | metrics |
+| 5000 (+Traefik) | ScadaBridge central | UI + Inbound API |
+| **4053** | OtOpcUa | Akka remoting |
+| **4840** | OtOpcUa | OPC UA endpoint |
+| OtOpcUa AdminUI port | OtOpcUa central (admin) | AdminUI (driver-only site nodes host no UI) |
+| OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) |
+
+**Aligned HA posture (both products, per VM — already true or landing via the selfform plan):**
+auto-down downing (15 s window), oldest-Up active/primary election, self-first seed ordering
+(the `SelfFormAfter` self-form fallback that briefly stood in for it was retired 2026-07-22),
+termination-watchdog → process exit → `sc.exe failure` restart recovery.
+One failover story for operators regardless of product.
+
+**Prerequisite ordering:** the fallback/manual-failover plan
+(`docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`) executes **before** this
+program — it is small, independent, and Phase 6 depends on its semantics (under per-pair meshes
+every node lists itself + partner as seeds, so the fallback covers both nodes of every pair;
+the site-node island guard then simply never triggers).
+
+---
+
+## What changes, in one table (from the design doc)
+
+| Aspect | Today (single fleet mesh) | Target (ScadaBridge shape) |
+|---|---|---|
+| Mesh | one gossip ring, all nodes, seeded by central-1 | one 2-node mesh per application `Cluster`; same ActorSystem name; separation by seed-node partitioning |
+| Roles | `admin` / `driver` fleet-wide | `driver` + cluster-specific `cluster-{ClusterId}`; singletons scoped to the cluster role |
+| Primary election | mesh-wide oldest Up driver (Phase 0b) | pair-local by construction — same rule, correct scope |
+| Command/control | 9 DPS topics + singleton over gossip | ClusterClient (one receptionist actor per side); central discovers nodes from `ClusterNode` rows |
+| Deploy | notify (DPS) + node fetches from ConfigDb | notify via ClusterClient; artifact fetched **from central**, cached in LocalDb |
+| Driver ConfigDb connection | direct EF connection to central SQL | **none** — LocalDb is the steady-state config store |
+| Live telemetry | 7 observability DPS topics | one gRPC stream contract (`oneof` event), central dials each cluster node |
+| Rig | six-node single mesh (`docker-dev`) | per-cluster meshes; rig models the real topology |
+
+---
+
+## Phases
+
+Each phase below is one row of design-doc §7, expanded with entry/exit gates. **Execution recipe
+per phase:** (1) invoke writing-plans in this repo to produce
+`docs/plans/2026-07-2X-mesh-phase-N-<name>.md` from the scope notes here + the design doc §
+references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit
+gate; (4) update this file's status column and the design doc's §7 table.
+
+### Phase 1 — `ClusterNode` address columns + DB-sourced ack set — **DONE 2026-07-22**
+Shipped per `2026-07-21-per-cluster-mesh-phase1.md`; see that plan's Task 6 gate record and the
+design doc's "Phase 1 as shipped" note for the two scope deviations (no cluster-scope filtering — no
+such deployment exists; no per-node role column) and the one addition (`ClusterNodeAddressReconciler`).
+**AdminUI node edit was deferred to Phase 2** — nothing reads the columns until then, and the
+migration default plus the rig seed cover every node today.
+
+**Scope:** `ClusterNode` gains Akka + gRPC address columns (mirroring ScadaBridge's `Site`
+entity `NodeAAddress`/`GrpcNodeAAddress` pattern, but per-node rows); EF migration; AdminUI node
+edit surfaces the fields; `ConfigPublishCoordinator` derives its expected-ack set from
+`ClusterNode` rows instead of `Akka.Cluster.State.Members` filtered by role (design §3 fact 3 —
+this removes the coordinator's one genuinely mesh-bound dependency).
+**Independent of the split:** yes — safe on the current mesh.
+**Exit gate:** deploy on the unchanged docker-dev rig completes with the coordinator's expected-ack
+set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node
+missing; a node up but row absent → its ack is not expected).
+
+### Phase 2 — Comm actors + ClusterClient transport
+**Scope:** one receptionist-registered actor per side (`/user/central-communication`,
+`/user/cluster-communication`), registered **per node, not as a singleton** (contact rotation);
+ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge
+idioms copied verbatim: sender-preserving `Tell(new ClusterClient.Send(...), Sender)` Ask relay,
+typed-failure reply for every unhandled message, no central buffering toward unreachable clusters
+(drop + warn), central discovers contacts from Phase 1's `ClusterNode` rows (60 s refresh +
+admin-change refresh), clusters know central from appsettings (static, restart to change).
+**Frame-size guard:** anything carrying payload sets both the frame limit and
+`log-frame-size-exceeding` (design §8) — the deploy path stays payload-free by design.
+**Exit gate:** on the still-single-mesh rig, deploy notify + acks and AdminUI Reconnect/Restart
+flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly
+against a stopped node.
+
+### Phase 3 — Config fetch-and-cache from central
+**Scope:** the deploy artifact is served **by central** (transport per its own phase plan — the
+ScadaBridge analogue is token-gated HTTP; decide HTTP vs a gRPC fetch RPC when planning) and
+cached in LocalDb; driver nodes read config exclusively from LocalDb (boot-from-cache becomes the
+normal path — design §6.1); chunking, SHA-256 verify, newest-2 retention, pair replication carry
+over unchanged. **This phase changes a running data path — it gets its own live gate** (design §7
+note): deploy lands on a driver pair with central SQL stopped mid-fetch → retry lands; #485
+last-known-good semantics re-proven on the new path.
+**Exit gate:** live gate green on the rig; a driver node with an empty LocalDb and reachable
+central boots into the current config; with central down it boots last-known-good.
+
+### Phase 4 — Cut the driver-side ConfigDb connection
+**Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local
+state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor`
+question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering,
+so define the replacement health input (candidate: central-reachability via the Phase 2/3
+transports) — this is a client-visible ServiceLevel semantics change and must be documented in
+`docs/Redundancy.md` + the interop playbook; audit `OpcUaPublishActor`'s ConfigDb use (TBD in the
+design) and re-source it; registration cleanup in `ServiceCollectionExtensions`; driver-role
+`Program.cs` branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only
+`AddConfigurationDatabase`).
+**Exit gate:** its own live gate — a driver pair runs a full deploy + alarm + historian cycle with
+**no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can
+resolve the ConfigDb context.
+
+### Phase 5 — gRPC stream contract for live telemetry
+**Scope:** one server-streaming contract carrying a `oneof` event, **cluster nodes host the gRPC
+server, central dials in** (the inverted direction is the load-bearing ScadaBridge finding —
+design §2); migrate the seven observability topics (`alerts`, `driver-health`,
+`driver-resilience-status`, `fleet-status`, `script-logs`, plus redundancy-state distribution and
+deployment-acks if Phase 2 left them on DPS); additive-only field evolution, contract locked by
+test; per-panel reconnect story for the AdminUI (design §8 — losing gossip loses free fleet
+observability).
+**Exit gate:** all AdminUI live panels green against a pair with DPS telemetry topics deleted;
+kill-and-reconnect of the central dialer recovers every stream.
+
+### Phase 6 — Mesh partition + co-location topology
+**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
+`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
+`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
+`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
+`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
+§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
+`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed
+of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
+central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
+including the co-location port table above (both products' compose files on shared per-site
+networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
+election is pair-local now) and the fallback's site-node island-guard docs note (moot — every
+node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair.
+**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair
+(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**);
+secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its
+topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot
+cross separate meshes).
+
+### Phase 7 — Failover drill + live gates
+**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
+directions; **close the two outstanding live gates**: (a) auto-down 1-vs-1 crash-the-oldest
+(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) self-first
+lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair;
+operator runbook for the co-located site (one page covering both products' failover on the same
+two VMs).
+**Exit gate:** drill green on every pair type (central, site); runbook merged; design doc §7
+table fully marked DONE. (Live gate (b) is now the **self-first cold-start-alone** drill, not
+`SelfFormAfter` — the watchdog it named was retired 2026-07-22.)
+
+---
+
+## Risks carried from the design (§8, unchanged — re-read before each phase plan)
+
+LocalDb becomes load-bearing for config (blast radius of the #485 class rises);
+`DbHealthProbeActor` feeds a client-visible value; the rig models the doomed topology until
+Phase 6; losing gossip loses free observability (7 panels); 128 KB ClusterClient frame drop is
+silent unless both knobs are set; never stack app-level LWW on LocalDb's HLC. **New (this
+program):** co-located VMs mean a VM loss now takes out one node of BOTH products at once — the
+drill in Phase 7 must include the shared-VM failure (both products fail over together), and
+resource sizing on the site VMs should be checked once both products run the full stack.
+
+## Tracking
+
+| Phase | Status |
+|---|---|
+| 0a downing strategy | DONE 2026-07-21 (live gate → Phase 7) |
+| 0b oldest-Up election | DONE 2026-07-21 |
+| Prereq: selfform-fallback + manual-failover plan | plan written 2026-07-22, not executed |
+| 1 ClusterNode columns + DB ack set | **detailed plan ALREADY WRITTEN** (`2026-07-21-per-cluster-mesh-phase1.md`, from the design session — do NOT write a new one), not executed. ⚠ Carries a decision gate: DB-derived expected-ack set changes deploy-seal semantics (a stopped-but-enabled node now FAILS the deploy at the apply deadline instead of sealing green without it; escape hatch = `ClusterNode.Enabled=0`) — confirm with the owner before its Task 3. |
+| 2 ClusterClient transport | not started |
+| 3 fetch-and-cache | not started |
+| 4 cut driver ConfigDb | not started |
+| 5 gRPC telemetry | not started |
+| 6 mesh partition + co-location | not started |
+| 7 drill + live gates | not started |

docs/plans/2026-07-22-per-cluster-mesh-program.md.tasks.json

@@ -0,0 +1,15 @@
+{
+  "planPath": "docs/plans/2026-07-22-per-cluster-mesh-program.md",
+  "note": "PROGRAM plan: each task = write that phase's detailed plan (writing-plans), execute it (executing-plans), run its exit gate, update the tracking tables. Prereq task 0 is the separate selfform-fallback plan in this repo.",
+  "tasks": [
+    {"id": 0, "subject": "Prereq: execute 2026-07-22-selfform-fallback-and-manual-failover.md (7 tasks)", "status": "pending"},
+    {"id": 1, "subject": "Phase 1: ClusterNode Akka+gRPC address columns; ConfigPublishCoordinator ack set from DB", "status": "pending"},
+    {"id": 2, "subject": "Phase 2: comm actors + receptionist + ClusterClient transport (deploy notify/acks, driver-control)", "status": "pending", "blockedBy": [1]},
+    {"id": 3, "subject": "Phase 3: config fetch-and-cache from central; LocalDb steady-state config store (live gate)", "status": "pending", "blockedBy": [2]},
+    {"id": 4, "subject": "Phase 4: cut driver-side ConfigDb — EfAlarmConditionStateStore to LocalDb, DbHealthProbeActor ServiceLevel input, OpcUaPublishActor audit (live gate)", "status": "pending", "blockedBy": [3]},
+    {"id": 5, "subject": "Phase 5: gRPC oneof stream contract; migrate 7 observability topics; AdminUI reconnect story", "status": "pending", "blockedBy": [2]},
+    {"id": 6, "subject": "Phase 6: mesh partition — per-pair seeds, cluster-scoped roles/singletons, co-location rig rewrite, secrets replication re-scope", "status": "pending", "blockedBy": [0, 4, 5]},
+    {"id": 7, "subject": "Phase 7: failover drill per pair (incl. shared-VM failure) + close auto-down 1v1 and self-first cold-start-alone live gates + operator runbook", "status": "pending", "blockedBy": [6]}
+  ],
+  "lastUpdated": "2026-07-22T00:00:00Z"
+}

docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md

@@ -0,0 +1,405 @@
+# OtOpcUa: InitJoin Self-Form Fallback + Manual Failover Control — Implementation Plan
+
+> ## ⚠ SUPERSEDED IN PART (2026-07-22, same day)
+>
+> **Half (1) of this plan — the `Cluster:SelfFormAfter` self-form watchdog — was retired within hours
+> of landing.** `ClusterBootstrapFallback`, the `SelfFormAfter` option and `SelfFormBootstrapTests`
+> are **deleted**; the cold-start-alone goal is now met by **self-first seed ordering** (each seed
+> node lists ITSELF as `seed-nodes[0]`), enforced at boot by `AkkaClusterOptionsValidator` and pinned
+> by `SelfFirstSeedBootstrapTests`. Reason: a timer outside Akka's join handshake cannot distinguish
+> "no seed answered" from "a seed answered and the join is in flight", and `Cluster.Join(SelfAddress)`
+> is not ignored mid-handshake — it wins. Task 8 below caught exactly that live and patched the one
+> observed shape with a TCP reachability guard; the race remained. Mesh-program Phase 6 had this
+> convergence queued and it was pulled forward. See `docs/Redundancy.md` §"Bootstrap: self-first seed
+> ordering" and ScadaBridge `4a6341d8`.
+>
+> **Half (2) — manual failover — is unaffected and still current.** Tasks below are kept verbatim as
+> the execution record, including the Task 8 live-gate evidence, which is the whole reason the
+> watchdog was rejected. Do not execute the Task 1/2/3 code as written.
+
+> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
+>
+> Shared cross-repo design: `~/Desktop/scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md` (design rationale, MNTR assessment, behavior spec). The ScadaBridge half lives in `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`. This plan is self-contained for execution.
+>
+> **⚠ This plan is expected to GROW** — see "Reserved for additions" at the end. Append new tasks there (and to the `.tasks.json`) rather than renumbering existing ones.
+
+**Goal:** (1) Either node of a 2-node OtOpcUa pair can cold-start alone and become operational, unattended — a configurable, fast (default 10 s) InitJoin timeout falls back to self-forming a cluster, with a hard guard so site nodes seeded only by central can never island themselves. (2) An admin-only "Trigger failover" control on the Cluster Redundancy page performs a graceful, audited swap of the driver Primary.
+
+**Architecture:** New `AkkaClusterOptions.SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables, bound from the `Cluster` section) arms `ClusterBootstrapFallback` via an Akka.Hosting startup task inside `WithOtOpcUaClusterBootstrap` — so production AND the Cluster.Tests hosts get it identically. Wait for membership via `RegisterOnMemberUp`; on expiry, `Cluster.Join(SelfAddress)` — only if this node's own address is in its own `SeedNodes`. Manual failover = graceful `Cluster.Leave(<oldest Up driver member>)` via a new `IManualFailoverService` in ControlPlane, surfaced on `/clusters/{id}/redundancy`.
+
+**Tech Stack:** .NET 10, Akka.NET 1.5.62, Akka.Hosting 1.5.62 (`AddStartup`), Blazor Server (AdminUI, InteractiveServer), bUnit, xunit. No new packages.
+
+**Branch:** `feat/selfform-fallback` off `master`.
+
+---
+
+## Design essentials (from the shared design doc)
+
+**The defect:** Akka only lets the FIRST listed seed self-join to form a *new* cluster; every other node loops on `InitJoin` forever — "auto-down removes the crash outage, not this one" (`docs/Redundancy.md:244-247`). A lone cold-starting `central-2` (or a pair node under the future per-cluster mesh) never comes Up.
+
+**Behavior spec:**
+
+| Scenario | Behavior with fallback |
+|---|---|
+| Peer alive (any boot order) | Normal seed join in ms — fallback never fires |
+| Lone cold-start, self IS in own `SeedNodes` | After `SelfFormAfter`: warn log + `Cluster.Join(SelfAddress)` → Up alone (`min-nr-of-members=1`) |
+| Lone cold-start, self NOT in own `SeedNodes` | Fallback inert (info log). **This is the docker-dev site-node topology** — site-a/b nodes list only `central-1`; self-forming there would island them permanently, so they must wait. |
+| Peer boots after survivor self-formed | Its InitJoin is answered → joins as youngest. No island. |
+| Both pair nodes cold-start simultaneously, mutually unreachable | Both self-form → dual-active (same partition class auto-down accepts; restart one side) |
+| `SelfFormAfter` null/`≤0` | Disabled — today's wait-forever behavior |
+| ~~Window expires mid-join-handshake~~ | ~~Benign: Akka ignores `Join` once joined~~ **FALSE — proven live (Task 8): `Join(self)` during an in-flight join abandons it and islands the node. Fixed by the TCP reachability guard: never self-form while any seed peer accepts a connection; wait a window and re-arm. ScadaBridge rejected the watchdog entirely for self-first seed ordering; OtOpcUa converges at mesh-program Phase 6.** |
+
+**Manual failover rules:** graceful `Leave` of the **oldest Up `driver` member** (identical query to `RedundancyStateActor.SelectDriverPrimary` — `Member.AgeOrdering`, `Leaving` excluded via `Status == Up`), never `Down`. The leaving node hands over via the cluster-leave phases, `CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it, and it rejoins as youngest → the survivor is Primary (ServiceLevel 250 moves; OPC UA clients re-select). Admin-only; peer guard (<2 Up driver members → disabled); confirmation dialog; audited via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam before the Leave. **Mesh-scope caveat:** until the per-cluster mesh lands, the election is mesh-wide — the button acts on THE Primary of the whole Akka cluster, not per-`ServerCluster` row; the UI must say so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md:84-99`).
+
+**Multi-node TestKit:** assessed, NOT used — everything is deterministic in-process via real hosts through `WithOtOpcUaClusterBootstrap` (the `SplitBrainResolverActivationTests` pattern), and the Akka TestKit family is xunit-v2-only while this repo's integration tree is xunit.v3. Full verdict in the shared design doc.
+
+---
+
+### Task 1 (B1): `SelfFormAfter` on `AkkaClusterOptions`
+
+**Classification:** small
+**Estimated implement time:** ~3 min
+**Parallelizable with:** none (first task)
+
+**Files:**
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs`
+
+**Step 1: Branch**
+
+```bash
+cd ~/Desktop/OtOpcUa && git checkout master && git checkout -b feat/selfform-fallback
+```
+
+**Step 2: Add the property** (after `SplitBrainResolverStrategy`, mirroring its XML-doc depth):
+
+```csharp
+    /// <summary>
+    /// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
+    /// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
+    /// while its peer is down loops on InitJoin forever ("auto-down removes the crash outage,
+    /// not this one" — docs/Redundancy.md). When this node has waited longer than this window
+    /// without cluster membership it joins itself — but ONLY if its own address appears in
+    /// <see cref="SeedNodes"/>; a non-seed node (e.g. a site node whose only seed is central-1)
+    /// stays waiting, because self-forming there creates a permanent island. Default 10s
+    /// (same-datacenter pair; a live peer answers InitJoin in milliseconds). <c>null</c> or a
+    /// non-positive value disables the fallback. Accepted trade: both pair nodes cold-starting
+    /// inside the window while mutually unreachable form two clusters — the same dual-active
+    /// class the auto-down strategy already accepts, same recovery (restart one side).
+    /// </summary>
+    public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);
+```
+
+**Step 3:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster` — 0 warnings. Commit:
+
+```bash
+git add -A && git commit -m "feat(cluster): SelfFormAfter option (bootstrap self-form fallback window)"
+```
+
+(The default-value pin test lands with Task 3's test file — the property is inert until Task 2.)
+
+---
+
+### Task 2 (B2): `ClusterBootstrapFallback` + `AddStartup` wiring
+
+**Classification:** high-risk (cluster formation behavior)
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterBootstrapFallback.cs`
+- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (`WithOtOpcUaClusterBootstrap`, after the `AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend)` call at line 93)
+
+**Step 1: Implement the fallback class** (deliberately duplicated from ScadaBridge's, like the termination watchdogs — see that repo's `ClusterBootstrapFallback.cs` for the full doc comment to adapt):
+
+```csharp
+using Akka.Actor;
+using Microsoft.Extensions.Logging;
+
+namespace ZB.MOM.WW.OtOpcUa.Cluster;
+
+/// <summary>
+/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1). Akka only
+/// lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin forever —
+/// "auto-down removes the crash outage, not this one" (docs/Redundancy.md). Waits
+/// <see cref="AkkaClusterOptions.SelfFormAfter"/> for membership, then joins itself.
+/// <para><b>Island safety:</b> fires ONLY when this node's own address is in its own
+/// <see cref="AkkaClusterOptions.SeedNodes"/> — a site node seeded only by central-1 must wait,
+/// not island. Sequential recovery is island-free: Akka's join protocol prefers an existing
+/// cluster (a booting node only self-forms when NO seed answers InitJoin).</para>
+/// <para><b>Benign race:</b> Akka ignores <c>Join</c> once the node has joined a cluster this
+/// incarnation. Residual risk: both pair nodes cold-starting inside the window while mutually
+/// unreachable self-form separately — the same dual-active class auto-down accepts.</para>
+/// </summary>
+public static class ClusterBootstrapFallback
+{
+    public static void Arm(ActorSystem system, AkkaClusterOptions options, ILogger logger)
+    {
+        if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero)
+        {
+            logger.LogInformation(
+                "Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting "
+                + "while its peer is down will wait on InitJoin indefinitely.");
+            return;
+        }
+
+        var cluster = Akka.Cluster.Cluster.Get(system);
+        var self = cluster.SelfAddress;
+        var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self));
+        if (!isSeed)
+        {
+            logger.LogInformation(
+                "Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list "
+                + "[{Seeds}] — self-forming here would island it from the real seeds.",
+                self, string.Join(", ", options.SeedNodes));
+            return;
+        }
+
+        var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
+        cluster.RegisterOnMemberUp(() => joined.TrySetResult());
+
+        _ = Task.Run(async () =>
+        {
+            var winner = await Task.WhenAny(joined.Task, Task.Delay(window));
+            if (winner == joined.Task || system.WhenTerminated.IsCompleted)
+                return;
+            logger.LogWarning(
+                "No cluster membership after {Window} — no seed answered InitJoin (peer down at boot). "
+                + "Self-forming a cluster at {Self} so this node becomes operational; if the peer was "
+                + "merely partitioned (not dead), the pair is now dual-active — restart one side after "
+                + "the partition heals (accepted availability-first trade, decision 2026-07-22).",
+                window, self);
+            cluster.Join(self);
+        });
+    }
+
+    private static bool TryParseAddress(string seed, out Address address)
+    {
+        try { address = Address.Parse(seed); return true; }
+        catch { address = default!; return false; }
+    }
+}
+```
+
+(If the Cluster project lacks a `Microsoft.Extensions.Logging.Abstractions` reference, add the `PackageReference` — check `Directory.Packages.props` for the pinned version first.)
+
+**Step 2: Wire via Akka.Hosting startup hook** — in `WithOtOpcUaClusterBootstrap`, after the downing HOCON line (~93):
+
+```csharp
+        // InitJoin self-form fallback (decision 2026-07-22): registered as an Akka.Hosting
+        // startup task so every host built through this bootstrap — production AND the test
+        // hosts in Cluster.Tests — arms it identically. Guarded inside Arm: disabled when
+        // SelfFormAfter is unset; inert when this node is not in its own seed list (site nodes
+        // seeded only by central-1 must wait, not island).
+        var fallbackLogger = (serviceProvider.GetService<ILoggerFactory>()
+                              ?? Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance)
+            .CreateLogger("ZB.MOM.WW.OtOpcUa.Cluster.ClusterBootstrapFallback");
+        builder.AddStartup((system, _) =>
+        {
+            ClusterBootstrapFallback.Arm(system, options, fallbackLogger);
+            return Task.CompletedTask;
+        });
+```
+
+(Akka.Hosting 1.5.62 has `AddStartup`. If the overload signature differs, alternative: register an `IHostedService` in `AddOtOpcUaCluster` mirroring `ActorSystemTerminationWatchdog`'s lazy-accessor pattern — but prefer `AddStartup` so test hosts arm it for free.)
+
+**Step 3:** `dotnet build src/Core/ZB.MOM.WW.OtOpcUa.Cluster` — 0 warnings.
+
+**Step 4: Commit**
+
+```bash
+git add -A && git commit -m "feat(cluster): InitJoin self-form fallback armed via Akka.Hosting startup task"
+```
+
+---
+
+### Task 3 (B3): `SelfFormBootstrapTests` — 4 scenarios through the production bootstrap
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** Task 4
+
+**Files:**
+- Create: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SelfFormBootstrapTests.cs`
+
+**Step 1: Write the tests** — follow `SplitBrainResolverActivationTests`' pattern (real `Host.CreateDefaultBuilder()` + `AddAkka` + `WithOtOpcUaClusterBootstrap`; that file's remarks explain why testing through the production bootstrap is mandatory — a test that re-creates the wiring pins the test's wiring, not the shipped one). Helper + scenarios:
+
+```csharp
+public sealed class SelfFormBootstrapTests
+{
+    private static int FreePort()
+    {
+        var l = new System.Net.Sockets.TcpListener(System.Net.IPAddress.Loopback, 0);
+        l.Start();
+        var p = ((System.Net.IPEndPoint)l.LocalEndpoint).Port;
+        l.Stop();
+        return p;
+    }
+
+    private static async Task<IHost> StartNodeAsync(
+        string systemName, int selfPort, int peerPort, TimeSpan? selfFormAfter, bool selfInSeeds = true)
+    {
+        var options = new AkkaClusterOptions
+        {
+            SystemName = systemName, Hostname = "127.0.0.1", PublicHostname = "127.0.0.1",
+            Port = selfPort, Roles = new[] { "admin", "driver" },
+            SelfFormAfter = selfFormAfter,
+            SeedNodes = selfInSeeds
+                ? new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}",
+                          $"akka.tcp://{systemName}@127.0.0.1:{selfPort}" }   // self SECOND — only the fallback can form it
+                : new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}" },
+        };
+        var builder = Host.CreateDefaultBuilder();
+        builder.ConfigureServices(services =>
+        {
+            services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
+            services.AddAkka(systemName, (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
+        });
+        var host = builder.Build();
+        await host.StartAsync();
+        return host;
+    }
+
+    // 1) SelfFormAfter_defaults_to_ten_seconds
+    //      new AkkaClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10));
+    // 2) Lone_non_first_seed_self_forms_after_the_window
+    //      dead peer port, window 2s → poll Cluster.Get(system).State until 1 Up member ≤ 20s.
+    // 3) Disabled_fallback_keeps_waiting  (window null → State.Members empty after 6s;
+    //      POSITIVE CONTROL: cluster.Join(cluster.SelfAddress) → 1 Up ≤ 20s — proves the node
+    //      was formable and only the fallback was absent.)
+    // 4) Site_node_seeded_only_by_central_never_self_forms  (selfInSeeds:false, window 1s →
+    //      Members empty after 5s; positive control join. THE guard test: this is the exact
+    //      docker-dev topology — site nodes list only central-1.)
+}
+```
+
+Use a unique `systemName` per test (`otopcua-selfform-1` …) so concurrent hosts cannot cross-join; dispose each host in `finally` (`await host.StopAsync(); host.Dispose();`).
+
+**Step 2: Prove the key assertion has teeth** — before relying on scenario 2, temporarily run it with `selfFormAfter: null` and confirm it times out (red), then restore. Record that in the task notes.
+
+**Step 3: Run**
+
+```bash
+dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~SelfFormBootstrap"
+```
+Expected: 4 PASS.
+
+**Step 4: Commit**
+
+```bash
+git add -A && git commit -m "test(cluster): self-form fallback — lone-seed forms, disabled waits, site-node guard"
+```
+
+---
+
+### Task 4 (B4): Docs — Redundancy.md + mesh design note
+
+**Classification:** small
+**Estimated implement time:** ~4 min
+**Parallelizable with:** Task 3
+
+**Files:**
+- Modify: `docs/Redundancy.md` (~lines 162, 244–247) — replace the bootstrap-constraint text ("Auto-down removes the crash outage, not this one") with `SelfFormAfter` semantics, the 10 s default, the site-node island guard (why driver-only nodes seeded by central-1 deliberately do NOT self-form under the CURRENT one-mesh topology), and the note that under the future per-cluster mesh each pair node is a seed so the fallback covers both.
+- Modify: `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2/§7 — one-line note: the "registered outage gap" is now closed by `SelfFormAfter` (link Redundancy.md).
+- Modify: `CLAUDE.md` — brief note in the redundancy-corrections area.
+
+Commit: `docs(redundancy): SelfFormAfter closes the seed-node bootstrap gap`.
+
+---
+
+### Task 5 (B5): Full verification + optional docker-dev live check
+
+**Classification:** standard
+**Estimated implement time:** ~5 min (suite runtime dominates)
+**Parallelizable with:** none
+
+```bash
+cd ~/Desktop/OtOpcUa
+dotnet build ZB.MOM.WW.OtOpcUa.slnx     # 0 warnings
+dotnet test  ZB.MOM.WW.OtOpcUa.slnx     # green vs the pre-existing baseline
+```
+
+Optional live check (docker-dev six-node mesh): `docker compose stop central-1 central-2 && docker compose start central-2` → central-2 (a listed seed) self-forms in ~10 s; site nodes (non-seeds) keep waiting — the island guard on the real topology. Then `docker compose start central-1` → joins central-2's cluster, mesh re-forms. If the rig isn't running, record deferred-live.
+
+---
+
+### Task 6 (D4): Manual failover service + cluster-level test
+
+**Classification:** high-risk
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs`
+- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/ManualFailoverService.cs`
+- Modify: the ControlPlane/Host DI registration point (find where `RedundancyStateActor`-adjacent services register; admin-role branch of `Program.cs`)
+- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ManualFailoverServiceTests.cs`
+
+**Step 1: Failing tests** — reuse the two-node fixture pattern from `RedundancyPrimaryElectionTests` (node A older on the **higher** port so age ≠ address order; that fixture even has a guard test asserting the divergence — keep it honest here too):
+
+- `FailOver_targets_the_oldest_driver_member_not_the_role_leader`: service picks node A (oldest, higher port) even though B is `RoleLeader("driver")`; A's system terminates (graceful exit); B's next `RedundancyStateActor` snapshot names B Primary.
+- `FailOver_refuses_when_no_driver_peer_exists`: single driver member → returns null, no Leave, node still Up (positive assert).
+- `Parity_with_SelectDriverPrimary`: on the same fixture, the service's target equals the member `RedundancyStateActor` names Primary — pins the two 5-line queries together.
+
+**Step 2: Implement** — same shape as ScadaBridge's `FailOverCore` but role `"driver"`:
+
+```csharp
+public static Address? FailOverCore(ActorSystem system, bool dryRun = false)
+{
+    var cluster = Akka.Cluster.Cluster.Get(system);
+    // Intentionally identical to RedundancyStateActor.SelectDriverPrimary (oldest Up driver,
+    // Member.AgeOrdering) — the node acted on must be exactly the one the election names.
+    // Pinned by ManualFailoverServiceTests.Parity_with_SelectDriverPrimary.
+    var drivers = cluster.State.Members
+        .Where(m => m.Status == MemberStatus.Up && m.Roles.Contains("driver"))
+        .OrderBy(m => m, Member.AgeOrdering)
+        .ToList();
+    if (drivers.Count < 2) return null;
+    var primary = drivers[0];
+    if (!dryRun) cluster.Leave(primary.Address);
+    return primary.Address;
+}
+```
+
+Async wrapper `FailOverDriverPrimaryAsync(string actor)`: dry-run peer guard → audit via the shared `ZB.MOM.WW.Audit.IAuditWriter` seam (action `cluster.manual-failover`, actor, target in `DetailsJson` — copy the call shape from an existing audited AdminUI admin action) → real Leave → warn log → return address string.
+
+**Step 3:** Tests PASS → commit `feat(redundancy): manual failover service — graceful Leave of the driver Primary`.
+
+---
+
+### Task 7 (D5): ClusterRedundancy page — live panel + failover button + bUnit tests
+
+**Classification:** standard
+**Estimated implement time:** ~5 min
+**Parallelizable with:** none
+
+**Files:**
+- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Clusters/ClusterRedundancy.razor` (page is already `@rendermode RenderMode.InteractiveServer` — no rendermode work; and it lives in the AdminUI host, not an RCL, so the static-router wrapper concern from lmxopcua#483 does not apply)
+- Test: the AdminUI bUnit suite (place beside the existing Clusters page tests)
+- Modify: `docs/Redundancy.md` — "Manual failover" subsection (button semantics, mesh-wide scope caveat)
+
+**Step 1: Failing bUnit tests:** (a) button hidden under the read-only policy (`AdminUiPolicies.AuthenticatedRead`), visible under the admin/write policy — check `AdminUiPolicies` for the exact policy the mutating Clusters pages use (e.g. `ClusterEdit.razor`) and use the same one; (b) disabled with tooltip when the live snapshot shows <2 Up driver members; (c) confirm flow calls a fake `IManualFailoverService` exactly once.
+
+**Step 2: Implement.** Add a "Live redundancy" panel above the static config table: current driver Primary + Up driver members from a `GetSnapshot()` read-only companion on `IManualFailoverService` (sourced from live `Cluster.State`; do NOT subscribe to the DPS topic for v1), the mesh-scope notice ("in the current single-mesh topology this acts on the whole mesh's Primary"), and the admin-gated **Trigger failover** button with confirm dialog (states: the Primary node restarts, ServiceLevel 250 moves to the survivor, OPC UA clients re-select). On confirm: call the service with the authenticated user name; surface the returned target address.
+
+**Step 3:** Tests PASS → commit `feat(adminui): manual failover control on the cluster redundancy page`.
+
+**Live check (fold into the Task 5 gate when docker-dev is up):** press the button, ServiceLevel 250 moves (old Primary drops off, survivor publishes 250), denial meters stay quiet, audit event recorded.
+
+---
+
+## Reserved for additions (this plan is expected to grow)
+
+Append new tasks below as **Task 8+** (and extend `.tasks.json`) — do not renumber Tasks 1–7. Candidate items already flagged in the family docs, awaiting a decision on which to include:
+
+- **Configurable downing window** — promote `ServiceCollectionExtensions.DowningStableAfter` (hard-coded 15 s) to `AkkaClusterOptions` (e.g. `Cluster:DowningStableAfter`), keeping the pinned invariant window > `acceptable-heartbeat-pause` (`akka_failover.md` §6.2 calls this out; ScadaBridge's equivalent `Cluster:StableAfter` is already config).
+- **Failure-detector tuning for faster failover** — expose `heartbeat-interval` / `acceptable-heartbeat-pause` (currently fixed in `Resources/akka.conf`) as options, per the availability-first tuning table in `akka_failover.md` §2.
+- ~~Auto-down 1-vs-1 live gate~~ / ~~per-cluster mesh interplay~~ — **now owned by the per-cluster mesh PROGRAM plan** (`docs/plans/2026-07-22-per-cluster-mesh-program.md`, added 2026-07-22): Phases 1–7 align OtOpcUa's mesh with ScadaBridge's shape (one 2-node mesh per application Cluster, co-located on the ScadaBridge VMs). That program lists THIS plan as its prerequisite (execute this one first); its Phase 6 removes the ClusterRedundancy mesh-scope caveat + the site-node island-guard docs note, and Phase 7 closes the auto-down 1-vs-1 and `SelfFormAfter` live gates on the real per-pair topology.
+- *(add further items here)*
+
+---
+
+## Completion
+
+- Merge decision via the finishing-a-development-branch flow (family convention: ff-merge to `master` + push to gitea `lmxopcua`, or PR — ask the user).
+- After merge: update `scadaproj/akka_failover.md` §6.1 status, `scadaproj/CLAUDE.md` index row, and memory `ha-availability-over-partition-safety` (tracked in the scadaproj index plan).
+- Verification-before-completion applies throughout; live gates may be recorded deferred-live if docker-dev is down.

docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md.tasks.json

@@ -0,0 +1,14 @@
+{
+  "planPath": "docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md",
+  "tasks": [
+    {"id": 0, "subject": "Task 1: SelfFormAfter on AkkaClusterOptions", "status": "completed"},
+    {"id": 1, "subject": "Task 2: ClusterBootstrapFallback + AddStartup wiring", "status": "completed", "blockedBy": [0]},
+    {"id": 2, "subject": "Task 3: SelfFormBootstrapTests — 4 scenarios incl. site-node island guard", "status": "completed", "blockedBy": [1], "notes": "Teeth proven: scenario 2 run with selfFormAfter:null failed (red) before restore. A 5th test was added by Task 8."},
+    {"id": 3, "subject": "Task 4: Docs — Redundancy.md bootstrap section + mesh design note", "status": "completed", "blockedBy": [1]},
+    {"id": 4, "subject": "Task 5: Full verification + optional docker-dev live check", "status": "completed", "blockedBy": [2, 3], "notes": "Live gate PASSED for the cold-start case (defect observed on old image, fix on new, island guard on a real site node, one six-node cluster after recovery)."},
+    {"id": 5, "subject": "Task 6: Manual failover service + cluster-level test", "status": "completed", "blockedBy": [4], "notes": "4/4. Teeth proven: sabotaging FailOverCore to address order turned the target + parity tests red."},
+    {"id": 6, "subject": "Task 7: ClusterRedundancy live panel + failover button + tests", "status": "completed", "blockedBy": [5], "notes": "PLAN DEVIATION: plan said bUnit; repo has no bUnit (PageAuthorizationGuardTests documents itself as the substitute). Adapted to a pure ManualFailoverPageModel + 9 unit tests, razor as thin shell. Policy is FleetAdmin, not the ConfigEditor the neighbouring cluster pages use (this restarts a node; ConfigEditor admits Designer). Live-verified: panel, confirm dialog, execution, singleton handover, audit row."},
+    {"id": 7, "subject": "Task 8 (NEW, from live gate): reachability guard — don't self-form while a seed peer is reachable", "status": "completed", "blockedBy": [6], "notes": "DEFECT: the Task 7 live drill caught the self-form fallback islanding the node the failover bounced. central-1 restarted, got InitJoinAck from central-2 at 10:49:05 (join in flight, healthy) but no Welcome inside the window (peer's ring still held the old incarnation until 10:49:06); at 10:49:15 the fallback fired and formed a SECOND cluster. The plan's 'Window expires mid-join-handshake -> benign, Akka ignores Join once joined' row is FALSE — the node had not joined, and Join(self) wins. Manual failover deliberately produces that restart, so the two halves of the plan collided. FIX (user chose the reachability guard): TCP-probe the other seed addresses before self-forming; a reachable peer means wait another window and re-arm. Proven three ways: (1) new regression test whose 'peer' is a bare TcpListener speaking no Akka — RED without the guard; (2) live, a throwaway node seeded at sql:1433 (TCP-reachable, not Akka) logged the guard every 10s and never self-formed; (3) positive control, the same node seeded at a dead port self-formed at +10s. Failover re-drill on the rig: no island, all six nodes one cluster."}
+  ],
+  "lastUpdated": "2026-07-22T00:00:00Z"
+}

docs/security.md

@@ -380,6 +380,55 @@ polling the node.
 
 ---
 
+## Config secrets (`${secret:}` delivery)
+
+OtOpcUa never commits secret values to `appsettings*.json`. Every real secret is
+supplied at deploy time — either as a plain environment variable, or as a
+`${secret:<name>}` token backed by the shared **`ZB.MOM.WW.Secrets`** encrypted store.
+
+A pre-host expander (`SecretReferenceExpander.ExpandConfigurationAsync`, wired in
+`OtOpcUa.Host/Program.cs`) walks the assembled configuration and rewrites every
+`${secret:<name>}` token into its resolved plaintext **before** the host is built —
+so every downstream binder/validator (`AddZbSerilog`, `AddOtOpcUaConfigDb`, the first
+`ValidateOnStart`) sees resolved values. The store's key-encryption key (KEK) comes from
+the `ZB_SECRETS_MASTER_KEY` environment variable (`Secrets:MasterKey:Source=Environment`);
+the encrypted SQLite store lives at `Secrets:SqlitePath`.
+
+The expander is **fail-closed and section-agnostic**: a `${secret:<name>}` token whose
+secret is absent throws `SecretNotFoundException` at startup, regardless of which feature
+owns the key or whether that feature is enabled. Keys prefixed with `_` (comment keys) are
+skipped, so a `${secret:...}` example inside a `_…Comment` value is never resolved.
+
+The five config secrets and their canonical secret names:
+
+| Config key | Owner | Secret name |
+|---|---|---|
+| `Security:Jwt:SigningKey` | `JwtOptions` | `otopcua/jwt/signing-key` |
+| `Security:Ldap:ServiceAccountPassword` | `LdapOptions` | `otopcua/ldap/service-account-password` |
+| `Security:DeployApiKey` | `DeployApiEndpoints` | `otopcua/deploy/api-key` |
+| `ConnectionStrings:ConfigDb` | `AddOtOpcUaConfigDb` | `otopcua/sql/configdb-connstr` |
+| `ServerHistorian:ApiKey` | `ServerHistorianOptions` | `otopcua/historian/api-key` |
+
+**Delivery.** By default these are delivered as plain environment variables (e.g.
+`ServerHistorian__ApiKey=histgw_…`), never committed. To deliver one from the encrypted
+store instead, seed it once with the `secret` CLI (from `ZB.MOM.WW.Secrets`), then supply
+the token in place of the literal — via env var or a deployment appsettings overlay:
+
+```
+secret set otopcua/historian/api-key <value>       # seed the encrypted store first
+ServerHistorian__ApiKey='${secret:otopcua/historian/api-key}'
+```
+
+Only switch a value to a `${secret:}` token **after** the secret is seeded — an unseeded
+token fails the boot fail-closed. Do not commit the KEK (`ZB_SECRETS_MASTER_KEY`) or any
+secret value.
+
+For the production posture needed so every clustered node (admin, driver, and any
+fused role) resolves the same secrets from the same store, see
+[`docs/operations/2026-07-16-secrets-clustered-master-key.md`](operations/2026-07-16-secrets-clustered-master-key.md).
+
+---
+
 ## Troubleshooting
 
 ### Certificate trust failure

docs/v2/Cluster.md

@@ -64,7 +64,7 @@ The Cluster.Tests project verifies these key values stay correct (`HoconLoaderTe
 - `Hostname`: interface to bind. `0.0.0.0` listens on every interface.
 - `Port`: TCP port for cluster gossip. Default 4053.
 - `PublicHostname`: address advertised in cluster gossip. Must be reachable by every other node.
-- `SeedNodes`: where new nodes go to join. List one (or two) stable nodes. First node bootstraps the cluster from its own address.
+- `SeedNodes`: where new nodes go to join. List one (or two) stable nodes. **Order is load-bearing:** Akka only lets `seed-nodes[0]` form a *new* cluster (`FirstSeedNodeProcess`); every other node retries `InitJoin` forever. A node that is one of its own seeds must therefore list ITSELF first, or it can never cold-start while its peer is down — enforced at boot by `AkkaClusterOptionsValidator`, explained in [Redundancy.md § Bootstrap: self-first seed ordering](../Redundancy.md#bootstrap-self-first-seed-ordering). Nodes seeded only by someone else (driver-only site nodes) are exempt.
 - `Roles`: free-form tags Akka gossip propagates. v2 uses `admin` + `driver`; per-role wiring in `Program.cs` reads `OTOPCUA_ROLES` env var, not this list — these two should stay in sync.
 
 Per-role overlay files (`appsettings.admin.json`, `appsettings.driver.json`, `appsettings.admin-driver.json`) layer on top of base `appsettings.json` based on the parsed `OTOPCUA_ROLES` (alphabetical, joined by `-`). See [ServiceHosting.md § Per-role configuration overlays](../ServiceHosting.md#per-role-configuration-overlays).

docs/v2/config-db-schema.md

@@ -148,10 +148,13 @@ CREATE TABLE dbo.ClusterNode (
     Host                            nvarchar(255)   NOT NULL,
     OpcUaPort                       int             NOT NULL DEFAULT 4840,
     DashboardPort                   int             NOT NULL DEFAULT 8081,
+    AkkaPort                        int             NOT NULL DEFAULT 4053,
+    GrpcPort                        int             NULL,
     ApplicationUri                  nvarchar(256)   NOT NULL,
     ServiceLevelBase                tinyint         NOT NULL DEFAULT 200,
     DriverConfigOverridesJson       nvarchar(max)   NULL CHECK (DriverConfigOverridesJson IS NULL OR ISJSON(DriverConfigOverridesJson) = 1),
     Enabled                         bit             NOT NULL DEFAULT 1,
+    MaintenanceMode                 bit             NOT NULL DEFAULT 0,
     LastSeenAt                      datetime2(3)    NULL,
     CreatedAt                       datetime2(3)    NOT NULL DEFAULT SYSUTCDATETIME(),
     CreatedBy                       nvarchar(128)   NOT NULL
@@ -167,6 +170,49 @@ CREATE UNIQUE INDEX UX_ClusterNode_Primary_Per_Cluster
     WHERE RedundancyRole = 'Primary';
 ```
 
+#### `AkkaPort` / `GrpcPort` — central's dial targets (per-cluster mesh Phase 1)
+
+These are **the addresses central dials**, not the node's own binding configuration. The node binds
+from `Cluster:Port` in its own appsettings; these columns duplicate that value for a reader that
+cannot see the node's config. Today central shares a gossip ring with every node and does not need
+them — [per-cluster mesh Phase 2](../plans/2026-07-21-per-cluster-mesh-design.md) splits the fleet
+into one mesh per `Cluster`, after which `Host` + `AkkaPort` is how central builds its ClusterClient
+contact points, and `GrpcPort` is the Phase 5 telemetry stream.
+
+`AkkaPort` is `NOT NULL DEFAULT 4053` because every node listens on a remoting port — `0` is never a
+truthful value, and rows predating the column must migrate to something real. `GrpcPort` is nullable
+with **no** default because nothing listens on it until Phase 5, and a non-null default would assert
+a port that does not exist.
+
+**The duplication against `Cluster:Port` is not enforced by the schema.** A node that binds 4054 while
+its row says 4053 is unreachable from central in Phase 2, and the symptom there is a silent absence of
+acks rather than an error. `ClusterNodeAddressReconcilerActor` (an admin-role singleton) compares the
+rows against observed cluster membership and logs an Error on mismatch. It reads membership rather
+than having each driver node assert its own row, because Phase 4 removes the driver nodes' ConfigDb
+connection entirely.
+
+#### `Enabled` vs `MaintenanceMode` — two flags, two questions
+
+Phase 1 made the rows the deploy path's **expected-ack set**: `ConfigPublishCoordinator` no longer
+derives it from cluster membership, so a row whose node is down now fails the deployment at the apply
+deadline instead of letting it seal green without that node.
+
+The escape hatch is **`MaintenanceMode = 1`**, not `Enabled = 0`:
+
+| Flag | Question it answers | Checked by |
+|---|---|---|
+| `Enabled` | Is this node part of the cluster's declared topology? | `DraftValidator.ValidateClusterTopology` — the enabled-node count must equal `ServerCluster.NodeCount` |
+| `MaintenanceMode` | Should we expect it to participate right now? | `ConfigPublishCoordinator` (expected-ack set) and `ClusterNodeAddressReconciler` (absence warnings) |
+
+`Enabled = 0` on one node of a Warm/Hot pair is **rejected at the deploy gate** with
+`ClusterEnabledNodeCountMismatch` — the enabled count drops to 1 while `NodeCount` stays 2. Since
+every cluster in the target topology is a 2-node pair, `Enabled` cannot serve as the maintenance
+hatch at all. The Phase 1 live gate found this; the two rules were independently reasonable and
+contradictory together, so the meanings were split rather than either rule being weakened.
+
+A node in maintenance is still enabled, so the topology gate still passes, and the runtime redundancy
+posture (`NodeCount` / `RedundancyMode`) is unchanged.
+
 `DriverConfigOverridesJson` shape:
 
 ```jsonc

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs

@@ -20,7 +20,31 @@ public sealed class AkkaClusterOptions
     /// </summary>
     public string PublicHostname { get; set; } = "127.0.0.1";
 
-    /// <summary>Gets or sets the seed nodes for cluster bootstrapping.</summary>
+    /// <summary>
+    /// Gets or sets the seed nodes for cluster bootstrapping.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b>ORDER IS LOAD-BEARING (decision 2026-07-22): a node that is one of its own seeds
+    ///         must list ITSELF first.</b> Akka runs <c>FirstSeedNodeProcess</c> — the only bootstrap
+    ///         path that can form a NEW cluster when no peer answers <c>InitJoin</c> — exclusively
+    ///         when <c>seed-nodes[0]</c> is this node's own address; any other node runs
+    ///         <c>JoinSeedNodeProcess</c>, which retries <c>InitJoin</c> forever and can never form a
+    ///         cluster. So listing both peers does NOT mean either can cold-start alone: a node that
+    ///         lists its partner first never comes Up while that partner is down. Self-first ordering
+    ///         closes that gap inside Akka's own handshake — unlike the retired
+    ///         <c>Cluster:SelfFormAfter</c> watchdog, which sat outside it and could not tell "no
+    ///         seed answered" from "a seed answered and the join is in flight".
+    ///     </para>
+    ///     <para>
+    ///         The rule is <b>conditional</b>: it binds only when this node's own address appears in
+    ///         this list. A node seeded exclusively by someone else — today's docker-dev site nodes,
+    ///         which list only <c>central-1</c> — is legitimately not a seed and is exempt. Enforced
+    ///         at boot by <see cref="AkkaClusterOptionsValidator"/>; identity is
+    ///         <see cref="PublicHostname"/> + <see cref="Port"/> (what Akka puts in
+    ///         <c>SelfAddress</c>), never the <see cref="Hostname"/> bind address.
+    ///     </para>
+    /// </remarks>
     public string[] SeedNodes { get; set; } = Array.Empty<string>();
 
     /// <summary>
@@ -29,4 +53,30 @@ public sealed class AkkaClusterOptions
     /// <c>admin</c>, <c>driver</c>, <c>dev</c>.
     /// </summary>
     public string[] Roles { get; set; } = Array.Empty<string>();
+
+    /// <summary>
+    /// How the cluster decides to down a node it can no longer reach. One of <c>auto-down</c>
+    /// (default) or <c>keep-oldest</c>; any other value fails the host at startup.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b><c>auto-down</c> — availability.</b> The leader among the <i>reachable</i> members
+    ///         downs the unreachable peer after
+    ///         <see cref="ServiceCollectionExtensions.DowningStableAfter"/>. A hard crash of either
+    ///         node — including the oldest — fails over to the survivor with no operator action.
+    ///         The trade is that a genuine network partition (both nodes alive, link cut) leaves
+    ///         both sides running active until an operator restarts one.
+    ///     </para>
+    ///     <para>
+    ///         <b><c>keep-oldest</c> — partition-safety.</b> The SBR resolver sacrifices the younger
+    ///         side of a split, so a partition can never run dual-active. <b>The cost is severe for a
+    ///         two-node pair:</b> it cannot survive a crash of the oldest node at all. Akka.NET's
+    ///         <c>KeepOldest.OldestDecision</c> only lets <c>down-if-alone</c> rescue a side holding
+    ///         &gt;= 2 members, so the 1-vs-1 survivor downs <i>itself</i> and shuts down — the
+    ///         redundancy pair turns a single-node crash into a total outage. Choose this only for
+    ///         clusters of three or more nodes, or where dual-active is genuinely worse than an
+    ///         outage.
+    ///     </para>
+    /// </remarks>
+    public string SplitBrainResolverStrategy { get; set; } = "auto-down";
 }

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptionsValidator.cs

@@ -0,0 +1,96 @@
+using Akka.Actor;
+using ZB.MOM.WW.Configuration;
+
+namespace ZB.MOM.WW.OtOpcUa.Cluster;
+
+/// <summary>
+///     Fail-fast startup validator for <see cref="AkkaClusterOptions"/>, built on the shared
+///     <c>ZB.MOM.WW.Configuration</c> <see cref="OptionsValidatorBase{TOptions}"/> and wired through
+///     <c>AddValidatedOptions</c> in <see cref="ServiceCollectionExtensions.AddOtOpcUaCluster"/>.
+/// </summary>
+/// <remarks>
+///     <para>
+///         <b>Self-first seed ordering (decision 2026-07-22).</b> Akka runs
+///         <c>FirstSeedNodeProcess</c> — the only bootstrap path that can form a NEW cluster when no
+///         peer answers <c>InitJoin</c> — exclusively when <c>seed-nodes[0]</c> is this node's own
+///         address. Every other node runs <c>JoinSeedNodeProcess</c> and retries <c>InitJoin</c>
+///         forever. A node that lists its partner first therefore cannot cold-start while that
+///         partner is down, and the failure is <i>silent</i>: the process runs, the port listens, the
+///         node simply never becomes a member. Enforced here so it is loud, at boot, on the node
+///         that has the problem.
+///     </para>
+///     <para>
+///         <b>The rule is conditional and must stay that way.</b> It binds only when this node's own
+///         address appears in its own seed list. A driver-only site node is seeded solely by
+///         <c>central-1</c> and is legitimately not a seed of anything; an unconditional "self must
+///         be first" rule would refuse to boot every one of them.
+///     </para>
+///     <para>
+///         <b>Identity is the advertised address, never the bind address.</b> In docker-dev
+///         <c>Cluster:Hostname</c> is <c>0.0.0.0</c> and <c>Cluster:PublicHostname</c> is the
+///         container DNS name — the value Akka puts in <c>SelfAddress</c> and the only one a seed URI
+///         can match. Comparing against <c>Hostname</c> would match nothing anywhere, silently exempt
+///         every node, and leave this rule inert. <c>PublicHostname</c> falling back to
+///         <c>Hostname</c> when blank mirrors Akka's own <c>public-hostname</c> semantics.
+///     </para>
+/// </remarks>
+public sealed class AkkaClusterOptionsValidator : OptionsValidatorBase<AkkaClusterOptions>
+{
+    /// <inheritdoc />
+    protected override void Validate(ValidationBuilder builder, AkkaClusterOptions options)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+
+        var seeds = options.SeedNodes ?? Array.Empty<string>();
+        if (seeds.Length == 0)
+        {
+            // Single-node / dev default: nothing is claimed about ordering, so there is nothing to
+            // enforce. (Akka bootstraps from an empty seed list by waiting for an explicit join.)
+            return;
+        }
+
+        var selfHost = string.IsNullOrWhiteSpace(options.PublicHostname)
+            ? options.Hostname
+            : options.PublicHostname;
+
+        var selfIndex = Array.FindIndex(seeds, s => IsSelf(s, selfHost, options.Port));
+        if (selfIndex <= 0)
+        {
+            // -1 = this node is not one of its own seeds (exempt); 0 = correctly ordered.
+            return;
+        }
+
+        builder.Add(
+            $"Cluster:SeedNodes must list this node itself first: entry {selfIndex} is this node "
+            + $"('{seeds[selfIndex]}') but entry 0 is '{seeds[0]}'. Akka only lets seed-nodes[0] form "
+            + "a new cluster (FirstSeedNodeProcess); every other node retries InitJoin forever, so "
+            + "with the partner listed first this node can never start while its peer is down. Swap "
+            + "the entries — see docs/Redundancy.md → 'Bootstrap: self-first seed ordering'.");
+    }
+
+    /// <summary>
+    /// True when <paramref name="seed"/> addresses this node itself — host AND port.
+    /// </summary>
+    /// <remarks>
+    /// Host comparison is case-insensitive because DNS names are, but is otherwise exact: Akka does
+    /// no DNS canonicalisation either, so <c>central-2</c> and <c>central-2.example.com</c> are
+    /// genuinely different seed identities to the cluster. A seed entry Akka itself cannot parse is
+    /// treated as "not this node" — reporting it is Akka's job, with Akka's wording, and this rule
+    /// must not turn a parse problem into a misleading ordering complaint.
+    /// </remarks>
+    private static bool IsSelf(string seed, string selfHost, int selfPort)
+    {
+        Address address;
+        try
+        {
+            address = Address.Parse(seed);
+        }
+        catch (Exception)
+        {
+            return false;
+        }
+
+        return address.Port == selfPort
+            && string.Equals(address.Host, selfHost, StringComparison.OrdinalIgnoreCase);
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Redundancy/ServiceLevelCalculator.cs

@@ -7,7 +7,7 @@ public readonly record struct NodeHealthInputs(
     bool DbReachable,
     bool OpcUaProbeOk,
     bool Stale,
-    bool IsDriverRoleLeader);
+    bool IsDriverPrimary);
 
 /// <summary>
 /// Pure ServiceLevel computation per design §6. Output range 0–255, where higher = "more
@@ -18,7 +18,9 @@ public readonly record struct NodeHealthInputs(
 /// - Member not Up/Joining: 0 (cluster cannot trust this node).
 /// - DB reachable + OPC UA probe ok + not stale: 240 (full service).
 /// - Stale config (DB reachable or not, OPC UA probe state ignored): 100 or 200 depending on DB.
-/// - +10 bonus when this node holds the role-leader lease for the "driver" role.
+/// - +10 bonus when this node is the driver Primary (the oldest Up member with the "driver" role —
+///   where the cluster singletons live). Previously this keyed off Akka's role *leader*, the
+///   lowest-addressed member, which names a different node once any node has restarted.
 /// </summary>
 public static class ServiceLevelCalculator
 {
@@ -38,6 +40,6 @@ public static class ServiceLevelCalculator
             _ => 0,
         };
 
-        return (byte)Math.Clamp(basis + (h.IsDriverRoleLeader ? 10 : 0), 0, 255);
+        return (byte)Math.Clamp(basis + (h.IsDriverPrimary ? 10 : 0), 0, 255);
     }
 }

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf

@@ -37,14 +37,26 @@ akka {
         roles = []
         min-nr-of-members = 1
 
-        # Split-brain resolver (arch-review 03/S1). This block IS active: Akka.Cluster.Hosting's
-        # WithClustering enables an SBR downing provider by default (it applies
-        # SplitBrainResolverOption.Default when ClusterOptions.SplitBrainResolver is null), and that
-        # provider reads this block. ServiceCollectionExtensions.BuildClusterOptions additionally sets
-        # the typed KeepOldestOption { DownIfAlone = true } to make the strategy EXPLICIT in code —
-        # its active-strategy + keep-oldest.down-if-alone MUST match this block. (The cluster is NOT
-        # NoDowning; only an explicit downing-provider-class = "" would disable failover.)
-        # stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin.
+        # DOWNING. The provider is NOT selected here — this file is added with HoconAddMode.Append,
+        # which loses to what Akka.Hosting's WithClustering emits, so a downing-provider-class set
+        # here would be silently overridden. Selection lives in
+        # ServiceCollectionExtensions.BuildDowningHocon (Prepended, which outranks WithClustering),
+        # driven by Cluster:SplitBrainResolverStrategy.
+        #
+        # The DEFAULT strategy is "auto-down", under which this whole block is inert: Akka's
+        # AutoDowning provider is installed with auto-down-unreachable-after = 15s, and the leader
+        # among the reachable members downs the unreachable peer — so a hard crash of EITHER node,
+        # oldest included, fails over. The trade is dual-active during a real network partition.
+        #
+        # The block below applies only when an operator selects "keep-oldest". Note that keep-oldest
+        # is NOT safe for a two-node pair: Akka.NET's KeepOldest.OldestDecision only lets
+        # down-if-alone rescue a side holding >= 2 members, so a 1-vs-1 survivor downs ITSELF and
+        # (with run-coordinated-shutdown-when-down) exits — a crash of the oldest node becomes a
+        # total outage. See docs/Redundancy.md.
+        #
+        # stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin, and
+        # must equal ServiceCollectionExtensions.DowningStableAfter so both strategies fail over at
+        # the same latency. Both constraints are pinned by SplitBrainResolverActivationTests.
         split-brain-resolver {
             active-strategy = "keep-oldest"
             stable-after = 15s

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs

@@ -7,6 +7,7 @@ using Akka.Remote.Hosting;
 using Microsoft.Extensions.Configuration;
 using Microsoft.Extensions.DependencyInjection;
 using Microsoft.Extensions.Options;
+using ZB.MOM.WW.Configuration;
 using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
 
 namespace ZB.MOM.WW.OtOpcUa.Cluster;
@@ -19,13 +20,19 @@ public static class ServiceCollectionExtensions
     /// configurator via <see cref="WithOtOpcUaClusterBootstrap"/> — keeping the entire Akka graph
     /// under Akka.Hosting's management so cluster singletons land on the same ActorSystem.
     /// </summary>
+    /// <remarks>
+    /// The binding is validated at startup (<c>ValidateOnStart</c>) by
+    /// <see cref="AkkaClusterOptionsValidator"/>, so a seed list that cannot bootstrap this node —
+    /// self listed behind its partner — fails the host loudly instead of leaving it running but
+    /// permanently outside the cluster.
+    /// </remarks>
     /// <param name="services">The service collection to configure.</param>
     /// <param name="configuration">The application configuration containing cluster options.</param>
     /// <returns>The same service collection, for chaining.</returns>
     public static IServiceCollection AddOtOpcUaCluster(this IServiceCollection services, IConfiguration configuration)
     {
-        services.AddOptions<AkkaClusterOptions>()
-            .Bind(configuration.GetSection(AkkaClusterOptions.SectionName));
+        services.AddValidatedOptions<AkkaClusterOptions, AkkaClusterOptionsValidator>(
+            configuration, AkkaClusterOptions.SectionName);
 
         services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
 
@@ -80,40 +87,133 @@ public static class ServiceCollectionExtensions
 
         builder.WithClustering(BuildClusterOptions(options));
 
+        // Must come AFTER WithClustering, which always emits a downing-provider-class of its own:
+        // Akka.Cluster.Hosting applies SplitBrainResolverOption.Default whenever the typed
+        // ClusterOptions.SplitBrainResolver is null, so there is always a competing value to beat.
+        // Prepend is the highest-precedence mode, which makes this block win irrespective of how
+        // many fragments are added or in what order.
+        //
+        // Do not take this comment's word for it. Which fragment actually wins is not obvious from
+        // the mode names — measured, Append also wins here, purely because it happens to be added
+        // last. SplitBrainResolverActivationTests reads the provider back off a *running*
+        // ActorSystem for exactly that reason: the effective value is the only one worth asserting.
+        builder.AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend);
+
+        // NOTE (2026-07-22): no bootstrap watchdog is armed here any more. The cold-start-alone gap
+        // it existed for — Akka lets only seed-nodes[0] form a NEW cluster — is now closed by
+        // self-first seed ordering (AkkaClusterOptions.SeedNodes), enforced at boot by
+        // AkkaClusterOptionsValidator. The retired ClusterBootstrapFallback timer sat OUTSIDE Akka's
+        // join handshake, so it could not distinguish "no seed answered" from "a seed answered and
+        // the join is in flight", and Cluster.Join(SelfAddress) is not ignored mid-handshake — it
+        // wins. See docs/Redundancy.md → "Bootstrap: self-first seed ordering".
+
         return builder;
     }
 
     /// <summary>
-    /// Builds the <see cref="ClusterOptions"/> for the fused-host cluster, setting the split-brain-resolver
-    /// strategy <b>explicitly in code</b>.
-    ///
-    /// <b>Activation note (arch-review 03/S1, corrected):</b> Akka.Cluster.Hosting's <c>WithClustering</c>
-    /// already enables an SBR downing provider <b>by default</b> — when <see cref="ClusterOptions.SplitBrainResolver"/>
-    /// is <c>null</c> it applies <c>SplitBrainResolverOption.Default</c>, which registers
-    /// <c>Akka.Cluster.SBR.SplitBrainResolverProvider</c> and reads the <c>split-brain-resolver</c> HOCON block in
-    /// <c>Resources/akka.conf</c>. So the cluster was <b>not</b> running <c>NoDowning</c> before this option was
-    /// set: the pre-existing akka.conf <c>keep-oldest</c> block was already active, and hard-crashed nodes already
-    /// failed over. Setting this typed option makes the strategy <b>explicit in code</b> (independent of the
-    /// framework default) rather than being the sole activator — it is reinforcing/belt-and-suspenders and
-    /// produces the same effective behavior. (Only an <i>explicit</i> <c>NoDowning</c>, e.g.
-    /// <c>downing-provider-class = ""</c>, would disable failover.)
-    ///
-    /// <see cref="KeepOldestOption"/> with <c>DownIfAlone=true</c> mirrors the HOCON intent and is the
-    /// correct strategy for a 2-node warm-redundancy pair: on an even split the oldest member (typically
-    /// the long-running primary) survives, and <c>down-if-alone</c> downs a singleton that loses its
-    /// peer. <c>keep-majority</c>/<c>static-quorum</c> are wrong for two nodes. The strategy + down-if-alone
-    /// here MUST stay consistent with the HOCON block; <c>stable-after</c> lives only in HOCON because the
-    /// typed option cannot express it (it must stay ≥ <c>failure-detector.acceptable-heartbeat-pause</c>).
+    /// How long a member must stay unreachable before it is downed. Shared by both strategies
+    /// (<c>auto-down-unreachable-after</c> and the SBR resolver's <c>stable-after</c>) so the two
+    /// have the same failover latency. Must stay above
+    /// <c>akka.cluster.failure-detector.acceptable-heartbeat-pause</c> or a merely-slow node gets
+    /// downed; both constraints are pinned by tests.
     /// </summary>
-    /// <param name="options">The bound cluster options carrying seed nodes and roles.</param>
-    /// <returns>The cluster options with seed nodes, roles, and the explicit split-brain-resolver strategy.</returns>
+    public static readonly TimeSpan DowningStableAfter = TimeSpan.FromSeconds(15);
+
+    /// <summary>
+    /// Builds the HOCON that selects the downing provider, per
+    /// <see cref="AkkaClusterOptions.SplitBrainResolverStrategy"/>.
+    /// </summary>
+    /// <param name="options">The bound cluster options carrying the strategy.</param>
+    /// <returns>
+    /// For <c>auto-down</c>, a block installing Akka's <c>AutoDowning</c> provider with a downing
+    /// window of <see cref="DowningStableAfter"/>. For <c>keep-oldest</c>, an empty string — the
+    /// typed <see cref="KeepOldestOption"/> from <see cref="BuildClusterOptions"/> already installs
+    /// the SBR provider, and the resolver's own settings live in <c>Resources/akka.conf</c>.
+    /// </returns>
+    /// <exception cref="ArgumentOutOfRangeException">The strategy is not a recognised value.</exception>
+    public static string BuildDowningHocon(AkkaClusterOptions options)
+    {
+        ArgumentNullException.ThrowIfNull(options);
+
+        if (IsAutoDown(options))
+        {
+            // auto-down-unreachable-after is load-bearing: it defaults to `off`, under which
+            // AutoDowning is installed but never downs anything — failover silently disabled.
+            return $$"""
+                akka.cluster {
+                    downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"
+                    auto-down-unreachable-after = {{(int)DowningStableAfter.TotalMilliseconds}}ms
+                }
+                """;
+        }
+
+        if (IsKeepOldest(options))
+        {
+            return string.Empty;
+        }
+
+        throw new ArgumentOutOfRangeException(
+            nameof(options),
+            options.SplitBrainResolverStrategy,
+            $"Unknown {AkkaClusterOptions.SectionName}:{nameof(AkkaClusterOptions.SplitBrainResolverStrategy)}. "
+            + "Expected 'auto-down' (default; survives a crash of either node) or 'keep-oldest' "
+            + "(partition-safe, but a two-node pair cannot survive a crash of the oldest node).");
+    }
+
+    private static bool IsAutoDown(AkkaClusterOptions options) =>
+        string.Equals(options.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase);
+
+    private static bool IsKeepOldest(AkkaClusterOptions options) =>
+        string.Equals(options.SplitBrainResolverStrategy, "keep-oldest", StringComparison.OrdinalIgnoreCase);
+
+    /// <summary>
+    /// Builds the <see cref="ClusterOptions"/> for the fused-host cluster.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         <b>Activation note (arch-review 03/S1, corrected twice).</b> Akka.Cluster.Hosting's
+    ///         <c>WithClustering</c> always installs a downing provider: when
+    ///         <see cref="ClusterOptions.SplitBrainResolver"/> is <c>null</c> it applies
+    ///         <c>SplitBrainResolverOption.Default</c>, which registers
+    ///         <c>Akka.Cluster.SBR.SplitBrainResolverProvider</c> and reads the
+    ///         <c>split-brain-resolver</c> block in <c>Resources/akka.conf</c>. The cluster is
+    ///         therefore never <c>NoDowning</c>. What this method chooses is only <i>which</i>
+    ///         provider — and under the default <c>auto-down</c> strategy the choice is not made
+    ///         here at all but by the Prepended HOCON in
+    ///         <see cref="WithOtOpcUaClusterBootstrap"/>, which outranks whatever
+    ///         <c>WithClustering</c> emits.
+    ///     </para>
+    ///     <para>
+    ///         <b>Why keep-oldest is no longer the default (2026-07-21).</b> The previous
+    ///         documentation here asserted that <see cref="KeepOldestOption"/> with
+    ///         <c>DownIfAlone=true</c> was "the correct strategy for a 2-node warm-redundancy pair"
+    ///         because <c>down-if-alone</c> would down a node that lost its peer. That is the
+    ///         opposite of what Akka.NET does. In <c>KeepOldest.OldestDecision</c> the
+    ///         <c>down-if-alone</c> branch requires the surviving side to hold &gt;= 2 members, so
+    ///         in a 1-vs-1 split the survivor falls through to <c>DownReachable</c> and downs
+    ///         <i>itself</i>; with <c>run-coordinated-shutdown-when-down = on</c> it then exits. A
+    ///         two-node pair running keep-oldest converts a crash of the oldest node into a total
+    ///         outage — precisely the failure redundancy exists to absorb. Live-proven on the sister
+    ///         project's rig; see <c>docs/Redundancy.md</c> and
+    ///         <see cref="AkkaClusterOptions.SplitBrainResolverStrategy"/> for the trade.
+    ///     </para>
+    /// </remarks>
+    /// <param name="options">The bound cluster options carrying seed nodes, roles and the strategy.</param>
+    /// <returns>
+    /// The cluster options with seed nodes and roles. <see cref="ClusterOptions.SplitBrainResolver"/>
+    /// is set only under <c>keep-oldest</c>, where the SBR provider is the one wanted; under
+    /// <c>auto-down</c> it is left null because the Prepended HOCON replaces the provider anyway,
+    /// and naming a resolver that is not in force would be misleading.
+    /// </returns>
     public static ClusterOptions BuildClusterOptions(AkkaClusterOptions options)
     {
+        ArgumentNullException.ThrowIfNull(options);
+
         return new ClusterOptions
         {
             SeedNodes = options.SeedNodes,
             Roles = options.Roles,
-            SplitBrainResolver = new KeepOldestOption { DownIfAlone = true },
+            SplitBrainResolver = IsKeepOldest(options) ? new KeepOldestOption { DownIfAlone = true } : null,
         };
     }
 }

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ZB.MOM.WW.OtOpcUa.Cluster.csproj

@@ -14,6 +14,10 @@
         <PackageReference Include="Akka.Remote.Hosting"/>
         <PackageReference Include="Microsoft.Extensions.Hosting"/>
         <PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions"/>
+        <!-- Shared options-validation primitives (OptionsValidatorBase / AddValidatedOptions), the
+             same seam the Host's Ldap/OpcUa/Historian validators use. Pulled in here because the
+             self-first seed-ordering rule belongs with the options it validates. -->
+        <PackageReference Include="ZB.MOM.WW.Configuration"/>
     </ItemGroup>
 
     <ItemGroup>

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Interfaces/NodeDiagnosticsSnapshot.cs

@@ -14,8 +14,16 @@ public sealed record DriverInstanceDiagnostics(
 /// Per-node diagnostics returned by <c>IFleetDiagnosticsClient</c>. Populated by the node's
 /// local <c>DriverHostActor</c> via a request/response over Akka.
 /// </summary>
+/// <param name="RunningFromCache">
+///     True when the node booted its configuration from the node-local artifact cache because the
+///     central ConfigDb was unreachable. Such a node looks entirely healthy — full address space,
+///     live values — but its configuration is frozen and no deployment can reach it, so the state
+///     needs to be explicitly visible rather than inferred. Defaults to false, which is also what
+///     every node that booted normally reports.
+/// </param>
 public sealed record NodeDiagnosticsSnapshot(
     NodeId NodeId,
     RevisionHash? CurrentRevision,
     IReadOnlyList<DriverInstanceDiagnostics> Drivers,
-    DateTime AsOfUtc);
+    DateTime AsOfUtc,
+    bool RunningFromCache = false);

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Alerts/AlarmTransitionEvent.cs

@@ -17,6 +17,7 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Alerts;
 /// <param name="AlarmTypeName">OPC UA Part 9 condition subtype name — one of <c>LimitAlarm</c> / <c>DiscreteAlarm</c> / <c>OffNormalAlarm</c> / <c>AlarmCondition</c> (the base type, used as the default). The historian feed maps this onto the durable alarm-type column.</param>
 /// <param name="Comment">Operator-supplied comment on ack / confirm / comment transitions; <c>null</c> for engine-driven transitions (Activated / Cleared / Shelved / …) that carry no comment.</param>
 /// <param name="HistorizeToAveva">When <c>false</c>, the durable historian sink suppresses this transition (the live <c>alerts</c> fan-out is unaffected); <c>null</c> or <c>true</c> historize. <c>null</c> is the cross-version/rolling-restart case: an old-format message missing the field deserializes to <c>null</c> (CLR default for <c>bool?</c>) and is historized (safe default-on), matching the <c>AlarmTypeName</c> null-coalesce in <c>HistorianAdapterActor.Translate</c>. The producer (<c>ScriptedAlarmHostActor</c>) always sets a concrete <c>true</c>/<c>false</c>.</param>
+/// <param name="ReferencingEquipmentPaths">v3 Batch 4 (multi-notifier native alarms) — the (possibly empty) list of UNS equipment-folder paths (<c>Area/Line/Equipment</c>) that reference the alarm's backing raw tag. A native alarm is a SINGLE Part 9 condition materialised once at the raw tag (its <see cref="AlarmId"/> is the RawPath); the same condition fans events to every referencing equipment's UNS folder via SDK notifiers, and this list is carried so <c>/alerts</c> shows the one condition row with all its referencing equipment as display metadata. Empty for scripted alarms (they are per-equipment) and for a native alarm whose raw tag no equipment references. Defaults empty so every existing producer + rolling-restart deserialization keeps compiling / working.</param>
 public sealed record AlarmTransitionEvent(
     string AlarmId,
     string EquipmentPath,
@@ -28,4 +29,5 @@ public sealed record AlarmTransitionEvent(
     DateTime TimestampUtc,
     string AlarmTypeName = "AlarmCondition",
     string? Comment = null,
-    bool? HistorizeToAveva = null);
+    bool? HistorizeToAveva = null,
+    IReadOnlyList<string>? ReferencingEquipmentPaths = null);

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Redundancy/NodeRedundancyState.cs

@@ -8,9 +8,20 @@ public enum RedundancyRole { Primary, Secondary, Detached }
 /// Snapshot of a single node's redundancy state. Aggregated by <c>RedundancyStateActor</c>
 /// to compute fleet-wide ServiceLevel.
 /// </summary>
+/// <param name="NodeId">Canonical <c>host:port</c> id of the node this entry describes.</param>
+/// <param name="Role">The node's redundancy role in the current snapshot.</param>
+/// <param name="IsClusterLeader">Whether the node is the Akka cluster leader.</param>
+/// <param name="IsDriverPrimary">
+/// Whether the node is the driver Primary — the <b>oldest</b> Up member carrying the <c>driver</c>
+/// role, which is where <c>ClusterSingletonManager</c> places singletons. Renamed from
+/// <c>IsRoleLeaderForDriver</c>: it was derived from <c>ClusterState.RoleLeader("driver")</c>, the
+/// lowest-<i>addressed</i> member, which diverges from the oldest after any node restart. The name
+/// now describes what the value means rather than how it used to be computed.
+/// </param>
+/// <param name="AsOfUtc">When the snapshot was computed.</param>
 public sealed record NodeRedundancyState(
     NodeId NodeId,
     RedundancyRole Role,
     bool IsClusterLeader,
-    bool IsRoleLeaderForDriver,
+    bool IsDriverPrimary,
     DateTime AsOfUtc);

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/AddressSpaceRealm.cs

@@ -0,0 +1,18 @@
+namespace ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
+
+/// <summary>
+///     Which of the two v3 OPC UA namespaces a node lives in. Carried explicitly alongside a
+///     node's <c>s=</c> identifier at every address-space sink seam so the namespace is chosen at
+///     the call site, never parsed back out of the id string (explicit beats inferred).
+///     <see cref="V3NodeIds.NamespaceUri"/> maps each realm to its namespace URI.
+/// </summary>
+public enum AddressSpaceRealm
+{
+    /// <summary>The device-oriented Raw tree (Folder→Driver→Device→TagGroup→Tag); a node's
+    /// <c>s=</c> id is its <see cref="ZB.MOM.WW.OtOpcUa.Commons.Types.RawPaths">RawPath</see>.</summary>
+    Raw,
+
+    /// <summary>The equipment-oriented UNS tree (Area→Line→Equipment→signal); a node's <c>s=</c>
+    /// id is the slash-joined <c>Area/Line/Equipment[/EffectiveName]</c>.</summary>
+    Uns,
+}

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/AlarmConditionSnapshot.cs

@@ -16,6 +16,12 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
 /// <param name="Shelving">The shelving mode (ShelvingState): unshelved, one-shot, or timed.</param>
 /// <param name="Severity">OPC UA severity on the 1..1000 scale (the SDK <c>SetSeverity</c> input).</param>
 /// <param name="Message">The human-readable condition message (LocalizedText payload).</param>
+/// <param name="Quality">
+///     Quality of the condition's source data (Part 9 <c>ConditionType.Quality</c>). Carried so a
+///     comms-lost native source can report a non-<c>Good</c> condition instead of the accidentally-Good
+///     default (issue #477). It is a pure annotation — it never alters Active/Acked/Retain. Defaults to
+///     <see cref="OpcUaQuality.Good"/> so scripted-alarm callers (which stay Good in v1) need not supply it.
+/// </param>
 public sealed record AlarmConditionSnapshot(
     bool Active,
     bool Acknowledged,
@@ -23,7 +29,8 @@ public sealed record AlarmConditionSnapshot(
     bool Enabled,
     AlarmShelvingKind Shelving,
     ushort Severity,
-    string Message);
+    string Message,
+    OpcUaQuality Quality = OpcUaQuality.Good);
 
 /// <summary>
 ///     Commons-local mirror of the Core <c>ShelvingKind</c> enum so this assembly carries no

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/DeferredAddressSpaceSink.cs

@@ -23,30 +23,44 @@ public sealed class DeferredAddressSpaceSink : IOpcUaAddressSpaceSink, ISurgical
         _inner = sink ?? NullOpcUaAddressSpaceSink.Instance;
 
     /// <inheritdoc />
-    public void WriteValue(string nodeId, object? value, OpcUaQuality quality, DateTime sourceTimestampUtc)
-        => _inner.WriteValue(nodeId, value, quality, sourceTimestampUtc);
+    public void WriteValue(string nodeId, object? value, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
+        => _inner.WriteValue(nodeId, value, quality, sourceTimestampUtc, realm);
 
     /// <inheritdoc />
-    public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc)
-        => _inner.WriteAlarmCondition(alarmNodeId, state, sourceTimestampUtc);
+    public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
+        => _inner.WriteAlarmCondition(alarmNodeId, state, sourceTimestampUtc, realm);
+
+    public void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm)
+        => _inner.WriteAlarmQuality(alarmNodeId, quality, sourceTimestampUtc, realm);
 
     /// <inheritdoc />
-    public void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, bool isNative = false)
-        => _inner.MaterialiseAlarmCondition(alarmNodeId, equipmentNodeId, displayName, alarmType, severity, isNative);
+    public void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, AddressSpaceRealm realm, bool isNative = false)
+        => _inner.MaterialiseAlarmCondition(alarmNodeId, equipmentNodeId, displayName, alarmType, severity, realm, isNative);
 
     /// <inheritdoc />
-    public void EnsureFolder(string folderNodeId, string? parentNodeId, string displayName)
-        => _inner.EnsureFolder(folderNodeId, parentNodeId, displayName);
+    // Forward the WP4 multi-notifier wiring to the inner sink. Without this the native-alarm fan-out to the
+    // referencing equipment folders ships INERT on every driver-role host (actors inject THIS wrapper, not the
+    // inner SdkAddressSpaceSink) — the F10b / PR#423 forwarding trap the reflection guard exists to catch.
+    public void WireAlarmNotifiers(string alarmNodeId, AddressSpaceRealm alarmRealm, IReadOnlyList<string> notifierFolderNodeIds, AddressSpaceRealm notifierFolderRealm)
+        => _inner.WireAlarmNotifiers(alarmNodeId, alarmRealm, notifierFolderNodeIds, notifierFolderRealm);
 
     /// <inheritdoc />
-    public void EnsureVariable(string variableNodeId, string? parentFolderNodeId, string displayName, string dataType, bool writable, string? historianTagname = null, bool isArray = false, uint? arrayLength = null)
-        => _inner.EnsureVariable(variableNodeId, parentFolderNodeId, displayName, dataType, writable, historianTagname, isArray, arrayLength);
+    public void EnsureFolder(string folderNodeId, string? parentNodeId, string displayName, AddressSpaceRealm realm)
+        => _inner.EnsureFolder(folderNodeId, parentNodeId, displayName, realm);
+
+    /// <inheritdoc />
+    public void EnsureVariable(string variableNodeId, string? parentFolderNodeId, string displayName, string dataType, bool writable, AddressSpaceRealm realm, string? historianTagname = null, bool isArray = false, uint? arrayLength = null)
+        => _inner.EnsureVariable(variableNodeId, parentFolderNodeId, displayName, dataType, writable, realm, historianTagname, isArray, arrayLength);
 
     /// <inheritdoc />
     public void RebuildAddressSpace() => _inner.RebuildAddressSpace();
 
     /// <inheritdoc />
-    public void RaiseNodesAddedModelChange(string affectedNodeId) => _inner.RaiseNodesAddedModelChange(affectedNodeId);
+    public void RaiseNodesAddedModelChange(string affectedNodeId, AddressSpaceRealm realm) => _inner.RaiseNodesAddedModelChange(affectedNodeId, realm);
+
+    /// <inheritdoc />
+    public void AddReference(string sourceNodeId, AddressSpaceRealm sourceRealm, string targetNodeId, AddressSpaceRealm targetRealm, string referenceType = "Organizes")
+        => _inner.AddReference(sourceNodeId, sourceRealm, targetNodeId, targetRealm, referenceType);
 
     /// <inheritdoc />
     // Forwards to the inner sink when it supports the surgical capability; returns false otherwise —
@@ -55,9 +69,9 @@ public sealed class DeferredAddressSpaceSink : IOpcUaAddressSpaceSink, ISurgical
     // Without this forward the surgical optimization is inert on every driver-role host, because
     // actors inject THIS wrapper, not the inner sink. ALL six args (including the DataType/array-shape
     // ones) MUST be forwarded — a partial forward silently drops the shape update on every driver-role host.
-    public bool UpdateTagAttributes(string variableNodeId, bool writable, string? historianTagname, string dataType, bool isArray, uint? arrayLength)
+    public bool UpdateTagAttributes(string variableNodeId, bool writable, string? historianTagname, string dataType, bool isArray, uint? arrayLength, AddressSpaceRealm realm)
         => _inner is ISurgicalAddressSpaceSink surgical
-           && surgical.UpdateTagAttributes(variableNodeId, writable, historianTagname, dataType, isArray, arrayLength);
+           && surgical.UpdateTagAttributes(variableNodeId, writable, historianTagname, dataType, isArray, arrayLength, realm);
 
     /// <inheritdoc />
     // Forwards to the inner sink when it supports the surgical capability; returns false otherwise —
@@ -65,9 +79,9 @@ public sealed class DeferredAddressSpaceSink : IOpcUaAddressSpaceSink, ISurgical
     // sink that isn't surgical — so the caller (AddressSpaceApplier) falls back to a full rebuild.
     // Without this forward the rename-refresh optimization is inert on every driver-role host, because
     // actors inject THIS wrapper, not the inner sink.
-    public bool UpdateFolderDisplayName(string folderNodeId, string displayName)
+    public bool UpdateFolderDisplayName(string folderNodeId, string displayName, AddressSpaceRealm realm)
         => _inner is ISurgicalAddressSpaceSink surgical
-           && surgical.UpdateFolderDisplayName(folderNodeId, displayName);
+           && surgical.UpdateFolderDisplayName(folderNodeId, displayName, realm);
 
     // R2-07 T7 — surgical remove forwards. Same capability-sniffing pattern as UpdateTagAttributes /
     // UpdateFolderDisplayName: forward when the inner sink supports the surgical capability; return false
@@ -75,14 +89,14 @@ public sealed class DeferredAddressSpaceSink : IOpcUaAddressSpaceSink, ISurgical
     // (AddressSpaceApplier) falls back to a full rebuild. Without these forwards the surgical remove path is
     // inert on every driver-role host — the F10b / PR#423 trap the reflection guard exists to catch.
     /// <inheritdoc />
-    public bool RemoveVariableNode(string variableNodeId)
-        => _inner is ISurgicalAddressSpaceSink surgical && surgical.RemoveVariableNode(variableNodeId);
+    public bool RemoveVariableNode(string variableNodeId, AddressSpaceRealm realm)
+        => _inner is ISurgicalAddressSpaceSink surgical && surgical.RemoveVariableNode(variableNodeId, realm);
 
     /// <inheritdoc />
-    public bool RemoveAlarmConditionNode(string alarmNodeId)
-        => _inner is ISurgicalAddressSpaceSink surgical && surgical.RemoveAlarmConditionNode(alarmNodeId);
+    public bool RemoveAlarmConditionNode(string alarmNodeId, AddressSpaceRealm realm)
+        => _inner is ISurgicalAddressSpaceSink surgical && surgical.RemoveAlarmConditionNode(alarmNodeId, realm);
 
     /// <inheritdoc />
-    public bool RemoveEquipmentSubtree(string equipmentNodeId)
-        => _inner is ISurgicalAddressSpaceSink surgical && surgical.RemoveEquipmentSubtree(equipmentNodeId);
+    public bool RemoveEquipmentSubtree(string equipmentNodeId, AddressSpaceRealm realm)
+        => _inner is ISurgicalAddressSpaceSink surgical && surgical.RemoveEquipmentSubtree(equipmentNodeId, realm);
 }

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/EquipmentNodeIds.cs

@@ -1,31 +0,0 @@
-namespace ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
-
-/// <summary>
-///     Single source of truth for equipment-namespace OPC UA NodeId strings. The variable NodeId is
-///     FOLDER-SCOPED (<c>{parent}/{Name}</c>), NOT the driver-side FullName — a driver wire ref is not
-///     unique across identical machines, so FullName-as-NodeId would collide in the sink. Used by the
-///     materialiser (AddressSpaceApplier), the VirtualTag publish map, and the driver live-value router so all
-///     three agree on the exact NodeId a variable was placed at.
-/// </summary>
-public static class EquipmentNodeIds
-{
-    /// <summary>The sub-folder NodeId under an equipment for a non-empty FolderPath: <c>{equipmentId}/{folderPath}</c>.</summary>
-    /// <param name="equipmentId">The owning equipment's NodeId.</param>
-    /// <param name="folderPath">The tag/vtag FolderPath (must be non-empty for this to be meaningful).</param>
-    /// <returns>The sub-folder NodeId string.</returns>
-    public static string SubFolder(string equipmentId, string folderPath) => $"{equipmentId}/{folderPath}";
-
-    /// <summary>
-    ///     The folder-scoped variable NodeId: <c>{parent}/{name}</c> where <c>parent = equipmentId</c> when
-    ///     <paramref name="folderPath"/> is null/empty, else <see cref="SubFolder"/>.
-    /// </summary>
-    /// <param name="equipmentId">The owning equipment's NodeId.</param>
-    /// <param name="folderPath">The tag/vtag FolderPath, or null/empty for "directly under the equipment".</param>
-    /// <param name="name">The tag/vtag Name (the leaf browse segment).</param>
-    /// <returns>The folder-scoped variable NodeId string.</returns>
-    public static string Variable(string equipmentId, string? folderPath, string name)
-    {
-        var parent = string.IsNullOrWhiteSpace(folderPath) ? equipmentId : SubFolder(equipmentId, folderPath);
-        return $"{parent}/{name}";
-    }
-}

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IOpcUaAddressSpaceSink.cs

@@ -13,7 +13,11 @@ public interface IOpcUaAddressSpaceSink
     /// <param name="value">The value to write.</param>
     /// <param name="quality">The quality status of the value.</param>
     /// <param name="sourceTimestampUtc">The source timestamp in UTC.</param>
-    void WriteValue(string nodeId, object? value, OpcUaQuality quality, DateTime sourceTimestampUtc);
+    /// <param name="realm">Which of the two v3 namespaces (<see cref="AddressSpaceRealm.Raw"/> device tree or
+    /// <see cref="AddressSpaceRealm.Uns"/> equipment tree) the <paramref name="nodeId"/> lives in — the sink
+    /// resolves the namespace index from the realm rather than parsing it out of the id string. WP3's fan-out
+    /// calls this once per NodeId (raw + every referencing UNS node) with the matching realm.</param>
+    void WriteValue(string nodeId, object? value, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm);
 
     /// <summary>Write an alarm-condition's full Part 9 state. When a real condition node has been
     /// materialised for <paramref name="alarmNodeId"/> via <see cref="MaterialiseAlarmCondition"/>,
@@ -25,7 +29,20 @@ public interface IOpcUaAddressSpaceSink
     /// <param name="alarmNodeId">The OPC UA node ID of the alarm (== ScriptedAlarmId for materialised conditions).</param>
     /// <param name="state">The full condition state to project onto the node.</param>
     /// <param name="sourceTimestampUtc">The source timestamp in UTC.</param>
-    void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc);
+    /// <param name="realm">The namespace realm <paramref name="alarmNodeId"/> lives in (must match the realm
+    /// the condition was materialised under).</param>
+    void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm);
+
+    /// <summary>#477 — annotate a materialised condition's source-data quality OUT OF BAND from any alarm
+    /// transition (used by the driver-connectivity path: comms lost → <see cref="OpcUaQuality.Bad"/>,
+    /// restored → <see cref="OpcUaQuality.Good"/>). Sets ONLY the condition's Quality — never
+    /// Active/Acked/Severity/Retain (a comms-lost active alarm stays active) — and fires one Part 9 event
+    /// only on a quality-bucket change. A no-op for an unmaterialised / non-condition node.</summary>
+    /// <param name="alarmNodeId">The condition node id (RawPath for a native alarm).</param>
+    /// <param name="quality">The source-data quality to annotate.</param>
+    /// <param name="sourceTimestampUtc">The connectivity transition timestamp in UTC.</param>
+    /// <param name="realm">The namespace realm the condition was materialised under.</param>
+    void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm);
 
     /// <summary>
     ///     Materialise a real OPC UA Part 9 alarm-condition node under its equipment folder so clients
@@ -41,7 +58,31 @@ public interface IOpcUaAddressSpaceSink
     /// <param name="isNative">When true this is a driver-fed (native, e.g. Galaxy) equipment-tag alarm; when
     /// false (default) it is a scripted alarm. The node manager tracks native condition node ids so a later
     /// task can route a native condition's inbound Acknowledge to the driver instead of the scripted engine.</param>
-    void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, bool isNative = false);
+    /// <param name="realm">The namespace realm the condition (and its parent folder <paramref name="equipmentNodeId"/>)
+    /// live in.</param>
+    void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, AddressSpaceRealm realm, bool isNative = false);
+
+    /// <summary>
+    ///     v3 Batch 4 (multi-notifier native alarms) — wire an already-materialised native alarm condition
+    ///     (<paramref name="alarmNodeId"/>, materialised at the raw tag via <see cref="MaterialiseAlarmCondition"/>
+    ///     with ConditionId = the RawPath) as an event notifier of EACH referencing equipment's UNS folder, so
+    ///     the condition's <b>single</b> <c>ReportEvent</c> fans one event to every referencing equipment root
+    ///     WITHOUT re-reporting per root (distinct EventIds would break Server-object dedup + Part 9 ack
+    ///     correlation). Per folder the sink wires the SDK's bidirectional notifier pattern
+    ///     (<c>alarm.AddNotifier(isInverse:true, folder)</c> + <c>folder.AddNotifier(isInverse:false, alarm)</c>)
+    ///     and promotes the folder to an event notifier. <b>Idempotent</b> (a re-wire of the same pair updates,
+    ///     never duplicates); a <b>missing endpoint is a no-op</b> (logged, never thrown) so a mid-rebuild race
+    ///     can't fault a deploy. The sink tracks each wired pair so a later rebuild / condition-removal /
+    ///     equipment-subtree-removal tears the notifier down bidirectionally (no inverse-notifier entry leaks
+    ///     across redeploys).
+    /// </summary>
+    /// <param name="alarmNodeId">The native alarm condition's node id (== the backing tag's RawPath).</param>
+    /// <param name="alarmRealm">The namespace realm the condition lives in (Raw for native alarms).</param>
+    /// <param name="notifierFolderNodeIds">The equipment folder node ids (their <c>s=</c> ids) to wire as
+    /// extra event-notifier roots for this condition. An unknown / not-yet-materialised folder id is skipped.</param>
+    /// <param name="notifierFolderRealm">The namespace realm the notifier folders live in (Uns for equipment folders).</param>
+    void WireAlarmNotifiers(string alarmNodeId, AddressSpaceRealm alarmRealm,
+                            IReadOnlyList<string> notifierFolderNodeIds, AddressSpaceRealm notifierFolderRealm);
 
     /// <summary>
     ///     Ensure a folder node exists under the given parent. Used by <c>AddressSpaceApplier</c> to
@@ -52,7 +93,8 @@ public interface IOpcUaAddressSpaceSink
     /// <param name="folderNodeId">The OPC UA node ID for the folder.</param>
     /// <param name="parentNodeId">The parent folder node ID, or null for namespace root.</param>
     /// <param name="displayName">The display name for the folder.</param>
-    void EnsureFolder(string folderNodeId, string? parentNodeId, string displayName);
+    /// <param name="realm">The namespace realm the folder (and its <paramref name="parentNodeId"/>) live in.</param>
+    void EnsureFolder(string folderNodeId, string? parentNodeId, string displayName, AddressSpaceRealm realm);
 
     /// <summary>
     ///     Ensure a Variable node exists at <paramref name="variableNodeId"/>, parented under
@@ -77,7 +119,10 @@ public interface IOpcUaAddressSpaceSink
     /// rank + dimensions carry the array-ness.</param>
     /// <param name="arrayLength">The declared length of the 1-D array when <paramref name="isArray"/> is
     /// true; ignored for scalars. Null ⇒ length 0 (unbounded).</param>
-    void EnsureVariable(string variableNodeId, string? parentFolderNodeId, string displayName, string dataType, bool writable, string? historianTagname = null, bool isArray = false, uint? arrayLength = null);
+    /// <param name="realm">The namespace realm the variable (and its <paramref name="parentFolderNodeId"/>) live
+    /// in. A historized UNS reference node registers the SAME <paramref name="historianTagname"/> as its backing
+    /// raw node — both NodeIds map to one tagname — so this call carries the shared tagname per realm.</param>
+    void EnsureVariable(string variableNodeId, string? parentFolderNodeId, string displayName, string dataType, bool writable, AddressSpaceRealm realm, string? historianTagname = null, bool isArray = false, uint? arrayLength = null);
 
     /// <summary>
     ///     Tear down + repopulate the address space. Called by <c>OpcUaPublishActor</c> after a
@@ -91,7 +136,28 @@ public interface IOpcUaAddressSpaceSink
     ///     (Part 3 GeneralModelChangeEvent, verb NodeAdded).
     /// </summary>
     /// <param name="affectedNodeId">The node under which discovered nodes were added.</param>
-    void RaiseNodesAddedModelChange(string affectedNodeId);
+    /// <param name="realm">The namespace realm <paramref name="affectedNodeId"/> lives in.</param>
+    void RaiseNodesAddedModelChange(string affectedNodeId, AddressSpaceRealm realm);
+
+    /// <summary>
+    ///     Add an OPC UA reference from an already-materialised source node to an already-materialised
+    ///     target node, both realm-qualified. Used by <c>AddressSpaceApplier</c> to link each UNS reference
+    ///     Variable (source, <see cref="AddressSpaceRealm.Uns"/>) to its backing Raw node (target,
+    ///     <see cref="AddressSpaceRealm.Raw"/>) with an <c>Organizes</c> edge, so the UNS variable
+    ///     browse-resolves to its raw source (and the raw node back via the inverse). The edge is wired
+    ///     bidirectionally (forward on the source, inverse on the target) so browse resolves both ways.
+    ///     <b>Idempotent</b> (an existing edge is not duplicated); a <b>missing endpoint is a no-op</b>
+    ///     (logged, never thrown) so a mid-rebuild race can't fault a deploy.
+    /// </summary>
+    /// <param name="sourceNodeId">The source node's <c>s=</c> id (the referencing node — e.g. the UNS variable).</param>
+    /// <param name="sourceRealm">The namespace realm <paramref name="sourceNodeId"/> lives in.</param>
+    /// <param name="targetNodeId">The target node's <c>s=</c> id (the referenced node — e.g. the backing Raw node).</param>
+    /// <param name="targetRealm">The namespace realm <paramref name="targetNodeId"/> lives in.</param>
+    /// <param name="referenceType">The hierarchical reference type name — <c>Organizes</c> (default),
+    /// <c>HasComponent</c>, or <c>HasProperty</c>; unknown names fall back to <c>Organizes</c>.</param>
+    void AddReference(string sourceNodeId, AddressSpaceRealm sourceRealm,
+                      string targetNodeId, AddressSpaceRealm targetRealm,
+                      string referenceType = "Organizes");
 }
 
 /// <summary>OPC UA status code projection — Good / Uncertain / Bad. Real SDK has finer-grained
@@ -106,23 +172,30 @@ public sealed class NullOpcUaAddressSpaceSink : IOpcUaAddressSpaceSink
     private NullOpcUaAddressSpaceSink() { }
 
     /// <inheritdoc />
-    public void WriteValue(string nodeId, object? value, OpcUaQuality quality, DateTime sourceTimestampUtc) { }
+    public void WriteValue(string nodeId, object? value, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
 
     /// <inheritdoc />
-    public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc) { }
+    public void WriteAlarmCondition(string alarmNodeId, AlarmConditionSnapshot state, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
+    public void WriteAlarmQuality(string alarmNodeId, OpcUaQuality quality, DateTime sourceTimestampUtc, AddressSpaceRealm realm) { }
 
     /// <inheritdoc />
-    public void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, bool isNative = false) { }
+    public void MaterialiseAlarmCondition(string alarmNodeId, string equipmentNodeId, string displayName, string alarmType, int severity, AddressSpaceRealm realm, bool isNative = false) { }
 
     /// <inheritdoc />
-    public void EnsureFolder(string folderNodeId, string? parentNodeId, string displayName) { }
+    public void WireAlarmNotifiers(string alarmNodeId, AddressSpaceRealm alarmRealm, IReadOnlyList<string> notifierFolderNodeIds, AddressSpaceRealm notifierFolderRealm) { }
 
     /// <inheritdoc />
-    public void EnsureVariable(string variableNodeId, string? parentFolderNodeId, string displayName, string dataType, bool writable, string? historianTagname = null, bool isArray = false, uint? arrayLength = null) { }
+    public void EnsureFolder(string folderNodeId, string? parentNodeId, string displayName, AddressSpaceRealm realm) { }
+
+    /// <inheritdoc />
+    public void EnsureVariable(string variableNodeId, string? parentFolderNodeId, string displayName, string dataType, bool writable, AddressSpaceRealm realm, string? historianTagname = null, bool isArray = false, uint? arrayLength = null) { }
 
     /// <inheritdoc />
     public void RebuildAddressSpace() { }
 
     /// <inheritdoc />
-    public void RaiseNodesAddedModelChange(string affectedNodeId) { }
+    public void RaiseNodesAddedModelChange(string affectedNodeId, AddressSpaceRealm realm) { }
+
+    /// <inheritdoc />
+    public void AddReference(string sourceNodeId, AddressSpaceRealm sourceRealm, string targetNodeId, AddressSpaceRealm targetRealm, string referenceType = "Organizes") { }
 }

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/IOpcUaNodeWriteGateway.cs

@@ -11,11 +11,17 @@ public interface IOpcUaNodeWriteGateway
 {
     /// <summary>Route a write of <paramref name="value"/> to the driver backing node
     /// <paramref name="nodeId"/>; the returned task resolves with the device-write outcome.</summary>
-    /// <param name="nodeId">The folder-scoped equipment-variable node id being written.</param>
+    /// <param name="nodeId">The full ns-qualified node id being written (<c>node.NodeId.ToString()</c>).</param>
     /// <param name="value">The value the client wrote.</param>
+    /// <param name="realm">Which of the two v3 namespaces (<see cref="AddressSpaceRealm.Raw"/> device tree or
+    /// <see cref="AddressSpaceRealm.Uns"/> equipment tree) <paramref name="nodeId"/> lives in — the node
+    /// manager resolves it from the node's namespace index (<c>RealmOf</c>). It is REQUIRED for correct
+    /// routing: a raw <c>s=&lt;RawPath&gt;</c> and a UNS <c>s=&lt;Area/Line/Equip/Eff&gt;</c> can collide as bare
+    /// strings, so the routing map is keyed by <c>(realm, bareId)</c> — dropping the realm would let an
+    /// operator write route to the WRONG driver ref.</param>
     /// <param name="ct">Cancellation token.</param>
     /// <returns>A task resolving to the device-write outcome.</returns>
-    Task<NodeWriteOutcome> WriteAsync(string nodeId, object? value, CancellationToken ct);
+    Task<NodeWriteOutcome> WriteAsync(string nodeId, object? value, AddressSpaceRealm realm, CancellationToken ct);
 }
 
 /// <summary>Outcome of routing an inbound node write to the backing driver.</summary>
@@ -31,6 +37,6 @@ public sealed class NullOpcUaNodeWriteGateway : IOpcUaNodeWriteGateway
     public static readonly NullOpcUaNodeWriteGateway Instance = new();
     private NullOpcUaNodeWriteGateway() { }
     /// <inheritdoc />
-    public Task<NodeWriteOutcome> WriteAsync(string nodeId, object? value, CancellationToken ct) =>
+    public Task<NodeWriteOutcome> WriteAsync(string nodeId, object? value, AddressSpaceRealm realm, CancellationToken ct) =>
         Task.FromResult(new NodeWriteOutcome(false, "writes unavailable"));
 }

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/ISurgicalAddressSpaceSink.cs

@@ -21,8 +21,10 @@ public interface ISurgicalAddressSpaceSink
     /// <param name="dataType">The OPC UA built-in data type name (e.g. "Boolean", "Int32", "Float").</param>
     /// <param name="isArray">When true the node is a 1-D array (ValueRank=OneDimension); when false scalar.</param>
     /// <param name="arrayLength">The declared length of the 1-D array when <paramref name="isArray"/> is true; ignored for scalars.</param>
+    /// <param name="realm">The namespace realm <paramref name="variableNodeId"/> lives in — the sink resolves the
+    /// namespace index from the realm rather than parsing it out of the id string.</param>
     /// <returns>True when the in-place update was applied; false when the node is missing (caller rebuilds).</returns>
-    bool UpdateTagAttributes(string variableNodeId, bool writable, string? historianTagname, string dataType, bool isArray, uint? arrayLength);
+    bool UpdateTagAttributes(string variableNodeId, bool writable, string? historianTagname, string dataType, bool isArray, uint? arrayLength, AddressSpaceRealm realm);
 
     /// <summary>Update an existing folder node's <c>DisplayName</c> IN PLACE, notifying subscribers
     /// (ClearChangeMasks) without a rebuild — used by AddressSpaceApplier to apply a UNS Area / Line
@@ -32,8 +34,9 @@ public interface ISurgicalAddressSpaceSink
     /// should rebuild instead).</summary>
     /// <param name="folderNodeId">The node id of the folder whose display name to update in place.</param>
     /// <param name="displayName">The new display name to apply.</param>
+    /// <param name="realm">The namespace realm <paramref name="folderNodeId"/> lives in.</param>
     /// <returns>True when the in-place update was applied; false when the folder is missing (caller rebuilds).</returns>
-    bool UpdateFolderDisplayName(string folderNodeId, string displayName);
+    bool UpdateFolderDisplayName(string folderNodeId, string displayName, AddressSpaceRealm realm);
 
     /// <summary>Remove ONE existing value-variable node IN PLACE (detach from its parent, drop it from the
     /// predefined-node set, drop any historized-tagname registration), then raise a Part 3
@@ -43,8 +46,9 @@ public interface ISurgicalAddressSpaceSink
     /// re-subscription attempts get BadNodeIdUnknown from the SDK. Returns false when the node id is unknown
     /// (the node-manager maps drifted from the plan — caller falls back to a full rebuild).</summary>
     /// <param name="variableNodeId">The folder-scoped node id of the variable to remove in place.</param>
+    /// <param name="realm">The namespace realm <paramref name="variableNodeId"/> lives in.</param>
     /// <returns>True when the node was removed; false when the id is unknown (caller rebuilds).</returns>
-    bool RemoveVariableNode(string variableNodeId);
+    bool RemoveVariableNode(string variableNodeId, AddressSpaceRealm realm);
 
     /// <summary>Remove ONE existing Part 9 alarm-condition node IN PLACE (detach, drop from the
     /// predefined-node set, clear the native-alarm flag), then raise NodeDeleted outside the lock. The
@@ -52,8 +56,9 @@ public interface ISurgicalAddressSpaceSink
     /// stops replaying on ConditionRefresh. Returns false when the condition id is unknown (caller
     /// rebuilds).</summary>
     /// <param name="alarmNodeId">The alarm condition node id (== ScriptedAlarmId, or a native alarm tag's folder-scoped id).</param>
+    /// <param name="realm">The namespace realm <paramref name="alarmNodeId"/> lives in.</param>
     /// <returns>True when the condition was removed; false when the id is unknown (caller rebuilds).</returns>
-    bool RemoveAlarmConditionNode(string alarmNodeId);
+    bool RemoveAlarmConditionNode(string alarmNodeId, AddressSpaceRealm realm);
 
     /// <summary>Remove an equipment folder AND its entire descendant subtree (sub-folders, value variables,
     /// condition nodes) IN PLACE: per-descendant map/registration cleanup (variables, historized tagnames,
@@ -62,6 +67,7 @@ public interface ISurgicalAddressSpaceSink
     /// the equipment folder outside the lock. Returns false when the folder id is unknown (caller
     /// rebuilds).</summary>
     /// <param name="equipmentNodeId">The equipment folder node id whose entire subtree to remove.</param>
+    /// <param name="realm">The namespace realm <paramref name="equipmentNodeId"/> lives in.</param>
     /// <returns>True when the subtree was removed; false when the id is unknown (caller rebuilds).</returns>
-    bool RemoveEquipmentSubtree(string equipmentNodeId);
+    bool RemoveEquipmentSubtree(string equipmentNodeId, AddressSpaceRealm realm);
 }

src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/V3NodeIds.cs

@@ -0,0 +1,110 @@
+using ZB.MOM.WW.OtOpcUa.Commons.Types;
+
+namespace ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
+
+/// <summary>
+///     v3 dual-namespace NodeId + namespace-URI authority. Two OPC UA namespaces expose the same
+///     underlying values under two identity schemes:
+///     <list type="bullet">
+///       <item><b>Raw</b> (<see cref="RawNamespaceUri"/>): the device-oriented tree
+///         Folder→Driver→Device→TagGroup→Tag. A node's <c>s=</c> identifier IS its
+///         <see cref="RawPaths">RawPath</see> — folders/drivers/devices/groups included (each keys
+///         on its own RawPath prefix).</item>
+///       <item><b>Uns</b> (<see cref="UnsNamespaceUri"/>): the equipment-oriented tree
+///         Area→Line→Equipment→signal. A node's <c>s=</c> identifier is the slash-joined
+///         <c>Area/Line/Equipment[/EffectiveName]</c>.</item>
+///     </list>
+///     These replace the single <c>https://zb.com/otopcua/ns</c> namespace and the retired
+///     <c>EquipmentNodeIds</c> ({equipmentId}/{folderPath}/{name}) scheme. Realm travels alongside
+///     the identifier as an <see cref="AddressSpaceRealm"/> at every sink seam — the namespace is
+///     never parsed back out of the id string. Both schemes share <see cref="RawPaths.Separator"/>
+///     and its ordinal, case-sensitive segment charset so identity is enforced in one place.
+/// </summary>
+public static class V3NodeIds
+{
+    /// <summary>The Raw namespace URI (device-oriented subtree).</summary>
+    public const string RawNamespaceUri = "https://zb.com/otopcua/raw";
+
+    /// <summary>The UNS namespace URI (equipment-oriented subtree).</summary>
+    public const string UnsNamespaceUri = "https://zb.com/otopcua/uns";
+
+    /// <summary>The namespace URI for a realm.</summary>
+    /// <param name="realm">The address-space realm.</param>
+    /// <returns>The realm's namespace URI.</returns>
+    /// <exception cref="ArgumentOutOfRangeException">Unknown realm.</exception>
+    public static string NamespaceUri(AddressSpaceRealm realm) => realm switch
+    {
+        AddressSpaceRealm.Raw => RawNamespaceUri,
+        AddressSpaceRealm.Uns => UnsNamespaceUri,
+        _ => throw new ArgumentOutOfRangeException(nameof(realm), realm, "Unknown address-space realm."),
+    };
+
+    // ----- Raw realm -----
+
+    /// <summary>
+    ///     The Raw-namespace <c>s=</c> identifier for a raw node — identical to its
+    ///     <see cref="RawPaths">RawPath</see> (folders, drivers, devices, groups, and tags all key
+    ///     on their own RawPath). A pass-through seam so call sites read intent rather than a bare
+    ///     string; the argument is expected to already be a RawPaths-built path.
+    /// </summary>
+    /// <param name="rawPath">The (already-built) RawPath.</param>
+    /// <returns>The Raw-namespace <c>s=</c> identifier (== <paramref name="rawPath"/>).</returns>
+    public static string Raw(string rawPath)
+    {
+        ArgumentException.ThrowIfNullOrEmpty(rawPath);
+        return rawPath;
+    }
+
+    // ----- Uns realm -----
+
+    /// <summary>
+    ///     The UNS-namespace <c>s=</c> identifier for a folder or variable, slash-joined from
+    ///     ordered segments (Area, then Line, then Equipment, then an optional EffectiveName).
+    ///     Every segment is validated via <see cref="RawPaths.ValidateSegment"/> (no embedded
+    ///     separator, no leading/trailing whitespace) so a UNS id round-trips like a RawPath.
+    /// </summary>
+    /// <param name="segments">The ordered, non-empty segments (Area → leaf).</param>
+    /// <returns>The slash-joined UNS <c>s=</c> identifier.</returns>
+    /// <exception cref="ArgumentException">A segment is invalid, or the sequence is empty.</exception>
+    public static string Uns(params string[] segments) => UnsFromSegments(segments);
+
+    /// <summary>Build a UNS <c>s=</c> identifier from ordered segments. See <see cref="Uns(string[])"/>.</summary>
+    /// <param name="segments">The ordered, non-empty segments (Area → leaf).</param>
+    /// <returns>The slash-joined UNS <c>s=</c> identifier.</returns>
+    public static string Uns(IEnumerable<string> segments) => UnsFromSegments(segments);
+
+    /// <summary>
+    ///     Append a child segment (a Line under an Area, an Equipment under a Line, an
+    ///     EffectiveName under an Equipment) to an already-built UNS path.
+    /// </summary>
+    /// <param name="parentUnsPath">The parent UNS path (assumed already valid).</param>
+    /// <param name="childSegment">The child segment to append.</param>
+    /// <returns>The combined UNS <c>s=</c> identifier.</returns>
+    /// <exception cref="ArgumentException">The child segment is invalid, or the parent is blank.</exception>
+    public static string UnsChild(string parentUnsPath, string childSegment)
+    {
+        ArgumentException.ThrowIfNullOrEmpty(parentUnsPath);
+        var error = RawPaths.ValidateSegment(childSegment);
+        if (error is not null)
+            throw new ArgumentException($"Invalid UNS NodeId segment ('{childSegment}'): {error}", nameof(childSegment));
+
+        return parentUnsPath + RawPaths.Separator + childSegment;
+    }
+
+    private static string UnsFromSegments(IEnumerable<string> segments)
+    {
+        ArgumentNullException.ThrowIfNull(segments);
+        var list = segments as IReadOnlyList<string> ?? segments.ToList();
+        if (list.Count == 0)
+            throw new ArgumentException("A UNS NodeId must have at least one segment.", nameof(segments));
+
+        for (var i = 0; i < list.Count; i++)
+        {
+            var error = RawPaths.ValidateSegment(list[i]);
+            if (error is not null)
+                throw new ArgumentException($"Invalid UNS NodeId segment at index {i} ('{list[i]}'): {error}", nameof(segments));
+        }
+
+        return string.Join(RawPaths.Separator, list);
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Types/EquipmentReferenceMap.cs

@@ -0,0 +1,94 @@
+namespace ZB.MOM.WW.OtOpcUa.Commons.Types;
+
+/// <summary>
+///     Builds the per-equipment reference map — <c>equipmentId → (effectiveName → backing-tag RawPath)</c> —
+///     the single authority both compose seams (<c>AddressSpaceComposer</c> + <c>DeploymentArtifact</c>)
+///     and the draft validator use to resolve <c>{{equip}}/&lt;RefName&gt;</c> script paths. A reference's
+///     <b>effective name</b> is its <c>DisplayNameOverride</c> else the backing raw tag's <c>Name</c>; the
+///     RawPath is computed through the shared <see cref="RawPathResolver"/> (the same identity authority the
+///     rest of v3 uses), so the entity side (authoring/validation) and the artifact-decode side agree
+///     byte-for-byte.
+///     <para>Input-shape agnostic + pure: callers flatten their references (EF entities on one side,
+///     artifact JSON on the other) into the same primitive tuple + tag lookup, so the join, the
+///     effective-name rule, and the collision policy live ONLY here. References are consumed in
+///     <c>UnsTagReferenceId</c>-ordinal order and effective-name collisions keep the FIRST — a valid draft
+///     has none (the uniqueness validator rejects them), and the deterministic first-wins keeps the two
+///     seams parity-stable on any input.</para>
+/// </summary>
+public static class EquipmentReferenceMap
+{
+    /// <summary>A flattened UNS tag reference: which equipment references which raw tag under an optional override.</summary>
+    /// <param name="UnsTagReferenceId">Stable logical id — the ordering key for deterministic first-wins.</param>
+    /// <param name="EquipmentId">The referencing equipment.</param>
+    /// <param name="TagId">The backing raw tag.</param>
+    /// <param name="DisplayNameOverride">Optional effective-name override; <see langword="null"/> = use the raw tag's Name.</param>
+    public readonly record struct ReferenceRow(string UnsTagReferenceId, string EquipmentId, string TagId, string? DisplayNameOverride);
+
+    /// <summary>The backing raw tag's identity inputs for its RawPath + effective name.</summary>
+    /// <param name="Name">The raw tag's leaf name (also the default effective name).</param>
+    /// <param name="DeviceId">The tag's owning device id.</param>
+    /// <param name="TagGroupId">The tag's owning tag-group id, or <see langword="null"/> when directly under the device.</param>
+    public readonly record struct TagRow(string Name, string DeviceId, string? TagGroupId);
+
+    /// <summary>
+    ///     Build the reference map. For each reference (ordered by <c>UnsTagReferenceId</c>), resolve the
+    ///     backing tag's RawPath via <paramref name="resolver"/> and key it under the effective name within
+    ///     the owning equipment. References whose backing tag is unknown or whose RawPath cannot be built
+    ///     (broken chain) are skipped. Effective-name collisions within an equipment keep the first.
+    /// </summary>
+    /// <param name="references">The flattened UNS tag references (any order; sorted internally).</param>
+    /// <param name="tagsById">Backing raw tags keyed by <c>TagId</c>.</param>
+    /// <param name="resolver">The shared RawPath resolver over the raw topology.</param>
+    /// <returns><c>equipmentId → (effectiveName → RawPath)</c>. Equipments with no resolvable reference are absent.</returns>
+    public static IReadOnlyDictionary<string, IReadOnlyDictionary<string, string>> Build(
+        IEnumerable<ReferenceRow> references,
+        IReadOnlyDictionary<string, TagRow> tagsById,
+        RawPathResolver resolver)
+    {
+        ArgumentNullException.ThrowIfNull(references);
+        ArgumentNullException.ThrowIfNull(tagsById);
+        ArgumentNullException.ThrowIfNull(resolver);
+
+        var byEquip = new Dictionary<string, Dictionary<string, string>>(StringComparer.Ordinal);
+
+        foreach (var r in references.OrderBy(x => x.UnsTagReferenceId, StringComparer.Ordinal))
+        {
+            if (string.IsNullOrEmpty(r.EquipmentId) || string.IsNullOrEmpty(r.TagId)) continue;
+            if (!tagsById.TryGetValue(r.TagId, out var tag)) continue;
+            var effectiveName = string.IsNullOrEmpty(r.DisplayNameOverride) ? tag.Name : r.DisplayNameOverride!;
+            if (string.IsNullOrEmpty(effectiveName)) continue;
+            var rawPath = resolver.TryBuildTagPath(tag.DeviceId, tag.TagGroupId, tag.Name);
+            if (rawPath is null) continue; // broken chain — dropped (deploy gate rejects invalid names)
+
+            if (!byEquip.TryGetValue(r.EquipmentId, out var map))
+                byEquip[r.EquipmentId] = map = new Dictionary<string, string>(StringComparer.Ordinal);
+            map.TryAdd(effectiveName, rawPath); // first-wins on collision (validator rejects real collisions)
+        }
+
+        return byEquip.ToDictionary(
+            kv => kv.Key,
+            kv => (IReadOnlyDictionary<string, string>)kv.Value,
+            StringComparer.Ordinal);
+    }
+
+    /// <summary>The set of effective names a single equipment's references contribute (for the authoring
+    /// gate + Monaco completion, which need only the names — not the RawPaths).</summary>
+    /// <param name="references">The flattened UNS tag references for (typically) one equipment.</param>
+    /// <param name="tagNameById">Backing raw tag Name keyed by <c>TagId</c>.</param>
+    /// <returns>The distinct effective names, ordinal.</returns>
+    public static IReadOnlySet<string> EffectiveNames(
+        IEnumerable<ReferenceRow> references,
+        IReadOnlyDictionary<string, string> tagNameById)
+    {
+        ArgumentNullException.ThrowIfNull(references);
+        ArgumentNullException.ThrowIfNull(tagNameById);
+        var names = new HashSet<string>(StringComparer.Ordinal);
+        foreach (var r in references)
+        {
+            var effectiveName = r.DisplayNameOverride
+                ?? (tagNameById.TryGetValue(r.TagId, out var n) ? n : null);
+            if (!string.IsNullOrEmpty(effectiveName)) names.Add(effectiveName);
+        }
+        return names;
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Commons/Types/EquipmentScriptPaths.cs

@@ -3,19 +3,29 @@ using System.Text.RegularExpressions;
 namespace ZB.MOM.WW.OtOpcUa.Commons.Types;
 
 /// <summary>
-/// Helpers for equipment-relative virtual-tag script paths. The reserved token
-/// <c>{{equip}}</c> inside a <c>ctx.GetTag</c>/<c>ctx.SetVirtualTag</c> path literal is
-/// replaced at the compose seams with the owning equipment's tag base prefix (derived
-/// from its child-tag <c>FullName</c>s). Pure + regex-based (no Roslyn) so the OpcUaServer
-/// composer and the Runtime artifact-decode path can both share it. Also the single home
-/// for the <c>ctx.GetTag("…")</c> dependency-ref extraction those two seams used to
-/// duplicate.
+/// Helpers for equipment-relative virtual-tag / scripted-alarm script paths. The reserved token
+/// <c>{{equip}}/&lt;RefName&gt;</c> inside a <c>ctx.GetTag</c>/<c>ctx.SetVirtualTag</c> path literal is
+/// replaced at the compose seams (<c>AddressSpaceComposer</c> + <c>DeploymentArtifact</c>) with the
+/// backing raw tag's <c>RawPath</c>, resolved through the owning equipment's
+/// <c>UnsTagReference</c> rows by effective name (<c>&lt;RefName&gt;</c>).
+/// <para><b>v3 syntax:</b> the token joint is a <b>slash</b> — <c>{{equip}}/&lt;RefName&gt;</c> — replacing
+/// the v2 dot-prefix derivation (<c>{{equip}}.X</c>). <c>&lt;RefName&gt;</c> is a reference's effective name
+/// (its <c>DisplayNameOverride</c> else the backing raw tag's <c>Name</c>). An unresolved
+/// <c>&lt;RefName&gt;</c> is left un-substituted (substitution never throws); the deploy-time validator +
+/// the authoring gate + the Monaco diagnostic reject it first, preserving the invariant
+/// <b>the editor accepts ⇔ publish accepts</b>.</para>
+/// Pure + regex-based (no Roslyn) so the OpcUaServer composer and the Runtime artifact-decode path can
+/// both share it. Also the single home for the <c>ctx.GetTag("…")</c> dependency-ref extraction those
+/// two seams used to duplicate.
 /// </summary>
 public static class EquipmentScriptPaths
 {
-    /// <summary>The reserved equipment-base token.</summary>
+    /// <summary>The reserved equipment token stem.</summary>
     public const string EquipToken = "{{equip}}";
 
+    /// <summary>The reserved equipment reference-relative prefix — the token stem plus the slash joint.</summary>
+    public const string EquipTokenPrefix = "{{equip}}/";
+
     // ctx.GetTag("ref") — reads only; the dependency graph subscribes to exactly these.
     private static readonly Regex GetTagRefRegex =
         new(@"ctx\s*\.\s*GetTag\s*\(\s*""([^""]+)""\s*\)", RegexOptions.Compiled);
@@ -32,6 +42,14 @@ public static class EquipmentScriptPaths
     private static readonly Regex PathLiteralRegex =
         new(@"(ctx\s*\.\s*(?:GetTag|SetVirtualTag)\s*\(\s*"")([^""]*)("")", RegexOptions.Compiled);
 
+    // {{equip}}/<RefName> occurrence in FREE TEXT (message templates). <RefName> runs until the next
+    // delimiter that cannot appear in an intra-literal RawPath segment used inline in a template —
+    // whitespace, quote, brace, paren, semicolon, or the '/' separator. (Reference effective names may
+    // contain internal spaces; that space-bearing form is unambiguous only inside a path literal, so the
+    // free-text scan is best-effort — see ExtractEquipReferenceNamesFromText.)
+    private static readonly Regex EquipRefTextRegex =
+        new(@"\{\{equip\}\}/([^\s""{}();/]+)", RegexOptions.Compiled);
+
     // A pure pass-through virtual-tag body: exactly `return ctx.GetTag("<ref>").Value;`
     // (whitespace-insensitive). Captures <ref> for relay→alias conversion. Anything with extra
     // statements, arithmetic, a different member than .Value, or multiple GetTag calls is NOT a relay.
@@ -46,44 +64,89 @@ public static class EquipmentScriptPaths
         !string.IsNullOrEmpty(source) && source.Contains(EquipToken, StringComparison.Ordinal);
 
     /// <summary>
-    /// Equipment tag base = the single shared substring-before-first-dot across the
-    /// equipment's child-tag <c>FullName</c>s. Returns <c>null</c> when there are no usable
-    /// FullNames or they don't agree on one prefix (equipment spanning multiple objects).
+    /// Replace each <c>{{equip}}/&lt;RefName&gt;</c> reference-relative path with the backing raw tag's
+    /// <c>RawPath</c> from <paramref name="referenceMap"/> (effective name → RawPath), inside
+    /// <c>ctx.GetTag</c>/<c>ctx.SetVirtualTag</c> path literals only. A path literal whose content is
+    /// <c>{{equip}}/&lt;RefName&gt;</c> is treated as a single reference: <c>&lt;RefName&gt;</c> is the
+    /// entire remainder after the prefix (so reference effective names may contain internal spaces).
+    /// Identity when <paramref name="source"/> is null/empty, <paramref name="referenceMap"/> is empty,
+    /// or the token is absent (so every existing script — none of which use the token — is byte-unchanged).
+    /// An <c>&lt;RefName&gt;</c> absent from the map is left un-substituted (never throws) — the validator
+    /// rejects it first at deploy.
     /// </summary>
-    /// <param name="childFullNames">The equipment's child-tag driver FullNames.</param>
-    /// <returns>The shared base prefix, or null when none/ambiguous.</returns>
-    public static string? DeriveEquipmentBase(IEnumerable<string?> childFullNames)
+    /// <param name="source">The script source.</param>
+    /// <param name="referenceMap">The equipment's reference map: effective name → backing-tag RawPath.</param>
+    /// <returns>The source with each resolvable <c>{{equip}}/&lt;RefName&gt;</c> substituted inside path literals.</returns>
+    public static string SubstituteEquipmentToken(string source, IReadOnlyDictionary<string, string> referenceMap)
     {
-        string? found = null;
-        foreach (var fn in childFullNames)
-        {
-            if (string.IsNullOrWhiteSpace(fn)) continue;
-            var dot = fn.IndexOf('.');
-            var prefix = dot < 0 ? fn : fn.Substring(0, dot);
-            if (prefix.Length == 0) continue;
-            if (found is null) found = prefix;
-            else if (!string.Equals(found, prefix, StringComparison.Ordinal)) return null;
-        }
-        return found;
+        if (string.IsNullOrEmpty(source) || referenceMap is null || referenceMap.Count == 0) return source;
+        if (!source.Contains(EquipTokenPrefix, StringComparison.Ordinal)) return source;
+        return PathLiteralRegex.Replace(source, m =>
+            m.Groups[1].Value
+            + SubstituteContent(m.Groups[2].Value, referenceMap)
+            + m.Groups[3].Value);
+    }
+
+    // Substitute the reference-relative token inside a single path-literal's content. The content is one
+    // whole tag path; when it starts with the {{equip}}/ prefix the remainder is the reference name.
+    private static string SubstituteContent(string content, IReadOnlyDictionary<string, string> referenceMap)
+    {
+        if (!content.StartsWith(EquipTokenPrefix, StringComparison.Ordinal)) return content;
+        var refName = content.Substring(EquipTokenPrefix.Length);
+        if (refName.Length == 0) return content;
+        return referenceMap.TryGetValue(refName, out var rawPath) ? rawPath : content;
     }
 
     /// <summary>
-    /// Replace <c>{{equip}}</c> with <paramref name="equipBase"/> inside
-    /// <c>ctx.GetTag</c>/<c>ctx.SetVirtualTag</c> path literals only. Identity when
-    /// <paramref name="equipBase"/> is null/empty or the token is absent (so every existing
-    /// script — none of which use the token — is byte-unchanged).
+    /// Distinct <c>&lt;RefName&gt;</c> values used via <c>{{equip}}/&lt;RefName&gt;</c> inside
+    /// <c>ctx.GetTag</c>/<c>ctx.SetVirtualTag</c> path literals, in first-seen order. Scoped to the SAME
+    /// path literals <see cref="SubstituteEquipmentToken"/> operates on (so a token in a comment / logger
+    /// string is not reported), and the entire post-prefix remainder is the reference name (matching
+    /// substitution) — this keeps the validator / authoring gate / Monaco diagnostic in lockstep with what
+    /// resolves. Feeds the deploy-gate + authoring rejection + editor completion.
     /// </summary>
-    /// <param name="source">The script source.</param>
-    /// <param name="equipBase">The equipment base prefix, or null/empty for no substitution.</param>
-    /// <returns>The source with the token substituted inside path literals.</returns>
-    public static string SubstituteEquipmentToken(string source, string? equipBase)
+    /// <param name="scriptSource">The virtual-tag / scripted-alarm predicate script source.</param>
+    /// <returns>Distinct reference names, first-seen order; empty when none / no token.</returns>
+    public static IReadOnlyList<string> ExtractEquipReferenceNames(string? scriptSource)
     {
-        if (string.IsNullOrEmpty(source) || string.IsNullOrEmpty(equipBase)) return source;
-        if (!source.Contains(EquipToken, StringComparison.Ordinal)) return source;
-        return PathLiteralRegex.Replace(source, m =>
-            m.Groups[1].Value
-            + m.Groups[2].Value.Replace(EquipToken, equipBase, StringComparison.Ordinal)
-            + m.Groups[3].Value);
+        if (string.IsNullOrEmpty(scriptSource)
+            || !scriptSource.Contains(EquipTokenPrefix, StringComparison.Ordinal))
+            return Array.Empty<string>();
+        var seen = new HashSet<string>(StringComparer.Ordinal);
+        var result = new List<string>();
+        foreach (Match m in PathLiteralRegex.Matches(scriptSource))
+        {
+            var content = m.Groups[2].Value;
+            if (!content.StartsWith(EquipTokenPrefix, StringComparison.Ordinal)) continue;
+            var refName = content.Substring(EquipTokenPrefix.Length);
+            if (refName.Length > 0 && seen.Add(refName)) result.Add(refName);
+        }
+        return result;
+    }
+
+    /// <summary>
+    /// Distinct <c>&lt;RefName&gt;</c> values used via <c>{{equip}}/&lt;RefName&gt;</c> in FREE TEXT — the
+    /// scripted-alarm <c>MessageTemplate</c>, which is not a path literal — in first-seen order. Best-effort:
+    /// a reference name is captured up to the next whitespace / quote / brace / paren / semicolon / slash, so
+    /// a space-bearing effective name used inline in a template resolves only up to its first space (a
+    /// documented, benign limitation — template rendering is dark until Batch 4). Feeds the deploy-gate rule
+    /// for alarm message tokens.
+    /// </summary>
+    /// <param name="text">The free text (e.g. a scripted-alarm message template).</param>
+    /// <returns>Distinct reference names, first-seen order; empty when none / no token.</returns>
+    public static IReadOnlyList<string> ExtractEquipReferenceNamesFromText(string? text)
+    {
+        if (string.IsNullOrEmpty(text)
+            || !text.Contains(EquipTokenPrefix, StringComparison.Ordinal))
+            return Array.Empty<string>();
+        var seen = new HashSet<string>(StringComparer.Ordinal);
+        var result = new List<string>();
+        foreach (Match m in EquipRefTextRegex.Matches(text))
+        {
+            var refName = m.Groups[1].Value;
+            if (refName.Length > 0 && seen.Add(refName)) result.Add(refName);
+        }
+        return result;
     }
 
     /// <summary>
@@ -115,10 +178,11 @@ public static class EquipmentScriptPaths
     /// <c>{{equip}}</c> double-brace form is excluded by the token regex. Deterministic so the live
     /// composer (<c>AddressSpaceComposer</c>) and the artifact-decode mirror (<c>DeploymentArtifact</c>)
     /// produce the exact same ordered list — the byte-parity contract <c>EquipmentScriptedAlarmPlan</c>
-    /// equality depends on. Scripted alarms do NOT use <c>{{equip}}</c> substitution (only virtual
-    /// tags do) — pass the predicate source as-is.
+    /// equality depends on. The predicate source passed here is the ALREADY-substituted source (the two
+    /// compose seams substitute <c>{{equip}}/&lt;RefName&gt;</c> before extraction, identically), so the
+    /// merged refs are resolved RawPaths.
     /// </summary>
-    /// <param name="predicateSource">The resolved predicate script source.</param>
+    /// <param name="predicateSource">The resolved (substituted) predicate script source.</param>
     /// <param name="messageTemplate">The alarm message template carrying <c>{TagPath}</c> tokens.</param>
     /// <returns>The merged, distinct, deterministically-ordered dependency refs.</returns>
     public static IReadOnlyList<string> ExtractAlarmDependencyRefs(string? predicateSource, string? messageTemplate)
@@ -154,7 +218,7 @@ public static class EquipmentScriptPaths
     /// <param name="source">The virtual-tag script source to inspect.</param>
     /// <param name="tagReference">
     /// When the method returns <see langword="true"/>, the captured <c>GetTag</c> path literal
-    /// (e.g. <c>TestMachine_020.TestChangingInt</c> or <c>{{equip}}.Speed</c>);
+    /// (e.g. <c>TestMachine_020.TestChangingInt</c> or <c>{{equip}}/Speed</c>);
     /// otherwise <see langword="null"/>.
     /// </param>
     /// <returns>

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Entities/ClusterNode.cs

@@ -18,6 +18,33 @@ public sealed class ClusterNode
     /// <summary>The dashboard HTTP port (default 8081).</summary>
     public int DashboardPort { get; set; } = 8081;
 
+    /// <summary>
+    ///     Akka remoting port (default 4053). <b>This is central's dial target, not the node's own
+    ///     binding configuration</b> — the node binds from <c>Cluster:Port</c> in its own appsettings,
+    ///     and this column duplicates that value for a reader that cannot see the node's config.
+    ///     Per-cluster mesh Phase 2 builds its ClusterClient contact points from
+    ///     <see cref="Host"/> + this port, at which point central and the site node no longer share a
+    ///     gossip ring and central has no other way to learn it.
+    /// </summary>
+    /// <remarks>
+    ///     The duplication is real and unenforced by the schema. <c>ClusterNodeAddressReconciler</c>
+    ///     on the admin node compares this row against observed cluster membership and logs an Error
+    ///     on mismatch — a node that binds 4054 while its row says 4053 is unreachable from central in
+    ///     Phase 2, and the symptom there is a silent absence of acks rather than an error.
+    /// </remarks>
+    public int AkkaPort { get; set; } = 4053;
+
+    /// <summary>
+    ///     gRPC port central dials for the Phase 5 telemetry stream, or <see langword="null"/> when the
+    ///     node exposes none. Also central's dial target rather than the node's binding config.
+    /// </summary>
+    /// <remarks>
+    ///     Nullable by intent: nothing listens on this port yet, and a non-null default would assert a
+    ///     port that does not exist. Phase 5 populates it; until then <see langword="null"/> is the
+    ///     honest value.
+    /// </remarks>
+    public int? GrpcPort { get; set; }
+
     /// <summary>
     ///     OPC UA <c>ApplicationUri</c> — MUST be unique per node per OPC UA spec. Clients pin trust here.
     ///     Fleet-wide unique index enforces no two nodes share a value.
@@ -36,8 +63,37 @@ public sealed class ClusterNode
     public string? DriverConfigOverridesJson { get; set; }
 
     /// <summary>Gets or sets a value indicating whether this node is enabled.</summary>
+    /// <remarks>
+    ///     <b>Part of the declared topology.</b> <c>DraftValidator.ValidateClusterTopology</c> requires
+    ///     the enabled-node count to equal <see cref="ServerCluster.NodeCount"/>, so disabling one node
+    ///     of a Warm/Hot pair is a deploy-blocking validation error by design — it would boot the
+    ///     runtime into the InvalidTopology band. To take a node out of service temporarily, use
+    ///     <see cref="MaintenanceMode"/> instead.
+    /// </remarks>
     public bool Enabled { get; set; } = true;
 
+    /// <summary>
+    ///     Node is temporarily out of service: still part of the cluster's declared topology, but not
+    ///     expected to participate. A deployment does not wait for its ack, and the address reconciler
+    ///     does not warn that it is absent.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         Separate from <see cref="Enabled"/> because the two answer different questions.
+    ///         <see cref="Enabled"/> means "this node is part of the declared topology" and is checked
+    ///         against <see cref="ServerCluster.NodeCount"/>; this flag means "do not expect it right
+    ///         now". Per-cluster mesh Phase 1 made an enabled row's ack mandatory, so without this a
+    ///         node down for maintenance would block every deployment to its cluster — and clearing
+    ///         <see cref="Enabled"/> to escape that is itself rejected by the topology gate on any
+    ///         2-node pair, which is every cluster in the target deployment.
+    ///     </para>
+    ///     <para>
+    ///         Found by the Phase 1 live gate: the two rules were independently reasonable and
+    ///         contradictory together.
+    ///     </para>
+    /// </remarks>
+    public bool MaintenanceMode { get; set; }
+
     /// <summary>Gets or sets the timestamp when this node was last seen.</summary>
     public DateTime? LastSeenAt { get; set; }
 

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/GenerationSealedCache.cs

@@ -1,197 +0,0 @@
-using LiteDB;
-
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Generation-sealed LiteDB cache per <c>docs/v2/plan.md</c> and Phase 6.1
-///     Stream D.1. Each published generation writes one <b>read-only</b> LiteDB file under
-///     <c>&lt;cache-root&gt;/&lt;clusterId&gt;/&lt;generationId&gt;.db</c>. A per-cluster
-///     <c>CURRENT</c> text file holds the currently-active generation id; it is updated
-///     atomically (temp file + <see cref="File.Replace(string, string, string?)"/>) only after
-///     the sealed file is fully written.
-/// </summary>
-/// <remarks>
-///     <para>Mixed-generation reads are impossible: any read opens the single file pointed to
-///     by <c>CURRENT</c>, which is a coherent snapshot. Corruption of the CURRENT file or the
-///     sealed file surfaces as <see cref="GenerationCacheUnavailableException"/> — the reader
-///     fails closed rather than silently falling back to an older generation. Recovery path
-///     is to re-fetch from the central DB (and the Phase 6.1 Stream C <c>UsingStaleConfig</c>
-///     flag goes true until that succeeds).</para>
-///
-///     <para>This cache is the read-path fallback when the central DB is unreachable. The
-///     write path (draft edits, publish) bypasses the cache and fails hard on DB outage per
-///     Stream D.2 — inconsistent writes are worse than a temporary inability to edit.</para>
-/// </remarks>
-public sealed class GenerationSealedCache
-{
-    private const string CollectionName = "generation";
-    private const string CurrentPointerFileName = "CURRENT";
-    private readonly string _cacheRoot;
-
-    // Private per-database BsonMapper with the entity pre-registered. LiteDB's default
-    // BsonMapper.Global is a process-wide singleton whose lazy per-type member registration is
-    // not thread-safe across concurrently-constructed LiteDatabase instances; a seal racing a
-    // read (or this cache racing LiteDbConfigCache) corrupts the global mapper, surfacing as
-    // "Member … not found on BsonMapper" or a bogus duplicate-_id insert.
-    private static BsonMapper BuildMapper()
-    {
-        var mapper = new BsonMapper();
-        mapper.Entity<GenerationSnapshot>();
-        return mapper;
-    }
-
-    /// <summary>Root directory for all clusters' sealed caches.</summary>
-    public string CacheRoot => _cacheRoot;
-
-    /// <summary>Initializes a new instance of the GenerationSealedCache class.</summary>
-    /// <param name="cacheRoot">The root directory for the cache.</param>
-    public GenerationSealedCache(string cacheRoot)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(cacheRoot);
-        _cacheRoot = cacheRoot;
-        Directory.CreateDirectory(_cacheRoot);
-    }
-
-    /// <summary>
-    ///     Seal a generation: write the snapshot to <c>&lt;cluster&gt;/&lt;generationId&gt;.db</c>,
-    ///     mark the file read-only, then atomically publish the <c>CURRENT</c> pointer. Existing
-    ///     sealed files for prior generations are preserved (prune separately).
-    /// </summary>
-    /// <param name="snapshot">The generation snapshot to seal.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task representing the asynchronous operation.</returns>
-    public async Task SealAsync(GenerationSnapshot snapshot, CancellationToken ct = default)
-    {
-        ArgumentNullException.ThrowIfNull(snapshot);
-        ct.ThrowIfCancellationRequested();
-
-        var clusterDir = Path.Combine(_cacheRoot, snapshot.ClusterId);
-        Directory.CreateDirectory(clusterDir);
-        var sealedPath = Path.Combine(clusterDir, $"{snapshot.GenerationId}.db");
-
-        if (File.Exists(sealedPath))
-        {
-            // Already sealed — idempotent. Treat as no-op + update pointer in case an earlier
-            // seal succeeded but the pointer update failed (crash recovery).
-            WritePointerAtomically(clusterDir, snapshot.GenerationId);
-            return;
-        }
-
-        var tmpPath = sealedPath + ".tmp";
-        try
-        {
-            using (var db = new LiteDatabase(new ConnectionString { Filename = tmpPath, Upgrade = false }, BuildMapper()))
-            {
-                var col = db.GetCollection<GenerationSnapshot>(CollectionName);
-                col.Insert(snapshot);
-            }
-
-            File.Move(tmpPath, sealedPath);
-            File.SetAttributes(sealedPath, File.GetAttributes(sealedPath) | FileAttributes.ReadOnly);
-            WritePointerAtomically(clusterDir, snapshot.GenerationId);
-        }
-        catch
-        {
-            try { if (File.Exists(tmpPath)) File.Delete(tmpPath); } catch { /* best-effort */ }
-            throw;
-        }
-
-        await Task.CompletedTask;
-    }
-
-    /// <summary>
-    ///     Read the current sealed snapshot for <paramref name="clusterId"/>. Throws
-    ///     <see cref="GenerationCacheUnavailableException"/> when the pointer is missing
-    ///     (first-boot-no-snapshot case) or when the sealed file is corrupt. Never silently
-    ///     falls back to a prior generation.
-    /// </summary>
-    /// <param name="clusterId">The cluster ID to read the snapshot for.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task representing the asynchronous operation containing the generation snapshot.</returns>
-    public Task<GenerationSnapshot> ReadCurrentAsync(string clusterId, CancellationToken ct = default)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
-        ct.ThrowIfCancellationRequested();
-
-        var clusterDir = Path.Combine(_cacheRoot, clusterId);
-        var pointerPath = Path.Combine(clusterDir, CurrentPointerFileName);
-        if (!File.Exists(pointerPath))
-            throw new GenerationCacheUnavailableException(
-                $"No sealed generation for cluster '{clusterId}' at '{clusterDir}'. First-boot case: the central DB must be reachable at least once before cache fallback is possible.");
-
-        long generationId;
-        try
-        {
-            var text = File.ReadAllText(pointerPath).Trim();
-            generationId = long.Parse(text, System.Globalization.CultureInfo.InvariantCulture);
-        }
-        catch (Exception ex)
-        {
-            throw new GenerationCacheUnavailableException(
-                $"CURRENT pointer at '{pointerPath}' is corrupt or unreadable.", ex);
-        }
-
-        var sealedPath = Path.Combine(clusterDir, $"{generationId}.db");
-        if (!File.Exists(sealedPath))
-            throw new GenerationCacheUnavailableException(
-                $"CURRENT points at generation {generationId} but '{sealedPath}' is missing — fails closed rather than serving an older generation.");
-
-        try
-        {
-            using var db = new LiteDatabase(new ConnectionString { Filename = sealedPath, ReadOnly = true }, BuildMapper());
-            var col = db.GetCollection<GenerationSnapshot>(CollectionName);
-            var snapshot = col.FindAll().FirstOrDefault()
-                ?? throw new GenerationCacheUnavailableException(
-                    $"Sealed file '{sealedPath}' contains no snapshot row — file is corrupt.");
-            return Task.FromResult(snapshot);
-        }
-        catch (GenerationCacheUnavailableException) { throw; }
-        catch (Exception ex) when (ex is LiteException or InvalidDataException or IOException
-                                      or NotSupportedException or FormatException)
-        {
-            throw new GenerationCacheUnavailableException(
-                $"Sealed file '{sealedPath}' is corrupt or unreadable — fails closed rather than falling back to an older generation.", ex);
-        }
-    }
-
-    /// <summary>Return the generation id the <c>CURRENT</c> pointer points at, or null if no pointer exists.</summary>
-    /// <param name="clusterId">The cluster ID to get the current generation ID for.</param>
-    /// <returns>The generation ID, or null if no pointer exists.</returns>
-    public long? TryGetCurrentGenerationId(string clusterId)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
-        var pointerPath = Path.Combine(_cacheRoot, clusterId, CurrentPointerFileName);
-        if (!File.Exists(pointerPath)) return null;
-        try
-        {
-            return long.Parse(File.ReadAllText(pointerPath).Trim(), System.Globalization.CultureInfo.InvariantCulture);
-        }
-        catch
-        {
-            return null;
-        }
-    }
-
-    private static void WritePointerAtomically(string clusterDir, long generationId)
-    {
-        var pointerPath = Path.Combine(clusterDir, CurrentPointerFileName);
-        var tmpPath = pointerPath + ".tmp";
-        File.WriteAllText(tmpPath, generationId.ToString(System.Globalization.CultureInfo.InvariantCulture));
-        if (File.Exists(pointerPath))
-            File.Replace(tmpPath, pointerPath, destinationBackupFileName: null);
-        else
-            File.Move(tmpPath, pointerPath);
-    }
-}
-
-/// <summary>Sealed cache is unreachable — caller must fail closed.</summary>
-public sealed class GenerationCacheUnavailableException : Exception
-{
-    /// <summary>Initializes a new instance of the GenerationCacheUnavailableException class.</summary>
-    /// <param name="message">The error message.</param>
-    public GenerationCacheUnavailableException(string message) : base(message) { }
-    /// <summary>Initializes a new instance of the GenerationCacheUnavailableException class with an inner exception.</summary>
-    /// <param name="message">The error message.</param>
-    /// <param name="inner">The inner exception.</param>
-    public GenerationCacheUnavailableException(string message, Exception inner) : base(message, inner) { }
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/GenerationSnapshot.cs

@@ -1,20 +0,0 @@
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     A self-contained snapshot of one generation — enough to rebuild the address space on a node
-///     that has lost DB connectivity. The payload is the JSON-serialized <c>sp_GetGenerationContent</c>
-///     result; the local cache doesn't inspect the shape, it just round-trips bytes.
-/// </summary>
-public sealed class GenerationSnapshot
-{
-    /// <summary>Gets or sets the auto-generated LiteDB ID.</summary>
-    public int Id { get; set; } // LiteDB auto-ID
-    /// <summary>Gets or sets the cluster identifier.</summary>
-    public required string ClusterId { get; set; }
-    /// <summary>Gets or sets the generation identifier.</summary>
-    public required long GenerationId { get; set; }
-    /// <summary>Gets or sets the time this snapshot was cached.</summary>
-    public required DateTime CachedAt { get; set; }
-    /// <summary>Gets or sets the JSON-serialized payload content.</summary>
-    public required string PayloadJson { get; set; }
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/ILocalConfigCache.cs

@@ -1,32 +0,0 @@
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Per-node local cache of the most-recently-applied generation(s). Used to bootstrap the
-///     address space when the central DB is unreachable (degraded-but-running).
-/// </summary>
-/// <remarks>
-///     <para><b>Concurrency contract:</b> implementations must serialize writes — specifically,
-///     <see cref="PutAsync"/> for the same <c>(ClusterId, GenerationId)</c> from concurrent
-///     callers must not produce duplicate rows. Reads may run concurrently with reads and writes.
-///     The <see cref="LiteDbConfigCache"/> implementation enforces this via an instance-level
-///     <see cref="SemaphoreSlim"/> around the find-then-insert/update window.</para>
-/// </remarks>
-public interface ILocalConfigCache
-{
-    /// <summary>Retrieves the most recent generation snapshot for the specified cluster.</summary>
-    /// <param name="clusterId">The cluster identifier.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>The most recent generation snapshot, or null if none exists.</returns>
-    Task<GenerationSnapshot?> GetMostRecentAsync(string clusterId, CancellationToken ct = default);
-    /// <summary>Stores a generation snapshot in the local cache.</summary>
-    /// <param name="snapshot">The generation snapshot to store.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task that represents the asynchronous operation.</returns>
-    Task PutAsync(GenerationSnapshot snapshot, CancellationToken ct = default);
-    /// <summary>Removes old generations, keeping only the most recent N.</summary>
-    /// <param name="clusterId">The cluster identifier.</param>
-    /// <param name="keepLatest">The number of latest generations to keep.</param>
-    /// <param name="ct">The cancellation token.</param>
-    /// <returns>A task that represents the asynchronous operation.</returns>
-    Task PruneOldGenerationsAsync(string clusterId, int keepLatest = 10, CancellationToken ct = default);
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/LiteDbConfigCache.cs

@@ -1,129 +0,0 @@
-using LiteDB;
-
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     LiteDB-backed <see cref="ILocalConfigCache"/>. One file per node (default
-///     <c>config_cache.db</c>), one collection per snapshot. Corruption surfaces as
-///     <see cref="LocalConfigCacheCorruptException"/> on construction or read — callers should
-///     delete and re-fetch from the central DB.
-/// </summary>
-public sealed class LiteDbConfigCache : ILocalConfigCache, IDisposable
-{
-    private const string CollectionName = "generations";
-
-    // LiteDB's default BsonMapper.Global is a process-wide singleton whose per-type member
-    // registration is lazy and NOT thread-safe across concurrently-constructed LiteDatabase
-    // instances. When several caches (this one + GenerationSealedCache) initialise in parallel
-    // the global mapper races, surfacing as "Member ClusterId not found on BsonMapper" or a
-    // bogus "duplicate key _id = 0" (the int auto-id mapping was lost so Insert writes a literal
-    // 0 twice). Give each database a private, pre-registered mapper so member
-    // resolution happens once, single-threaded, at construction and never touches the global.
-    private static BsonMapper BuildMapper()
-    {
-        var mapper = new BsonMapper();
-        mapper.Entity<GenerationSnapshot>();
-        return mapper;
-    }
-
-    private readonly LiteDatabase _db;
-    private readonly ILiteCollection<GenerationSnapshot> _col;
-    // PutAsync is a find-then-insert/update; without serialization, two concurrent puts for the
-    // same (ClusterId, GenerationId) can both observe `existing is null` and both Insert,
-    // producing duplicate rows. Serialize writes through this semaphore so
-    // the read-modify-write block is atomic for a given instance. LiteDB itself only locks the
-    // page-level write, not the find-then-insert window.
-    private readonly SemaphoreSlim _writeGate = new(initialCount: 1, maxCount: 1);
-
-    /// <summary>Initializes a new instance of the <see cref="LiteDbConfigCache"/> class.</summary>
-    /// <param name="dbPath">Path to the LiteDB database file.</param>
-    public LiteDbConfigCache(string dbPath)
-    {
-        // LiteDB can be tolerant of header-only corruption at construction time (it may overwrite
-        // the header and "recover"), so we force a write + read probe to fail fast on real corruption.
-        try
-        {
-            _db = new LiteDatabase(new ConnectionString { Filename = dbPath, Upgrade = true }, BuildMapper());
-            _col = _db.GetCollection<GenerationSnapshot>(CollectionName);
-            _col.EnsureIndex(s => s.ClusterId);
-            _col.EnsureIndex(s => s.GenerationId);
-            _ = _col.Count();
-        }
-        catch (Exception ex) when (ex is LiteException or InvalidDataException or IOException
-                                      or NotSupportedException or UnauthorizedAccessException
-                                      or ArgumentOutOfRangeException or FormatException)
-        {
-            _db?.Dispose();
-            throw new LocalConfigCacheCorruptException(
-                $"LiteDB cache at '{dbPath}' is corrupt or unreadable — delete the file and refetch from the central DB.",
-                ex);
-        }
-    }
-
-    /// <inheritdoc />
-    public Task<GenerationSnapshot?> GetMostRecentAsync(string clusterId, CancellationToken ct = default)
-    {
-        ct.ThrowIfCancellationRequested();
-        var snapshot = _col
-            .Find(s => s.ClusterId == clusterId)
-            .OrderByDescending(s => s.GenerationId)
-            .FirstOrDefault();
-        return Task.FromResult<GenerationSnapshot?>(snapshot);
-    }
-
-    /// <inheritdoc />
-    public async Task PutAsync(GenerationSnapshot snapshot, CancellationToken ct = default)
-    {
-        ct.ThrowIfCancellationRequested();
-        // Serialize the find-then-insert/update so concurrent callers do not observe a stale
-        // `existing is null` and both Insert. LiteDB's per-call lock is not enough — the
-        // read and the write are independent calls.
-        await _writeGate.WaitAsync(ct).ConfigureAwait(false);
-        try
-        {
-            // upsert by (ClusterId, GenerationId) — replace in place if already cached
-            var existing = _col
-                .Find(s => s.ClusterId == snapshot.ClusterId && s.GenerationId == snapshot.GenerationId)
-                .FirstOrDefault();
-
-            if (existing is null)
-                _col.Insert(snapshot);
-            else
-            {
-                snapshot.Id = existing.Id;
-                _col.Update(snapshot);
-            }
-        }
-        finally
-        {
-            _writeGate.Release();
-        }
-    }
-
-    /// <inheritdoc />
-    public Task PruneOldGenerationsAsync(string clusterId, int keepLatest = 10, CancellationToken ct = default)
-    {
-        ct.ThrowIfCancellationRequested();
-        var doomed = _col
-            .Find(s => s.ClusterId == clusterId)
-            .OrderByDescending(s => s.GenerationId)
-            .Skip(keepLatest)
-            .Select(s => s.Id)
-            .ToList();
-
-        foreach (var id in doomed)
-            _col.Delete(id);
-
-        return Task.CompletedTask;
-    }
-
-    /// <summary>Releases all resources used by the cache.</summary>
-    public void Dispose()
-    {
-        _writeGate.Dispose();
-        _db.Dispose();
-    }
-}
-
-public sealed class LocalConfigCacheCorruptException(string message, Exception inner)
-    : Exception(message, inner);

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/ResilientConfigReader.cs

@@ -1,140 +0,0 @@
-using System.Text.RegularExpressions;
-using Microsoft.Extensions.Logging;
-using Polly;
-using Polly.Retry;
-using Polly.Timeout;
-
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Wraps a central-DB fetch function with Phase 6.1 Stream D.2 resilience:
-///     <b>timeout 2 s → retry 3× jittered → fallback to sealed cache</b>. Maintains the
-///     <see cref="StaleConfigFlag"/> — fresh on central-DB success, stale on cache fallback.
-/// </summary>
-/// <remarks>
-///     <para>Read-path only per plan. The write path (draft save, publish) bypasses this
-///     wrapper entirely and fails hard on DB outage so inconsistent writes never land.</para>
-///
-///     <para>Fallback is triggered by <b>any exception</b> the fetch raises (central-DB
-///     unreachable, SqlException, timeout). If the sealed cache also fails (no pointer,
-///     corrupt file, etc.), <see cref="GenerationCacheUnavailableException"/> surfaces — caller
-///     must fail the current request (InitializeAsync for a driver, etc.).</para>
-/// </remarks>
-public sealed class ResilientConfigReader
-{
-    private readonly GenerationSealedCache _cache;
-    private readonly StaleConfigFlag _staleFlag;
-    private readonly ResiliencePipeline _pipeline;
-    private readonly ILogger<ResilientConfigReader> _logger;
-
-    /// <summary>Initializes a resilient config reader with the given cache and options.</summary>
-    /// <param name="cache">The sealed cache for fallback.</param>
-    /// <param name="staleFlag">The stale config flag to manage.</param>
-    /// <param name="logger">The logger instance.</param>
-    /// <param name="timeout">The timeout for central fetch (default 2s).</param>
-    /// <param name="retryCount">The number of retries (default 3).</param>
-    public ResilientConfigReader(
-        GenerationSealedCache cache,
-        StaleConfigFlag staleFlag,
-        ILogger<ResilientConfigReader> logger,
-        TimeSpan? timeout = null,
-        int retryCount = 3)
-    {
-        _cache = cache;
-        _staleFlag = staleFlag;
-        _logger = logger;
-        var builder = new ResiliencePipelineBuilder()
-            .AddTimeout(new TimeoutStrategyOptions { Timeout = timeout ?? TimeSpan.FromSeconds(2) });
-
-        if (retryCount > 0)
-        {
-            builder.AddRetry(new RetryStrategyOptions
-            {
-                MaxRetryAttempts = retryCount,
-                BackoffType = DelayBackoffType.Exponential,
-                UseJitter = true,
-                Delay = TimeSpan.FromMilliseconds(100),
-                MaxDelay = TimeSpan.FromSeconds(1),
-                // Handle ALL exceptions including OperationCanceledException. A SQL command-level
-                // timeout surfaces as TaskCanceledException (derives from OperationCanceledException)
-                // when the caller's token is NOT cancelled, and must be retried just like any other
-                // transient error. Polly itself checks the cancellation token between retries and
-                // stops with OperationCanceledException on genuine caller cancellation regardless of
-                // this predicate.
-                ShouldHandle = new PredicateBuilder().Handle<Exception>(),
-            });
-        }
-
-        _pipeline = builder.Build();
-    }
-
-    /// <summary>
-    ///     Redacts connection-string fragments (Password, User Id, Pwd, etc.)
-    ///     that a caller's exception message could carry. Conservative regex pass — anything
-    ///     matching <c>Key=Value</c> with a known credential key gets its value replaced.
-    /// </summary>
-    private static readonly Regex SecretsRegex = new(
-        @"(?ix)\b(Password|Pwd|User\s*Id|Uid|AccessToken|Authorization|Api[-_]?Key)\s*=\s*[^;,)\s]*",
-        RegexOptions.Compiled);
-
-    /// <summary>Redacts sensitive credential information from a message.</summary>
-    /// <param name="message">The message to scrub.</param>
-    /// <returns>The message with redacted credentials.</returns>
-    internal static string ScrubSecrets(string? message)
-    {
-        if (string.IsNullOrEmpty(message)) return message ?? string.Empty;
-        // Replace the entire matched fragment (key + value) with a redaction marker so the
-        // key name itself doesn't leak — log scrapers grep for "Password=" too.
-        return SecretsRegex.Replace(message, "[redacted credential]");
-    }
-
-    /// <summary>
-    ///     Executes a central fetch through the resilience pipeline. On full failure
-    ///     (post-retry), reads the sealed cache and extracts the requested shape.
-    /// </summary>
-    /// <typeparam name="T">The type of configuration to read.</typeparam>
-    /// <param name="clusterId">The cluster ID to fetch for.</param>
-    /// <param name="centralFetch">Function to fetch from central DB.</param>
-    /// <param name="fromSnapshot">Function to extract the config from a snapshot.</param>
-    /// <param name="cancellationToken">Cancellation token.</param>
-    /// <returns>The configuration of type T.</returns>
-    public async ValueTask<T> ReadAsync<T>(
-        string clusterId,
-        Func<CancellationToken, ValueTask<T>> centralFetch,
-        Func<GenerationSnapshot, T> fromSnapshot,
-        CancellationToken cancellationToken)
-    {
-        ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
-        ArgumentNullException.ThrowIfNull(centralFetch);
-        ArgumentNullException.ThrowIfNull(fromSnapshot);
-
-        try
-        {
-            var result = await _pipeline.ExecuteAsync(centralFetch, cancellationToken).ConfigureAwait(false);
-            _staleFlag.MarkFresh();
-            return result;
-        }
-        // Catch all exceptions that are NOT genuine caller cancellations. A SQL command-level
-        // timeout surfaces as TaskCanceledException (derives from OperationCanceledException)
-        // but the caller's token is NOT cancelled — we must fall back to the sealed cache for
-        // that case, not propagate. Only rethrow if the caller actually requested cancellation.
-        catch (Exception ex) when (ex is not OperationCanceledException || !cancellationToken.IsCancellationRequested)
-        {
-            // Do NOT pass the raw exception object — it carries the stack
-            // and inner-exception chain, and SqlException/wrapping delegates can surface
-            // connection-string fragments (Password=…, User Id=…) embedded in messages.
-            // Log only the exception type and a scrubbed message so secrets stay out of logs.
-            _logger.LogWarning(
-                "Central-DB read failed after retries ({ExceptionType}: {SanitizedMessage}); falling back to sealed cache for cluster {ClusterId}",
-                ex.GetType().Name,
-                ScrubSecrets(ex.Message),
-                clusterId);
-            // GenerationCacheUnavailableException surfaces intentionally — fails the caller's
-            // operation. StaleConfigFlag stays unchanged; the flag only flips when we actually
-            // served a cache snapshot.
-            var snapshot = await _cache.ReadCurrentAsync(clusterId, cancellationToken).ConfigureAwait(false);
-            _staleFlag.MarkStale();
-            return fromSnapshot(snapshot);
-        }
-    }
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/LocalCache/StaleConfigFlag.cs

@@ -1,20 +0,0 @@
-namespace ZB.MOM.WW.OtOpcUa.Configuration.LocalCache;
-
-/// <summary>
-///     Thread-safe <c>UsingStaleConfig</c> signal per Phase 6.1 Stream D.3. Flips true whenever
-///     a read falls back to a sealed cache snapshot; flips false on the next successful central-DB
-///     round-trip. Surfaced on <c>/healthz</c> body and on the Admin <c>/hosts</c> page.
-/// </summary>
-public sealed class StaleConfigFlag
-{
-    private int _stale;
-
-    /// <summary>True when the last config read was served from the sealed cache, not the central DB.</summary>
-    public bool IsStale => Volatile.Read(ref _stale) != 0;
-
-    /// <summary>Mark the current config as stale (a read fell back to the cache).</summary>
-    public void MarkStale() => Volatile.Write(ref _stale, 1);
-
-    /// <summary>Mark the current config as fresh (a central-DB read succeeded).</summary>
-    public void MarkFresh() => Volatile.Write(ref _stale, 0);
-}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Migrations/20260722122301_AddClusterNodeTransportPorts.cs

@@ -0,0 +1,39 @@
+using Microsoft.EntityFrameworkCore.Migrations;
+
+#nullable disable
+
+namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
+{
+    /// <inheritdoc />
+    public partial class AddClusterNodeTransportPorts : Migration
+    {
+        /// <inheritdoc />
+        protected override void Up(MigrationBuilder migrationBuilder)
+        {
+            migrationBuilder.AddColumn<int>(
+                name: "AkkaPort",
+                table: "ClusterNode",
+                type: "int",
+                nullable: false,
+                defaultValue: 4053);
+
+            migrationBuilder.AddColumn<int>(
+                name: "GrpcPort",
+                table: "ClusterNode",
+                type: "int",
+                nullable: true);
+        }
+
+        /// <inheritdoc />
+        protected override void Down(MigrationBuilder migrationBuilder)
+        {
+            migrationBuilder.DropColumn(
+                name: "AkkaPort",
+                table: "ClusterNode");
+
+            migrationBuilder.DropColumn(
+                name: "GrpcPort",
+                table: "ClusterNode");
+        }
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Migrations/20260722125506_AddClusterNodeMaintenanceMode.cs

@@ -0,0 +1,29 @@
+using Microsoft.EntityFrameworkCore.Migrations;
+
+#nullable disable
+
+namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
+{
+    /// <inheritdoc />
+    public partial class AddClusterNodeMaintenanceMode : Migration
+    {
+        /// <inheritdoc />
+        protected override void Up(MigrationBuilder migrationBuilder)
+        {
+            migrationBuilder.AddColumn<bool>(
+                name: "MaintenanceMode",
+                table: "ClusterNode",
+                type: "bit",
+                nullable: false,
+                defaultValue: false);
+        }
+
+        /// <inheritdoc />
+        protected override void Down(MigrationBuilder migrationBuilder)
+        {
+            migrationBuilder.DropColumn(
+                name: "MaintenanceMode",
+                table: "ClusterNode");
+        }
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Migrations/OtOpcUaConfigDbContextModelSnapshot.cs

@@ -47,6 +47,11 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
                         .HasMaxLength(64)
                         .HasColumnType("nvarchar(64)");
 
+                    b.Property<int>("AkkaPort")
+                        .ValueGeneratedOnAdd()
+                        .HasColumnType("int")
+                        .HasDefaultValue(4053);
+
                     b.Property<string>("ApplicationUri")
                         .IsRequired()
                         .HasMaxLength(256)
@@ -76,6 +81,9 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
                     b.Property<bool>("Enabled")
                         .HasColumnType("bit");
 
+                    b.Property<int?>("GrpcPort")
+                        .HasColumnType("int");
+
                     b.Property<string>("Host")
                         .IsRequired()
                         .HasMaxLength(255)
@@ -84,6 +92,9 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Migrations
                     b.Property<DateTime?>("LastSeenAt")
                         .HasColumnType("datetime2(3)");
 
+                    b.Property<bool>("MaintenanceMode")
+                        .HasColumnType("bit");
+
                     b.Property<int>("OpcUaPort")
                         .HasColumnType("int");
 

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/OtOpcUaConfigDbContext.cs

@@ -142,6 +142,11 @@ public sealed class OtOpcUaConfigDbContext(DbContextOptions<OtOpcUaConfigDbConte
             e.Property(x => x.Host).HasMaxLength(255);
             e.Property(x => x.ApplicationUri).HasMaxLength(256);
             e.Property(x => x.DriverConfigOverridesJson).HasColumnType("nvarchar(max)");
+            // Central's dial targets (per-cluster mesh Phase 1). AkkaPort carries a DB-side default so
+            // rows migrated from before the column existed come out at 4053 rather than 0 — every node
+            // in the fleet does listen on a remoting port, so 0 is never a truthful value for it.
+            // GrpcPort has no default: nothing listens on it until Phase 5.
+            e.Property(x => x.AkkaPort).HasDefaultValue(4053);
             e.Property(x => x.LastSeenAt).HasColumnType("datetime2(3)");
             e.Property(x => x.CreatedAt).HasColumnType("datetime2(3)").HasDefaultValueSql("SYSUTCDATETIME()");
             e.Property(x => x.CreatedBy).HasMaxLength(128);

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Services/ILdapGroupRoleMappingService.cs

@@ -10,10 +10,18 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Services;
 ///     Phase 6.2 compliance check on control/data-plane separation).
 /// </summary>
 /// <remarks>
-///     Per Phase 6.2 Stream A.2 this service is expected to run behind the Phase 6.1
-///     <c>ResilientConfigReader</c> pipeline (timeout → retry → fallback-to-cache) so a
-///     transient DB outage during sign-in falls back to the sealed snapshot rather than
-///     denying every login.
+///     <para>
+///         This service has no local-cache fallback: a DB outage during sign-in denies logins.
+///     </para>
+///     <para>
+///         The Phase 6.1 <c>ResilientConfigReader</c> pipeline (timeout → retry →
+///         fallback-to-sealed-snapshot) this was once expected to run behind was never wired to
+///         anything and was deleted along with the rest of the dormant LiteDB local cache, which
+///         <c>ZB.MOM.WW.LocalDb</c> supersedes. LocalDb caches the deployed-configuration artifact
+///         for driver-role nodes; it does not currently cover admin-plane reads like this one.
+///         Reviving the fallback means adding an admin-side cache on LocalDb, not restoring the
+///         old pipeline.
+///     </para>
 /// </remarks>
 public interface ILdapGroupRoleMappingService
 {

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Validation/DraftValidator.cs

@@ -39,6 +39,7 @@ public static class DraftValidator
         ValidateRawNameCharset(draft, errors);
         ValidateHistorizedTagnameLength(draft, errors);
         ValidateUnsEffectiveLeafUniqueness(draft, errors);
+        ValidateEquipReferenceResolution(draft, errors);
         ValidateCalculationTags(draft, errors);
         return errors;
     }
@@ -238,6 +239,64 @@ public static class DraftValidator
         }
     }
 
+    /// <summary>v3: every <c>{{equip}}/&lt;RefName&gt;</c> used in an equipment's VirtualTag script,
+    /// ScriptedAlarm predicate script, or ScriptedAlarm message template must resolve to one of the owning
+    /// equipment's <c>UnsTagReference</c> effective names (<c>DisplayNameOverride</c> else the backing raw
+    /// tag's <c>Name</c>) — the same set the compose seams substitute against. An unresolved <c>&lt;RefName&gt;</c>
+    /// is a deploy error (replacing the v2 silent null-base no-substitution), naming the script/alarm, the
+    /// equipment, and the missing ref. This is the deploy-gate half of the invariant the Monaco diagnostic +
+    /// the authoring gate enforce (editor accepts ⇔ publish accepts).</summary>
+    private static void ValidateEquipReferenceResolution(DraftSnapshot draft, List<ValidationError> errors)
+    {
+        var tagNameById = draft.Tags.GroupBy(t => t.TagId, StringComparer.Ordinal)
+            .ToDictionary(g => g.Key, g => g.First().Name, StringComparer.Ordinal);
+
+        // equipmentId → set of reference effective names ({{equip}}/<RefName> resolves through REFERENCES
+        // only — VirtualTags/ScriptedAlarms are computed signals with no backing RawPath).
+        var refNamesByEquip = new Dictionary<string, HashSet<string>>(StringComparer.Ordinal);
+        foreach (var r in draft.UnsTagReferences)
+        {
+            var effective = r.DisplayNameOverride ?? (tagNameById.TryGetValue(r.TagId, out var n) ? n : null);
+            if (string.IsNullOrEmpty(effective)) continue;
+            if (!refNamesByEquip.TryGetValue(r.EquipmentId, out var set))
+                refNamesByEquip[r.EquipmentId] = set = new HashSet<string>(StringComparer.Ordinal);
+            set.Add(effective!);
+        }
+
+        var scriptSourceById = draft.Scripts.GroupBy(s => s.ScriptId, StringComparer.Ordinal)
+            .ToDictionary(g => g.Key, g => g.First().SourceCode, StringComparer.Ordinal);
+
+        void CheckRefs(string equipmentId, IEnumerable<string> refNames, string source)
+        {
+            var have = refNamesByEquip.TryGetValue(equipmentId, out var set) ? set : null;
+            foreach (var name in refNames)
+            {
+                if (have is not null && have.Contains(name)) continue;
+                errors.Add(new("EquipReferenceUnresolved",
+                    $"{source} uses '{EquipmentScriptPaths.EquipTokenPrefix}{name}' but equipment '{equipmentId}' has " +
+                    $"no reference named '{name}'; add a reference with that effective name or correct the script.",
+                    equipmentId));
+            }
+        }
+
+        foreach (var v in draft.VirtualTags)
+        {
+            var src = scriptSourceById.TryGetValue(v.ScriptId, out var s) ? s : null;
+            var used = EquipmentScriptPaths.ExtractEquipReferenceNames(src);
+            if (used.Count > 0) CheckRefs(v.EquipmentId, used, $"VirtualTag '{v.VirtualTagId}'");
+        }
+
+        foreach (var a in draft.ScriptedAlarms)
+        {
+            var src = scriptSourceById.TryGetValue(a.PredicateScriptId, out var s) ? s : null;
+            var used = EquipmentScriptPaths.ExtractEquipReferenceNames(src)
+                .Concat(EquipmentScriptPaths.ExtractEquipReferenceNamesFromText(a.MessageTemplate))
+                .Distinct(StringComparer.Ordinal)
+                .ToList();
+            if (used.Count > 0) CheckRefs(a.EquipmentId, used, $"ScriptedAlarm '{a.ScriptedAlarmId}'");
+        }
+    }
+
     private static bool IsValidSegment(string? s) =>
         s is not null && (UnsSegment.IsMatch(s) || s == UnsDefaultSegment);
 

src/Core/ZB.MOM.WW.OtOpcUa.Configuration/ZB.MOM.WW.OtOpcUa.Configuration.csproj

@@ -24,9 +24,12 @@
             <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
         </PackageReference>
         <PackageReference Include="Microsoft.AspNetCore.DataProtection.EntityFrameworkCore"/>
+        <!-- Surgical transitive pin — see the System.Security.Cryptography.Xml block in
+             Directory.Packages.props. This is the one project where the DataProtection chain
+             enters the repo; every other consumer reaches it through a ProjectReference here. -->
+        <PackageReference Include="System.Security.Cryptography.Xml"/>
         <PackageReference Include="Microsoft.Extensions.Configuration.Abstractions"/>
         <PackageReference Include="Microsoft.Extensions.Logging.Abstractions"/>
-        <PackageReference Include="LiteDB"/>
         <PackageReference Include="Polly.Core"/>
     </ItemGroup>
 

src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/AlarmSfSchema.cs

@@ -0,0 +1,105 @@
+using System.Security.Cryptography;
+using System.Text;
+using Microsoft.Data.Sqlite;
+
+namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
+
+/// <summary>
+///     DDL for the alarm store-and-forward buffer: one row per alarm event awaiting delivery to
+///     the historian gateway.
+/// </summary>
+/// <remarks>
+///     <para>
+///         Replaces the standalone <c>alarm-historian.db</c> the sink used to own outright. Living
+///         in the consolidated LocalDb file is what lets the buffer replicate to the redundant pair
+///         peer, so a node that dies with undelivered alarm history no longer takes it to the grave.
+///     </para>
+///     <para>
+///         Deliberately depends on nothing but <see cref="SqliteConnection"/> so it can be applied
+///         to any connection — the host's <c>LocalDbSetup.OnReady</c> in production, and a bare
+///         connection in a test — without dragging the DI graph along. Mirrors
+///         <c>DeploymentCacheSchema</c>.
+///     </para>
+///     <para>
+///         <b>Why <see cref="IdColumn"/> is TEXT and app-minted.</b> The legacy queue keyed on
+///         <c>RowId INTEGER PRIMARY KEY AUTOINCREMENT</c>. Convergence is last-writer-wins over the
+///         primary key, so two nodes independently allocating rowid 7 for different alarms would
+///         silently overwrite one another. The sink mints the id from a hash of the payload, which
+///         additionally makes the same event converge to one row when both nodes of a pair enqueue
+///         it — as they legitimately do in the window before the first redundancy snapshot arrives.
+///     </para>
+/// </remarks>
+public static class AlarmSfSchema
+{
+    /// <summary>Table holding queued alarm events awaiting delivery.</summary>
+    public const string EventsTable = "alarm_sf_events";
+
+    /// <summary>The single primary-key column.</summary>
+    public const string IdColumn = "id";
+
+    /// <summary>
+    ///     Derives a row's primary key from its serialized payload.
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         Deterministic rather than a fresh GUID, so that the same event arriving twice
+    ///         converges to one row under last-writer-wins instead of duplicating. That happens for
+    ///         real in two places: <c>HistorianAdapterActor</c> default-writes while its redundancy
+    ///         role is unknown, so both nodes of a pair accept the same fanned transition in the
+    ///         window before the first snapshot; and both nodes independently migrate their own
+    ///         legacy queue file, which for a warm pair holds overlapping history.
+    ///     </para>
+    ///     <para>
+    ///         Two genuinely distinct events cannot collide. <c>AlarmHistorianEvent</c> carries a
+    ///         full-precision timestamp alongside the alarm id, transition kind, message and user,
+    ///         so an equal hash means an equal event.
+    ///     </para>
+    /// </remarks>
+    /// <param name="payloadJson">The serialized <c>AlarmHistorianEvent</c>.</param>
+    /// <returns>The row's primary key.</returns>
+    public static string DeriveId(string payloadJson)
+    {
+        ArgumentNullException.ThrowIfNull(payloadJson);
+        return Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(payloadJson)));
+    }
+
+    /// <summary>
+    ///     Creates the buffer table if it does not already exist. Idempotent.
+    /// </summary>
+    /// <param name="connection">
+    ///     An already-open connection. <c>ILocalDb.CreateConnection()</c> hands out open,
+    ///     pragma-configured connections — do not call <c>Open()</c> on one.
+    /// </param>
+    public static void Apply(SqliteConnection connection)
+    {
+        ArgumentNullException.ThrowIfNull(connection);
+
+        using var cmd = connection.CreateCommand();
+
+        // Columns map 1:1 onto the legacy Queue table so the one-time migrator is a straight copy
+        // and the drain keeps its existing semantics: dead_lettered is the same 0/1 flag, and an
+        // acknowledged row is DELETEd rather than marked. Deleting keeps the table bounded without
+        // a second sweeper, and the replication engine carries the delete as a tombstone, so the
+        // peer drops its copy too.
+        //
+        // The drain index orders by enqueued_at_utc because a hashed TEXT id carries no insertion
+        // order the way the legacy AUTOINCREMENT RowId did. Timestamps are round-trip ("O") format,
+        // so lexicographic ordering is chronological; id breaks ties so the order is total and the
+        // index covers the drain's read.
+        cmd.CommandText = """
+            CREATE TABLE IF NOT EXISTS alarm_sf_events (
+                id               TEXT    NOT NULL PRIMARY KEY,
+                alarm_id         TEXT    NOT NULL,
+                enqueued_at_utc  TEXT    NOT NULL,
+                payload_json     TEXT    NOT NULL,
+                attempt_count    INTEGER NOT NULL DEFAULT 0,
+                last_attempt_utc TEXT    NULL,
+                last_error       TEXT    NULL,
+                dead_lettered    INTEGER NOT NULL DEFAULT 0
+            );
+            CREATE INDEX IF NOT EXISTS ix_alarm_sf_events_drain
+                ON alarm_sf_events (dead_lettered, enqueued_at_utc, id);
+            """;
+        cmd.ExecuteNonQuery();
+    }
+}

src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/IAlarmHistorianSink.cs

@@ -4,13 +4,13 @@ namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
 ///     The historian sink contract — where qualifying alarm events land. Ingestion routes
 ///     through the HistorianGateway alarm writer (the gateway's <c>SendEvent</c> gRPC path)
 ///     behind the durable store-and-forward queue. Tests use an in-memory fake; production uses
-///     <see cref="SqliteStoreAndForwardSink"/>.
+///     <see cref="LocalDbStoreAndForwardSink"/>.
 /// </summary>
 /// <remarks>
 ///     <para>
 ///         <see cref="EnqueueAsync"/> is fire-and-forget from the engine's perspective —
 ///         the sink MUST NOT block the emitting thread. Production implementations
-///         (<see cref="SqliteStoreAndForwardSink"/>) persist to a local SQLite queue
+///         (<see cref="LocalDbStoreAndForwardSink"/>) persist to the node's local SQLite queue
 ///         first, then drain asynchronously to the actual historian. Per the Phase 7 plan,
 ///         failed downstream writes replay with exponential backoff;
 ///         operator actions are never blocked waiting on the historian.
@@ -79,6 +79,18 @@ public enum HistorianDrainState
     Idle,
     Draining,
     BackingOff,
+
+    /// <summary>
+    ///     Ticking, but not draining: this node does not hold the Primary role, so it leaves the
+    ///     (replicated) queue for the node that does.
+    /// </summary>
+    /// <remarks>
+    ///     Distinct from <see cref="Idle"/> so a rising queue depth on a Secondary reads as the
+    ///     designed behaviour rather than a stalled drain. It is also how a misconfigured pair
+    ///     becomes diagnosable: if BOTH nodes report this, no one is draining and alarm history is
+    ///     silently accumulating toward the capacity ceiling.
+    /// </remarks>
+    NotPrimary,
 }
 
 /// <summary>Returned by the historian alarm writer per event — drain worker uses this to decide retry cadence.</summary>

src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/LocalDbStoreAndForwardSink.cs

@@ -1,53 +1,54 @@
 using System.Text.Json;
-using Microsoft.Data.Sqlite;
 using Serilog;
+using ZB.MOM.WW.LocalDb;
 
 namespace ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
 
 /// <summary>
-///     Durable SQLite queue on the node absorbs every qualifying alarm event, a drain
-///     worker batches rows to the Wonderware historian sidecar via
-///     <see cref="IAlarmHistorianWriter"/> on an exponential-backoff cadence, and
-///     operator acks never block on the historian being reachable.
+///     Durable queue in the node's consolidated LocalDb absorbs every qualifying alarm event, a
+///     drain worker batches rows to the historian gateway via <see cref="IAlarmHistorianWriter"/>
+///     on an exponential-backoff cadence, and operator acks never block on the historian being
+///     reachable.
 /// </summary>
 /// <remarks>
 ///     <para>
-///         Queue schema:
-///         <code>
-///         CREATE TABLE Queue (
-///             RowId INTEGER PRIMARY KEY AUTOINCREMENT,
-///             AlarmId TEXT NOT NULL,
-///             EnqueuedUtc TEXT NOT NULL,
-///             PayloadJson TEXT NOT NULL,
-///             AttemptCount INTEGER NOT NULL DEFAULT 0,
-///             LastAttemptUtc TEXT NULL,
-///             LastError TEXT NULL,
-///             DeadLettered INTEGER NOT NULL DEFAULT 0
-///         );
-///         </code>
-///         Dead-lettered rows stay in place for the configured retention window (default
-///         30 days) so operators can inspect + manually
-///         retry before the sweeper purges them. Regular queue capacity is bounded —
-///         overflow evicts the oldest non-dead-lettered rows with a WARN log. The
-///         durability guarantee is therefore bounded by <see cref="DefaultCapacity"/>:
-///         under a sustained historian outage, accepted events may be evicted before
-///         delivery. The <see cref="HistorianSinkStatus.EvictedCount"/> counter makes
-///         overflow visible to operators without requiring the WARN log to be scraped.
+///         Rows live in <c>alarm_sf_events</c> (see <see cref="AlarmSfSchema"/>), which the host
+///         registers for replication. That is the point of this type living on
+///         <see cref="ILocalDb"/> rather than owning its own file: the buffer mirrors to the
+///         redundant pair peer, so a node that dies holding undelivered alarm history no longer
+///         takes it to the grave.
+///     </para>
+///     <para>
+///         <b>Only one node of a pair may drain.</b> Replication puts the Primary's queued rows in
+///         the Secondary's table too; an ungated drain there would re-deliver every event,
+///         continuously. The <c>drainGate</c> constructor argument is how the caller scopes
+///         draining to the Primary. Enqueue is gated separately and upstream, by
+///         <c>HistorianAdapterActor</c>.
+///     </para>
+///     <para>
+///         Dead-lettered rows stay in place for the configured retention window (default 30 days)
+///         so operators can inspect + manually retry before the sweeper purges them. Regular queue
+///         capacity is bounded — overflow evicts the oldest non-dead-lettered rows with a WARN log.
+///         The durability guarantee is therefore bounded by <see cref="DefaultCapacity"/>: under a
+///         sustained historian outage, accepted events may be evicted before delivery. The
+///         <see cref="HistorianSinkStatus.EvictedCount"/> counter makes overflow visible to
+///         operators without requiring the WARN log to be scraped.
 ///     </para>
 ///     <para>
 ///         Drain runs on a self-rescheduling one-shot <see cref="System.Threading.Timer"/>.
 ///         Exponential backoff on <see cref="HistorianWriteOutcome.RetryPlease"/>:
-///         1s → 2s → 5s → 15s → 60s cap — the backoff is applied to the timer's next
-///         due-time, so a historian outage genuinely slows the drain cadence.
-///         <see cref="HistorianWriteOutcome.PermanentFail"/> rows flip
-///         the <c>DeadLettered</c> flag on the individual row; neighbors in the batch
-///         still retry on their own cadence.
+///         1s → 2s → 5s → 15s → 60s cap — the backoff is applied to the timer's next due-time, so a
+///         historian outage genuinely slows the drain cadence.
+///         <see cref="HistorianWriteOutcome.PermanentFail"/> rows flip the <c>dead_lettered</c> flag
+///         on the individual row; neighbors in the batch still retry on their own cadence.
 ///     </para>
 /// </remarks>
-public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
+public sealed class LocalDbStoreAndForwardSink : IAlarmHistorianSink, IDisposable
 {
     /// <summary>Default queue capacity — oldest non-dead-lettered rows evicted past this.</summary>
     public const long DefaultCapacity = 1_000_000;
+
+    /// <summary>Default window dead-lettered rows are retained for before the sweeper purges them.</summary>
     public static readonly TimeSpan DefaultDeadLetterRetention = TimeSpan.FromDays(30);
 
     /// <summary>Default max delivery attempts before a perpetually-retrying (poison) row is dead-lettered.</summary>
@@ -62,7 +63,7 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
         TimeSpan.FromSeconds(60),
     ];
 
-    private readonly string _connectionString;
+    private readonly ILocalDb _db;
     private readonly IAlarmHistorianWriter _writer;
     private readonly ILogger _logger;
     private readonly int _batchSize;
@@ -70,8 +71,9 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     private readonly TimeSpan _deadLetterRetention;
     private readonly int _maxAttempts;
     private readonly Func<DateTime> _clock;
+    private readonly Func<bool> _drainGate;
 
-    private readonly SemaphoreSlim _drainGate = new(1, 1);
+    private readonly SemaphoreSlim _drainGateLock = new(1, 1);
     private Timer? _drainTimer;
     private TimeSpan _tickInterval;
     private volatile int _backoffIndex;
@@ -85,13 +87,16 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     private string? _lastError;
     private HistorianDrainState _drainState = HistorianDrainState.Idle;
     private long _evictedCount;
+    // Tracks the last gate decision so a denial is logged on the transition rather than on every
+    // tick. Null until the first drain runs.
+    private bool? _lastGateDecision;
 
     private long _queuedRowCount;
     // Probe counter — incremented every time we actually issue a real COUNT(*) for
     // capacity enforcement. Public for test instrumentation only.
     private long _capacityProbeCount;
     // After every Nth enqueue we resync the in-memory counter from storage to defend
-    // against silent drift (e.g. an external process editing the DB).
+    // against silent drift (e.g. the peer replicating rows in underneath us).
     private const long ResyncEnqueueInterval = 10_000;
     private long _enqueuesSinceResync;
 
@@ -99,9 +104,13 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     public long DebugCapacityProbeCount => Interlocked.Read(ref _capacityProbeCount);
 
     /// <summary>
-    ///     Initializes a new instance of the <see cref="SqliteStoreAndForwardSink"/> class with the specified configuration.
+    ///     Initializes a new instance of the <see cref="LocalDbStoreAndForwardSink"/> class.
     /// </summary>
-    /// <param name="databasePath">The filesystem path to the SQLite database file.</param>
+    /// <param name="db">
+    ///     The node's local database. Its <c>alarm_sf_events</c> table must already exist and be
+    ///     registered for replication — the host does both in <c>LocalDbSetup.OnReady</c>, before
+    ///     any consumer can resolve this sink.
+    /// </param>
     /// <param name="writer">The alarm historian writer to handle batch forwarding.</param>
     /// <param name="logger">The logger for diagnostic output.</param>
     /// <param name="batchSize">The maximum number of rows to forward in a single batch. Defaults to 100.</param>
@@ -109,18 +118,24 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     /// <param name="deadLetterRetention">The timespan to retain dead-lettered rows before purging. Defaults to 30 days.</param>
     /// <param name="maxAttempts">The maximum number of delivery attempts before a perpetually-retrying (poison) row is dead-lettered. Defaults to 10.</param>
     /// <param name="clock">Optional clock function for testing; defaults to <see cref="DateTime.UtcNow"/>.</param>
-    public SqliteStoreAndForwardSink(
-        string databasePath,
+    /// <param name="drainGate">
+    ///     Consulted at the top of every drain tick; <c>false</c> skips the tick entirely, leaving
+    ///     the queue intact. Callers running a redundant pair MUST supply a gate that is true on at
+    ///     most one node, because the queue replicates — see the class remarks. The default
+    ///     (<c>null</c> ⇒ always drain) is the correct single-node and test posture.
+    /// </param>
+    public LocalDbStoreAndForwardSink(
+        ILocalDb db,
         IAlarmHistorianWriter writer,
         ILogger logger,
         int batchSize = 100,
         long capacity = DefaultCapacity,
         TimeSpan? deadLetterRetention = null,
         int maxAttempts = DefaultMaxAttempts,
-        Func<DateTime>? clock = null)
+        Func<DateTime>? clock = null,
+        Func<bool>? drainGate = null)
     {
-        if (string.IsNullOrWhiteSpace(databasePath))
-            throw new ArgumentException("Database path required.", nameof(databasePath));
+        _db = db ?? throw new ArgumentNullException(nameof(db));
         _writer = writer ?? throw new ArgumentNullException(nameof(writer));
         _logger = logger ?? throw new ArgumentNullException(nameof(logger));
         _batchSize = batchSize > 0 ? batchSize : throw new ArgumentOutOfRangeException(nameof(batchSize));
@@ -128,50 +143,11 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
         _deadLetterRetention = deadLetterRetention ?? DefaultDeadLetterRetention;
         _maxAttempts = maxAttempts > 0 ? maxAttempts : throw new ArgumentOutOfRangeException(nameof(maxAttempts));
         _clock = clock ?? (() => DateTime.UtcNow);
-        // DefaultTimeout gives ADO.NET command-level retry; the PRAGMA busy_timeout
-        // applied in OpenConnection backs it with SQLite's own busy-handler so an
-        // enqueue/drain collision waits out the file lock instead of throwing
-        // SQLITE_BUSY immediately.
-        _connectionString = new SqliteConnectionStringBuilder
-        {
-            DataSource = databasePath,
-            DefaultTimeout = 5,
-        }.ToString();
+        _drainGate = drainGate ?? (static () => true);
 
-        InitializeSchema();
         _queuedRowCount = ProbeQueuedRowCount();
     }
 
-    /// <summary>
-    ///     Open a connection with the busy timeout + WAL journal applied. SQLite
-    ///     serializes writers with a file lock; the busy_timeout lets a writer wait
-    ///     out a competing lock (default is 0 — fail fast), and WAL lets readers and
-    ///     the single writer proceed without blocking each other.
-    /// </summary>
-    private SqliteConnection OpenConnection()
-    {
-        var conn = new SqliteConnection(_connectionString);
-        conn.Open();
-        ApplyPragmas(conn);
-        return conn;
-    }
-
-    /// <summary>Apply busy_timeout + WAL pragmas to an already-open connection (sync).</summary>
-    private static void ApplyPragmas(SqliteConnection conn)
-    {
-        using var pragma = conn.CreateCommand();
-        pragma.CommandText = "PRAGMA busy_timeout=5000; PRAGMA journal_mode=WAL;";
-        pragma.ExecuteNonQuery();
-    }
-
-    /// <summary>Apply busy_timeout + WAL pragmas to an already-open connection (async).</summary>
-    private static async Task ApplyPragmasAsync(SqliteConnection conn, CancellationToken ct)
-    {
-        using var pragma = conn.CreateCommand();
-        pragma.CommandText = "PRAGMA busy_timeout=5000; PRAGMA journal_mode=WAL;";
-        await pragma.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
-    }
-
     /// <summary>
     ///     Start the background drain worker. Not started automatically so tests can
     ///     drive <see cref="DrainOnceAsync"/> deterministically.
@@ -188,7 +164,7 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     /// <param name="tickInterval">The base interval between drain attempts.</param>
     public void StartDrainLoop(TimeSpan tickInterval)
     {
-        if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
+        if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
         _tickInterval = tickInterval;
         _drainTimer?.Dispose();
         // One-shot: dueTime = tickInterval, period = Infinite. RescheduleDrain re-arms
@@ -239,25 +215,31 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     public async Task EnqueueAsync(AlarmHistorianEvent evt, CancellationToken cancellationToken)
     {
         if (evt is null) throw new ArgumentNullException(nameof(evt));
-        if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
+        if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
 
-        using var conn = new SqliteConnection(_connectionString);
-        await conn.OpenAsync(cancellationToken).ConfigureAwait(false);
-        await ApplyPragmasAsync(conn, cancellationToken).ConfigureAwait(false);
+        await EnforceCapacityFastPathAsync(cancellationToken).ConfigureAwait(false);
 
-        await EnforceCapacityFastPathAsync(conn, cancellationToken).ConfigureAwait(false);
+        var payload = JsonSerializer.Serialize(evt);
 
-        using var cmd = conn.CreateCommand();
-        cmd.CommandText = """
-            INSERT INTO Queue (AlarmId, EnqueuedUtc, PayloadJson, AttemptCount)
-            VALUES ($alarmId, $enqueued, $payload, 0);
-            """;
-        cmd.Parameters.AddWithValue("$alarmId", evt.AlarmId);
-        cmd.Parameters.AddWithValue("$enqueued", _clock().ToString("O"));
-        cmd.Parameters.AddWithValue("$payload", JsonSerializer.Serialize(evt));
-        await cmd.ExecuteNonQueryAsync(cancellationToken).ConfigureAwait(false);
+        // ON CONFLICT DO NOTHING: re-enqueuing an identical event (the pair's boot-window
+        // double-accept) must not resurrect a row the drain already bumped or dead-lettered.
+        var inserted = await _db.ExecuteAsync(
+            """
+            INSERT INTO alarm_sf_events
+                (id, alarm_id, enqueued_at_utc, payload_json, attempt_count)
+            VALUES (@Id, @AlarmId, @EnqueuedAtUtc, @PayloadJson, 0)
+            ON CONFLICT(id) DO NOTHING
+            """,
+            new
+            {
+                Id = AlarmSfSchema.DeriveId(payload),
+                AlarmId = evt.AlarmId,
+                EnqueuedAtUtc = _clock().ToString("O"),
+                PayloadJson = payload,
+            },
+            cancellationToken).ConfigureAwait(false);
 
-        Interlocked.Increment(ref _queuedRowCount);
+        if (inserted > 0) Interlocked.Increment(ref _queuedRowCount);
     }
 
     /// <summary>
@@ -268,15 +250,17 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     ///     through <see cref="EnforceCapacityAsync"/> which still runs a precise
     ///     COUNT to compute the exact number of rows to evict.
     /// </summary>
-    private async Task EnforceCapacityFastPathAsync(SqliteConnection conn, CancellationToken ct)
+    private async Task EnforceCapacityFastPathAsync(CancellationToken ct)
     {
         var enqueuesSinceResync = Interlocked.Increment(ref _enqueuesSinceResync);
         var cached = Interlocked.Read(ref _queuedRowCount);
 
-        // Periodic resync — bounded amount of drift even under exotic conditions.
+        // Periodic resync — bounded amount of drift even under exotic conditions. Now doubly
+        // warranted: replication applies the peer's rows straight into the table, so the counter
+        // has a second way to fall behind that no local code path can observe.
         if (enqueuesSinceResync >= ResyncEnqueueInterval)
         {
-            await ResyncQueuedRowCountAsync(conn, ct).ConfigureAwait(false);
+            await ResyncQueuedRowCountAsync(ct).ConfigureAwait(false);
             cached = Interlocked.Read(ref _queuedRowCount);
             Interlocked.Exchange(ref _enqueuesSinceResync, 0);
         }
@@ -286,55 +270,70 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
 
         // Cached counter says we're at or above the capacity wall — fall back to the
         // precise path which probes COUNT(*) and evicts whatever's needed.
-        await EnforceCapacityAsync(conn, ct).ConfigureAwait(false);
+        await EnforceCapacityAsync(ct).ConfigureAwait(false);
     }
 
     /// <summary>Synchronously query <c>COUNT(*)</c> of non-dead-lettered rows. Used at startup.</summary>
     private long ProbeQueuedRowCount()
     {
         Interlocked.Increment(ref _capacityProbeCount);
-        using var conn = OpenConnection();
+        using var conn = _db.CreateConnection();
         using var cmd = conn.CreateCommand();
-        cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
+        cmd.CommandText = "SELECT COUNT(*) FROM alarm_sf_events WHERE dead_lettered = 0";
         return (long)(cmd.ExecuteScalar() ?? 0L);
     }
 
     /// <summary>Re-sync the in-memory counter from storage (async path).</summary>
-    private async Task ResyncQueuedRowCountAsync(SqliteConnection conn, CancellationToken ct)
+    private async Task ResyncQueuedRowCountAsync(CancellationToken ct)
     {
         Interlocked.Increment(ref _capacityProbeCount);
-        using var cmd = conn.CreateCommand();
-        cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
-        var live = (long)(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false) ?? 0L);
+        var live = await CountAsync("dead_lettered = 0", ct).ConfigureAwait(false);
         Interlocked.Exchange(ref _queuedRowCount, live);
     }
 
+    private async Task<long> CountAsync(string predicate, CancellationToken ct)
+    {
+        var rows = await _db.QueryAsync(
+            $"SELECT COUNT(*) FROM alarm_sf_events WHERE {predicate}",
+            r => r.GetInt64(0),
+            parameters: null,
+            ct).ConfigureAwait(false);
+        return rows.Count > 0 ? rows[0] : 0L;
+    }
+
     /// <summary>
     ///     Read up to <see cref="_batchSize"/> queued rows, forward through the writer,
     ///     remove Ack'd rows, dead-letter PermanentFail rows, and extend the backoff
     ///     on RetryPlease. Safe to call from multiple threads; the semaphore enforces
     ///     serial execution.
     /// </summary>
+    /// <remarks>
+    ///     Returns without touching the queue when the drain gate is closed. On a redundant pair
+    ///     that is the Secondary's steady state: its table fills by replication and stays put until
+    ///     it is promoted.
+    /// </remarks>
     /// <param name="ct">Cancellation token for the operation.</param>
     /// <returns>A task that represents the asynchronous operation.</returns>
     public async Task DrainOnceAsync(CancellationToken ct)
     {
         if (_disposed) return;
-        if (!await _drainGate.WaitAsync(0, ct).ConfigureAwait(false)) return;
+        if (!await _drainGateLock.WaitAsync(0, ct).ConfigureAwait(false)) return;
         try
         {
+            if (!EvaluateDrainGate())
+            {
+                lock (_statusLock) { _drainState = HistorianDrainState.NotPrimary; }
+                return;
+            }
+
             lock (_statusLock)
             {
                 _drainState = HistorianDrainState.Draining;
                 _lastDrainUtc = _clock();
             }
 
-            // One connection per drain tick — used by purge, read, corrupt-dead-letter,
-            // and the outcome-applying transaction.
-            using var conn = OpenConnection();
-
-            PurgeAgedDeadLetters(conn);
-            var batch = ReadBatch(conn);
+            await PurgeAgedDeadLettersAsync(ct).ConfigureAwait(false);
+            var batch = await ReadBatchAsync(ct).ConfigureAwait(false);
             if (batch.Count == 0)
             {
                 lock (_statusLock) { _drainState = HistorianDrainState.Idle; }
@@ -342,22 +341,24 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
             }
 
             // A null/un-deserializable payload can never succeed — dead-letter it
-            // immediately for its own RowId so it cannot stall the queue head, and
+            // immediately for its own id so it cannot stall the queue head, and
             // exclude it from the batch handed to the writer.
-            var corruptRowIds = batch.Where(r => r.Event is null).Select(r => r.RowId).ToList();
+            var corruptIds = batch.Where(r => r.Event is null).Select(r => r.Id).ToList();
             var liveRows = batch.Where(r => r.Event is not null).ToList();
             var events = liveRows.Select(r => r.Event!).ToList();
 
-            if (corruptRowIds.Count > 0)
+            if (corruptIds.Count > 0)
             {
-                using var corruptTx = conn.BeginTransaction();
-                foreach (var rowId in corruptRowIds)
-                    DeadLetterRow(conn, corruptTx, rowId, $"corrupt payload at {_clock():O}");
-                corruptTx.Commit();
-                Interlocked.Add(ref _queuedRowCount, -corruptRowIds.Count);
+                await using (var corruptTx = await _db.BeginTransactionAsync(ct).ConfigureAwait(false))
+                {
+                    foreach (var id in corruptIds)
+                        await DeadLetterRowAsync(corruptTx, id, $"corrupt payload at {_clock():O}", ct).ConfigureAwait(false);
+                    await corruptTx.CommitAsync(ct).ConfigureAwait(false);
+                }
+                Interlocked.Add(ref _queuedRowCount, -corruptIds.Count);
                 _logger.Warning(
                     "Dead-lettered {Count} historian queue row(s) with un-deserializable payload",
-                    corruptRowIds.Count);
+                    corruptIds.Count);
             }
 
             if (events.Count == 0)
@@ -404,41 +405,44 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
             }
 
             int rowsLeavingQueue = 0;
-            using (var tx = conn.BeginTransaction())
+            await using (var tx = await _db.BeginTransactionAsync(ct).ConfigureAwait(false))
             {
                 for (var i = 0; i < outcomes.Count; i++)
                 {
                     var outcome = outcomes[i];
-                    var rowId = liveRows[i].RowId;
+                    var id = liveRows[i].Id;
                     switch (outcome)
                     {
                         case HistorianWriteOutcome.Ack:
-                            DeleteRow(conn, tx, rowId);
+                            // Deleted, not marked delivered. The replication engine carries the
+                            // delete as a tombstone, so the peer drops its copy and cannot
+                            // re-deliver the row after a failover.
+                            await DeleteRowAsync(tx, id, ct).ConfigureAwait(false);
                             rowsLeavingQueue++;
                             break;
                         case HistorianWriteOutcome.PermanentFail:
-                            DeadLetterRow(conn, tx, rowId, $"permanent fail at {_clock():O}");
+                            await DeadLetterRowAsync(tx, id, $"permanent fail at {_clock():O}", ct).ConfigureAwait(false);
                             rowsLeavingQueue++;
                             break;
                         case HistorianWriteOutcome.RetryPlease:
                             // finding 002: cap retries so a perpetually-RetryPlease (poison)
                             // row cannot retry forever at the 60s backoff floor. The incoming
-                            // AttemptCount is the count BEFORE this attempt; +1 accounts for the
+                            // attempt count is the count BEFORE this attempt; +1 accounts for the
                             // bump this drain represents. At the cap, dead-letter instead of
                             // bumping — and count it as leaving the live queue like PermanentFail.
                             if (liveRows[i].AttemptCount + 1 >= _maxAttempts)
                             {
-                                DeadLetterRow(conn, tx, rowId, $"max attempts ({_maxAttempts}) exceeded");
+                                await DeadLetterRowAsync(tx, id, $"max attempts ({_maxAttempts}) exceeded", ct).ConfigureAwait(false);
                                 rowsLeavingQueue++;
                             }
                             else
                             {
-                                BumpAttempt(conn, tx, rowId, "retry-please");
+                                await BumpAttemptAsync(tx, id, "retry-please", ct).ConfigureAwait(false);
                             }
                             break;
                     }
                 }
-                tx.Commit();
+                await tx.CommitAsync(ct).ConfigureAwait(false);
             }
             // Ack-deleted + PermanentFail-dead-lettered rows both leave the
             // non-dead-lettered queue, as do RetryPlease rows that hit the max-attempts
@@ -464,22 +468,66 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
         }
         finally
         {
-            _drainGate.Release();
+            _drainGateLock.Release();
         }
     }
 
+    /// <summary>
+    ///     Consults the injected gate, logging only when the decision changes.
+    /// </summary>
+    /// <remarks>
+    ///     Logged at all because the closed state is indistinguishable from a healthy idle queue
+    ///     from the outside, and one way to reach it is a redundancy snapshot that never names this
+    ///     node — the documented identity-mismatch shape. Without this line, that misconfiguration
+    ///     would present as alarm history quietly ceasing on both nodes of a pair. Logged on the
+    ///     transition rather than per tick because the drain runs every few seconds.
+    /// </remarks>
+    /// <returns><c>true</c> when this tick may proceed.</returns>
+    private bool EvaluateDrainGate()
+    {
+        bool open;
+        try
+        {
+            open = _drainGate();
+        }
+        catch (Exception ex)
+        {
+            // A throwing gate must not wedge the queue silently, and must not be treated as
+            // permission either: skip this tick, say why, and re-ask on the next one.
+            _logger.Error(ex, "Historian drain gate threw; skipping this tick");
+            return false;
+        }
+
+        bool? previous;
+        lock (_statusLock)
+        {
+            previous = _lastGateDecision;
+            _lastGateDecision = open;
+        }
+
+        if (previous == open) return open;
+
+        if (open)
+            _logger.Information("Historian drain resumed — this node now services the alarm queue");
+        else
+            _logger.Information(
+                "Historian drain suspended — this node is not the Primary. Queued alarm events are retained and will drain from whichever node holds the Primary role");
+
+        return open;
+    }
+
     /// <inheritdoc />
     public HistorianSinkStatus GetStatus()
     {
-        if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
+        if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
         var queued = Interlocked.Read(ref _queuedRowCount);
         if (queued < 0) queued = 0;
 
         long deadlettered;
-        using (var conn = OpenConnection())
+        using (var conn = _db.CreateConnection())
         using (var cmd = conn.CreateCommand())
         {
-            cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 1";
+            cmd.CommandText = "SELECT COUNT(*) FROM alarm_sf_events WHERE dead_lettered = 1";
             deadlettered = (long)(cmd.ExecuteScalar() ?? 0L);
         }
 
@@ -510,10 +558,13 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     /// <returns>The number of rows revived from the dead-letter state.</returns>
     public int RetryDeadLettered()
     {
-        if (_disposed) throw new ObjectDisposedException(nameof(SqliteStoreAndForwardSink));
-        using var conn = OpenConnection();
+        if (_disposed) throw new ObjectDisposedException(nameof(LocalDbStoreAndForwardSink));
+        // Through a LocalDb connection, not a private one: the capture triggers are what make this
+        // revival replicate to the peer, and they only exist on the registered table.
+        using var conn = _db.CreateConnection();
         using var cmd = conn.CreateCommand();
-        cmd.CommandText = "UPDATE Queue SET DeadLettered = 0, AttemptCount = 0, LastError = NULL WHERE DeadLettered = 1";
+        cmd.CommandText =
+            "UPDATE alarm_sf_events SET dead_lettered = 0, attempt_count = 0, last_error = NULL WHERE dead_lettered = 1";
         var revived = cmd.ExecuteNonQuery();
         // Dead-lettered rows rejoin the non-dead-lettered queue — keep the in-memory
         // counter aligned.
@@ -523,29 +574,31 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
 
     /// <summary>
     ///     One queued row paired with its deserialized event. <see cref="Event"/> is
-    ///     <c>null</c> when the row's <c>PayloadJson</c> is corrupt or un-deserializable —
-    ///     the <see cref="RowId"/> always stays bound to its own row so outcomes can
+    ///     <c>null</c> when the row's <c>payload_json</c> is corrupt or un-deserializable —
+    ///     the <see cref="Id"/> always stays bound to its own row so outcomes can
     ///     never be mapped to the wrong row.
     /// </summary>
-    private readonly record struct QueueRow(long RowId, AlarmHistorianEvent? Event, long AttemptCount);
+    private readonly record struct QueueRow(string Id, AlarmHistorianEvent? Event, long AttemptCount);
 
-    private List<QueueRow> ReadBatch(SqliteConnection conn)
+    private async Task<List<QueueRow>> ReadBatchAsync(CancellationToken ct)
     {
-        var rows = new List<QueueRow>();
-        using var cmd = conn.CreateCommand();
-        cmd.CommandText = """
-            SELECT RowId, PayloadJson, AttemptCount FROM Queue
-            WHERE DeadLettered = 0
-            ORDER BY RowId ASC
-            LIMIT $limit
-            """;
-        cmd.Parameters.AddWithValue("$limit", _batchSize);
-        using var reader = cmd.ExecuteReader();
-        while (reader.Read())
+        // Ordered by enqueue time rather than insertion order: a hashed TEXT primary key carries
+        // no sequence the way the legacy AUTOINCREMENT rowid did. Round-trip ("O") timestamps sort
+        // lexicographically in chronological order; id makes the ordering total.
+        var raw = await _db.QueryAsync(
+            """
+            SELECT id, payload_json, attempt_count FROM alarm_sf_events
+            WHERE dead_lettered = 0
+            ORDER BY enqueued_at_utc ASC, id ASC
+            LIMIT @Limit
+            """,
+            r => (Id: r.GetString(0), Payload: r.GetString(1), AttemptCount: r.GetInt64(2)),
+            new { Limit = _batchSize },
+            ct).ConfigureAwait(false);
+
+        var rows = new List<QueueRow>(raw.Count);
+        foreach (var (id, payload, attemptCount) in raw)
         {
-            var rowId = reader.GetInt64(0);
-            var payload = reader.GetString(1);
-            var attemptCount = reader.GetInt64(2);
             AlarmHistorianEvent? evt;
             try
             {
@@ -556,77 +609,58 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
                 // Malformed JSON — carry a null event so the caller dead-letters this row.
                 evt = null;
             }
-            rows.Add(new QueueRow(rowId, evt, attemptCount));
+            rows.Add(new QueueRow(id, evt, attemptCount));
         }
         return rows;
     }
 
-    private static void DeleteRow(SqliteConnection conn, SqliteTransaction tx, long rowId)
-    {
-        using var cmd = conn.CreateCommand();
-        cmd.Transaction = tx;
-        cmd.CommandText = "DELETE FROM Queue WHERE RowId = $id";
-        cmd.Parameters.AddWithValue("$id", rowId);
-        cmd.ExecuteNonQuery();
-    }
+    private static Task DeleteRowAsync(ILocalDbTransaction tx, string id, CancellationToken ct) =>
+        tx.ExecuteAsync("DELETE FROM alarm_sf_events WHERE id = @Id", new { Id = id }, ct);
 
-    private void DeadLetterRow(SqliteConnection conn, SqliteTransaction tx, long rowId, string reason)
-    {
-        using var cmd = conn.CreateCommand();
-        cmd.Transaction = tx;
-        cmd.CommandText = """
-            UPDATE Queue SET DeadLettered = 1, LastAttemptUtc = $now, LastError = $err, AttemptCount = AttemptCount + 1
-            WHERE RowId = $id
-            """;
-        cmd.Parameters.AddWithValue("$now", _clock().ToString("O"));
-        cmd.Parameters.AddWithValue("$err", reason);
-        cmd.Parameters.AddWithValue("$id", rowId);
-        cmd.ExecuteNonQuery();
-    }
+    private Task DeadLetterRowAsync(ILocalDbTransaction tx, string id, string reason, CancellationToken ct) =>
+        tx.ExecuteAsync(
+            """
+            UPDATE alarm_sf_events
+            SET dead_lettered = 1, last_attempt_utc = @Now, last_error = @Err,
+                attempt_count = attempt_count + 1
+            WHERE id = @Id
+            """,
+            new { Now = _clock().ToString("O"), Err = reason, Id = id },
+            ct);
 
-    private void BumpAttempt(SqliteConnection conn, SqliteTransaction tx, long rowId, string reason)
-    {
-        using var cmd = conn.CreateCommand();
-        cmd.Transaction = tx;
-        cmd.CommandText = """
-            UPDATE Queue SET LastAttemptUtc = $now, LastError = $err, AttemptCount = AttemptCount + 1
-            WHERE RowId = $id
-            """;
-        cmd.Parameters.AddWithValue("$now", _clock().ToString("O"));
-        cmd.Parameters.AddWithValue("$err", reason);
-        cmd.Parameters.AddWithValue("$id", rowId);
-        cmd.ExecuteNonQuery();
-    }
+    private Task BumpAttemptAsync(ILocalDbTransaction tx, string id, string reason, CancellationToken ct) =>
+        tx.ExecuteAsync(
+            """
+            UPDATE alarm_sf_events
+            SET last_attempt_utc = @Now, last_error = @Err, attempt_count = attempt_count + 1
+            WHERE id = @Id
+            """,
+            new { Now = _clock().ToString("O"), Err = reason, Id = id },
+            ct);
 
-    // Async variant used by EnqueueAsync.
-    private async Task EnforceCapacityAsync(SqliteConnection conn, CancellationToken ct)
+    private async Task EnforceCapacityAsync(CancellationToken ct)
     {
         Interlocked.Increment(ref _capacityProbeCount);
-        long count;
-        using (var cmd = conn.CreateCommand())
-        {
-            cmd.CommandText = "SELECT COUNT(*) FROM Queue WHERE DeadLettered = 0";
-            count = (long)(await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false) ?? 0L);
-        }
+        var count = await CountAsync("dead_lettered = 0", ct).ConfigureAwait(false);
+
         // Resync the in-memory counter while we have a fresh number.
         Interlocked.Exchange(ref _queuedRowCount, count);
         if (count < _capacity) return;
 
         var toEvict = count - _capacity + 1;
-        using (var cmd = conn.CreateCommand())
-        {
-            cmd.CommandText = """
-                DELETE FROM Queue
-                WHERE RowId IN (
-                    SELECT RowId FROM Queue
-                    WHERE DeadLettered = 0
-                    ORDER BY RowId ASC
-                    LIMIT $n
-                )
-                """;
-            cmd.Parameters.AddWithValue("$n", toEvict);
-            await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
-        }
+        await _db.ExecuteAsync(
+            """
+            DELETE FROM alarm_sf_events
+            WHERE id IN (
+                SELECT id FROM alarm_sf_events
+                WHERE dead_lettered = 0
+                ORDER BY enqueued_at_utc ASC, id ASC
+                LIMIT @N
+            )
+            """,
+            new { N = toEvict },
+            ct).ConfigureAwait(false);
+
         Interlocked.Add(ref _queuedRowCount, -toEvict);
         long lifetimeEvicted;
         lock (_statusLock) { _evictedCount += toEvict; lifetimeEvicted = _evictedCount; }
@@ -635,40 +669,21 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
             _capacity, toEvict, lifetimeEvicted);
     }
 
-    private void PurgeAgedDeadLetters(SqliteConnection conn)
+    private async Task PurgeAgedDeadLettersAsync(CancellationToken ct)
     {
         var cutoff = (_clock() - _deadLetterRetention).ToString("O");
-        using var cmd = conn.CreateCommand();
-        cmd.CommandText = """
-            DELETE FROM Queue
-            WHERE DeadLettered = 1 AND LastAttemptUtc IS NOT NULL AND LastAttemptUtc < $cutoff
-            """;
-        cmd.Parameters.AddWithValue("$cutoff", cutoff);
-        var purged = cmd.ExecuteNonQuery();
+        var purged = await _db.ExecuteAsync(
+            """
+            DELETE FROM alarm_sf_events
+            WHERE dead_lettered = 1 AND last_attempt_utc IS NOT NULL AND last_attempt_utc < @Cutoff
+            """,
+            new { Cutoff = cutoff },
+            ct).ConfigureAwait(false);
+
         if (purged > 0)
             _logger.Information("Purged {Count} dead-lettered row(s) past retention window", purged);
     }
 
-    private void InitializeSchema()
-    {
-        using var conn = OpenConnection();
-        using var cmd = conn.CreateCommand();
-        cmd.CommandText = """
-            CREATE TABLE IF NOT EXISTS Queue (
-                RowId INTEGER PRIMARY KEY AUTOINCREMENT,
-                AlarmId TEXT NOT NULL,
-                EnqueuedUtc TEXT NOT NULL,
-                PayloadJson TEXT NOT NULL,
-                AttemptCount INTEGER NOT NULL DEFAULT 0,
-                LastAttemptUtc TEXT NULL,
-                LastError TEXT NULL,
-                DeadLettered INTEGER NOT NULL DEFAULT 0
-            );
-            CREATE INDEX IF NOT EXISTS IX_Queue_Drain ON Queue (DeadLettered, RowId);
-            """;
-        cmd.ExecuteNonQuery();
-    }
-
     private void BumpBackoff() => _backoffIndex = Math.Min(_backoffIndex + 1, BackoffLadder.Length - 1);
     private void ResetBackoff() => _backoffIndex = 0;
 
@@ -676,12 +691,16 @@ public sealed class SqliteStoreAndForwardSink : IAlarmHistorianSink, IDisposable
     public TimeSpan CurrentBackoff => BackoffLadder[_backoffIndex];
 
     /// <summary>Disposes the sink and releases all held resources including the drain timer and the writer.</summary>
+    /// <remarks>
+    ///     The <see cref="ILocalDb"/> is NOT disposed here — it is a shared singleton owned by the
+    ///     host and used by the deployment cache too.
+    /// </remarks>
     public void Dispose()
     {
         if (_disposed) return;
         _disposed = true;
         _drainTimer?.Dispose();
-        _drainGate.Dispose();
+        _drainGateLock.Dispose();
         if (_writer is IDisposable writerDisposable) writerDisposable.Dispose();
     }
 }

src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.csproj

@@ -20,6 +20,12 @@
         -->
         <PackageReference Include="SQLitePCLRaw.bundle_e_sqlite3"/>
         <PackageReference Include="Serilog"/>
+        <!--
+          The store-and-forward buffer lives in the node's consolidated LocalDb file so it
+          replicates to the redundant pair peer. Only the ILocalDb SQL surface is used here; the
+          Host owns the DI registration and the replication package.
+        -->
+        <PackageReference Include="ZB.MOM.WW.LocalDb"/>
     </ItemGroup>
 
     <ItemGroup>

src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/Part9StateMachine.cs

@@ -377,4 +377,9 @@ public enum EmissionKind
     Enabled,
     Disabled,
     CommentAdded,
+    /// <summary>#478 — the worst-of-input source quality changed bucket (Good/Uncertain/Bad) with no
+    /// accompanying Part 9 state transition. Delivered out of band via the dedicated
+    /// <c>WriteAlarmQuality</c> node path, never through the <c>IAlarmSource</c> fan-out (it is not a
+    /// state change and must not materialize or historize a condition).</summary>
+    QualityChanged,
 }

src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmEngine.cs

@@ -62,6 +62,17 @@ public sealed class ScriptedAlarmEngine : IDisposable
     private readonly ConcurrentDictionary<string, AlarmScratch> _scratchByAlarmId =
         new(StringComparer.Ordinal);
 
+    /// <summary>
+    ///     #478 — last emitted worst-of-input quality bucket per alarm (0 = Good, 1 = Uncertain,
+    ///     2 = Bad), computed each evaluation over the refilled read cache. A change in this bucket
+    ///     with no accompanying Part 9 state transition drives a standalone
+    ///     <see cref="EmissionKind.QualityChanged"/> emission (a Bad input freezes the condition — no
+    ///     transition — so quality can't ride one, exactly like a comms-lost native driver). Only ever
+    ///     mutated under <c>_evalGate</c>; cleared alongside <see cref="_alarms"/> on load/dispose.
+    /// </summary>
+    private readonly ConcurrentDictionary<string, int> _lastQualityBucketByAlarmId =
+        new(StringComparer.Ordinal);
+
     /// <summary>
     ///     Compile cache for every alarm predicate. Routes <see cref="LoadAsync"/>'s
     ///     <see cref="ScriptEvaluator{TContext, TResult}.Compile"/> calls through the
@@ -203,6 +214,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
             // have changed (different Inputs, different Logger), so any reuse would be
             // unsafe.
             _scratchByAlarmId.Clear();
+            _lastQualityBucketByAlarmId.Clear();
             // Dispose every compiled-predicate ALC from the prior generation BEFORE we
             // recompile this one. Skipping this is what made the earlier fix a
             // no-op in production.
@@ -412,7 +424,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
             // OnEvent dispatch until after Release() so a slow subscriber or a
             // subscriber that re-enters the engine doesn't block / deadlock.
             if (result.Emission != EmissionKind.None)
-                pending = BuildEmission(state, result.State, result.Emission);
+                pending = BuildEmission(state, result.State, result.Emission, LastWorstStatus(alarmId));
             else if (result.NoOpReason is { } reason)
             {
                 // The Part9StateMachine remarks promise a diagnostic log line for
@@ -513,13 +525,27 @@ public sealed class ScriptedAlarmEngine : IDisposable
             _ => new AlarmScratch(state.Inputs, state.Logger, _clock));
         RefillReadCache(scratch.ReadCache, state.Inputs);
 
+        // #478 — worst OPC UA quality across the alarm's inputs, computed BEFORE the readiness
+        // short-circuit so an outright-Bad input is still observed. A bucket change with no state
+        // transition is delivered out of band as a QualityChanged emission (see below).
+        var worstStatus = WorstInputStatus(scratch.ReadCache);
+        var qualityBucketChanged = TrackQualityBucket(state.Definition.AlarmId, worstStatus);
+
         // Cold-start guard — skip the predicate when any referenced upstream tag has no
         // cached value yet (the upstream subscription hasn't delivered its first push).
         // Without this, predicates that cast `(double)ctx.GetTag(path).Value` throw NRE on
         // every tick until the cache fills, spamming the log with identical stack traces.
         // Bad quality is treated the same: the input isn't available at the predicate's
         // expected type, so the only defensible move is to hold the prior condition state.
-        if (!AreInputsReady(scratch.ReadCache)) return seed;
+        if (!AreInputsReady(scratch.ReadCache))
+        {
+            // The condition is frozen (can't trust its state), but its source quality just changed
+            // bucket — annotate it out of band so a comms-lost / Bad-input scripted condition reports
+            // Bad, mirroring the native OT path.
+            if (qualityBucketChanged)
+                pendingEmissions.Add(BuildQualityEmission(state, seed, worstStatus));
+            return seed;
+        }
 
         var context = scratch.Context;
 
@@ -544,10 +570,19 @@ public sealed class ScriptedAlarmEngine : IDisposable
         }
 
         var result = Part9StateMachine.ApplyPredicate(seed, predicateTrue, nowUtc);
-        if (result.Emission != EmissionKind.None)
+        var transition = result.Emission != EmissionKind.None
+            ? BuildEmission(state, result.State, result.Emission, worstStatus)
+            : null;
+        if (transition is not null)
         {
-            var evt = BuildEmission(state, result.State, result.Emission);
-            if (evt is not null) pendingEmissions.Add(evt);
+            // A real transition carries the current worst quality so the projected full-snapshot
+            // write doesn't clobber quality back to Good (e.g. a transition while an input is Uncertain).
+            pendingEmissions.Add(transition);
+        }
+        else if (qualityBucketChanged)
+        {
+            // No transition (or a Suppressed one) but the quality bucket moved — annotate out of band.
+            pendingEmissions.Add(BuildQualityEmission(state, result.State, worstStatus));
         }
         return result.State;
     }
@@ -599,7 +634,8 @@ public sealed class ScriptedAlarmEngine : IDisposable
     ///     done by <see cref="FireEvent(ScriptedAlarmEvent)"/> AFTER the gate is
     ///     released.
     /// </summary>
-    private ScriptedAlarmEvent? BuildEmission(AlarmState state, AlarmConditionState condition, EmissionKind kind)
+    private ScriptedAlarmEvent? BuildEmission(
+        AlarmState state, AlarmConditionState condition, EmissionKind kind, uint worstInputStatus)
     {
         // Suppressed kind means shelving ate the emission — we don't fire for subscribers
         // but the state record still advanced so startup recovery reflects reality.
@@ -629,9 +665,89 @@ public sealed class ScriptedAlarmEngine : IDisposable
             // Carry the per-alarm durable-historization opt-out through to subscribers. The historian
             // adapter honors it to suppress ONLY the durable sink write; the live alerts fan-out is
             // unaffected (it is not gated on this flag).
-            HistorizeToAveva: state.Definition.HistorizeToAveva);
+            HistorizeToAveva: state.Definition.HistorizeToAveva,
+            // #478 — the worst input quality at evaluation time rides the transition so the projected
+            // full snapshot keeps quality consistent (no clobber-to-Good).
+            WorstInputStatusCode: worstInputStatus);
     }
 
+    /// <summary>
+    ///     #478 — build a standalone <see cref="EmissionKind.QualityChanged"/> event carrying the new
+    ///     worst-of-input quality. Emitted when the quality bucket moved but no Part 9 transition fired
+    ///     (a Bad input freezes the condition; a Suppressed/None transition also leaves state unchanged).
+    ///     The host routes it to the dedicated <c>WriteAlarmQuality</c> node path (annotate quality only,
+    ///     no <c>/alerts</c> row, no historian write); the <see cref="IAlarmSource"/> fan-out skips it.
+    /// </summary>
+    private ScriptedAlarmEvent BuildQualityEmission(
+        AlarmState state, AlarmConditionState condition, uint worstInputStatus)
+        => new(
+            AlarmId: state.Definition.AlarmId,
+            EquipmentPath: state.Definition.EquipmentPath,
+            AlarmName: state.Definition.AlarmName,
+            Kind: state.Definition.Kind,
+            Severity: state.Definition.Severity,
+            Message: MessageTemplate.Resolve(state.Definition.MessageTemplate, TryLookup),
+            Condition: condition,
+            Emission: EmissionKind.QualityChanged,
+            TimestampUtc: _clock(),
+            HistorizeToAveva: state.Definition.HistorizeToAveva,
+            WorstInputStatusCode: worstInputStatus);
+
+    /// <summary>Worst OPC UA StatusCode across a refilled read cache — the entry with the highest severity
+    /// bits (top 2). An input with no value yet (null snapshot/value — the cold-start placeholder, or a
+    /// not-yet-published upstream) is NOT a quality signal: it means "no data", which the
+    /// <see cref="AreInputsReady"/> guard already handles by holding the condition. Counting it as Bad here
+    /// would flash every scripted condition Bad at deploy until the first push and would flood the quality
+    /// path with load-time annotations, so unread inputs are skipped (contribute Good). Empty / all-unread
+    /// cache ⇒ Good (0).</summary>
+    private static uint WorstInputStatus(IReadOnlyDictionary<string, DataValueSnapshot> cache)
+    {
+        uint worst = 0u;
+        var worstSeverity = 0u;
+        foreach (var kv in cache)
+        {
+            if (kv.Value is null || kv.Value.Value is null) continue; // no data yet — not a quality signal
+            var status = kv.Value.StatusCode;
+            var severity = status >> 30;
+            if (severity > worstSeverity)
+            {
+                worstSeverity = severity;
+                worst = status;
+            }
+        }
+        return worst;
+    }
+
+    /// <summary>Update the tracked worst-quality bucket for an alarm; return true iff the 3-state bucket
+    /// (0 = Good, 1 = Uncertain, 2 = Bad) changed from the last observed value. Only called under
+    /// <c>_evalGate</c>.</summary>
+    private bool TrackQualityBucket(string alarmId, uint worstStatus)
+    {
+        var bucket = QualityBucket(worstStatus);
+        var prior = _lastQualityBucketByAlarmId.TryGetValue(alarmId, out var b) ? b : 0; // default Good
+        _lastQualityBucketByAlarmId[alarmId] = bucket;
+        return bucket != prior;
+    }
+
+    /// <summary>Collapse an OPC UA StatusCode's 2 severity bits (00/01/10/11) to a 3-state quality bucket
+    /// (0 = Good, 1 = Uncertain, 2 = Bad).</summary>
+    private static int QualityBucket(uint statusCode)
+    {
+        var severity = statusCode >> 30;
+        return severity >= 2 ? 2 : (int)severity;
+    }
+
+    /// <summary>#478 — a canonical worst StatusCode for an alarm's last-observed quality bucket, used by
+    /// the operator-command + shelving-timer emission paths (which don't re-read inputs) so an ack /
+    /// shelve while an input is Bad still carries Bad rather than resetting the condition to Good.</summary>
+    private uint LastWorstStatus(string alarmId)
+        => (_lastQualityBucketByAlarmId.TryGetValue(alarmId, out var bucket) ? bucket : 0) switch
+        {
+            2 => 0x80000000u, // Bad
+            1 => 0x40000000u, // Uncertain
+            _ => 0u,          // Good
+        };
+
     /// <summary>
     ///     Invoke the <see cref="OnEvent"/> handler for a built emission. Must be
     ///     called OUTSIDE <c>_evalGate</c>: a slow subscriber would otherwise
@@ -708,7 +824,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
                         _alarms[id] = state with { Condition = result.State };
                         if (result.Emission != EmissionKind.None)
                         {
-                            var evt = BuildEmission(state, result.State, result.Emission);
+                            var evt = BuildEmission(state, result.State, result.Emission, LastWorstStatus(id));
                             if (evt is not null) pending.Add(evt);
                         }
                     }
@@ -780,6 +896,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
         _alarms.Clear();
         _alarmsReferencing.Clear();
         _scratchByAlarmId.Clear();
+        _lastQualityBucketByAlarmId.Clear();
         // Dispose every compiled-predicate ALC so the engine's shutdown actually
         // releases the emitted assemblies. The drain above ensures no evaluator is
         // mid-call; CompiledScriptCache.Dispose internally guards against use-after-
@@ -851,7 +968,11 @@ public sealed record ScriptedAlarmEvent(
     EmissionKind Emission,
     DateTime TimestampUtc,
     string? Comment = null,
-    bool HistorizeToAveva = true);
+    bool HistorizeToAveva = true,
+    // #478 — the worst OPC UA StatusCode across the alarm's input tags at evaluation time. A raw uint
+    // (Core.ScriptedAlarms does not reference Commons/OpcUaQuality); the host maps it to OpcUaQuality by
+    // the top-2 severity bits. Default 0u == Good keeps every existing constructor call unchanged.
+    uint WorstInputStatusCode = 0u);
 
 /// <summary>
 ///     Upstream source abstraction — intentionally identical shape to the virtual-tag

src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ScriptedAlarmSource.cs

@@ -81,6 +81,11 @@ public sealed class ScriptedAlarmSource : IAlarmSource, IDisposable
     {
         if (_disposed) return;
 
+        // #478 — QualityChanged is a source-quality annotation, not a Part 9 state change. It is delivered
+        // out of band via the dedicated WriteAlarmQuality node path; surfacing it here would fabricate a
+        // phantom AlarmEventArgs that materializes / historizes a condition. Swallow it.
+        if (evt.Emission == EmissionKind.QualityChanged) return;
+
         foreach (var sub in _subscriptions.Values)
         {
             if (!Matches(sub, evt)) continue;

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser/GalaxyDriverBrowser.cs

@@ -5,6 +5,7 @@ using Microsoft.Extensions.Logging.Abstractions;
 using ZB.MOM.WW.MxGateway.Client;
 using ZB.MOM.WW.OtOpcUa.Commons.Browsing;
 using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
+using ZB.MOM.WW.Secrets.Abstractions;
 
 namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browser;
 
@@ -34,11 +35,14 @@ public sealed class GalaxyDriverBrowser : IDriverBrowser
     };
 
     private readonly ILogger<GalaxyDriverBrowser> _logger;
+    private readonly ISecretResolver _secretResolver;
 
     /// <summary>Creates a new browser. Logger defaults to <see cref="NullLogger{T}"/>.</summary>
+    /// <param name="secretResolver">Resolves the <c>secret:</c> arm of <c>Gateway.ApiKeySecretRef</c> (DI-injected).</param>
     /// <param name="logger">Optional logger; null is allowed for unit-test construction.</param>
-    public GalaxyDriverBrowser(ILogger<GalaxyDriverBrowser>? logger = null)
+    public GalaxyDriverBrowser(ISecretResolver secretResolver, ILogger<GalaxyDriverBrowser>? logger = null)
     {
+        _secretResolver = secretResolver ?? throw new ArgumentNullException(nameof(secretResolver));
         _logger = logger ?? NullLogger<GalaxyDriverBrowser>.Instance;
     }
 
@@ -68,7 +72,7 @@ public sealed class GalaxyDriverBrowser : IDriverBrowser
         if (string.IsNullOrWhiteSpace(opts.MxAccess.ClientName))
             throw new InvalidOperationException("Galaxy browser requires MxAccess.ClientName.");
 
-        var clientOpts = BuildClientOptions(opts.Gateway);
+        var clientOpts = await BuildClientOptionsAsync(opts.Gateway, cancellationToken).ConfigureAwait(false);
 
         // 30s wall-clock budget for the connect phase, linked to the caller's token so
         // an AdminUI cancel still wins early.
@@ -116,19 +120,28 @@ public sealed class GalaxyDriverBrowser : IDriverBrowser
     ///     Build the gateway client options from the form's Gateway section. Mirrors the
     ///     runtime driver's <c>GalaxyDriver.BuildClientOptions</c> field-for-field so the
     ///     gateway sees an identical option shape. The API-key reference is resolved via
-    ///     the shared <see cref="GalaxySecretRef.ResolveApiKey"/> in Driver.Galaxy.Contracts
+    ///     the shared <see cref="GalaxySecretRef.ResolveApiKeyAsync"/> in Driver.Galaxy.Contracts
     ///     (the same resolver the runtime driver uses), so browse and runtime stay in lock-step.
+    ///     The <c>secret:</c> arm is async, so the key is resolved into a local before the
+    ///     object initializer (you can't await inside one).
     /// </summary>
-    private MxGatewayClientOptions BuildClientOptions(GalaxyGatewayOptions gw) => new()
+    private async Task<MxGatewayClientOptions> BuildClientOptionsAsync(
+        GalaxyGatewayOptions gw, CancellationToken cancellationToken)
     {
-        Endpoint = new Uri(gw.Endpoint, UriKind.Absolute),
-        ApiKey = GalaxySecretRef.ResolveApiKey(gw.ApiKeySecretRef, _logger),
-        UseTls = gw.UseTls,
-        CaCertificatePath = gw.CaCertificatePath,
-        ConnectTimeout = TimeSpan.FromSeconds(gw.ConnectTimeoutSeconds),
-        DefaultCallTimeout = TimeSpan.FromSeconds(gw.DefaultCallTimeoutSeconds),
-        StreamTimeout = gw.StreamTimeoutSeconds > 0
-            ? TimeSpan.FromSeconds(gw.StreamTimeoutSeconds)
-            : null,
-    };
+        var apiKey = await GalaxySecretRef
+            .ResolveApiKeyAsync(gw.ApiKeySecretRef, _secretResolver, _logger, cancellationToken)
+            .ConfigureAwait(false);
+        return new MxGatewayClientOptions
+        {
+            Endpoint = new Uri(gw.Endpoint, UriKind.Absolute),
+            ApiKey = apiKey,
+            UseTls = gw.UseTls,
+            CaCertificatePath = gw.CaCertificatePath,
+            ConnectTimeout = TimeSpan.FromSeconds(gw.ConnectTimeoutSeconds),
+            DefaultCallTimeout = TimeSpan.FromSeconds(gw.DefaultCallTimeoutSeconds),
+            StreamTimeout = gw.StreamTimeoutSeconds > 0
+                ? TimeSpan.FromSeconds(gw.StreamTimeoutSeconds)
+                : null,
+        };
+    }
 }

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/GalaxyDriverOptions.cs

@@ -43,12 +43,14 @@ public sealed record GalaxyDriverOptions(
 
 /// <summary>
 ///     Connection details for the MxAccess gateway. <see cref="ApiKeySecretRef"/> is
-///     resolved by <see cref="GalaxySecretRef.ResolveApiKey"/> at InitializeAsync time. Four forms
+///     resolved by <see cref="GalaxySecretRef.ResolveApiKeyAsync"/> at InitializeAsync time. Five forms
 ///     supported, in priority order:
 ///     <list type="bullet">
 ///       <item><c>env:NAME</c> — read from an environment variable (recommended for
 ///         production; the central config DB holds only the indirection, not the key).</item>
 ///       <item><c>file:PATH</c> — read from an ACL'd file outside the repo.</item>
+///       <item><c>secret:NAME</c> — resolved through the shared <c>ZB.MOM.WW.Secrets</c>
+///         encrypted store; fail-closed if absent (the production path).</item>
 ///       <item><c>dev:KEY</c> — explicit cleartext opt-in for dev rigs / parity tests;
 ///         no startup warning.</item>
 ///       <item>Anything else — treated as a literal cleartext API key for back-compat.

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/GalaxySecretRef.cs

@@ -1,9 +1,10 @@
 using Microsoft.Extensions.Logging;
+using ZB.MOM.WW.Secrets.Abstractions;
 
 namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
 
 /// <summary>
-///     Resolves <c>Gateway.ApiKeySecretRef</c> to the actual API-key string. Four
+///     Resolves <c>Gateway.ApiKeySecretRef</c> to the actual API-key string. Five
 ///     forms supported, evaluated in order:
 ///     <list type="number">
 ///       <item><c>env:NAME</c> — reads <c>Environment.GetEnvironmentVariable(NAME)</c>.
@@ -15,12 +16,17 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
 ///       <item><c>dev:KEY</c> — explicit cleartext literal. The <c>dev:</c> prefix
 ///         is a deliberate opt-in signal (dev box, parity rig) so the resolver
 ///         doesn't emit a warning; production should never use this arm.</item>
+///       <item><c>secret:NAME</c> — resolves NAME through the shared
+///         <c>ZB.MOM.WW.Secrets</c> <see cref="ISecretResolver"/> (the encrypted-at-rest
+///         store). Fail-closed: a <c>secret:</c> ref whose secret is absent/tombstoned
+///         throws rather than falling through to the literal arm — the production path
+///         that retires the cleartext <c>dev:</c>/literal-in-DB model.</item>
 ///       <item>Anything else — used as the literal API key for back-compat with
 ///         configs that pre-date this resolver. When a logger is supplied the
 ///         resolver emits a startup warning so an operator who accidentally
 ///         committed a cleartext key sees it.</item>
 ///     </list>
-///     A future PR can swap any of these arms for a DPAPI-backed lookup without
+///     A future PR can swap any of these arms for a different backing store without
 ///     changing the call site.
 /// </summary>
 /// <remarks>
@@ -31,18 +37,26 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
 public static class GalaxySecretRef
 {
     /// <summary>
-    ///     Resolves the supplied secret reference. When the ref falls through to the
-    ///     back-compat literal arm (an unprefixed cleartext API key in
-    ///     <c>DriverConfig</c> JSON) and a <paramref name="logger"/> is supplied, emits
-    ///     a <see cref="LogLevel.Warning"/>. The <c>dev:</c> prefix is the explicit
-    ///     opt-in path that doesn't warn.
+    ///     Resolves the supplied secret reference. The <c>secret:NAME</c> arm resolves
+    ///     through <paramref name="resolver"/> and is fail-closed (throws when the secret
+    ///     is absent). When the ref falls through to the back-compat literal arm (an
+    ///     unprefixed cleartext API key in <c>DriverConfig</c> JSON) and a
+    ///     <paramref name="logger"/> is supplied, emits a <see cref="LogLevel.Warning"/>.
+    ///     The <c>dev:</c> prefix is the explicit opt-in path that doesn't warn.
     /// </summary>
     /// <param name="secretRef">The secret reference string to resolve.</param>
+    /// <param name="resolver">The shared secret resolver used by the <c>secret:</c> arm.</param>
     /// <param name="logger">Optional logger for warning on cleartext keys.</param>
+    /// <param name="ct">Cancellation token for the async <c>secret:</c> resolution.</param>
     /// <returns>The resolved API-key string.</returns>
-    public static string ResolveApiKey(string secretRef, ILogger? logger = null)
+    public static async Task<string> ResolveApiKeyAsync(
+        string secretRef,
+        ISecretResolver resolver,
+        ILogger? logger = null,
+        CancellationToken ct = default)
     {
         ArgumentException.ThrowIfNullOrEmpty(secretRef);
+        ArgumentNullException.ThrowIfNull(resolver);
 
         if (secretRef.StartsWith("env:", StringComparison.OrdinalIgnoreCase))
         {
@@ -76,6 +90,19 @@ public static class GalaxySecretRef
             return secretRef[4..];
         }
 
+        if (secretRef.StartsWith("secret:", StringComparison.OrdinalIgnoreCase))
+        {
+            // Production path: resolve the name through the shared encrypted secret store.
+            // Fail-closed — an absent/tombstoned secret throws rather than falling through
+            // to the literal arm (which would silently treat the ref string as the key).
+            var name = secretRef["secret:".Length..];
+            var value = await resolver.GetAsync(new SecretName(name), ct).ConfigureAwait(false);
+            return !string.IsNullOrEmpty(value)
+                ? value
+                : throw new InvalidOperationException(
+                    $"Galaxy.Gateway.ApiKeySecretRef='{secretRef}' resolves secret '{name}', but it is absent from the store (fail-closed).");
+        }
+
         // Back-compat literal arm. An unprefixed string is treated as the literal
         // API key — but emit a warning so an operator who accidentally committed a
         // cleartext key into DriverConfig sees it. Use the dev: prefix to suppress

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts.csproj

@@ -13,5 +13,6 @@
         <ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Abstractions\ZB.MOM.WW.OtOpcUa.Core.Abstractions.csproj" />
         <!-- Logging abstraction needed by GalaxySecretRef.ResolveApiKey's optional warning logger. -->
         <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
+        <PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" />
     </ItemGroup>
 </Project>

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/GalaxyDriver.cs

@@ -8,6 +8,7 @@ using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Browse;
 using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
 using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Health;
 using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Runtime;
+using ZB.MOM.WW.Secrets.Abstractions;
 
 namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy;
 
@@ -43,6 +44,12 @@ public sealed class GalaxyDriver
     private readonly GalaxyDriverOptions _options;
     private readonly ILogger<GalaxyDriver> _logger;
 
+    // Resolves the Gateway.ApiKeySecretRef secret: arm through the shared encrypted store.
+    // Injected via ctor (the production factory pulls it from DI). The internal test ctor
+    // defaults it to a null-object resolver so the 45 seam-injecting test call sites keep
+    // compiling; those tests never use a secret: ref (they inject seams or use env/file/literal).
+    private readonly ISecretResolver _secretResolver;
+
     // PR 4.1 — IGalaxyHierarchySource is the test seam for browse. When null, the driver
     // lazily builds a GatewayGalaxyHierarchySource around a GalaxyRepositoryClient on
     // first DiscoverAsync. Tests inject a fake source via the internal ctor to exercise
@@ -147,14 +154,17 @@ public sealed class GalaxyDriver
     /// <summary>Initializes a new instance of the <see cref="GalaxyDriver"/> class.</summary>
     /// <param name="driverInstanceId">The unique identifier for this driver instance.</param>
     /// <param name="options">The Galaxy driver configuration options.</param>
+    /// <param name="secretResolver">Resolves the <c>secret:</c> arm of <c>Gateway.ApiKeySecretRef</c>.</param>
     /// <param name="logger">Optional logger instance for diagnostics.</param>
     public GalaxyDriver(
         string driverInstanceId,
         GalaxyDriverOptions options,
+        ISecretResolver secretResolver,
         ILogger<GalaxyDriver>? logger = null)
         : this(driverInstanceId, options,
               hierarchySource: null, dataReader: null, dataWriter: null, subscriber: null,
-              alarmAcknowledger: null, alarmFeed: null, logger)
+              alarmAcknowledger: null, alarmFeed: null, logger,
+              secretResolver: secretResolver ?? throw new ArgumentNullException(nameof(secretResolver)))
     {
     }
 
@@ -173,6 +183,11 @@ public sealed class GalaxyDriver
     /// <param name="alarmAcknowledger">Optional custom alarm acknowledger for testing.</param>
     /// <param name="alarmFeed">Optional custom alarm feed for testing.</param>
     /// <param name="logger">Optional logger instance for diagnostics.</param>
+    /// <param name="secretResolver">
+    ///     Optional secret resolver for the <c>secret:</c> API-key arm. Defaults to a
+    ///     null-object resolver (returns null for every name) so seam-injecting tests that
+    ///     don't exercise a <c>secret:</c> ref need not supply one.
+    /// </param>
     internal GalaxyDriver(
         string driverInstanceId,
         GalaxyDriverOptions options,
@@ -182,13 +197,15 @@ public sealed class GalaxyDriver
         IGalaxySubscriber? subscriber = null,
         IGalaxyAlarmAcknowledger? alarmAcknowledger = null,
         IGalaxyAlarmFeed? alarmFeed = null,
-        ILogger<GalaxyDriver>? logger = null)
+        ILogger<GalaxyDriver>? logger = null,
+        ISecretResolver? secretResolver = null)
     {
         _driverInstanceId = !string.IsNullOrWhiteSpace(driverInstanceId)
             ? driverInstanceId
             : throw new ArgumentException("Driver instance id required.", nameof(driverInstanceId));
         _options = options ?? throw new ArgumentNullException(nameof(options));
         _logger = logger ?? NullLogger<GalaxyDriver>.Instance;
+        _secretResolver = secretResolver ?? NullSecretResolver.Instance;
         _hierarchySource = hierarchySource;
         _dataReader = dataReader;
         _dataWriter = dataWriter;
@@ -316,7 +333,7 @@ public sealed class GalaxyDriver
                 _driverInstanceId);
         }
 
-        StartDeployWatcher();
+        await StartDeployWatcherAsync(cancellationToken).ConfigureAwait(false);
         _logger.LogInformation(
             "GalaxyDriver {InstanceId} initialized — endpoint={Endpoint} clientName={ClientName}",
             _driverInstanceId, _options.Gateway.Endpoint, _options.MxAccess.ClientName);
@@ -331,7 +348,7 @@ public sealed class GalaxyDriver
     /// </summary>
     private async Task BuildProductionRuntimeAsync(CancellationToken cancellationToken)
     {
-        var clientOptions = BuildClientOptions(_options.Gateway);
+        var clientOptions = await BuildClientOptionsAsync(_options.Gateway, cancellationToken).ConfigureAwait(false);
         _ownedMxClient = MxGatewayClient.Create(clientOptions);
         _ownedMxSession = new GalaxyMxSession(_options.MxAccess, _logger);
         await _ownedMxSession.ConnectAsync(clientOptions, cancellationToken).ConfigureAwait(false);
@@ -386,7 +403,7 @@ public sealed class GalaxyDriver
     private async Task ReopenAsync(CancellationToken cancellationToken)
     {
         if (_ownedMxSession is null) return;
-        var clientOptions = BuildClientOptions(_options.Gateway);
+        var clientOptions = await BuildClientOptionsAsync(_options.Gateway, cancellationToken).ConfigureAwait(false);
         await _ownedMxSession.RecreateAsync(clientOptions, cancellationToken).ConfigureAwait(false);
         // The recreated session invalidates every prior gw item handle; drop the writer's handle/advise
         // caches so the next write re-AddItems + re-AdviseSupervisory against the fresh session.
@@ -527,32 +544,42 @@ public sealed class GalaxyDriver
         }
     }
 
-    private MxGatewayClientOptions BuildClientOptions(GalaxyGatewayOptions gw) => new()
+    private async Task<MxGatewayClientOptions> BuildClientOptionsAsync(
+        GalaxyGatewayOptions gw, CancellationToken cancellationToken)
     {
-        Endpoint = new Uri(gw.Endpoint, UriKind.Absolute),
-        // Pass the logger so the literal-arm cleartext fallback surfaces a startup
-        // warning rather than silently shipping the key. The resolver lives in
-        // Driver.Galaxy.Contracts (GalaxySecretRef) so the runtime driver and the
-        // AdminUI browser share one implementation.
-        ApiKey = GalaxySecretRef.ResolveApiKey(gw.ApiKeySecretRef, _logger),
-        UseTls = gw.UseTls,
-        CaCertificatePath = gw.CaCertificatePath,
-        ConnectTimeout = TimeSpan.FromSeconds(gw.ConnectTimeoutSeconds),
-        DefaultCallTimeout = TimeSpan.FromSeconds(gw.DefaultCallTimeoutSeconds),
-        StreamTimeout = gw.StreamTimeoutSeconds > 0 ? TimeSpan.FromSeconds(gw.StreamTimeoutSeconds) : null,
-    };
+        // Resolve the API-key ref BEFORE the object initializer — the secret: arm is
+        // async and you can't await inside an initializer. Pass the logger so the
+        // literal-arm cleartext fallback surfaces a startup warning rather than
+        // silently shipping the key. The resolver lives in Driver.Galaxy.Contracts
+        // (GalaxySecretRef) so the runtime driver and the AdminUI browser share one
+        // implementation; the secret: arm resolves through the shared ISecretResolver.
+        var apiKey = await GalaxySecretRef
+            .ResolveApiKeyAsync(gw.ApiKeySecretRef, _secretResolver, _logger, cancellationToken)
+            .ConfigureAwait(false);
+        return new MxGatewayClientOptions
+        {
+            Endpoint = new Uri(gw.Endpoint, UriKind.Absolute),
+            ApiKey = apiKey,
+            UseTls = gw.UseTls,
+            CaCertificatePath = gw.CaCertificatePath,
+            ConnectTimeout = TimeSpan.FromSeconds(gw.ConnectTimeoutSeconds),
+            DefaultCallTimeout = TimeSpan.FromSeconds(gw.DefaultCallTimeoutSeconds),
+            StreamTimeout = gw.StreamTimeoutSeconds > 0 ? TimeSpan.FromSeconds(gw.StreamTimeoutSeconds) : null,
+        };
+    }
 
-    private void StartDeployWatcher()
+    private async Task StartDeployWatcherAsync(CancellationToken cancellationToken)
     {
         if (!_options.Repository.WatchDeployEvents) return;
         if (_ownedRepositoryClient is null && _hierarchySource is null) return;
 
         // Reuse the lazily-built repository client (DiscoverAsync constructs it on demand).
         // If discovery hasn't run yet, build the client here so the watcher has a target.
-        // Guard with ??= so if BuildDefaultHierarchySource later runs it reuses this client
-        // rather than overwriting the field and leaking the first instance.
+        // Build under a null-check (not ??=) so if BuildDefaultHierarchySourceAsync later
+        // runs it reuses this client rather than overwriting the field and leaking the
+        // first instance — the client-options build is now async (secret: arm).
         _ownedRepositoryClient ??= ZB.MOM.WW.MxGateway.Client.GalaxyRepositoryClient.Create(
-            BuildClientOptions(_options.Gateway));
+            await BuildClientOptionsAsync(_options.Gateway, cancellationToken).ConfigureAwait(false));
 
         var source = new GatewayGalaxyDeployWatchSource(_ownedRepositoryClient);
         _deployWatcher = new DeployWatcher(source, _logger);
@@ -691,7 +718,8 @@ public sealed class GalaxyDriver
         // After discovery, SyncPlatformsAsync refreshes the probe watcher's membership so
         // newly added $WinPlatform / $AppEngine objects start advising their ScanState attribute.
         var capturingBuilder = new SecurityCapturingBuilder(builder, _securityByFullRef);
-        var source = _hierarchySource ??= BuildDefaultHierarchySource();
+        var source = _hierarchySource ??=
+            await BuildDefaultHierarchySourceAsync(cancellationToken).ConfigureAwait(false);
         // Thread the v3 reverse resolver so the discoverer emits each authored attribute's RawPath as
         // its node reference (FullName) — the security map SecurityCapturingBuilder captures is then
         // keyed by RawPath, matching what WriteAsync resolves against.
@@ -1292,12 +1320,14 @@ public sealed class GalaxyDriver
     ///     <see cref="Dispose"/>. Tests bypass this by injecting their own source via the
     ///     internal ctor.
     /// </summary>
-    private IGalaxyHierarchySource BuildDefaultHierarchySource()
+    private async Task<IGalaxyHierarchySource> BuildDefaultHierarchySourceAsync(CancellationToken cancellationToken)
     {
-        // Reuse a client that StartDeployWatcher may have already created (??=) rather
+        // Reuse a client that StartDeployWatcherAsync may have already created (??=) rather
         // than always overwriting the field and leaking the first instance. Both paths
-        // produce equivalent clients from the same options.
-        _ownedRepositoryClient ??= GalaxyRepositoryClient.Create(BuildClientOptions(_options.Gateway));
+        // produce equivalent clients from the same options. The client-options build is
+        // async now (secret: arm resolves through the shared ISecretResolver).
+        _ownedRepositoryClient ??= GalaxyRepositoryClient.Create(
+            await BuildClientOptionsAsync(_options.Gateway, cancellationToken).ConfigureAwait(false));
         return new TracedGalaxyHierarchySource(
             new GatewayGalaxyHierarchySource(_ownedRepositoryClient), _options.MxAccess.ClientName);
     }

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/GalaxyDriverFactoryExtensions.cs

@@ -4,6 +4,7 @@ using Microsoft.Extensions.Logging;
 using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
 using ZB.MOM.WW.OtOpcUa.Core.Hosting;
 using ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Config;
+using ZB.MOM.WW.Secrets.Abstractions;
 
 namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy;
 
@@ -23,30 +24,45 @@ public static class GalaxyDriverFactoryExtensions
 {
     public const string DriverTypeName = "GalaxyMxGateway";
 
-    /// <summary>Registers the Galaxy driver factory with the given registry and optional logger factory.</summary>
+    /// <summary>Registers the Galaxy driver factory with the given registry, secret resolver, and optional logger factory.</summary>
     /// <param name="registry">The driver factory registry to register with.</param>
+    /// <param name="secretResolver">
+    ///     The shared secret resolver (from DI) used to resolve the <c>secret:</c> arm of
+    ///     each instance's <c>Gateway.ApiKeySecretRef</c>.
+    /// </param>
     /// <param name="loggerFactory">The optional logger factory for creating drivers.</param>
-    public static void Register(DriverFactoryRegistry registry, ILoggerFactory? loggerFactory = null)
+    public static void Register(
+        DriverFactoryRegistry registry,
+        ISecretResolver secretResolver,
+        ILoggerFactory? loggerFactory = null)
     {
         ArgumentNullException.ThrowIfNull(registry);
-        registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory));
+        ArgumentNullException.ThrowIfNull(secretResolver);
+        registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory, secretResolver));
     }
 
-    /// <summary>Convenience for tests + standalone callers.</summary>
+    /// <summary>
+    ///     Convenience for tests + standalone callers. Uses the <see cref="NullSecretResolver"/>
+    ///     null-object, so the <c>secret:</c> API-key arm resolves fail-closed (absent) — callers
+    ///     that need a working <c>secret:</c> ref must use the <see cref="Register"/> path that
+    ///     threads the DI resolver.
+    /// </summary>
     /// <param name="driverInstanceId">The unique identifier for the driver instance.</param>
     /// <param name="driverConfigJson">The driver configuration in JSON format.</param>
     /// <returns>The constructed Galaxy driver instance.</returns>
     public static GalaxyDriver CreateInstance(string driverInstanceId, string driverConfigJson)
-        => CreateInstance(driverInstanceId, driverConfigJson, loggerFactory: null);
+        => CreateInstance(driverInstanceId, driverConfigJson, loggerFactory: null, NullSecretResolver.Instance);
 
-    /// <summary>Creates a Galaxy driver instance from configuration JSON with optional logger factory.</summary>
+    /// <summary>Creates a Galaxy driver instance from configuration JSON with optional logger factory and a secret resolver.</summary>
     /// <param name="driverInstanceId">The unique identifier for the driver instance.</param>
     /// <param name="driverConfigJson">The driver configuration in JSON format.</param>
     /// <param name="loggerFactory">The optional logger factory for creating drivers.</param>
+    /// <param name="secretResolver">The secret resolver for the driver's <c>secret:</c> API-key arm.</param>
     /// <returns>The constructed Galaxy driver instance.</returns>
     public static GalaxyDriver CreateInstance(
-        string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory)
+        string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory, ISecretResolver secretResolver)
     {
+        ArgumentNullException.ThrowIfNull(secretResolver);
         ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
         ArgumentException.ThrowIfNullOrWhiteSpace(driverConfigJson);
 
@@ -88,7 +104,7 @@ public static class GalaxyDriverFactoryExtensions
             RawTags = dto.RawTags ?? [],
         };
 
-        return new GalaxyDriver(driverInstanceId, options, loggerFactory?.CreateLogger<GalaxyDriver>());
+        return new GalaxyDriver(driverInstanceId, options, secretResolver, loggerFactory?.CreateLogger<GalaxyDriver>());
     }
 
     private static readonly JsonSerializerOptions JsonOptions = new()

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy/NullSecretResolver.cs

@@ -0,0 +1,26 @@
+using ZB.MOM.WW.Secrets.Abstractions;
+
+namespace ZB.MOM.WW.OtOpcUa.Driver.Galaxy;
+
+/// <summary>
+///     A null-object <see cref="ISecretResolver"/> that resolves every name to <c>null</c>
+///     (absent). Used as the default for the internal test ctor and the parse-only
+///     <c>CreateInstance</c> path so callers that never exercise a <c>secret:</c> API-key
+///     ref need not thread a real resolver. A production <c>GalaxyDriver</c> always receives
+///     the DI-registered resolver via the factory; a <c>secret:</c> ref resolved against this
+///     null object throws fail-closed (the secret is reported absent), which is the correct
+///     behaviour for a mis-wired deployment.
+/// </summary>
+internal sealed class NullSecretResolver : ISecretResolver
+{
+    /// <summary>The shared singleton instance.</summary>
+    public static readonly NullSecretResolver Instance = new();
+
+    private NullSecretResolver()
+    {
+    }
+
+    /// <inheritdoc />
+    public Task<string?> GetAsync(SecretName name, CancellationToken ct) =>
+        Task.FromResult<string?>(null);
+}

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayAlarmHistorianWriter.cs

@@ -9,7 +9,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
 
 /// <summary>
 ///     <see cref="IAlarmHistorianWriter"/> backed by the HistorianGateway <c>SendEvent</c> path. The
-///     drain worker behind <c>SqliteStoreAndForwardSink</c> calls
+///     drain worker behind <c>LocalDbStoreAndForwardSink</c> calls
 ///     <see cref="WriteBatchAsync"/> and uses the returned per-event
 ///     <see cref="HistorianWriteOutcome"/> to decide retry vs. dead-letter, so this writer maps every
 ///     gateway result — success ack, the published client's typed exception hierarchy, raw

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianServiceCollectionExtensions.cs

@@ -47,7 +47,7 @@ public static class GatewayHistorian
     ///     <see cref="ServerHistorianOptions"/> — the <b>same single gateway</b> the read path
     ///     (<see cref="CreateDataSource"/>) targets. The Host's <c>AddAlarmHistorian</c> wiring supplies
     ///     this as the concrete <see cref="IAlarmHistorianWriter"/> the durable
-    ///     <c>SqliteStoreAndForwardSink</c> drain worker delegates to, sourcing the connection from the
+    ///     <c>LocalDbStoreAndForwardSink</c> drain worker delegates to, sourcing the connection from the
     ///     <c>ServerHistorian</c> section (endpoint/key/TLS) rather than the legacy Wonderware-shaped
     ///     <c>AlarmHistorian</c> host/port. Resolves an <see cref="ILoggerFactory"/> and the writer's
     ///     <see cref="ILogger{TCategoryName}"/> from <paramref name="services"/>, falling back to the null

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/HistorianGatewayClientAdapter.cs

@@ -57,16 +57,23 @@ public sealed class HistorianGatewayClientAdapter : IHistorianGatewayClient, IDi
                 $"ServerHistorian:Endpoint must be an absolute http(s) URI (e.g. https://host:5222); got '{options.Endpoint}'.");
         }
 
+        // TLS-only options must stay at their defaults on a plaintext h2c connection: the client rejects
+        // each of them outright when UseTls=false ("<name> is a TLS-only option and requires
+        // UseTls=true"). Forwarding them unconditionally made h2c UNREACHABLE — AllowUntrustedServer-
+        // Certificate defaults to false, so RequireCertificateValidation was always sent as true and
+        // every http:// deployment crashed at startup, even though the scheme is documented as the
+        // supported way to select h2c. There is no certificate to have a posture about here.
         var clientOptions = new HistorianGatewayClientOptions
         {
             Endpoint = endpointUri,
             ApiKey = options.ApiKey,
             UseTls = options.UseTls,
-            CaCertificatePath = options.CaCertificatePath,
-            // INVERTED mapping: ServerHistorianOptions.AllowUntrustedServerCertificate (opt-in to accept
-            // a self-signed cert) is the negation of the client's RequireCertificateValidation. Allowing
-            // an untrusted cert == not requiring validation; a pinned CaCertificatePath always verifies.
-            RequireCertificateValidation = !options.AllowUntrustedServerCertificate,
+            CaCertificatePath = options.UseTls ? options.CaCertificatePath : null,
+            // INVERTED mapping (TLS only): ServerHistorianOptions.AllowUntrustedServerCertificate
+            // (opt-in to accept a self-signed cert) is the negation of the client's
+            // RequireCertificateValidation. Allowing an untrusted cert == not requiring validation; a
+            // pinned CaCertificatePath always verifies.
+            RequireCertificateValidation = options.UseTls && !options.AllowUntrustedServerCertificate,
             DefaultCallTimeout = options.CallTimeout,
             LoggerFactory = loggerFactory,
         };

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Contracts/OpcUaClientDriverOptions.cs

@@ -14,7 +14,9 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
 ///     exposes PLC data" flow. Tier A (pure managed, OPC Foundation reference SDK); universal
 ///     protections cover it.
 /// </remarks>
-public sealed class OpcUaClientDriverOptions
+// A record (not a plain class) so G-2b's OpcUaClientSecretResolution can produce a
+// credential-resolved copy with a `with` expression — every property keeps its init setter.
+public sealed record OpcUaClientDriverOptions
 {
     /// <summary>
     ///     Remote OPC UA endpoint URL, e.g. <c>opc.tcp://plc.internal:4840</c>. Convenience

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/NullSecretResolver.cs

@@ -0,0 +1,26 @@
+using ZB.MOM.WW.Secrets.Abstractions;
+
+namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
+
+/// <summary>
+///     A null-object <see cref="ISecretResolver"/> that resolves every name to <c>null</c>
+///     (absent). Backs the driver's default (test/parse-only) construction path so callers that
+///     never author a <c>secret:</c> credential ref need not thread a real resolver. A production
+///     <c>OpcUaClientDriver</c> always receives the DI-registered resolver via the factory; a
+///     <c>secret:</c> ref resolved against this null object throws fail-closed (the secret is
+///     reported absent), which is the correct behaviour for a mis-wired deployment — the
+///     <c>secret:</c> literal is never sent verbatim as a password.
+/// </summary>
+internal sealed class NullSecretResolver : ISecretResolver
+{
+    /// <summary>The shared singleton instance.</summary>
+    public static readonly NullSecretResolver Instance = new();
+
+    private NullSecretResolver()
+    {
+    }
+
+    /// <inheritdoc />
+    public Task<string?> GetAsync(SecretName name, CancellationToken ct) =>
+        Task.FromResult<string?>(null);
+}

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriver.cs

@@ -4,6 +4,7 @@ using Opc.Ua;
 using Opc.Ua.Client;
 using Opc.Ua.Configuration;
 using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
+using ZB.MOM.WW.Secrets.Abstractions;
 
 namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
 
@@ -36,15 +37,25 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
     /// <param name="options">Driver configuration.</param>
     /// <param name="driverInstanceId">Stable logical ID from the config DB.</param>
     /// <param name="logger">Optional logger; defaults to NullLogger when not supplied.</param>
+    /// <param name="secretResolver">
+    ///     Optional shared secret resolver used to resolve <c>secret:</c>-prefixed
+    ///     <see cref="OpcUaClientDriverOptions.Password"/> /
+    ///     <see cref="OpcUaClientDriverOptions.UserCertificatePassword"/> at session-open.
+    ///     Defaults to the <see cref="NullSecretResolver"/> null-object (which reports every
+    ///     secret absent) for the test/parse-only path; the production factory threads the
+    ///     DI-registered resolver. A <c>secret:</c> ref against the null-object fails closed.
+    /// </param>
     public OpcUaClientDriver(OpcUaClientDriverOptions options, string driverInstanceId,
-        ILogger<OpcUaClientDriver>? logger = null)
+        ILogger<OpcUaClientDriver>? logger = null, ISecretResolver? secretResolver = null)
     {
         _options = options;
         _driverInstanceId = driverInstanceId;
         _logger = logger ?? NullLogger<OpcUaClientDriver>.Instance;
+        _secretResolver = secretResolver ?? NullSecretResolver.Instance;
     }
 
     private readonly OpcUaClientDriverOptions _options;
+    private readonly ISecretResolver _secretResolver;
     private readonly string _driverInstanceId;
     // ---- IAlarmSource state ----
 
@@ -159,7 +170,18 @@ public sealed class OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWrit
             var appConfig = await BuildApplicationConfigurationAsync(cancellationToken).ConfigureAwait(false);
             var candidates = ResolveEndpointCandidates(_options);
 
-            var identity = BuildUserIdentity(_options);
+            // G-2b: resolve any secret:-prefixed Password / UserCertificatePassword through the
+            // shared secret store ONCE, here in the async connect flow, just before the credentials
+            // are consumed. _options stays the raw config (with secret: refs) — only this
+            // connect-scoped local carries plaintext, and only for the duration of the connect.
+            // Resolving on every InitializeAsync (a full reconnect calls it) picks up rotations.
+            // Both credential consumers — the Username password below and the Certificate password
+            // in BuildCertificateIdentity — flow through BuildUserIdentity(resolvedOptions), so a
+            // single resolve covers both paths.
+            var resolvedOptions = await OpcUaClientSecretResolution
+                .ResolveSecretRefsAsync(_options, _secretResolver, cancellationToken).ConfigureAwait(false);
+
+            var identity = BuildUserIdentity(resolvedOptions);
 
             // Failover sweep: try each endpoint in order, return the session from the first
             // one that successfully connects. Per-endpoint failures are captured so the final

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriverFactoryExtensions.cs

@@ -2,6 +2,7 @@ using System.Text.Json;
 using System.Text.Json.Serialization;
 using Microsoft.Extensions.Logging;
 using ZB.MOM.WW.OtOpcUa.Core.Hosting;
+using ZB.MOM.WW.Secrets.Abstractions;
 
 namespace ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient;
 
@@ -31,22 +32,37 @@ public static class OpcUaClientDriverFactoryExtensions
         Converters = { new JsonStringEnumConverter() },
     };
 
-    /// <summary>Register the OpcUaClient factory with the driver registry.</summary>
+    /// <summary>Register the OpcUaClient factory with the driver registry, threading the shared secret resolver.</summary>
     /// <param name="registry">The driver factory registry to register with.</param>
     /// <param name="loggerFactory">Optional logger factory used to create per-instance loggers.</param>
-    public static void Register(DriverFactoryRegistry registry, ILoggerFactory? loggerFactory = null)
+    /// <param name="secretResolver">
+    ///     The shared secret resolver (from DI) used to resolve the <c>secret:</c> arm of each
+    ///     instance's <c>Password</c> / <c>UserCertificatePassword</c> at session-open. Defaults to
+    ///     the <see cref="NullSecretResolver"/> null-object for the test/standalone path — callers
+    ///     that need a working <c>secret:</c> credential ref must thread the DI resolver.
+    /// </param>
+    public static void Register(
+        DriverFactoryRegistry registry,
+        ILoggerFactory? loggerFactory = null,
+        ISecretResolver? secretResolver = null)
     {
         ArgumentNullException.ThrowIfNull(registry);
-        registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory));
+        var resolver = secretResolver ?? NullSecretResolver.Instance;
+        registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory, resolver));
     }
 
     /// <summary>Public for the Server-side bootstrapper + test consumers.</summary>
     /// <param name="driverInstanceId">The unique identifier for the driver instance.</param>
     /// <param name="driverConfigJson">The JSON configuration string for the driver.</param>
     /// <param name="loggerFactory">Optional logger factory for the per-instance logger.</param>
+    /// <param name="secretResolver">
+    ///     Optional shared secret resolver for the driver's <c>secret:</c> credential arm; defaults
+    ///     to the <see cref="NullSecretResolver"/> null-object (fail-closed on a <c>secret:</c> ref).
+    /// </param>
     /// <returns>A configured <see cref="OpcUaClientDriver"/>.</returns>
     public static OpcUaClientDriver CreateInstance(
-        string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory = null)
+        string driverInstanceId, string driverConfigJson, ILoggerFactory? loggerFactory = null,
+        ISecretResolver? secretResolver = null)
     {
         ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
         ArgumentException.ThrowIfNullOrWhiteSpace(driverConfigJson);
@@ -55,6 +71,8 @@ public static class OpcUaClientDriverFactoryExtensions
             ?? throw new InvalidOperationException(
                 $"OpcUaClient driver config for '{driverInstanceId}' deserialised to null");
 
-        return new OpcUaClientDriver(options, driverInstanceId, loggerFactory?.CreateLogger<OpcUaClientDriver>());
+        return new OpcUaClientDriver(
+            options, driverInstanceId, loggerFactory?.CreateLogger<OpcUaClientDriver>(),
+            secretResolver ?? NullSecretResolver.Instance);
     }
 }