docs(design): settle the three open questions on the per-cluster mesh

(1) One central DB, driver nodes disconnected from it. Verified in ScadaBridge
before adopting: AddConfigurationDatabase is called at Program.cs:264, inside the
Central branch (89-478); the Site branch starts at 479, so site nodes never
register the central database at all. There is still exactly one authoritative
configuration database — it is central-only, and sites fetch from central and
cache locally.

The consequence worth naming: this promotes LocalDb from an outage cache to the
driver node's steady-state configuration store. Boot-from-cache stops being the
exceptional path and becomes the normal one. Phase 1's chunking, SHA-256
verification, retention and pair replication all carry over — but a cache defect
that used to be a degraded-mode bug becomes a total-availability one, so the #485
unreadable-artifact class now has a larger blast radius.

Audited the five driver-side ConfigDb consumers that must be re-homed. Two are
more than mechanical: EfAlarmConditionStateStore holds pair-local Part 9 condition
state and should follow the alarm S&F buffer into LocalDb, and DbHealthProbeActor
feeds ServiceLevel — a driver node with no DB to probe changes what a
client-visible value means.

(2) The keep-oldest gap: corrected the status rather than the conclusion. What was
resolved is the documentation and the drill, not the hole. Gitea PR #12 (closed)
carried T1 badc97af, which made the drill "measure the registered keep-oldest
outage instead of pretending recovery", and T2 d5364506, which dropped "the
undeliverable ~25s active-crash promise". The gap itself is still the registered
deferred keep-oldest topology decision on the 2026-07-08 master tracker, with no
open issue. Accepted here with the same posture, recorded as a known risk.

(3) Auth matches ScadaBridge for now — unauthenticated inter-cluster transports on
a trusted network, recorded as an accepted risk with the fail-closed interceptor
already in our tree noted as the template for whenever it is revisited.

Sequencing grows from six phases to eight; the two that change a running system's
data path rather than its wiring each get their own live gate.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

docs/plans/2026-07-21-per-cluster-mesh-design.md

@@ -1,7 +1,9 @@
 # Per-Cluster Akka Mesh — design
 
 > **Status:** design for review. No implementation planned yet.
-> **Decision requested:** the three open questions in §6, before any per-phase plan is written.
+> **Decisions settled 2026-07-21** — see §6: one central DB with driver nodes disconnected from it,
+> the two-node keep-oldest gap accepted as ScadaBridge accepts it, and their auth posture matched for
+> now.
 
 ## 1. What this changes
 
@@ -153,9 +155,12 @@ Mirror ScadaBridge's shape, adapted to OtOpcUa's existing substrate:
 - **Central ↔ cluster command/control over ClusterClient**, with exactly one receptionist-registered
   actor per side and central discovering cluster node addresses from `ClusterNode` rows (extended
   with Akka + gRPC addresses, mirroring their `Site` entity).
-- **Deploy stays notify-and-fetch.** The notify crosses via ClusterClient; the artifact continues to
-  come from the shared ConfigDb (see §6.1). The coordinator's expected-ack set comes from
+- **Deploy stays notify-and-fetch, but the fetch changes source.** The notify crosses via
+  ClusterClient; the artifact then comes **from central**, not from the ConfigDb the driver node no
+  longer connects to (§6.1), and is cached in LocalDb. The coordinator's expected-ack set comes from
   `ClusterNode` rows instead of cluster membership.
+- **Driver nodes hold no ConfigDb connection.** LocalDb becomes their steady-state configuration
+  store; five current DB consumers are re-homed or re-sourced per §6.1.
 - **Live telemetry over gRPC**, central dialling into each cluster node, replacing the seven
   observability topics with one stream contract carrying a `oneof` event — additive-only field
   evolution, contract-locked by test.
@@ -163,53 +168,96 @@ Mirror ScadaBridge's shape, adapted to OtOpcUa's existing substrate:
 Deliberately **not** copied: their transient-only central alarm cache and site-local-primary storage
 (see §6.1), and their unauthenticated transports (§6.3).
 
-## 6. Open questions — these need answering before a plan
+## 6. Decisions (settled 2026-07-21)
 
-### 6.1 Shared ConfigDb, or autonomous clusters?
+### 6.1 DECIDED — one central DB, and driver nodes never connect to it
 
-"Mimic ScadaBridge" splits into two very different commitments:
+**Verified in ScadaBridge before adopting.** `Program.cs` calls
+`AddConfigurationDatabase(configDbConnectionString)` at line 264 — inside the Central branch
+(89–478). The Site branch begins at line 479. **Site nodes never register the central configuration
+database at all**; they receive config from central (notify → token-gated HTTP fetch) and persist it
+locally. There is still exactly one authoritative configuration database — it is simply
+central-only.
 
-- **Topology** — separate mesh per cluster. Contained, given §3.
-- **Data architecture** — ScadaBridge sites are *autonomous*: their own SQLite, config fetched from
-  central over token-gated HTTP, because a site is not assumed to reach central's database at all.
+OtOpcUa adopts the same shape: **the ConfigDb stays the single source of truth, and driver nodes stop
+connecting to it.** They obtain their configuration from the central nodes and cache it locally in
+LocalDb.
 
-OtOpcUa's premise is the opposite: one shared ConfigDb every node reads directly, with LocalDb as an
-**outage cache** — the entire basis of Phase 1's boot-from-cache.
+**This promotes LocalDb from an outage cache to the driver node's steady-state configuration store.**
+That is a reframing of the Phase 1 work rather than a contradiction of it: boot-from-cache stops
+being the exceptional path and becomes the normal one, and "central SQL unreachable" stops being a
+degraded mode for driver nodes because they never held a connection to lose. The chunking,
+SHA-256 verification, newest-2 retention and pair replication all carry over unchanged.
 
-**Recommendation: take the topology, keep the shared ConfigDb.** Sites are autonomous in ScadaBridge
-because they genuinely cannot rely on reaching central. If OtOpcUa's nodes can — and Phase 1 assumes
-they normally can — then site-local-primary buys a large amount of machinery (staged pending rows,
-fetch tokens, TTL purge singleton, reconciliation service) for a problem we do not have. Adopting it
-would also reverse the premise of the LocalDb work just shipped.
+**Scope — five driver-side ConfigDb consumers must be re-homed or re-sourced.** Audited:
 
-### 6.2 The two-node keep-oldest outage gap
+| Consumer | Current use | Disposition |
+|---|---|---|
+| `DriverHostActor` | artifact fetch + ack writes (5 `CreateDbContext` sites) | artifact via fetch-and-cache; acks via the ack transport |
+| `EfAlarmConditionStateStore` | Part 9 alarm condition state | **move to LocalDb** — it is pair-local state, the same journey Phase 2 made for the alarm S&F buffer |
+| `DbHealthProbeActor` | DB reachability feeding ServiceLevel tiering | **semantics change** — a driver node has no DB to probe; decide what health input replaces it |
+| `OpcUaPublishActor` | (audit required) | TBD |
+| `ServiceCollectionExtensions` | registration | follows the above |
 
-Inheriting 2-node keep-oldest inherits the registered hole in §2: the younger survivor cannot
-re-bootstrap alone after the oldest dies. Options: accept it with an operator runbook (their choice);
-add a third lightweight seed/witness per cluster; or use a different downing strategy. **This needs
-an explicit decision — it is a total-outage hole, not a degradation.**
+`DbHealthProbeActor` deserves particular care: DB health is currently a ServiceLevel input, and
+under this model it stops existing for driver nodes. That is a behaviour change on a client-visible
+value, not just an internal refactor.
 
-### 6.3 Authentication on the new transports
+### 6.2 DECIDED — accept the two-node keep-oldest gap, as ScadaBridge does
 
-Their boundary assumes a trusted network. OtOpcUa's clusters may span less trusted links. If so we
-need what they did not build: Akka.Remote TLS and a gRPC interceptor covering the whole service
-surface. `LocalDbSyncAuthInterceptor` is already in our tree as a fail-closed template.
+**Status corrected.** What was resolved is the *documentation and the drill*, not the gap. Gitea
+**PR #12** (closed) — <https://gitea.dohertylan.com/dohertj2/ScadaBridge/pulls/12> — "R2-01: Cluster,
+Host & Failover round-2 fixes" carried:
+
+- **T1** (`badc97af`) — the failover drill now kills the STANDBY by default, and its `active` mode
+  *"measures the registered keep-oldest outage instead of pretending recovery"*.
+- **T2** (`d5364506`) — per-direction failover docs, dropping what the commit message calls *"the
+  undeliverable ~25s active-crash promise"*.
+
+So the closed work made the gap **honest**, not absent. `Component-ClusterInfrastructure.md:113`
+still reads: *"Removing the gap itself is the **registered deferred keep-oldest topology/strategy
+decision** (master tracker 2026-07-08, owner: user)."* There is no open Gitea issue for it — it is
+tracked on that master tracker, not in the issue list.
+
+OtOpcUa inherits the same gap and, for now, the same posture. Worth recording so it is a known
+accepted risk rather than a surprise: after the oldest node of a two-node mesh dies, the survivor
+self-downs and cannot re-bootstrap alone, because only the first-listed seed may self-join. Recovery
+is operator-driven — restart the dead first seed, or restart the survivor with a self-first seed
+override.
+
+### 6.3 DECIDED — match ScadaBridge's auth posture for now
+
+Inter-cluster transports stay unauthenticated and unencrypted, as theirs are: the boundary assumes a
+trusted network. Recorded as an accepted risk rather than an oversight, with two notes for whenever
+it is revisited: `LocalDbSyncAuthInterceptor` is already in our tree as a fail-closed template, and
+the surface most worth gating first is any Pull-style RPC that returns historical rows.
 
 ## 7. Sequencing sketch
 
-Deliberately not a task plan — per-phase plans follow once §6 is settled.
+Deliberately not a task plan — per-phase plans follow, one at a time.
 
 | Phase | Content | Independent of the split? |
 |---|---|---|
 | 0 | Oldest-Up role derivation (§4) | **Yes** — ship first |
 | 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | Yes |
 | 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No |
-| 3 | gRPC stream contract; migrate the seven observability topics | No |
-| 4 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No |
-| 5 | Auth per §6.3; failover drill; live gate | No |
+| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |
+| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No |
+| 5 | gRPC stream contract; migrate the seven observability topics | No |
+| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No |
+| 7 | Failover drill (both directions, per §6.2); live gate | No |
+
+Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each
+deserves its own live gate.
 
 ## 8. Risks
 
+- **LocalDb becomes load-bearing for configuration, not just resilient.** Phase 1 built it as a
+  fallback; §6.1 makes it the driver node's only config store. A cache defect that was previously a
+  degraded-mode bug becomes a total-availability bug — the #485 unreadable-artifact class, with the
+  blast radius raised.
+- **`DbHealthProbeActor` feeds a client-visible value.** Removing the driver-side DB connection
+  changes what ServiceLevel means on those nodes; that is an interop-visible change, not internal.
 - **The rig currently models the topology that would be abolished.** Phase 4 rewrites
   `docker-dev/docker-compose.yml` substantially, and every live gate that depends on it.
 - **Losing gossip loses free fleet-wide observability.** Seven AdminUI panels are fed by DPS today;