From fe12ed9d3467edb8159192c468d22f17afcd8245 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 21 Jul 2026 10:07:00 -0400 Subject: [PATCH] docs(design): settle the three open questions on the per-cluster mesh MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit (1) One central DB, driver nodes disconnected from it. Verified in ScadaBridge before adopting: AddConfigurationDatabase is called at Program.cs:264, inside the Central branch (89-478); the Site branch starts at 479, so site nodes never register the central database at all. There is still exactly one authoritative configuration database — it is central-only, and sites fetch from central and cache locally. The consequence worth naming: this promotes LocalDb from an outage cache to the driver node's steady-state configuration store. Boot-from-cache stops being the exceptional path and becomes the normal one. Phase 1's chunking, SHA-256 verification, retention and pair replication all carry over — but a cache defect that used to be a degraded-mode bug becomes a total-availability one, so the #485 unreadable-artifact class now has a larger blast radius. Audited the five driver-side ConfigDb consumers that must be re-homed. Two are more than mechanical: EfAlarmConditionStateStore holds pair-local Part 9 condition state and should follow the alarm S&F buffer into LocalDb, and DbHealthProbeActor feeds ServiceLevel — a driver node with no DB to probe changes what a client-visible value means. (2) The keep-oldest gap: corrected the status rather than the conclusion. What was resolved is the documentation and the drill, not the hole. Gitea PR #12 (closed) carried T1 badc97af, which made the drill "measure the registered keep-oldest outage instead of pretending recovery", and T2 d5364506, which dropped "the undeliverable ~25s active-crash promise". The gap itself is still the registered deferred keep-oldest topology decision on the 2026-07-08 master tracker, with no open issue. Accepted here with the same posture, recorded as a known risk. (3) Auth matches ScadaBridge for now — unauthenticated inter-cluster transports on a trusted network, recorded as an accepted risk with the fail-closed interceptor already in our tree noted as the template for whenever it is revisited. Sequencing grows from six phases to eight; the two that change a running system's data path rather than its wiring each get their own live gate. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- .../2026-07-21-per-cluster-mesh-design.md | 106 +++++++++++++----- 1 file changed, 77 insertions(+), 29 deletions(-) diff --git a/docs/plans/2026-07-21-per-cluster-mesh-design.md b/docs/plans/2026-07-21-per-cluster-mesh-design.md index f19b7ebf..89f1e22e 100644 --- a/docs/plans/2026-07-21-per-cluster-mesh-design.md +++ b/docs/plans/2026-07-21-per-cluster-mesh-design.md @@ -1,7 +1,9 @@ # Per-Cluster Akka Mesh — design > **Status:** design for review. No implementation planned yet. -> **Decision requested:** the three open questions in §6, before any per-phase plan is written. +> **Decisions settled 2026-07-21** — see §6: one central DB with driver nodes disconnected from it, +> the two-node keep-oldest gap accepted as ScadaBridge accepts it, and their auth posture matched for +> now. ## 1. What this changes @@ -153,9 +155,12 @@ Mirror ScadaBridge's shape, adapted to OtOpcUa's existing substrate: - **Central ↔ cluster command/control over ClusterClient**, with exactly one receptionist-registered actor per side and central discovering cluster node addresses from `ClusterNode` rows (extended with Akka + gRPC addresses, mirroring their `Site` entity). -- **Deploy stays notify-and-fetch.** The notify crosses via ClusterClient; the artifact continues to - come from the shared ConfigDb (see §6.1). The coordinator's expected-ack set comes from +- **Deploy stays notify-and-fetch, but the fetch changes source.** The notify crosses via + ClusterClient; the artifact then comes **from central**, not from the ConfigDb the driver node no + longer connects to (§6.1), and is cached in LocalDb. The coordinator's expected-ack set comes from `ClusterNode` rows instead of cluster membership. +- **Driver nodes hold no ConfigDb connection.** LocalDb becomes their steady-state configuration + store; five current DB consumers are re-homed or re-sourced per §6.1. - **Live telemetry over gRPC**, central dialling into each cluster node, replacing the seven observability topics with one stream contract carrying a `oneof` event — additive-only field evolution, contract-locked by test. @@ -163,53 +168,96 @@ Mirror ScadaBridge's shape, adapted to OtOpcUa's existing substrate: Deliberately **not** copied: their transient-only central alarm cache and site-local-primary storage (see §6.1), and their unauthenticated transports (§6.3). -## 6. Open questions — these need answering before a plan +## 6. Decisions (settled 2026-07-21) -### 6.1 Shared ConfigDb, or autonomous clusters? +### 6.1 DECIDED — one central DB, and driver nodes never connect to it -"Mimic ScadaBridge" splits into two very different commitments: +**Verified in ScadaBridge before adopting.** `Program.cs` calls +`AddConfigurationDatabase(configDbConnectionString)` at line 264 — inside the Central branch +(89–478). The Site branch begins at line 479. **Site nodes never register the central configuration +database at all**; they receive config from central (notify → token-gated HTTP fetch) and persist it +locally. There is still exactly one authoritative configuration database — it is simply +central-only. -- **Topology** — separate mesh per cluster. Contained, given §3. -- **Data architecture** — ScadaBridge sites are *autonomous*: their own SQLite, config fetched from - central over token-gated HTTP, because a site is not assumed to reach central's database at all. +OtOpcUa adopts the same shape: **the ConfigDb stays the single source of truth, and driver nodes stop +connecting to it.** They obtain their configuration from the central nodes and cache it locally in +LocalDb. -OtOpcUa's premise is the opposite: one shared ConfigDb every node reads directly, with LocalDb as an -**outage cache** — the entire basis of Phase 1's boot-from-cache. +**This promotes LocalDb from an outage cache to the driver node's steady-state configuration store.** +That is a reframing of the Phase 1 work rather than a contradiction of it: boot-from-cache stops +being the exceptional path and becomes the normal one, and "central SQL unreachable" stops being a +degraded mode for driver nodes because they never held a connection to lose. The chunking, +SHA-256 verification, newest-2 retention and pair replication all carry over unchanged. -**Recommendation: take the topology, keep the shared ConfigDb.** Sites are autonomous in ScadaBridge -because they genuinely cannot rely on reaching central. If OtOpcUa's nodes can — and Phase 1 assumes -they normally can — then site-local-primary buys a large amount of machinery (staged pending rows, -fetch tokens, TTL purge singleton, reconciliation service) for a problem we do not have. Adopting it -would also reverse the premise of the LocalDb work just shipped. +**Scope — five driver-side ConfigDb consumers must be re-homed or re-sourced.** Audited: -### 6.2 The two-node keep-oldest outage gap +| Consumer | Current use | Disposition | +|---|---|---| +| `DriverHostActor` | artifact fetch + ack writes (5 `CreateDbContext` sites) | artifact via fetch-and-cache; acks via the ack transport | +| `EfAlarmConditionStateStore` | Part 9 alarm condition state | **move to LocalDb** — it is pair-local state, the same journey Phase 2 made for the alarm S&F buffer | +| `DbHealthProbeActor` | DB reachability feeding ServiceLevel tiering | **semantics change** — a driver node has no DB to probe; decide what health input replaces it | +| `OpcUaPublishActor` | (audit required) | TBD | +| `ServiceCollectionExtensions` | registration | follows the above | -Inheriting 2-node keep-oldest inherits the registered hole in §2: the younger survivor cannot -re-bootstrap alone after the oldest dies. Options: accept it with an operator runbook (their choice); -add a third lightweight seed/witness per cluster; or use a different downing strategy. **This needs -an explicit decision — it is a total-outage hole, not a degradation.** +`DbHealthProbeActor` deserves particular care: DB health is currently a ServiceLevel input, and +under this model it stops existing for driver nodes. That is a behaviour change on a client-visible +value, not just an internal refactor. -### 6.3 Authentication on the new transports +### 6.2 DECIDED — accept the two-node keep-oldest gap, as ScadaBridge does -Their boundary assumes a trusted network. OtOpcUa's clusters may span less trusted links. If so we -need what they did not build: Akka.Remote TLS and a gRPC interceptor covering the whole service -surface. `LocalDbSyncAuthInterceptor` is already in our tree as a fail-closed template. +**Status corrected.** What was resolved is the *documentation and the drill*, not the gap. Gitea +**PR #12** (closed) — — "R2-01: Cluster, +Host & Failover round-2 fixes" carried: + +- **T1** (`badc97af`) — the failover drill now kills the STANDBY by default, and its `active` mode + *"measures the registered keep-oldest outage instead of pretending recovery"*. +- **T2** (`d5364506`) — per-direction failover docs, dropping what the commit message calls *"the + undeliverable ~25s active-crash promise"*. + +So the closed work made the gap **honest**, not absent. `Component-ClusterInfrastructure.md:113` +still reads: *"Removing the gap itself is the **registered deferred keep-oldest topology/strategy +decision** (master tracker 2026-07-08, owner: user)."* There is no open Gitea issue for it — it is +tracked on that master tracker, not in the issue list. + +OtOpcUa inherits the same gap and, for now, the same posture. Worth recording so it is a known +accepted risk rather than a surprise: after the oldest node of a two-node mesh dies, the survivor +self-downs and cannot re-bootstrap alone, because only the first-listed seed may self-join. Recovery +is operator-driven — restart the dead first seed, or restart the survivor with a self-first seed +override. + +### 6.3 DECIDED — match ScadaBridge's auth posture for now + +Inter-cluster transports stay unauthenticated and unencrypted, as theirs are: the boundary assumes a +trusted network. Recorded as an accepted risk rather than an oversight, with two notes for whenever +it is revisited: `LocalDbSyncAuthInterceptor` is already in our tree as a fail-closed template, and +the surface most worth gating first is any Pull-style RPC that returns historical rows. ## 7. Sequencing sketch -Deliberately not a task plan — per-phase plans follow once §6 is settled. +Deliberately not a task plan — per-phase plans follow, one at a time. | Phase | Content | Independent of the split? | |---|---|---| | 0 | Oldest-Up role derivation (§4) | **Yes** — ship first | | 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | Yes | | 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No | -| 3 | gRPC stream contract; migrate the seven observability topics | No | -| 4 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No | -| 5 | Auth per §6.3; failover drill; live gate | No | +| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No | +| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No | +| 5 | gRPC stream contract; migrate the seven observability topics | No | +| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No | +| 7 | Failover drill (both directions, per §6.2); live gate | No | + +Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each +deserves its own live gate. ## 8. Risks +- **LocalDb becomes load-bearing for configuration, not just resilient.** Phase 1 built it as a + fallback; §6.1 makes it the driver node's only config store. A cache defect that was previously a + degraded-mode bug becomes a total-availability bug — the #485 unreadable-artifact class, with the + blast radius raised. +- **`DbHealthProbeActor` feeds a client-visible value.** Removing the driver-side DB connection + changes what ServiceLevel means on those nodes; that is an interop-visible change, not internal. - **The rig currently models the topology that would be abolished.** Phase 4 rewrites `docker-dev/docker-compose.yml` substantially, and every live gate that depends on it. - **Losing gossip loses free fleet-wide observability.** Seven AdminUI panels are fed by DPS today;