lmxopcua/docs/plans/2026-07-22-per-cluster-mesh-program.md

# Per-Cluster Mesh Program — Align OtOpcUa's Akka Topology with ScadaBridge

> **For Claude:** This is a PROGRAM plan (phase roadmap + gates), not a bite-sized task plan.
> The authoritative design is `docs/plans/2026-07-21-per-cluster-mesh-design.md` (decisions settled
> 2026-07-21; Phases 0a/0b DONE). Per that design's own instruction, **each phase gets its own
> detailed plan, written when the phase starts** (superpowers writing-plans → executing-plans),
> so plans are authored against current code, not against a forecast. This document sequences the
> phases, fixes the deployment topology (co-location with ScadaBridge — NEW constraint 2026-07-22),
> and defines each phase's entry/exit gates.

**Goal:** OtOpcUa runs the same Akka.NET mesh shape as ScadaBridge — **one Akka mesh per
application `Cluster`, two nodes max**, central ↔ cluster joined by explicit transports
(ClusterClient + gRPC + fetch), driver nodes holding **no ConfigDb connection** — so that every
Primary-gated decision and every gated resource share one pair-local scope, and both products
present one operational model on the shared site hardware.

**Why now (2026-07-22):** OtOpcUa cluster pairs will run **on the same two Windows VMs as the
ScadaBridge site nodes**. That deployment makes the current single-fleet-mesh design actively
wrong for sites (a site's OtOpcUa nodes would gossip across the WAN to central and elect ONE
Primary fleet-wide), and makes the ScadaBridge shape the obviously correct one: each site's two
VMs host two independent, identically-postured 2-node clusters (one per product), surviving alone
on the site LAN. It also means driver nodes must not depend on reaching central SQL — **sites
have no SQL Server**, so §6.1's "driver nodes never connect to the ConfigDb" stops being an
architectural preference and becomes a deployment requirement.

---

## Deployment topology (the co-location constraint, NEW)

Each **site**: 2 Windows VMs. Each VM runs one ScadaBridge site node AND one OtOpcUa driver node.
Two independent 2-node Akka clusters per site — they share hardware, never a mesh. **Central**:
2 Windows VMs, each running a ScadaBridge central node and an OtOpcUa central node
(`admin,driver`), again as separate pairs.

**Per-VM port allocation (no collisions — verify against actual deployment configs in Phase 6):**

| Port | Owner | Purpose |
|---|---|---|
| 8081 / 8082 | ScadaBridge | Akka remoting (central / site) |
| 8083 | ScadaBridge site | gRPC h2c (streams + LocalDb sync) |
| 8084 | ScadaBridge site | metrics |
| 5000 (+Traefik) | ScadaBridge central | UI + Inbound API |
| **4053** | OtOpcUa | Akka remoting |
| **4840** | OtOpcUa | OPC UA endpoint |
| OtOpcUa AdminUI port | OtOpcUa central (admin) | AdminUI (driver-only site nodes host no UI) |
| OtOpcUa LocalDb sync port | OtOpcUa driver | h2c LocalDb pair replication (default off/0 today — Phase 6 assigns a real per-site port) |

**Aligned HA posture (both products, per VM — already true or landing via the selfform plan):**
auto-down downing (15 s window), oldest-Up active/primary election, self-first seed ordering
(the `SelfFormAfter` self-form fallback that briefly stood in for it was retired 2026-07-22),
termination-watchdog → process exit → `sc.exe failure` restart recovery.
One failover story for operators regardless of product.

**Prerequisite ordering:** the fallback/manual-failover plan
(`docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`) executes **before** this
program — it is small, independent, and Phase 6 depends on its semantics (under per-pair meshes
every node lists itself + partner as seeds, so the fallback covers both nodes of every pair;
the site-node island guard then simply never triggers).

---

## What changes, in one table (from the design doc)

| Aspect | Today (single fleet mesh) | Target (ScadaBridge shape) |
|---|---|---|
| Mesh | one gossip ring, all nodes, seeded by central-1 | one 2-node mesh per application `Cluster`; same ActorSystem name; separation by seed-node partitioning |
| Roles | `admin` / `driver` fleet-wide | `driver` + cluster-specific `cluster-{ClusterId}`; singletons scoped to the cluster role |
| Primary election | mesh-wide oldest Up driver (Phase 0b) | pair-local by construction — same rule, correct scope |
| Command/control | 9 DPS topics + singleton over gossip | ClusterClient (one receptionist actor per side); central discovers nodes from `ClusterNode` rows |
| Deploy | notify (DPS) + node fetches from ConfigDb | notify via ClusterClient; artifact fetched **from central**, cached in LocalDb |
| Driver ConfigDb connection | direct EF connection to central SQL | **none** — LocalDb is the steady-state config store |
| Live telemetry | 7 observability DPS topics | one gRPC stream contract (`oneof` event), central dials each cluster node |
| Rig | six-node single mesh (`docker-dev`) | per-cluster meshes; rig models the real topology |

---

## Phases

Each phase below is one row of design-doc §7, expanded with entry/exit gates. **Execution recipe
per phase:** (1) invoke writing-plans in this repo to produce
`docs/plans/2026-07-2X-mesh-phase-N-<name>.md` from the scope notes here + the design doc §
references, exploring current code first; (2) execute it task-by-task; (3) run the phase's exit
gate; (4) update this file's status column and the design doc's §7 table.

### Phase 1 — `ClusterNode` address columns + DB-sourced ack set — **DONE 2026-07-22**
Shipped per `2026-07-21-per-cluster-mesh-phase1.md`; see that plan's Task 6 gate record and the
design doc's "Phase 1 as shipped" note for the two scope deviations (no cluster-scope filtering — no
such deployment exists; no per-node role column) and the one addition (`ClusterNodeAddressReconciler`).
**AdminUI node edit was deferred to Phase 2** — nothing reads the columns until then, and the
migration default plus the rig seed cover every node today.

**Scope:** `ClusterNode` gains Akka + gRPC address columns (mirroring ScadaBridge's `Site`
entity `NodeAAddress`/`GrpcNodeAAddress` pattern, but per-node rows); EF migration; AdminUI node
edit surfaces the fields; `ConfigPublishCoordinator` derives its expected-ack set from
`ClusterNode` rows instead of `Akka.Cluster.State.Members` filtered by role (design §3 fact 3 —
this removes the coordinator's one genuinely mesh-bound dependency).
**Independent of the split:** yes — safe on the current mesh.
**Exit gate:** deploy on the unchanged docker-dev rig completes with the coordinator's expected-ack
set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node
missing; a node up but row absent → its ack is not expected).

### Phase 2 — Comm actors + ClusterClient transport — **DONE 2026-07-22 (live gate PASSED)**
Shipped per [`2026-07-22-mesh-phase2-clusterclient-transport.md`](2026-07-22-mesh-phase2-clusterclient-transport.md)
as a **dark switch**: `MeshTransport:Mode` defaults to `Dps`, so nothing changes on any deployment
until the flag flips. The live gate ([`2026-07-22-mesh-phase2-live-gate.md`](2026-07-22-mesh-phase2-live-gate.md))
flipped the whole rig to `ClusterClient` and passed for the transport, with four findings — most
notably that `buffer-size = 0`'s stated rationale was overstated (a node applies a `TimedOut`
deployment anyway via boot-time orphan-row replay) and that stopping both centrals strands non-seed
nodes at the membership layer (a Phase 7 drill item, transport-independent). Both comm actors are spawned and receptionist-registered in **both** modes, so
the flip is a config change rather than a redeploy. Three scope corrections landed against the
design doc (that plan's header carries them in full):

- the node path is `/user/node-communication`, not `/user/cluster-communication` (there is no
  per-cluster scoping until Phase 6);
- the substitution for `Publish` is **`SendToAll`**, not `Send` — `Send` reaches exactly one node and
  would deploy to one member of the fleet while the rest silently kept their old config;
- central creates **exactly one fleet-wide ClusterClient**. On a single mesh a receptionist serves
  its whole cluster, so a client per application `Cluster` would fan every command out once per
  cluster. Per-cluster clients become correct only when Phase 6 splits the meshes (`TODO(Phase 6)`
  in `CentralCommunicationActor.RebuildClient`).

**Exit-gate deviation — "an Ask timing out cleanly against a stopped node" cannot be run as written.**
There is no cross-boundary Ask in Phase 2: every migrated command is fire-and-forget on the wire.
`DispatchDeployment` is a `Tell` whose ack returns later as an independent `ApplyAck`;
`RestartDriver` / `ReconnectDriver` / `AlarmCommand` reply `Ok = true` from the admin node the moment
they are dispatched (`Ok` has always meant *dispatched*, not *applied* — the AdminUI string is
literally "Restart dispatched"). The honest equivalents, both in the live gate: a **stopped node**
fails the deploy at the apply deadline naming it (Phase 1 semantics), and **no reachable contact**
drops the command with a Warning and never buffers it.

**Scope:** one receptionist-registered actor per side (`/user/central-communication`,
`/user/cluster-communication`), registered **per node, not as a singleton** (contact rotation);
ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge
idioms copied verbatim: sender-preserving `Tell(new ClusterClient.Send(...), Sender)` Ask relay,
typed-failure reply for every unhandled message, no central buffering toward unreachable clusters
(drop + warn), central discovers contacts from Phase 1's `ClusterNode` rows (60 s refresh +
admin-change refresh), clusters know central from appsettings (static, restart to change).
**Frame-size guard:** anything carrying payload sets both the frame limit and
`log-frame-size-exceeding` (design §8) — the deploy path stays payload-free by design.
**Exit gate:** on the still-single-mesh rig, deploy notify + acks and AdminUI Reconnect/Restart
flow over ClusterClient (DPS paths deleted or dark-switched), including an Ask timing out cleanly
against a stopped node.

### Phase 3 — Config fetch-and-cache from central — **DONE 2026-07-23 (merged `d01b0695`, live gate PASSED, pushed to origin)**
**Decisions taken:** the deploy artifact is served **by central over a gRPC fetch RPC**
(`DeploymentArtifactService.Fetch`, streamed chunks — chosen over token-gated HTTP), authenticated by
a **shared node bearer key** (`ConfigServe:ApiKey == ConfigSource:ApiKey`, `FixedTimeEquals`,
fail-closed — chosen over a per-deployment token, so **no migration** and `DispatchDeployment` stays
payload-free). Cutover is a **config-flagged dark switch** `ConfigSource:Mode` (`Direct` default |
`FetchAndCache`), per node. Plan + task ledger:
[`docs/plans/2026-07-22-mesh-phase3-config-fetch-and-cache.md`](2026-07-22-mesh-phase3-config-fetch-and-cache.md).
**Shipped (all green, sabotage-checked):** `ConfigSource`/`ConfigServe` options + validator; the repo's
first in-repo `.proto` + `Grpc.Tools` codegen; central `DeploymentArtifactGrpcService` (streams the
sealed blob, NotFound-hides unknown/not-sealed/empty); `ConfigServeAuthInterceptor` + a dedicated h2c
Kestrel listener merged with the LocalDb-sync listener (re-binds the existing surface once, adds both
dedicated ports); node `GrpcDeploymentArtifactFetcher` (SHA-verified reassembly, endpoint failover,
null-on-any-failure); the `FetchAndCache` apply path (`PipeTo(Self)`, fetch→cache→apply-from-bytes,
null = apply-failure keeping last-known-good) and bootstrap-from-the-LocalDb-pointer (no driver-side
config SQL read); #485 coverage; and a **real two-Host gRPC boundary test** (right-key reassembles,
wrong-key control returns null, failover). Chunking, SHA-256 verify, newest-2 retention, pair
replication carry over unchanged. **Phase boundary held:** config **reads** only — the
`NodeDeploymentState` ack-row write, `DbHealthProbe`, `EfAlarmConditionStateStore` stay for Phase 4.
**Live gate PASSED:** deployed on a site pair flipped to `FetchAndCache` with central SQL stopped
mid-fetch — retry landed; #485 last-known-good re-proven on the new path. Merged to master
`d01b0695` and pushed to origin; the scadaproj umbrella index was updated + pushed (`b5e7bc8`) in the
same pass.
**Exit gate — MET:** live gate green on the rig; a driver node with an empty LocalDb and reachable
central boots into the current config; with central down it boots last-known-good.

### Phase 4 — Cut the driver-side ConfigDb connection — **DONE 2026-07-23 (live gate PASSED; Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred)**
**Scope (design §6.1 audit table):** re-home `EfAlarmConditionStateStore` to LocalDb (pair-local
state, same journey as the Phase-2 alarm S&F buffer); **resolve the `DbHealthProbeActor`
question** — driver nodes have no DB to probe, and DB health currently feeds ServiceLevel tiering,
so define the replacement health input (candidate: central-reachability via the Phase 2/3
transports) — this is a client-visible ServiceLevel semantics change and must be documented in
`docs/Redundancy.md` + the interop playbook; audit `OpcUaPublishActor`'s ConfigDb use (TBD in the
design) and re-source it; registration cleanup in `ServiceCollectionExtensions`; driver-role
`Program.cs` branch registers no EF ConfigDb context at all (mirror ScadaBridge's central-only
`AddConfigurationDatabase`).
**Shipped (branch `feat/mesh-phase4`, Tasks 0–7 + 1b):** `Program.cs` gates `AddOtOpcUaConfigDb` +
the ConfigDb health check on `hasAdmin`, so a driver-only node registers no ConfigDb connection
string at all; `ConfigSourceOptionsValidator` fails host start if a driver-only node's
`ConfigSource:Mode` is left `Direct` (fused `admin,driver` may stay `Direct`); the DB-health
question resolved as **retire, don't replace** — `DbHealthProbeActor` is simply not spawned when no
ConfigDb is present, and `OpcUaPublishActor` runs a `dbLess` mode feeding `DbReachable=true`
constant with staleness from the redundancy-snapshot age alone (documented as the client-visible
change in `docs/Redundancy.md`); `LocalDbAlarmConditionStateStore` (replicated
`alarm_condition_state` table) replaces `EfAlarmConditionStateStore` on every driver-role node;
`DriverHostActor` no longer writes `NodeDeploymentState` on a driver-only node (central's
`ConfigPublishCoordinator.PersistNodeAck` was already the ack system of record); driver-only nodes
register `NullLdapGroupRoleMappingService` so LDAP group→role falls back to the
`Security:Ldap:GroupToRole` appsettings baseline (DB grants were never delivered to driver nodes via
config anyway). Docs updated: `docs/Redundancy.md`, `docs/Configuration.md`, `CLAUDE.md`.
**Deferred:** Task 9 — dropping the now-unused `ScriptedAlarmState` ConfigDb table (a follow-up, not
a blocker; the table is simply unread).
**Remaining (Tasks 10–11):** rig config for a driver-only node with no `ConfigDb` connection string,
and the live gate itself — a driver pair runs a full deploy + alarm + historian cycle with no
ConfigDb configured at all, confirming ServiceLevel 240/250 while central SQL is down and
grep-level proof no driver-branch service can resolve the ConfigDb context. **Not yet run — treat
Phase 4 as code-complete, not verified, until this lands.**
**Exit gate:** its own live gate — a driver pair runs a full deploy + alarm + historian cycle with
**no ConfigDb connection string configured at all**; grep-level proof no driver-branch service can
resolve the ConfigDb context.

### Phase 5 — gRPC stream contract for live telemetry — **DONE 2026-07-23 (live gate PASSED)**
**Scope as shipped (narrowed from the program sketch's "seven observability topics," settled with the
user 2026-07-23 — mirrors how Phases 1 and 3 also honestly scoped down):** one server-streaming gRPC
contract carrying a `oneof` event with **four** kinds, **driver nodes host the gRPC server (dedicated
Kestrel h2c port), central dials in** — the inverted direction is the load-bearing ScadaBridge finding
(design §2), required so telemetry survives the Phase 6 mesh split without shared gossip membership.

**The four migrated channels:**

| DPS topic | Domain record | Central sink |
|---|---|---|
| `alerts` | `AlarmTransitionEvent` | `IInProcessBroadcaster<AlarmTransitionEvent>` (+ `AlertHub`) |
| `script-logs` | `ScriptLogEntry` | `IInProcessBroadcaster<ScriptLogEntry>` (+ `ScriptLogHub`) |
| `driver-health` | `DriverHealthChanged` | `IDriverStatusSnapshotStore` |
| `driver-resilience-status` | `DriverResilienceStatusChanged` | `IDriverResilienceStatusStore` |

**Three deferred, each with a settled reason (NOT migrated — do not read the program sketch's original
"seven" as the delivered scope):**
- `redundancy-state` — bidirectional, built from `Cluster.State`, pair-local control plane driving
  ServiceLevel + the Primary gate. Stays on DPS in every mode; under Phase 6 it works in-mesh per pair.
  Central's cross-pair *display* of redundancy is a Phase 6 concern, not a Phase 5 panel.
- `fleet-status` — central-internal; `FleetStatusBroadcaster` builds it from the admin node's own
  cluster membership events and `Fleet.razor` polls the Config DB, ignoring the feed. Not a
  node→central stream at all. Revisit in Phase 6 once central loses site-node gossip visibility.
- `deployment-acks` — already migrated, but onto the Phase 2 `ClusterClient` transport, as a
  command-plane reply, not an observability broadcast.

**Dark switch:** `Telemetry:Mode` (node/serve, default `Dps`) and `TelemetryDial:Mode` (central/dial,
default `Dps`) flip independently to `Grpc`; both code paths are compiled into every binary, so
flipping is a config + restart, not a redeploy. The node **always** hosts the gRPC server whenever
`Telemetry:GrpcListenPort > 0` and **always** emits into its node-local hub + publishes DPS, in both
modes — only central's ingest source (`TelemetryDial:Mode`) actually switches which of the two
always-available sources feeds the AdminUI sinks. The AdminUI panels/components are unchanged; only
the sinks' upstream feed swaps.

**Auth from day one, fail-closed:** a shared node bearer key (`Telemetry:ApiKey` ==
`TelemetryDial:ApiKey`), gated by `TelemetryStreamAuthInterceptor`
(`CryptographicOperations.FixedTimeEquals`, `PermissionDenied` on mismatch, unset key rejects every
call). This supersedes design §6.3's "match ScadaBridge's unauthenticated posture for now" — ScadaBridge
itself has since closed that gap with this identical interceptor pattern (see the supersede note added
to design §6.3, and `docs/Telemetry.md` § Authentication).

**Reconnect story:** central's `TelemetryDialSupervisor` runs one reconnecting dialer per driver node,
discovered from `ClusterNode` rows (`Host`+`GrpcPort`, refreshed every
`TelemetryDial:ContactRefreshSeconds`); immediate first retry then ~5s fixed backoff, generation-stamped
so a superseded stream's late error/event is ignored; the dialer never permanently gives up
(observability, not data plane). The node-local hub replays last-value snapshots for
`driver-health`/`driver-resilience-status` on every (re)subscribe so those stores re-prime immediately;
`alerts`/`script-logs` are append logs and tolerate the gap. The `/alerts`/`/script-log` pill reflects
aggregate stream health (live when ≥1 node stream is up), matching today's DPS pill semantics.

**Docs shipped:** `docs/Telemetry.md` (new, canonical guide), `docs/Configuration.md` §
`Telemetry`/`TelemetryDial`, `docs/Redundancy.md` cross-reference, this section, and the design §6.3
supersede note.

**Exit gate (not yet run):** all AdminUI live panels green against the docker-dev rig with
`Telemetry:Mode=Grpc`/`TelemetryDial:Mode=Grpc` and the four DPS telemetry bridges NOT spawned;
kill-and-reconnect of a site node's dialer recovers every stream (driver-health/resilience re-prime
from the node hub's snapshot replay). Treat Phase 5 as **code-complete, not verified**, until this
lands — see `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` Task 12.

### Phase 6 — Mesh partition + co-location topology — **DONE 2026-07-24**
**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
§"Bootstrap: self-first seed ordering"). The rest of the phase — actually splitting the single
fleet-wide mesh into three independent 2-node meshes (central `cluster-MAIN`, site-a
`cluster-SITE-A`, site-b `cluster-SITE-B`) — **shipped 2026-07-24**. See
`docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md` for the full plan and task list.
Delivered:
- Per-pair `Cluster:SeedNodes` (each node lists itself first, its pair partner second — no
  cross-pair seeds) + `cluster-{ClusterId}` roles on every driver node + singleton re-scoping:
  `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver
  node (falls back to plain `driver` for legacy nodes), so **each pair elects its own Primary —
  two or more Primaries fleet-wide, by design**; `ClusterNodeAddressReconciler` is own-cluster-scoped.
- `CentralCommunicationActor` now creates one `ClusterClient` per application `Cluster` and fans
  commands across them (closes the `TODO(Phase 6)` left by Phase 2).
- New `SplitTopologyTransportValidator` fails host start on a `cluster-{ClusterId}`-role node left
  on a DPS transport (`MeshTransport:Mode` must be `ClusterClient`; `Telemetry:Mode`/`TelemetryDial:Mode`
  must be `Grpc`) — see `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)".
- **Three decisions worth recording:**
  1. **Secrets replication went pair-local, not SQL-hub.** The design flagged "likely SQL-hub mode
     like ScadaBridge" as the fallback if DPS couldn't cross the split meshes — in the event, each
     pair simply keeps replicating its own secrets within itself (no cross-cluster sync needed, and
     no SQL-hub was built, consistent with Phase 4: site nodes have no SQL to hub through).
  2. **The compiled transport-mode defaults (`MeshTransport:Mode`, `Telemetry:Mode`,
     `TelemetryDial:Mode`) were deliberately left at `Dps`**, not flipped to the split-safe modes —
     a blast-radius assessment found flipping the default breaks every integration test and every
     appsettings profile that leaves the key unset. The guardrail is
     `SplitTopologyTransportValidator` plus explicit per-deployment config, not a default flip.
  3. **The docker-dev rig models three OtOpcUa-only meshes** (central/site-a/site-b), not the
     ScadaBridge co-location topology sketched in the design's port table — co-location with
     ScadaBridge's own site nodes stayed out of scope for this phase.
**Exit gate:** rig up in the new three-mesh shape; every existing live-gated behavior re-verified
per pair (deploy, redundancy 250/240 per pair, two-Primaries-fleet-wide confirmed); the
ClusterRedundancy page's mesh-scope caveat and `IManualFailoverService`'s doc-comment caveat
removed (the election is pair-local now); `docs/Redundancy.md`'s per-Akka-cluster KNOWN LIMITATION
marked resolved.

### Phase 7 — Failover drill + live gates
**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
directions; **close the two outstanding live gates**: (a) auto-down 1-vs-1 crash-the-oldest
(deferred since Phase 0a — finally testable, every mesh is exactly two nodes), (b) self-first
lone-cold-start on the real per-pair topology; manual-failover button re-verified per pair;
operator runbook for the co-located site (one page covering both products' failover on the same
two VMs).
**Exit gate:** drill green on every pair type (central, site); runbook merged; design doc §7
table fully marked DONE. (Live gate (b) is now the **self-first cold-start-alone** drill, not
`SelfFormAfter` — the watchdog it named was retired 2026-07-22.)

---

## Risks carried from the design (§8, unchanged — re-read before each phase plan)

LocalDb becomes load-bearing for config (blast radius of the #485 class rises);
`DbHealthProbeActor` feeds a client-visible value; the rig models the doomed topology until
Phase 6; losing gossip loses free observability (7 panels); 128 KB ClusterClient frame drop is
silent unless both knobs are set; never stack app-level LWW on LocalDb's HLC. **New (this
program):** co-located VMs mean a VM loss now takes out one node of BOTH products at once — the
drill in Phase 7 must include the shared-VM failure (both products fail over together), and
resource sizing on the site VMs should be checked once both products run the full stack.

## Tracking

| Phase | Status |
|---|---|
| 0a downing strategy | DONE 2026-07-21 (live gate → Phase 7) |
| 0b oldest-Up election | DONE 2026-07-21 |
| Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. |
| 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
| 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
| 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. |
| 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. |
| 6 mesh partition + co-location | **DONE 2026-07-24** — three independent 2-node meshes (central/site-a/site-b), `cluster-{ClusterId}` roles, per-pair self-first seeds, cluster-scoped `RedundancyStateActor` singleton (two+ Primaries fleet-wide, by design), one `ClusterClient` per `Cluster`, `SplitTopologyTransportValidator`, pair-local secrets replication, compiled transport defaults intentionally kept `Dps`. See `2026-07-24-mesh-phase6-partition-and-colocation.md`. |
| 7 drill + live gates | **DONE 2026-07-24 — all 6 drills PASSED** on the docker-dev three-mesh rig. D1 MAIN graceful manual-failover via the AdminUI button (Primary leaves, peer→250, restarts+rejoins no split); D2 SITE-A auto-down failover both directions; D3 SITE-B crash-the-oldest; D4 **auto-down 1-vs-1 survival (closes the gate deferred since Phase 0a)** — every survivor stayed Up; D5 **self-first cold-start-alone (gate b)** — a node boots alone and forms its mesh; D6 recovery/rejoin. Manual-failover button confirmed MAIN-only by construction (site pairs are driver-only, fail over via auto-down). One-page co-located operator runbook shipped. Carried Phase-6 finding: simultaneous cold-start of both pair VMs can split — operational mitigation (staggered start) in the runbook; product-level guard is a candidate follow-up, not shipped. See `2026-07-24-mesh-phase7-failover-drills.md`. **Per-cluster mesh program COMPLETE.** |