Phase 2 of the per-cluster mesh program: move the three central->node command
channels (deployments, driver-control, alarm-commands) and the deployment-acks
reply channel off DistributedPubSub onto Akka ClusterClient, behind a config
flag.
Recon of both repos turned up three corrections to the design doc's account of
the sister project, all folded into the plan:
- Design doc S2 claims "every unhandled message replies with a typed failure".
ScadaBridge has no ReceiveAny/Unhandled override in either comm actor; the
typed-failure idiom fires only for a MISSING REGISTRATION, per message type.
- "No central buffering" was never a setting they wrote. They have zero
akka.cluster.client HOCON and run the defaults (buffer-size 1000,
reconnect-timeout off). We choose buffer-size = 0 deliberately.
- Publish -> Send is the wrong substitution. Today's deploy notify is a
broadcast every DriverHostActor receives (no ClusterId filter exists on the
node side), so it must become SendToAll. Send would deploy to exactly one
node of the fleet and seal green on partial acks.
Also records the single-mesh duplicate-delivery trap (one ClusterClient in
Phase 2, not one per Cluster -- a receptionist serves its whole mesh) and one
deviation from the program plan's exit gate: there is no cross-boundary Ask in
Phase 2, because every migrated command is fire-and-forget with a local reply,
so "an Ask timing out cleanly" cannot be run as written.
Marks the program tracking tables current: prereq + Phase 1 are DONE (they
still read "not executed"), Phase 2 in progress.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
# Per-Cluster Mesh Phase 2 — Comm Actors + ClusterClient Transport
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
**Goal:** Move the three central→node command channels (`deployments`, `driver-control`, `alarm-commands`) and the `deployment-acks` reply channel off DistributedPubSub and onto Akka **ClusterClient**, behind a config flag, so central can command a cluster it does not share a gossip ring with.
**Architecture:** One receptionist-registered comm actor per side — `/user/central-communication` (on admin nodes) and `/user/node-communication` (on driver nodes), each registered **per node, not as a singleton**, so contact rotation reaches whichever answers. Central's singletons stop touching the mediator: they `Tell` a `MeshCommand` envelope to a node-local router actor that decides DPS or ClusterClient from `MeshTransport:Mode`. Inbound commands land on the node's **EventStream** — genuinely node-local, unlike DPS — and existing subscribers add one `EventStream.Subscribe` line beside their existing DPS `Subscribe`, so the dark switch touches only publishers.
**Tech Stack:** .NET 10, Akka.NET 1.5.62 (`Akka.Cluster.Tools` already referenced by Cluster, Runtime, and ControlPlane — no new package), xUnit + Shouldly, Akka.TestKit.Xunit2.
---
## Decisions already made (do not re-litigate)
| Decision | Choice | Why |
|---|---|---|
| Cutover | **Config-flagged dark switch** (`MeshTransport:Mode` = `Dps` default \| `ClusterClient`) | Lets the rig gate A/B the same deploy and roll back with a config change, not a revert. Phase 1's live gate is the argument: the defect only appeared against real infrastructure. |
| Buffering | **`buffer-size = 0`** (drop immediately) | Akka's default buffers 1000 and replays on reconnect. With Phase 1 semantics a deploy recorded `TimedOut` would still be applied minutes later when the node returns — the node runs a config the DB says failed. Fail-fast keeps the DB the single record of truth. |
| `alarm-commands` central leg | **In scope** | Same shape as driver-control; leaves no half-migrated command plane. The topic stays alive for the node-local publisher. |
---
## Three corrections to the design doc, found during recon
These change what "copy ScadaBridge verbatim" means. Fold them into the code comments as you go.
1.**`docs/plans/2026-07-21-per-cluster-mesh-design.md` §2 is wrong about typed failure replies.** It says *"Every unhandled message replies with a typed failure rather than dropping."* ScadaBridge has **no `ReceiveAny` / `Unhandled` override in either comm actor.** The typed-failure idiom is per-message-type and fires only when a *registration* is missing (no ClusterClient yet, no handler registered). A genuinely unknown message type still dead-letters. Do not build a catch-all failure reply believing it is the sister project's pattern — Task 9 corrects the design doc.
2.**"No central buffering" was never a setting they wrote.** ScadaBridge has *zero*`akka.cluster.client.*` HOCON; production runs Akka defaults (`buffer-size = 1000`, `reconnect-timeout = off`). Their "no buffering" is app-level only: no ClusterClient entry for a site ⇒ drop + warn. We are choosing `buffer-size = 0` deliberately (Task 1), which they did not.
3.**`Publish` → `Send` is the wrong substitution; it must be `SendToAll`.** Today's deploy notify is a DPS broadcast that **every**`DriverHostActor` receives — there is no ClusterId or node filter anywhere on the node side (`DriverHostActor.cs:476-485`); scoping happens later, inside the artifact (`DeploymentArtifact.ResolveClusterScope`, `DriverHostActor.cs:1801`). `ClusterClient.Send` delivers to exactly **one** registered actor and would silently deploy to one node of the fleet.
---
## The single-mesh duplicate-delivery trap — read before Task 3
Phase 2 ships while the fleet is **still one Akka mesh**. A `ClusterClientReceptionist` serves its whole cluster, so `SendToAll("/user/node-communication")` reaches every registered node-comm actor **in the entire mesh**, regardless of which node's address was used as the contact point.
Therefore central must create **exactly ONE ClusterClient** in Phase 2, with contact points drawn from all enabled non-maintenance `ClusterNode` rows. Creating one client *per application Cluster* — which is what Phase 6 will want, and what ScadaBridge does — would fan the same command out once per cluster, producing N× duplicate `DispatchDeployment` and N× duplicate `ApplyAck`.
That is not a bug to fix later; on the single mesh the fleet-wide client is *correct*, and it produces exactly today's DPS fan-out. Phase 6 splits the meshes and converts this to per-cluster clients. Mark the code with `TODO(Phase 6)`.
---
## Deviation from the program plan's exit gate
`docs/plans/2026-07-22-per-cluster-mesh-program.md` §Phase 2 asks the gate to include *"an Ask timing out cleanly against a stopped node."*
**There is no cross-boundary Ask in Phase 2, so that gate item cannot be run as written.** Every command being migrated is fire-and-forget on the wire:
-`DispatchDeployment` — `Tell`; the ack returns later as a separate `ApplyAck` message, not an Ask reply (`ConfigPublishCoordinator.cs:111` subscribes to a topic).
-`RestartDriver` / `ReconnectDriver` — `AdminOperationsActor.cs:397,427` publish, then reply `Ok = true` immediately from the admin node. `Ok` means *dispatched*, not *applied*; the AdminUI string is literally "Restart dispatched".
The honest equivalents, both of which Task 10's gate runs:
- **a stopped node** ⇒ the deploy fails at the apply deadline naming that node (Phase 1 semantics), not an Ask timeout;
- **no ClusterClient / no reachable contact** ⇒ the command is dropped with a Warning and never buffered, and the AdminUI still reports "dispatched" (a pre-existing lie, now on a new transport — record it, do not fix it here).
Task 9 records this deviation in the program plan.
`ZB.MOM.WW.OtOpcUa.Cluster` is referenced by both ControlPlane (`ControlPlane.csproj:23`) and Runtime, so this is the right home — next to `AkkaClusterOptions`.
In `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs`, inside `AddOtOpcUaCluster`, directly after the existing `AddValidatedOptions<AkkaClusterOptions, …>` call:
**The test must read the effective config off a running `ActorSystem`, never the HOCON text.** This is a settled lesson in this repo (`SplitBrainResolverActivationTests`, and the comment at `ServiceCollectionExtensions.cs:96-99`): several fragments compete and the losing one still reads correctly in the file.
# DELIBERATE, and NOT what the sister project runs (they never wrote this section, so
# they inherit buffer-size = 1000). Zero means: if the receptionist is unknown, drop the
# message now. Buffering would replay a DispatchDeployment on reconnect and apply a
# deployment that ConfigPublishCoordinator already sealed as TimedOut — the node would
# then be running a configuration the database says failed. Fail-fast keeps the DB the
# single record of truth, and matches today's DPS behaviour exactly (a node that is down
# misses the publish and self-heals from the DB on restart).
buffer-size = 0
# Retry forever rather than self-stopping the client. A central node that is down for an
# hour must not require a driver-node restart to become reachable again.
reconnect-timeout = off
receptionist {
# Empty = every member hosts a receptionist. This is what makes the comm actors
# "registered per node, not as a singleton" work — contact rotation reaches whichever
# node answers. Do not scope this to a role.
role = ""
}
}
```
**Step 4: Run the tests**
Run: `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~MeshTransportHoconTests"`
Expected: PASS, 4/4.
**Step 5: Sabotage-verify**
Change `buffer-size = 0` to `buffer-size = 1000`, re-run: `Cluster_client_buffering_is_disabled` must go RED. Restore, re-run, green.
> **MSBuild mtime trap (cost several cycles in Phase 1):** `akka.conf` is an **embedded resource**. If you restore a file with `mv` or `git checkout` its mtime may land older than the build output and MSBuild will skip recompiling, so the test keeps reading the sabotaged value. After restoring, run `touch src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf` and rebuild.
1.`Receive<MeshCommand>` — under `Dps`, `mediator.Tell(new Publish(cmd.Topic, cmd.Message))`; under `ClusterClient`, `client.Tell(new ClusterClient.SendToAll(NodeCommunicationPath, cmd.Message))`. **`SendToAll`, never `Send`** — see the corrections section.
2.`Receive<ApplyAck>` — inbound from a node's ClusterClient; `Forward` to the `ConfigPublishCoordinatorKey` singleton proxy so `Sender` survives the hop.
3. Contact-point cache — rebuild from enabled non-maintenance `ClusterNode` rows on a `ContactRefreshSeconds` timer and on demand; **exactly one** client (single-mesh trap above).
4.`Receive<Status.Failure>` — a faulted DB load piped back. Without this handler the refresh fails at Debug level and an operator cannot distinguish "no nodes configured" from "the database is down".
Key implementation notes to put in the code:
```csharp
/// <summary>Path the node-side comm actor registers under. This string is the wire contract.</summary>
// TODO(Phase 6): one client per application Cluster. Today ONE client, fleet-wide contacts.
// A ClusterClientReceptionist serves its whole cluster, and the fleet is still a single mesh,
// so SendToAll from ANY contact point reaches every registered node-comm actor in the mesh.
// One client per Cluster would therefore fan each command out once per cluster — N x duplicate
// DispatchDeployment and N x duplicate ApplyAck. On the single mesh the fleet-wide client is
// not a compromise: it reproduces today's DPS fan-out exactly.
```
Dropping when unroutable (the app-level "no buffering" decision — Akka's own buffer is already 0 from Task 1):
```csharp
if(_clientisnull)
{
_log.Warning(
"No ClusterClient — dropping {MessageType}. No enabled ClusterNode rows produced a "
+"usable contact point at the last refresh; the command is NOT buffered and will "
+"not be retried",
cmd.Message.GetType().Name);
return;
}
```
Address parsing — per-row try/catch so one malformed row cannot abort the whole refresh and leave the cache half-built (a shipped sister-project bug, with a regression test that deliberately orders the bad row first):
"ClusterNode {NodeId} has an unusable address {Host}:{Port}; skipping it in "
+"this refresh (other nodes are unaffected)",nodeId,host,akkaPort);
}
}
```
**Tests (all must exist):**
| Test | Asserts |
|---|---|
| `Dps_mode_publishes_on_the_carried_topic` | mediator sees `Publish(topic, msg)`; no client created |
| `ClusterClient_mode_sends_to_all_not_to_one` | probe receives `ClusterClient.SendToAll` with `Path == "/user/node-communication"` — **assert the type is `SendToAll`, not merely that something was sent** |
| `Exactly_one_client_is_created_for_a_multi_cluster_fleet` | seed `ClusterNode` rows across two `ServerCluster`s; factory invoked once |
| `ApplyAck_is_forwarded_to_the_coordinator_preserving_sender` | coordinator probe receives `ApplyAck` with `LastSender` == the original |
| `A_malformed_node_row_does_not_abort_the_refresh` | bad row ordered **first**; the good row still yields a contact |
| `A_faulted_db_load_logs_a_warning` | `Status.Failure` → Warning, actor still alive |
Use `AwaitAssert`/`ExpectMsg` with explicit timeouts. **Do not use `Thread.Sleep`** — the sister project's older comm-actor tests do, and it is their documented flake source.
**Sabotage-verify:** change `SendToAll` to `Send`; `ClusterClient_mode_sends_to_all_not_to_one` must go RED. This is the single most important assertion in the phase — `Send` would deploy to one node of the fleet and every other node would silently keep its old config.
**Why inbound commands go to the `EventStream` and not back onto DPS.** Re-publishing an inbound command onto its DPS topic would fan it across the *whole mesh* — and central has already `SendToAll`-ed it to every node — so every subscriber would receive N copies. `Context.System.EventStream` is node-local by construction, which is exactly the needed semantics, needs no registration handshake with actors that are spawned later (`ScriptedAlarmHostActor` is a **child of**`DriverHostActor` and has no registry key), and survives the Phase 6 mesh split unchanged.
```csharp
/// <summary>
/// Node-side end of the central↔node boundary, registered with the
/// <c>ClusterClientReceptionist</c> at <c>/user/node-communication</c> — <b>per node, not as a
/// singleton</b>, so contact rotation reaches whichever node answers.
/// </summary>
/// <remarks>
/// <para>
/// Inbound commands are re-emitted on <see cref="EventStream"/> rather than on their
/// DistributedPubSub topic. DPS is mesh-wide: since central <c>SendToAll</c>s to every node's
/// comm actor, a DPS re-publish would deliver N copies to every subscriber. The EventStream
/// is node-local by construction.
/// </para>
/// <para>
/// Outbound is <see cref="ApplyAck"/> only. There is no Ask across this boundary in Phase 2 —
/// every migrated command is fire-and-forget, and the deploy ack returns as an independent
/// message the coordinator already subscribes for.
Handlers: `DispatchDeployment`, `RestartDriver`, `ReconnectDriver`, `AlarmCommand` → `Context.System.EventStream.Publish(msg)`; `ApplyAck` → `_centralClient.Tell(new ClusterClient.Send(CentralCommunicationPath, ack))`, or Warning + drop when `_centralClient is null`.
**Tests:** each inbound type reaches an EventStream subscriber probe exactly once; `ApplyAck` produces a `ClusterClient.Send` (not `SendToAll` — the coordinator is one singleton) with the right path; a null client logs a Warning and drops.
**Sabotage-verify:** delete the `AlarmCommand` handler — its test must go RED (guards against a handler being added to the plan but not the actor).
**Commit**
```bash
git commit -m "feat(mesh): NodeCommunicationActor — inbound commands to the node-local EventStream"
```
---
## Task 5: Node-side subscribers accept both transports
**Classification:** standard
**Estimated implement time:** ~3 min
**Parallelizable with:** none
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs:476-485` and `:2410-2424`
**Subscribe to both, unconditionally — no mode plumbing on the node's subscribers.** In `Dps` mode nothing publishes to the EventStream; in `ClusterClient` mode nothing publishes to the topic. Adding both means the dark switch touches **publishers only**, which is what keeps it cheap to flip and cheap to revert.
In `DriverHostActor.PreStart`, after the existing three `Subscribe` calls:
```csharp
// Phase 2 dark switch: under MeshTransport:Mode = ClusterClient these arrive from
// NodeCommunicationActor on the node-local EventStream instead of the mesh-wide DPS topic.
// Subscribing to both is safe and deliberate — exactly one of the two ever publishes, so
// the transport flag lives entirely on the publishing side.
Same shape in `ScriptedAlarmHostActor.PreStart` for `typeof(AlarmCommand)`.
**Rename `_coordinatorOverride` → `_ackRouter`** (field, ctor parameter `coordinator:` → `ackRouter:`, and the `Props` parameter). Under ClusterClient mode this ref is the `NodeCommunicationActor`, not a coordinator, and the old name would actively mislead the next reader. The `SendAck` branch at `:2413` needs no logic change — only the comment:
```csharp
if(_ackRouterisnotnull)
{
// Either a direct coordinator handle (tests) or, under MeshTransport:Mode =
// ClusterClient, the NodeCommunicationActor that relays this across the boundary.
_ackRouter.Tell(ack);
}
```
Update the call site at `Runtime/ServiceCollectionExtensions.cs:~403` (`coordinator: null`) and every test constructing `DriverHostActor.Props`.
**Test:** publish a `DispatchDeployment` on the EventStream and assert the driver host acts on it; ditto `AlarmCommand` → `ScriptedAlarmHostActor`.
Node side, in `WithOtOpcUaRuntimeActors`, **after**`driverHost` is spawned: build the node's ClusterClient from `MeshTransport:CentralContactPoints` (only when `Mode = ClusterClient`), spawn `NodeCommunicationActor`, register it with the receptionist, and pass it as `ackRouter:` to `DriverHostActor.Props`.
Then thread the router into the two singletons (Task 7 uses it): `ConfigPublishCoordinator.Props(dbFactory, applyDeadline, meshRouter)` and `AdminOperationsActor.Props(..., meshRouter)`.
**Pin the paths.**`/user/central-communication` and `/user/node-communication` are the wire contract and nothing else asserts them — a rename would compile, deploy, and silently deliver nothing:
`_meshRouter` must be **nullable with a mediator fallback**, so the ~12 existing tests that call `ConfigPublishCoordinator.Props(dbFactory)` and `AdminOperationsActor.Props(...)` keep compiling and keep exercising the DPS path:
```csharp
// Null in tests and on any node wired before Phase 2: fall back to the mediator, which is
// exactly the pre-Phase-2 behaviour. The fallback is not a permanent seam — Phase 6 deletes it
**Test:** with a router probe injected, dispatching a deployment yields `MeshCommand(DeploymentsTopic, DispatchDeployment)` at the probe and **nothing** on the mediator; with no router, the mediator still sees the `Publish`.
**Sabotage-verify:** make `Dispatch` always take the mediator branch — the router test must go RED. (Phase 1 shipped a test that passed with the production mapping deleted; assume nothing here is load-bearing until you have watched it fail.)
**Commit**
```bash
git commit -m "feat(mesh): route deploy, driver-control and alarm commands through the mesh router"
**This is the deliberate divergence from the sister project, and the most valuable task in the phase.** They have **no** test that puts two real ActorSystems on either side of a ClusterClient — they substitute a 6-line in-process `ClusterClientRelay` that unwraps `ClusterClient.Send` and forwards. That relay tests actor *logic* and cannot fail for any transport reason: not a missing receptionist extension, not `SendToAll` behaving unlike `Send`, not a malformed contact path, not a serialization break. Those are precisely the Phase 2 failure modes.
Build two genuinely separate systems — same ActorSystem name `otopcua` (required: Akka.Remote address matching means a ClusterClient cannot reach a differently-named system), **non-overlapping seed lists**, each self-seeded so neither joins the other:
```csharp
// Two clusters, one ActorSystem name, joined ONLY by ClusterClient. If this test ever starts
// passing because the two systems gossiped into one mesh, it is testing nothing — assert the
1.`SendToAll` from central reaches the node's registered comm actor and lands on the node's EventStream.
2. An `ApplyAck` sent from the node reaches central's comm actor.
3.**Falsifiability control** — with the receptionist extension removed from the node's config, (1) must NOT arrive within the timeout. Without this control the test cannot distinguish "the boundary works" from "something else delivered it".
Use `AwaitAssert` with explicit timeouts throughout; no `Thread.Sleep`.
> The Phase 1 lesson that earns this task: *"the bad thing didn't happen"* is not evidence the mechanism works. Assertion 3 is what converts assertions 1–2 from a hope into a measurement.
If this test proves irreducibly flaky, **do not delete it and do not weaken it into a relay test** — mark it `[Trait("Category","MeshBoundary")]`, exclude it from the default run, and record the flakiness in the live-gate doc as an open item. A quarantined honest test beats a green vacuous one.
**Commit**
```bash
git commit -m "test(mesh): real two-ActorSystem ClusterClient boundary test with falsifiability control"
```
---
## Task 9: Rig config + docs
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 8
**Files:**
- Modify: `docker-dev/docker-compose.yml` (add `MeshTransport__Mode` + `MeshTransport__CentralContactPoints__0/1` to all six nodes; leave `Mode=Dps` so the rig comes up on the old path)
- Modify: `docs/Configuration.md` (new `MeshTransport` section)
- Modify: `docs/Redundancy.md` (the command boundary)
- Modify: `docs/plans/2026-07-21-per-cluster-mesh-design.md` — **correct §2's typed-failure claim** and add a "Phase 2 as shipped" note
- Modify: `docs/plans/2026-07-22-per-cluster-mesh-program.md` — record the no-cross-boundary-Ask gate deviation; flip the Phase 2 row
- Modify: `CLAUDE.md` — a short "Mesh command transport" note beside the Redundancy section
**Run every step and record the actual output, including anything that fails.** Phase 1's gate found a defect at step 4 that steps 1–3 had convincingly hidden; the value is entirely in running the steps that prove *recovery*, not the ones that prove the happy path.
| # | Step | Pass condition |
|---|---|---|
| 1 | Rig up with `Mode=Dps`; deploy via `POST :9200/api/deployments` (`X-Api-Key: docker-dev-deploy-key`) | Seals green — baseline on the old transport |
| 2 | Flip all six nodes to `Mode=ClusterClient`, rebuild, redeploy | Seals green; coordinator logs the same ack count |
| 3 | Confirm the transport actually changed | `docker logs` shows the `SendToAll`/receptionist path; **no**`deployments` DPS publish |
| 4 | Stop one node of the site-a pair, deploy | Fails at the apply deadline **naming that node** (Phase 1 semantics), not a hang |
| 5 | Set that node's `MaintenanceMode = 1`, redeploy | Seals green without it |
| 6 | AdminUI Restart + Reconnect on a driver | Reaches the owning node; UI reports "dispatched" |
| 7 | AdminUI alarm ack on `/alerts` | Reaches the owning node's engine |
| 8 | Stop **both** central nodes, then start a driver node | Node boots; ack is dropped with a Warning and **not** buffered; no crash loop |
| 9 | Restart central, redeploy | Recovers with no operator action beyond the redeploy |
| 10 | Grep every node's logs for frame-size warnings | None (the deploy path is payload-free — this is the canary that stays quiet) |
Step 8 is the one that proves the `buffer-size = 0` decision. If a command *does* arrive late after central returns, buffering is still on somewhere and the DB is no longer the single record of truth.
The rig's AdminUI runs with `Security__Auth__DisableLogin=true` — **drive steps 6 and 7 yourself via browser automation at `http://localhost:9200`; do not defer them as needing the user to sign in.**
- **`SendToAll` vs `Send`** is a one-word difference between "every node deploys" and "one node deploys and the rest silently keep their old config, while the deployment seals green on partial acks". Pinned by a sabotage-verified test in Task 3.
- **The single-mesh duplicate trap** turns "one client per cluster" — the obviously-right-looking shape, and the one Phase 6 wants — into N× delivery today.
- **`buffer-size = 0` is a behaviour choice, not a tuning knob.** If someone later "fixes" it back to Akka's default to make a flaky test pass, they reintroduce apply-after-timeout divergence between node and DB.
- **The AdminUI's `Ok = true` already means "dispatched", not "applied".** Phase 2 moves that lie to a new transport without making it worse; do not let a reviewer believe Phase 2 introduced it.
- **`redundancy-state` is bidirectional and carries two unrelated message types** (`RedundancyStateChanged` central→node, `OpcUaProbeResult` node→node on the same topic). It is explicitly **out of scope** here — but note `RedundancyStateActor` builds its snapshot from `Cluster.State`, so it is genuinely mesh-bound and is the hardest thing standing between here and Phase 6.
- **`alerts` has a node-local subscriber** (`HistorianAdapterActor`), so Phase 5 cannot simply move it to gRPC without keeping a node-local leg.
"note":"Per-cluster mesh Phase 2. Decisions pre-made: config-flagged dark switch (MeshTransport:Mode), buffer-size=0, alarm-commands central leg in scope. Three design-doc corrections and the single-mesh duplicate-delivery trap are documented in the plan header — read them before Task 3.",
{"id":6,"subject":"Task 6: wire both sides, receptionist registration, pin the wire-contract actor paths","status":"pending","classification":"standard","blockedBy":[3,5]},
{"id":7,"subject":"Task 7: route the five publish sites through the mesh router (nullable, mediator fallback)","status":"pending","classification":"standard","blockedBy":[6]},
{"id":8,"subject":"Task 8: real two-ActorSystem ClusterClient boundary test + falsifiability control","status":"pending","classification":"high-risk","blockedBy":[7],"parallelizableWith":[9]},
{"id":10,"subject":"Task 10: live gate on docker-dev (10 steps; step 4 and step 8 are the ones that matter)","status":"pending","classification":"high-risk","blockedBy":[8,9]}
| Prereq: selfform-fallback + manual-failover plan | plan written 2026-07-22, not executed |
| 1 ClusterNode columns + DB ack set | **detailed plan ALREADY WRITTEN** (`2026-07-21-per-cluster-mesh-phase1.md`, from the design session — do NOT write a new one), not executed. ⚠ Carries a decision gate: DB-derived expected-ack set changes deploy-seal semantics (a stopped-but-enabled node now FAILS the deploy at the apply deadline instead of sealing green without it; escape hatch = `ClusterNode.Enabled=0`) — confirm with the owner before its Task 3. |
| 2 ClusterClient transport | not started |
| Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. |
| 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
| 2 ClusterClient transport | **IN PROGRESS** — plan being written 2026-07-22 |
"note":"PROGRAM plan: each task = write that phase's detailed plan (writing-plans), execute it (executing-plans), run its exit gate, update the tracking tables. Prereq task 0 is the separate selfform-fallback plan in this repo.",
{"id":1,"subject":"Phase 1: ClusterNode Akka+gRPC address columns; ConfigPublishCoordinator ack set from DB","status":"pending"},
{"id":2,"subject":"Phase 2: comm actors + receptionist + ClusterClient transport (deploy notify/acks, driver-control)","status":"pending","blockedBy":[1]},
{"id":0,"subject":"Prereq: execute 2026-07-22-selfform-fallback-and-manual-failover.md (7 tasks)","status":"completed","note":"Shipped as self-first seed ordering + AkkaClusterOptionsValidator, NOT the planned SelfFormAfter watchdog (live gate caught it islanding a failed-over node). Merged a78425ea."},
{"id":1,"subject":"Phase 1: ClusterNode Akka+gRPC address columns; ConfigPublishCoordinator ack set from DB","status":"completed","note":"Merged 7654f24d. Escape hatch is ClusterNode.MaintenanceMode, not Enabled=0."},
{"id":2,"subject":"Phase 2: comm actors + receptionist + ClusterClient transport (deploy notify/acks, driver-control)","status":"in_progress","blockedBy":[1]},
{"id":3,"subject":"Phase 3: config fetch-and-cache from central; LocalDb steady-state config store (live gate)","status":"pending","blockedBy":[2]},
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Joseph Doherty