# Per-Cluster Mesh Phase 2 — Comm Actors + ClusterClient Transport

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

**Goal:** Move the three central→node command channels (`deployments`, `driver-control`, `alarm-commands`) and the `deployment-acks` reply channel off DistributedPubSub and onto Akka **ClusterClient**, behind a config flag, so central can command a cluster it does not share a gossip ring with.

**Architecture:** One receptionist-registered comm actor per side — `/user/central-communication` (on admin nodes) and `/user/node-communication` (on driver nodes), each registered **per node, not as a singleton**, so contact rotation reaches whichever answers. Central's singletons stop touching the mediator: they `Tell` a `MeshCommand` envelope to a node-local router actor that decides DPS or ClusterClient from `MeshTransport:Mode`. Inbound commands land on the node's **EventStream** — genuinely node-local, unlike DPS — and existing subscribers add one `EventStream.Subscribe` line beside their existing DPS `Subscribe`, so the dark switch touches only publishers.

**Tech Stack:** .NET 10, Akka.NET 1.5.62 (`Akka.Cluster.Tools` already referenced by Cluster, Runtime, and ControlPlane — no new package), xUnit + Shouldly, Akka.TestKit.Xunit2.

---

## Decisions already made (do not re-litigate)

| Decision | Choice | Why |
|---|---|---|
| Cutover | **Config-flagged dark switch** (`MeshTransport:Mode` = `Dps` default \| `ClusterClient`) | Lets the rig gate A/B the same deploy and roll back with a config change, not a revert. Phase 1's live gate is the argument: the defect only appeared against real infrastructure. |
| Buffering | **`buffer-size = 0`** (drop immediately) | Akka's default buffers 1000 and replays on reconnect. With Phase 1 semantics a deploy recorded `TimedOut` would still be applied minutes later when the node returns — the node runs a config the DB says failed. Fail-fast keeps the DB the single record of truth. **CORRECTED by the live gate ([finding B](2026-07-22-mesh-phase2-live-gate.md)):** this justification was overstated. A node applies a `TimedOut` deployment *anyway*, via `DriverHostActor.Bootstrap()` replaying the orphan `Applying` row on restart (live-observed at +44 s). `buffer-size = 0` removes the *silent, timer-driven* replay, not the boot-time DB one. The decision stands; the rationale narrowed. |
| `alarm-commands` central leg | **In scope** | Same shape as driver-control; leaves no half-migrated command plane. The topic stays alive for the node-local publisher. |

---

## Three corrections to the design doc, found during recon

These change what "copy ScadaBridge verbatim" means. Fold them into the code comments as you go.

1. **`docs/plans/2026-07-21-per-cluster-mesh-design.md` §2 is wrong about typed failure replies.** It says *"Every unhandled message replies with a typed failure rather than dropping."* ScadaBridge has **no `ReceiveAny` / `Unhandled` override in either comm actor.** The typed-failure idiom is per-message-type and fires only when a *registration* is missing (no ClusterClient yet, no handler registered). A genuinely unknown message type still dead-letters. Do not build a catch-all failure reply believing it is the sister project's pattern — Task 9 corrects the design doc.

2. **"No central buffering" was never a setting they wrote.** ScadaBridge has *zero* `akka.cluster.client.*` HOCON; production runs Akka defaults (`buffer-size = 1000`, `reconnect-timeout = off`). Their "no buffering" is app-level only: no ClusterClient entry for a site ⇒ drop + warn. We are choosing `buffer-size = 0` deliberately (Task 1), which they did not.

3. **`Publish` → `Send` is the wrong substitution; it must be `SendToAll`.** Today's deploy notify is a DPS broadcast that **every** `DriverHostActor` receives — there is no ClusterId or node filter anywhere on the node side (`DriverHostActor.cs:476-485`); scoping happens later, inside the artifact (`DeploymentArtifact.ResolveClusterScope`, `DriverHostActor.cs:1801`). `ClusterClient.Send` delivers to exactly **one** registered actor and would silently deploy to one node of the fleet.

---

## The single-mesh duplicate-delivery trap — read before Task 3

Phase 2 ships while the fleet is **still one Akka mesh**. A `ClusterClientReceptionist` serves its whole cluster, so `SendToAll("/user/node-communication")` reaches every registered node-comm actor **in the entire mesh**, regardless of which node's address was used as the contact point.

Therefore central must create **exactly ONE ClusterClient** in Phase 2, with contact points drawn from all enabled non-maintenance `ClusterNode` rows. Creating one client *per application Cluster* — which is what Phase 6 will want, and what ScadaBridge does — would fan the same command out once per cluster, producing N× duplicate `DispatchDeployment` and N× duplicate `ApplyAck`.

That is not a bug to fix later; on the single mesh the fleet-wide client is *correct*, and it produces exactly today's DPS fan-out. Phase 6 splits the meshes and converts this to per-cluster clients. Mark the code with `TODO(Phase 6)`.

---

## Deviation from the program plan's exit gate

`docs/plans/2026-07-22-per-cluster-mesh-program.md` §Phase 2 asks the gate to include *"an Ask timing out cleanly against a stopped node."*

**There is no cross-boundary Ask in Phase 2, so that gate item cannot be run as written.** Every command being migrated is fire-and-forget on the wire:

- `DispatchDeployment` — `Tell`; the ack returns later as a separate `ApplyAck` message, not an Ask reply (`ConfigPublishCoordinator.cs:111` subscribes to a topic).
- `RestartDriver` / `ReconnectDriver` — `AdminOperationsActor.cs:397,427` publish, then reply `Ok = true` immediately from the admin node. `Ok` means *dispatched*, not *applied*; the AdminUI string is literally "Restart dispatched".
- `AlarmCommand` — same, `AdminOperationsActor.cs:88,135`.

The honest equivalents, both of which Task 10's gate runs:

- **a stopped node** ⇒ the deploy fails at the apply deadline naming that node (Phase 1 semantics), not an Ask timeout;
- **no ClusterClient / no reachable contact** ⇒ the command is dropped with a Warning and never buffered, and the AdminUI still reports "dispatched" (a pre-existing lie, now on a new transport — record it, do not fix it here).

Task 9 records this deviation in the program plan.

---

## Task 0: `MeshTransportOptions` + validator

**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 1

**Files:**
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptionsValidator.cs`
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs:32-40` (`AddOtOpcUaCluster`)
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportOptionsValidatorTests.cs`

`ZB.MOM.WW.OtOpcUa.Cluster` is referenced by both ControlPlane (`ControlPlane.csproj:23`) and Runtime, so this is the right home — next to `AkkaClusterOptions`.

**Step 1: Write the failing tests**

```csharp
// tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportOptionsValidatorTests.cs
using Microsoft.Extensions.Options;
using Shouldly;
using ZB.MOM.WW.OtOpcUa.Cluster;

namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;

public class MeshTransportOptionsValidatorTests
{
    private static ValidateOptionsResult Validate(MeshTransportOptions o) =>
        new MeshTransportOptionsValidator().Validate(MeshTransportOptions.SectionName, o);

    [Fact]
    public void Default_options_are_valid()
    {
        Validate(new MeshTransportOptions()).Succeeded.ShouldBeTrue();
    }

    [Fact]
    public void Unknown_mode_fails()
    {
        var result = Validate(new MeshTransportOptions { Mode = "grpc" });
        result.Failed.ShouldBeTrue();
        result.FailureMessage.ShouldContain("grpc");
    }

    [Fact]
    public void ClusterClient_mode_requires_central_contact_points()
    {
        var result = Validate(new MeshTransportOptions
        {
            Mode = "ClusterClient",
            CentralContactPoints = [],
        });
        result.Failed.ShouldBeTrue();
        result.FailureMessage.ShouldContain(nameof(MeshTransportOptions.CentralContactPoints));
    }

    [Fact]
    public void A_contact_point_that_is_not_an_akka_address_fails()
    {
        var result = Validate(new MeshTransportOptions
        {
            Mode = "ClusterClient",
            CentralContactPoints = ["central-1:4053"],   // missing akka.tcp:// scheme
        });
        result.Failed.ShouldBeTrue();
        result.FailureMessage.ShouldContain("akka.tcp://");
    }

    // The ScadaBridge footgun, ported. Their shipped site template listed the site's OWN
    // remoting port as the second central contact — "a permanent failure in the initial-contact
    // rotation" (appsettings.Site.json:45). Catch it at boot instead of in an incident.
    [Fact]
    public void A_contact_point_carrying_an_actor_path_suffix_fails()
    {
        var result = Validate(new MeshTransportOptions
        {
            Mode = "ClusterClient",
            CentralContactPoints = ["akka.tcp://otopcua@central-1:4053/system/receptionist"],
        });
        result.Failed.ShouldBeTrue();
        result.FailureMessage.ShouldContain("/system/receptionist");
    }

    [Fact]
    public void Well_formed_ClusterClient_options_succeed()
    {
        Validate(new MeshTransportOptions
        {
            Mode = "ClusterClient",
            CentralContactPoints =
            [
                "akka.tcp://otopcua@central-1:4053",
                "akka.tcp://otopcua@central-2:4053",
            ],
        }).Succeeded.ShouldBeTrue();
    }
}
```

**Step 2: Run to verify they fail**

Run: `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~MeshTransportOptionsValidatorTests"`
Expected: FAIL to compile — `MeshTransportOptions` does not exist.

**Step 3: Write `MeshTransportOptions.cs`**

```csharp
namespace ZB.MOM.WW.OtOpcUa.Cluster;

/// <summary>
///     Selects and configures the transport carrying central→node commands and node→central
///     deployment acks (per-cluster mesh Phase 2).
/// </summary>
/// <remarks>
///     <para>
///         The <see cref="Mode"/> flag is a deliberate dark switch, not a permanent knob: both
///         transports are live in the tree for one phase so the docker-dev rig can run the same
///         deployment over each and compare, and so a bad Phase 2 rolls back with a config change
///         rather than a revert. Phase 6 deletes the DPS branch along with the single mesh.
///     </para>
///     <para>
///         Discovery is deliberately asymmetric, matching the sister project: <b>central discovers
///         nodes from the database</b> (enabled, non-maintenance <c>ClusterNode</c> rows, refreshed
///         on a timer), <b>nodes discover central from this section</b> — static, restart to change.
///         Central's set changes as operators add nodes; central's own address does not.
///     </para>
/// </remarks>
public sealed class MeshTransportOptions
{
    public const string SectionName = "MeshTransport";

    public const string ModeDps = "Dps";
    public const string ModeClusterClient = "ClusterClient";

    /// <summary>
    ///     <c>Dps</c> (default) keeps every command on DistributedPubSub; <c>ClusterClient</c> routes
    ///     it over the receptionist boundary. Any other value fails the host at startup.
    /// </summary>
    public string Mode { get; set; } = ModeDps;

    /// <summary>
    ///     Akka remoting addresses of the central nodes, e.g.
    ///     <c>akka.tcp://otopcua@central-1:4053</c>. Node addresses only — the
    ///     <c>/system/receptionist</c> path is appended at construction time.
    /// </summary>
    /// <remarks>
    ///     Each entry MUST be a <b>central</b> node's remoting endpoint, never this node's own port.
    ///     The sister project shipped a template whose second contact was the site's own port; it is
    ///     a permanent failure in the initial-contact rotation and is invisible until a command goes
    ///     missing. <see cref="MeshTransportOptionsValidator"/> rejects the malformed shapes at boot.
    /// </remarks>
    public string[] CentralContactPoints { get; set; } = [];

    /// <summary>How often central re-reads <c>ClusterNode</c> rows to rebuild its contact points.</summary>
    public int ContactRefreshSeconds { get; set; } = 60;
}
```

**Step 4: Write `MeshTransportOptionsValidator.cs`**

```csharp
using Microsoft.Extensions.Options;

namespace ZB.MOM.WW.OtOpcUa.Cluster;

/// <summary>
///     Fails the host at startup on a <see cref="MeshTransportOptions"/> shape that would leave
///     commands silently undelivered. Every failure here is one that otherwise surfaces as an
///     absence — a deploy that never arrives, an alarm ack that does nothing — which is the hardest
///     class of fault to attribute in the field.
/// </summary>
public sealed class MeshTransportOptionsValidator : IValidateOptions<MeshTransportOptions>
{
    public ValidateOptionsResult Validate(string? name, MeshTransportOptions options)
    {
        ArgumentNullException.ThrowIfNull(options);
        var errors = new List<string>();

        var isDps = string.Equals(options.Mode, MeshTransportOptions.ModeDps, StringComparison.OrdinalIgnoreCase);
        var isClusterClient = string.Equals(
            options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);

        if (!isDps && !isClusterClient)
        {
            errors.Add(
                $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)} is "
                + $"'{options.Mode}'. Expected '{MeshTransportOptions.ModeDps}' or "
                + $"'{MeshTransportOptions.ModeClusterClient}'.");
        }

        if (options.ContactRefreshSeconds <= 0)
        {
            errors.Add(
                $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.ContactRefreshSeconds)} "
                + $"must be > 0 (was {options.ContactRefreshSeconds}).");
        }

        if (isClusterClient)
        {
            if (options.CentralContactPoints.Length == 0)
            {
                errors.Add(
                    $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.CentralContactPoints)} "
                    + "is empty. Under ClusterClient mode a driver node cannot reach central without at "
                    + "least one contact point, and its deployment acks would be dropped silently.");
            }

            foreach (var cp in options.CentralContactPoints)
            {
                if (!cp.StartsWith("akka.tcp://", StringComparison.Ordinal))
                {
                    errors.Add($"Contact point '{cp}' must start with 'akka.tcp://'.");
                }
                else if (cp.Contains("/user/", StringComparison.Ordinal)
                         || cp.Contains("/system/receptionist", StringComparison.Ordinal))
                {
                    errors.Add(
                        $"Contact point '{cp}' carries an actor-path suffix. Configure the node "
                        + "address only — '/system/receptionist' is appended at construction time.");
                }
            }
        }

        return errors.Count == 0
            ? ValidateOptionsResult.Success
            : ValidateOptionsResult.Fail(string.Join(" ", errors));
    }
}
```

**Step 5: Register it**

In `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs`, inside `AddOtOpcUaCluster`, directly after the existing `AddValidatedOptions<AkkaClusterOptions, …>` call:

```csharp
        services.AddValidatedOptions<MeshTransportOptions, MeshTransportOptionsValidator>(
            configuration, MeshTransportOptions.SectionName);
```

**Step 6: Run the tests**

Run: `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~MeshTransportOptionsValidatorTests"`
Expected: PASS, 6/6.

**Step 7: Commit**

```bash
git add src/Core/ZB.MOM.WW.OtOpcUa.Cluster tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests
git commit -m "feat(mesh): MeshTransportOptions — dark switch between DPS and ClusterClient"
```

---

## Task 1: Receptionist extension, zero buffering, frame-size logging

**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 0

**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:16-33`
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportHoconTests.cs`

**The test must read the effective config off a running `ActorSystem`, never the HOCON text.** This is a settled lesson in this repo (`SplitBrainResolverActivationTests`, and the comment at `ServiceCollectionExtensions.cs:96-99`): several fragments compete and the losing one still reads correctly in the file.

**Step 1: Write the failing test**

```csharp
// tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportHoconTests.cs
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Shouldly;
using ZB.MOM.WW.OtOpcUa.Cluster;

namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;

/// <summary>
///     Asserts the ClusterClient / receptionist / frame-size settings that are in force on a REAL
///     ActorSystem built from the shipped base config. Asserting the akka.conf text instead would
///     pass while a competing fragment silently overrode the value.
/// </summary>
public class MeshTransportHoconTests : IDisposable
{
    private readonly ActorSystem _sys;

    public MeshTransportHoconTests()
    {
        var config = Akka.Configuration.ConfigurationFactory
            .ParseString("akka.remote.dot-netty.tcp.port = 0\nakka.cluster.seed-nodes = []")
            .WithFallback(HoconLoader.LoadBaseConfig());
        _sys = ActorSystem.Create("hocon-probe", config);
    }

    [Fact]
    public void Cluster_client_buffering_is_disabled()
    {
        // buffer-size = 0 means "drop immediately if the receptionist is unknown". Akka's default
        // of 1000 would replay a DispatchDeployment minutes later, applying a config the DB has
        // already recorded as TimedOut.
        _sys.Settings.Config.GetInt("akka.cluster.client.buffer-size").ShouldBe(0);
    }

    [Fact]
    public void Receptionist_serves_every_member_role()
    {
        // role = "" is what makes "registered per node, not as a singleton" work: every member
        // hosts a receptionist, so contact rotation reaches whichever node answers.
        _sys.Settings.Config.GetString("akka.cluster.client.receptionist.role").ShouldBe(string.Empty);
    }

    [Fact]
    public void Oversized_frames_are_logged_rather_than_dropped_silently()
    {
        // The sister project's most expensive incident: a message over the 128 KB frame limit is
        // dropped WITHOUT tearing down the association — heartbeats keep flowing, the node still
        // reports healthy, and the only symptom is an absence. log-frame-size-exceeding turns that
        // into a log line with the offending size.
        _sys.Settings.Config.GetByteSize("akka.remote.dot-netty.tcp.maximum-frame-size").ShouldBe(128000);
        _sys.Settings.Config.GetByteSize("akka.remote.dot-netty.tcp.log-frame-size-exceeding").ShouldBe(32000);
    }

    [Fact]
    public void Receptionist_extension_is_registered()
    {
        // Not auto-started: without the extensions entry the receptionist only materialises if some
        // code path happens to call ClusterClientReceptionist.Get(system) first.
        _sys.Settings.Config.GetStringList("akka.extensions")
            .ShouldContain(s => s.Contains("ClusterClientReceptionistExtensionProvider"));
    }

    public void Dispose() => _sys.Terminate().Wait(TimeSpan.FromSeconds(5));
}
```

**Step 2: Run to verify it fails**

Run: `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~MeshTransportHoconTests"`
Expected: FAIL — `buffer-size` reads 1000, `log-frame-size-exceeding` is unset, extension missing.

**Step 3: Edit `akka.conf`**

Replace the `extensions` block (lines 16-18) with:

```hocon
    extensions = [
        "Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider, Akka.Cluster.Tools"
        # Per-cluster mesh Phase 2. NOT auto-started: without this entry the receptionist only
        # materialises if something happens to call ClusterClientReceptionist.Get(system) first,
        # which makes "is the boundary listening?" depend on startup ordering.
        "Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"
    ]
```

Add to the `remote.dot-netty.tcp` block (after `port = 4053`):

```hocon
            # Explicit rather than inherited. Both values ARE the Akka defaults for maximum-frame-size;
            # what changes is log-frame-size-exceeding, which defaults to `off`. Over the limit the
            # transport drops that ONE message without tearing down the association: heartbeats keep
            # flowing, the node still reports healthy, and the only symptom is a command that never
            # arrived. The sister project lost a deploy path to exactly this and could not attribute
            # it — see docs/plans/2026-07-21-per-cluster-mesh-design.md §8.
            maximum-frame-size = 128000b
            log-frame-size-exceeding = 32000b
```

Add a new block inside `akka.cluster` (after the `pub-sub`-adjacent settings, before `singleton`):

```hocon
        # ClusterClient / receptionist — the central↔node command boundary (Phase 2).
        client {
            # DELIBERATE, and NOT what the sister project runs (they never wrote this section, so
            # they inherit buffer-size = 1000). Zero means: if the receptionist is unknown, drop the
            # message now. Buffering would replay a DispatchDeployment on reconnect and apply a
            # deployment that ConfigPublishCoordinator already sealed as TimedOut — the node would
            # then be running a configuration the database says failed. Fail-fast keeps the DB the
            # single record of truth, and matches today's DPS behaviour exactly (a node that is down
            # misses the publish and self-heals from the DB on restart).
            buffer-size = 0

            # Retry forever rather than self-stopping the client. A central node that is down for an
            # hour must not require a driver-node restart to become reachable again.
            reconnect-timeout = off

            receptionist {
                # Empty = every member hosts a receptionist. This is what makes the comm actors
                # "registered per node, not as a singleton" work — contact rotation reaches whichever
                # node answers. Do not scope this to a role.
                role = ""
            }
        }
```

**Step 4: Run the tests**

Run: `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~MeshTransportHoconTests"`
Expected: PASS, 4/4.

**Step 5: Sabotage-verify**

Change `buffer-size = 0` to `buffer-size = 1000`, re-run: `Cluster_client_buffering_is_disabled` must go RED. Restore, re-run, green.

> **MSBuild mtime trap (cost several cycles in Phase 1):** `akka.conf` is an **embedded resource**. If you restore a file with `mv` or `git checkout` its mtime may land older than the build output and MSBuild will skip recompiling, so the test keeps reading the sabotaged value. After restoring, run `touch src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf` and rebuild.

**Step 6: Commit**

```bash
git add src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportHoconTests.cs
git commit -m "feat(mesh): receptionist extension, zero ClusterClient buffering, frame-size logging"
```

---

## Task 2: `MeshCommand` envelope + ClusterClient factory

**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (Task 3 depends on it)

**Files:**
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Mesh/MeshCommand.cs`
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/IMeshClusterClientFactory.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/MeshClusterClientFactoryTests.cs`

**Step 1: Write `MeshCommand.cs`**

```csharp
namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;

/// <summary>
///     A central→node command plus the DistributedPubSub topic it would have been published on.
/// </summary>
/// <param name="Topic">
///     The legacy DPS topic. Carried so the router can publish under <c>MeshTransport:Mode = Dps</c>
///     without a second dispatch table; ignored entirely under ClusterClient mode, where the
///     destination is an actor path rather than a topic.
/// </param>
/// <param name="Message">The command itself — never wrapped, so node-side handlers are unchanged.</param>
/// <remarks>
///     The envelope exists so the admin singletons stop knowing which transport is in force. Before
///     Phase 2 each publisher held its own <c>DistributedPubSub.Get(Context.System).Mediator</c>
///     call, which meant the dark switch would have had to be threaded through four call sites in
///     two actors. It is central-side only and never crosses the wire.
/// </remarks>
public sealed record MeshCommand(string Topic, object Message);
```

**Step 2: Write the factory**

```csharp
// src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/IMeshClusterClientFactory.cs
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;

namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;

/// <summary>Creates the ClusterClient central uses to reach driver nodes. Seam for tests.</summary>
public interface IMeshClusterClientFactory
{
    IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts);
}

/// <inheritdoc />
public sealed class DefaultMeshClusterClientFactory : IMeshClusterClientFactory
{
    /// <summary>
    ///     Per-incarnation generation counter. <c>Context.Stop</c> of the previous client is
    ///     asynchronous — its actor name stays reserved until termination completes — so recreating
    ///     the same name while handling one message throws <see cref="InvalidActorNameException"/>.
    ///     A generation suffix makes every incarnation's name unique by construction. Ported from the
    ///     sister project, where it was a shipped bug fix rather than foresight.
    /// </summary>
    private long _generation;

    public IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts)
    {
        ArgumentNullException.ThrowIfNull(system);
        var settings = ClusterClientSettings.Create(system).WithInitialContacts(contacts);
        var name = $"mesh-client-{Interlocked.Increment(ref _generation)}";
        return system.ActorOf(ClusterClient.Props(settings), name);
    }
}
```

**Step 3: Write the test**

```csharp
// tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/MeshClusterClientFactoryTests.cs
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Configuration;
using Akka.TestKit.Xunit2;
using Shouldly;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;

namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;

public class MeshClusterClientFactoryTests : TestKit
{
    private static readonly Config TestConfig = ConfigurationFactory.ParseString(@"
        akka.actor.provider = cluster
        akka.remote.dot-netty.tcp.port = 0
        akka.remote.dot-netty.tcp.hostname = 127.0.0.1")
        .WithFallback(ClusterClientReceptionist.DefaultConfig());

    public MeshClusterClientFactoryTests() : base(TestConfig) { }

    // A real ClusterClient against an unreachable contact is safe in TestKit — it just retries.
    [Fact]
    public void Recreating_a_client_in_one_pass_does_not_collide_on_the_actor_name()
    {
        var factory = new DefaultMeshClusterClientFactory();
        var contacts = ImmutableHashSet.Create(
            ActorPath.Parse("akka.tcp://otopcua@127.0.0.1:65501/system/receptionist"));

        var first = factory.Create(Sys, contacts);
        Sys.Stop(first);
        // Stop is async: the name is still reserved here. Without the generation suffix this throws.
        var second = factory.Create(Sys, contacts);

        second.Path.Name.ShouldNotBe(first.Path.Name);
    }
}
```

**Step 4: Run**

Run: `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests --filter "FullyQualifiedName~MeshClusterClientFactoryTests"`
Expected: PASS.

**Step 5: Sabotage-verify** — drop the `-{generation}` suffix from the name; the test must go RED with `InvalidActorNameException`. Restore.

**Step 6: Commit**

```bash
git add src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Mesh src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication
git commit -m "feat(mesh): MeshCommand envelope + generation-suffixed ClusterClient factory"
```

---

## Task 3: `CentralCommunicationActor`

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 4

**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/CentralCommunicationActorTests.cs`

Responsibilities, in order of importance:

1. `Receive<MeshCommand>` — under `Dps`, `mediator.Tell(new Publish(cmd.Topic, cmd.Message))`; under `ClusterClient`, `client.Tell(new ClusterClient.SendToAll(NodeCommunicationPath, cmd.Message))`. **`SendToAll`, never `Send`** — see the corrections section.
2. `Receive<ApplyAck>` — inbound from a node's ClusterClient; `Forward` to the `ConfigPublishCoordinatorKey` singleton proxy so `Sender` survives the hop.
3. Contact-point cache — rebuild from enabled non-maintenance `ClusterNode` rows on a `ContactRefreshSeconds` timer and on demand; **exactly one** client (single-mesh trap above).
4. `Receive<Status.Failure>` — a faulted DB load piped back. Without this handler the refresh fails at Debug level and an operator cannot distinguish "no nodes configured" from "the database is down".

Key implementation notes to put in the code:

```csharp
    /// <summary>Path the node-side comm actor registers under. This string is the wire contract.</summary>
    public const string NodeCommunicationPath = "/user/node-communication";

    // TODO(Phase 6): one client per application Cluster. Today ONE client, fleet-wide contacts.
    // A ClusterClientReceptionist serves its whole cluster, and the fleet is still a single mesh,
    // so SendToAll from ANY contact point reaches every registered node-comm actor in the mesh.
    // One client per Cluster would therefore fan each command out once per cluster — N x duplicate
    // DispatchDeployment and N x duplicate ApplyAck. On the single mesh the fleet-wide client is
    // not a compromise: it reproduces today's DPS fan-out exactly.
```

Dropping when unroutable (the app-level "no buffering" decision — Akka's own buffer is already 0 from Task 1):

```csharp
        if (_client is null)
        {
            _log.Warning(
                "No ClusterClient — dropping {MessageType}. No enabled ClusterNode rows produced a "
                + "usable contact point at the last refresh; the command is NOT buffered and will "
                + "not be retried",
                cmd.Message.GetType().Name);
            return;
        }
```

Address parsing — per-row try/catch so one malformed row cannot abort the whole refresh and leave the cache half-built (a shipped sister-project bug, with a regression test that deliberately orders the bad row first):

```csharp
            foreach (var (nodeId, host, akkaPort) in rows)
            {
                try
                {
                    paths.Add(ActorPath.Parse($"akka.tcp://{systemName}@{host}:{akkaPort}/system/receptionist"));
                }
                catch (Exception ex)
                {
                    _log.Warning(ex,
                        "ClusterNode {NodeId} has an unusable address {Host}:{Port}; skipping it in "
                        + "this refresh (other nodes are unaffected)", nodeId, host, akkaPort);
                }
            }
```

**Tests (all must exist):**

| Test | Asserts |
|---|---|
| `Dps_mode_publishes_on_the_carried_topic` | mediator sees `Publish(topic, msg)`; no client created |
| `ClusterClient_mode_sends_to_all_not_to_one` | probe receives `ClusterClient.SendToAll` with `Path == "/user/node-communication"` — **assert the type is `SendToAll`, not merely that something was sent** |
| `Exactly_one_client_is_created_for_a_multi_cluster_fleet` | seed `ClusterNode` rows across two `ServerCluster`s; factory invoked once |
| `ApplyAck_is_forwarded_to_the_coordinator_preserving_sender` | coordinator probe receives `ApplyAck` with `LastSender` == the original |
| `A_malformed_node_row_does_not_abort_the_refresh` | bad row ordered **first**; the good row still yields a contact |
| `MeshCommand_with_no_client_is_dropped_with_a_warning` | `EventFilter.Warning(contains: "dropping").ExpectOne(...)` |
| `A_faulted_db_load_logs_a_warning` | `Status.Failure` → Warning, actor still alive |

Use `AwaitAssert`/`ExpectMsg` with explicit timeouts. **Do not use `Thread.Sleep`** — the sister project's older comm-actor tests do, and it is their documented flake source.

**Sabotage-verify:** change `SendToAll` to `Send`; `ClusterClient_mode_sends_to_all_not_to_one` must go RED. This is the single most important assertion in the phase — `Send` would deploy to one node of the fleet and every other node would silently keep its old config.

**Commit**

```bash
git commit -m "feat(mesh): CentralCommunicationActor — dark-switched fan-out, DB-sourced contacts"
```

---

## Task 4: `NodeCommunicationActor`

**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** Task 3

**Files:**
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Communication/NodeCommunicationActor.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Communication/NodeCommunicationActorTests.cs`

**Why inbound commands go to the `EventStream` and not back onto DPS.** Re-publishing an inbound command onto its DPS topic would fan it across the *whole mesh* — and central has already `SendToAll`-ed it to every node — so every subscriber would receive N copies. `Context.System.EventStream` is node-local by construction, which is exactly the needed semantics, needs no registration handshake with actors that are spawned later (`ScriptedAlarmHostActor` is a **child of** `DriverHostActor` and has no registry key), and survives the Phase 6 mesh split unchanged.

```csharp
/// <summary>
///     Node-side end of the central↔node boundary, registered with the
///     <c>ClusterClientReceptionist</c> at <c>/user/node-communication</c> — <b>per node, not as a
///     singleton</b>, so contact rotation reaches whichever node answers.
/// </summary>
/// <remarks>
///     <para>
///         Inbound commands are re-emitted on <see cref="EventStream"/> rather than on their
///         DistributedPubSub topic. DPS is mesh-wide: since central <c>SendToAll</c>s to every node's
///         comm actor, a DPS re-publish would deliver N copies to every subscriber. The EventStream
///         is node-local by construction.
///     </para>
///     <para>
///         Outbound is <see cref="ApplyAck"/> only. There is no Ask across this boundary in Phase 2 —
///         every migrated command is fire-and-forget, and the deploy ack returns as an independent
///         message the coordinator already subscribes for.
///     </para>
/// </remarks>
public sealed class NodeCommunicationActor : ReceiveActor
{
    public const string ActorName = "node-communication";
    public const string CentralCommunicationPath = "/user/central-communication";
```

Handlers: `DispatchDeployment`, `RestartDriver`, `ReconnectDriver`, `AlarmCommand` → `Context.System.EventStream.Publish(msg)`; `ApplyAck` → `_centralClient.Tell(new ClusterClient.Send(CentralCommunicationPath, ack))`, or Warning + drop when `_centralClient is null`.

**Tests:** each inbound type reaches an EventStream subscriber probe exactly once; `ApplyAck` produces a `ClusterClient.Send` (not `SendToAll` — the coordinator is one singleton) with the right path; a null client logs a Warning and drops.

**Sabotage-verify:** delete the `AlarmCommand` handler — its test must go RED (guards against a handler being added to the plan but not the actor).

**Commit**

```bash
git commit -m "feat(mesh): NodeCommunicationActor — inbound commands to the node-local EventStream"
```

---

## Task 5: Node-side subscribers accept both transports

**Classification:** standard
**Estimated implement time:** ~3 min
**Parallelizable with:** none

**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs:476-485` and `:2410-2424`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs:522-530`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostTransportSubscriptionTests.cs`

**Subscribe to both, unconditionally — no mode plumbing on the node's subscribers.** In `Dps` mode nothing publishes to the EventStream; in `ClusterClient` mode nothing publishes to the topic. Adding both means the dark switch touches **publishers only**, which is what keeps it cheap to flip and cheap to revert.

In `DriverHostActor.PreStart`, after the existing three `Subscribe` calls:

```csharp
        // Phase 2 dark switch: under MeshTransport:Mode = ClusterClient these arrive from
        // NodeCommunicationActor on the node-local EventStream instead of the mesh-wide DPS topic.
        // Subscribing to both is safe and deliberate — exactly one of the two ever publishes, so
        // the transport flag lives entirely on the publishing side.
        Context.System.EventStream.Subscribe(Self, typeof(DispatchDeployment));
        Context.System.EventStream.Subscribe(Self, typeof(RestartDriver));
        Context.System.EventStream.Subscribe(Self, typeof(ReconnectDriver));
```

Same shape in `ScriptedAlarmHostActor.PreStart` for `typeof(AlarmCommand)`.

**Rename `_coordinatorOverride` → `_ackRouter`** (field, ctor parameter `coordinator:` → `ackRouter:`, and the `Props` parameter). Under ClusterClient mode this ref is the `NodeCommunicationActor`, not a coordinator, and the old name would actively mislead the next reader. The `SendAck` branch at `:2413` needs no logic change — only the comment:

```csharp
        if (_ackRouter is not null)
        {
            // Either a direct coordinator handle (tests) or, under MeshTransport:Mode =
            // ClusterClient, the NodeCommunicationActor that relays this across the boundary.
            _ackRouter.Tell(ack);
        }
```

Update the call site at `Runtime/ServiceCollectionExtensions.cs:~403` (`coordinator: null`) and every test constructing `DriverHostActor.Props`.

**Test:** publish a `DispatchDeployment` on the EventStream and assert the driver host acts on it; ditto `AlarmCommand` → `ScriptedAlarmHostActor`.

**Commit**

```bash
git commit -m "feat(mesh): node subscribers accept EventStream commands; _coordinatorOverride -> _ackRouter"
```

---

## Task 6: Wire both sides + pin the actor paths

**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none

**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:~380-421`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshCommActorPathTests.cs`

Central side, in `WithOtOpcUaControlPlaneSingletons` — a plain `WithActors`, **not** a singleton:

```csharp
        // Per node, NOT a cluster singleton: a node's ClusterClient rotates across contact points
        // and must find a live comm actor at whichever central node answers. A singleton would be
        // reachable only through the one node hosting it, and the rotation would silently fail
        // against the other.
        builder.WithActors((system, registry, resolver) =>
        {
            var actor = system.ActorOf(CentralCommunicationActor.Props(...), CentralCommunicationActor.ActorName);
            ClusterClientReceptionist.Get(system).RegisterService(actor);
            registry.Register<CentralCommunicationActorKey>(actor);
        });
```

Node side, in `WithOtOpcUaRuntimeActors`, **after** `driverHost` is spawned: build the node's ClusterClient from `MeshTransport:CentralContactPoints` (only when `Mode = ClusterClient`), spawn `NodeCommunicationActor`, register it with the receptionist, and pass it as `ackRouter:` to `DriverHostActor.Props`.

Then thread the router into the two singletons (Task 7 uses it): `ConfigPublishCoordinator.Props(dbFactory, applyDeadline, meshRouter)` and `AdminOperationsActor.Props(..., meshRouter)`.

**Pin the paths.** `/user/central-communication` and `/user/node-communication` are the wire contract and nothing else asserts them — a rename would compile, deploy, and silently deliver nothing:

```csharp
    private async Task AssertActorExists(ActorSystem system, string path)
    {
        var identity = await system.ActorSelection(path)
            .Ask<ActorIdentity>(new Identify(path), TimeSpan.FromSeconds(5));
        identity.Subject.ShouldNotBeNull($"{path} is the ClusterClient wire contract");
    }
```

**Sabotage-verify:** rename the actor to `node-comm`; the path test must go RED.

**Commit**

```bash
git commit -m "feat(mesh): register comm actors with the receptionist; pin the wire-contract paths"
```

---

## Task 7: Route the publishers through the router

**Classification:** standard
**Estimated implement time:** ~4 min
**Parallelizable with:** none

**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs:78-101,175`
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs:88,135,397,427`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ConfigPublishCoordinatorTransportTests.cs`

Five publish sites become one call each. `ConfigPublishCoordinator.cs:175`:

```csharp
        _meshRouter.Tell(new MeshCommand(DeploymentsTopic, msg));
```

`AdminOperationsActor.cs:88` / `:135` / `:397` / `:427` likewise, with `AlarmCommandsTopic.Name` / `DriverControlTopic.Name`.

`_meshRouter` must be **nullable with a mediator fallback**, so the ~12 existing tests that call `ConfigPublishCoordinator.Props(dbFactory)` and `AdminOperationsActor.Props(...)` keep compiling and keep exercising the DPS path:

```csharp
    // Null in tests and on any node wired before Phase 2: fall back to the mediator, which is
    // exactly the pre-Phase-2 behaviour. The fallback is not a permanent seam — Phase 6 deletes it
    // with the DPS branch.
    private void Dispatch(string topic, object message)
    {
        if (_meshRouter is not null) _meshRouter.Tell(new MeshCommand(topic, message));
        else DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(topic, message));
    }
```

**Test:** with a router probe injected, dispatching a deployment yields `MeshCommand(DeploymentsTopic, DispatchDeployment)` at the probe and **nothing** on the mediator; with no router, the mediator still sees the `Publish`.

**Sabotage-verify:** make `Dispatch` always take the mediator branch — the router test must go RED. (Phase 1 shipped a test that passed with the production mapping deleted; assume nothing here is load-bearing until you have watched it fail.)

**Commit**

```bash
git commit -m "feat(mesh): route deploy, driver-control and alarm commands through the mesh router"
```

---

## Task 8: Real two-ActorSystem boundary test

**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** none

**Files:**
- Create: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshClusterClientBoundaryTests.cs`

**This is the deliberate divergence from the sister project, and the most valuable task in the phase.** They have **no** test that puts two real ActorSystems on either side of a ClusterClient — they substitute a 6-line in-process `ClusterClientRelay` that unwraps `ClusterClient.Send` and forwards. That relay tests actor *logic* and cannot fail for any transport reason: not a missing receptionist extension, not `SendToAll` behaving unlike `Send`, not a malformed contact path, not a serialization break. Those are precisely the Phase 2 failure modes.

Build two genuinely separate systems — same ActorSystem name `otopcua` (required: Akka.Remote address matching means a ClusterClient cannot reach a differently-named system), **non-overlapping seed lists**, each self-seeded so neither joins the other:

```csharp
    // Two clusters, one ActorSystem name, joined ONLY by ClusterClient. If this test ever starts
    // passing because the two systems gossiped into one mesh, it is testing nothing — assert the
    // separation explicitly.
    Cluster.Get(_central).State.Members.Count.ShouldBe(1);
    Cluster.Get(_node).State.Members.Count.ShouldBe(1);
```

Assertions:

1. `SendToAll` from central reaches the node's registered comm actor and lands on the node's EventStream.
2. An `ApplyAck` sent from the node reaches central's comm actor.
3. **Falsifiability control** — with the receptionist extension removed from the node's config, (1) must NOT arrive within the timeout. Without this control the test cannot distinguish "the boundary works" from "something else delivered it".

Use `AwaitAssert` with explicit timeouts throughout; no `Thread.Sleep`.

> The Phase 1 lesson that earns this task: *"the bad thing didn't happen"* is not evidence the mechanism works. Assertion 3 is what converts assertions 1–2 from a hope into a measurement.

> **Scope limit — what this test can NOT see:** with exactly one node per side, "registered per
> node" and "registered as a singleton" are indistinguishable by construction, so this test does
> NOT cover the per-node property whose structural assertions Task 6 dropped (both discriminators
> were unfalsifiable — see the dead-end record there). That property's discriminating coverage is
> **live-gate step 8** (stop the OLDEST central; the survivor's comm actor must receive a node ack
> inside the downing window — a singleton comm actor, hosted on the stopped oldest, cannot). Do not
> close the phase with step 8 skipped: it is the honest replacement for the dropped Task 6 test.

If this test proves irreducibly flaky, **do not delete it and do not weaken it into a relay test** — mark it `[Trait("Category","MeshBoundary")]`, exclude it from the default run, and record the flakiness in the live-gate doc as an open item. A quarantined honest test beats a green vacuous one.

**Commit**

```bash
git commit -m "test(mesh): real two-ActorSystem ClusterClient boundary test with falsifiability control"
```

---

## Task 9: Rig config + docs

**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 8

**Files:**
- Modify: `docker-dev/docker-compose.yml` (add `MeshTransport__Mode` + `MeshTransport__CentralContactPoints__0/1` to all six nodes; leave `Mode=Dps` so the rig comes up on the old path)
- Modify: `docs/Configuration.md` (new `MeshTransport` section)
- Modify: `docs/Redundancy.md` (the command boundary)
- Modify: `docs/plans/2026-07-21-per-cluster-mesh-design.md` — **correct §2's typed-failure claim** and add a "Phase 2 as shipped" note
- Modify: `docs/plans/2026-07-22-per-cluster-mesh-program.md` — record the no-cross-boundary-Ask gate deviation; flip the Phase 2 row
- Modify: `CLAUDE.md` — a short "Mesh command transport" note beside the Redundancy section

**Commit**

```bash
git commit -m "docs(mesh): MeshTransport config, Phase 2 as shipped, design-doc correction"
```

---

## Task 10: Live gate on docker-dev

**Classification:** high-risk
**Estimated implement time:** ~5 min (plus rig time)
**Parallelizable with:** none

**Files:**
- Create: `docs/plans/2026-07-22-mesh-phase2-live-gate.md`

**Run every step and record the actual output, including anything that fails.** Phase 1's gate found a defect at step 4 that steps 1–3 had convincingly hidden; the value is entirely in running the steps that prove *recovery*, not the ones that prove the happy path.

| # | Step | Pass condition |
|---|---|---|
| 1 | Rig up with `Mode=Dps`; deploy via `POST :9200/api/deployments` (`X-Api-Key: docker-dev-deploy-key`) | Seals green — baseline on the old transport |
| 2 | Flip all six nodes to `Mode=ClusterClient`, rebuild, redeploy | Seals green; coordinator logs the same ack count |
| 3 | Confirm the transport actually changed | `docker logs` shows the `SendToAll`/receptionist path; **no** `deployments` DPS publish |
| 4 | Stop one node of the site-a pair, deploy | Fails at the apply deadline **naming that node** (Phase 1 semantics), not a hang |
| 5 | Set that node's `MaintenanceMode = 1`, redeploy | Seals green without it |
| 6 | AdminUI Restart + Reconnect on a driver | Reaches the owning node; UI reports "dispatched" |
| 7 | AdminUI alarm ack on `/alerts` | Reaches the owning node's engine |
| 8 | **Contact rotation / the per-node discriminator.** Stop the **OLDEST** central node only (the singleton host — stopping the younger proves nothing, a singleton would also live on the still-up oldest), then immediately restart one driver node so it emits its boot-time ack | The **surviving** central's `central-communication` actor logs receipt of the ack **within seconds — inside the 15 s downing window**, before singleton migration could explain delivery. This is the discriminating test for "registered per node, NOT a singleton" — the property Task 6's structural assertions could not falsify (both dead ends recorded there): a singleton-shaped comm actor was hosted on the node just stopped, so nothing could receive on the survivor until downing + migration (~25 s). Receipt at the comm actor is the pass signal; the onward `Forward` to the coordinator singleton proxy is EXPECTED to buffer until the singleton migrates — do not count that against the step (full recovery is step 10's job) |
| 9 | Stop **both** central nodes, then start a driver node | Node boots; ack is dropped with a Warning and **not** buffered; no crash loop |
| 10 | Restart central, redeploy | Recovers with no operator action beyond the redeploy |
| 11 | Grep every node's logs for frame-size warnings | None (the deploy path is payload-free — this is the canary that stays quiet) |

Step 8 is the one that proves the `buffer-size = 0` decision. If a command *does* arrive late after central returns, buffering is still on somewhere and the DB is no longer the single record of truth.

The rig's AdminUI runs with `Security__Auth__DisableLogin=true` — **drive steps 6 and 7 yourself via browser automation at `http://localhost:9200`; do not defer them as needing the user to sign in.**

**Commit**

```bash
git commit -m "docs(mesh): Phase 2 live-gate record"
```

---

## Risks specific to this phase

- **`SendToAll` vs `Send`** is a one-word difference between "every node deploys" and "one node deploys and the rest silently keep their old config, while the deployment seals green on partial acks". Pinned by a sabotage-verified test in Task 3.
- **The single-mesh duplicate trap** turns "one client per cluster" — the obviously-right-looking shape, and the one Phase 6 wants — into N× delivery today.
- **`buffer-size = 0` is a behaviour choice, not a tuning knob.** If someone later "fixes" it back to Akka's default to make a flaky test pass, they reintroduce apply-after-timeout divergence between node and DB.
- **The AdminUI's `Ok = true` already means "dispatched", not "applied".** Phase 2 moves that lie to a new transport without making it worse; do not let a reviewer believe Phase 2 introduced it.
- **`redundancy-state` is bidirectional and carries two unrelated message types** (`RedundancyStateChanged` central→node, `OpcUaProbeResult` node→node on the same topic). It is explicitly **out of scope** here — but note `RedundancyStateActor` builds its snapshot from `Cluster.State`, so it is genuinely mesh-bound and is the hardest thing standing between here and Phase 6.
- **`alerts` has a node-local subscriber** (`HistorianAdapterActor`), so Phase 5 cannot simply move it to gRPC without keeping a node-local leg.

---

## Follow-ups found during execution (not fixed in Phase 2)

- **`DefaultMeshClusterClientFactory`'s generation counter is per factory instance, and its clients
  are top-level actors.** Production creates one factory per `CentralCommunicationActor` and captures
  it in the props closure, so the counter survives a supervision restart and names stay unique — that
  part is sound. What is *not* handled: the clients are `system.ActorOf`, not children, so a restart
  of the comm actor loses `_client` without stopping it and leaves an orphaned ClusterClient dialling
  receptionists forever. Harmless today (nothing sends to it) but it is a leak per restart, and it
  will matter in Phase 6 when there is one client per cluster. Found by
  `MeshClusterClientBoundaryTests`, which hit the name collision the moment a second factory was
  constructed on the same system.
