lmxopcua/docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md

Per-Cluster Mesh Phase 2 — Comm Actors + ClusterClient Transport

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

Goal: Move the three central→node command channels (deployments, driver-control, alarm-commands) and the deployment-acks reply channel off DistributedPubSub and onto Akka ClusterClient, behind a config flag, so central can command a cluster it does not share a gossip ring with.

Architecture: One receptionist-registered comm actor per side — /user/central-communication (on admin nodes) and /user/node-communication (on driver nodes), each registered per node, not as a singleton, so contact rotation reaches whichever answers. Central's singletons stop touching the mediator: they Tell a MeshCommand envelope to a node-local router actor that decides DPS or ClusterClient from MeshTransport:Mode. Inbound commands land on the node's EventStream — genuinely node-local, unlike DPS — and existing subscribers add one EventStream.Subscribe line beside their existing DPS Subscribe, so the dark switch touches only publishers.

Tech Stack: .NET 10, Akka.NET 1.5.62 (Akka.Cluster.Tools already referenced by Cluster, Runtime, and ControlPlane — no new package), xUnit + Shouldly, Akka.TestKit.Xunit2.

---

Decisions already made (do not re-litigate)

Decision	Choice	Why
Cutover	Config-flagged dark switch (MeshTransport:Mode = Dps default | ClusterClient)	Lets the rig gate A/B the same deploy and roll back with a config change, not a revert. Phase 1's live gate is the argument: the defect only appeared against real infrastructure.
Buffering	buffer-size = 0 (drop immediately)	Akka's default buffers 1000 and replays on reconnect. With Phase 1 semantics a deploy recorded TimedOut would still be applied minutes later when the node returns — the node runs a config the DB says failed. Fail-fast keeps the DB the single record of truth. CORRECTED by the live gate (finding B): this justification was overstated. A node applies a TimedOut deployment anyway, via DriverHostActor.Bootstrap() replaying the orphan Applying row on restart (live-observed at +44 s). buffer-size = 0 removes the silent, timer-driven replay, not the boot-time DB one. The decision stands; the rationale narrowed.
alarm-commands central leg	In scope	Same shape as driver-control; leaves no half-migrated command plane. The topic stays alive for the node-local publisher.

---

Three corrections to the design doc, found during recon

These change what "copy ScadaBridge verbatim" means. Fold them into the code comments as you go.

1. docs/plans/2026-07-21-per-cluster-mesh-design.md §2 is wrong about typed failure replies. It says "Every unhandled message replies with a typed failure rather than dropping." ScadaBridge has no ReceiveAny / Unhandled override in either comm actor. The typed-failure idiom is per-message-type and fires only when a registration is missing (no ClusterClient yet, no handler registered). A genuinely unknown message type still dead-letters. Do not build a catch-all failure reply believing it is the sister project's pattern — Task 9 corrects the design doc.

2. "No central buffering" was never a setting they wrote. ScadaBridge has zero akka.cluster.client.* HOCON; production runs Akka defaults (buffer-size = 1000, reconnect-timeout = off). Their "no buffering" is app-level only: no ClusterClient entry for a site ⇒ drop + warn. We are choosing buffer-size = 0 deliberately (Task 1), which they did not.

3. Publish → Send is the wrong substitution; it must be SendToAll. Today's deploy notify is a DPS broadcast that every DriverHostActor receives — there is no ClusterId or node filter anywhere on the node side (DriverHostActor.cs:476-485); scoping happens later, inside the artifact (DeploymentArtifact.ResolveClusterScope, DriverHostActor.cs:1801). ClusterClient.Send delivers to exactly one registered actor and would silently deploy to one node of the fleet.

---

The single-mesh duplicate-delivery trap — read before Task 3

Phase 2 ships while the fleet is still one Akka mesh. A ClusterClientReceptionist serves its whole cluster, so SendToAll("/user/node-communication") reaches every registered node-comm actor in the entire mesh, regardless of which node's address was used as the contact point.

Therefore central must create exactly ONE ClusterClient in Phase 2, with contact points drawn from all enabled non-maintenance ClusterNode rows. Creating one client per application Cluster — which is what Phase 6 will want, and what ScadaBridge does — would fan the same command out once per cluster, producing N× duplicate DispatchDeployment and N× duplicate ApplyAck.

That is not a bug to fix later; on the single mesh the fleet-wide client is correct, and it produces exactly today's DPS fan-out. Phase 6 splits the meshes and converts this to per-cluster clients. Mark the code with TODO(Phase 6).

---

Deviation from the program plan's exit gate

docs/plans/2026-07-22-per-cluster-mesh-program.md §Phase 2 asks the gate to include "an Ask timing out cleanly against a stopped node."

There is no cross-boundary Ask in Phase 2, so that gate item cannot be run as written. Every command being migrated is fire-and-forget on the wire:

• DispatchDeployment — Tell; the ack returns later as a separate ApplyAck message, not an Ask reply (ConfigPublishCoordinator.cs:111 subscribes to a topic).
• RestartDriver / ReconnectDriver — AdminOperationsActor.cs:397,427 publish, then reply Ok = true immediately from the admin node. Ok means dispatched, not applied; the AdminUI string is literally "Restart dispatched".
• AlarmCommand — same, AdminOperationsActor.cs:88,135.

The honest equivalents, both of which Task 10's gate runs:

• a stopped node ⇒ the deploy fails at the apply deadline naming that node (Phase 1 semantics), not an Ask timeout;
• no ClusterClient / no reachable contact ⇒ the command is dropped with a Warning and never buffered, and the AdminUI still reports "dispatched" (a pre-existing lie, now on a new transport — record it, do not fix it here).

Task 9 records this deviation in the program plan.

---

Task 0: MeshTransportOptions + validator

Classification: small Estimated implement time: ~4 min Parallelizable with: Task 1

Files:

• Create: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs
• Create: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptionsValidator.cs
• Modify: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs:32-40 (AddOtOpcUaCluster)
• Test: tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportOptionsValidatorTests.cs

ZB.MOM.WW.OtOpcUa.Cluster is referenced by both ControlPlane (ControlPlane.csproj:23) and Runtime, so this is the right home — next to AkkaClusterOptions.

Step 1: Write the failing tests

// tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportOptionsValidatorTests.cs
using Microsoft.Extensions.Options;
using Shouldly;
using ZB.MOM.WW.OtOpcUa.Cluster;

namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;

public class MeshTransportOptionsValidatorTests
{
    private static ValidateOptionsResult Validate(MeshTransportOptions o) =>
        new MeshTransportOptionsValidator().Validate(MeshTransportOptions.SectionName, o);

    [Fact]
    public void Default_options_are_valid()
    {
        Validate(new MeshTransportOptions()).Succeeded.ShouldBeTrue();
    }

    [Fact]
    public void Unknown_mode_fails()
    {
        var result = Validate(new MeshTransportOptions { Mode = "grpc" });
        result.Failed.ShouldBeTrue();
        result.FailureMessage.ShouldContain("grpc");
    }

    [Fact]
    public void ClusterClient_mode_requires_central_contact_points()
    {
        var result = Validate(new MeshTransportOptions
        {
            Mode = "ClusterClient",
            CentralContactPoints = [],
        });
        result.Failed.ShouldBeTrue();
        result.FailureMessage.ShouldContain(nameof(MeshTransportOptions.CentralContactPoints));
    }

    [Fact]
    public void A_contact_point_that_is_not_an_akka_address_fails()
    {
        var result = Validate(new MeshTransportOptions
        {
            Mode = "ClusterClient",
            CentralContactPoints = ["central-1:4053"],   // missing akka.tcp:// scheme
        });
        result.Failed.ShouldBeTrue();
        result.FailureMessage.ShouldContain("akka.tcp://");
    }

    // The ScadaBridge footgun, ported. Their shipped site template listed the site's OWN
    // remoting port as the second central contact — "a permanent failure in the initial-contact
    // rotation" (appsettings.Site.json:45). Catch it at boot instead of in an incident.
    [Fact]
    public void A_contact_point_carrying_an_actor_path_suffix_fails()
    {
        var result = Validate(new MeshTransportOptions
        {
            Mode = "ClusterClient",
            CentralContactPoints = ["akka.tcp://otopcua@central-1:4053/system/receptionist"],
        });
        result.Failed.ShouldBeTrue();
        result.FailureMessage.ShouldContain("/system/receptionist");
    }

    [Fact]
    public void Well_formed_ClusterClient_options_succeed()
    {
        Validate(new MeshTransportOptions
        {
            Mode = "ClusterClient",
            CentralContactPoints =
            [
                "akka.tcp://otopcua@central-1:4053",
                "akka.tcp://otopcua@central-2:4053",
            ],
        }).Succeeded.ShouldBeTrue();
    }
}

Step 2: Run to verify they fail

Run: dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~MeshTransportOptionsValidatorTests" Expected: FAIL to compile — MeshTransportOptions does not exist.

Step 3: Write MeshTransportOptions.cs

namespace ZB.MOM.WW.OtOpcUa.Cluster;

/// <summary>
///     Selects and configures the transport carrying central→node commands and node→central
///     deployment acks (per-cluster mesh Phase 2).
/// </summary>
/// <remarks>
///     <para>
///         The <see cref="Mode"/> flag is a deliberate dark switch, not a permanent knob: both
///         transports are live in the tree for one phase so the docker-dev rig can run the same
///         deployment over each and compare, and so a bad Phase 2 rolls back with a config change
///         rather than a revert. Phase 6 deletes the DPS branch along with the single mesh.
///     </para>
///     <para>
///         Discovery is deliberately asymmetric, matching the sister project: <b>central discovers
///         nodes from the database</b> (enabled, non-maintenance <c>ClusterNode</c> rows, refreshed
///         on a timer), <b>nodes discover central from this section</b> — static, restart to change.
///         Central's set changes as operators add nodes; central's own address does not.
///     </para>
/// </remarks>
public sealed class MeshTransportOptions
{
    public const string SectionName = "MeshTransport";

    public const string ModeDps = "Dps";
    public const string ModeClusterClient = "ClusterClient";

    /// <summary>
    ///     <c>Dps</c> (default) keeps every command on DistributedPubSub; <c>ClusterClient</c> routes
    ///     it over the receptionist boundary. Any other value fails the host at startup.
    /// </summary>
    public string Mode { get; set; } = ModeDps;

    /// <summary>
    ///     Akka remoting addresses of the central nodes, e.g.
    ///     <c>akka.tcp://otopcua@central-1:4053</c>. Node addresses only — the
    ///     <c>/system/receptionist</c> path is appended at construction time.
    /// </summary>
    /// <remarks>
    ///     Each entry MUST be a <b>central</b> node's remoting endpoint, never this node's own port.
    ///     The sister project shipped a template whose second contact was the site's own port; it is
    ///     a permanent failure in the initial-contact rotation and is invisible until a command goes
    ///     missing. <see cref="MeshTransportOptionsValidator"/> rejects the malformed shapes at boot.
    /// </remarks>
    public string[] CentralContactPoints { get; set; } = [];

    /// <summary>How often central re-reads <c>ClusterNode</c> rows to rebuild its contact points.</summary>
    public int ContactRefreshSeconds { get; set; } = 60;
}

Step 4: Write MeshTransportOptionsValidator.cs

using Microsoft.Extensions.Options;

namespace ZB.MOM.WW.OtOpcUa.Cluster;

/// <summary>
///     Fails the host at startup on a <see cref="MeshTransportOptions"/> shape that would leave
///     commands silently undelivered. Every failure here is one that otherwise surfaces as an
///     absence — a deploy that never arrives, an alarm ack that does nothing — which is the hardest
///     class of fault to attribute in the field.
/// </summary>
public sealed class MeshTransportOptionsValidator : IValidateOptions<MeshTransportOptions>
{
    public ValidateOptionsResult Validate(string? name, MeshTransportOptions options)
    {
        ArgumentNullException.ThrowIfNull(options);
        var errors = new List<string>();

        var isDps = string.Equals(options.Mode, MeshTransportOptions.ModeDps, StringComparison.OrdinalIgnoreCase);
        var isClusterClient = string.Equals(
            options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);

        if (!isDps && !isClusterClient)
        {
            errors.Add(
                $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)} is "
                + $"'{options.Mode}'. Expected '{MeshTransportOptions.ModeDps}' or "
                + $"'{MeshTransportOptions.ModeClusterClient}'.");
        }

        if (options.ContactRefreshSeconds <= 0)
        {
            errors.Add(
                $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.ContactRefreshSeconds)} "
                + $"must be > 0 (was {options.ContactRefreshSeconds}).");
        }

        if (isClusterClient)
        {
            if (options.CentralContactPoints.Length == 0)
            {
                errors.Add(
                    $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.CentralContactPoints)} "
                    + "is empty. Under ClusterClient mode a driver node cannot reach central without at "
                    + "least one contact point, and its deployment acks would be dropped silently.");
            }

            foreach (var cp in options.CentralContactPoints)
            {
                if (!cp.StartsWith("akka.tcp://", StringComparison.Ordinal))
                {
                    errors.Add($"Contact point '{cp}' must start with 'akka.tcp://'.");
                }
                else if (cp.Contains("/user/", StringComparison.Ordinal)
                         || cp.Contains("/system/receptionist", StringComparison.Ordinal))
                {
                    errors.Add(
                        $"Contact point '{cp}' carries an actor-path suffix. Configure the node "
                        + "address only — '/system/receptionist' is appended at construction time.");
                }
            }
        }

        return errors.Count == 0
            ? ValidateOptionsResult.Success
            : ValidateOptionsResult.Fail(string.Join(" ", errors));
    }
}

Step 5: Register it

In src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs, inside AddOtOpcUaCluster, directly after the existing AddValidatedOptions<AkkaClusterOptions, …> call:

        services.AddValidatedOptions<MeshTransportOptions, MeshTransportOptionsValidator>(
            configuration, MeshTransportOptions.SectionName);

Step 6: Run the tests

Run: dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~MeshTransportOptionsValidatorTests" Expected: PASS, 6/6.

Step 7: Commit

git add src/Core/ZB.MOM.WW.OtOpcUa.Cluster tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests
git commit -m "feat(mesh): MeshTransportOptions — dark switch between DPS and ClusterClient"

---

Task 1: Receptionist extension, zero buffering, frame-size logging

Classification: small Estimated implement time: ~4 min Parallelizable with: Task 0

Files:

• Modify: src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:16-33
• Test: tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportHoconTests.cs

The test must read the effective config off a running ActorSystem, never the HOCON text. This is a settled lesson in this repo (SplitBrainResolverActivationTests, and the comment at ServiceCollectionExtensions.cs:96-99): several fragments compete and the losing one still reads correctly in the file.

Step 1: Write the failing test

// tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportHoconTests.cs
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Shouldly;
using ZB.MOM.WW.OtOpcUa.Cluster;

namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;

/// <summary>
///     Asserts the ClusterClient / receptionist / frame-size settings that are in force on a REAL
///     ActorSystem built from the shipped base config. Asserting the akka.conf text instead would
///     pass while a competing fragment silently overrode the value.
/// </summary>
public class MeshTransportHoconTests : IDisposable
{
    private readonly ActorSystem _sys;

    public MeshTransportHoconTests()
    {
        var config = Akka.Configuration.ConfigurationFactory
            .ParseString("akka.remote.dot-netty.tcp.port = 0\nakka.cluster.seed-nodes = []")
            .WithFallback(HoconLoader.LoadBaseConfig());
        _sys = ActorSystem.Create("hocon-probe", config);
    }

    [Fact]
    public void Cluster_client_buffering_is_disabled()
    {
        // buffer-size = 0 means "drop immediately if the receptionist is unknown". Akka's default
        // of 1000 would replay a DispatchDeployment minutes later, applying a config the DB has
        // already recorded as TimedOut.
        _sys.Settings.Config.GetInt("akka.cluster.client.buffer-size").ShouldBe(0);
    }

    [Fact]
    public void Receptionist_serves_every_member_role()
    {
        // role = "" is what makes "registered per node, not as a singleton" work: every member
        // hosts a receptionist, so contact rotation reaches whichever node answers.
        _sys.Settings.Config.GetString("akka.cluster.client.receptionist.role").ShouldBe(string.Empty);
    }

    [Fact]
    public void Oversized_frames_are_logged_rather_than_dropped_silently()
    {
        // The sister project's most expensive incident: a message over the 128 KB frame limit is
        // dropped WITHOUT tearing down the association — heartbeats keep flowing, the node still
        // reports healthy, and the only symptom is an absence. log-frame-size-exceeding turns that
        // into a log line with the offending size.
        _sys.Settings.Config.GetByteSize("akka.remote.dot-netty.tcp.maximum-frame-size").ShouldBe(128000);
        _sys.Settings.Config.GetByteSize("akka.remote.dot-netty.tcp.log-frame-size-exceeding").ShouldBe(32000);
    }

    [Fact]
    public void Receptionist_extension_is_registered()
    {
        // Not auto-started: without the extensions entry the receptionist only materialises if some
        // code path happens to call ClusterClientReceptionist.Get(system) first.
        _sys.Settings.Config.GetStringList("akka.extensions")
            .ShouldContain(s => s.Contains("ClusterClientReceptionistExtensionProvider"));
    }

    public void Dispose() => _sys.Terminate().Wait(TimeSpan.FromSeconds(5));
}

Step 2: Run to verify it fails

Run: dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~MeshTransportHoconTests" Expected: FAIL — buffer-size reads 1000, log-frame-size-exceeding is unset, extension missing.

Step 3: Edit akka.conf

Replace the extensions block (lines 16-18) with:

    extensions = [
        "Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider, Akka.Cluster.Tools"
        # Per-cluster mesh Phase 2. NOT auto-started: without this entry the receptionist only
        # materialises if something happens to call ClusterClientReceptionist.Get(system) first,
        # which makes "is the boundary listening?" depend on startup ordering.
        "Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"
    ]

Add to the remote.dot-netty.tcp block (after port = 4053):

            # Explicit rather than inherited. Both values ARE the Akka defaults for maximum-frame-size;
            # what changes is log-frame-size-exceeding, which defaults to `off`. Over the limit the
            # transport drops that ONE message without tearing down the association: heartbeats keep
            # flowing, the node still reports healthy, and the only symptom is a command that never
            # arrived. The sister project lost a deploy path to exactly this and could not attribute
            # it — see docs/plans/2026-07-21-per-cluster-mesh-design.md §8.
            maximum-frame-size = 128000b
            log-frame-size-exceeding = 32000b

Add a new block inside akka.cluster (after the pub-sub-adjacent settings, before singleton):

        # ClusterClient / receptionist — the central↔node command boundary (Phase 2).
        client {
            # DELIBERATE, and NOT what the sister project runs (they never wrote this section, so
            # they inherit buffer-size = 1000). Zero means: if the receptionist is unknown, drop the
            # message now. Buffering would replay a DispatchDeployment on reconnect and apply a
            # deployment that ConfigPublishCoordinator already sealed as TimedOut — the node would
            # then be running a configuration the database says failed. Fail-fast keeps the DB the
            # single record of truth, and matches today's DPS behaviour exactly (a node that is down
            # misses the publish and self-heals from the DB on restart).
            buffer-size = 0

            # Retry forever rather than self-stopping the client. A central node that is down for an
            # hour must not require a driver-node restart to become reachable again.
            reconnect-timeout = off

            receptionist {
                # Empty = every member hosts a receptionist. This is what makes the comm actors
                # "registered per node, not as a singleton" work — contact rotation reaches whichever
                # node answers. Do not scope this to a role.
                role = ""
            }
        }

Step 4: Run the tests

Run: dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests --filter "FullyQualifiedName~MeshTransportHoconTests" Expected: PASS, 4/4.

Step 5: Sabotage-verify

Change buffer-size = 0 to buffer-size = 1000, re-run: Cluster_client_buffering_is_disabled must go RED. Restore, re-run, green.

MSBuild mtime trap (cost several cycles in Phase 1): akka.conf is an embedded resource. If you restore a file with mv or git checkout its mtime may land older than the build output and MSBuild will skip recompiling, so the test keeps reading the sabotaged value. After restoring, run touch src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf and rebuild.

Step 6: Commit

git add src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportHoconTests.cs
git commit -m "feat(mesh): receptionist extension, zero ClusterClient buffering, frame-size logging"

---

Task 2: MeshCommand envelope + ClusterClient factory

Classification: small Estimated implement time: ~4 min Parallelizable with: none (Task 3 depends on it)

Files:

• Create: src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Mesh/MeshCommand.cs
• Create: src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/IMeshClusterClientFactory.cs
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/MeshClusterClientFactoryTests.cs

Step 1: Write MeshCommand.cs

namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;

/// <summary>
///     A central→node command plus the DistributedPubSub topic it would have been published on.
/// </summary>
/// <param name="Topic">
///     The legacy DPS topic. Carried so the router can publish under <c>MeshTransport:Mode = Dps</c>
///     without a second dispatch table; ignored entirely under ClusterClient mode, where the
///     destination is an actor path rather than a topic.
/// </param>
/// <param name="Message">The command itself — never wrapped, so node-side handlers are unchanged.</param>
/// <remarks>
///     The envelope exists so the admin singletons stop knowing which transport is in force. Before
///     Phase 2 each publisher held its own <c>DistributedPubSub.Get(Context.System).Mediator</c>
///     call, which meant the dark switch would have had to be threaded through four call sites in
///     two actors. It is central-side only and never crosses the wire.
/// </remarks>
public sealed record MeshCommand(string Topic, object Message);

Step 2: Write the factory

// src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/IMeshClusterClientFactory.cs
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;

namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;

/// <summary>Creates the ClusterClient central uses to reach driver nodes. Seam for tests.</summary>
public interface IMeshClusterClientFactory
{
    IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts);
}

/// <inheritdoc />
public sealed class DefaultMeshClusterClientFactory : IMeshClusterClientFactory
{
    /// <summary>
    ///     Per-incarnation generation counter. <c>Context.Stop</c> of the previous client is
    ///     asynchronous — its actor name stays reserved until termination completes — so recreating
    ///     the same name while handling one message throws <see cref="InvalidActorNameException"/>.
    ///     A generation suffix makes every incarnation's name unique by construction. Ported from the
    ///     sister project, where it was a shipped bug fix rather than foresight.
    /// </summary>
    private long _generation;

    public IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts)
    {
        ArgumentNullException.ThrowIfNull(system);
        var settings = ClusterClientSettings.Create(system).WithInitialContacts(contacts);
        var name = $"mesh-client-{Interlocked.Increment(ref _generation)}";
        return system.ActorOf(ClusterClient.Props(settings), name);
    }
}

Step 3: Write the test

// tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/MeshClusterClientFactoryTests.cs
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Configuration;
using Akka.TestKit.Xunit2;
using Shouldly;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;

namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;

public class MeshClusterClientFactoryTests : TestKit
{
    private static readonly Config TestConfig = ConfigurationFactory.ParseString(@"
        akka.actor.provider = cluster
        akka.remote.dot-netty.tcp.port = 0
        akka.remote.dot-netty.tcp.hostname = 127.0.0.1")
        .WithFallback(ClusterClientReceptionist.DefaultConfig());

    public MeshClusterClientFactoryTests() : base(TestConfig) { }

    // A real ClusterClient against an unreachable contact is safe in TestKit — it just retries.
    [Fact]
    public void Recreating_a_client_in_one_pass_does_not_collide_on_the_actor_name()
    {
        var factory = new DefaultMeshClusterClientFactory();
        var contacts = ImmutableHashSet.Create(
            ActorPath.Parse("akka.tcp://otopcua@127.0.0.1:65501/system/receptionist"));

        var first = factory.Create(Sys, contacts);
        Sys.Stop(first);
        // Stop is async: the name is still reserved here. Without the generation suffix this throws.
        var second = factory.Create(Sys, contacts);

        second.Path.Name.ShouldNotBe(first.Path.Name);
    }
}

Step 4: Run

Run: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests --filter "FullyQualifiedName~MeshClusterClientFactoryTests" Expected: PASS.

Step 5: Sabotage-verify — drop the -{generation} suffix from the name; the test must go RED with InvalidActorNameException. Restore.

Step 6: Commit

git add src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Mesh src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication
git commit -m "feat(mesh): MeshCommand envelope + generation-suffixed ClusterClient factory"

---

Task 3: CentralCommunicationActor

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: Task 4

Files:

• Create: src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/CentralCommunicationActorTests.cs

Responsibilities, in order of importance:

1. Receive<MeshCommand> — under Dps, mediator.Tell(new Publish(cmd.Topic, cmd.Message)); under ClusterClient, client.Tell(new ClusterClient.SendToAll(NodeCommunicationPath, cmd.Message)). SendToAll, never Send — see the corrections section.
2. Receive<ApplyAck> — inbound from a node's ClusterClient; Forward to the ConfigPublishCoordinatorKey singleton proxy so Sender survives the hop.
3. Contact-point cache — rebuild from enabled non-maintenance ClusterNode rows on a ContactRefreshSeconds timer and on demand; exactly one client (single-mesh trap above).
4. Receive<Status.Failure> — a faulted DB load piped back. Without this handler the refresh fails at Debug level and an operator cannot distinguish "no nodes configured" from "the database is down".

Key implementation notes to put in the code:

    /// <summary>Path the node-side comm actor registers under. This string is the wire contract.</summary>
    public const string NodeCommunicationPath = "/user/node-communication";

    // TODO(Phase 6): one client per application Cluster. Today ONE client, fleet-wide contacts.
    // A ClusterClientReceptionist serves its whole cluster, and the fleet is still a single mesh,
    // so SendToAll from ANY contact point reaches every registered node-comm actor in the mesh.
    // One client per Cluster would therefore fan each command out once per cluster — N x duplicate
    // DispatchDeployment and N x duplicate ApplyAck. On the single mesh the fleet-wide client is
    // not a compromise: it reproduces today's DPS fan-out exactly.

Dropping when unroutable (the app-level "no buffering" decision — Akka's own buffer is already 0 from Task 1):

        if (_client is null)
        {
            _log.Warning(
                "No ClusterClient — dropping {MessageType}. No enabled ClusterNode rows produced a "
                + "usable contact point at the last refresh; the command is NOT buffered and will "
                + "not be retried",
                cmd.Message.GetType().Name);
            return;
        }

Address parsing — per-row try/catch so one malformed row cannot abort the whole refresh and leave the cache half-built (a shipped sister-project bug, with a regression test that deliberately orders the bad row first):

            foreach (var (nodeId, host, akkaPort) in rows)
            {
                try
                {
                    paths.Add(ActorPath.Parse($"akka.tcp://{systemName}@{host}:{akkaPort}/system/receptionist"));
                }
                catch (Exception ex)
                {
                    _log.Warning(ex,
                        "ClusterNode {NodeId} has an unusable address {Host}:{Port}; skipping it in "
                        + "this refresh (other nodes are unaffected)", nodeId, host, akkaPort);
                }
            }

Tests (all must exist):

Test	Asserts
Dps_mode_publishes_on_the_carried_topic	mediator sees Publish(topic, msg); no client created
ClusterClient_mode_sends_to_all_not_to_one	probe receives ClusterClient.SendToAll with Path == "/user/node-communication" — assert the type is SendToAll, not merely that something was sent
Exactly_one_client_is_created_for_a_multi_cluster_fleet	seed ClusterNode rows across two ServerClusters; factory invoked once
ApplyAck_is_forwarded_to_the_coordinator_preserving_sender	coordinator probe receives ApplyAck with LastSender == the original
A_malformed_node_row_does_not_abort_the_refresh	bad row ordered first; the good row still yields a contact
MeshCommand_with_no_client_is_dropped_with_a_warning	EventFilter.Warning(contains: "dropping").ExpectOne(...)
A_faulted_db_load_logs_a_warning	Status.Failure → Warning, actor still alive

Use AwaitAssert/ExpectMsg with explicit timeouts. Do not use Thread.Sleep — the sister project's older comm-actor tests do, and it is their documented flake source.

Sabotage-verify: change SendToAll to Send; ClusterClient_mode_sends_to_all_not_to_one must go RED. This is the single most important assertion in the phase — Send would deploy to one node of the fleet and every other node would silently keep its old config.

Commit

git commit -m "feat(mesh): CentralCommunicationActor — dark-switched fan-out, DB-sourced contacts"

---

Task 4: NodeCommunicationActor

Classification: standard Estimated implement time: ~4 min Parallelizable with: Task 3

Files:

• Create: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Communication/NodeCommunicationActor.cs
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Communication/NodeCommunicationActorTests.cs

Why inbound commands go to the EventStream and not back onto DPS. Re-publishing an inbound command onto its DPS topic would fan it across the whole mesh — and central has already SendToAll-ed it to every node — so every subscriber would receive N copies. Context.System.EventStream is node-local by construction, which is exactly the needed semantics, needs no registration handshake with actors that are spawned later (ScriptedAlarmHostActor is a child of DriverHostActor and has no registry key), and survives the Phase 6 mesh split unchanged.

/// <summary>
///     Node-side end of the central↔node boundary, registered with the
///     <c>ClusterClientReceptionist</c> at <c>/user/node-communication</c> — <b>per node, not as a
///     singleton</b>, so contact rotation reaches whichever node answers.
/// </summary>
/// <remarks>
///     <para>
///         Inbound commands are re-emitted on <see cref="EventStream"/> rather than on their
///         DistributedPubSub topic. DPS is mesh-wide: since central <c>SendToAll</c>s to every node's
///         comm actor, a DPS re-publish would deliver N copies to every subscriber. The EventStream
///         is node-local by construction.
///     </para>
///     <para>
///         Outbound is <see cref="ApplyAck"/> only. There is no Ask across this boundary in Phase 2 —
///         every migrated command is fire-and-forget, and the deploy ack returns as an independent
///         message the coordinator already subscribes for.
///     </para>
/// </remarks>
public sealed class NodeCommunicationActor : ReceiveActor
{
    public const string ActorName = "node-communication";
    public const string CentralCommunicationPath = "/user/central-communication";

Handlers: DispatchDeployment, RestartDriver, ReconnectDriver, AlarmCommand → Context.System.EventStream.Publish(msg); ApplyAck → _centralClient.Tell(new ClusterClient.Send(CentralCommunicationPath, ack)), or Warning + drop when _centralClient is null.

Tests: each inbound type reaches an EventStream subscriber probe exactly once; ApplyAck produces a ClusterClient.Send (not SendToAll — the coordinator is one singleton) with the right path; a null client logs a Warning and drops.

Sabotage-verify: delete the AlarmCommand handler — its test must go RED (guards against a handler being added to the plan but not the actor).

Commit

git commit -m "feat(mesh): NodeCommunicationActor — inbound commands to the node-local EventStream"

---

Task 5: Node-side subscribers accept both transports

Classification: standard Estimated implement time: ~3 min Parallelizable with: none

Files:

• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs:476-485 and :2410-2424
• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs:522-530
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostTransportSubscriptionTests.cs

Subscribe to both, unconditionally — no mode plumbing on the node's subscribers. In Dps mode nothing publishes to the EventStream; in ClusterClient mode nothing publishes to the topic. Adding both means the dark switch touches publishers only, which is what keeps it cheap to flip and cheap to revert.

In DriverHostActor.PreStart, after the existing three Subscribe calls:

        // Phase 2 dark switch: under MeshTransport:Mode = ClusterClient these arrive from
        // NodeCommunicationActor on the node-local EventStream instead of the mesh-wide DPS topic.
        // Subscribing to both is safe and deliberate — exactly one of the two ever publishes, so
        // the transport flag lives entirely on the publishing side.
        Context.System.EventStream.Subscribe(Self, typeof(DispatchDeployment));
        Context.System.EventStream.Subscribe(Self, typeof(RestartDriver));
        Context.System.EventStream.Subscribe(Self, typeof(ReconnectDriver));

Same shape in ScriptedAlarmHostActor.PreStart for typeof(AlarmCommand).

Rename _coordinatorOverride → _ackRouter (field, ctor parameter coordinator: → ackRouter:, and the Props parameter). Under ClusterClient mode this ref is the NodeCommunicationActor, not a coordinator, and the old name would actively mislead the next reader. The SendAck branch at :2413 needs no logic change — only the comment:

        if (_ackRouter is not null)
        {
            // Either a direct coordinator handle (tests) or, under MeshTransport:Mode =
            // ClusterClient, the NodeCommunicationActor that relays this across the boundary.
            _ackRouter.Tell(ack);
        }

Update the call site at Runtime/ServiceCollectionExtensions.cs:~403 (coordinator: null) and every test constructing DriverHostActor.Props.

Test: publish a DispatchDeployment on the EventStream and assert the driver host acts on it; ditto AlarmCommand → ScriptedAlarmHostActor.

Commit

git commit -m "feat(mesh): node subscribers accept EventStream commands; _coordinatorOverride -> _ackRouter"

---

Task 6: Wire both sides + pin the actor paths

Classification: standard Estimated implement time: ~5 min Parallelizable with: none

Files:

• Modify: src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs
• Modify: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:~380-421
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshCommActorPathTests.cs

Central side, in WithOtOpcUaControlPlaneSingletons — a plain WithActors, not a singleton:

        // Per node, NOT a cluster singleton: a node's ClusterClient rotates across contact points
        // and must find a live comm actor at whichever central node answers. A singleton would be
        // reachable only through the one node hosting it, and the rotation would silently fail
        // against the other.
        builder.WithActors((system, registry, resolver) =>
        {
            var actor = system.ActorOf(CentralCommunicationActor.Props(...), CentralCommunicationActor.ActorName);
            ClusterClientReceptionist.Get(system).RegisterService(actor);
            registry.Register<CentralCommunicationActorKey>(actor);
        });

Node side, in WithOtOpcUaRuntimeActors, after driverHost is spawned: build the node's ClusterClient from MeshTransport:CentralContactPoints (only when Mode = ClusterClient), spawn NodeCommunicationActor, register it with the receptionist, and pass it as ackRouter: to DriverHostActor.Props.

Then thread the router into the two singletons (Task 7 uses it): ConfigPublishCoordinator.Props(dbFactory, applyDeadline, meshRouter) and AdminOperationsActor.Props(..., meshRouter).

Pin the paths. /user/central-communication and /user/node-communication are the wire contract and nothing else asserts them — a rename would compile, deploy, and silently deliver nothing:

    private async Task AssertActorExists(ActorSystem system, string path)
    {
        var identity = await system.ActorSelection(path)
            .Ask<ActorIdentity>(new Identify(path), TimeSpan.FromSeconds(5));
        identity.Subject.ShouldNotBeNull($"{path} is the ClusterClient wire contract");
    }

Sabotage-verify: rename the actor to node-comm; the path test must go RED.

Commit

git commit -m "feat(mesh): register comm actors with the receptionist; pin the wire-contract paths"

---

Task 7: Route the publishers through the router

Classification: standard Estimated implement time: ~4 min Parallelizable with: none

Files:

• Modify: src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs:78-101,175
• Modify: src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs:88,135,397,427
• Test: tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ConfigPublishCoordinatorTransportTests.cs

Five publish sites become one call each. ConfigPublishCoordinator.cs:175:

        _meshRouter.Tell(new MeshCommand(DeploymentsTopic, msg));

AdminOperationsActor.cs:88 / :135 / :397 / :427 likewise, with AlarmCommandsTopic.Name / DriverControlTopic.Name.

_meshRouter must be nullable with a mediator fallback, so the ~12 existing tests that call ConfigPublishCoordinator.Props(dbFactory) and AdminOperationsActor.Props(...) keep compiling and keep exercising the DPS path:

    // Null in tests and on any node wired before Phase 2: fall back to the mediator, which is
    // exactly the pre-Phase-2 behaviour. The fallback is not a permanent seam — Phase 6 deletes it
    // with the DPS branch.
    private void Dispatch(string topic, object message)
    {
        if (_meshRouter is not null) _meshRouter.Tell(new MeshCommand(topic, message));
        else DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(topic, message));
    }

Test: with a router probe injected, dispatching a deployment yields MeshCommand(DeploymentsTopic, DispatchDeployment) at the probe and nothing on the mediator; with no router, the mediator still sees the Publish.

Sabotage-verify: make Dispatch always take the mediator branch — the router test must go RED. (Phase 1 shipped a test that passed with the production mapping deleted; assume nothing here is load-bearing until you have watched it fail.)

Commit

git commit -m "feat(mesh): route deploy, driver-control and alarm commands through the mesh router"

---

Task 8: Real two-ActorSystem boundary test

Classification: high-risk Estimated implement time: ~5 min Parallelizable with: none

Files:

• Create: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshClusterClientBoundaryTests.cs

This is the deliberate divergence from the sister project, and the most valuable task in the phase. They have no test that puts two real ActorSystems on either side of a ClusterClient — they substitute a 6-line in-process ClusterClientRelay that unwraps ClusterClient.Send and forwards. That relay tests actor logic and cannot fail for any transport reason: not a missing receptionist extension, not SendToAll behaving unlike Send, not a malformed contact path, not a serialization break. Those are precisely the Phase 2 failure modes.

Build two genuinely separate systems — same ActorSystem name otopcua (required: Akka.Remote address matching means a ClusterClient cannot reach a differently-named system), non-overlapping seed lists, each self-seeded so neither joins the other:

    // Two clusters, one ActorSystem name, joined ONLY by ClusterClient. If this test ever starts
    // passing because the two systems gossiped into one mesh, it is testing nothing — assert the
    // separation explicitly.
    Cluster.Get(_central).State.Members.Count.ShouldBe(1);
    Cluster.Get(_node).State.Members.Count.ShouldBe(1);

Assertions:

1. SendToAll from central reaches the node's registered comm actor and lands on the node's EventStream.
2. An ApplyAck sent from the node reaches central's comm actor.
3. Falsifiability control — with the receptionist extension removed from the node's config, (1) must NOT arrive within the timeout. Without this control the test cannot distinguish "the boundary works" from "something else delivered it".

Use AwaitAssert with explicit timeouts throughout; no Thread.Sleep.

The Phase 1 lesson that earns this task: "the bad thing didn't happen" is not evidence the mechanism works. Assertion 3 is what converts assertions 1–2 from a hope into a measurement.

Scope limit — what this test can NOT see: with exactly one node per side, "registered per node" and "registered as a singleton" are indistinguishable by construction, so this test does NOT cover the per-node property whose structural assertions Task 6 dropped (both discriminators were unfalsifiable — see the dead-end record there). That property's discriminating coverage is live-gate step 8 (stop the OLDEST central; the survivor's comm actor must receive a node ack inside the downing window — a singleton comm actor, hosted on the stopped oldest, cannot). Do not close the phase with step 8 skipped: it is the honest replacement for the dropped Task 6 test.

If this test proves irreducibly flaky, do not delete it and do not weaken it into a relay test — mark it [Trait("Category","MeshBoundary")], exclude it from the default run, and record the flakiness in the live-gate doc as an open item. A quarantined honest test beats a green vacuous one.

Commit

git commit -m "test(mesh): real two-ActorSystem ClusterClient boundary test with falsifiability control"

---

Task 9: Rig config + docs

Classification: small Estimated implement time: ~5 min Parallelizable with: Task 8

Files:

• Modify: docker-dev/docker-compose.yml (add MeshTransport__Mode + MeshTransport__CentralContactPoints__0/1 to all six nodes; leave Mode=Dps so the rig comes up on the old path)
• Modify: docs/Configuration.md (new MeshTransport section)
• Modify: docs/Redundancy.md (the command boundary)
• Modify: docs/plans/2026-07-21-per-cluster-mesh-design.md — correct §2's typed-failure claim and add a "Phase 2 as shipped" note
• Modify: docs/plans/2026-07-22-per-cluster-mesh-program.md — record the no-cross-boundary-Ask gate deviation; flip the Phase 2 row
• Modify: CLAUDE.md — a short "Mesh command transport" note beside the Redundancy section

Commit

git commit -m "docs(mesh): MeshTransport config, Phase 2 as shipped, design-doc correction"

---

Task 10: Live gate on docker-dev

Classification: high-risk Estimated implement time: ~5 min (plus rig time) Parallelizable with: none

Files:

• Create: docs/plans/2026-07-22-mesh-phase2-live-gate.md

Run every step and record the actual output, including anything that fails. Phase 1's gate found a defect at step 4 that steps 1–3 had convincingly hidden; the value is entirely in running the steps that prove recovery, not the ones that prove the happy path.

#	Step	Pass condition
1	Rig up with Mode=Dps; deploy via POST :9200/api/deployments (X-Api-Key: docker-dev-deploy-key)	Seals green — baseline on the old transport
2	Flip all six nodes to Mode=ClusterClient, rebuild, redeploy	Seals green; coordinator logs the same ack count
3	Confirm the transport actually changed	docker logs shows the SendToAll/receptionist path; no deployments DPS publish
4	Stop one node of the site-a pair, deploy	Fails at the apply deadline naming that node (Phase 1 semantics), not a hang
5	Set that node's MaintenanceMode = 1, redeploy	Seals green without it
6	AdminUI Restart + Reconnect on a driver	Reaches the owning node; UI reports "dispatched"
7	AdminUI alarm ack on /alerts	Reaches the owning node's engine
8	Contact rotation / the per-node discriminator. Stop the OLDEST central node only (the singleton host — stopping the younger proves nothing, a singleton would also live on the still-up oldest), then immediately restart one driver node so it emits its boot-time ack	The surviving central's central-communication actor logs receipt of the ack within seconds — inside the 15 s downing window, before singleton migration could explain delivery. This is the discriminating test for "registered per node, NOT a singleton" — the property Task 6's structural assertions could not falsify (both dead ends recorded there): a singleton-shaped comm actor was hosted on the node just stopped, so nothing could receive on the survivor until downing + migration (~25 s). Receipt at the comm actor is the pass signal; the onward Forward to the coordinator singleton proxy is EXPECTED to buffer until the singleton migrates — do not count that against the step (full recovery is step 10's job)
9	Stop both central nodes, then start a driver node	Node boots; ack is dropped with a Warning and not buffered; no crash loop
10	Restart central, redeploy	Recovers with no operator action beyond the redeploy
11	Grep every node's logs for frame-size warnings	None (the deploy path is payload-free — this is the canary that stays quiet)

Step 8 is the one that proves the buffer-size = 0 decision. If a command does arrive late after central returns, buffering is still on somewhere and the DB is no longer the single record of truth.

The rig's AdminUI runs with Security__Auth__DisableLogin=true — drive steps 6 and 7 yourself via browser automation at http://localhost:9200; do not defer them as needing the user to sign in.

Commit

git commit -m "docs(mesh): Phase 2 live-gate record"

---

Risks specific to this phase

• SendToAll vs Send is a one-word difference between "every node deploys" and "one node deploys and the rest silently keep their old config, while the deployment seals green on partial acks". Pinned by a sabotage-verified test in Task 3.
• The single-mesh duplicate trap turns "one client per cluster" — the obviously-right-looking shape, and the one Phase 6 wants — into N× delivery today.
• buffer-size = 0 is a behaviour choice, not a tuning knob. If someone later "fixes" it back to Akka's default to make a flaky test pass, they reintroduce apply-after-timeout divergence between node and DB.
• The AdminUI's Ok = true already means "dispatched", not "applied". Phase 2 moves that lie to a new transport without making it worse; do not let a reviewer believe Phase 2 introduced it.
• redundancy-state is bidirectional and carries two unrelated message types (RedundancyStateChanged central→node, OpcUaProbeResult node→node on the same topic). It is explicitly out of scope here — but note RedundancyStateActor builds its snapshot from Cluster.State, so it is genuinely mesh-bound and is the hardest thing standing between here and Phase 6.
• alerts has a node-local subscriber (HistorianAdapterActor), so Phase 5 cannot simply move it to gRPC without keeping a node-local leg.

---

Follow-ups found during execution (not fixed in Phase 2)

• DefaultMeshClusterClientFactory's generation counter is per factory instance, and its clients are top-level actors. Production creates one factory per CentralCommunicationActor and captures it in the props closure, so the counter survives a supervision restart and names stay unique — that part is sound. What is not handled: the clients are system.ActorOf, not children, so a restart of the comm actor loses _client without stopping it and leaves an orphaned ClusterClient dialling receptionists forever. Harmless today (nothing sends to it) but it is a leak per restart, and it will matter in Phase 6 when there is one client per cluster. Found by MeshClusterClientBoundaryTests, which hit the name collision the moment a second factory was constructed on the same system.