refactor(mesh): step 6 — delete the ClusterClient mode value; split topology now requires Grpc

MeshTransportOptions.ModeClusterClient is gone. MeshTransportOptionsValidator accepts Dps or
Grpc and nothing else, so a node left carrying the string "ClusterClient" is now REFUSED at
boot naming MeshTransport:Mode and both remaining values — deliberately fail-closed rather
than falling back to Dps, which on a split-topology node would deliver nothing at all. A new
fact pins exactly that.

SplitTopologyTransportValidator tightens from an allow-list of two to Grpc only, as its own
comment said it would once ClusterClient was deleted. Dps on a cluster-role node stays
refused; the rule is still an explicit allow-list, so a typo'd mode fails too.

Tests: the two ClusterClient-mode facts in MeshTransportOptionsValidatorTests are deleted
(their Dps siblings already assert the same rule for the one remaining non-Grpc mode), the
SplitTopology fixtures move to Grpc, and the three-mode Theories in
ControlPlaneStarterOrderTests / CentralCommunicationActorTests become two-mode.

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs

@@ -6,9 +6,11 @@ namespace ZB.MOM.WW.OtOpcUa.Cluster;
 /// </summary>
 /// <remarks>
 ///     <para>
-///         <see cref="Mode"/> is a deliberate <b>dark switch</b>: the transports live in the tree
-///         side by side so the docker-dev rig can run the same deployment over each and compare, and
-///         so a bad rollout rolls back with a config change rather than a revert.
+///         <see cref="Mode"/> selects between a <b>pair-local</b> transport and a <b>cross-mesh</b>
+///         one. <see cref="ModeDps"/> is the default and cannot cross an application-<c>Cluster</c>
+///         boundary; <see cref="ModeGrpc"/> is the only mode that can, and
+///         <see cref="SplitTopologyTransportValidator"/> requires it on any node carrying a
+///         <c>cluster-{ClusterId}</c> role.
 ///     </para>
 ///     <para>
 ///         <b>The DistributedPubSub branch is retained, not scheduled for deletion.</b> An earlier
@@ -16,15 +18,15 @@ namespace ZB.MOM.WW.OtOpcUa.Cluster;
 ///         deliberately reversed — see <c>docs/Configuration.md</c> § "The compiled transport-mode
 ///         defaults were deliberately NOT flipped". DPS remains the correct and default transport for
 ///         a single-mesh topology, which every integration test and every profile that leaves this
-///         key unset still runs. What is being replaced is <see cref="ModeClusterClient"/>, by
-///         <see cref="ModeGrpc"/>.
+///         key unset still runs.
 ///     </para>
 ///     <para>
 ///         <b>Discovery is deliberately asymmetric</b>, matching the sister project. Central
 ///         discovers nodes <i>from the database</i> — enabled, non-maintenance <c>ClusterNode</c>
 ///         rows, rebuilt every <see cref="ContactRefreshSeconds"/>. Nodes discover central <i>from
-///         this section</i>: static, restart to change. The asymmetry follows the rate of change —
-///         operators add and retire nodes, but central's own address is part of the deployment.
+///         this section</i> (<see cref="CentralGrpcEndpoints"/>): static, restart to change. The
+///         asymmetry follows the rate of change — operators add and retire nodes, but central's own
+///         address is part of the deployment.
 ///     </para>
 /// </remarks>
 public sealed class MeshTransportOptions
@@ -32,26 +34,26 @@ public sealed class MeshTransportOptions
     /// <summary>Configuration section name.</summary>
     public const string SectionName = "MeshTransport";
 
-    /// <summary>Every command stays on DistributedPubSub — the pre-Phase-2 behaviour.</summary>
+    /// <summary>
+    ///     Every command stays on DistributedPubSub — pair-local, and the default. Cannot cross an
+    ///     application-<c>Cluster</c> boundary.
+    /// </summary>
     public const string ModeDps = "Dps";
 
-    /// <summary>Commands cross the ClusterClient receptionist boundary.</summary>
-    public const string ModeClusterClient = "ClusterClient";
-
     /// <summary>
     ///     Commands cross the mesh boundary as unary gRPC calls on the <c>mesh_transport.v1</c>
     ///     contract — central dials each node's <c>NodeCommandService</c>, a node posts its
-    ///     deployment ack to central's <c>CentralCommandService</c>. The replacement for
-    ///     <see cref="ModeClusterClient"/>; unlike it, the command plane is authenticated
-    ///     (<see cref="GrpcPsk"/>).
+    ///     deployment ack to central's <c>CentralCommandService</c>. The command plane is
+    ///     authenticated per application <c>Cluster</c> (<see cref="GrpcPsk"/>).
     /// </summary>
     public const string ModeGrpc = "Grpc";
 
     /// <summary>
-    ///     <see cref="ModeDps"/> (default), <see cref="ModeClusterClient"/> or
-    ///     <see cref="ModeGrpc"/>. Any other value fails the host at startup rather than silently
-    ///     falling back — a transport that is none of these delivers nothing, and the symptom would
-    ///     be commands that vanish.
+    ///     <see cref="ModeDps"/> (default) or <see cref="ModeGrpc"/>. Any other value fails the host
+    ///     at startup rather than silently falling back — a transport that is neither delivers
+    ///     nothing, and the symptom would be commands that vanish. That includes the retired
+    ///     <c>ClusterClient</c> value: a node still carrying it is refused at boot rather than
+    ///     quietly demoted to DPS, which on a split-topology node would deliver nothing at all.
     /// </summary>
     public string Mode { get; set; } = ModeDps;
 

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptionsValidator.cs

@@ -40,16 +40,18 @@ public sealed class MeshTransportOptionsValidator : IValidateOptions<MeshTranspo
         var errors = new List<string>();
 
         var isDps = string.Equals(options.Mode, MeshTransportOptions.ModeDps, StringComparison.OrdinalIgnoreCase);
-        var isClusterClient = string.Equals(
-            options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
         var isGrpc = string.Equals(options.Mode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase);
 
-        if (!isDps && !isClusterClient && !isGrpc)
+        // An unrecognised mode is refused, never defaulted. The retired 'ClusterClient' value lands
+        // here: a node left carrying it fails to start with the two remaining values named, rather
+        // than silently falling back to Dps — which on a split-topology node cannot cross the mesh
+        // boundary and would deliver nothing at all.
+        if (!isDps && !isGrpc)
         {
             errors.Add(
                 $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)} is "
-                + $"'{options.Mode}'. Expected '{MeshTransportOptions.ModeDps}', "
-                + $"'{MeshTransportOptions.ModeClusterClient}' or '{MeshTransportOptions.ModeGrpc}'.");
+                + $"'{options.Mode}'. Expected '{MeshTransportOptions.ModeDps}' or "
+                + $"'{MeshTransportOptions.ModeGrpc}'.");
         }
 
         if (options.ContactRefreshSeconds <= 0)
@@ -64,6 +66,7 @@ public sealed class MeshTransportOptionsValidator : IValidateOptions<MeshTranspo
         // The first Grpc block below is node-side: the PSK a driver presents (and requires), the
         // central endpoints it posts its ack to, and the two call deadlines (the admin-side rule that
         // follows it is a separate scope). Scoped to the driver role because an admin-only central node
+        // dials nodes rather than being dialled, and resolves per-cluster keys at runtime through
         // MeshTransportOptions.ClusterPsks / the Secrets store, so requiring a single GrpcPsk and an
         // endpoint list of it would demand configuration it never reads. Roles come from
         // configuration, not IClusterRoleInfo (see the ctor remark), and through
@@ -140,7 +143,7 @@ public sealed class MeshTransportOptionsValidator : IValidateOptions<MeshTranspo
         // every node's deployment ack has nowhere to land, and nothing errors: the deployment simply
         // times out naming nodes that applied perfectly. Owned here rather than by ConfigServe's own
         // validation because the requirement comes from MeshTransport:Mode, not from config-serve — a
-        // Dps or ClusterClient central node is legitimately portless.
+        // Dps central node is legitimately portless.
         if (isGrpc && isAdmin)
         {
             // Read as a string, the way SplitTopologyTransportValidator reads the modes:

src/Core/ZB.MOM.WW.OtOpcUa.Cluster/SplitTopologyTransportValidator.cs

@@ -98,22 +98,19 @@ public sealed class SplitTopologyTransportValidator : IValidateOptions<MeshTrans
         var telemetryMode = _configuration["Telemetry:Mode"] ?? TelemetryOptions.ModeDps;
         var telemetryDialMode = _configuration["TelemetryDial:Mode"] ?? TelemetryDialOptions.ModeDps;
 
-        // Two accepted values, not one, for the duration of the ClusterClient→gRPC migration: both
-        // cross a mesh boundary, which is the only property this rule cares about. Without the
-        // relaxation the very first rollout step — flipping a split-topology node to Grpc — would be a
-        // host-start failure. It is NOT widened to "anything but Dps": a typo'd mode delivers nothing
-        // just as surely, so the check stays an explicit allow-list and tightens to Grpc-only once
-        // ClusterClient is deleted.
-        if (!string.Equals(meshMode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase)
-            && !string.Equals(meshMode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase))
+        // Exactly ONE accepted value now that the Akka ClusterClient transport is deleted: Grpc is
+        // the only mode that crosses a mesh boundary, which is the only property this rule cares
+        // about. Deliberately an explicit allow-list rather than "anything but Dps" — a typo'd mode
+        // delivers nothing just as surely as DPS does, and the retired 'ClusterClient' value is
+        // caught here as well as by MeshTransportOptionsValidator's own unknown-mode rule.
+        if (!string.Equals(meshMode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase))
         {
             errors.Add(
                 $"Cluster:Roles declares a cluster-{clusterId} role (split topology) but "
                 + $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)} is "
                 + $"'{meshMode}'; a split mesh cannot use DistributedPubSub — set "
                 + $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)}="
-                + $"{MeshTransportOptions.ModeGrpc} (or {MeshTransportOptions.ModeClusterClient}, the "
-                + "transport it replaces).");
+                + $"{MeshTransportOptions.ModeGrpc}.");
         }
 
         if (isDriver

tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportOptionsValidatorTests.cs

@@ -71,15 +71,26 @@ public class MeshTransportOptionsValidatorTests
     }
 
     [Fact]
-    public void Unknown_mode_message_names_all_three_accepted_values()
+    public void Unknown_mode_message_names_both_accepted_values()
     {
-        // An operator who mistypes the mode gets the full menu, not a partial one — Grpc became a
-        // legal value in Phase 0 and an error that omitted it would send them back to ClusterClient.
-        var result = Validate(new MeshTransportOptions { Mode = "receptionist" });
+        // An operator who mistypes the mode gets the full menu, not a partial one.
+        var result = Validate(new MeshTransportOptions { Mode = "not-a-mode" });
 
         result.Failed.ShouldBeTrue();
         result.FailureMessage.ShouldContain(MeshTransportOptions.ModeDps);
-        result.FailureMessage.ShouldContain(MeshTransportOptions.ModeClusterClient);
+        result.FailureMessage.ShouldContain(MeshTransportOptions.ModeGrpc);
+    }
+
+    [Fact]
+    public void The_retired_ClusterClient_mode_fails_rather_than_falling_back_to_Dps()
+    {
+        // A node left carrying the deleted transport's mode value must be refused at boot. Silently
+        // treating it as Dps would leave a split-topology node running, listening, and delivering
+        // nothing across its mesh boundary — the exact absence this validator exists to prevent.
+        var result = Validate(new MeshTransportOptions { Mode = "ClusterClient" });
+
+        result.Failed.ShouldBeTrue();
+        result.FailureMessage.ShouldContain("MeshTransport:Mode");
         result.FailureMessage.ShouldContain(MeshTransportOptions.ModeGrpc);
     }
 
@@ -293,15 +304,6 @@ public class MeshTransportOptionsValidatorTests
             "admin", "cluster-MAIN").Succeeded.ShouldBeTrue();
     }
 
-    [Fact]
-    public void ClusterClient_admin_node_needs_no_config_serve_listener()
-    {
-        // Fails only if the ConfigServe rule wrongly fires outside Grpc mode.
-        Validate(
-            new MeshTransportOptions { Mode = MeshTransportOptions.ModeClusterClient },
-            "admin", "cluster-MAIN").Succeeded.ShouldBeTrue();
-    }
-
     [Fact]
     public void The_admin_role_is_also_read_from_OTOPCUA_ROLES()
     {
@@ -348,14 +350,6 @@ public class MeshTransportOptionsValidatorTests
             "driver", "cluster-SITE-A").Succeeded.ShouldBeTrue();
     }
 
-    [Fact]
-    public void ClusterClient_mode_requires_none_of_the_new_Grpc_keys()
-    {
-        Validate(
-            new MeshTransportOptions { Mode = MeshTransportOptions.ModeClusterClient },
-            "driver", "cluster-SITE-A").Succeeded.ShouldBeTrue();
-    }
-
     [Fact]
     public void Grpc_fused_admin_and_driver_node_must_satisfy_BOTH_surfaces()
     {

tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitTopologyTransportValidatorTests.cs

@@ -51,7 +51,6 @@ public class SplitTopologyTransportValidatorTests
 
         result.Failed.ShouldBeTrue();
         result.FailureMessage.ShouldContain("MeshTransport:Mode");
-        result.FailureMessage.ShouldContain(MeshTransportOptions.ModeClusterClient);
         result.FailureMessage.ShouldContain(MeshTransportOptions.ModeGrpc);
         result.FailureMessage.ShouldContain("SITE-A");
     }
@@ -59,9 +58,8 @@ public class SplitTopologyTransportValidatorTests
     [Fact]
     public void Cluster_role_driver_node_on_Grpc_mesh_transport_passes()
     {
-        // The migration window: Grpc is the transport this rule exists to arrive at, so a rig node
-        // flipped to it must boot. Without this relaxation the very first Grpc rollout step — setting
-        // the mode on a split-topology node — is a host-start failure.
+        // Grpc is now the ONLY mode that satisfies this rule: it is the only transport that crosses
+        // a mesh boundary.
         Validate(
             meshMode: MeshTransportOptions.ModeGrpc,
             telemetryMode: TelemetryOptions.ModeGrpc,
@@ -92,8 +90,9 @@ public class SplitTopologyTransportValidatorTests
     [Fact]
     public void Cluster_role_node_on_an_unrecognised_mesh_transport_still_fails()
     {
-        // Widening the rule from one accepted value to two must not widen it to "anything that is not
-        // Dps" — a typo'd mode is still a mode that delivers nothing across the boundary.
+        // The rule is an explicit allow-list, not "anything that is not Dps" — a typo'd mode is
+        // still a mode that delivers nothing across the boundary. The retired 'ClusterClient' value
+        // lands here too.
         var result = Validate(
             meshMode: "receptionist",
             telemetryMode: TelemetryOptions.ModeGrpc,
@@ -108,7 +107,7 @@ public class SplitTopologyTransportValidatorTests
     public void Cluster_role_driver_node_fully_configured_passes()
     {
         Validate(
-            meshMode: MeshTransportOptions.ModeClusterClient,
+            meshMode: MeshTransportOptions.ModeGrpc,
             telemetryMode: TelemetryOptions.ModeGrpc,
             telemetryDialMode: null,
             "driver", "cluster-SITE-A").Succeeded.ShouldBeTrue();
@@ -118,7 +117,7 @@ public class SplitTopologyTransportValidatorTests
     public void Cluster_role_admin_node_fully_configured_passes()
     {
         Validate(
-            meshMode: MeshTransportOptions.ModeClusterClient,
+            meshMode: MeshTransportOptions.ModeGrpc,
             telemetryMode: null,
             telemetryDialMode: TelemetryDialOptions.ModeGrpc,
             "admin", "cluster-SITE-A").Succeeded.ShouldBeTrue();
@@ -128,7 +127,7 @@ public class SplitTopologyTransportValidatorTests
     public void Cluster_role_driver_node_on_Dps_telemetry_fails()
     {
         var result = Validate(
-            meshMode: MeshTransportOptions.ModeClusterClient,
+            meshMode: MeshTransportOptions.ModeGrpc,
             telemetryMode: TelemetryOptions.ModeDps,
             telemetryDialMode: null,
             "driver", "cluster-SITE-A");
@@ -142,7 +141,7 @@ public class SplitTopologyTransportValidatorTests
     public void Cluster_role_admin_node_on_Dps_telemetry_dial_fails()
     {
         var result = Validate(
-            meshMode: MeshTransportOptions.ModeClusterClient,
+            meshMode: MeshTransportOptions.ModeGrpc,
             telemetryMode: null,
             telemetryDialMode: TelemetryDialOptions.ModeDps,
             "admin", "cluster-SITE-A");
@@ -156,7 +155,7 @@ public class SplitTopologyTransportValidatorTests
     public void Fused_cluster_role_node_needs_all_three_to_pass()
     {
         Validate(
-            meshMode: MeshTransportOptions.ModeClusterClient,
+            meshMode: MeshTransportOptions.ModeGrpc,
             telemetryMode: TelemetryOptions.ModeGrpc,
             telemetryDialMode: TelemetryDialOptions.ModeGrpc,
             "admin", "driver", "cluster-MAIN").Succeeded.ShouldBeTrue();
@@ -166,7 +165,7 @@ public class SplitTopologyTransportValidatorTests
     public void Fused_cluster_role_node_missing_telemetry_grpc_fails()
     {
         var result = Validate(
-            meshMode: MeshTransportOptions.ModeClusterClient,
+            meshMode: MeshTransportOptions.ModeGrpc,
             telemetryMode: TelemetryOptions.ModeDps,
             telemetryDialMode: TelemetryDialOptions.ModeGrpc,
             "admin", "driver", "cluster-MAIN");
@@ -179,7 +178,7 @@ public class SplitTopologyTransportValidatorTests
     public void Fused_cluster_role_node_missing_telemetry_dial_grpc_fails()
     {
         var result = Validate(
-            meshMode: MeshTransportOptions.ModeClusterClient,
+            meshMode: MeshTransportOptions.ModeGrpc,
             telemetryMode: TelemetryOptions.ModeGrpc,
             telemetryDialMode: TelemetryDialOptions.ModeDps,
             "admin", "driver", "cluster-MAIN");
@@ -229,7 +228,7 @@ public class SplitTopologyTransportValidatorTests
     public void Mode_matching_is_case_insensitive()
     {
         Validate(
-            meshMode: "clusterclient",
+            meshMode: "grpc",
             telemetryMode: "grpc",
             telemetryDialMode: "grpc",
             "admin", "driver", "cluster-MAIN").Succeeded.ShouldBeTrue();
@@ -287,13 +286,13 @@ public class SplitTopologyTransportValidatorTests
     }
 
     [Fact]
-    public void Cluster_role_node_with_neither_admin_nor_driver_only_needs_ClusterClient()
+    public void Cluster_role_node_with_neither_admin_nor_driver_only_needs_the_mesh_mode()
     {
         // A cluster-role node that is neither admin nor driver has no telemetry surface to constrain —
-        // only the always-on MeshTransport=ClusterClient rule applies. Telemetry/TelemetryDial left on
+        // only the always-on MeshTransport=Grpc rule applies. Telemetry/TelemetryDial left on
         // their Dps defaults must not fail it.
         Validate(
-            meshMode: MeshTransportOptions.ModeClusterClient,
+            meshMode: MeshTransportOptions.ModeGrpc,
             telemetryMode: null,
             telemetryDialMode: null,
             "cluster-SITE-A").Succeeded.ShouldBeTrue();

tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/CentralCommunicationActorTests.cs

@@ -515,17 +515,16 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
 
     [Theory]
     [InlineData(MeshTransportOptions.ModeDps)]
-    [InlineData(MeshTransportOptions.ModeClusterClient)]
     [InlineData(MeshTransportOptions.ModeGrpc)]
     public void The_known_cluster_gate_is_published_in_every_transport_mode(string mode)
     {
         // Replaces an earlier fact asserting the OPPOSITE, whose premise did not survive Phase 1B: the
         // gate feeds CentralCommandAuthInterceptor, which is registered on every admin node, and the
         // ack service it protects is mapped by LISTENER PORT — neither consults MeshTransport:Mode.
-        // Gating the publication on Grpc meant a dark-switch rollback to Dps left that endpoint
-        // serving with a permanently inactive gate, silently restoring both the timing
-        // oracle and the pre-authentication store/audit flood. The set is central's cluster
-        // inventory, not transport state.
+        // Gating the publication on Grpc meant a rollback to Dps left that endpoint serving with a
+        // permanently inactive gate, silently restoring both the timing oracle and the
+        // pre-authentication store/audit flood. The set is central's cluster inventory, not
+        // transport state.
         var factory = NewInMemoryDbFactory();
         SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
         var psk = new RecordingPskProvider();

tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/ControlPlaneStarterOrderTests.cs

@@ -55,11 +55,10 @@ public sealed class ControlPlaneStarterOrderTests
 {
     [Theory]
     [InlineData(MeshTransportOptions.ModeDps)]
-    [InlineData(MeshTransportOptions.ModeClusterClient)]
     [InlineData(MeshTransportOptions.ModeGrpc)]
     public async Task An_admin_node_boots_with_the_real_mesh_dependency_graph(string mode)
     {
-        // Every mode, because the eager resolve was mode-independent: the starter resolved both
+        // Both modes, because the eager resolve was mode-independent: the starter resolved both
         // dependencies before anything looked at MeshTransport:Mode, so Dps — the DEFAULT, and the
         // rollback target — died exactly like Grpc did.
         using var host = BuildAdminHost(mode);
@@ -82,7 +81,6 @@ public sealed class ControlPlaneStarterOrderTests
 
     [Theory]
     [InlineData(MeshTransportOptions.ModeDps)]
-    [InlineData(MeshTransportOptions.ModeClusterClient)]
     [InlineData(MeshTransportOptions.ModeGrpc)]
     public async Task The_deferred_dependencies_are_resolved_and_used_once_the_host_is_up(string mode)
     {
@@ -90,7 +88,7 @@ public sealed class ControlPlaneStarterOrderTests
         // silent one, with the known-cluster gate permanently unpublished and the dialer permanently
         // absent. So this asserts the far end of the deferral: the actor resolved the PSK provider
         // FROM THE CONTAINER and drove it, through the ordinary contact-refresh path. A Theory over
-        // all three modes because publication is mode-independent (the gate follows the endpoint).
+        // both modes because publication is mode-independent (the gate follows the endpoint).
         //
         // The provider is registered as a TYPE with an ISecretResolver constructor, so the resolve
         // walks the same registry-bound chain the production MeshPskProvider does — which makes