dohertj2/lmxopcua
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(mesh): step 1 — delete the central ClusterClient factory and per-cluster client cache

Browse Source 
IMeshClusterClientFactory/DefaultMeshClusterClientFactory are gone, along with
CentralCommunicationActor's _clients map, RebuildClient, the SendToAll fan and the
receptionist-address half of the contact load. LoadContactsFromDb SURVIVES — it feeds the
gRPC dialer's endpoint resolution and the known-cluster gate — but now projects only the
columns those two need (the AkkaPort/ActorPath.TryParse parse and its Malformed list went
with the receptionist addresses).

Ripples fixed in this step because the build must be green at every commit:
  - ControlPlane/ServiceCollectionExtensions drops the factory argument.
  - CentralCommunicationActorTests: the six ClusterClient-only facts are deleted (each has a
    Grpc counterpart that already asserted the same property independently), the three
    transport-agnostic facts move to Grpc mode, and the Dps fact keeps its own coverage.
  - MeshClusterClientFactoryTests deleted (its subject is gone).
  - MeshClusterClientBoundaryTests deleted here rather than in step 7: it drives
    DefaultMeshClusterClientFactory directly, so it cannot compile past this commit. Its
    replacement, MeshGrpcBoundaryTests, lands in step 7 and keeps the load-bearing assertion
    (two separate meshes, no cross-mesh Akka association).
This commit is contained in:
Joseph Doherty
2026-08-06 15:35:54 -04:00
parent 5510452662
commit 0db40a5e7f
6 changed files with 78 additions and 1108 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs
+46 -260
View File
@@ -1,6 +1,4 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using Grpc.Core;
@@ -20,39 +18,30 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
/// <summary>
/// Central-side end of the central↔node command boundary (per-cluster mesh Phase 2). Routes
/// every central→node command out — over DistributedPubSub, ClusterClient or gRPC, per
/// <see cref="MeshTransportOptions.Mode"/> — and forwards inbound <see cref="ApplyAck"/>s to the
/// deploy coordinator singleton.
/// every central→node command out — over DistributedPubSub (pair-local) or gRPC (cross-mesh),
/// per <see cref="MeshTransportOptions.Mode"/> — and forwards inbound <see cref="ApplyAck"/>s to
/// the deploy coordinator singleton.
/// </summary>
/// <remarks>
/// <para>
/// <b>Registered per node, NOT as a cluster singleton.</b> A driver node's ClusterClient
/// rotates across contact points and must find a live comm actor at whichever central node
/// answers. A singleton would be reachable only through the one node hosting it, and the
/// rotation would fail silently against the other.
/// <b>Registered per node, NOT as a cluster singleton.</b> A driver node posts its ack to
/// whichever central node answers first, so a live comm actor must exist on both. A singleton
/// would be reachable only through the one node hosting it, and the other central node's ack
/// ingress would have nothing to forward to.
/// </para>
/// <para>
/// <b>One ClusterClient per application <c>Cluster</c> (Phase 6).</b> The fleet is now one
/// Akka mesh per cluster, so a single fleet-wide client's <c>SendToAll</c> would reach only
/// the one mesh whose receptionist answered. This actor therefore holds a client per cluster
/// and fans <c>SendToAll</c> across all of them. See <see cref="RebuildClient"/> — this is the
/// single most important thing to understand before changing this actor.
/// <b>The gRPC branch fans node by node over the contact load's target set.</b> That load
/// selects exactly the enabled, non-maintenance <c>ClusterNode</c> rows carrying a
/// <c>GrpcPort</c>, and the fan dials every one of them, one call per node. It is deliberately
/// NOT narrower: the node side has no <c>ClusterId</c> or node filter at all — scoping happens
/// later, inside the artifact — so a node left out here silently keeps its old configuration
/// while the deployment seals green.
/// </para>
/// <para>
/// <b>The gRPC branch fans over the same recipient set, node by node</b>
/// (ClusterClient→gRPC Phase 1A). <c>SendToAll</c> reaches every node registered on a mesh's
/// receptionist, and the contact load already selects exactly the enabled, non-maintenance
/// rows those nodes come from; the gRPC branch dials that same row set directly, one call per
/// node, so the recipient set is identical by construction rather than by coincidence. It is
/// deliberately NOT narrower: the node side has no <c>ClusterId</c> or node filter at all —
/// scoping happens later, inside the artifact — so a node left out here silently keeps its
/// old configuration while the deployment seals green.
/// </para>
/// <para>
/// <b>Fire-and-forget, on every transport.</b> The gRPC branch pipes per-node results back to
/// <b>Fire-and-forget, on both transports.</b> The gRPC branch pipes per-node results back to
/// itself for logging ONLY: a failure names the node and the status and is dropped. No retry,
/// no queue, no ordering guarantee across nodes. That is not an omission — it is the
/// ClusterClient semantic preserved exactly, and the backstops are unchanged (the deploy
/// DistributedPubSub semantic preserved exactly, and the backstops are unchanged (the deploy
/// coordinator's apply deadline, and a node's boot-time replay).
/// </para>
/// </remarks>
@@ -62,7 +51,6 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
private readonly MeshTransportOptions _options;
private readonly IMeshClusterClientFactory _clientFactory;
private readonly Func<IActorRef?> _coordinator;
// Resolved in PreStart, never at construction. See the Props doc: the DI graph behind these two
@@ -88,15 +76,8 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
_dialer ??= Resolve(_dialerFactory, nameof(IMeshCommandDialer), firstUse: true);
private readonly ILoggingAdapter _log = Context.GetLogger();
private readonly bool _useClusterClient;
private readonly bool _useGrpc;
// One client per application Cluster, keyed by ClusterId. A ClusterClient reaches only the mesh
// whose receptionist it dialled, and after Phase 6 every cluster is its own mesh — so central must
// hold a client per cluster and fan SendToAll across all of them.
private readonly Dictionary<string, (IActorRef Client, ImmutableHashSet<string> Contacts)> _clients =
new();
// Node ids already warned about for a null GrpcPort. A persistently portless row would otherwise
// produce one warning per refresh forever; same warn-once shape as TelemetryNodeSource. Actor
// thread only.
@@ -121,7 +102,6 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// <summary>Creates the props for this actor.</summary>
/// <param name="dbFactory">Factory for the config database holding <c>ClusterNode</c> rows.</param>
/// <param name="options">The bound mesh-transport options.</param>
/// <param name="clientFactory">Creates the ClusterClient; substituted in tests.</param>
/// <param name="coordinator">
/// Resolves the deploy-coordinator singleton proxy. A <see cref="Func{TResult}"/> rather than
/// an <see cref="IActorRef"/> so this actor never depends on registration order at wiring
@@ -129,9 +109,9 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// </param>
/// <param name="dialer">
/// Resolves the thing that places one command call on one node, under
/// <see cref="MeshTransportOptions.ModeGrpc"/>. Optional so the two older transports need
/// nothing new wired; a <see langword="null"/> dialer under <c>Grpc</c> is a mis-wiring and
/// is reported loudly rather than silently dropping every command.
/// <see cref="MeshTransportOptions.ModeGrpc"/>. Optional so the Dps transport needs nothing
/// new wired; a <see langword="null"/> dialer under <c>Grpc</c> is a mis-wiring and is
/// reported loudly rather than silently dropping every command.
/// </param>
/// <param name="pskProvider">
/// Resolves central's per-cluster key lookup, told on every contact refresh which clusters
@@ -163,37 +143,30 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
public static Props Props(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
MeshTransportOptions options,
IMeshClusterClientFactory clientFactory,
Func<IActorRef?> coordinator,
Func<IMeshCommandDialer?>? dialer = null,
Func<IMeshPskProvider?>? pskProvider = null) =>
Akka.Actor.Props.Create(() =>
new CentralCommunicationActor(
dbFactory, options, clientFactory, coordinator, dialer, pskProvider));
new CentralCommunicationActor(dbFactory, options, coordinator, dialer, pskProvider));
/// <summary>Initializes a new instance of the <see cref="CentralCommunicationActor"/> class.</summary>
/// <param name="dbFactory">Factory for the config database.</param>
/// <param name="options">The bound mesh-transport options.</param>
/// <param name="clientFactory">Creates the ClusterClient.</param>
/// <param name="coordinator">Lazily resolves the deploy-coordinator singleton proxy.</param>
/// <param name="dialer">Resolves the gRPC command dialer; invoked in <see cref="PreStart"/>.</param>
/// <param name="pskProvider">Resolves the PSK provider; invoked in <see cref="PreStart"/>.</param>
public CentralCommunicationActor(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
MeshTransportOptions options,
IMeshClusterClientFactory clientFactory,
Func<IActorRef?> coordinator,
Func<IMeshCommandDialer?>? dialer = null,
Func<IMeshPskProvider?>? pskProvider = null)
{
_dbFactory = dbFactory;
_options = options;
_clientFactory = clientFactory;
_coordinator = coordinator;
_dialerFactory = dialer;
_pskProviderFactory = pskProvider;
_useClusterClient = string.Equals(
options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
_useGrpc = string.Equals(
options.Mode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase);
@@ -246,13 +219,13 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
_dialer = Resolve(_dialerFactory, nameof(IMeshCommandDialer), firstUse: false);
_pskProvider = Resolve(_pskProviderFactory, nameof(IMeshPskProvider), firstUse: false);
if (!_useClusterClient && !_useGrpc)
if (!_useGrpc)
{
_log.Info(
"Mesh transport is {Mode}: commands are published on DistributedPubSub and no "
+ "ClusterClient is created. The contact refresh still runs — it is what keeps the "
+ "known-cluster gate current, and that gate belongs to the inbound ack endpoint, "
+ "which is mapped by port and does not care about this mode", _options.Mode);
"Mesh transport is {Mode}: commands are published on DistributedPubSub and nothing "
+ "is dialled. The contact refresh still runs — it is what keeps the known-cluster "
+ "gate current, and that gate belongs to the inbound ack endpoint, which is mapped "
+ "by port and does not care about this mode", _options.Mode);
}
// Deliberately NO null guard for either dependency here. At this point a still-null value
@@ -266,7 +239,7 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
// Started in EVERY mode. The refresh is not transport state: it is central's ClusterNode
// inventory, and the known-cluster gate it feeds protects an endpoint mapped by PORT, not by
// MeshTransport:Mode. Gating the timer on the mode would mean a dark-switch rollback to
// ClusterClient or Dps left that endpoint live with the gate permanently inactive — silently
// Dps left that endpoint live with the gate permanently inactive — silently
// reopening both the timing oracle and the pre-auth store/audit flood. Under Dps this is a
// ConfigDb query every ContactRefreshSeconds that did not run before; central always has a
// ConfigDb, and one indexed read a minute is the cheaper half of the trade.
@@ -357,9 +330,6 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// <inheritdoc />
protected override void PostStop()
{
foreach (var (client, _) in _clients.Values) Context.Stop(client);
_clients.Clear();
// Cancelling the lifecycle token also cancels every gRPC dial in flight. The dialer itself is
// deliberately NOT disposed here: it is a container-owned singleton shared with anything else
// that may dial, and an actor restart must not tear its channel pool down.
@@ -369,43 +339,15 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
private void HandleMeshCommand(MeshCommand cmd)
{
// Checked before the DPS arm: _useClusterClient is false under Grpc too, so the order is
// what keeps a Grpc-mode command off DistributedPubSub.
// Checked first, so a Grpc-mode command never reaches DistributedPubSub. DPS is mesh-wide and
// cannot cross an application-Cluster boundary at all in the split topology.
if (_useGrpc)
{
DispatchOverGrpc(cmd);
return;
}
if (!_useClusterClient)
{
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(cmd.Topic, cmd.Message));
return;
}
if (_clients.Count == 0)
{
// App-level drop, matching the sister project's decision and this repo's buffer-size = 0.
// The command is NOT queued and will NOT be retried: a deploy fails at the coordinator's
// apply deadline naming the silent nodes, which is a better operator signal than a
// command arriving minutes late against a config the DB already recorded as failed.
_log.Warning(
"No mesh ClusterClient — dropping {MessageType}. The last contact refresh produced "
+ "no usable contact point from the enabled, non-maintenance ClusterNode rows in any "
+ "cluster; the command is not buffered and will not be retried",
cmd.Message.GetType().Name);
return;
}
// SendToAll, NEVER Send, fanned across EVERY cluster's client. Today's DPS publish reaches
// EVERY DriverHostActor — there is no ClusterId or node filter on the node side at all; scoping
// happens later, inside the artifact (DeploymentArtifact.ResolveClusterScope). Each ClusterClient
// reaches only the one mesh it dialled, so central must fan across all of them to preserve the
// fleet-wide reach DPS gives. ClusterClient.Send delivers to exactly ONE registered actor, which
// would deploy to a single node while every other node silently kept its old configuration — and
// the deployment would still seal green if the ack set happened to be satisfied.
foreach (var (client, _) in _clients.Values)
client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, cmd.Message));
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(cmd.Topic, cmd.Message));
}
/// <summary>
@@ -453,10 +395,10 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
var targets = _targets;
if (targets.Count == 0)
{
// App-level drop, exactly as the ClusterClient arm does with no client. The command is
// NOT queued and will NOT be retried: a deploy fails at the coordinator's apply deadline
// naming the silent nodes, which is a better operator signal than a command arriving
// minutes late against a config the DB already recorded as failed.
// App-level drop. The command is NOT queued and will NOT be retried: a deploy fails at
// the coordinator's apply deadline naming the silent nodes, which is a better operator
// signal than a command arriving minutes late against a config the DB already recorded
// as failed.
_log.Warning(
"No mesh gRPC targets — dropping {MessageType}. The last contact refresh produced no "
+ "enabled, non-maintenance ClusterNode row carrying a GrpcPort in any cluster; the "
@@ -632,7 +574,6 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
return; // Stopping.
}
var systemName = Context.System.Name;
var dbFactory = _dbFactory;
// Off the actor thread, result piped back as a message. A synchronous DB read here would
@@ -641,29 +582,21 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
{
await using var db = await dbFactory.CreateDbContextAsync(ct).ConfigureAwait(false);
// UNFILTERED, with the eligibility flags projected and applied per row below. The two
// outputs of this read want different row sets: the contact/dispatch sets are the
// enabled, non-maintenance nodes central talks TO, while the known-cluster gate answers
// "is this a cluster central has ever heard of" and must include a cluster whose every
// node is disabled or in maintenance — such a cluster can still send an ack, and denying
// it would make maintenance mode an authentication switch.
// outputs of this read want different row sets: the dispatch set is the enabled,
// non-maintenance nodes central talks TO, while the known-cluster gate answers "is this a
// cluster central has ever heard of" and must include a cluster whose every node is
// disabled or in maintenance — such a cluster can still send an ack, and denying it would
// make maintenance mode an authentication switch.
var rows = await db.ClusterNodes
.AsNoTracking()
.Select(n => new
{
n.NodeId, n.ClusterId, n.Host, n.AkkaPort, n.GrpcPort, n.Enabled, n.MaintenanceMode,
n.NodeId, n.ClusterId, n.Host, n.GrpcPort, n.Enabled, n.MaintenanceMode,
})
.ToListAsync(ct)
.ConfigureAwait(false);
// Grouped BY ClusterId: central now dials one ClusterClient per application Cluster, so the
// contact set must be partitioned the same way. A row's contacts join only its own cluster's
// client.
var byCluster = new Dictionary<string, List<string>>();
var malformed = new List<string>();
// The gRPC dispatch set, built from the SAME row set in the SAME pass. Sharing the read is
// what makes "the gRPC fan reaches exactly the ClusterClient recipient set" structural
// rather than a claim two separate queries would have to keep agreeing on.
// The gRPC dispatch set: one entry per eligible row that carries a port.
var targets = new List<MeshNodeTarget>(rows.Count);
var portless = new List<string>();
@@ -677,8 +610,7 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
clusterIds.Add(row.ClusterId);
// The eligibility filter that used to be the query's WHERE clause. Everything below
// this line — receptionist contacts and gRPC targets — sees exactly the rows it saw
// before.
// this line — the gRPC targets — sees exactly the rows it saw before.
if (!row.Enabled || row.MaintenanceMode)
continue;
@@ -687,29 +619,9 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
row.NodeId, row.ClusterId, $"http://{row.Host}:{grpcPort}"));
else
portless.Add(row.NodeId);
var address = $"akka.tcp://{systemName}@{row.Host}:{row.AkkaPort}/system/receptionist";
// Parse up front, per row, inside the loop's own guard. A single malformed row must
// not abort the whole refresh and leave the contact set half-built — the sister
// project shipped that bug and its regression test deliberately orders the bad row
// first.
if (ActorPath.TryParse(address, out _))
{
if (!byCluster.TryGetValue(row.ClusterId, out var list))
byCluster[row.ClusterId] = list = new List<string>();
list.Add(address);
}
else
{
malformed.Add($"{row.NodeId} ({row.ClusterId}) -> {address}");
}
}
var contactsByCluster = byCluster.ToDictionary(
kv => kv.Key, kv => (IReadOnlyList<string>)kv.Value);
return new ContactsLoaded(
contactsByCluster, malformed, targets, portless, clusterIds.ToList());
return new ContactsLoaded(targets, portless, clusterIds.ToList());
}, ct).PipeTo(self);
}
@@ -722,66 +634,6 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
// EVERY mode. See PreStart — the gate protects an endpoint mapped by port, not by mode.
PublishKnownClusters(msg);
if (!_useClusterClient)
return; // Dps or Grpc: no ClusterClient to rebuild.
foreach (var bad in msg.Malformed)
{
_log.Warning(
"ClusterNode {Row} does not yield a usable receptionist address; skipping it in this "
+ "refresh (other nodes are unaffected). Check the row's Host and AkkaPort", bad);
}
var byCluster = msg.ContactsByCluster.ToDictionary(
kv => kv.Key, kv => kv.Value.ToImmutableHashSet());
// Clusters that dropped out of the DB entirely, or whose only rows were malformed this refresh:
// stop and forget their clients so a vanished cluster stops receiving. LoadContactsFromDb only
// creates a ByCluster key when at least one of its rows parsed, so an all-malformed cluster has
// NO key at all and is caught here by the absent-key (!TryGetValue) arm — not by an empty value.
var gone = _clients.Keys
.Where(id => !byCluster.TryGetValue(id, out var c) || c.IsEmpty)
.ToList();
foreach (var clusterId in gone)
{
_log.Info(
"Cluster {ClusterId} no longer yields any usable contact point; stopping its "
+ "ClusterClient", clusterId);
Context.Stop(_clients[clusterId].Client);
_clients.Remove(clusterId);
}
var anyUsable = false;
foreach (var (clusterId, contacts) in byCluster)
{
if (contacts.IsEmpty)
{
// Defensive: LoadContactsFromDb never emits an empty-but-present cluster (a key is
// created only on a first successfully-parsed row), so this arm is unreachable for the
// DB producer — an all-malformed cluster is handled by the absent-key branch above. It
// guards a ContactsLoaded (a public record) constructed elsewhere with an empty list.
_log.Warning(
"Cluster {ClusterId} has no usable ClusterNode contact points; no ClusterClient "
+ "was created for it and its nodes' commands will be dropped until a refresh finds "
+ "one", clusterId);
continue;
}
anyUsable = true;
if (_clients.TryGetValue(clusterId, out var existing) && existing.Contacts.SetEquals(contacts))
continue;
RebuildClient(clusterId, contacts);
}
if (!anyUsable && _clients.Count == 0)
{
_log.Warning(
"No usable ClusterNode contact points in any cluster; no mesh ClusterClient was "
+ "created and every central→node command will be dropped until a refresh finds one");
}
}
/// <summary>
@@ -790,11 +642,9 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// </summary>
/// <param name="msg">The load result.</param>
/// <remarks>
/// The malformed-receptionist-address warnings are deliberately NOT logged here: nothing on
/// this transport dials a receptionist, and reporting an unusable Akka address as a
/// command-plane problem would send an operator after the wrong column. A host that is
/// unusable as a gRPC endpoint surfaces instead as a named per-node dial failure on every
/// dispatch, which is a truer signal.
/// A host that is unusable as a gRPC endpoint is deliberately NOT screened here: it surfaces
/// as a named per-node dial failure on every dispatch, which is a truer signal than a
/// refresh-time warning about a string that may well be fine.
/// </remarks>
private void AdoptGrpcTargets(ContactsLoaded msg)
{
@@ -837,7 +687,7 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// <b>Not gated on <see cref="MeshTransportOptions.Mode"/>, on purpose.</b> The gate feeds
/// <c>CentralCommandAuthInterceptor</c>, which is registered on every admin node, and the
/// ack service it protects is mapped by listener port — neither consults the mode. Gating
/// this on <c>Grpc</c> would mean a dark-switch rollback to <c>ClusterClient</c> or
/// this on <c>Grpc</c> would mean a dark-switch rollback to
/// <c>Dps</c> left that endpoint serving with a permanently inactive gate: the timing
/// oracle and the pre-authentication store/audit flood would both come back, silently,
/// as a side effect of a rollback nobody would connect to authentication.
@@ -887,75 +737,13 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
pskProvider.InvalidateAll();
}
/// <summary>Stops a cluster's existing client and creates a replacement for its new contact set.</summary>
/// <remarks>
/// <para>
/// <b>One client per application <c>Cluster</c> (delivered in Phase 6).</b> This actor now
/// holds one <c>ClusterClient</c> per cluster and <see cref="HandleMeshCommand"/> fans
/// <c>SendToAll</c> across all of them. Once the fleet is split into one Akka mesh per
/// cluster, a <c>ClusterClientReceptionist</c> serves only <i>its own</i> mesh, so a single
/// client's <c>SendToAll</c> would reach the one mesh whose receptionist answered and leave
/// every other cluster silently on its old configuration. Fanning across a client per
/// cluster restores the fleet-wide reach the pre-split DPS broadcast gave.
/// </para>
/// <para>
/// Why this is not N× duplicate dispatch: each client dials only its own cluster's
/// receptionists, so a command is delivered <b>once per mesh</b>, not once per cluster into
/// every mesh. (While the fleet was a single mesh, the same code would have duplicated —
/// which is exactly why the fan-out could not land until the meshes were partitioned.)
/// </para>
/// <para>
/// Corollary worth stating because it is easy to assume otherwise: <b>the contact set
/// does not scope delivery within a mesh.</b> Excluding a maintenance-mode node from a
/// cluster's contacts does not stop it receiving commands — its receptionist still delivers
/// to every registered node-comm actor in that mesh. The filter is about which receptionists
/// are worth <i>dialling</i> (a node in maintenance may be switched off), not about who gets
/// the message.
/// </para>
/// </remarks>
/// <param name="clusterId">The application <c>Cluster</c> whose client is being (re)built.</param>
/// <param name="contacts">The receptionist addresses to dial for that cluster.</param>
private void RebuildClient(string clusterId, ImmutableHashSet<string> contacts)
{
if (_clients.TryGetValue(clusterId, out var existing))
{
_log.Info("Contact set for cluster {ClusterId} changed; replacing its ClusterClient", clusterId);
Context.Stop(existing.Client);
// Removed now: if the create below throws, a stale ref would route commands into a
// stopping actor and they would vanish without a warning.
_clients.Remove(clusterId);
}
try
{
var paths = contacts.Select(ActorPath.Parse).ToImmutableHashSet();
var client = _clientFactory.Create(Context.System, clusterId, paths);
_clients[clusterId] = (client, contacts);
_log.Info(
"Mesh ClusterClient for cluster {ClusterId} created with {Count} contact point(s)",
clusterId, contacts.Count);
}
catch (Exception ex)
{
_log.Error(ex,
"Failed to create the mesh ClusterClient for cluster {ClusterId}; its nodes' commands "
+ "are dropped until the next contact refresh succeeds", clusterId);
}
}
/// <summary>Internal tick asking for a contact-point refresh.</summary>
public sealed record RefreshContacts;
/// <summary>Result of a contact-point load, piped back from the DB read.</summary>
/// <param name="ContactsByCluster">
/// Receptionist addresses that parsed cleanly, grouped by the owning application
/// <c>Cluster</c>'s id. One <c>ClusterClient</c> is built per key.
/// </param>
/// <param name="Malformed">Rows that did not parse, described for the log.</param>
/// <param name="Targets">
/// The gRPC dispatch set: one entry per enabled, non-maintenance row that carries a
/// <c>GrpcPort</c>. Built from the same rows as <paramref name="ContactsByCluster"/>.
/// <c>GrpcPort</c>.
/// </param>
/// <param name="PortlessNodes">
/// Node ids skipped because their row has no <c>GrpcPort</c>. Carried rather than logged in
@@ -968,8 +756,6 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// still send an ack, and refusing it would make maintenance mode an authentication switch.
/// </param>
public sealed record ContactsLoaded(
IReadOnlyDictionary<string, IReadOnlyList<string>> ContactsByCluster,
IReadOnlyList<string> Malformed,
IReadOnlyList<MeshNodeTarget> Targets,
IReadOnlyList<string> PortlessNodes,
IReadOnlyList<string> ClusterIds);
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/IMeshClusterClientFactory.cs
-67
View File
@@ -1,67 +0,0 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
/// <summary>
/// Creates the <c>ClusterClient</c> central uses to reach driver nodes. Exists as a seam so
/// <c>CentralCommunicationActor</c> can be tested against a <c>TestProbe</c> without standing up
/// a second cluster.
/// </summary>
public interface IMeshClusterClientFactory
{
/// <summary>Creates a client for the given receptionist contact points.</summary>
/// <param name="system">The actor system to spawn the client in.</param>
/// <param name="clusterId">
/// The application <c>Cluster</c> these contacts belong to. Central now creates one client per
/// cluster (Phase 6), so the id is woven into the client's actor name to keep the per-cluster
/// clients distinct and diagnosable.
/// </param>
/// <param name="contacts">Receptionist actor paths, one per reachable node.</param>
/// <returns>The new client's <see cref="IActorRef"/>.</returns>
IActorRef Create(ActorSystem system, string clusterId, ImmutableHashSet<ActorPath> contacts);
}
/// <inheritdoc />
public sealed class DefaultMeshClusterClientFactory : IMeshClusterClientFactory
{
/// <summary>
/// Per-incarnation generation counter, appended to the actor name.
/// </summary>
/// <remarks>
/// <c>Context.Stop</c> of the previous client is <b>asynchronous</b> — its actor name stays
/// reserved until termination completes — so recreating the same name while handling a single
/// message throws <see cref="InvalidActorNameException"/>. That is exactly what a contact-set
/// change does: stop the old client, create the replacement, both in one handler. A
/// generation suffix makes every incarnation's name unique by construction. Ported from the
/// sister project, where it was a shipped bug fix rather than foresight.
/// </remarks>
private long _generation;
/// <inheritdoc />
public IActorRef Create(ActorSystem system, string clusterId, ImmutableHashSet<ActorPath> contacts)
{
ArgumentNullException.ThrowIfNull(system);
ArgumentException.ThrowIfNullOrEmpty(clusterId);
ArgumentNullException.ThrowIfNull(contacts);
var settings = ClusterClientSettings.Create(system).WithInitialContacts(contacts);
var name = $"mesh-client-{SanitizeForActorName(clusterId)}-{Interlocked.Increment(ref _generation)}";
return system.ActorOf(ClusterClient.Props(settings), name);
}
/// <summary>
/// Reduces a cluster id to characters safe in an Akka actor name. The id is operator-authored
/// (e.g. a GUID or a slug), so it may carry characters Akka rejects; anything outside
/// <c>[A-Za-z0-9-]</c> becomes <c>-</c>. The trailing generation counter still guarantees
/// uniqueness, so a lossy sanitisation cannot cause a name collision.
/// </summary>
/// <param name="clusterId">The raw cluster id.</param>
/// <returns>An actor-name-safe rendering of the id.</returns>
private static string SanitizeForActorName(string clusterId)
{
var chars = clusterId.Select(c => char.IsAsciiLetterOrDigit(c) || c == '-' ? c : '-');
return new string(chars.ToArray());
}
}
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs
+5 -6
View File
@@ -56,11 +56,11 @@ public static class ServiceCollectionExtensions
// Per-cluster mesh Phase 2: the central end of the central↔node command boundary.
//
// A plain WithActors, NOT a singleton — and that is the load-bearing part. A driver node's
// ClusterClient rotates across contact points and must find a live comm actor at whichever
// central node answers. As a singleton it would exist on one central node only, and the
// rotation would fail silently against the other: acks would land sometimes, depending on
// which contact the client happened to settle on.
// A plain WithActors, NOT a singleton — and that is the load-bearing part. A driver node
// walks MeshTransport:CentralGrpcEndpoints in order and posts its ack to whichever central
// node answers first, so a live comm actor must exist on both. As a singleton it would exist
// on one central node only, and the other node's ack ingress would have nothing to forward
// to: acks would land sometimes, depending on which endpoint answered.
builder.WithActors((system, registry, resolver) =>
{
var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
@@ -85,7 +85,6 @@ public static class ServiceCollectionExtensions
CentralCommunicationActor.Props(
dbFactory,
options,
new DefaultMeshClusterClientFactory(),
// Resolved lazily, per message, so this actor never depends on the order in
// which Akka.Hosting materialises the singleton proxy relative to this block.
() => registry.TryGet<ConfigPublishCoordinatorKey>(out var coordinator)