feat(mesh): gRPC command fan-out behind Mode=Grpc
MeshGrpcCommandDialer places one NodeCommandService.Execute call on one node: a GrpcChannel cached per endpoint (the GrpcDeploymentArtifactFetcher shape), the TARGET node's cluster's bearer plus x-otopcua-cluster naming that cluster, and a per-call deadline linked to the caller's token. A cluster with no resolvable key is refused locally with Unauthenticated — there is no arm on which a command leaves central unauthenticated, even one the far side would have rejected. CentralCommunicationActor gains the third mode branch. The contact load now also projects GrpcPort and, from the SAME rows in the SAME pass, builds the dispatch set — which is what makes 'the gRPC fan reaches exactly the SendToAll recipient set' structural rather than two queries that have to keep agreeing. A row with a null GrpcPort is skipped warn-once (the TelemetryNodeSource precedent). Per-node results are piped back for LOGGING ONLY: named node, gRPC status, dropped. No retry, no queue. Every dial is scheduled on the thread pool rather than called from the receive. Found by review and pinned red-first: an async method runs synchronously to its first await, and the dialer's pre-await work is TryGetPsk, which blocks on the secret store — bounded but seconds, and repeated after every refresh since the refresh drops the cache. Dialing inline parked the actor's mailbox, so deployment acks queued behind a slow node; the regression test tells an ApplyAck while a dial is held and fails without the hop. The contact refresh also wires the Phase 1A register: it publishes the loaded cluster ids as the PSK provider's known-cluster set and drops the key cache, so a rotated key takes effect within one ContactRefreshSeconds. Neither happens under Dps or ClusterClient. Both existing mode branches are behaviourally untouched and every existing fact passes unmodified. The one edit to shared text: the Status.Failure warning drops the word 'ClusterClient', since that handler now serves all three transports. ControlPlane.Tests 205 -> 230, Host.Tests 92, Cluster.Tests 184, 0 warnings.
This commit is contained in:
Joseph Doherty
+417
-10
@@ -3,17 +3,24 @@ using Akka.Actor;
|
||||
using Akka.Cluster.Tools.Client;
|
||||
using Akka.Cluster.Tools.PublishSubscribe;
|
||||
using Akka.Event;
|
||||
using Grpc.Core;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||
// Grpc.Core (imported for RpcException on the dial path) declares a Status of its own. This file's
|
||||
// bare Status is Akka's — the Status.Failure a faulted piped task arrives as; the gRPC one is only
|
||||
// ever reached through an RpcException instance.
|
||||
using Status = Akka.Actor.Status;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
|
||||
|
||||
/// <summary>
|
||||
/// Central-side end of the central↔node command boundary (per-cluster mesh Phase 2). Routes
|
||||
/// every central→node command out — over DistributedPubSub or ClusterClient, per
|
||||
/// every central→node command out — over DistributedPubSub, ClusterClient or gRPC, per
|
||||
/// <see cref="MeshTransportOptions.Mode"/> — and forwards inbound <see cref="ApplyAck"/>s to the
|
||||
/// deploy coordinator singleton.
|
||||
/// </summary>
|
||||
@@ -31,6 +38,23 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
|
||||
/// and fans <c>SendToAll</c> across all of them. See <see cref="RebuildClient"/> — this is the
|
||||
/// single most important thing to understand before changing this actor.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>The gRPC branch fans over the same recipient set, node by node</b>
|
||||
/// (ClusterClient→gRPC Phase 1A). <c>SendToAll</c> reaches every node registered on a mesh's
|
||||
/// receptionist, and the contact load already selects exactly the enabled, non-maintenance
|
||||
/// rows those nodes come from; the gRPC branch dials that same row set directly, one call per
|
||||
/// node, so the recipient set is identical by construction rather than by coincidence. It is
|
||||
/// deliberately NOT narrower: the node side has no <c>ClusterId</c> or node filter at all —
|
||||
/// scoping happens later, inside the artifact — so a node left out here silently keeps its
|
||||
/// old configuration while the deployment seals green.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Fire-and-forget, on every transport.</b> The gRPC branch pipes per-node results back to
|
||||
/// itself for logging ONLY: a failure names the node and the status and is dropped. No retry,
|
||||
/// no queue, no ordering guarantee across nodes. That is not an omission — it is the
|
||||
/// ClusterClient semantic preserved exactly, and the backstops are unchanged (the deploy
|
||||
/// coordinator's apply deadline, and a node's boot-time replay).
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
||||
{
|
||||
@@ -40,8 +64,11 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
||||
private readonly MeshTransportOptions _options;
|
||||
private readonly IMeshClusterClientFactory _clientFactory;
|
||||
private readonly Func<IActorRef?> _coordinator;
|
||||
private readonly IMeshCommandDialer? _dialer;
|
||||
private readonly IMeshPskProvider? _pskProvider;
|
||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||
private readonly bool _useClusterClient;
|
||||
private readonly bool _useGrpc;
|
||||
|
||||
// One client per application Cluster, keyed by ClusterId. A ClusterClient reaches only the mesh
|
||||
// whose receptionist it dialled, and after Phase 6 every cluster is its own mesh — so central must
|
||||
@@ -49,6 +76,15 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
||||
private readonly Dictionary<string, (IActorRef Client, ImmutableHashSet<string> Contacts)> _clients =
|
||||
new();
|
||||
|
||||
// Node ids already warned about for a null GrpcPort. A persistently portless row would otherwise
|
||||
// produce one warning per refresh forever; same warn-once shape as TelemetryNodeSource. Actor
|
||||
// thread only.
|
||||
private readonly HashSet<string> _warnedPortlessNodes = new(StringComparer.Ordinal);
|
||||
|
||||
// The gRPC dispatch set, replaced wholesale on every contact load. Actor thread only; the fan
|
||||
// captures the list reference, so a refresh mid-fan cannot mutate a dispatch already under way.
|
||||
private IReadOnlyList<MeshNodeTarget> _targets = [];
|
||||
|
||||
private CancellationTokenSource _lifecycle = new();
|
||||
|
||||
/// <summary>Gets the timer scheduler driving the periodic contact refresh.</summary>
|
||||
@@ -63,46 +99,73 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
||||
/// an <see cref="IActorRef"/> so this actor never depends on registration order at wiring
|
||||
/// time — it resolves on first use and caches.
|
||||
/// </param>
|
||||
/// <param name="dialer">
|
||||
/// Places one command call on one node, under <see cref="MeshTransportOptions.ModeGrpc"/>.
|
||||
/// Optional so the two older transports need nothing new wired; a <see langword="null"/>
|
||||
/// dialer under <c>Grpc</c> is a mis-wiring and is reported loudly rather than silently
|
||||
/// dropping every command.
|
||||
/// </param>
|
||||
/// <param name="pskProvider">
|
||||
/// Central's per-cluster key lookup, told on every contact refresh which clusters exist (the
|
||||
/// known-cluster gate) and to drop its cache (so a rotated key takes effect within one
|
||||
/// refresh). Optional for the same reason as <paramref name="dialer"/>.
|
||||
/// </param>
|
||||
/// <returns>The props.</returns>
|
||||
public static Props Props(
|
||||
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
|
||||
MeshTransportOptions options,
|
||||
IMeshClusterClientFactory clientFactory,
|
||||
Func<IActorRef?> coordinator) =>
|
||||
Func<IActorRef?> coordinator,
|
||||
IMeshCommandDialer? dialer = null,
|
||||
IMeshPskProvider? pskProvider = null) =>
|
||||
Akka.Actor.Props.Create(() =>
|
||||
new CentralCommunicationActor(dbFactory, options, clientFactory, coordinator));
|
||||
new CentralCommunicationActor(
|
||||
dbFactory, options, clientFactory, coordinator, dialer, pskProvider));
|
||||
|
||||
/// <summary>Initializes a new instance of the <see cref="CentralCommunicationActor"/> class.</summary>
|
||||
/// <param name="dbFactory">Factory for the config database.</param>
|
||||
/// <param name="options">The bound mesh-transport options.</param>
|
||||
/// <param name="clientFactory">Creates the ClusterClient.</param>
|
||||
/// <param name="coordinator">Lazily resolves the deploy-coordinator singleton proxy.</param>
|
||||
/// <param name="dialer">Places one gRPC command call on one node; <c>Grpc</c> mode only.</param>
|
||||
/// <param name="pskProvider">Central's per-cluster key lookup; <c>Grpc</c> mode only.</param>
|
||||
public CentralCommunicationActor(
|
||||
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
|
||||
MeshTransportOptions options,
|
||||
IMeshClusterClientFactory clientFactory,
|
||||
Func<IActorRef?> coordinator)
|
||||
Func<IActorRef?> coordinator,
|
||||
IMeshCommandDialer? dialer = null,
|
||||
IMeshPskProvider? pskProvider = null)
|
||||
{
|
||||
_dbFactory = dbFactory;
|
||||
_options = options;
|
||||
_clientFactory = clientFactory;
|
||||
_coordinator = coordinator;
|
||||
_dialer = dialer;
|
||||
_pskProvider = pskProvider;
|
||||
_useClusterClient = string.Equals(
|
||||
options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
|
||||
_useGrpc = string.Equals(
|
||||
options.Mode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase);
|
||||
|
||||
Receive<MeshCommand>(HandleMeshCommand);
|
||||
Receive<ApplyAck>(HandleApplyAck);
|
||||
Receive<RefreshContacts>(_ => LoadContactsFromDb());
|
||||
Receive<ContactsLoaded>(HandleContactsLoaded);
|
||||
Receive<CommandDispatched>(HandleCommandDispatched);
|
||||
|
||||
// A faulted LoadContactsFromDb task is piped here as a Status.Failure. Without this handler
|
||||
// the failure is an unhandled message (debug level only) and the refresh fails silently —
|
||||
// an operator cannot distinguish "no nodes configured" from "the database is down", and the
|
||||
// contact set silently freezes at whatever it last held.
|
||||
//
|
||||
// Under Grpc that freeze extends to the known-cluster gate, deliberately: a failed load must
|
||||
// neither widen the gate nor narrow it, and the previous set is the last thing central
|
||||
// actually knew.
|
||||
Receive<Status.Failure>(f => _log.Warning(
|
||||
f.Cause,
|
||||
"Failed to load ClusterNode contact points; the mesh ClusterClient contact set was NOT "
|
||||
+ "refreshed and may be stale or empty. Commands may be dropped until the next refresh"));
|
||||
"Failed to load ClusterNode contact points; the mesh contact set was NOT refreshed and "
|
||||
+ "may be stale or empty. Commands may be dropped until the next refresh"));
|
||||
|
||||
Receive<SubscribeAck>(_ => { /* DPS subscribe confirmation */ });
|
||||
}
|
||||
@@ -110,7 +173,7 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
||||
/// <inheritdoc />
|
||||
protected override void PreStart()
|
||||
{
|
||||
if (!_useClusterClient)
|
||||
if (!_useClusterClient && !_useGrpc)
|
||||
{
|
||||
_log.Info(
|
||||
"Mesh transport is {Mode}: commands are published on DistributedPubSub and no "
|
||||
@@ -118,6 +181,19 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
||||
return;
|
||||
}
|
||||
|
||||
if (_useGrpc && _dialer is null)
|
||||
{
|
||||
// Not a fatal throw: the ack path (HandleApplyAck) is transport-independent and still
|
||||
// works, so taking the actor down would turn a command-plane mis-wiring into an ack
|
||||
// outage as well. Every dispatch logs its own error, so this cannot go unnoticed.
|
||||
_log.Error(
|
||||
"Mesh transport is Grpc but no command dialer was supplied to this actor; EVERY "
|
||||
+ "central→node command will be dropped. Check the ControlPlane registration wires "
|
||||
+ "IMeshCommandDialer");
|
||||
}
|
||||
|
||||
// The refresh drives more than ClusterClient contacts under Grpc: it is also what publishes
|
||||
// the dispatch target set and keeps the PSK provider's known-cluster gate current.
|
||||
Timers.StartPeriodicTimer(
|
||||
RefreshTimerKey,
|
||||
new RefreshContacts(),
|
||||
@@ -131,12 +207,23 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
||||
foreach (var (client, _) in _clients.Values) Context.Stop(client);
|
||||
_clients.Clear();
|
||||
|
||||
// Cancelling the lifecycle token also cancels every gRPC dial in flight. The dialer itself is
|
||||
// deliberately NOT disposed here: it is a container-owned singleton shared with anything else
|
||||
// that may dial, and an actor restart must not tear its channel pool down.
|
||||
_lifecycle.Cancel();
|
||||
_lifecycle.Dispose();
|
||||
}
|
||||
|
||||
private void HandleMeshCommand(MeshCommand cmd)
|
||||
{
|
||||
// Checked before the DPS arm: _useClusterClient is false under Grpc too, so the order is
|
||||
// what keeps a Grpc-mode command off DistributedPubSub.
|
||||
if (_useGrpc)
|
||||
{
|
||||
DispatchOverGrpc(cmd);
|
||||
return;
|
||||
}
|
||||
|
||||
if (!_useClusterClient)
|
||||
{
|
||||
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(cmd.Topic, cmd.Message));
|
||||
@@ -168,6 +255,199 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
||||
client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, cmd.Message));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Encodes one command once and fans it to every target node, one unary gRPC call each.
|
||||
/// </summary>
|
||||
/// <param name="cmd">The command envelope carrying the domain record.</param>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Encoded once, before the fan.</b> Not per node — partly because the wire form is
|
||||
/// identical for every recipient, but mostly because <c>AlarmCommand</c> has no
|
||||
/// correlation member of its own and the envelope mints one: minting inside the loop
|
||||
/// would give one operator action N unrelated correlation ids and make it untraceable.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Nothing captured from the actor.</b> Each dial is a static call taking only locals,
|
||||
/// so no per-node continuation can touch actor state off the actor's thread. Results come
|
||||
/// back as ordinary messages via <c>PipeTo</c> and are handled on the actor thread like
|
||||
/// anything else.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Every dial starts on the thread pool, not here.</b> An <c>async</c> method runs
|
||||
/// synchronously up to its first <c>await</c>, and the dialer's pre-await work includes
|
||||
/// <c>IMeshPskProvider.TryGetPsk</c>, which blocks on the secret store (bounded, but
|
||||
/// seconds) the first time a cluster is looked up — and again after every refresh, since
|
||||
/// the refresh drops the cache to pick up rotations. Calling straight into it would park
|
||||
/// this actor's mailbox, and deployment acks queue behind that mailbox.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private void DispatchOverGrpc(MeshCommand cmd)
|
||||
{
|
||||
var commandType = cmd.Message.GetType().Name;
|
||||
|
||||
var dialer = _dialer;
|
||||
if (dialer is null)
|
||||
{
|
||||
_log.Error(
|
||||
"Mesh transport is Grpc but no command dialer is wired — dropping {MessageType}. The "
|
||||
+ "command is not buffered and will not be retried",
|
||||
commandType);
|
||||
return;
|
||||
}
|
||||
|
||||
// Captured before the fan: a contact refresh between here and the last dial must not change
|
||||
// the recipient set of a dispatch already under way.
|
||||
var targets = _targets;
|
||||
if (targets.Count == 0)
|
||||
{
|
||||
// App-level drop, exactly as the ClusterClient arm does with no client. The command is
|
||||
// NOT queued and will NOT be retried: a deploy fails at the coordinator's apply deadline
|
||||
// naming the silent nodes, which is a better operator signal than a command arriving
|
||||
// minutes late against a config the DB already recorded as failed.
|
||||
_log.Warning(
|
||||
"No mesh gRPC targets — dropping {MessageType}. The last contact refresh produced no "
|
||||
+ "enabled, non-maintenance ClusterNode row carrying a GrpcPort in any cluster; the "
|
||||
+ "command is not buffered and will not be retried",
|
||||
commandType);
|
||||
return;
|
||||
}
|
||||
|
||||
CommandEnvelope envelope;
|
||||
try
|
||||
{
|
||||
envelope = MeshTransportDtoMapper.ToEnvelope(cmd.Message, new CorrelationId(Guid.NewGuid()));
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// The mapper refuses a payload the far side could not decode. Dropping here is the same
|
||||
// outcome the old transport gave for such a message (nothing on the node matched it), but
|
||||
// loud instead of silent.
|
||||
_log.Error(
|
||||
ex,
|
||||
"Cannot encode {MessageType} for the gRPC command plane; the command is dropped and "
|
||||
+ "was NOT delivered to any node",
|
||||
commandType);
|
||||
return;
|
||||
}
|
||||
|
||||
CancellationToken ct;
|
||||
try
|
||||
{
|
||||
ct = _lifecycle.Token;
|
||||
}
|
||||
catch (ObjectDisposedException)
|
||||
{
|
||||
return; // Stopping.
|
||||
}
|
||||
|
||||
var self = Self;
|
||||
var correlationId = envelope.CorrelationId;
|
||||
foreach (var target in targets)
|
||||
{
|
||||
var node = target;
|
||||
|
||||
// CancellationToken.None on Task.Run, deliberately: cancelling the SCHEDULING would make
|
||||
// the task complete as canceled, which PipeTo converts into a Status.Failure — the one
|
||||
// message this actor already reads as "the contact load failed". The lifecycle token is
|
||||
// passed into the dial instead, where it cancels the call and comes back as an ordinary
|
||||
// per-node outcome.
|
||||
Task.Run(
|
||||
() => DispatchOneAsync(dialer, node, envelope, commandType, correlationId, ct),
|
||||
CancellationToken.None)
|
||||
.PipeTo(self);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Dials one node and turns every possible outcome into a <see cref="CommandDispatched"/>.
|
||||
/// </summary>
|
||||
/// <param name="dialer">The dialer seam.</param>
|
||||
/// <param name="target">The node to dial.</param>
|
||||
/// <param name="envelope">The already-encoded command.</param>
|
||||
/// <param name="commandType">CLR type name of the command, for the log.</param>
|
||||
/// <param name="correlationId">Transport correlation, for the log.</param>
|
||||
/// <param name="ct">The actor's lifecycle token.</param>
|
||||
/// <returns>The outcome, always — this method does not fault.</returns>
|
||||
/// <remarks>
|
||||
/// Catching everything is what keeps the result on ONE channel. A faulting task piped to an
|
||||
/// actor arrives as <c>Status.Failure</c>, which this actor already handles as "the contact
|
||||
/// load failed" — so an unhandled dial fault would log a database-outage warning about a node
|
||||
/// that simply did not answer.
|
||||
/// </remarks>
|
||||
private static async Task<CommandDispatched> DispatchOneAsync(
|
||||
IMeshCommandDialer dialer,
|
||||
MeshNodeTarget target,
|
||||
CommandEnvelope envelope,
|
||||
string commandType,
|
||||
string correlationId,
|
||||
CancellationToken ct)
|
||||
{
|
||||
try
|
||||
{
|
||||
await dialer.DispatchAsync(target.Endpoint, envelope, target.ClusterId, ct)
|
||||
.ConfigureAwait(false);
|
||||
return new CommandDispatched(target.NodeId, commandType, correlationId, null, null);
|
||||
}
|
||||
catch (RpcException ex)
|
||||
{
|
||||
return new CommandDispatched(
|
||||
target.NodeId,
|
||||
commandType,
|
||||
correlationId,
|
||||
ex.StatusCode.ToString(),
|
||||
Truncate(ex.Status.Detail));
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Named rather than swept into the generic arm: this is the ordinary shape of a node that
|
||||
// did not answer in time, and "Unknown / TaskCanceledException" would send an operator
|
||||
// looking for a bug instead of at DispatchTimeoutSeconds.
|
||||
return new CommandDispatched(
|
||||
target.NodeId,
|
||||
commandType,
|
||||
correlationId,
|
||||
"Cancelled",
|
||||
"the dial did not complete within MeshTransport:DispatchTimeoutSeconds, or central "
|
||||
+ "is shutting down");
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// Type name only, never the message: this path also covers whatever the dialer's channel
|
||||
// construction can throw, and a message is the one place a misconfigured endpoint string
|
||||
// could carry something an operator did not mean to put in a log.
|
||||
return new CommandDispatched(
|
||||
target.NodeId, commandType, correlationId, "Unknown", ex.GetType().Name);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Bounds a status detail before it reaches the log. The string comes from the far side, and
|
||||
/// the far side is a node — trusted, but not a reason to let it choose how many lines of
|
||||
/// central's log one failed command occupies.
|
||||
/// </summary>
|
||||
/// <param name="detail">The received status detail.</param>
|
||||
/// <returns>The detail, at most 200 characters.</returns>
|
||||
private static string Truncate(string? detail)
|
||||
=> detail is null || detail.Length <= 200 ? detail ?? "" : detail[..200] + "…";
|
||||
|
||||
private void HandleCommandDispatched(CommandDispatched result)
|
||||
{
|
||||
if (result.FailureStatus is null)
|
||||
{
|
||||
_log.Debug(
|
||||
"Mesh: {CommandType} (correlationId={CorrelationId}) accepted by {NodeId} over gRPC",
|
||||
result.CommandType, result.CorrelationId, result.NodeId);
|
||||
return;
|
||||
}
|
||||
|
||||
_log.Warning(
|
||||
"Mesh: {CommandType} (correlationId={CorrelationId}) was NOT delivered to {NodeId} "
|
||||
+ "({Status}: {Detail}). It is dropped and will NOT be retried — the deployment apply "
|
||||
+ "deadline and the node's boot-time replay are the backstops",
|
||||
result.CommandType, result.CorrelationId, result.NodeId,
|
||||
result.FailureStatus, result.FailureDetail);
|
||||
}
|
||||
|
||||
private void HandleApplyAck(ApplyAck ack)
|
||||
{
|
||||
var coordinator = _coordinator();
|
||||
@@ -210,7 +490,7 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
||||
var rows = await db.ClusterNodes
|
||||
.AsNoTracking()
|
||||
.Where(n => n.Enabled && !n.MaintenanceMode)
|
||||
.Select(n => new { n.NodeId, n.ClusterId, n.Host, n.AkkaPort })
|
||||
.Select(n => new { n.NodeId, n.ClusterId, n.Host, n.AkkaPort, n.GrpcPort })
|
||||
.ToListAsync(ct)
|
||||
.ConfigureAwait(false);
|
||||
|
||||
@@ -219,8 +499,28 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
||||
// client.
|
||||
var byCluster = new Dictionary<string, List<string>>();
|
||||
var malformed = new List<string>();
|
||||
|
||||
// The gRPC dispatch set, built from the SAME row set in the SAME pass. Sharing the read is
|
||||
// what makes "the gRPC fan reaches exactly the ClusterClient recipient set" structural
|
||||
// rather than a claim two separate queries would have to keep agreeing on.
|
||||
var targets = new List<MeshNodeTarget>(rows.Count);
|
||||
var portless = new List<string>();
|
||||
|
||||
// Every cluster with at least one enabled, non-maintenance row — the known-cluster gate's
|
||||
// membership set. Deliberately independent of GrpcPort: the gate answers "is this a cluster
|
||||
// central knows", and a cluster central cannot currently dial can still send it an ack.
|
||||
var clusterIds = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
foreach (var row in rows)
|
||||
{
|
||||
clusterIds.Add(row.ClusterId);
|
||||
|
||||
if (row.GrpcPort is { } grpcPort)
|
||||
targets.Add(new MeshNodeTarget(
|
||||
row.NodeId, row.ClusterId, $"http://{row.Host}:{grpcPort}"));
|
||||
else
|
||||
portless.Add(row.NodeId);
|
||||
|
||||
var address = $"akka.tcp://{systemName}@{row.Host}:{row.AkkaPort}/system/receptionist";
|
||||
|
||||
// Parse up front, per row, inside the loop's own guard. A single malformed row must
|
||||
@@ -241,12 +541,19 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
||||
|
||||
var contactsByCluster = byCluster.ToDictionary(
|
||||
kv => kv.Key, kv => (IReadOnlyList<string>)kv.Value);
|
||||
return new ContactsLoaded(contactsByCluster, malformed);
|
||||
return new ContactsLoaded(
|
||||
contactsByCluster, malformed, targets, portless, clusterIds.ToList());
|
||||
}, ct).PipeTo(self);
|
||||
}
|
||||
|
||||
private void HandleContactsLoaded(ContactsLoaded msg)
|
||||
{
|
||||
if (_useGrpc)
|
||||
{
|
||||
HandleGrpcContactsLoaded(msg);
|
||||
return;
|
||||
}
|
||||
|
||||
foreach (var bad in msg.Malformed)
|
||||
{
|
||||
_log.Warning(
|
||||
@@ -305,6 +612,59 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Adopts a contact load under <see cref="MeshTransportOptions.ModeGrpc"/>: the dispatch
|
||||
/// target set, and the two things the PSK provider needs kept current.
|
||||
/// </summary>
|
||||
/// <param name="msg">The load result.</param>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The malformed-receptionist-address warnings are deliberately NOT logged here: nothing
|
||||
/// on this transport dials a receptionist, and reporting an unusable Akka address as a
|
||||
/// command-plane problem would send an operator after the wrong column. A host that is
|
||||
/// unusable as a gRPC endpoint surfaces instead as a named per-node dial failure on every
|
||||
/// dispatch, which is a truer signal.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b><c>InvalidateAll</c>, not per-cluster <c>Invalidate</c>.</b> The requirement is that
|
||||
/// a rotated key takes effect within one refresh, and only a whole-cache drop delivers
|
||||
/// that: a key can rotate without the cluster's membership changing at all, so invalidating
|
||||
/// only departed clusters would serve the pre-rotation key forever. The cost is one store
|
||||
/// read per cluster per refresh — the cache holds a single entry per cluster — and it is
|
||||
/// paid only by clusters that are actually dialled.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private void HandleGrpcContactsLoaded(ContactsLoaded msg)
|
||||
{
|
||||
foreach (var nodeId in msg.PortlessNodes)
|
||||
{
|
||||
// Warn-once per node id: a persistently portless row is re-skipped every refresh, and the
|
||||
// same repetition problem TelemetryNodeSource solved this way.
|
||||
if (_warnedPortlessNodes.Add(nodeId))
|
||||
{
|
||||
_log.Warning(
|
||||
"ClusterNode {NodeId} has no GrpcPort; it exposes no command endpoint and is "
|
||||
+ "skipped in this dispatch set (further skips of this node are silent). Commands "
|
||||
+ "will not reach it until the row carries a port", nodeId);
|
||||
}
|
||||
}
|
||||
|
||||
_targets = msg.Targets;
|
||||
|
||||
// Published AFTER the target set is adopted, so anything observing the gate can rely on the
|
||||
// dispatch set already being current.
|
||||
_pskProvider?.SetKnownClusters(msg.ClusterIds);
|
||||
_pskProvider?.InvalidateAll();
|
||||
|
||||
if (_targets.Count == 0)
|
||||
{
|
||||
_log.Warning(
|
||||
"No enabled, non-maintenance ClusterNode row carries a GrpcPort in any cluster; there "
|
||||
+ "are no gRPC command targets and every central→node command will be dropped until a "
|
||||
+ "refresh finds one");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Stops a cluster's existing client and creates a replacement for its new contact set.</summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
@@ -371,7 +731,54 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
||||
/// <c>Cluster</c>'s id. One <c>ClusterClient</c> is built per key.
|
||||
/// </param>
|
||||
/// <param name="Malformed">Rows that did not parse, described for the log.</param>
|
||||
/// <param name="Targets">
|
||||
/// The gRPC dispatch set: one entry per enabled, non-maintenance row that carries a
|
||||
/// <c>GrpcPort</c>. Built from the same rows as <paramref name="ContactsByCluster"/>.
|
||||
/// </param>
|
||||
/// <param name="PortlessNodes">
|
||||
/// Node ids skipped because their row has no <c>GrpcPort</c>. Carried rather than logged in
|
||||
/// the load task so the warn-once bookkeeping stays on the actor's thread.
|
||||
/// </param>
|
||||
/// <param name="ClusterIds">
|
||||
/// Every application <c>Cluster</c> with at least one enabled, non-maintenance row —
|
||||
/// the PSK provider's known-cluster gate membership.
|
||||
/// </param>
|
||||
public sealed record ContactsLoaded(
|
||||
IReadOnlyDictionary<string, IReadOnlyList<string>> ContactsByCluster,
|
||||
IReadOnlyList<string> Malformed);
|
||||
IReadOnlyList<string> Malformed,
|
||||
IReadOnlyList<MeshNodeTarget> Targets,
|
||||
IReadOnlyList<string> PortlessNodes,
|
||||
IReadOnlyList<string> ClusterIds);
|
||||
|
||||
/// <summary>One node the gRPC command plane can dial.</summary>
|
||||
/// <param name="NodeId">The <c>ClusterNode</c> row's id — what a failure log names.</param>
|
||||
/// <param name="ClusterId">
|
||||
/// The node's application <c>Cluster</c>. Selects the pre-shared key presented to it: a node
|
||||
/// dialled with another cluster's id is graded against the wrong key and refuses the command.
|
||||
/// </param>
|
||||
/// <param name="Endpoint">The h2c base address, <c>http://{Host}:{GrpcPort}</c>.</param>
|
||||
public sealed record MeshNodeTarget(string NodeId, string ClusterId, string Endpoint);
|
||||
|
||||
/// <summary>
|
||||
/// The outcome of one node's dial, piped back to the actor <b>for logging only</b>. Nothing
|
||||
/// retries, counts or waits on these — a deployment's completion is judged by the acks nodes
|
||||
/// send back, never by whether central's outbound call succeeded.
|
||||
/// </summary>
|
||||
/// <param name="NodeId">The node dialled.</param>
|
||||
/// <param name="CommandType">CLR type name of the command.</param>
|
||||
/// <param name="CorrelationId">The envelope's transport correlation.</param>
|
||||
/// <param name="FailureStatus">
|
||||
/// <see langword="null"/> on success; otherwise the gRPC status code's name, or
|
||||
/// <c>Unknown</c> for a non-gRPC fault.
|
||||
/// </param>
|
||||
/// <param name="FailureDetail">
|
||||
/// The gRPC status detail, or the exception's type name for a non-gRPC fault. Never a key,
|
||||
/// and never a raw exception message.
|
||||
/// </param>
|
||||
public sealed record CommandDispatched(
|
||||
string NodeId,
|
||||
string CommandType,
|
||||
string CorrelationId,
|
||||
string? FailureStatus,
|
||||
string? FailureDetail);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Grpc.Core;
|
||||
using Grpc.Net.Client;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
|
||||
|
||||
/// <summary>
|
||||
/// Places one central→node command call on one node's <c>NodeCommandService</c>
|
||||
/// (ClusterClient→gRPC migration Phase 1A). The seam <c>CentralCommunicationActor</c> fans
|
||||
/// across its target set, kept as an interface so the actor's tests never open a socket.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// One dial, one node, no retry and no fan-out of its own — the fan and its per-node failure
|
||||
/// logging belong to the actor, which is the component that knows what the recipient set is.
|
||||
/// </remarks>
|
||||
public interface IMeshCommandDialer
|
||||
{
|
||||
/// <summary>Delivers one command envelope to one node.</summary>
|
||||
/// <param name="nodeEndpoint">The node's gRPC base address, e.g. <c>http://site-a-1:4056</c>.</param>
|
||||
/// <param name="envelope">The wire envelope, already mapped from the domain record.</param>
|
||||
/// <param name="clusterId">
|
||||
/// The application <c>Cluster</c> the <i>target node</i> belongs to. Selects which pre-shared
|
||||
/// key is presented, and is echoed to the node in <c>x-otopcua-cluster</c>.
|
||||
/// </param>
|
||||
/// <param name="ct">Cancels the dial; a per-call deadline is applied on top of it.</param>
|
||||
/// <returns>A task completing when the node has accepted the command.</returns>
|
||||
/// <exception cref="RpcException">
|
||||
/// The node rejected the call, was unreachable, or the deadline expired — and also when no
|
||||
/// key could be established for <paramref name="clusterId"/>, which is refused locally with
|
||||
/// <see cref="StatusCode.Unauthenticated"/> rather than dialled unauthenticated.
|
||||
/// </exception>
|
||||
Task DispatchAsync(string nodeEndpoint, CommandEnvelope envelope, string clusterId, CancellationToken ct);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Fail-closed on a missing key, always.</b> There is no arm on which a command leaves
|
||||
/// central without the target cluster's bearer. A node's <c>MeshCommandAuthInterceptor</c>
|
||||
/// would reject an unauthenticated call anyway, but "the far side would have refused it" is
|
||||
/// not a reason to make the call — an unauthenticated dial is a credential-free command
|
||||
/// attempt against a node's control surface, and it should not exist even as a rejected one.
|
||||
/// The refusal is reported as an <see cref="RpcException"/> so the actor's per-node failure
|
||||
/// logging covers it on the same channel as a remote rejection; the operator response is
|
||||
/// identical either way (seed the cluster's key), and a second channel would need a second
|
||||
/// handler that could drift.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>One channel per endpoint, cached.</b> Same shape as
|
||||
/// <c>GrpcDeploymentArtifactFetcher</c>: a <see cref="GrpcChannel"/> owns a connection pool
|
||||
/// and creating one per call leaks sockets. The cache is keyed by the endpoint string
|
||||
/// ordinally — endpoints come from <c>ClusterNode.Host</c> + <c>GrpcPort</c>, so two spellings
|
||||
/// of one host would merely cost a second channel, never a wrong destination.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>The deadline is per call, not per fan.</b> A dispatch dials every node in the fleet;
|
||||
/// an unbounded call to one unreachable node would hold a task and a socket indefinitely
|
||||
/// while the deploy path's own 2-minute apply deadline ticked down.
|
||||
/// <see cref="MeshTransportOptions.DispatchTimeoutSeconds"/> bounds it, linked to the
|
||||
/// caller's token so actor shutdown still cancels everything in flight.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Registered as a singleton and disposed by the container. Disposal is one-way: a dial
|
||||
/// after it throws rather than silently repopulating the cache with channels nobody will
|
||||
/// ever dispose.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class MeshGrpcCommandDialer : IMeshCommandDialer, IDisposable
|
||||
{
|
||||
/// <summary>
|
||||
/// Header naming the cluster the presented bearer belongs to. Central's own inbound gate
|
||||
/// reads the same header on the ack path; sending it outbound keeps one convention and tells
|
||||
/// a node which key central graded itself against.
|
||||
/// </summary>
|
||||
private const string ClusterHeader = "x-otopcua-cluster";
|
||||
|
||||
private readonly IMeshPskProvider _pskProvider;
|
||||
private readonly int _dispatchTimeoutSeconds;
|
||||
private readonly Func<string, NodeCommandService.NodeCommandServiceClient> _clientFactory;
|
||||
private readonly ConcurrentDictionary<string, GrpcChannel> _channels = new(StringComparer.Ordinal);
|
||||
|
||||
private volatile bool _disposed;
|
||||
|
||||
/// <summary>Initializes a new instance of the <see cref="MeshGrpcCommandDialer"/> class.</summary>
|
||||
/// <param name="pskProvider">Supplies the bearer for the target node's cluster.</param>
|
||||
/// <param name="options">Mesh transport options; supplies the per-call deadline.</param>
|
||||
/// <param name="clientFactory">
|
||||
/// Seam for tests: builds the generated client for an endpoint. Defaults to a real h2c
|
||||
/// <see cref="GrpcChannel"/>, one cached per endpoint.
|
||||
/// </param>
|
||||
public MeshGrpcCommandDialer(
|
||||
IMeshPskProvider pskProvider,
|
||||
IOptions<MeshTransportOptions> options,
|
||||
Func<string, NodeCommandService.NodeCommandServiceClient>? clientFactory = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(pskProvider);
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
_pskProvider = pskProvider;
|
||||
_dispatchTimeoutSeconds = options.Value.DispatchTimeoutSeconds;
|
||||
_clientFactory = clientFactory ?? DefaultClientFactory;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task DispatchAsync(
|
||||
string nodeEndpoint, CommandEnvelope envelope, string clusterId, CancellationToken ct)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(nodeEndpoint);
|
||||
ArgumentNullException.ThrowIfNull(envelope);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
|
||||
ObjectDisposedException.ThrowIf(_disposed, this);
|
||||
|
||||
if (!_pskProvider.TryGetPsk(clusterId, out var psk))
|
||||
{
|
||||
// The cluster id is operator-authored configuration, not a secret, so naming it is what
|
||||
// makes this actionable. The key never appears here or anywhere else in this class.
|
||||
throw new RpcException(new Status(
|
||||
StatusCode.Unauthenticated,
|
||||
$"No mesh pre-shared key could be established for cluster '{clusterId}'; the dial was "
|
||||
+ "refused locally rather than made unauthenticated. Seed the secret "
|
||||
+ "OT-MESH-PSK-<clusterId> on this node, or set MeshTransport:ClusterPsks:<clusterId> "
|
||||
+ "on a rig."));
|
||||
}
|
||||
|
||||
var headers = new Metadata
|
||||
{
|
||||
{ "authorization", $"Bearer {psk}" },
|
||||
{ ClusterHeader, clusterId },
|
||||
};
|
||||
|
||||
using var deadlineCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
|
||||
if (_dispatchTimeoutSeconds > 0)
|
||||
deadlineCts.CancelAfter(TimeSpan.FromSeconds(_dispatchTimeoutSeconds));
|
||||
|
||||
var client = _clientFactory(nodeEndpoint);
|
||||
using var call = client.ExecuteAsync(envelope, headers, cancellationToken: deadlineCts.Token);
|
||||
await call.ResponseAsync.ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Dispose()
|
||||
{
|
||||
_disposed = true;
|
||||
|
||||
foreach (var channel in _channels.Values)
|
||||
channel.Dispose();
|
||||
_channels.Clear();
|
||||
}
|
||||
|
||||
private NodeCommandService.NodeCommandServiceClient DefaultClientFactory(string endpoint)
|
||||
{
|
||||
// h2c: GrpcChannel over an http:// address negotiates prior-knowledge HTTP/2.
|
||||
var channel = _channels.GetOrAdd(endpoint, GrpcChannel.ForAddress);
|
||||
return new NodeCommandService.NodeCommandServiceClient(channel);
|
||||
}
|
||||
}
|
||||
@@ -66,6 +66,15 @@ public static class ServiceCollectionExtensions
|
||||
var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
|
||||
var options = resolver.GetService<IOptions<MeshTransportOptions>>().Value;
|
||||
|
||||
// Through IServiceProvider rather than the Akka resolver, deliberately: these two are
|
||||
// OPTIONAL (Grpc mode only, and admin-role only within that), and GetService's
|
||||
// null-for-absent contract is the thing being relied on. The Akka resolver's
|
||||
// GetService<T> makes no such promise, and this same block runs in test harnesses whose
|
||||
// service collections register neither.
|
||||
var serviceProvider = resolver.GetService<IServiceProvider>();
|
||||
var dialer = serviceProvider.GetService<IMeshCommandDialer>();
|
||||
var pskProvider = serviceProvider.GetService<IMeshPskProvider>();
|
||||
|
||||
var actor = system.ActorOf(
|
||||
CentralCommunicationActor.Props(
|
||||
dbFactory,
|
||||
@@ -75,7 +84,9 @@ public static class ServiceCollectionExtensions
|
||||
// which Akka.Hosting materialises the singleton proxy relative to this block.
|
||||
() => registry.TryGet<ConfigPublishCoordinatorKey>(out var coordinator)
|
||||
? coordinator
|
||||
: null),
|
||||
: null,
|
||||
dialer,
|
||||
pskProvider),
|
||||
MeshPaths.CentralCommunicationName);
|
||||
|
||||
// Registered unconditionally, in both transport modes. Under Dps nothing dials it, and
|
||||
|
||||
@@ -534,6 +534,14 @@ if (hasAdmin)
|
||||
// Depends on ISecretResolver, which AddOtOpcUaSecrets registers unconditionally below (DI
|
||||
// registration order is irrelevant; resolution order is not).
|
||||
builder.Services.AddSingleton<IMeshPskProvider, MeshPskProvider>();
|
||||
// Central's outbound command dial (ClusterClient→gRPC Phase 1A). Admin-only for the same reason
|
||||
// as the provider above — a node never dials another node — and singleton because it owns a
|
||||
// GrpcChannel per node endpoint; per-call channels would leak sockets across a fleet-wide fan.
|
||||
// Registered in every mode: CentralCommunicationActor only reaches it under Mode=Grpc, and an
|
||||
// idle registration costs nothing, whereas gating it here would mean flipping the mode on a
|
||||
// running fleet needs a central restart before any command lands — the same reasoning that
|
||||
// registers the ClusterClient receptionist unconditionally.
|
||||
builder.Services.AddSingleton<IMeshCommandDialer, MeshGrpcCommandDialer>();
|
||||
}
|
||||
|
||||
// Registered unconditionally: driver-role nodes resolve Layer-B DriverConfig secrets and have no auth/DP/AdminUI.
|
||||
|
||||
Reference in New Issue
Block a user