feat(mesh): gRPC ApplyAck path behind Mode=Grpc

Task 1B.2 — the node→central half of the ClusterClient→gRPC migration.

GrpcApplyAckSender walks MeshTransport:CentralGrpcEndpoints in order and
STOPS AT THE FIRST SUCCESS. That single `return` is the whole single-count
guarantee: the transport it replaces was ClusterClient.Send, deliberately not
SendToAll, because both central nodes forward to the SAME
ConfigPublishCoordinator singleton proxy — an ack posted to both would have the
coordinator count one node twice and could seal a deployment on a doubled count.
The list is an availability list, not a fan-out list.

No retry beyond that one ordered pass, no queue: the receptionist buffered
nothing and retried nothing, and this is like-for-like, not an upgrade to
delivery guarantees. All endpoints failing is still a DROP — but a loud one,
naming the node, deployment, correlation id and the status each endpoint
answered with, which is what separates "central is down" from "central rejected
this node's key". The bearer never reaches a log; only endpoints and status
codes do. Per-endpoint GrpcChannel cache and the endpoint loop clone
GrpcDeploymentArtifactFetcher; the cluster id in x-otopcua-cluster comes from
this node's own cluster-<id> role via RoleParser, off the same bound
Cluster:Roles ClusterRoleInfo reads.

SendAsync never throws: the caller discards the task, so an escaping exception
would become an unobserved fault — a drop with no log line, the exact failure
mode this migration removes.

NodeCommunicationActor gains an OPTIONAL ackSender; every existing test compiles
and passes unedited. It is checked before the ClusterClient so the two can never
both fire. The registration arm creates no ClusterClient under Grpc.

NOTE — one wiring change beyond the actor: DriverHostActor's ackRouter is now
`useClusterClient || useGrpc ? nodeComm : null`. Left at ClusterClient-only, a
Grpc node's ack would have gone out on the DPS deployment-acks topic and never
reached the sender at all.
This commit is contained in:
Joseph Doherty
2026-08-06 05:48:29 -04:00
parent e1db90f9c9
commit 308c6aaa0f
6 changed files with 775 additions and 13 deletions
@@ -0,0 +1,203 @@
using System.Collections.Concurrent;
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication;
/// <summary>
/// Posts this node's <see cref="ApplyAck"/> to central's <c>CentralCommandService</c> as a unary
/// gRPC call (ClusterClient→gRPC migration Task 1B.2), the <c>MeshTransport:Mode = Grpc</c>
/// replacement for the ClusterClient <c>Send</c>.
/// </summary>
/// <remarks>
/// <para>
/// <b>One ordered pass, stopping at the first success.</b>
/// <see cref="MeshTransportOptions.CentralGrpcEndpoints"/> is an availability list, not a
/// fan-out list: either central node hands the ack to the same
/// <c>ConfigPublishCoordinator</c> singleton proxy, so posting to both would have the
/// coordinator count this node twice — and a doubled count can seal a deployment other nodes
/// never applied. That is exactly why the transport being replaced used
/// <c>ClusterClient.Send</c> and not <c>SendToAll</c>.
/// </para>
/// <para>
/// <b>No retry beyond that single pass, and no queue.</b> The receptionist buffered nothing
/// and retried nothing, and this is a like-for-like replacement, not an upgrade to delivery
/// guarantees. The deploy path's own apply deadline stays the only recovery mechanism; a
/// retry loop here would instead pile acks up behind an unreachable central and deliver a
/// burst of stale ones when it returned.
/// </para>
/// <para>
/// <b>All endpoints failing is a DROP — but a loud one.</b> That is the same outcome the
/// ClusterClient path had when it could not reach central; what changes is that the operator
/// now gets the node, the deployment, the correlation id and the status each endpoint
/// answered with, which is what distinguishes "central is down" from "central rejected this
/// node's key".
/// </para>
/// <para>
/// <b>The bearer never reaches a log.</b> It travels in the call headers on every attempt and
/// the failure log names endpoints and status codes only — a per-attempt dump of the
/// <see cref="Metadata"/> would write the cluster's pre-shared key into the node log of every
/// deployment that could not reach central.
/// </para>
/// <para>
/// <b>The cluster id comes from this node's own <c>cluster-&lt;id&gt;</c> Akka role</b>, read
/// through <see cref="RoleParser"/> off the same bound <c>Cluster:Roles</c> that
/// <c>ClusterRoleInfo</c> reads (the <c>OTOPCUA_ROLES</c> fallback is already applied to those
/// options by <c>AddOtOpcUaCluster</c>'s <c>PostConfigure</c>, so nothing is re-resolved
/// here). The two must agree: central grades the bearer against the key of the cluster named
/// in the header, so a node that authenticated as a different cluster than it gossips would be
/// rejected on every ack.
/// </para>
/// </remarks>
public sealed class GrpcApplyAckSender : IApplyAckSender, IDisposable
{
/// <summary>The cluster the caller claims to be; central grades the bearer against ITS key.</summary>
private const string ClusterHeader = "x-otopcua-cluster";
private readonly MeshTransportOptions _options;
private readonly string? _clusterId;
private readonly ILogger<GrpcApplyAckSender> _log;
private readonly Func<string, CentralCommandService.CentralCommandServiceClient> _clientFactory;
private readonly ConcurrentDictionary<string, GrpcChannel> _channels = new(StringComparer.Ordinal);
/// <summary>Initializes a new instance of the <see cref="GrpcApplyAckSender"/> class.</summary>
/// <param name="options">Mesh-transport options; supplies the endpoints, the PSK and the deadline.</param>
/// <param name="clusterOptions">
/// Cluster options, read only for this node's roles — the <c>cluster-&lt;id&gt;</c> one names
/// the cluster this node authenticates as.
/// </param>
/// <param name="log">Logger.</param>
/// <param name="clientFactory">
/// Seam for tests: builds the generated client for an endpoint. Defaults to a real
/// <see cref="GrpcChannel"/> (one cached per endpoint).
/// </param>
public GrpcApplyAckSender(
IOptions<MeshTransportOptions> options,
IOptions<AkkaClusterOptions> clusterOptions,
ILogger<GrpcApplyAckSender> log,
Func<string, CentralCommandService.CentralCommandServiceClient>? clientFactory = null)
{
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(clusterOptions);
ArgumentNullException.ThrowIfNull(log);
_options = options.Value;
_log = log;
_clusterId = RoleParser.ClusterIdFromRoles(clusterOptions.Value.Roles);
_clientFactory = clientFactory ?? DefaultClientFactory;
if (_clusterId is null)
{
// Said once, at construction, rather than on every ack. Central cannot grade a bearer it
// has no cluster for, so every ack this node posts will come back PermissionDenied — and
// without this line that reads as a key mismatch rather than a missing role.
_log.LogWarning(
"This node carries no '{Prefix}<ClusterId>' role, so its deployment acks will be "
+ "posted without the {Header} header and central will reject every one of them. Add "
+ "the cluster-scoped role to Cluster:Roles (or OTOPCUA_ROLES).",
RoleParser.ClusterRolePrefix, ClusterHeader);
}
}
/// <inheritdoc />
public async Task SendAsync(ApplyAck ack, CancellationToken cancellationToken = default)
{
ArgumentNullException.ThrowIfNull(ack);
try
{
await PostAsync(ack, cancellationToken);
}
catch (Exception ex)
{
// Deliberately unbounded, and deliberately not a rethrow. The caller discards this task
// (see IApplyAckSender), so anything escaping would be swallowed by the runtime and
// resurface as an unobserved exception at GC — a drop with no line in the node log, which
// is precisely the failure mode this transport replaces. Encoding faults and non-RpcException
// transport faults both land here.
_log.LogWarning(
ex,
"Dropping the ApplyAck for {DeploymentId} (correlation {CorrelationId}) from {NodeId}: "
+ "posting it to central failed unexpectedly. It is not buffered and will not be "
+ "retried, so the deployment will time out at central naming this node even though it "
+ "applied successfully.",
ack.DeploymentId, ack.CorrelationId, ack.NodeId);
}
}
private async Task PostAsync(ApplyAck ack, CancellationToken cancellationToken)
{
var endpoints = _options.CentralGrpcEndpoints;
if (endpoints.Length == 0)
{
_log.LogWarning(
"Dropping the ApplyAck for {DeploymentId} from {NodeId}: no "
+ "MeshTransport:CentralGrpcEndpoints are configured, so there is nowhere to post it. "
+ "The deployment will time out at central naming this node even though it applied.",
ack.DeploymentId, ack.NodeId);
return;
}
var message = MeshTransportDtoMapper.ToProto(ack);
var headers = new Metadata { { "authorization", $"Bearer {_options.GrpcPsk}" } };
if (_clusterId is not null)
headers.Add(ClusterHeader, _clusterId);
var deadline = _options.AckTimeoutSeconds > 0
? DateTime.UtcNow.AddSeconds(_options.AckTimeoutSeconds)
: (DateTime?)null;
var failures = new List<string>(endpoints.Length);
foreach (var endpoint in endpoints)
{
try
{
var client = _clientFactory(endpoint);
await client.ReportApplyAsync(message, headers, deadline, cancellationToken);
_log.LogDebug(
"Posted the ApplyAck for {DeploymentId} ({Outcome}) to {Endpoint}.",
ack.DeploymentId, ack.Outcome, endpoint);
// The single-count guarantee, and it is this `return`: the pass stops at the first
// endpoint that accepted the ack and never walks back.
return;
}
catch (RpcException ex)
{
// Status code only — never the exception's own message and never the headers, either
// of which could echo credential material back into the log.
failures.Add($"{endpoint} => {ex.StatusCode}");
}
}
_log.LogWarning(
"Dropping the ApplyAck for {DeploymentId} (correlation {CorrelationId}) from {NodeId}: all "
+ "{Count} central endpoint(s) refused it ({Failures}). It is not buffered and will not be "
+ "retried, so the deployment will time out at central naming this node even though it "
+ "applied successfully.",
ack.DeploymentId, ack.CorrelationId, ack.NodeId, endpoints.Length, string.Join("; ", failures));
}
private CentralCommandService.CentralCommandServiceClient DefaultClientFactory(string endpoint)
{
// h2c: GrpcChannel over an http:// address negotiates prior-knowledge HTTP/2. One channel per
// endpoint, cached and reused (creating a channel per ack would leak a socket per deployment).
var channel = _channels.GetOrAdd(endpoint, ep => GrpcChannel.ForAddress(ep));
return new CentralCommandService.CentralCommandServiceClient(channel);
}
/// <inheritdoc />
public void Dispose()
{
foreach (var channel in _channels.Values)
channel.Dispose();
_channels.Clear();
}
}
@@ -0,0 +1,36 @@
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication;
/// <summary>
/// Posts this node's deployment acknowledgment to central over the <c>mesh_transport.v1</c>
/// command plane (ClusterClient→gRPC migration Phase 1B). The seam
/// <see cref="NodeCommunicationActor"/> uses under <c>MeshTransport:Mode = Grpc</c>, in place of
/// the <c>ClusterClient.Send</c> it uses under <c>ClusterClient</c>.
/// </summary>
/// <remarks>
/// <para>
/// <b>One ack per node per dispatch.</b> Whatever the implementation does about redundancy, it
/// must deliver the ack to <i>at most one</i> central endpoint: both central nodes forward to
/// the same <c>ConfigPublishCoordinator</c> singleton proxy, so a fan-out would have the
/// coordinator count one node twice. That is the semantic of the <c>ClusterClient.Send</c>
/// this replaces — <c>Send</c>, deliberately not <c>SendToAll</c>.
/// </para>
/// <para>
/// <b>Fire-and-forget, and it never throws.</b> The caller is an actor that discards the
/// returned task, so an exception escaping here would become an unobserved task fault — the
/// silent drop this migration exists to remove. An implementation reports failure by logging
/// and returning, never by faulting the task.
/// </para>
/// </remarks>
public interface IApplyAckSender
{
/// <summary>Posts <paramref name="ack"/> to central, or logs why it could not be.</summary>
/// <param name="ack">The per-node apply acknowledgment.</param>
/// <param name="cancellationToken">Cancels the in-flight call.</param>
/// <returns>
/// A task that completes when the attempt finishes, successfully or not. It never faults —
/// see the remarks.
/// </returns>
Task SendAsync(ApplyAck ack, CancellationToken cancellationToken = default);
}
@@ -37,23 +37,31 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication;
public sealed class NodeCommunicationActor : ReceiveActor
{
private readonly IActorRef? _centralClient;
private readonly IApplyAckSender? _ackSender;
private readonly ILoggingAdapter _log = Context.GetLogger();
/// <summary>Creates the props for this actor.</summary>
/// <param name="centralClient">
/// ClusterClient dialling central's receptionist, or <see langword="null"/> under
/// <c>MeshTransport:Mode = Dps</c>, where acks still go out over DistributedPubSub from
/// <c>DriverHostActor</c> and this actor never sees them.
/// <c>DriverHostActor</c> and this actor never sees them — and under
/// <c>Mode = Grpc</c>, where no ClusterClient is created at all.
/// </param>
/// <param name="ackSender">
/// The gRPC ack transport under <c>MeshTransport:Mode = Grpc</c>, or <see langword="null"/>
/// under <c>Dps</c> and <c>ClusterClient</c>.
/// </param>
/// <returns>The props.</returns>
public static Props Props(IActorRef? centralClient) =>
Akka.Actor.Props.Create(() => new NodeCommunicationActor(centralClient));
public static Props Props(IActorRef? centralClient, IApplyAckSender? ackSender = null) =>
Akka.Actor.Props.Create(() => new NodeCommunicationActor(centralClient, ackSender));
/// <summary>Initializes a new instance of the <see cref="NodeCommunicationActor"/> class.</summary>
/// <param name="centralClient">ClusterClient dialling central, or <see langword="null"/>.</param>
public NodeCommunicationActor(IActorRef? centralClient)
/// <param name="ackSender">The gRPC ack transport, or <see langword="null"/>.</param>
public NodeCommunicationActor(IActorRef? centralClient, IApplyAckSender? ackSender = null)
{
_centralClient = centralClient;
_ackSender = ackSender;
// Inbound from central. Each type is listed explicitly rather than caught by a ReceiveAny:
// an unknown type must dead-letter loudly during development rather than be republished
@@ -75,13 +83,28 @@ public sealed class NodeCommunicationActor : ReceiveActor
private void HandleApplyAck(ApplyAck ack)
{
if (_ackSender is not null)
{
// Fire-and-forget by construction, exactly as the ClusterClient Tell below is. The sender
// never throws and never retries (see IApplyAckSender), so there is nothing to await and
// nothing to pipe back: awaiting would stall this mailbox behind a network call to central,
// and an Ask would invent a request/response the one-way contract does not have.
//
// Checked BEFORE the ClusterClient. The registration arm builds exactly one transport, so
// both are never set — but were they, posting over both would deliver two acks for one
// apply and the coordinator would count this node twice.
_ = _ackSender.SendAsync(ack);
return;
}
if (_centralClient is null)
{
_log.Warning(
"No central ClusterClient — dropping ApplyAck for {DeploymentId}. The ack is not "
"No central ack transport — dropping ApplyAck for {DeploymentId}. The ack is not "
+ "buffered and will not be retried, so the deployment will time out at central "
+ "naming this node even though it applied successfully. Check "
+ "MeshTransport:CentralContactPoints",
+ "MeshTransport:CentralContactPoints (ClusterClient mode) or "
+ "MeshTransport:CentralGrpcEndpoints (Grpc mode)",
ack.DeploymentId);
return;
}
@@ -1,6 +1,7 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Event;
using Akka.Hosting;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Configuration;
@@ -99,6 +100,26 @@ public static class ServiceCollectionExtensions
// enabled, so piggybacking would leave a redundant pair that historizes values but not alarms
// with no gate at all, and both nodes would write every sample twice with nothing to say so.
services.TryAddSingleton<IHistorizationRoleView>(sp => sp.GetRequiredService<RedundancyRoleView>());
// ClusterClient→gRPC migration Task 1B.2: the node→central ack transport under
// MeshTransport:Mode = Grpc. Registered unconditionally but resolved only by the Grpc arm of
// WithOtOpcUaRuntimeActors — the mode is not knowable here (this runs before configuration is
// read into options), and a singleton nobody resolves is never constructed, so a Dps node
// opens no channel.
//
// Registered in the CONTAINER rather than new-ed up beside the actor so its per-endpoint
// GrpcChannel cache is disposed at host shutdown; an actor-scoped instance would leak the
// channels on every restart.
//
// Built by a factory, not by type: MS.DI does not honour optional constructor parameters, so
// the clientFactory test seam would make type-based activation fail outright. The logger is
// resolved leniently for the same reason DriverHostActor's is — a harness that skipped logging
// should not fail to build the graph.
services.TryAddSingleton<IApplyAckSender>(sp => new GrpcApplyAckSender(
sp.GetRequiredService<IOptions<MeshTransportOptions>>(),
sp.GetRequiredService<IOptions<AkkaClusterOptions>>(),
sp.GetService<ILoggerFactory>()?.CreateLogger<GrpcApplyAckSender>()
?? NullLogger<GrpcApplyAckSender>.Instance));
return services;
}
@@ -389,6 +410,8 @@ public static class ServiceCollectionExtensions
var meshOptions = resolver.GetService<IOptions<MeshTransportOptions>>().Value;
var useClusterClient = string.Equals(
meshOptions.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
var useGrpc = string.Equals(
meshOptions.Mode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase);
IActorRef? centralClient = null;
if (useClusterClient)
@@ -406,12 +429,33 @@ public static class ServiceCollectionExtensions
CentralClusterClientName);
}
// ClusterClient→gRPC migration Task 1B.2. Deliberately EXCLUSIVE with the branch above —
// under Grpc no ClusterClient is created at all, which is half the point of the migration
// (it is the cross-WAN Akka association this program removes). The two are never both
// handed to the actor, and it prefers the sender if they ever were.
IApplyAckSender? ackSender = null;
if (useGrpc)
{
ackSender = resolver.GetService<IApplyAckSender>();
if (ackSender is null)
{
// Reachable only in a graph that skipped AddOtOpcUaRuntime (a test harness), never
// in the Host. Said here rather than left to the actor's per-ack drop warning,
// because the cause is a missing registration and the symptom would otherwise
// appear once per deployment with no hint of why.
system.Log.Warning(
"MeshTransport:Mode is {Mode} but no {Sender} is registered — every ApplyAck "
+ "this node produces will be dropped. AddOtOpcUaRuntime registers it.",
meshOptions.Mode, nameof(IApplyAckSender));
}
}
var nodeComm = system.ActorOf(
NodeCommunicationActor.Props(centralClient),
NodeCommunicationActor.Props(centralClient, ackSender),
MeshPaths.NodeCommunicationName);
// Registered with the receptionist in BOTH modes: under Dps nothing dials it, and an
// idle registration costs nothing — whereas registering only under ClusterClient would
// Registered with the receptionist in ALL modes: under Dps and Grpc nothing dials it, and
// an idle registration costs nothing — whereas registering only under ClusterClient would
// make flipping the flag require a node restart before central could reach it.
ClusterClientReceptionist.Get(system).RegisterService(nodeComm);
registry.Register<NodeCommunicationActorKey>(nodeComm);
@@ -523,9 +567,12 @@ public static class ServiceCollectionExtensions
: null;
var driverHost = system.ActorOf(
// ackRouter: under ClusterClient mode the node comm actor relays ApplyAcks across
// the boundary; under Dps it stays null and the host publishes on the
// deployment-acks topic exactly as before.
// ackRouter: under ClusterClient AND Grpc the node comm actor relays ApplyAcks across
// the boundary (which transport it then uses is its own business — the ClusterClient
// ref or the gRPC sender it was built with); under Dps it stays null and the host
// publishes on the deployment-acks topic exactly as before. Grpc MUST be included
// here: routed to DPS instead, a Grpc node's ack would never reach the sender at all
// and every deployment to it would time out while it applied perfectly.
// dbFactory is nullable (Phase 4): a driver-only node passes null here — ConfigDb is
// gated on hasAdmin and driver-only nodes are forced to FetchAndCache. DriverHostActor
// guards every ConfigDb-touching path on it (UpsertNodeDeploymentState no-ops; central
@@ -533,7 +580,7 @@ public static class ServiceCollectionExtensions
// alarmStateStore (there is no ConfigDb-backed EF fallback; the host skips the alarm
// host when no store was wired).
DriverHostActor.Props(dbFactory, roleInfo.LocalNode,
ackRouter: useClusterClient ? nodeComm : null,
ackRouter: useClusterClient || useGrpc ? nodeComm : null,
driverFactory: driverFactory, localRoles: roleInfo.LocalRoles,
dependencyMux: mux,
opcUaPublishActor: publishActor,