dohertj2/lmxopcua
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(mesh): gRPC ApplyAck path behind Mode=Grpc

Browse Source 
Task 1B.2 — the node→central half of the ClusterClient→gRPC migration.

GrpcApplyAckSender walks MeshTransport:CentralGrpcEndpoints in order and
STOPS AT THE FIRST SUCCESS. That single `return` is the whole single-count
guarantee: the transport it replaces was ClusterClient.Send, deliberately not
SendToAll, because both central nodes forward to the SAME
ConfigPublishCoordinator singleton proxy — an ack posted to both would have the
coordinator count one node twice and could seal a deployment on a doubled count.
The list is an availability list, not a fan-out list.

No retry beyond that one ordered pass, no queue: the receptionist buffered
nothing and retried nothing, and this is like-for-like, not an upgrade to
delivery guarantees. All endpoints failing is still a DROP — but a loud one,
naming the node, deployment, correlation id and the status each endpoint
answered with, which is what separates "central is down" from "central rejected
this node's key". The bearer never reaches a log; only endpoints and status
codes do. Per-endpoint GrpcChannel cache and the endpoint loop clone
GrpcDeploymentArtifactFetcher; the cluster id in x-otopcua-cluster comes from
this node's own cluster-<id> role via RoleParser, off the same bound
Cluster:Roles ClusterRoleInfo reads.

SendAsync never throws: the caller discards the task, so an escaping exception
would become an unobserved fault — a drop with no log line, the exact failure
mode this migration removes.

NodeCommunicationActor gains an OPTIONAL ackSender; every existing test compiles
and passes unedited. It is checked before the ClusterClient so the two can never
both fire. The registration arm creates no ClusterClient under Grpc.

NOTE — one wiring change beyond the actor: DriverHostActor's ackRouter is now
`useClusterClient || useGrpc ? nodeComm : null`. Left at ClusterClient-only, a
Grpc node's ack would have gone out on the DPS deployment-acks topic and never
reached the sender at all.
This commit is contained in:
Joseph Doherty
2026-08-06 05:48:29 -04:00
parent e1db90f9c9
commit 308c6aaa0f
6 changed files with 775 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs
+54 -7
View File
@@ -1,6 +1,7 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Event;
using Akka.Hosting;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Configuration;
@@ -99,6 +100,26 @@ public static class ServiceCollectionExtensions
// enabled, so piggybacking would leave a redundant pair that historizes values but not alarms
// with no gate at all, and both nodes would write every sample twice with nothing to say so.
services.TryAddSingleton<IHistorizationRoleView>(sp => sp.GetRequiredService<RedundancyRoleView>());
// ClusterClient→gRPC migration Task 1B.2: the node→central ack transport under
// MeshTransport:Mode = Grpc. Registered unconditionally but resolved only by the Grpc arm of
// WithOtOpcUaRuntimeActors — the mode is not knowable here (this runs before configuration is
// read into options), and a singleton nobody resolves is never constructed, so a Dps node
// opens no channel.
//
// Registered in the CONTAINER rather than new-ed up beside the actor so its per-endpoint
// GrpcChannel cache is disposed at host shutdown; an actor-scoped instance would leak the
// channels on every restart.
//
// Built by a factory, not by type: MS.DI does not honour optional constructor parameters, so
// the clientFactory test seam would make type-based activation fail outright. The logger is
// resolved leniently for the same reason DriverHostActor's is — a harness that skipped logging
// should not fail to build the graph.
services.TryAddSingleton<IApplyAckSender>(sp => new GrpcApplyAckSender(
sp.GetRequiredService<IOptions<MeshTransportOptions>>(),
sp.GetRequiredService<IOptions<AkkaClusterOptions>>(),
sp.GetService<ILoggerFactory>()?.CreateLogger<GrpcApplyAckSender>()
?? NullLogger<GrpcApplyAckSender>.Instance));
return services;
}
@@ -389,6 +410,8 @@ public static class ServiceCollectionExtensions
var meshOptions = resolver.GetService<IOptions<MeshTransportOptions>>().Value;
var useClusterClient = string.Equals(
meshOptions.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
var useGrpc = string.Equals(
meshOptions.Mode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase);
IActorRef? centralClient = null;
if (useClusterClient)
@@ -406,12 +429,33 @@ public static class ServiceCollectionExtensions
CentralClusterClientName);
}
// ClusterClient→gRPC migration Task 1B.2. Deliberately EXCLUSIVE with the branch above —
// under Grpc no ClusterClient is created at all, which is half the point of the migration
// (it is the cross-WAN Akka association this program removes). The two are never both
// handed to the actor, and it prefers the sender if they ever were.
IApplyAckSender? ackSender = null;
if (useGrpc)
{
ackSender = resolver.GetService<IApplyAckSender>();
if (ackSender is null)
{
// Reachable only in a graph that skipped AddOtOpcUaRuntime (a test harness), never
// in the Host. Said here rather than left to the actor's per-ack drop warning,
// because the cause is a missing registration and the symptom would otherwise
// appear once per deployment with no hint of why.
system.Log.Warning(
"MeshTransport:Mode is {Mode} but no {Sender} is registered — every ApplyAck "
+ "this node produces will be dropped. AddOtOpcUaRuntime registers it.",
meshOptions.Mode, nameof(IApplyAckSender));
}
}
var nodeComm = system.ActorOf(
NodeCommunicationActor.Props(centralClient),
NodeCommunicationActor.Props(centralClient, ackSender),
MeshPaths.NodeCommunicationName);
// Registered with the receptionist in BOTH modes: under Dps nothing dials it, and an
// idle registration costs nothing — whereas registering only under ClusterClient would
// Registered with the receptionist in ALL modes: under Dps and Grpc nothing dials it, and
// an idle registration costs nothing — whereas registering only under ClusterClient would
// make flipping the flag require a node restart before central could reach it.
ClusterClientReceptionist.Get(system).RegisterService(nodeComm);
registry.Register<NodeCommunicationActorKey>(nodeComm);
@@ -523,9 +567,12 @@ public static class ServiceCollectionExtensions
: null;
var driverHost = system.ActorOf(
// ackRouter: under ClusterClient mode the node comm actor relays ApplyAcks across
// the boundary; under Dps it stays null and the host publishes on the
// deployment-acks topic exactly as before.
// ackRouter: under ClusterClient AND Grpc the node comm actor relays ApplyAcks across
// the boundary (which transport it then uses is its own business — the ClusterClient
// ref or the gRPC sender it was built with); under Dps it stays null and the host
// publishes on the deployment-acks topic exactly as before. Grpc MUST be included
// here: routed to DPS instead, a Grpc node's ack would never reach the sender at all
// and every deployment to it would time out while it applied perfectly.
// dbFactory is nullable (Phase 4): a driver-only node passes null here — ConfigDb is
// gated on hasAdmin and driver-only nodes are forced to FetchAndCache. DriverHostActor
// guards every ConfigDb-touching path on it (UpsertNodeDeploymentState no-ops; central
@@ -533,7 +580,7 @@ public static class ServiceCollectionExtensions
// alarmStateStore (there is no ConfigDb-backed EF fallback; the host skips the alarm
// host when no store was wired).
DriverHostActor.Props(dbFactory, roleInfo.LocalNode,
ackRouter: useClusterClient ? nodeComm : null,
ackRouter: useClusterClient || useGrpc ? nodeComm : null,
driverFactory: driverFactory, localRoles: roleInfo.LocalRoles,
dependencyMux: mux,
opcUaPublishActor: publishActor,