feat(comm): extract SiteCommandDispatcher; site serves commands over gRPC too (T1B.2)
Refactor SiteCommunicationActor's central→site routing table into one SiteCommandDispatcher — the single routing truth for the 28 migrated commands (IntegrationCallRequest, the dead 29th, stays on the actor and out of the dispatcher). The Akka actor and the new SiteCommandGrpcService both route through one dispatcher instance so the two transports can never drift on where a command goes. Server-side only: nothing central flips to gRPC yet (that is T1B.3); ClusterClient remains the live path. Decisions worth recording: - Targets preserved byte-for-byte. Lifecycle/OPC UA/query/route → the Deployment Manager singleton proxy; DeployArtifacts/EventLog/parked → their null-guarded handlers with the exact same "handler not available" replies; the parked handler stays NODE-LOCAL (per-node replicated-store owner), never the singleton proxy — pinned by a dispatcher test that asserts the target is the parked probe and NOT the dm proxy. - Sender preservation intact. The actor's command handlers became thin DispatchCommand delegations that still Forward (central Ask → reply routes straight back); the existing SiteCommunicationActorTests pass unchanged, which is the regression guard for that plumbing. UnsubscribeDebugView keeps its fire-and-forget shape: the actor Forwards, the gRPC service Tells + returns the synthetic UnsubscribeDebugViewAck so a unary RPC still answers. - Ack-before-Leave on failover. The dispatcher's PrepareFailover resolves the standby with a DRY-RUN (no leave) to build the ack, and hands back a deferred CommitLeave; the gRPC service returns the ack, then schedules the real Cluster.Leave — so a caller reaching the very node about to leave still gets its ack instead of a broken stream. The actor path keeps today's coupled resolve-and-leave (over ClusterClient the ack Tell only enqueues, so order is immaterial). Proven at both levels: a dispatcher test asserts the ack is built before CommitLeave runs, and a TestServer test asserts the recorded seam order is resolve-then-leave. - ControlPlaneAuthInterceptor gates SiteCommandService by EXTENDING DefaultGatedPrefixes (descriptor-derived), not by adding a constructor — the one-public-ctor invariant and its test stay green. Tests: SiteCommandDispatcherTests (28-command routing incl. parked node-locality and both failover paths) and SiteCommandGrpcService TestServer tests (auth, readiness→Unavailable, one command per oneof group, failover ordering). Full solution build 0/0; Communication.Tests 574 and Host.Tests 377 green. No active <Protobuf> item.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,227 @@
|
||||
using Akka.Actor;
|
||||
using Grpc.Core;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
|
||||
using GrpcStatus = Grpc.Core.Status;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// gRPC front door for the central→site command plane: decodes a proto command (via
|
||||
/// <see cref="SiteCommandDtoMapper"/>), routes it through the ONE
|
||||
/// <see cref="SiteCommandDispatcher"/> the Akka <c>SiteCommunicationActor</c> also uses, and
|
||||
/// encodes the reply. One routing truth, two transports.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Mapped in the site branch next to <c>SiteStreamGrpcServer</c>, behind the same
|
||||
/// <c>ControlPlaneAuthInterceptor</c> PSK gate and the same readiness convention: calls are
|
||||
/// rejected with <see cref="StatusCode.Unavailable"/> until <see cref="SetReady"/> is called
|
||||
/// once the site actor system is up (mirrors <c>SiteStreamGrpcServer.SetReady</c>).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Coexistence — server-side only.</b> This makes the site ALSO listen on gRPC for commands;
|
||||
/// nothing central flips to gRPC here (that is T1B.3). Central still dials sites via ClusterClient.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Fire-and-forget.</b> The Akka path never acks <c>UnsubscribeDebugView</c>, but a unary RPC
|
||||
/// must answer, so the dispatcher marks it <see cref="SiteCommandDispatcher.RouteDisposition.TellFireAndForget"/>
|
||||
/// and this service Tells the target then returns the synthetic ack.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Ack-before-Leave.</b> <c>TriggerFailover</c> resolves the standby WITHOUT leaving
|
||||
/// (<see cref="SiteCommandDispatcher.PrepareFailover"/>), returns the ack, and only THEN schedules
|
||||
/// the real <c>Cluster.Leave</c> — so a caller reaching the very node that is about to leave still
|
||||
/// receives its ack instead of a broken stream.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class SiteCommandGrpcService : SiteCommandService.SiteCommandServiceBase
|
||||
{
|
||||
// A local Ask has no client deadline of its own; fall back to a generous ceiling when the
|
||||
// caller set none (a deadline-less client is a test or an internal caller). When the caller
|
||||
// DID set a gRPC deadline, honour the remaining time instead.
|
||||
private static readonly TimeSpan DefaultAskTimeout = TimeSpan.FromMinutes(2);
|
||||
|
||||
private readonly ILogger<SiteCommandGrpcService> _logger;
|
||||
private readonly Action<Action> _leaveScheduler;
|
||||
|
||||
// Set once by SetReady after the site actor system and dispatcher exist. Read on Kestrel
|
||||
// threads, written on the host bring-up thread — volatile.
|
||||
private volatile SiteCommandDispatcher? _dispatcher;
|
||||
private volatile bool _ready;
|
||||
|
||||
/// <summary>DI constructor.</summary>
|
||||
/// <param name="logger">Logger for denial/dispatch diagnostics.</param>
|
||||
public SiteCommandGrpcService(ILogger<SiteCommandGrpcService> logger)
|
||||
: this(logger, DeferLeaveUntilAfterReply)
|
||||
{
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Test constructor letting a test observe when the deferred leave runs. Internal so DI sees a
|
||||
/// single public constructor.
|
||||
/// </summary>
|
||||
/// <param name="logger">Logger.</param>
|
||||
/// <param name="leaveScheduler">Runs the deferred <c>Cluster.Leave</c> after the ack is returned.</param>
|
||||
internal SiteCommandGrpcService(ILogger<SiteCommandGrpcService> logger, Action<Action> leaveScheduler)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(logger);
|
||||
ArgumentNullException.ThrowIfNull(leaveScheduler);
|
||||
_logger = logger;
|
||||
_leaveScheduler = leaveScheduler;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Marks the service ready and injects the shared routing table. Called once the site actor
|
||||
/// system, the Deployment Manager singleton, and the local handlers are up — the same point
|
||||
/// <c>SiteStreamGrpcServer.SetReady</c> is called.
|
||||
/// </summary>
|
||||
/// <param name="dispatcher">The shared dispatcher (the same instance the actor routes through).</param>
|
||||
public void SetReady(SiteCommandDispatcher dispatcher)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(dispatcher);
|
||||
_dispatcher = dispatcher;
|
||||
_ready = true;
|
||||
}
|
||||
|
||||
/// <summary>Whether the service is accepting commands. Exposed for tests.</summary>
|
||||
internal bool IsReady => _ready;
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async Task<LifecycleReply> ExecuteLifecycle(LifecycleRequest request, ServerCallContext context)
|
||||
{
|
||||
var command = SiteCommandDtoMapper.FromLifecycleRequest(request);
|
||||
var reply = await DispatchAsync(command, context).ConfigureAwait(false);
|
||||
return SiteCommandDtoMapper.ToLifecycleReply(reply);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async Task<OpcUaReply> ExecuteOpcUa(OpcUaRequest request, ServerCallContext context)
|
||||
{
|
||||
var command = SiteCommandDtoMapper.FromOpcUaRequest(request);
|
||||
var reply = await DispatchAsync(command, context).ConfigureAwait(false);
|
||||
return SiteCommandDtoMapper.ToOpcUaReply(reply);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async Task<QueryReply> ExecuteQuery(QueryRequest request, ServerCallContext context)
|
||||
{
|
||||
var command = SiteCommandDtoMapper.FromQueryRequest(request);
|
||||
var reply = await DispatchAsync(command, context).ConfigureAwait(false);
|
||||
return SiteCommandDtoMapper.ToQueryReply(reply);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async Task<ParkedReply> ExecuteParked(ParkedRequest request, ServerCallContext context)
|
||||
{
|
||||
var command = SiteCommandDtoMapper.FromParkedRequest(request);
|
||||
var reply = await DispatchAsync(command, context).ConfigureAwait(false);
|
||||
return SiteCommandDtoMapper.ToParkedReply(reply);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async Task<RouteReply> ExecuteRoute(RouteRequest request, ServerCallContext context)
|
||||
{
|
||||
var command = SiteCommandDtoMapper.FromRouteRequest(request);
|
||||
var reply = await DispatchAsync(command, context).ConfigureAwait(false);
|
||||
return SiteCommandDtoMapper.ToRouteReply(reply);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task<SiteFailoverAckDto> TriggerFailover(TriggerSiteFailoverDto request, ServerCallContext context)
|
||||
{
|
||||
var dispatcher = EnsureReady();
|
||||
var msg = SiteCommandDtoMapper.FromProto(request);
|
||||
|
||||
// Resolve the standby and build the ack WITHOUT leaving. The real leave is deferred so the
|
||||
// ack is on the wire first (ack-before-Leave).
|
||||
var outcome = dispatcher.PrepareFailover(msg);
|
||||
var dto = SiteCommandDtoMapper.ToProto(outcome.Ack);
|
||||
|
||||
if (outcome.CommitLeave is not null)
|
||||
{
|
||||
_leaveScheduler(outcome.CommitLeave);
|
||||
}
|
||||
|
||||
return Task.FromResult(dto);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Routes a decoded command through the shared dispatcher and returns the reply record.
|
||||
/// Forward → local <c>Ask</c>; fire-and-forget → local <c>Tell</c> + synthetic ack; no target →
|
||||
/// the dispatcher's synthetic "handler not available" reply.
|
||||
/// </summary>
|
||||
private async Task<object> DispatchAsync(object command, ServerCallContext context)
|
||||
{
|
||||
var dispatcher = EnsureReady();
|
||||
var route = dispatcher.ResolveRoute(command);
|
||||
|
||||
switch (route.Disposition)
|
||||
{
|
||||
case SiteCommandDispatcher.RouteDisposition.ImmediateReply:
|
||||
return route.Reply!;
|
||||
|
||||
case SiteCommandDispatcher.RouteDisposition.TellFireAndForget:
|
||||
route.Target!.Tell(command);
|
||||
return route.Reply!;
|
||||
|
||||
default:
|
||||
try
|
||||
{
|
||||
return await route.Target!.Ask<object>(
|
||||
command, AskTimeout(context), context.CancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (RpcException)
|
||||
{
|
||||
throw;
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
// Client cancelled or the deadline elapsed.
|
||||
throw new RpcException(new GrpcStatus(
|
||||
StatusCode.DeadlineExceeded, "Site did not answer within the deadline."));
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"Local dispatch of {Command} faulted", command.GetType().Name);
|
||||
throw new RpcException(new GrpcStatus(StatusCode.Internal, ex.Message));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private SiteCommandDispatcher EnsureReady()
|
||||
{
|
||||
var dispatcher = _dispatcher;
|
||||
if (!_ready || dispatcher is null)
|
||||
{
|
||||
throw new RpcException(new GrpcStatus(
|
||||
StatusCode.Unavailable, "Site command plane not ready."));
|
||||
}
|
||||
return dispatcher;
|
||||
}
|
||||
|
||||
private static TimeSpan AskTimeout(ServerCallContext context)
|
||||
{
|
||||
var deadline = context.Deadline;
|
||||
if (deadline == DateTime.MaxValue)
|
||||
{
|
||||
return DefaultAskTimeout;
|
||||
}
|
||||
|
||||
var remaining = deadline - DateTime.UtcNow;
|
||||
return remaining > TimeSpan.Zero ? remaining : TimeSpan.FromMilliseconds(1);
|
||||
}
|
||||
|
||||
// Default deferred-leave: return control (so the ack serialises) before the graceful Leave
|
||||
// runs. A yield hands the current continuation back before the leave begins; the leave itself
|
||||
// is slow-async (member marked Leaving, CoordinatedShutdown over seconds), so the ack is long
|
||||
// gone by the time Kestrel actually stops.
|
||||
private static void DeferLeaveUntilAfterReply(Action commitLeave)
|
||||
=> _ = Task.Run(async () =>
|
||||
{
|
||||
await Task.Yield();
|
||||
commitLeave();
|
||||
});
|
||||
}
|
||||
Reference in New Issue
Block a user