Files
ScadaBridge/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationService.cs
T
Joseph Doherty c8e2f4da02 feat(cluster): site-pair manual failover relayed from the central UI (Task 10)
Central and each site are SEPARATE Akka clusters, so central cannot act on a
site's membership -- it asks. New TriggerSiteFailover/SiteFailoverAck contract
travels the existing ClusterClient command/control channel (mirroring the
RetryParkedOperation relay); the site's own SiteCommunicationActor performs the
graceful Leave and acks the outcome.

- ClusterFailoverCoordinator moved out of Host into Communication/ClusterState,
  beside ActiveNodeEvaluator. Both paths now share ONE oldest-Up implementation;
  SiteCommunicationActor cannot reference Host, and the two definitions must not
  drift or the node asked to leave stops being the singleton host.
- Site scope is the SITE-SPECIFIC role (site-{SiteId}), not the base Site role --
  site singletons are placed on the former, so the base role would move the wrong
  node. Pinned by a unit test asserting the role string and by a real-cluster test.
- Site-side guards: refuses a command addressed to another site (a misroute must
  never fail over a site the operator did not select), refuses when there is no
  peer, and reports a fault as an ack rather than throwing into supervision --
  a restart there would drop central's Ask into a bare timeout and lose the reason.
- Ack is sent before the Leave takes effect so it still reaches central.
- UI: the same control now serves both scopes via a SiteId parameter. The site
  confirmation deliberately does NOT claim the admin's page will disconnect --
  it won't, and crying wolf there devalues the central warning that is real. A
  site refusal and an unreachable site surface distinctly.
- Rolling upgrade: a site on an older binary has no handler, so the message
  dead-letters and the Ask times out, reported as "site did not respond". That
  is the honest outcome; documented on the contract.

Fallout fixed: HealthPageTests now renders the page inside
CascadingAuthenticationState with the real policy set and IAuthorizationService,
because the cards embed an AuthorizeView. That mirrors production, where the
layout supplies the cascading value.
2026-07-22 07:48:56 -04:00

847 lines
40 KiB
C#

using Akka.Actor;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Integration;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Lifecycle;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
namespace ZB.MOM.WW.ScadaBridge.Communication;
/// <summary>
/// Central-side service that wraps the Akka Ask pattern with per-pattern timeouts.
/// Provides a typed API for sending messages to sites and awaiting responses.
/// On connection drop, the ask times out (no central buffering per design).
/// </summary>
public class CommunicationService
{
private readonly CommunicationOptions _options;
private readonly ILogger<CommunicationService> _logger;
private IActorRef? _centralCommunicationActor;
private IActorRef? _notificationOutboxProxy;
private IActorRef? _siteCallAuditProxy;
/// <summary>
/// Initializes a new instance of the CommunicationService.
/// </summary>
/// <param name="options">Communication service configuration options.</param>
/// <param name="logger">Logger instance.</param>
public CommunicationService(
IOptions<CommunicationOptions> options,
ILogger<CommunicationService> logger)
{
_options = options.Value;
_logger = logger;
}
/// <summary>
/// Sets the central communication actor reference. Called during actor system startup.
/// </summary>
/// <param name="centralCommunicationActor">The central communication actor reference.</param>
public void SetCommunicationActor(IActorRef centralCommunicationActor)
{
_centralCommunicationActor = centralCommunicationActor;
}
/// <summary>
/// Sets the notification-outbox singleton proxy reference. Called during actor
/// system startup. The outbox actor is central-local, so outbox calls Ask this
/// proxy directly (no SiteEnvelope routing).
/// </summary>
/// <param name="notificationOutboxProxy">The notification outbox proxy reference.</param>
public void SetNotificationOutbox(IActorRef notificationOutboxProxy)
{
_notificationOutboxProxy = notificationOutboxProxy;
}
/// <summary>
/// Sets the Site Call Audit singleton proxy reference. Called during
/// actor system startup. The Site Call Audit actor is central-local, so Site
/// Calls read calls Ask this proxy directly (no SiteEnvelope routing), the
/// same pattern as <see cref="SetNotificationOutbox"/>.
/// </summary>
/// <param name="siteCallAuditProxy">The Site Call Audit proxy reference.</param>
public void SetSiteCallAudit(IActorRef siteCallAuditProxy)
{
_siteCallAuditProxy = siteCallAuditProxy;
}
/// <summary>
/// Triggers an immediate refresh of the site address cache from the database.
/// </summary>
public void RefreshSiteAddresses()
{
GetActor().Tell(new RefreshSiteAddresses());
}
/// <summary>
/// Gets the central communication actor reference. Throws if not yet initialized.
/// </summary>
/// <returns>The <see cref="IActorRef"/> for the central communication actor.</returns>
public IActorRef GetCommunicationActor()
{
return _centralCommunicationActor
?? throw new InvalidOperationException("CommunicationService not initialized. CentralCommunicationActor not set.");
}
private IActorRef GetActor() => GetCommunicationActor();
/// <summary>
/// Gets the notification-outbox proxy reference. Throws if not yet initialized.
/// </summary>
private IActorRef GetNotificationOutbox()
{
return _notificationOutboxProxy
?? throw new InvalidOperationException("CommunicationService not initialized. NotificationOutbox proxy not set.");
}
/// <summary>
/// Gets the Site Call Audit proxy reference. Throws if not yet initialized.
/// </summary>
private IActorRef GetSiteCallAudit()
{
return _siteCallAuditProxy
?? throw new InvalidOperationException("CommunicationService not initialized. SiteCallAudit proxy not set.");
}
// ── Pattern 1: Instance Deployment ──
/// <summary>
/// Sends a small "refresh deployment" notify to a site (notify-and-fetch):
/// the site fetches the config over HTTP rather than receiving it inline.
/// Reply is the existing DeploymentStatusResponse, bounded by the deployment timeout.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="command">The refresh-deployment notify.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The deployment status response.</returns>
public async Task<DeploymentStatusResponse> RefreshDeploymentAsync(
string siteId, RefreshDeploymentCommand command, CancellationToken cancellationToken = default)
{
_logger.LogInformation(
"Sending RefreshDeploymentCommand to site {SiteId}, instance={Instance}, deploymentId={DeploymentId}",
siteId, command.InstanceUniqueName, command.DeploymentId);
var envelope = new SiteEnvelope(siteId, command);
return await GetActor().Ask<DeploymentStatusResponse>(
envelope, _options.DeploymentTimeout, cancellationToken);
}
/// <summary>
/// Queries a site for the currently-applied deployment
/// identity of a single instance. Used by the Deployment Manager before a
/// re-deploy to reconcile against the site's actual state. Sent over the
/// existing ClusterClient command/control transport; the Ask times out (no
/// central buffering) if the site is unreachable, and the caller falls
/// through to a normal deploy.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="request">The deployment state query request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The deployment state query response.</returns>
public async Task<DeploymentStateQueryResponse> QueryDeploymentStateAsync(
string siteId, DeploymentStateQueryRequest request, CancellationToken cancellationToken = default)
{
_logger.LogDebug(
"Sending DeploymentStateQueryRequest to site {SiteId}, instance={Instance}, correlationId={CorrelationId}",
siteId, request.InstanceUniqueName, request.CorrelationId);
var envelope = new SiteEnvelope(siteId, request);
return await GetActor().Ask<DeploymentStateQueryResponse>(
envelope, _options.QueryTimeout, cancellationToken);
}
// ── Pattern 2: Lifecycle ──
/// <summary>
/// Sends a disable command for an instance to a site.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="command">The disable command.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The instance lifecycle response.</returns>
public async Task<InstanceLifecycleResponse> DisableInstanceAsync(
string siteId, DisableInstanceCommand command, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, command);
return await GetActor().Ask<InstanceLifecycleResponse>(
envelope, _options.LifecycleTimeout, cancellationToken);
}
/// <summary>
/// Sends an enable command for an instance to a site.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="command">The enable command.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The instance lifecycle response.</returns>
public async Task<InstanceLifecycleResponse> EnableInstanceAsync(
string siteId, EnableInstanceCommand command, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, command);
return await GetActor().Ask<InstanceLifecycleResponse>(
envelope, _options.LifecycleTimeout, cancellationToken);
}
/// <summary>
/// Sends a delete command for an instance to a site.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="command">The delete command.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The instance lifecycle response.</returns>
public async Task<InstanceLifecycleResponse> DeleteInstanceAsync(
string siteId, DeleteInstanceCommand command, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, command);
return await GetActor().Ask<InstanceLifecycleResponse>(
envelope, _options.LifecycleTimeout, cancellationToken);
}
// ── Pattern 3: Artifact Deployment ──
/// <summary>
/// Sends a system-wide artifact deployment command to a site.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="command">The artifact deployment command.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The artifact deployment response.</returns>
public async Task<ArtifactDeploymentResponse> DeployArtifactsAsync(
string siteId, DeployArtifactsCommand command, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, command);
return await GetActor().Ask<ArtifactDeploymentResponse>(
envelope, _options.ArtifactDeploymentTimeout, cancellationToken);
}
// ── Pattern 4: Integration Routing ──
/// <summary>
/// Routes an integration call to a site.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="request">The integration call request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The integration call response.</returns>
public async Task<IntegrationCallResponse> RouteIntegrationCallAsync(
string siteId, IntegrationCallRequest request, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, request);
return await GetActor().Ask<IntegrationCallResponse>(
envelope, _options.IntegrationTimeout, cancellationToken);
}
// ── Pattern 5: Debug View ──
/// <summary>
/// Subscribes to debug view events from a site.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="request">The debug view subscription request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>A snapshot of the debug view.</returns>
public async Task<DebugViewSnapshot> SubscribeDebugViewAsync(
string siteId, SubscribeDebugViewRequest request, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, request);
return await GetActor().Ask<DebugViewSnapshot>(
envelope, _options.DebugViewTimeout, cancellationToken);
}
/// <summary>
/// Unsubscribes from debug view events for a site.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="request">The debug view unsubscription request.</param>
public void UnsubscribeDebugView(string siteId, UnsubscribeDebugViewRequest request)
{
// Tell (fire-and-forget) — no response expected
GetActor().Tell(new SiteEnvelope(siteId, request));
}
// ── Pattern 6a: Debug Snapshot (one-shot, request/response) ──
/// <summary>
/// Requests a snapshot of the debug view from a site.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="request">The debug snapshot request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>A snapshot of the debug view.</returns>
public async Task<DebugViewSnapshot> RequestDebugSnapshotAsync(
string siteId, DebugSnapshotRequest request, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, request);
return await GetActor().Ask<DebugViewSnapshot>(
envelope, _options.QueryTimeout, cancellationToken);
}
// ── Pattern 6b: Health Reporting (site→central, Tell) ──
// Health reports are received by central, not sent. No method needed here.
// ── Pattern 7: Remote Queries ──
/// <summary>
/// Queries event logs from a site.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="request">The event log query request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The event log query response.</returns>
public async Task<EventLogQueryResponse> QueryEventLogsAsync(
string siteId, EventLogQueryRequest request, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, request);
return await GetActor().Ask<EventLogQueryResponse>(
envelope, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Queries parked messages from a site.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="request">The parked message query request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The parked message query response.</returns>
public async Task<ParkedMessageQueryResponse> QueryParkedMessagesAsync(
string siteId, ParkedMessageQueryRequest request, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, request);
return await GetActor().Ask<ParkedMessageQueryResponse>(
envelope, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Retries a parked message at a site.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="request">The parked message retry request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The parked message retry response.</returns>
public async Task<ParkedMessageRetryResponse> RetryParkedMessageAsync(
string siteId, ParkedMessageRetryRequest request, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, request);
return await GetActor().Ask<ParkedMessageRetryResponse>(
envelope, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Discards a parked message at a site.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="request">The parked message discard request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The parked message discard response.</returns>
public async Task<ParkedMessageDiscardResponse> DiscardParkedMessageAsync(
string siteId, ParkedMessageDiscardRequest request, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, request);
return await GetActor().Ask<ParkedMessageDiscardResponse>(
envelope, _options.QueryTimeout, cancellationToken);
}
// ── OPC UA Tag Browser (interactive design-time query) ──
/// <summary>
/// Asks a site to enumerate the immediate children of an OPC UA node on the
/// live server backing the given data connection. Used by the CentralUI OPC UA
/// Tag Browser dialog. The Ask is bounded by <see cref="CommunicationOptions.QueryTimeout"/>
/// — interactive browse expansions are short, one-shot queries that share the
/// same latency budget as other remote queries (event logs, parked messages).
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="command">The OPC UA browse command.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The browse result (children + truncation flag + structured failure).</returns>
public virtual Task<BrowseNodeResult> BrowseNodeAsync(
string siteId,
BrowseNodeCommand command,
CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, command);
return GetActor().Ask<BrowseNodeResult>(
envelope, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Asks a site to run a read-only verification probe of an endpoint
/// configuration without persisting it: connect, capture the server certificate
/// if it is untrusted, then disconnect. Backs the CentralUI "Verify endpoint"
/// button. The Ask is bounded by <see cref="CommunicationOptions.QueryTimeout"/>
/// — verification is a short, one-shot query that shares the same latency budget
/// as the other interactive design-time queries (browse, search, test bindings).
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="command">The verify-endpoint command (connection name + protocol + serialized config).</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The verification result (success or classified failure, with a captured certificate when the failure is an untrusted certificate).</returns>
public virtual Task<VerifyEndpointResult> VerifyEndpointAsync(
string siteId,
VerifyEndpointCommand command,
CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, command);
return GetActor().Ask<VerifyEndpointResult>(
envelope, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Asks a site to run a bounded recursive search of the address space on the
/// live server backing the given data connection. The address-space analogue
/// of <see cref="BrowseNodeAsync"/> — used by the CentralUI OPC UA tag picker's
/// "find a tag" box. The Ask is bounded by <see cref="CommunicationOptions.QueryTimeout"/>,
/// the same latency budget as browse and the other interactive design-time
/// queries.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="command">The address-space search command (connection name + query + depth/result caps).</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The search result (matches + cap-reached flag + structured failure).</returns>
public virtual Task<SearchAddressSpaceResult> SearchAddressSpaceAsync(
string siteId,
SearchAddressSpaceCommand command,
CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, command);
return GetActor().Ask<SearchAddressSpaceResult>(
envelope, _options.QueryTimeout, cancellationToken);
}
// ── OPC UA Server-Certificate Trust (node-wide site PKI store) ──
/// <summary>
/// Asks the owning site to trust an OPC UA server certificate at every site
/// node. Backs the CentralUI cert-management Trust action. The site Deployment
/// Manager singleton broadcasts the captured DER bytes to every site node's
/// <c>CertStoreActor</c>; the result reflects whether every reachable node
/// acked. The Ask is bounded by <see cref="CommunicationOptions.QueryTimeout"/>,
/// mirroring <see cref="BrowseNodeAsync"/> and the other interactive
/// design-time site queries.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="command">The trust-server-cert command (connection name + DER + thumbprint).</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The cert-trust result (per-node aggregate success + first error).</returns>
public virtual Task<CertTrustResult> TrustServerCertAsync(
string siteId,
TrustServerCertCommand command,
CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, command);
return GetActor().Ask<CertTrustResult>(
envelope, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Asks the owning site to list the certificates in its trusted-peer and
/// rejected PKI stores. Backs the CentralUI cert-management list page. Answered
/// by the site Deployment Manager singleton from its own node (the store is
/// node-wide per site node). The Ask is bounded by
/// <see cref="CommunicationOptions.QueryTimeout"/>.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="command">The list-server-certs command.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The cert-trust result carrying the listed certificates on success.</returns>
public virtual Task<CertTrustResult> ListServerCertsAsync(
string siteId,
ListServerCertsCommand command,
CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, command);
return GetActor().Ask<CertTrustResult>(
envelope, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Asks the owning site to remove a previously-trusted OPC UA server
/// certificate from every site node, identified by thumbprint. Backs the
/// CentralUI cert-management Remove action. The site Deployment Manager
/// singleton broadcasts the removal to every site node's <c>CertStoreActor</c>.
/// The Ask is bounded by <see cref="CommunicationOptions.QueryTimeout"/>.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="command">The remove-server-cert command (thumbprint).</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The cert-trust result (per-node aggregate success + first error).</returns>
public virtual Task<CertTrustResult> RemoveServerCertAsync(
string siteId,
RemoveServerCertCommand command,
CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, command);
return GetActor().Ask<CertTrustResult>(
envelope, _options.QueryTimeout, cancellationToken);
}
// ── Test Bindings (one-shot live read of bound tags) ──
/// <summary>
/// Asks a site to read the current value of one or more tags on the live
/// server backing the given data connection. Used by the CentralUI "Test
/// Bindings" dialog on the Configure Instance page. The Ask is bounded by
/// <see cref="CommunicationOptions.QueryTimeout"/> — same latency budget
/// as <see cref="BrowseNodeAsync"/> (both are interactive one-shot
/// design-time queries).
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="command">The read-tag-values command (connection name + tag paths).</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The read result — per-tag outcomes plus an optional connection-level failure.</returns>
public Task<ReadTagValuesResult> ReadTagValuesAsync(
string siteId,
ReadTagValuesCommand command,
CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, command);
return GetActor().Ask<ReadTagValuesResult>(
envelope, _options.QueryTimeout, cancellationToken);
}
// ── Secured Write Relay (approve → site MxGateway write) ──
/// <summary>
/// Relays a single tag write to the site's data connection and awaits the
/// outcome. Used by the secured-write (two-person) approve flow: once a
/// Verifier's approval wins the compare-and-swap, the central ManagementActor
/// calls this to execute the write against the site's MxGateway connection.
/// The request is the existing <see cref="WriteTagRequest"/>, already handled
/// site-side by <c>DataConnectionActor</c> (routed to the MxGateway adapter) —
/// no site-side change is required. The Ask is bounded by
/// <see cref="CommunicationOptions.QueryTimeout"/>, mirroring
/// <see cref="BrowseNodeAsync"/> and the other one-shot site queries.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="request">The tag write request (correlation id + connection + tag + value + timestamp).</param>
/// <param name="ct">Cancellation token.</param>
/// <returns>The write response (success flag + optional error message).</returns>
public virtual Task<WriteTagResponse> WriteTagAsync(
string siteId,
WriteTagRequest request,
CancellationToken ct = default)
{
var envelope = new SiteEnvelope(siteId, request);
return GetActor().Ask<WriteTagResponse>(
envelope, _options.QueryTimeout, ct);
}
// ── Pattern 8: Heartbeat (site→central, Tell) ──
// Heartbeats are received by central, not sent. No method needed here.
// ── Inbound API Cross-Site Routing ──
/// <summary>
/// Routes an inbound API call to a site.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="request">The call route request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The call route response.</returns>
public async Task<RouteToCallResponse> RouteToCallAsync(
string siteId, RouteToCallRequest request, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, request);
return await GetActor().Ask<RouteToCallResponse>(
envelope, _options.IntegrationTimeout, cancellationToken);
}
/// <summary>
/// Routes an inbound API get-attributes request to a site.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="request">The get-attributes route request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The get-attributes route response.</returns>
public async Task<RouteToGetAttributesResponse> RouteToGetAttributesAsync(
string siteId, RouteToGetAttributesRequest request, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, request);
return await GetActor().Ask<RouteToGetAttributesResponse>(
envelope, _options.IntegrationTimeout, cancellationToken);
}
/// <summary>
/// Routes an inbound API set-attributes request to a site.
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="request">The set-attributes route request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The set-attributes route response.</returns>
public async Task<RouteToSetAttributesResponse> RouteToSetAttributesAsync(
string siteId, RouteToSetAttributesRequest request, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, request);
return await GetActor().Ask<RouteToSetAttributesResponse>(
envelope, _options.IntegrationTimeout, cancellationToken);
}
/// <summary>
/// Routes an inbound API wait-for-attribute request to a site (spec §6).
/// </summary>
/// <param name="siteId">The target site identifier.</param>
/// <param name="request">The wait-for-attribute route request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The wait-for-attribute route response.</returns>
public async Task<RouteToWaitForAttributeResponse> RouteToWaitForAttributeAsync(
string siteId, RouteToWaitForAttributeRequest request, CancellationToken cancellationToken = default)
{
var envelope = new SiteEnvelope(siteId, request);
// A wait legitimately blocks up to request.Timeout on the site, so the cluster
// Ask must be bounded by the WAIT deadline (plus integration-timeout slack for
// the round trip), not the generic IntegrationTimeout used by the other routes.
var askTimeout = request.Timeout + _options.IntegrationTimeout;
return await GetActor().Ask<RouteToWaitForAttributeResponse>(
envelope, askTimeout, cancellationToken);
}
// ── Notification Outbox (central-local actor — Asked directly, no SiteEnvelope) ──
/// <summary>
/// Queries the notification outbox.
/// </summary>
/// <param name="request">The notification outbox query request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The notification outbox query response.</returns>
public async Task<NotificationOutboxQueryResponse> QueryNotificationOutboxAsync(
NotificationOutboxQueryRequest request, CancellationToken cancellationToken = default)
{
return await GetNotificationOutbox().Ask<NotificationOutboxQueryResponse>(
request, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Retries a notification from the outbox.
/// </summary>
/// <param name="request">The retry notification request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The retry notification response.</returns>
public async Task<RetryNotificationResponse> RetryNotificationAsync(
RetryNotificationRequest request, CancellationToken cancellationToken = default)
{
return await GetNotificationOutbox().Ask<RetryNotificationResponse>(
request, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Discards a notification from the outbox.
/// </summary>
/// <param name="request">The discard notification request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The discard notification response.</returns>
public async Task<DiscardNotificationResponse> DiscardNotificationAsync(
DiscardNotificationRequest request, CancellationToken cancellationToken = default)
{
return await GetNotificationOutbox().Ask<DiscardNotificationResponse>(
request, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Gets details about a specific notification.
/// </summary>
/// <param name="request">The notification detail request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The notification detail response.</returns>
public async Task<NotificationDetailResponse> GetNotificationDetailAsync(
NotificationDetailRequest request, CancellationToken cancellationToken = default)
{
return await GetNotificationOutbox().Ask<NotificationDetailResponse>(
request, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Gets KPI metrics for the notification outbox.
/// </summary>
/// <param name="request">The notification KPI request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The notification KPI response.</returns>
public async Task<NotificationKpiResponse> GetNotificationKpisAsync(
NotificationKpiRequest request, CancellationToken cancellationToken = default)
{
return await GetNotificationOutbox().Ask<NotificationKpiResponse>(
request, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Gets per-site KPI metrics for the notification outbox.
/// </summary>
/// <param name="request">The per-site notification KPI request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The per-site notification KPI response.</returns>
public async Task<PerSiteNotificationKpiResponse> GetPerSiteNotificationKpisAsync(
PerSiteNotificationKpiRequest request, CancellationToken cancellationToken = default)
{
return await GetNotificationOutbox().Ask<PerSiteNotificationKpiResponse>(
request, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Gets per-node KPI metrics for the notification outbox.
/// Groups by <c>SourceNode</c> (e.g. <c>node-a</c>/<c>node-b</c>); rows with
/// a <c>NULL</c> node are omitted. Additive alongside
/// <see cref="GetPerSiteNotificationKpisAsync"/>.
/// </summary>
/// <param name="request">The per-node notification KPI request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The per-node notification KPI response.</returns>
public async Task<PerNodeNotificationKpiResponse> GetPerNodeNotificationKpisAsync(
PerNodeNotificationKpiRequest request, CancellationToken cancellationToken = default)
{
return await GetNotificationOutbox().Ask<PerNodeNotificationKpiResponse>(
request, _options.QueryTimeout, cancellationToken);
}
// ── Site Call Audit (central-local actor — Asked directly, no SiteEnvelope) ──
/// <summary>
/// Queries site call audit records.
/// </summary>
/// <param name="request">The site call query request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The site call query response.</returns>
public async Task<SiteCallQueryResponse> QuerySiteCallsAsync(
SiteCallQueryRequest request, CancellationToken cancellationToken = default)
{
return await GetSiteCallAudit().Ask<SiteCallQueryResponse>(
request, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Gets details about a specific site call.
/// </summary>
/// <param name="request">The site call detail request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The site call detail response.</returns>
public async Task<SiteCallDetailResponse> GetSiteCallDetailAsync(
SiteCallDetailRequest request, CancellationToken cancellationToken = default)
{
return await GetSiteCallAudit().Ask<SiteCallDetailResponse>(
request, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Gets KPI metrics for site calls.
/// </summary>
/// <param name="request">The site call KPI request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The site call KPI response.</returns>
public async Task<SiteCallKpiResponse> GetSiteCallKpisAsync(
SiteCallKpiRequest request, CancellationToken cancellationToken = default)
{
return await GetSiteCallAudit().Ask<SiteCallKpiResponse>(
request, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Gets per-site KPI metrics for site calls.
/// </summary>
/// <param name="request">The per-site site call KPI request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The per-site site call KPI response.</returns>
public async Task<PerSiteSiteCallKpiResponse> GetPerSiteSiteCallKpisAsync(
PerSiteSiteCallKpiRequest request, CancellationToken cancellationToken = default)
{
return await GetSiteCallAudit().Ask<PerSiteSiteCallKpiResponse>(
request, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Gets per-node KPI metrics for site calls. Groups by <c>SourceNode</c>
/// (e.g. <c>node-a</c>/<c>node-b</c>); rows with a <c>NULL</c> node are
/// omitted. Additive alongside <see cref="GetPerSiteSiteCallKpisAsync"/>.
/// </summary>
/// <param name="request">The per-node site call KPI request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The per-node site call KPI response.</returns>
public async Task<PerNodeSiteCallKpiResponse> GetPerNodeSiteCallKpisAsync(
PerNodeSiteCallKpiRequest request, CancellationToken cancellationToken = default)
{
return await GetSiteCallAudit().Ask<PerNodeSiteCallKpiResponse>(
request, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Relays an operator Retry of a parked cached call to its
/// owning site. The <c>SiteCallAuditActor</c> is Asked directly (it is
/// central-local); it in turn relays a <c>RetryParkedOperation</c> to the
/// owning site and replies a <see cref="RetrySiteCallResponse"/> carrying a
/// distinct site-unreachable outcome. Central never mutates the central
/// <c>SiteCalls</c> mirror row.
/// <para>
/// This outer Ask uses <see cref="CommunicationOptions.QueryTimeout"/>
/// (default 30s), which must outlive the inner site relay Ask the
/// <c>SiteCallAuditActor</c> issues with <c>SiteCallAuditOptions.RelayTimeout</c>
/// (default 10s). The inner relay must time out first so its distinct
/// <c>SiteUnreachable</c> outcome reaches us; were this outer Ask to expire
/// first, that outcome would be lost to a generic Ask-timeout exception.
/// </para>
/// </summary>
/// <param name="request">The retry site call request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The retry site call response.</returns>
public async Task<RetrySiteCallResponse> RetrySiteCallAsync(
RetrySiteCallRequest request, CancellationToken cancellationToken = default)
{
return await GetSiteCallAudit().Ask<RetrySiteCallResponse>(
request, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Relays an operator Discard of a parked cached call to its
/// owning site. See <see cref="RetrySiteCallAsync"/> for the routing and
/// source-of-truth rationale.
/// </summary>
/// <param name="request">The discard site call request.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The discard site call response.</returns>
public async Task<DiscardSiteCallResponse> DiscardSiteCallAsync(
DiscardSiteCallRequest request, CancellationToken cancellationToken = default)
{
return await GetSiteCallAudit().Ask<DiscardSiteCallResponse>(
request, _options.QueryTimeout, cancellationToken);
}
/// <summary>
/// Asks a site to gracefully fail over its own two-node pair (Task 10). Central and each
/// site are SEPARATE Akka clusters, so central cannot act on site membership directly —
/// the site's own <c>SiteCommunicationActor</c> performs the <c>Cluster.Leave</c> against
/// its site-specific role and acks the outcome.
/// <para>
/// A site running a binary older than the <see cref="TriggerSiteFailover"/> contract has no
/// handler for it, so the message dead-letters and this Ask times out. That surfaces to the
/// operator as "site did not respond", which is the honest outcome — an old site genuinely
/// cannot honour the request.
/// </para>
/// </summary>
/// <param name="siteId">Target site.</param>
/// <param name="correlationId">Correlation id echoed on the ack.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The site's ack.</returns>
public async Task<SiteFailoverAck> TriggerSiteFailoverAsync(
string siteId, string correlationId, CancellationToken cancellationToken = default)
{
_logger.LogWarning(
"Relaying TriggerSiteFailover to site {SiteId}, correlationId={CorrelationId}",
siteId, correlationId);
var envelope = new SiteEnvelope(siteId, new TriggerSiteFailover(correlationId, siteId));
return await GetActor().Ask<SiteFailoverAck>(
envelope, _options.QueryTimeout, cancellationToken);
}
}
/// <summary>
/// Envelope that wraps any message with a target site ID for routing.
/// Used by CentralCommunicationActor to resolve the site actor path.
/// </summary>
public record SiteEnvelope(string SiteId, object Message);