feat(comm): Phase 4 — delete Akka ClusterClient site↔central transport, gRPC-only
ClusterClient→gRPC migration Phase 4 (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Phases 2/3 proved both directions on gRPC; this removes the Akka transport underneath. Deleted: - AkkaCentralTransport, AkkaSiteTransport (+ their dedicated tests) - ISiteClientFactory + DefaultSiteClientFactory; CentralCommunicationActor legacy ctor + SelectTransport (Host now builds GrpcSiteTransport and injects it) - ClusterClient creation + both ClusterClientReceptionist.RegisterService calls in AkkaHostedService; the RegisterCentralClient message + receive block - CommunicationOptions.CentralContactPoints; the CentralTransport/SiteTransport coexistence flags; the CentralTransportMode/SiteTransportKind enums gRPC is now the only site↔central transport (site→central CentralControlService via GrpcCentralTransport; central→site SiteCommandService via GrpcSiteTransport), both built unconditionally by the Host. NoOpCentralTransport is the fail-loud null-default so TestKit command-dispatch suites still construct the site actor without a wired transport; production always injects GrpcCentralTransport. Config: CentralGrpcEndpoints is now unconditional — CommunicationOptionsValidator rejects blank entries (role-agnostic), and StartupValidator requires a Site node to list >=1 endpoint (fail-fast, mirrors GrpcPsk). Rig configs moved CentralContactPoints -> CentralGrpcEndpoints (docker x6, docker-env2 x2, Host default, deploy/wonder-app-vd03). Kept Akka.Cluster.Tools (ClusterSingleton still used). Tests: build 0/0; Communication.Tests 640, Host.Tests 421 green. Removed the ClusterClient.Send per-site-routing tests (covered by the transport suites), swapped the ISiteClientFactory-based ctors to a substitute ISiteCommandTransport, converted the audit-push integration relay to an in-process bridge transport. Docs: Component-Communication/Host/StoreAndForward, components/Communication, topology-guide, grpc_streams (SUPERSEDED note), the frame-size known-issue (retired amendment), and CLAUDE.md transport decisions. Not included: the dead IntegrationCallRequest path (#32) is a separate user-owned behavioral decision — SiteEnvelope routing is transport-agnostic so it still compiles.
This commit is contained in:
@@ -1,7 +1,5 @@
|
||||
using System.Collections.Immutable;
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Cluster.Tools.Client;
|
||||
using Akka.Cluster.Tools.Singleton;
|
||||
using Akka.Configuration;
|
||||
using Microsoft.Extensions.Options;
|
||||
@@ -428,15 +426,21 @@ akka {{
|
||||
var centralHealthCollector = _serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.HealthMonitoring.ISiteHealthCollector>();
|
||||
centralHealthCollector?.SetNodeHostname(_nodeOptions.NodeHostname);
|
||||
|
||||
var siteClientFactory = new DefaultSiteClientFactory();
|
||||
// Central→site command transport: gRPC SiteCommandService. This is the ONLY transport since
|
||||
// the ClusterClient→gRPC migration's Phase 4 removed the Akka path; the actor no longer
|
||||
// chooses a transport, the Host builds it. Constructed once from the DI
|
||||
// SitePairChannelProvider and shared across CentralCommunicationActor incarnations (it holds
|
||||
// only the shared provider + options); the actor reconciles its per-site channel pairs from
|
||||
// the DB refresh loop.
|
||||
var siteCommandTransport = new GrpcSiteTransport(
|
||||
_serviceProvider.GetRequiredService<SitePairChannelProvider>(),
|
||||
_communicationOptions,
|
||||
_serviceProvider.GetRequiredService<ILoggerFactory>().CreateLogger<GrpcSiteTransport>());
|
||||
_logger.LogInformation("central→site command transport: gRPC (SiteCommandService)");
|
||||
var centralCommActor = _actorSystem!.ActorOf(
|
||||
Props.Create(() => new CentralCommunicationActor(_serviceProvider, siteClientFactory)),
|
||||
Props.Create(() => new CentralCommunicationActor(_serviceProvider, siteCommandTransport)),
|
||||
"central-communication");
|
||||
|
||||
// Register CentralCommunicationActor with ClusterClientReceptionist so site ClusterClients can reach it
|
||||
ClusterClientReceptionist.Get(_actorSystem).RegisterService(centralCommActor);
|
||||
_logger.LogInformation("CentralCommunicationActor registered with ClusterClientReceptionist");
|
||||
|
||||
// Hand the same actor to the central-hosted gRPC control plane (T1A.2) and open its
|
||||
// readiness gate — the gRPC face Asks this exact actor, so both transports resolve to
|
||||
// one handler implementation. Mirrors SiteStreamGrpcServer.SetReady on the site side:
|
||||
@@ -833,35 +837,25 @@ akka {{
|
||||
_logger, role: siteRole);
|
||||
var dmProxy = dm.Proxy;
|
||||
|
||||
// Select the site→central transport behind the coexistence flag (default Akka
|
||||
// ClusterClient). When gRPC is chosen the site dials CentralControlService directly with
|
||||
// a sticky-failover channel pair, presenting its own preshared key; the ClusterClient
|
||||
// below is then not created at all.
|
||||
ICentralTransport? centralTransport = null;
|
||||
if (_communicationOptions.CentralTransport == CentralTransportMode.Grpc)
|
||||
{
|
||||
var loggerFactory = _serviceProvider.GetRequiredService<ILoggerFactory>();
|
||||
var channelProvider = new CentralChannelProvider(
|
||||
_communicationOptions.CentralGrpcEndpoints,
|
||||
new StaticSitePskProvider(_communicationOptions.GrpcPsk),
|
||||
_nodeOptions.SiteId!,
|
||||
_communicationOptions,
|
||||
loggerFactory.CreateLogger<CentralChannelProvider>());
|
||||
_trackedDisposables.Add(channelProvider);
|
||||
centralTransport = new GrpcCentralTransport(
|
||||
channelProvider,
|
||||
_communicationOptions,
|
||||
loggerFactory.CreateLogger<GrpcCentralTransport>());
|
||||
_logger.LogInformation(
|
||||
"Site→central transport: gRPC to {Count} central endpoint(s) for site {SiteId}",
|
||||
_communicationOptions.CentralGrpcEndpoints.Count, _nodeOptions.SiteId);
|
||||
}
|
||||
else
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"Site→central transport: Akka ClusterClient (default) for site {SiteId}",
|
||||
_nodeOptions.SiteId);
|
||||
}
|
||||
// Site→central transport: a gRPC dial of CentralControlService with a sticky
|
||||
// central-a→central-b channel pair, presenting this site's preshared key. This is the ONLY
|
||||
// transport since the ClusterClient→gRPC migration's Phase 4 removed the Akka path;
|
||||
// StartupValidator guarantees a Site node lists at least one CentralGrpcEndpoint.
|
||||
var loggerFactory = _serviceProvider.GetRequiredService<ILoggerFactory>();
|
||||
var channelProvider = new CentralChannelProvider(
|
||||
_communicationOptions.CentralGrpcEndpoints,
|
||||
new StaticSitePskProvider(_communicationOptions.GrpcPsk),
|
||||
_nodeOptions.SiteId!,
|
||||
_communicationOptions,
|
||||
loggerFactory.CreateLogger<CentralChannelProvider>());
|
||||
_trackedDisposables.Add(channelProvider);
|
||||
ICentralTransport centralTransport = new GrpcCentralTransport(
|
||||
channelProvider,
|
||||
_communicationOptions,
|
||||
loggerFactory.CreateLogger<GrpcCentralTransport>());
|
||||
_logger.LogInformation(
|
||||
"Site→central transport: gRPC to {Count} central endpoint(s) for site {SiteId}",
|
||||
_communicationOptions.CentralGrpcEndpoints.Count, _nodeOptions.SiteId);
|
||||
|
||||
// The ONE routing table for central→site commands, shared by the Akka
|
||||
// SiteCommunicationActor (below) and the gRPC SiteCommandGrpcService (SetReady at the end
|
||||
@@ -997,36 +991,10 @@ akka {{
|
||||
siteCommActor.Tell(new RegisterLocalHandler(LocalHandlerType.ParkedMessages, parkedMessageHandler));
|
||||
}
|
||||
|
||||
// Register SiteCommunicationActor with ClusterClientReceptionist so central ClusterClients can reach it
|
||||
ClusterClientReceptionist.Get(_actorSystem).RegisterService(siteCommActor);
|
||||
|
||||
_logger.LogInformation(
|
||||
"Site actors registered. DeploymentManager singleton scoped to role={SiteRole}, SiteCommunicationActor created.",
|
||||
siteRole);
|
||||
|
||||
// Create ClusterClient to central if contact points are configured — but only on the Akka
|
||||
// transport. On the gRPC transport the SiteCommunicationActor already holds a
|
||||
// GrpcCentralTransport and never receives RegisterCentralClient, so a ClusterClient here
|
||||
// would be dead weight (and keep an unwanted cross-cluster Akka association alive).
|
||||
if (_communicationOptions.CentralTransport == CentralTransportMode.Akka
|
||||
&& _communicationOptions.CentralContactPoints.Count > 0)
|
||||
{
|
||||
var contacts = _communicationOptions.CentralContactPoints
|
||||
.Select(cp => ActorPath.Parse($"{cp}/system/receptionist"))
|
||||
.ToImmutableHashSet();
|
||||
var clientSettings = ClusterClientSettings.Create(_actorSystem)
|
||||
.WithInitialContacts(contacts);
|
||||
var centralClient = _actorSystem.ActorOf(
|
||||
ClusterClient.Props(clientSettings), "central-cluster-client");
|
||||
|
||||
var siteCommSelection = _actorSystem.ActorSelection("/user/site-communication");
|
||||
siteCommSelection.Tell(new RegisterCentralClient(centralClient));
|
||||
|
||||
_logger.LogInformation(
|
||||
"Created ClusterClient to central with {Count} contact point(s) for site {SiteId}",
|
||||
contacts.Count, _nodeOptions.SiteId);
|
||||
}
|
||||
|
||||
// Per-node startup reconciliation. Created on EVERY site node (NOT a
|
||||
// singleton) so a standby that was DOWN during a deploy self-heals on its next
|
||||
// restart: it reports its local deployed inventory to central via the
|
||||
|
||||
@@ -143,6 +143,22 @@ public static class StartupValidator
|
||||
+ "production as ${secret:SB-GRPC-PSK-<siteId>}) and under the secret "
|
||||
+ "name SB-GRPC-PSK-<siteId> in central's secret store");
|
||||
|
||||
// gRPC (CentralControlService) is the only site→central transport after the
|
||||
// ClusterClient→gRPC migration's Phase 4 — the Akka ClusterClient path and its
|
||||
// CentralContactPoints option are gone. A site with no central gRPC endpoint has
|
||||
// nothing to dial: heartbeats, health reports, notification forwards and audit
|
||||
// ingest all silently fail. The shared CommunicationOptionsValidator only rejects
|
||||
// BLANK entries (it is role-agnostic, and central nodes legitimately leave the list
|
||||
// empty), so the "a Site must have at least one" rule lives here, where the role is
|
||||
// known. The predicate reads index :0 directly, so its non-empty presence proves the
|
||||
// list has a usable first endpoint.
|
||||
p.Require("ScadaBridge:Communication:CentralGrpcEndpoints:0",
|
||||
value => !string.IsNullOrWhiteSpace(value),
|
||||
"is required for Site nodes: gRPC (CentralControlService) is the only "
|
||||
+ "site→central transport, so each site must list at least one central gRPC "
|
||||
+ "endpoint under ScadaBridge:Communication:CentralGrpcEndpoints "
|
||||
+ "(e.g. http://scadabridge-central-a:8083). Central nodes leave it empty.");
|
||||
|
||||
// ScadaBridge:Database:SiteDbPath was required here until LocalDb
|
||||
// Phase 2. The site's tables now live in the consolidated LocalDb
|
||||
// database (LocalDb:Path, which SiteServiceRegistration requires),
|
||||
|
||||
@@ -44,9 +44,9 @@
|
||||
"Communication": {
|
||||
"_grpcPsk": "REQUIRED on Site nodes (StartupValidator fails the boot without it). The preshared key the gRPC control plane authenticates with: ControlPlaneAuthInterceptor is fail-closed, so an unset key refuses every SiteStream call — live subscriptions, audit pulls, cached-telemetry ingest — while the node still reports healthy. Supply it as ${secret:SB-GRPC-PSK-<siteId>} so the plaintext never sits in this file, and seed the SAME value on central: either as the secret SB-GRPC-PSK-<siteId> in its store, or as ScadaBridge:Communication:SitePsks:<siteId>. BOTH nodes of the pair carry the same key. Distinct from LocalDb:Replication:ApiKey, which authenticates the pair partner, not central — never share the two.",
|
||||
"GrpcPsk": "${secret:SB-GRPC-PSK-site-1}",
|
||||
"_centralContactPoints": "Host-016: each entry MUST be a central node's remoting endpoint, NOT this site's own remoting port. The single dev-loopback default below points only at central-a (localhost:8081). In a multi-central deployment add the second central node here (e.g. 'akka.tcp://scadabridge@central-b-host:8081') so ClusterClient can fail over when central-a is down. The previous template listed localhost:8082 as the second contact — that is THIS site's own RemotingPort and is a permanent failure in the initial-contact rotation.",
|
||||
"CentralContactPoints": [
|
||||
"akka.tcp://scadabridge@localhost:8081"
|
||||
"_centralGrpcEndpoints": "gRPC (CentralControlService) is the only site→central transport since the ClusterClient→gRPC migration's Phase 4. Each entry MUST be a central node's gRPC (h2c) endpoint on its CentralGrpcPort (default 8083) — NOT this site's own gRPC port, and NOT via Traefik (HTTP/1 only). The single dev-loopback default below points only at central-a (localhost:8083). In a multi-central deployment add the second central node here (e.g. 'http://central-b-host:8083') so the channel pair can fail over when central-a is down. StartupValidator requires a Site node to list at least one endpoint.",
|
||||
"CentralGrpcEndpoints": [
|
||||
"http://localhost:8083"
|
||||
],
|
||||
"DeploymentTimeout": "00:02:00",
|
||||
"LifecycleTimeout": "00:00:30",
|
||||
|
||||
Reference in New Issue
Block a user