fix(host): NodeOptions-bound ports, central ESG cache TTL, hosted warm-up wiring; document AllowSingleNodeCluster as validation-only (review 2026-10-21 08/O5, 08/O4, 05/U3)

This commit is contained in:
Joseph Doherty
2026-10-22 08:41:55 -04:00
parent fa73ddb453
commit 1e3765ba6e
4 changed files with 40 additions and 16 deletions
@@ -197,6 +197,8 @@ Differences from central failover, all deliberate:
`akka.cluster.min-nr-of-members` must be set to **1**. After failover, only one node is running. If set to 2, the surviving node waits for a second member before allowing the Cluster Singleton (Site Runtime Deployment Manager) to start — blocking all data collection and script execution indefinitely.
`ScadaBridge:Cluster:AllowSingleNodeCluster` is **validation-only**: it merely relaxes the seed-count check (one seed instead of two) in `ClusterOptionsValidator`. It changes no runtime behaviour — not `min-nr-of-members` (always 1), not downing, not the bootstrap guard; a lone node forms its cluster because every node lists itself first in its seed list. The name is retained to avoid breaking deployed configs.
## Failure Detection Timing
Configurable defaults for heartbeat and failure detection:
+7
View File
@@ -345,3 +345,10 @@ built)* section for where the shipped code deviates from the original design).
- **CentralUI / InboundAPI**: The Host maps their web endpoints into the ASP.NET Core pipeline on central nodes.
- **ManagementService**: The Host registers the ManagementActor on central nodes; the CLI reaches it over the HTTP `/management` endpoints (in-process via `ManagementActorHolder`), not via any cluster transport. No ClusterClientReceptionist is configured (removed in Phase 4).
- **HealthMonitoring**: The Host's startup validation and logging configuration provide the foundation for health reporting.
## Review 2026-10-21 notes (Host edge)
- **Port options come from the bound `NodeOptions`.** `Program.cs` binds `ScadaBridge:Node` once and reads `GrpcPort` / `CentralGrpcPort` / `MetricsPort` from it; the 8083/8084 defaults live only in `NodeOptions`. Central's dedicated metrics listener remains an explicit *presence* check on the raw key (a defaulted read would bind a new port on every existing central node).
- **Inbound method warm-up is a hosted service** (`InboundMethodWarmupService`, registered after `AkkaHostedService`), not a pre-`RunAsync` loop — see Component-InboundAPI.md.
- **`SitePskProvider` distinguishes a secret-store fault from a missing secret.** A store fault logs a Warning with the exception and throws `SecretStoreUnavailableException` (not cached); only a genuinely absent/empty/tombstoned secret produces the "seed `SB-GRPC-PSK-{site}`" message.
- Central registers `ExternalSystemGatewayOptions.DefinitionCacheTtl = 10 s` ahead of `AddExternalSystemGateway()` so an explicit config value wins; sites leave it off.
@@ -112,6 +112,13 @@ public class ClusterOptions
/// node (normally ≥2 are required so either node can start first). Without this
/// flag a single-node install must list a phantom second seed, which the node
/// dials forever — log noise and a defeated validation intent (review 01).
/// <para>
/// <b>Validation-only.</b> This flag changes nothing at runtime: it only relaxes
/// <c>ClusterOptionsValidator</c>'s seed-count check. It does not touch
/// <c>min-nr-of-members</c>, the downing strategy or the bootstrap guard — a lone node
/// forms its cluster because it lists ITSELF first (self-first seed ordering), not because
/// of this flag. The name is kept (renaming would break deployed configs) (review 2026-10-21 08/O4).
/// </para>
/// </summary>
public bool AllowSingleNodeCluster { get; set; }
+24 -16
View File
@@ -131,6 +131,11 @@ await using (var secretsProvider = expanderServices.BuildServiceProvider())
// Full startup validation — fail fast before any DI or actor system setup
StartupValidator.Validate(configuration);
// Bound node options: the single home of port defaults (NodeOptions), so the Kestrel setup below
// never repeats literal 8083/8084 defaults at the call sites (review 2026-10-21 08/O5).
var nodeOptions = configuration.GetSection("ScadaBridge:Node").Get<ZB.MOM.WW.ScadaBridge.Host.NodeOptions>()
?? new ZB.MOM.WW.ScadaBridge.Host.NodeOptions();
// Read node options for Serilog enrichment
var nodeRole = configuration["ScadaBridge:Node:Role"]!;
var nodeHostname = configuration["ScadaBridge:Node:NodeHostname"] ?? "unknown";
@@ -188,7 +193,7 @@ try
// The site branch has the same ConfigureKestrel shape but no ASPNETCORE_URLS surface
// to lose (it binds every port it needs — gRPC + metrics — explicitly). Central must
// therefore RE-BIND its HTTP port(s) here alongside the gRPC port.
var centralGrpcPort = configuration.GetValue<int>("ScadaBridge:Node:CentralGrpcPort", 8083);
var centralGrpcPort = nodeOptions.CentralGrpcPort;
// "urls" is WebHostDefaults.ServerUrlsKey — the host setting ASPNETCORE_URLS/--urls
// populate. Read it as a literal so this needs no extra Hosting using.
var httpUrls = configuration["ASPNETCORE_URLS"]
@@ -296,6 +301,12 @@ try
builder.Services.AddHealthMonitoring();
builder.Services.AddCentralHealthAggregation();
// Central resolves definitions from SQL Server on every call; cache them briefly
// (review 2026-10-21 05/P2). Registered BEFORE AddExternalSystemGateway so an explicit
// ScadaBridge:ExternalSystemGateway:DefinitionCacheTtl binding still wins. Sites keep it
// off — their repository is already a deploy-invalidated snapshot.
builder.Services.Configure<ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.ExternalSystemGatewayOptions>(
o => o.DefinitionCacheTtl = TimeSpan.FromSeconds(10));
builder.Services.AddExternalSystemGateway();
builder.Services.AddNotificationService();
@@ -593,6 +604,12 @@ try
// routes against agree on which node is active.
builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.InboundAPI.IActiveNodeGate, ActiveNodeGate>();
// Warm the inbound method compile cache in the BACKGROUND once the host is running and
// this node is the active one (review 2026-10-21 05/U3). Registered after
// AkkaHostedService so it never sits in front of the cluster join; the eager compile that
// used to block here before app.RunAsync() is gone.
builder.Services.AddHostedService<ZB.MOM.WW.ScadaBridge.InboundAPI.InboundMethodWarmupService>();
// Admin-triggered manual failover of the central pair (Health page control,
// decision 2026-07-22). Central-only: the seam is declared in CentralUI so that
// project stays Akka-free, and only this branch has a cluster to act on.
@@ -745,17 +762,8 @@ try
app.MapDeploymentConfigAPI();
app.MapHub<ZB.MOM.WW.ScadaBridge.ManagementService.DebugStreamHub>("/hubs/debug-stream");
// Compile and register all Inbound API method scripts at startup
using (var scope = app.Services.CreateScope())
{
var apiRepo = scope.ServiceProvider.GetRequiredService<ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories.IInboundApiRepository>();
var executor = app.Services.GetRequiredService<ZB.MOM.WW.ScadaBridge.InboundAPI.InboundScriptExecutor>();
var methods = await apiRepo.GetAllApiMethodsAsync();
// Parallel: CompileAndRegister is safe for concurrent callers (ConcurrentDictionary
// caches, no shared mutable compile state); serial Roslyn compiles stretch the
// failover-recovery window at hundreds of methods.
Parallel.ForEach(methods, method => executor.CompileAndRegister(method));
}
// Inbound API method scripts are pre-compiled by InboundMethodWarmupService after startup
// (background, active node only); the per-request lazy compile guarantees correctness.
await app.RunAsync();
}
@@ -778,15 +786,15 @@ try
// Windows Service support (no-op when not running as a Windows Service)
builder.Host.UseWindowsService();
// Read GrpcPort from config (NodeOptions already has default 8083)
var grpcPort = configuration.GetValue<int>("ScadaBridge:Node:GrpcPort", 8083);
// GrpcPort from the bound NodeOptions (default 8083)
var grpcPort = nodeOptions.GrpcPort;
// Read MetricsPort from config (NodeOptions already has default 8084).
// MetricsPort from the bound NodeOptions (default 8084).
// Separate HTTP/1.1 listener so a standard HTTP/1.1 Prometheus scraper can
// reach /metrics; the gRPC port stays HTTP/2-only below. The default is
// 8084 — distinct from RemotingPort (8082, Akka) and GrpcPort (8083) so the
// metrics listener never collides with the Akka remoting port on site nodes.
var metricsPort = configuration.GetValue<int>("ScadaBridge:Node:MetricsPort", 8084);
var metricsPort = nodeOptions.MetricsPort;
// Configure Kestrel for HTTP/2 only on the gRPC port (plus the h2 keepalive that
// makes a half-open inbound stream detectable) — see SiteKestrelSetup.